Year-end notes on generative AI and work

What have we learned? What's next?

1. Five Lessons, Five Unknowns

2. Research Highlights

3. Ask MIT

Five Lessons, Five Unknowns

As a capstone to a year filled with new models, new use cases, and new data about how generative AI works in the real world, we are providing a summary of our notes and lessons from the last three months of meetings and research. First, a quick summary of what we have learned – and where there are still blindspots.

Lessons

1. ‘Sweet spot’ use cases are beginning to scale. There are key use cases like customer support, marketing, and software development where generative AI tools seem to offer consistent productivity benefits and have seen widespread adoption. However, it is unclear whether the productivity gains from these domains are generalizable to the economy more broadly.

2. Uses of generative AI vary at the individual level. Some use generative AI infrequently to perform specific tasks, whereas others use LLMs as a more consistent source of knowledge and input. Others opt not to use it at all, even in occupations where it can be highly beneficial. We saw this in our wide-ranging survey of how workers think about automation technologies. Organizations are beginning to adapt their AI applications to different user profiles, although it is not yet clear what determines how individuals engage with generative AI tools.

3. Success is measured in performance, not in usage. Previously, some organizations were focused on the scale and impact of their generative AI tools in terms of user adoption within their companies. The scale-up from a pilot to a full-scale deployment of a tool with many users was cast as a success. However, focus has shifted to how generative AI can augment workers to improve productivity and quality or expand capabilities. A user might employ generative AI infrequently, but with high impact each time. Similarly, only a small team might use generative AI, but the benefits of their usage might have outsized benefits.

4. Generative AI is not a free lunch. In multiple studies and cases, there are apparent tradeoffs with deploying AI. In a study of the impact of GitHub copilot, it appears that generative AI tools can boost programmer output, but can also increase the error rate. A new paper on the impact of AI in R&D settings suggests that the tools can improve researcher productivity, but might decrease their job satisfaction. In our working group discussions, we have considered the potential impact of generative AI adoption on skill atrophy and reduced collaboration within firms.

5. Building trust is a calibration problem – not an optimization function. Some organizations have focused on building trust with users so that they adopt and use generative AI tools. However, users can put too much trust in algorithms that might lead them astray. One of the challenges for responsible AI teams is to understand the appropriate level of trust should be afforded to generative AI tools, and what level of usage is most appropriate.

Unknowns

1. Is generative AI more helpful to experts or novices? Early on, there was evidence that LLM-based tools were skill leveling – in experiments in writing and customer service, they boosted the performance of less experienced and skilled individuals more than they did for experts. However, our case studies and new research on AI tools in R&D settings show that this is not consistently the case. In some user settings, experts are able to make more productive use of the generative AI tools. The open question is: under what conditions do these tools advantage experts, and when do they help novices?

2. Does training on AI tools help? There has been a flurry of interest in training individuals to use AI tools – particularly to understand when the uses of the tools are most appropriate and where they might go awry. But this study shows that even when users were given training on how to use generative AI models, they still suffered from the same mistakes as users who were directed to use the tools without training. Perhaps the next question is: what kinds of training might lead to responsible AI use?

3. How much better will these tools get? One of the early challenges for generative AI users is the rapid introduction of new models, which requires internal teams to test and adjust their applications based on the comparative advantages of each. It is unclear how long this will last. Will users be able to lock their applications into a model that they can trust will be relatively consistent – perhaps with some updates – over a year or more? The big question – relevant for AI companies and users alike – is where LLMs are at in the technology development curve: a mature state with diminishing returns to new investments in compute, or a state where big new investments in model development can lead to even bigger performance gains?

4. What will be the dominant design for generative AI interfaces? There are several potential use case patterns that we have recognized: general-purpose chat bots, task-specific AI agents, and applications built on another core technology with a natural language interface powered by generative AI (and many more). Some AI users have suggested that the real impact of these tools will be beyond “chatbots,” but it is unclear what that means (see below for a deeper discussion of OpenAI’s o models). Will the best tools be task- and industry-specific? Will general-purpose AI agents be useful in business settings? On these questions, experimentation still rules the day.

5. What are the most relevant skills to use AI well? We have heard many times this year something to the effect of: “AI won’t take your job, but someone who knows how to use AI will.” This might seem obvious: we should be teaching more about how generative AI works – and how to develop AI applications that suit our needs. Certainly these technical skills will be important for a narrow band of technical jobs, but for most workers, what is important to know about AI? Consider the population of workers that use software as part of their daily work, but have no software development skills. They may learn how to use software that becomes more and more powered by AI, but they might not learn much about the underlying principles behind the underlying technology. That could be ok. The success of those workers might still depend on problem-solving skills and domain expertise: whether they are expert in the business process for which they are responsible – and are able to interpret the information that the software provides.

Research Highlights

There continues to be a steady stream of interesting research that teaches us how organizations are using AI tools — and how workers engage with these technologies. Below are some (along with their abstracts) that have challenged our assumptions.

The Toner-Rodgers paper is the first account we have seen of the causal effect of AI tools on innovation within a firm. The productivity gains are consistent with other studies of the impact of AI on task performance, but there are two dimensions that make this paper interesting and surprising: the first is that the biggest gains were among senior R&D experts. And the second piece — the part of the paper that has lingered with us — is how R&D personnel did not seem to like what their jobs had become with the introduction of generative AI.

The Cui et al. paper on the effects of GitHub Copilot on software engineers is one of multiple papers to estimate the impact of AI tools on work tasks. What makes this paper interesting is how it suggests significant heterogeneity in how the effects of these tools play out within firms. The next step for this research, it seems, is to understand why some software engineers are able to extract more productivity benefits from these tools than others. That would begin to illuminate not just how AI is changing work — but what makes us good at our work in the first place.

The Kellogg et al. and Armstrong et al. papers emerge from some of the core research interests motivating our Working Group on Generative AI and the Work of the Future (they include the three working group co-leads as co-authors).

These papers grapple with the factors that make workers successful users of generative AI, as well as the type of workers who might be more open to using these tools as part of their jobs. Both papers start from a set of assumptions that we havfe seen organizations embrace: individual workers will use these tools differently based not only on their skills and roles, but also based on their personal experience and orientation toward work. Organizations should try to adapt their approach to implementing these tools based on the diversity of perspectives and attitudes in their workforce.

Artificial Intelligence, Scientific Discovery, and Product Innovation

Aidan Toner-Rodgers

This paper studies the impact of artificial intelligence on innovation, exploiting the randomized introduction of a new materials discovery technology to 1,018 scientists in the R&D lab of a large U.S. firm. AI-assisted researchers discover 44% more materials, resulting in a 39% increase in patent filings and a 17% rise in downstream product innovation. These compounds possess more novel chemical structures and lead to more radical inventions. However, the technology has strikingly disparate effects across the productivity distribution: while the bottom third of scientists see little benefit, the output of top researchers nearly doubles. Investigating the mechanisms behind these results, I show that AI automates 57% of "idea-generation" tasks, reallocating researchers to the new task of evaluating model-produced candidate materials. Top scientists leverage their domain knowledge to prioritize promising AI suggestions, while others waste significant resources testing false positives. Together, these findings demonstrate the potential of AI-augmented research and highlight the complementarity between algorithms and expertise in the innovative process. Survey evidence reveals that these gains come at a cost, however, as 82% of scientists report reduced satisfaction with their work due to decreased creativity and skill underutilization.

The Effects of Generative AI on High Skilled Work: Evidence from Three Field Experiments with Software Developers

Kevin Zheyuan Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng, and Tobias Salz

This study evaluates the impact of generative AI on software developer productivity by analyzing data from three randomized controlled trials conducted at Microsoft, Accenture, and an anonymous Fortune 100 electronics manufacturing company. These field experiments, which were run by the companies as part of their ordinary course of business, provided a randomly selected subset of developers with access to GitHub Copilot, an AI-based coding assistant that suggests intelligent code completions. Though each separate experiment is noisy, combined across all three experiments and 4,867 software developers, our analysis reveals a 26.08% increase (SE: 10.3%) in the number of completed tasks among developers using the AI tool. Notably, less experienced developers showed higher adoption rates and greater productivity gains.

Don't Expect Juniors to Teach Senior Professionals to Use Generative AI: Emerging Technology Risks and Novice AI Risk Mitigation Tactics

Katherine Kellogg, Hila Lifshitz-Assaf, Steven Randazzo, Ethan R. Mollick, Fabrizio Dell'Acqua, Edward McFowland III, Francois Candelon, Karim R. Lakhani

The literature on communities of practice demonstrates that a proven way for senior professionals to upskill themselves in the use of new technologies that undermine existing expertise is to learn from junior professionals. It notes that juniors may be better able than seniors to engage in real-time experimentation close to the work itself, and may be more willing to learn innovative methods that conflict with traditional identities and norms. However, this literature has not explored emerging technologies, which are seen to pose new risks to valued outcomes because of their uncertain and wide-ranging capabilities, exponential rate of change, potential for outperforming humans in a wide variety of skilled and cognitive tasks, and dependence on a vast, varied, and high volume of data and other inputs from a broad ecosystem of actors. It has also not explored obstacles to junior professionals being a source of expertise in the use of new technologies for more senior members in contexts where the juniors themselves are not technical experts, and where technology is so new and rapidly changing that the juniors have had little experience with using it. However, such contexts may be increasingly common. In our study conducted with Boston Consulting Group, a global management consulting firm, we interviewed 78 such junior consultants in July-August 2023 who had recently participated in a field experiment that gave them access to generative AI (GPT-4) for a business problem solving task. Drawing from junior professionals’ in situ reflections soon after the experiment, we argue that such juniors may fail to be a source of expertise in the use of emerging technologies for more senior professionals; instead, they may recommend three kinds of novice AI risk mitigation tactics that: 1) are grounded in a lack of deep understanding of the emerging technology’s capabilities, 2) focus on change to human routines rather than system design, and 3) focus on interventions at the project-level rather than system deployer- or ecosystem-level.

Automation from the Worker's Perspective

Ben Armstrong, Valerie K. Chen, Alex Cuellar, Alexandra Forsey-Smerek, Julie A. Shah

Common narratives about automation often pit new technologies against workers. The introduction of advanced machine tools, industrial robots, and AI have all been met with concern that technological progress will mean fewer jobs. However, workers themselves offer a more optimistic, nuanced perspective. Drawing on a far-reaching 2024 survey of more than 9,000 workers across nine countries, this paper finds that more workers report potential benefits from new technologies like robots and AI for their safety and comfort at work, their pay, and their autonomy on the job than report potential costs. Workers with jobs that ask them to solve complex problems, workers who feel valued by their employers, and workers who are motivated to move up in their careers are all more likely to see new technologies as beneficial. In contrast to assumptions in previous research, more formal education is in some cases associated with more negative attitudes toward automation and its impact on work. In an experimental setting, the prospect of financial incentives for workers improve their perceptions of automation technologies, whereas the prospect of increased input about how new technologies are used does not have a significant effect on workers' attitudes toward automation.

Ask MIT

What’s different about OpenAI’s new o1 model?

OpenAI claims that the o1 and o1 pro models mimic human-like reasoning processes, spending more time “thinking” before responding. The release of the o1 series also marks a transition from OpenAI releasing increasingly larger, more capable, one-size-fits-all models to a tiered approach. The o1-mini, for instance, is meant to balance faster processing times and low cost with lower breadth of knowledge. Given the o1 model’s unpredictably long “thinking” steps, it may be difficult to build reliable applications on top of it. It is also important to note that the o1 model is not a foundation model in the sense that GPT-4 was. It can be helpful to visualize the o1 model as an agentic system that orchestrates foundation models internally to produce its outputs (which partly explains its high costs). This also means that techniques that work well for foundation models such as prompt engineering and retrieval-augmented generation (RAG) may actually worsen o1’s performance.

Given LLM improvements over the last year, traditional benchmarks such as MATH and GSM8K have virtually been maxed out. Consequently, the o1 models are benchmarked on more advanced STEM problems such as PhD-level science questions (GPQA). The o1 model demonstrates a marked increase in problem-solving ability that may advance use cases that were previously considered too complex for LLMs (ex. medical diagnosis, math problems, code debugging etc). However, because this model lacks a subset of the parameters that GPT-4o has, it is not better at all tasks when compared to preceding models. It seems that interoperability and choosing models that are appropriate to the problem at hand remains important.

The advanced reasoning ability is largely tied to the train-time reinforcement learning algorithm that was used. The hope is that this approach can also help align models to human values better using reward functions, as well as reduce the likelihood of hallucinations. However, it remains an open question whether these new chains of thought will truly recalibrate trust in LLMs because the chains of thought themselves remain hidden from the user.