From Nerds to Needs: How product managers bridge the AI gap in research

The Crucial Role of Product Managers in ML Research Teams: Lessons from Meta's Galactica and OpenAI ’s ChatGPT

In the ever-evolving landscape of machine learning (ML), the quest for innovation often eclipses a fundamental truth: no matter how advanced the technology, it ultimately serves a human purpose. This notion is particularly poignant in the case of ML-driven minimum viable products (MVPs). The role of a product manager (PM) in ML research teams is not just beneficial but essential. To underscore this point, let’s delve into the contrasting trajectories of Meta's Galactica and OpenAI’s ChatGPT, drawing insights highlighting why ML research teams should never operate without a PM.

The Galactica Fiasco: A Cautionary Tale

In November 2022, Meta unveiled Galactica, a large language model (LLM) designed to assist scientists. With capabilities like summarizing academic papers and solving math problems, Galactica was intended to be a groundbreaking tool for researchers. However, within three days of its release, Meta pulled the plug on Galactica due to widespread criticism over its propensity to generate biased and incorrect content.

Galactica's downfall wasn’t just a technological hiccup; it was a product management catastrophe. This model targeted a highly specialized and discerning audience—scientists—who dwell in a world where hallucinations are unacceptable, plagiarism is a nightmare, and peer review is sacred. Galactica’s inability to distinguish truth from falsehood was disastrous. Hallucinations, while sometimes amusing in casual use, become a significant issue in the scientific realm, where precision and accuracy are paramount.

Meta ’s team at AI at Meta (FAIR), led by brilliant researchers like Joelle Pineau and Yann LeCun, are exceptional scientists but not product managers. Galactica’s failure lay in a fundamental misunderstanding of the model’s purpose. Releasing it to a community that would scrutinize every output for factual accuracy was a misstep. The initial excitement quickly turned to backlash as users discovered that Galactica’s outputs, while often plausible, were factually incorrect and occasionally offensive.

Yann LeCun’s reaction underscored the lack of product DNA in the approach. His defensive tweets, like “It’s no longer possible to have some fun by casually misusing it. Happy?”, missed the point entirely. Similarly, Joelle Pineau’s reflections, stating that Galactica was never meant to be a product and was released for casual fun, revealed a disconnect from market reality. When you’re Meta, claiming a release isn’t a product or expecting casual fun with an advanced AI model is hiding from the truth.

Joelle Pineau’s analysis that the gap between expectations and research was too big, and that hallucinations wouldn’t be news later, showed a lack of understanding of product-market fit. Galactica’s level of hallucination might have been lower than other models, but it was still unacceptable for its intended use. Claiming that users suddenly had product expectations—like actually writing papers—missed the critical point of aligning the model’s capabilities with user needs.

The lessons Meta claimed to have learned from Galactica’s failure were more about managing public relations than genuinely reassessing their product approach. Joelle Pineau mentioned that if they were to do it again, they would just manage the release better. This perspective only highlights a superficial change. The subsequent launches of Llama models were handled with more caution, but the fundamental lesson about integrating PMs into the process seems to have been missed.

Yann LeCun's remark that releasing an LLM to everyone led to public vitriol, as seen with Galactica, reflected a defensive stance rather than a proactive shift towards user-centered design. The careful release of Llama and its subsequent versions showed a lesson learned in PR, but not necessarily in product development. The scarcity tactic might prevent immediate backlash but does not address the core issue of aligning product capabilities with user expectations.

In conclusion, the Galactica fiasco highlights the critical importance of having product managers in ML research teams. These PMs bring a necessary perspective on user needs, ensuring that even the most advanced models are not just technically impressive but also practically useful. By integrating PMs, teams can bridge the gap between innovative research and real-world applications, preventing missteps like those seen with Galactica and turning technical achievements into meaningful, user-friendly products.

OpenAI’s ChatGPT: A Masterclass in Product Management

Contrast Galactica’s implosion with the meteoric rise of OpenAI’s ChatGPT, and the difference becomes as stark as a plot twist in a Pixar Animation Studios movie. While Galactica floundered, ChatGPT soared, not because it was technically superior, but because it was strategically sound. OpenAI’s approach was a product management masterclass, one that understood the power of user-centric design and the simple yet profound magic of a good chat.

OpenAI’s secret sauce wasn’t just GPT; it was the chat. Picture this: in the realm of the Magnificent Seven of machine learning, where every model is almost neck-and-neck in technology, the real challenge is finding the right application and making it resonate with users. This is where OpenAI hit the bullseye. They recognized that, at its core, AI must serve people, not just dazzle them with its technical prowess.

The genius of ChatGPT’s release lay in its focus on the most human of interactions: conversation. Unlike Galactica, which aimed at a niche market intolerant of errors, ChatGPT targeted everyone. And what do people expect from AI, thanks to a lifetime of sci-fi books and movies? They expect to speak to a machine. Not outrun it, outswim it, or outfly it, but speak to it. Our species, Homo sapiens sapiens, is unique in its ability to communicate through complex language. If an AI can chat like a human, even if it’s not perfect, it feels like a real AI.

The choice to emphasize chat was a stroke of product genius. The UI was simple, intuitive, and engaging—Don Norman would have nodded in approval. It wasn’t about dazzling users with a plethora of features but about doing one thing exceptionally well. The chat interface provided the right affordances and signifiers, making it easy for users to interact without needing a manual. This was everyday AI design at its finest.

OpenAI’s PMs understood that to create a compelling user experience, they needed to tackle the challenge of making the chat feel natural. They used Reinforcement Learning from Human Feedback (RLHF) to fine-tune the interactions, ensuring the AI responses felt more human. This wasn’t about having a more robust or ready model than Galactica; it was about creating an MVP that showcased the LLM’s capabilities in a simple, user-friendly format. And in the end, the most important aspect wasn’t the GPT but the Chat.

OpenAI redefined what an MVP is for ML products. The interface was not cluttered with multiple functionalities; it focused on delivering a seamless chat experience. They even addressed potential issues creatively. For example, latency in response times was masked by writing the response letter by letter, just like in the movies, adding to the perception of quality. Google’s Bard didn’t do that, and as a result, it was perceived as less polished.

They didn't stop there. OpenAI also released an iOS app, tapping into another usage scenario: speaking to an AI on the go. This app transformed the chat experience, allowing users to converse with the AI while walking or at the office. The app didn’t need to handle the full complexity of human-to-human conversation but provided a tremendous speech-to-text transcript, which then launched GPT. It was a brilliant way to MVP AI interaction.

Of course, ChatGPT had its own share of hallucinations. Like Galactica, it often generated responses that sounded plausible but weren’t always factually correct. OpenAI was transparent about this challenge, acknowledging it in their blog. Despite these shortcomings, ChatGPT’s user-friendly design and strategic focus on chat propelled it to become one of the fastest-growing services ever, reaching 100 million monthly users in just two months and now 100 million weekly users.

Ultimately, ChatGPT’s success story is a testament to the power of product management in AI. By focusing on user needs and leveraging simple, intuitive design, OpenAI created a product that not only showcased its technological capabilities but also connected deeply with users. This is the real lesson from ChatGPT: it’s not just about the tech; it’s about how you package it, present it, and, most importantly, how you make it talk.

The Role of the PM in ML Research Teams

The contrasting outcomes of Galactica and ChatGPT illustrate a critical lesson: integrating a PM in ML research teams can bridge the gap between cutting-edge technology and practical, user-friendly products.

In the complex world of AI, integrating research scientists, engineers, and product managers is no easy feat. Engineers and PMs have had their share of growing pains before finding a rhythm, thanks partly to Technical Program Managers (TPMs) and tools like Atlassian Jira (🤣😱🤪). Now, AI introduces a new challenge: bringing research scientists into this collaborative mix. This integration is more than a procedural adjustment; it’s a fundamental shift in mindset and process.

AI and ML are inherently complex, often requiring simplifications based on assumptions. To make the research relevant and impactful, these assumptions must be user-centric. Scientists must learn to draw these assumptions from actual user needs, ensuring their models address real-world problems. This isn’t about dumbing down the science; it’s about making the science work for the people who will use it.

In this context, PMs play a vital role in adjusting the traditional research mindset. The industry demands quality, speed, and velocity—parameters that differ significantly from academic research. Here’s how PMs contribute:

User-Centric Design: PMs ensure that user needs and pain points are at the forefront of algorithm and model development. By involving users early and often, PMs guide research teams to refine their models meaningfully for end users.

Iterative Development: PMs champion the MVP approach, advocating for early, iterative releases and allowing real-world testing and feedback. This process is crucial for refining the product and ensuring it meets user expectations. It’s about showing models as soon as possible, testing them, and iterating quickly.

Adjusting Mindsets: PMs help research teams transition from an academic mindset to an industrial one. In academia, a model might be considered an isolated algorithm. In industry, every model is seen as a product—a prototype of something more than just code. This shift encourages researchers to think beyond the theoretical and consider practical applications from the get-go.

Quality and Velocity: The industrial world values not just quality but also speed and execution. PMs bring the scientists’ execution speed and velocity, a crucial currency in industrial research. It’s about balancing the depth of research with the pace of industry demands.

By integrating PMs into ML research teams, companies can ensure that their models are not only technically advanced but also practically valuable and user-friendly. This integration prevents missteps like those seen with Galactica and turns technical achievements into meaningful, real-world products. In essence, PMs help bridge the gap between nerds and needs, transforming groundbreaking research into accessible innovations.

Learning from the Past: Moving Forward

The lesson from Galactica is unmistakable: integrating a PM into ML research teams isn't just about smoother communication or faster development. It’s about embedding a strategic, user-centered mindset that can transform a technically impressive model into a genuinely impactful product.

For ML-driven MVPs, a PM brings the user into the heart of the research process, navigating the intricate landscape of expectations, market demands, and technological possibilities. They ensure that the 'P' in MVP is never forgotten—that every product, no matter how advanced, is ultimately designed for people.

Meta’s Galactica, juxtaposed with OpenAI ’s ChatGPT, underscores the necessity of this shift. Research scientists must do more than think outside the box; they must also embrace the constraints and opportunities within this box. By welcoming PMs into the fold, ML research teams can strike a balance between innovation and usability, creating products that not only demonstrate technological prowess but also resonate deeply with their users.

Ultimately, it’s not just about making smarter algorithms; it’s also about making smarter decisions on how those algorithms are deployed. PMs are the bridge between the brilliant minds of research scientists and the practical needs of users, turning groundbreaking research into accessible, meaningful innovations. This is the true path forward: a seamless blend of cutting-edge technology and user-focused design, led by the insightful guidance of product managers.