• Daniel Gross and Nat Friedman discussed their new AI toy, a teleprompter that prompts users for the next thing to say in different styles.
• They discussed the explosion of ChatGPT, which has validated the idea that good enough technology is available, but there is still a product hole.
• They discussed why people are lagging behind in using AI technology, citing lack of awareness, cost, and the perception that specialized knowledge is needed.
• They also discussed the potential of AI as a platform, and the need for people to explore and tinker with the technology.
• ChatGPT was successful due to reinforcement learning with human feedback (RLHF).
• RLHF made ChatGPT more accessible because the answers were more predictable and concise.
• AI alignment is the idea of getting AI to do more of what we want.
• Alignment is about making sure the AI does what it is supposed to do and not something else.
• Alignment is rooted in the issue of models “hallucinating” and sometimes outputting something that is not the answer.
• Alignment is both a religious movement and a pragmatic one.
• Truth, Hallucination, and Bias are key components of language models.
• It is difficult to create an algorithm that only outputs true things.
• Techniques such as document embedding and retrieval can reduce hallucination.
• Bias is fundamental to making a good product, and data and feedback may be a moat.
• Models are able to interpolate through an embedding space and learn meta lessons from fine tuning.
• GPT-4 is expected to be a noticeable improvement over GPT-3.
• AI generated text and images are becoming commodities, increasing the value of branding and reputation.
• GPT-3.5 is a branding tool that could increase the returns to trust and thoughtfulness.
• AI models can cause us to downgrade the appearance of authoritativeness, leading to a more truth-seeking society.
• AI models can be used for brainstorming and semantic search, but are not yet capable of producing big out of distribution insights.
• Apple has implemented optimizations for Stable Diffusion into their operating system, allowing for faster local processing of images.
• This has enabled Lensa to provide a great user experience and become a successful product.
• Apple’s strategy of commoditizing open source complements is a great gift to their business.
• Apple was surprised by the success of the M1 chip, which was a result of Intel’s poor performance.
• Stable Diffusion is an open source product that is surprisingly good and runs locally, making it a great surprise in the space of ChatGPT.
• It has been optimized to run on an iPhone and can make a hundred frames a second on server hardware.
• It has been used to fine-tune the model on objects, oneself, and even music.
• Text is inherently a good match with deterministic thinking, but images are more biologically oriented and may be the future of computing.
• Text is a hack, as humans are visually oriented and the printing press created a deficit of knowledge.
• Stable Diffusion has drastically compressed our representations of reality and figured out the minimum viable thing to output an image.
• AI as a platform is a concept similar to Windows, where applications can benefit from new processor technology without needing to be rewritten.
• Stable Diffusion 2 caused backlash due to its different clip model and more aggressive dataset filtering.
• There is a debate over whether the moat in AI is technical innovation or access to data.
• OpenAI’s Whisper model was trained using YouTube captions, demonstrating the potential for models to be “sucked” over the internet.
• K-shot learning is a middle layer between open source models and API use, allowing for more use cases.
• Middleware companies may be necessary to bridge the gap between research and product, as the API created by large language model companies may not be as user-friendly.
• AI capabilities are outpacing products and tinkerers, and the pace of change will not slow down.
• We will see more multimodal models that can consume and produce images as well as words.
• The cost of data will become increasingly important compared to the cost of compute.
• There will be a backlash against AI, particularly in non-fiction applications.
• Startups may have an opportunity to disrupt incumbents by taking advantage of AI to reduce costs.
• We may see AI native products emerge, such as automated music streaming services.
• Billion parameter functions and 100 trillion parameter AGIs are interesting, but it is unclear if the space in between is useful.
• The GPU paradox is the discrepancy between the high demand for GPUs due to the explosion of AI and Nvidia’s decision to write down their existing inventory and future purchase orders from TSMC.
• Possible explanations for this paradox include GPUs being more efficient than expected, Meta’s success in building their own silicon, and AI not being as big of a deal as it appears to be in Silicon Valley.
• Diffusion of AI technology is taking longer than expected, and regulators and conversations about AI will take time to develop.
• Demand for GPUs comes from inference, not training, and there are not yet many products with wide usage that leverage AI.
• Nvidia may have over-corrected after being financially destroyed by over-forecasting for the previous generation.
• Stable Diffusion is a breakthrough that allows for product exploration and experimentation without cost, and may be even more important if vision turns out to be more important than text.
Published December 22, 2022.
Visit Stratechery to read Ben Thompson’s original post