The Five Best Things: Dec 12, 2020

Something smells

Last week, I attempted to explain how the parallelism exposed by the attention mechanism allowed transformer-based natural language models to grow larger and larger. One of the earliest large language models was OpenAI’s GPT, which was followed by GPT-2 last year, and GPT-3 this year. I have a fun demo for GPT-2 today!

The Five Best Things

  1. Weird A.I Yankovic Colab notebook

    • OpenAI’s GPT-2 model was considered the “state of the art” in language models last year. During the initial unveiling, OpenAI refused to release the dataset, training code, or model weights, claiming the potential for societal harm; subsequently they did a staged release to a few hand-picked partners. A brief history of OpenAI: it was created in 2015 as a non-profit, with funding from Elon Musk and Sam Altman, among others, to “advance digital intelligence in the way that is most likely to benefit humanity as a whole, unconstrained by a need to generate financial returns”. In early 2019, they converted to a “capped” profit model, to attract and retain AI researchers with equity, and subsequently announced a $1 Billion funding round from Microsoft. In June this year, they announced the world’s largest language model - GPT-3. More on GPT-3 next week.

  2. The Tech Monopolies Go Vertical

    • This post talks about how the lines between software and hardware companies are blurring, as the traditionally software-driven FAANG companies are now implementing custom hardware roadmaps in an effort to a) reduce margins paid out to hardware vendors like Intel, ARM and Nvidia, and b) keep the hardware innovation proprietary, in order to bring more customers into their ecosystems. At the scale at which these companies operate, this is going to have a profound impact on hardware designs in the future. In particular, the hyperscalers, who sell their services to other companies, can end up with even wider moats.

    • This post was written a month ago, before Amazon announced its own ML training chip (Trainium, lol) last week. Regardless, this is a good summary of the state of SW companies going in to Hardware; I’ve previously written about the geopolitics of the hardware industry of late. However, I think the conclusion that the SW companies will cast a long shadow over hardware vendors is hardly a foregone one. There are more startups in HW now than ever before in my lifetime, and incumbents who are sharpening their focus and competing ferociously. Personally, I am an optimist and I think good things will come from the competition!

  3. Kate Petrova: Why are people complaining about scented candles all of a sudden?

  4. WSJ: Plant-Based Milk, Built by Machine Learning, Hits Whole Foods Shelves

    • A Chilean food-tech startup called NotCo developed an ML model called Giuseppe, which finds combinations of plant products that taste like meat and dairy. By analyzing the molecules in the food, it learns which combinations make, say, cow’s milk and then generates formulas to match. Its first product, NotMilk debuted in Whole Foods stores this past month. NotMilk is made from cabbage and pineapple.

    • The plant-based proteins industry is expected to grow at 14% in the next 5 years. For an industry that traditionally operated on very thin margins, a proven ML model can hugely accelerate new product introduction, and perhaps lead to breakthroughs in reducing the environmental impact of meat production.

  5. The rise of the gaming industry

    • A remarkable chart, courtesy Chris Fralic. CyberPunk 2077 - a game that took 8 years to develop - came out this week and recouped development costs in ONE day. Differentiated content creators are having a fun week - Disney announced a huge slate of new releases for Disney+ and revised forecast for subscribers from 60M to 260M! And Warner Bros will release all of its movies simultaneously on HBO Max and theatres. Will we think of movie theatres in a few years as we think of vaudeville now?

Honorable Mentions

Disclaimer: The views and opinions expressed in this post are my own and do not represent my employer.