Last week, I attempted to explain how the parallelism exposed by the attention mechanism allowed transformer-based natural language models to grow larger and larger. One of the earliest large language models was OpenAI’s GPT, which was followed by GPT-2 last year, and GPT-3 this year. I have a fun demo for GPT-2 today!
The Five Best Things
This song lyric generator, based on the natural language models GPT-2 and XLNet, takes in a prompt and sample lyrics, and creates an entirely new song in the style of Weird Al Yankovic. Click on Runtime > Run All, and then click on the little arrow next to Basic User Interface (see below). I seeded mine with a song my daughter likes, I’ll leave it to you to play around and decide if the results sound like a Weird Al song :-)
OpenAI’s GPT-2 model was considered the “state of the art” in language models last year. During the initial unveiling, OpenAI refused to release the dataset, training code, or model weights, claiming the potential for societal harm; subsequently they did a staged release to a few hand-picked partners. A brief history of OpenAI: it was created in 2015 as a non-profit, with funding from Elon Musk and Sam Altman, among others, to “advance digital intelligence in the way that is most likely to benefit humanity as a whole, unconstrained by a need to generate financial returns”. In early 2019, they converted to a “capped” profit model, to attract and retain AI researchers with equity, and subsequently announced a $1 Billion funding round from Microsoft. In June this year, they announced the world’s largest language model - GPT-3. More on GPT-3 next week.
This post talks about how the lines between software and hardware companies are blurring, as the traditionally software-driven FAANG companies are now implementing custom hardware roadmaps in an effort to a) reduce margins paid out to hardware vendors like Intel, ARM and Nvidia, and b) keep the hardware innovation proprietary, in order to bring more customers into their ecosystems. At the scale at which these companies operate, this is going to have a profound impact on hardware designs in the future. In particular, the hyperscalers, who sell their services to other companies, can end up with even wider moats.
This post was written a month ago, before Amazon announced its own ML training chip (Trainium, lol) last week. Regardless, this is a good summary of the state of SW companies going in to Hardware; I’ve previously written about the geopolitics of the hardware industry of late. However, I think the conclusion that the SW companies will cast a long shadow over hardware vendors is hardly a foregone one. There are more startups in HW now than ever before in my lifetime, and incumbents who are sharpening their focus and competing ferociously. Personally, I am an optimist and I think good things will come from the competition!
- I couldn’t just walk past this Tweet, so here is some fun Scented candles: An unexpected victim of the COVID-19 pandemic 1/n
Terri Nelson @TerriDrawsStuffThere are angry ladies all over Yankee Candle’s site reporting that none of the candles they just got had any smell at all. I wonder if they’re feeling a little hot and nothing has much taste for the last couple days too.
This thread went viral last week; since the beginning of this year, reviews for scented candles have been dropping at a much faster rate than unscented ones. One of the signature symptoms of Covid-19 is the loss of a sense of smell. Researcher Kate Petrova compellingly puts two-and-two together.
A Chilean food-tech startup called NotCo developed an ML model called Giuseppe, which finds combinations of plant products that taste like meat and dairy. By analyzing the molecules in the food, it learns which combinations make, say, cow’s milk and then generates formulas to match. Its first product, NotMilk debuted in Whole Foods stores this past month. NotMilk is made from cabbage and pineapple.
The plant-based proteins industry is expected to grow at 14% in the next 5 years. For an industry that traditionally operated on very thin margins, a proven ML model can hugely accelerate new product introduction, and perhaps lead to breakthroughs in reducing the environmental impact of meat production.
A remarkable chart, courtesy Chris Fralic. CyberPunk 2077 - a game that took 8 years to develop - came out this week and recouped development costs in ONE day. Differentiated content creators are having a fun week - Disney announced a huge slate of new releases for Disney+ and revised forecast for subscribers from 60M to 260M! And Warner Bros will release all of its movies simultaneously on HBO Max and theatres. Will we think of movie theatres in a few years as we think of vaudeville now?
AirBnB went public this week; the stock immediately jumped from $68 to $139 (see CEO Brian Chesky’s shocked face when he heard that price here), capping a wild year for the company. They released a nice video of AirBnb hosts ringing their doorbells, in lieu of missing out on the opening bell on the NYSE.
Remember when I posted about Amoako Boafo and the heist he pulled on art dealers that tried to exploit him? This is a nice spotlight on Gallerist Mariane Ibrahim, who discovered him and many other African artists.
WSJ: Barnes & Noble’s New Boss Tries to Save the Chain—and Traditional Bookselling Surprised and delighted to read that not only is Barnes and Noble alive, but thriving during the pandemic. The new CEO brought in a host of changes that empowered local store owners.
Princess Diana is in the news again because of the latest season of the Crown. Here’s a good read in Elle on how significant she was to destigmatizing AIDS.
This is a lovely thread to brighten your week!
Proud of my friend and classmate Katie Neal for this shout out from Business Insider!
Disclaimer: The views and opinions expressed in this post are my own and do not represent my employer.