The Five Best Things: Nov 22, 2020

A grab bag this week

Sorry about the delay in getting this one out. We picked up a cold around the household, and in our year of 2020, this is an extremely frantic experience. So I needed a day to catch my breath.

I will also be taking a break next weekend due to the Thanksgiving holiday in the U.S.

Today I am kicking off the road-to-GPT-3 series, with an explainer of the attention mechanism - the fundamental component of current era giant Natural Language Processing ML models.

Nvidia announced their Q3 earnings this week; they were as expected. Professional visualization segment revenues dropped 27% YoY, which is puzzling; the 10-Q pins it on weaker demand for desktop workstations vs laptops. The newest product in this segment launched in October, and Nvidia has been pretty supply constrained, so my sense is that Q4 results will round out the story on the graphics segment.

The news of Nvidia partnering with Epic games to launch a Safari-webpage only version of Fortnite broke over the weekend. Earlier this year, Fortnite was kicked out of the App store for offering to charge users a lower price if purchases were made outside of the App store, to get around paying Apple’s 30% tax on any in-app purchases. If the webpage-only version works out, Apple will end up making $0, ; might it finally concede that the 30% take rate is excessive?

The Five Best Things

  1. Gabriel Ilharco & Co: High Performance Natural Language Processing

    • This was a ~3 hour tutorial on the current state of Natural Language Processing, with accompanying slides here. The first 30 minutes catch you up to the current state of the art in Transformer models. I found the slides to be very well done, though the talk is academic for a layperson audience.

    • The key mechanism underpinning transformers is the “Attention” block; an intuitive explainer here: when we hear a sentence such as “The dog is a Labrador” we immediately grasp that the two most important words are “dog” and “Labrador”. Attention mechanisms capture the way our minds pick up on this relative importance between words in a sentence. Further, attention blocks can be computed independently, making their processing very parallelizable. This breakthrough has led to order of magnitude improvements in NLP, which I will cover next time.

  2. James Allworth: Intel’s Disruption is Now Complete

    • James Allworth is the Head of Innovation at Cloudflare and was a student and collaborator of Clayton Christensen, the creator of Disruption theory. In this short article, he first provides a history of Intel being on the winning side of market disruption. Apple announced that its latest Macbook Pro laptops will be powered by the in-house M1 processor, replacing Intel’s processors. Allworth argues that Intel is now on the losing end of low-cost disruption.

    • When you see the two charts in this article, Allworth’s point is made starkly clear. ARM-based chips have now scaled up from cellphones to laptops, are making inroads in cloud-based server computing and Supercomputing. The server market is at an interesting crossroads - one one hand, ARM-based servers offer customizability and arguably lower operating costs, but need to overcome software compatibility issues, since the vast majority of software has been written to be x86-native. On the other hand, AMD offers a similar cost structure without the customizability, but with software compatibility built-in. There is also the consideration that ARM is getting acquired by Nvidia, which has monopolistic tendencies of its own. I suspect there will be wins for both, but secular losses for Intel in the next few years.

  3. ML Art

    • A curated collection of art generated by machine learning, by the Google Arts & Culture team. My favorite is this one where you can add random objects to paintings. I’ll play around with the code one of these days and create my own!

  4. 10KDiver: Stock Based Compensation (click below to read the entire thread)

    • A great thread about how to impute the real cost of stock-based compensation, also known as RSUs, which are handed to executives and employees in lieu of salaries at many companies. It walks through an example of how to infer the cost of SBC, using a real example (Microsoft), and the negative implications for “normal” shareholders of a company.

  5. Texas Monthly: Meet an Intrepid Texan Who Spends Winters at the South Pole

    • A great write up about Texas native Wayne White, site manager of the Amundsen-Scott South Pole Station in Antarctica, and the only person to hold this post 3 years in a row. A couple of my favorite quotes below -

      Nobody at the station wears a mask—with no outsiders since February, there has been no need—and the crew doesn’t social distance. “Some people will say to me, ‘You’re lucky to be down there during this,’” White says. “But I say it’s hellish. We can’t do anything for people back home. We’ve had deaths, unforeseen divorces, hurricanes, and fires back home, and we can’t do a goddamn thing to help … I’d much rather be home than dealing with it here.”

      After eating almost exclusively frozen food since February, White is hankering for some Mexican food and a nice fresh salad—and a trip to the grocery store. “You can’t imagine what an H-E-B seems like after being here for a year,” he laughs.

Honorable Mentions

Disclaimer: The views and opinions expressed in this post are my own and do not represent my employer.