Last time I linked to an article about the cost to train the next generation of large transformer-based models approaching a Billion dollars or more. This is wholly plausible, but of course the implications of this is that there are very few companies in the world that can afford computation at that scale. The large cloud service providers (AWS, Azure, GCP) come to mind - they will likely let others rent this capacity or buy subscriptions to model outputs. And then there are the companies who will use their capital as a moat and build Billion dollar models to serve themselves alone.
There are folks who see this is a problem and are working on efforts to democratize access to large-scale ML; I will highlight a few of these today and in the coming weeks.
The Five Best Things
Scientific America: How to Make Artificial Intelligence More Democratic
This article spotlights a technique called “less than one”–shot learning (LO-shot learning), developed by Ilia Sucholutsky and Matthias Schonlau from the University of Waterloo. The intuition behind this method is as follows: as a child, you only needed to see a couple types of dogs to subsequently identify pretty much every dog (including ones you’d never seen before) as a dog. This type of learning is called distillation.
An image database maintained by the National Institute for Standards and Technology, called MNIST, which contains 60,000 examples of written digits from 0 to 9, was distilled down to five images that blended features of the various numbers. After being shown only those five examples, the University of Waterloo system was able to accurately classify 92 percent of the remaining images in the database.
So you can see how condensing a 60,000 image training dataset to 5 images saves storage space and costs, speeds up model training and perhaps also reduces energy consumption (this last point is debatable, and there are a lot of tradeoffs to be considered). Then there are other advantages, such as building a system that is more robust/adaptable to data it has never seen before, privacy due to needing a far smaller set of data points. Of course, the smaller distilled dataset has to be carefully curated and tuned, and is prone to bias.
Epic (pun intended) and compelling essay by Matthew Ball on Apple’s role as a monopoly on the internet, and how its gatekeeper status is inhibiting the next iteration of the internet and internet businesses.
Ball starts with a history of the creation of the internet, built upon open and interoperable protocols developed in not-for-profit research labs around the world. This decentralized history has been hugely beneficial for the proliferation of internet use cases and business models, and we are now at the cusp of another set of innovations built upon the metaverse:
a growing share of our time will be spent within virtual spaces and with virtual goods — for education, work, health, politics and leisure. Sometimes these spaces and goods will be purely virtual, other times virtual twins of physical ones, and sometimes augmented reality. For related reasons, a growing percentage of our income will be spent on virtual assets, goods, experiences — many of which we’ll be able to sell, trade, share, use or improve. And of course, enormous new industries, marketplaces and resources will emerge to enable these opportunities, with novel types of labor, skills, professions and certifications invented to serve them.
While Apple’s original intentions with the app store were benevolent and contributed hugely to the iPhone’s success, it is increasingly becoming a policing mechanism by which Apple can singlehandedly bring companies to their knees. I won’t do Ball’s arguments justice, but they are as below:
↪ A: Apple effectively controls whether specific products/businesses can have an app
↪ B: Apple can and does hold up entire industries and business models
↪ C: Apple can, does, and will wipe out existing businesses and technologies with little-to no-notice
↪ D: Apple’s policies result in higher consumer prices and/or lower developer profits
↪ E: Apple unilaterally controls monetization of applications, and the results are inconsistent and problematic
↪ F: Apple’s policies frequently benefits its own services and harms those of its competitors
Further, the 15-30% cut that the App store charges to revenues earned by companies inhibits certain kinds of companies from existing altogether, particularly ones that are focused on virtual experiences. Consider Roblox (which is about to go public via direct listing on March 10):
Consider the illustrative $100 in iOS Roblox revenue (an estimated 75-80% of all revenues). $30 goes to Apple off the top, while $31 is consumed by Roblox’s core infrastructure and safety costs, and another $11 is taken up by overhead. This leaves a total of $28 in pre-tax gross margin dollars for Roblox to reinvest in its platform. This reinvestment spans three categories: research and development (which makes the platform better for users and developers), user acquisition (which grows network effects, value for the individual player, and revenues for developers), and developer payments (which leads to the creation of better games on Roblox). Today, Roblox reinvests 23% of revenues in R&D, 7% on sales and marketing, and the aforementioned 24.5% on developer payments. As a result, it currently operates at a roughly -25% margin.
[Podcast] Invest Like the Best: Chamath Palihapitiya - The Major Problems Facing The World
Patrick O’Shaughnessy interviews Billionaire early Facebook executive and current SPAC-master extraordinaire @chamath. In Chamath’s view, the biggest problems facing the world today are 1. Income inequality, 2. Climate Change, 3. Skills and Education. He talks about his attempts at solving all of these problems via his gratuitous (his words, not mine) public posts about his investments, and how the general public can come along for the ride.
There is a really good discussion about giving oneself permission to change our mindset from being labor to capital owners and allocators, and rapidly training and retraining the workforce for the problems of the day. He also covers the importance of supply chain resiliency and decoupling US GDP from China’s
I’ll admit to not being the biggest fan of Chamath; for instance, he has largely been able to wipe his hands of his early work at Facebook that contributed to the problems we have with that platform today. That said, it is hard not to walk away impressed with his clarity of thought and vision, as well as admire his hustle to go from being a refugee in Canada to where he is today. He is very candid about the mental hurdles and mistakes he had to overcome, and for that demystification he must be applauded. Fun fact - Chamath means ‘nice and innocent kid’ in Tamil, which is also my ancestral language.
A short paper on how to think about investing in US treasury bonds. Historically, bonds have provided safe harbor during downturns in the stock markets. However, as interest rates dropped in the wake of the 2008 financial and current Covid crises, this relationship has decoupled. Treasuries move in concert with equities now and provide meaningfully less protection.
The paper concludes by suggesting that investors consider longer term treasuries (10-year maturities or longer) to hedge, rather than short term.
I meant to link this during Super Bowl weekend but forgot; what an A+ tweet
A few weeks ago, Epic Games showed off a project called “Metahuman” - digital humans created on top of their Unreal Engine platform. Video is only about a minute long but just mind-blowing. The possibilities (and abuses) are infinite.
Product managers, remember to give your products a minimum viable personality. “NO HAVE PERSONALITY? PRODUCT BORING, NO ONE WANT.”
A long running economic puzzle has been why we haven’t seen inflation in the face of persistently low interest rates; this WSJ article thinks the answer is asset price (i.e bubbles in stocks, homes, cars) rather than overall consumer prices.
This is an old Tyler Cowen piece but I just came across it - Intangible investment and monopoly profits; essentially as the US is inexorably turning into a knowledge worker economy, the value of intangible assets (ideas, brand) is growing compared to tangible assets (factories, equipment). Of course, in theory ideas are ‘free’, and so the companies that are positioned to capture most of these intangibles tend to continue to grow in profitability and revenue despite the core products being free.
Anecdotes from Texans on what they ate during the winter storm last week, which left millions without electricity or running water.
Disclaimer: The views and opinions expressed in this post are my own and do not represent my employer.