What Will AGI Be?

As 2023 wraps up I think it’s time to look at next year and a little beyond 2024. Let’s ask the most consequential question of our life. What will AGI literally be? I think we actually know. This timeline fits with the @elonmusk timeline of AGI by 2029. Let’s start with a short history

2012: It used to be that one machine learning model was used to tell if something is a cat or not a cat. Another model to tell if something is a dog or not a dog. Another model to tell if something is a hot dog or not a hot dog. This changed about a decade ago. The first glimpse of AGI came with AlexNet being able to generalize to multiple object types in the world's most competitive image recognition competition at the time, called ImageNet, with just one model. @geoffreyhinton @ilyasut and Alex Krizhevsky won the competition over other specialist models by utilizing two GPUs and deep learning on a dataset small enough that it can probably be trained on a cell phone today. This shocked the world. I remember the day it happened and the news dropped that now we had a single model that could learn multiple things at once. It felt like this was special.

Alex Krizhevsky, Ilya Sutskever (also famous for cofounding OpenAI and firing Sam Altman), and Geoffrey Hinton won the ImageNet Large Scale Visual Recognition Challenge with AlexNet by training GTX 580 GPUs depicted here for around 6 days

2019: the second glimpse of AGI happened with The Bitter Lesson from @RichardSSutton. The bitter lesson is that methods which learn to generalize with more compute combined with search will always eventually win over the best heuristics.

Richard Sutton's influential book on reinforcement learning

2020: the third glimpse of AGI I would argue actually came about with an often overlooked paper from @OpenAI called AI and efficiency. This paper lays out that algorithmic efficiency improvements are actually more important than classical hardware efficiency improvements. In short our AI algorithms get much much more efficient over time, even if the amount of compute GPUs can give you doesn’t improve. Wrights law improves the cost of transistors. Wrights law means the more transistors you make, the cheaper the price of each of those transistors on average. These two laws are what power AI today, and likely will in the future as long as we keep making more fabs and as long as we keep making more software layers of abstraction in AI. This means as long as we don't run out of physical resources for making more fabs and as long as we don't run out of more high quality software code changes we will achieve exponentially better AI. Algorithmic efficiency has actually outcompeted wright's law for the past decade. There's a lot of evidence that AI will increase the pace of this algorithmic efficiency even further when AI is making high quality software. Eventually AI will make higher quality fabs too, but software will happen first.

Wright's Law applied to the transistor industry for the past 70 years and will likely apply in the future too

2023: the fourth glimpse of AGI happened with OpenAI’s Q* likely using search and reinforcement learning alongside a theorem prover that can check math answers automatically to solve high school math on a small scale. This fits with the bitter lesson.

Q* is likely a name that is a nod to the A* search algorithm for weighted graphs that can be seen as an extension of Dijkstra's algorithm but works far faster in time, especially for more complex graphs. Here is a diagram of A* working on finding the shortest path in New York city

2024: the key is such a system generalizes with more compute and uses search to improve after self-checking just like in The Bitter Lesson from @RichardSSutton. Let’s look at an example of Q* generalizing with more compute. If we solve math better than any human with Q* in 2024 we can next make more sophisticated versions of Q* that don’t just work for math but also work for software.

Richard Sutton idea's are some of the most influential in AI

2025: software can be automatically generated, then run and evaluated with automatically generated tests, just like math theorems can be checked with an automated theorem prover. You can easily imagine more sophisticated software being created, run, and improved by a near future more advanced Q*. This is the obvious logical next step after applying Q* to math.

An automated DevOps cycle for AI to build, test, deploy, and update code & tests automatically is almost here

2026: now we can imagine more advanced software, like low fidelity simulations, running simulated robots and improving them in simulation until there’s a sim2real transfer, which is a method of robotics demonstrated by @pabbeel lab and others working in reinforcement learning. Here’s a @ETH_en robot doing just that last year depicted by my favorite YouTube channel @twominutepapers go watch their latest video for more details.

Two Minute Papers has a video you should watch about reinforcement learning and sim2real to make AI applied to robotics

2027: with a Q* like algorithm leveraging sim2real and simulations we could have robots that work orders of magnitude better than any robot ever has. And we can even put Q* like algorithms to work making higher fidelity simulations and comparing experiments in simulation automatically to observations in videos, images, labs, and recordings to improve simulation fidelity.

Embodiment of AI with robotics

2028: we could have high fidelity simulations of very complex systems like robotics, computer chips, manufacturing processes, homes, businesses, battlefields, economic processes, the climate, our microbiome, the biosphere, and even medically relevant processes for our AIs to explore.

High fidelity simulations being constantly improved and utilized to evaluate Q* like systems automatically will lead to extremely high fidelity and efficient simulations

2029: ensembles of simulations could be run and used by an AI that makes Q* look quaint, lower fidelity simulations run most of the time to preserve compute efficiency, the highest fidelity simulations only used when needed to maximize sample efficiency, and real world examples/experiments used as a last resort to improve simulation quality because they’re the most expensive.

Ensembles of different fidelity simulations will become normal within years. Real world experiments are just too expensive most of the time

2030: the world will be absolutely unrecognizable to us due to AGI that improves and generalizes with more compute. You only need wrights law, the algorithmic efficiency law, and a Q* like system that takes advantage of The Bitter Lesson. New diseases are cured monthly, new science discovered daily, sometimes even new fields discovered, new factories are orders of magnitude more efficient, twenty million new self driving electric cars manufactured per year, millions of affordable low cost robots like the @tesla_Optimus widely available for work in all developed nations leading to a standard of living an order of magnitude higher almost everywhere, lower home construction costs, and billions of jobs automatable but billions of new jobs that just weren’t economically possible before too.

Robots in the near future will work orders of magnitudes better than today's robots which would be shocking to someone from 2024

This is for all intents and purposes AGI. This is one optimistic but realistic path of many possible paths. one thing is certain, soon humans will never outcompete AGI and robots at any job ever again. And while it’s a complex world with many tradeoffs, a world with AGI is a very good world in the same way a world with an industrial revolution is good today though it may not have seemed like it at the time. It’s happening, I just hope the conversation about UBI and who gets what in a world of prosperity goes over well. In 2024 we can choose to, in the words of @sama, push back the veil of ignorance.

A world with AGI will be complicated, but we're optimistic

Get updates to your inbox.

Learn all our industry secrets
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.