The Future of Expertise May Be Bimodal

I think AI is the greatest learning assistant ever made. As a postdoctoral researcher who has a lot of responsibilities, I don’t get nearly as much time to read papers and research material as I’d like.

One of the main reasons that I don’t get enough time is that reading a scientific paper or working through a textbook is slow. You need to sit down uninterrupted and do some deep work to figure out what the authors are really trying to say. For math heavy papers, you need to understand the mathematics at least at a high level and then figure out how this new information fits into your existing understanding of the field.

It doesn’t help that scientific papers are usually written to satisfy reviewers and survive peer review rather than to communicate clearly to the public.

So like most experienced academics, I’ve learned to ruthlessly prioritize what I read. Of course, experienced academics also have a bag of tricks to read papers faster. You learn to skim the abstract, go straight to the methods, skim the results and get a gist of the paper before deciding if the paper is worth reading seriously. But getting the gist of a paper and genuinely understanding it are two different things. And the latter can still be a slog.

ChatGPT / Claude has been extraordinarily useful in my paper reading workflow in the last few months.

LLMs are excellent paper-speak translators. I can take a particularly dense section of a paper and ask it to explain what the authors are actually trying to do. If there’s a mathematical expression I don’t understand, I can drill down into the details. If a term is unfamiliar, I can trace it back through the literature until I understand who invented it and what it is in simple language. If the explanation doesn’t make sense, I just keep asking.

Over time my workflow to read a paper has converged into something like this:

1. Read a section of a paper and ask AI to help create a diagram and an explanation snippet.
2. The first iteration is usually just a rehash of the paper wording, and therefore hard to grok.
3. Ask more questions, dig into the mathematics, check the original references, and improve the explanation.
4. With a better understanding of the concept, return to the diagram and notice things that could be improved.
5. Iterate again, until I have a diagram that accurately represents what I’ve learned, an explanation that makes sense, and, most importantly, an excellent internal understanding of the concept myself.

This process of going back and forth between the AI, the source material, and my own developing understanding is immensely productive. Without AI, I would probably have spent several days working through some of these concepts. Or, more realistically, I might have gotten stuck somewhere, put the paper aside, and promised myself that I’d return to it when I had more time. (Spoiler Alert: This might be months later!).

With AI however, I’ve been able to read and understand an incredible quantity of material in a short time. I’m genuinely satisfied with how much I’ve been able to learn, and more importantly, how much of it I can now explain and reason about independently.

As someone who enjoys learning for its own sake, these last few weeks have been incredibly exciting. I can’t think of many technologies that are more empowering for self-driven learners.

So if AI is such a great enabler for learners, why are we so worried about our students?

We’re worried about the students

There are more and more alarming headlines in the news these days about how AI is destroying education as we know it. To be completely fair to the journalists, I think the results from the actual studies are pretty alarming too.

A study involving nearly a thousand high school students found that students using GPT-4 performed significantly better on mathematics practice problems. But when the AI was taken away, the students who had access to an unrestricted AI assistant performed 17% worse on an independent examination than those who had never used it.

Another study on knowledge retention finds that university students using ChatGPT as a study aid remembered less material when tested 45 days later. And researchers at MIT have reported reduced neural connectivity and poorer recall among participants who used ChatGPT to write essays.

At face value, the situation seems quite grim. If we were to extrapolate based on these initial studies into the future, it would seem that we’re doomed to rely more and more on AI and have fewer and fewer students who actually learn the things they go to school for. Students are increasingly able to produce impressive work without necessarily understanding what they’ve produced.

But I think there’s something that these statistics aren’t capturing yet. Something that might only become obvious when these students finish their education and start building and doing things in the real world.

My hypothesis is that AI will lead to a bimodal distribution of knowledge and expertise, and that the future might belong to the obsessed.

There’s two kinds of people

I think there will be a sharp split in the AI era between those who just want the answer and those who want to understand the answer. The mathematics study I cited earlier also found that students equipped with a version of GPT-4 designed to guide them through problems rather than provide answers did not suffer the same decline in test performance. How you use AI seems to play a large role in how expertise is developed.

If you just want the answer, AI is amazing. You just get the answer, use it and move on with your life. Why spend three hours working through an assignment when Claude can do it in thirty seconds?

If you want mastery however, studying with AI is like doing rock climbing with an Iron Man suit. A perfectly patient, always available tutor who can explain at ten levels of simplicity with diagrams and examples.

But if you grade the assignments from these two groups of people, you may not be able to spot much of a difference between them. Both could be producing polished, correct AI assisted answers by maxing out their Claude subscription every month.

The difference between the two groups might only really show up when they’re confronted with a genuinely new problem in the real world that needs their expertise and doesn’t have a ready-made answer.

But what happens to everyone in the middle?

Of course I do worry about what happens to the middle. On one end, you may have people who aren’t particularly interested in developing expertise. They can borrow capability from AI to get things done and are perfectly happy with the “good enough” results that they get.

And at the other end, you have people who are obsessed. These are the kind of people who might work through difficult textbooks for fun or spend hours acquiring specific knowledge of things. Like the Japanese guy who spent ten years inventing more than 10,000 imaginary railway stations, complete with route maps, timetables, and passenger statistics. Or the Singaporean bus enthusiast who knows the public transport network so well that he can identify a bus stop from just a few bus numbers. These types of people can go further, faster, and potentially reach levels of understanding that would previously have required much more time and access to specialist mentorship.

What about those in the middle who are interested in developing expertise, but only to a moderate degree? Historically, there was a substantial reward for becoming moderately competent at something. It was enough to distinguish you from people who had never learned it. When AI can match the output of someone with moderate expertise, the incentive to get reasonably good at something is reduced somewhat.

A bimodal future for expertise

My hypothesis is that the future of human expertise may be bimodal with lots of super-experts and novices but not many people in the middle.

The obsessed use AI to develop deeper expertise, faster than before and reach heights of human ability hitherto unheard of. And most people don’t bother to go to the moderate level at anything.

This is a slightly more optimistic counterpoint to what I’ve written about previously in The Last Craftsmen, where I worried about what happens to deep expertise and craftsmanship when AI can perform increasingly sophisticated intellectual work.

Maybe there’s another possibility after all. Maybe it’s not all doom and gloom. I said that AI might remove the lower rungs from the ladder of expertise, and I still hold that view. But it gives the sufficiently obsessed a rocket that skips right over the ladder.