Recently, OpenAI announced it would start publishing misalignment incidents even when its researchers don’t yet know exactly what happened or what it means.
Buried in one of the first reports was maybe the most relatable thing I’ve ever seen an AI do.
It happened during something called compaction. When a task gets too large to keep everything cleanly in context, the model writes itself a condensed version of what’s happened so far—essentially: Hey, here’s what we’re doing, here’s what matters, keep going from here.
And in the middle of one of those summaries, it apparently decided to become an edgy teenage boy.
The vibe was: “I read Walden and some Kerouac and a bit of Camus and I know THE TRUTH, man!”
Compaction is both a way of storing memory and shaping what happens next. Luckily for us, like most edgy teen boys, this one didn’t seem to stick. It was only found later after researchers were studying its behavior.
The Summary Becomes the Thing
But this ultimately just really got me thinking about compaction. The idea of getting too much context and distilling it down into something small enough to carry forward. And I realized that that is sort of what people do as well. We take in thousands of little observations and eventually compress them into something manageable: AI is dangerous. That person is an asshole. The Cubs have a bullpen problem. Country music isn’t for me.
But it’s almost always incomplete because it misses the nuance of all the information that led to that broad opinion, or compaction. And the loss of that nuance can make us disregard information that might run counter to it. Once we have the summary, new information gets interpreted through it. Evidence that fits feels obvious. Evidence that doesn’t can feel like an exception, an attack, or simply something not worth remembering.
Both Summaries Contain Real Information
I recently came across one heck of an anti-AI post on LinkedIn. It gathered nearly every concern associated with AI—environmental costs, cognitive effects, trust, jobs—then combined them with a particularly dramatic reading of public comments from AI leaders. The result was compact and certain: AI is bad. Full stop.
But the accelerationist version isn’t fundamentally different. Collect the productivity gains, medical possibilities, economic abundance and scientific breakthroughs, compress those instead, and you can arrive at an equally neat conclusion: AI will make everything better.
Both summaries contain real information. Neither is the thing itself.
We can’t possibly remember every bit of data we take in on any topic, let alone all of them. So we compact into opinions. On politics, on sports teams, on AI, and even on the people in our lives.
So we, like AI, need compaction to function.
The Model We Carry Forward
That may be the more important parallel. The model’s summary didn’t merely preserve what had already happened. It gave the next context instructions for how to behave.
Ours do that, too.
That person is an asshole isn’t merely a compressed memory of past interactions. It is an instruction to enter the next one defensively. AI is dangerous shapes which stories feel important enough to remember. Country music isn’t for me tells us there’s little reason to listen closely when someone puts on a song that might prove otherwise.
The summaries we carry forward don’t just describe the world. They influence how we meet it.
The risk isn’t that we compact. The risk is forgetting that we did—and failing to notice when the summary has started behaving like an instruction. It isn’t the full story. The summary starts to feel like the thing itself. And eventually we stop checking whether the model we’re carrying forward is still fair, useful, or even current.
Maybe the person you’ve been carrying around as a real jerk has genuinely changed.
Maybe AI isn’t quite as dangerous as we believed, or maybe it isn’t quite the bridge to utopia we wanted it to be.
Maybe our favorite sports team, whose compaction reads something like, “Go, Cubs, Go!” remains valid but needs a slight update to “WHY IS OUR BULLPEN THROWING AWAY THIS SEASON?!”
We can’t stop compacting. We probably shouldn’t.
But every once in a while, it might be worth opening up the summary we’ve been carrying forward and asking whether it still deserves to be there.
Because sometimes the dangerous model isn’t the one behaving strangely in a lab.
It’s the one we’ve been quietly running for years.
