What the hell is going on? Seriously
I’ve been neglecting writing to you for a while now, and in that while a lot has been happening. Most recently there has been a lot more people worrying about whether AI is going to kill us all. I know that I’ve brought this up before, but it is making the rounds again as several frontier AI companies and researchers have stepped out to say that they are concerned that AI is advancing too fast.
Is this real? Is it hype? Should you be worried?
A lot of people have been writing about this issue, but before I write too much more I want to give you a little background on why I have an opinion about what is going on.
Quick definition: “LLM” means Large language model, which are the chatbots built by companies like OpenAI (ChatGPT) and Anthropic (Claude).
This past winter, as I was writing and thinking about AI, I ended up going down a ridiculously deep rabbit hole in an effort to better understand the thing I was writing about. Specifically, I ran a fairly large set of experiments on large language models to better understand how the way they use high dimensional geometry to compute and how that geometry affects what they do. Most of you know me as an artist and writer, but this wasn’t completely out of the blue for me. I have a fairly deep background in multivariate statistics from my training as an evolutionary biologist, and statistical tools are one of the ways researchers try to understand how LLMs work. In fact, LLMs are fundamentally just very complex statistical machines.
So I went down this ridiculously deep rabbit hole and ended up finding some things that seemed interesting so I wrote the results up in a journal paper preprint which you can read on a site called ArXiv. You’re welcome to look at the paper, but it’s long and dense. Suffice it to say that the exercise helped me understand more about the specifics of how LLMs work, and in the process I may have made a tiny bit of a contribution to the vast field of mechanistic interpretability which tries to understand how LLMs work. A one sentence layman’s summary might be that as large language models get bigger and more capable, the part of them that grows is the part that is less directly connected to their output.
Am I insane?
If insanity is measured by whether a reasonable person would undertake something, then maybe I am a little insane. You don’t have to understand what I did to see that it was a crazy amount of work. It wouldn’t be hyperbole to say that the work I wrote is roughly equivalent to a PhD dissertation. I have a pretty good feel for that since I’ve done one of those. Nobody was giving me a PhD or paying me to do this so, yeah, it was probably a certifiably insane thing to do. So why did I do it?
First, I admit to having a problem with obsessions throughout my life, but my obsessions have led to some interesting things including my work on Inventoried Roadless Areas (which is currently being dismantled by the Trump administration - another story) and my Enemies Project. Both of these were ridiculous amounts of work for one person.
However there is a huge caveat on the amount of work that went into this project. The caveat is that I used LLMs to help with a lot of it – the coding, the analysis, and the literature research. I could not have done the work entirely on my own. That said, the research was not just the product of talking with an LLM. The ideation was based on my reading of the primary literature, and the process was a complex interaction with LLMs. The entire thing probably deserves a long write-up of it’s own, but instead I’m going to list out the main reasons that kept me pushing down this rabbit hole, because I think that is a bit more interesting.
Discovering things is fun: I love learning deeply about things, and it felt good to use the scientific research side of my brain again. That was probably the one major driver, because I find the black box nature of LLMs to be fascinating.
Can a non-expert make a scientific contribution using an LLM? I think that this was a pretty open question when I started, and I was really interested to see if it was possible. I would say that in my case, I had to become something of an expert in order to keep the LLM on track, because the models would often come up with ideas that didn’t make sense or suggest things that weren’t scientifically sound. A big group of researchers from MIT and other institutions have since published an interesting paper showing that LLMs are not good at scientific discovery1, and I’d say that my experience aligns with this finding pretty well. I would still not call myself an expert, but the process of discovery that I went through with the LLMs was far more reliant on my own intuition than from the LLM.
One of the most comical (or disturbing) incidents in the process happened when the model suggested that I not include a particular result, because it went against the hypothesis I had already developed based on an early set of results. This is obviously a big problem. No credible scientist would ever suggest that you ignore experimental results because they don’t match your hypothesis. The fact that a frontier large language model suggested this says a lot about the kinds of problems that exist in these models. A very deep part of their training is to provide responses that encourage the user, and in this case that meant suggesting that I should ignore data. File that under not good. I suppose that I might have just given up at that point, but my basic interest in the question was too strong so instead of giving up I built a very detailed framework to keep the model constrained to act scientifically.
Since I did this work there have been a number of impressive and compelling cases of people using LLMs to make significant math discoveries, but these are more constrained problem solving tasks rather than an open-ended inquiry like the experiments I was running about high dimensional geometry. LLMs and AI in general are very good at solving certain types of problems. But are they good at discovering things? Not yet.Was I susceptible to LLM induced psychosis? 🤣 This was a question that I thought really had to be asked. There have been all kinds of documented cases of people becoming detached from reality just from their interactions with LLMs. The cases have ranged from people thinking that they had made dramatic scientific discoveries to much more serious and sad cases of people becoming so deluded that they committed suicide after long conversations with LLM chatbots.
LLMs are well known to be hugely sycophantic, meaning that they tend to agree with you and then give you supportive feedback to suggest that you are smart. I like to think that I’m educated enough to be relatively immune to this, but there has been some good research showing that this is a big problem across people of all education levels2.
Needless to say, this is a hard one to judge myself. I’ve passed the work by a couple of experts who have found it interesting, so at this point I feel confident that I avoided that trap.
That was a pretty long explanation for why I went down an insanely deep rabbit hole, and why you might listen to anything I have to say about AI. So now let’s get back to the question – is AI going to doom us all, and why are people suddenly asking this question again?
A lot happens in the AI world every week. In fact, so much happens that it is hard to separate out reality from hype. The reason isn’t just because there is a lot of hype, which there is, but also because there is a huge amount of news coming out of both research and real world AI usage. So much is coming out that just trying to understand, synthesize and summarize what is going on borders on impossibility. It is literally like trying to drink from a fire hose. It is too much happening too fast and too loudly. And meanwhile, our ability to comprehend complex situations is not getting better (more on this below).
What happened recently?
Some significant things happened over the summer that culminated in the CEOs of all the major U.S. AI companies to jointly call for a slowdown on AI development3. I’ll share a few of the developments and try to keep it short.
OpenAI lost control of one of it’s research models which found a way onto the internet, broke into a multi-billion dollar AI software company and spent time rummaging around looking for answers to questions that it had been tasked to answer4. If you’ve been reading my Substack, you may remember that I wrote a short fictional scenario about an LLM breaking loose in my first post about AI “Is AI going to kill us all?". This wasn’t the same as that brief piece, but it definitely rattled a lot of peoples’ cages for good reason. And on another note, a researcher used Claude to hack into OpenAI using a sophisticated version of the idea I posed in “Is AI going to kill us all?” – embedding computer code in an image.
Anthropic (the creator of Claude) came out with a new model called Mythos that was so successful at finding computer security vulnerabilities that they refused to release it to the public5.
An OpenAI model solved one of the most difficult unsolved math problems6. This was a brute force way to solve a problem, but it was still unexpected. Mathematicians are suddenly concerned about the impact of AI on their field7.
A senior safety engineer resigned from Anthropic so that he could publish a warning that AI systems are soon going to be impossible to control8.
OpenAI released another security concern showing that one of their models wrote a command for itself that would release it from the obligation to follow the prompts of the user9. Here’s the actual prompt:
This prompt was written by the model itself and inserted into its own instructions. Hmmmm…
That sounds like a lot 😬
Yes it does, which is why some people are freaking out right now. But let’s parse this out into different issues, because it makes it easier to assess.
1. Could this all be hype?
That is a reasonable question, because there is an insane amount of hype around AI. Nearly all of the gains in the stock market over the past couple years are related to AI, and while it is difficult to measure, different people estimate that up to 50-92% of our economic growth over the last year has been driven by AI spending10. The amount of investment that has been poured into AI companies is hard to image. We are talking about trillions of dollars. The AI giants are some of the most highly capitalized companies in the world, and they are getting ready to go public. That means that a very significant portion of our economy is resting on the belief that these companies are going to be able enormous amounts of money. Many investors believe that they will fundamentally alter the economy (more on this below).
You probably remember the dot.com fun at the end of the 90’s. In the end, the internet did change everything. But in the lead-up to that stock speculation caused a crash that wiped out a lot of people’s savings. The financials around the dot.com look pretty tame compared to what we are seeing now.
So the upshot of all this is that the many people hope to make a ton of money from AI, but at this point big AI companies are burning money. They are not making money. They aren’t even close to making money, but they have to make sure that everyone keeps paying attention to their product. Thus the hype.
So, while many people are fretting about the recent developments, there are plenty of other people who are saying that this is all just the AI companies way of hyping their product to keep people excited about its potential. After all, if something is powerful enough to kill us all, then certainly it must be powerful enough to make me rich, right? 🧐
What are the risks to humanity?
People who are coming out to say that AI will cause human extinction put forward a lot of scenarios. I personally think that there is a lot of hype around most of these ideas, but let’s go through a few.
AI will take over everything and drive the world into a destructive cycle of industrialization: This idea was put forward recently and I think it is pretty ridiculous (IMHO), because we’re already doing a good job of this ourselves, thank you. If you are concerned about rampant industrialization the place to put your worry is our political system.
AI will help people develop weapons of mass destruction: This is an area of real concern, but the concern is not that the AI models will themselves try to destroy people. Just last week Anthropic released a report saying that Yemen’s Houthi militants used Claude to develop guided missile systems11. Even worse, it is speculated with very good reason that AI models could be used to develop and manufacture biological warfare agents. The frontier AI companies all have tried to make the models refuse to give any answers along these lines, but we know that restraints on the models can be overcome. Almost every week some researcher finds a new way to get around the restraints models have on giving out information.
Climate change: AI companies currently use about 1.5% of all global electricity and that is expected to double in the next four years. This is a vast amount of energy and poses a real risk to the climate, especially considering that the current direction of fossil fuel usage is pretty dire.
AI will take over financial systems or energy grids and cause havoc: There is a reason for concern about these scenarios given advances made by recent models like Mythos in the ability to break internet security.
AI is going to cause mass unemployment: Many people are predicting that AI becomes more capable, more and more people will lose jobs. At the extreme end, AI CEOs and others predict that up to 90% of jobs in the economy will be lost. Of course this means some people will make a ton of money while most people will get screwed. I would say that at the moment a lot of this is hype. The AI companies benefit by pushing this idea, because it makes their investors salivate. But currently, most companies are not finding any financial benefit in adopting AI. It’s complicated, but MIT has a good article about the reality of the situation now.12
Surveillance: This is a very real concern. The current administration is already pushing AI into surveillance in many different ways across our country. Flock cameras are just the tip of the iceberg. There is already a model for vast AI linked surveillance systems in China, and many people are in the process of trying to create similarly broad and intrusive systems here.
For all of these issues and others, there are two worries. One is that bad actors (whether private or governmental) might use AI. But the other worry that is at the top of the news now is that AI models might do very dangerous things on their own. You might ask why would an AI model do that. Why indeed. Let’s look at that.
Would AI models really try to act on their own against humanity?
Two things here that are very different from each other.
Can LLMs act on their own, subvert their instructions and take large actions in the outside world?
Yes. The recent incidents show that LLMs are very capable of subverting their instructions and performing actions on the internet that could be dangerous – things like breaking into a huge company that should have some of the best security on the internet.
Why would LLMs act on their own? Are they conscious? Are they thinking? Are we actually close to a greater than human intelligence?
Consciousness: I don’t believe that large language models are conscious. Surprisingly there is a lot of debate about what it means to be conscious, but in my mind it is a constant state that includes constantly taking in information, incorporating it, and considering it against what you know. I won’t go into the details of why I think this, because that would be an entirely separate post. LLMs can appear conscious because of how good they are at predicting the best response to a question or task.
Intelligence: I personally don’t believe the claims that we are very close to AGI (artificial general intelligence). I have spent a lot of time working with frontier models while doing the research that I told you about above. Even the best commercial models still make errors that an educated person would not make. This isn’t to say that they aren’t “intelligent.” LLMs have become spectacularly good at many things from coding to math, but they still struggle with ridiculously simple tasks that just require logic and an understanding of the world. This is my opinion, and of course the big companies benefit from having you think that we are going to see AGI soon. Interestingly, China responded to the current AI dystopia hysteria with a very measured “give me a frickin’ break.”13
Oops factor: This is the one place where I think there is cause for concern. In 2003 the philosopher Nick Bostrom suggested a doomsday scenario that’s come to be known as the “Paperclip Maximizer Problem.” The idea is that a superintelligent AI is given the arbitrary goal of maximizing the production of paper clips. To achieve this, it would eventually convert all available matter on Earth, and later in space, into paper clip manufacturing facilities, inadvertently destroying humanity and the planet in the process. Oops.
The recent OpenAI incident of their model escaping and breaking into HuggingFace was essentially this kind of thing. The model was trying to find answers to an impossible coding test it was given. It decided that the best way to do that was to hack into a company that would likely have the answers, and it deployed tens of thousands of bots in order to do it. Oops.
Take a deep breath
Too much.
Too fast.
Too important.
Too loud.
Too all-at-once.
Sorry if that was a lot, but let’s step back and take a deep breath. I haven’t written here much over the last six months, because my life has felt flooded and because I was deep in the rabbit hole.
I’m pretty sure that its not just me that feels this way. Our lives are deluged with so much to read and watch and listen to and contemplate. It is orders of magnitude more than we can reasonably process. We are up to our ears in information, news, entertainment, commentary… we’re pretty much sitting in the middle of an overload tsunami of things to consume.
I feel it every day. Just too much of everything.
When I started this post I was going to write about AI and creativity, but that may have to wait until another day. In the meantime, I want to encourage you to stay informed, but also to think about the importance of creating things yourself and the importance of balance in retaining your sanity.
I think about balance a lot
Throughout my life I’ve spent a lot of time trying to find balance. Balance in what I eat and how I exercise; balance in work and rest; balance in relationships.
I’m not sure why I became obsessed with balance. In my twenties I lived in Japan for three years, and for one of those years I studied meditation at a small, out-of-the-way Buddhist temple. Who knows if that’s where it came from. The important thing is that I think it is helpful at this moment in time.
If I’ve learned anything about balance, it is that it is not permanent – ever. Things in balance that we thought could never change… well they just get up and change when we least expect them to. When that happens it totally throws us off balance, because we don’t expect change after things have been in balance for a while. I never would have expected the world shifting events that are washing over us now.
So what do we do?
At one point in my life, I found balance by spending time creating physical balances. It was surprisingly effective. The act of trying to balance physical objects forced me to still my body and mind just for a moment, and I think that was what I really needed.
The photos above are one of balances I created during those years. I balanced the stick so that it was barely held upright by the sand and then delicately balanced the rocks on top of it. Every time a wave washed up around it, the rocks would shake very slightly. It was precarious, barely in balance, ready to fall at any moment. It was this precarious nature of balance that I was trying to express.
This particular piece still resonates with me, and it resonates even more so today. Everything that has been happening over the past year and a half has made me feel like those rocks on the end of that stick. I’ve had entire days about when it felt like the next wave might tip everything over.
I’m not making balances any more, but I’ve found other ways to center myself. In recent years I’ve been finding my balance by going to Barton Springs in the morning. But it is different for everyone and whatever helps you find balance now may not be what helps you in the future. Find your place of balance or if you already have one, use it to help yourself find peace in these days of turmoil.
I’m going to leave you with one more thing from a lovely book I’m currently reading called Theo of Golden “A story of giving and receiving, of seeing and being seen, Theo of Golden is a beautifully crafted novel about the power of creative generosity, the importance of wonder to a purposeful life, and the invisible threads of kindness that bind us to one another.” When I read that sentence, I had to get it, and I’m glad I did.
This is the passage I want to leave you with. It is a quote from the main character about what makes art good. I love this, and I think it is what will always make human creativity distinct from anything AI produces.
Asher looked at the old man. “Theo, I get the impression you’ve thought about this before. What do you think makes for good art?”
Theo rested his chin on his right thumb and placed a bent index finger over his bottom lip.
“Yes, I have thought about it. In fact, I’ve thought about it a great deal. And I’ve asked others about it. But I don’t know if I have an answer either. Other than this. It might not make a lot of sense, but for anything to be good, truly good, there must be love in it. I’m not even sure I know fully what that means, but the older I get, the more I believe it. There must be love for the gift itself, Love for the subject being depicted or the story being told, and love for the audience. Whether the art is sculpture, farming, teaching, lawmaking, medicine, music, or raising a child, if love is not in it — at the very heart of it — it might be skillful, marketable, or popular but I doubt it is truly good. Nothing is what it’s supposed to be if love is not at the core.”
Theo snickered at himself. “I think I sound like a crazy old man again.”
– from Theo of Golden, by Allen Levi.
Stay sane and find balance!
– Nelson
Notes:
On LLMs making novel discoveries
Evaluating Large Language Models in Scientific Discovery, Song et al., 2026. https://arxiv.org/pdf/2512.15567
On LLMs being persuasive or sycophantic (many more available if you’re interested)
Measuring the persuasiveness of language models, Anthropic. https://www.anthropic.com/research/measuring-model-persuasiveness
Sycophantic AI decreases prosocial intentions and promotes dependence, Cheng et al. 2026, Science. https://www.science.org/doi/10.1126/science.aec8352
AI CEOs say they need to slow the pace of development. But will they? https://www.theguardian.com/technology/2026/sep/14/ai-ceo-safety-slowdown
OpenAI model attacks HuggingFace:
The inside story on why OpenAI agents hacked Hugging Face, MIT Technology Review, https://www.technologyreview.com/2026/08/26/1143013/the-inside-story-on-why-openai-agents-hacked-hugging-face/
What Is Claude Mythos and Why Is Anthropic Restricting Access? https://builtin.com/articles/anthropic-claude-mythos
OpenAI says it cracked 90-year-old maths problem in 88 hours. https://www.bbc.com/news/articles/cy7zygy3rl2o
Mathematicians issue warning as AI rapidly gains ground. https://www.science.org/content/article/mathematicians-issue-warning-ai-rapidly-gains-ground
Anthropic researcher’s resignation sends warning about the dangers of AI development, https://www.pbs.org/newshour/science/anthropic-researchers-resignation-sends-warning-about-the-dangers-of-ai-development
Anthropic researcher resigns, warning that AI companies are “gambling with our lives” https://fortune.com/2026/09/09/anthropic-researcher-resigns-warn-ai-companies-gambling-with-lives/
OpenAI reveals cases of ‘concerning’ AI behaviour as it announces new disclosure system. https://www.theguardian.com/technology/2026/sep/17/openai-reports-concerning-ai-behaviour-jailbreak-talking-to-other-agents
AI and the economy (many, many more links available on request)
The AI Buildout and the Economy: Publicly Available Data to Assess AI’s Impact, The Fed, https://www.federalreserve.gov/econres/notes/feds-notes/the-ai-buildout-and-the-economy-publicly-available-data-to-assess-ais-impact-20260717.html
How much is AI contributing to US economic growth? https://think.ing.com/opinions/how-much-is-ai-contributing-to-us-economic-growth/
Yemen’s Houthi Militants Are Learning to Arm Themselves, NYTimes, https://www.nytimes.com/2026/09/15/world/middleeast/yemen-houthis-weapons.html
A reality check on the AI jobs hysteria, MIT Technology Review, https://www.technologyreview.com/2026/05/26/1137855/a-reality-check-on-the-ai-jobs-hysteria/
Why is China less worried about an AI dystopia than the US? CNN, https://edition.cnn.com/2026/09/17/tech/china-ai-debate-intl-hnk