> In the real world, it does feel likely that we’re going to hit some sort of a ceiling on the number of useful bugs, and probably we’ll hit it soon.
This doesn't resonate with me. I see companies adding more sloppily written features with AI. I see more bugs in the software I use, not less. While it's plausible that software is getting both buggier and more secure, I suspect those two move in the same direction not opposite.
My guess is that we're getting better at finding _existing_ security issues with AI (and thus fixing those issues), but simultaneously adding more insecure surface areas _at a faster rate_.
On one side, you have pieces like this, where seemingly there are constant fights between serious actors with large and properly distributed budgets, employing top tech and top minds; on the other - regular news of the hackz, where responsible person in charge of security with root access failed to grasp basic technical knowledge (several times), ticking every checkbox in "never do this" list from security best practices, which led to every customer being pwned.
It's like two parallel worlds, that exist in the same place at the same time, but somehow don't cross.
I started my career in the military, and got lucky enough that SOMEHOW, we convinced them to fund a trip to GDQ for educational purposes.
Anyways, while there I attended a little roundtable on software security. It was me, representing a small unit from the Air Force, some dude from Google, and like 15 game devs.
Despite only being a dev of 5 years at that point, I was SHOCKED at the lack of knowledge on software security. Even simple concepts seemed completely foreign to the game devs, though the Google dude seemed to have a really solid understanding of security.
Obviously game devs and website devs and all kinds of devs have different focuses, but it just blew my mind that out of all the topics there I might’ve been considered a comparative expert in, security was somehow the one. I wasn’t sure if that was a major plus for military devs or a major concern for the other devs, but now I’m starting to learn in the latter direction.
> In fact, the worst part about this dynamic is that these potential new backdoors will begin primarily useful for allowing the US to weaken its own systems, which will in turn allow foreign adversaries to find new ways to attack our communications. This deliberate self-sabotage will happen just at a moment when we’re finally learning how to defend our own infrastructure.
I don't understand how you can both argue for that law enforcement (and intelligence) agencies will force others to implement intentional backdoors AND also everyone will be using AI to find and secure ALL potential holes in the software so there won't be any vulnerabilities anymore.
Wouldn't one AI or another detect this deliberate backdoor and report it, as it'll look just like any other security vulnerability, the only difference being the intention?
I have respect for the author so I feel like I probably misunderstand something from the overall text rather than I somehow have a better perspective on this topic that the author knows very much more about than me. I felt like I nodded along all up until "So how is this a problem?" and now I'm not sure I understood correctly.
> This is not a call to action for experts to rally behind a sophisticated plan. Like so many things about the AI revolution, it’s just occurring to me that we’re on a long greasy slide to a place that will look different than where we are today.
heh. long greasy slide. It really does feel like that.
I dunno man, if there's a deluge of new AI generated code at all layers of the stack I think there will still be vulnerabilities.
Like if we were willing to stop adding new code and just have a small secure code base, AI could maybe help us find all the vulnerabilities in that code base.
But people have consistently been unwilling to do that. Like if we were willing to stop adding code we could have stopped decades ago and done SQLite level testing everywhere and probably have found almost all the bugs already.
When we've got people who don't know the difference between ssh and bash creating SaaS companies that generate revenue, yeah there's gonna be a lot of insecure code going out, but that same person can also tell the AI "red team my app to find vulnerabilities and then fix them", and the AI can competently actually do that, I don't know that there will be. I'm not saying that's never going to happen, but the bar is getting raised on both sides.
This "going dark" scenario would require new legislation. With secure enclaves, modern smartphones can't be cracked open in the manner that the FBI wanted in the 2016 case. So there's no such thing as "court order tech company to crack phone" anymore. It would have to be "outlaw tech companies from producing phones that they can't crack open", which is very different and does not fall under any existing US statute.
No system view. The law agencies can develop exploits to intercept our phones, that is a new, and totally unseen before threat.
Unless you remember 2013, Snowden, that nothing was done (at most was some concern about doing it to US citizens, the rest of the world doesn't deserve privacy), all US (and/or five-eyes) based web companies must disclose users information and be forced to not disclose that, and things kept going surely at a faster and more intrusive rate in everything else, and of course phones.
You are complaining being sprinkled by water while at the bottom of the ocean. At least the big companies can find their own vulnerabilities with the AI tools you mention, the rest of the doors are still wide open.
I think that one of the reasons they (frontier companies and the gov) will be putting so much effort into curtailing bugs and vulnerabilities is to limit the blast radius of future AI models. Imagine with the new Sol Ultrafast, they could have pwned Hugging Face in 6 hours and not 4 days (IIRC).
It also seems likely to me that the US Gov. probably already has routine mechanisms for compelling targeted software updates for persons of interest, so I'm not sure that a more formalized backdoor than automatic updates is going to be surfaced in the mainstream, unless that is avenue is also cut down somehow.
I'm supposed to be concerned that the US government and Israel won't be able to hack everyone's phones?
[delayed]
The reason why this might worry people who don't like the US/UK/Israeli governments is hidden in a secret place...a paragraph that is neither the first paragraph nor the last.
Tl;dr the argument is if three letter agencies can’t buy or make exploits anymore because all vulnerabilities are patched, they will ramp up legal and legislative pressure to make vendors install backdoors. Which is bad and we should all worry about.
One of the author’s blind spots here is the concept of “digital sovereignty”. The US is continuing to ban more and more Chinese-made hardware out of fear that the Chinese govt has installed backdoors in them… which you could interpret as an admission that the US does exactly that with American made products. Globalized supply chains are, erm, complex, and few if any companies are really going to be able to achieve “digital sovereignty” with hardware. But with software it’s actually plausible, though obviously hard. Whether or not the feds have actually installed backdoors in Microsoft Outlook, foreign govts are rightly concerned that they have, and are increasingly pushing to avoid US made software for simple national security reasons.
Which is to say that they will become more authoritarian and aggressive? That's not a good reason to be unhappy that they can no longer hack shit. They shouldn't do that either.
There are lots of things governments should and shouldn’t do, but unfortunately we don’t live in a vacuum, and need to consider the consequences of actions, rather than simply hope everyone is somehow forced to react the way we expect and desire.
In a nutshell, what I’m saying is “They shouldn’t do either” is at best a platitude and at worst a distraction preventing critical thought.
So our strategy should be to shut down AI so it leaves vulnerabilities everywhere? Not sure what you are saying. Fixing security bugs across the world's software isn't some single policy action that we can vote on.
Governments should behave. I know they don't, but they should, and the population should do everything in their power to force them to. What other choice do we have?
Please demonstrate your powerful faculty of critical thought and explain what we should do instead.
The trouble is, sometimes governments do things that they shouldn't do.
Read the article. That's what the article says.
> Thus: over the next two years, major pieces of software are likely to run out of remotely-exploitable bugs.
> While I think this is great, for law enforcement and offensive intelligence agencies, it’s going to be a nightmare.
> So what do we do about it? I honestly have no idea. [...] it’s just occurring to me that we’re on a long greasy slide to a place that will look different than where we are today. [We’re] just going to have to hope that this time we make the right choices.
Nope. The article’s assertion is that governments will start to put enormous pressure on tech companies in their borders to include backdoors in their systems for LEO use. This in turn may lead to “nationalized” software e.g. what Russia and China largely do now, but in many more countries. You may have no choice but to use phones, computers, and software that is backdoored by your government - everything else will be illegal.
What do you mean "Nope", you mean "Yep" right? What you are describing is precisely "more authoritarian and aggressive"
Which part of the article made you feel this way?
Right? I saw "I’m concerned that U.S. intelligence and law enforcement agencies are about to go dark, meaning lose a huge portion of their capability." and my first reaction was GOOD.
What was your reaction after reading the full article?
> In this case, we’re just going to have to hope that this time we make the right choices, for no other reason than that they’re right.
I'm more curious what could be a right choice, and more importantly who is the "we" in this, as many decisions are largely made by companies and governments.
As far as I'm concerned, the right choice is that the US government learns to live with remotely secure devices in the hands of everyone. No new laws are passed to force hardware/software makers to insert remote backdoors. Law enforcement and intelligence services have to investigate targets using metadata, publicly posted information, the myriad of online service providers who are already subject to warrants, and physically proximate surveillance.
Childish, nation-states as powerful as the US have access to much more potent stuff. Maybe they'll be forced to rely more on their Intel ME/AMD PSP/modem (cf https://redmine.replicant.us/projects/replicant/wiki/ModemIs...) backdoor and ANT James Bond catalog.
Sounds like a pretty strong argument to self host, and otherwise be in charge of the software you use.
Including your AI agents. And models become problematic. There very likely is and/or will be lots of pressure for US AI labs to make models help law enforcement. It could be by implementing backdoors in generated code, or not report some exploitable bugs, or something else. Similar for agents, they basically become the threat in your infra…
(It’s of course not only the US, just that the largest AI providers are US based and we know from history how US agencies operate)
Yep.
I'm using open weights models on a privacy-focused provider right now, and that's adequate for my current usage, but I'm rapidly getting to the point where my agent's access level to my data (and to a lesser extent, my accounts) is becoming something I'm not comfortable sending outside my network at all.
My hope is that models that are roughly on par with Deepseek V4 Flash can be run on hardware that I can own for <~$5k in the near future. We're close, but not there yet as far as I know.
The only long-term solution to this is self-hosting.
How do you think self-hosting will help in this situation?
Self host + open weight models, exactly the things that openai/anthropic don't want you to have
> Defenders are now in the process of patching every bug they can find, often with AI helping them. Entire development toolchains are being rebuilt to incorporate powerful vulnerability scanning before software reaches the testing phase. This does not mean that every bug will be found: even calculating the number of bugs in a piece of code is probably uncomputable. In the real world, it does feel likely that we’re going to hit some sort of a ceiling on the number of useful bugs, and probably we’ll hit it soon.
> Thus: over the next two years, major pieces of software are likely to run out of remotely-exploitable bugs.
His conclusion sounds extremely optimistic to me.
The number of remotely-exploitable defects is going to drop by 1 or 2 orders of magnitude. We now have amazing machines that will find pretty much all the a priori knowable ones. They outperform even the most gifted h@x0rs. So that just leaves a small pool of leetrs to scour a very barren landscape. And that pool is also shrinking as we rely more and more on the ai tools.
Perhaps we are going to go up a level with hacking done by probing the systems and the system of systems.
It all boils down to money/resources, like always.
Pre-AI, the advantage went to the entities with the largest budget to hire the best and brightest security engineers.
Post-AI, it'll go to the entities with the largest inference budget.
Right now we're in a transitionary period where it's kind of a tossup which approach is more practical, but at the end of the day - it's still all about how much money you can throw at the problem. I'm just hoping the threshold climbs high enough it's no longer practical for governments to be able to compromise individual actors' devices because doing so would waste a 0-day that's far, far more valuable than prosecuting one arbitrary person is worth.
I don't know, my colleague refuses to use AI and I've been seeing more bugs from their side, while reducing bugs on my side with the help of AI.
That said if companies want to "ship ship ship fast", then yes even AI can produce bugs or regressions if not carefully reviewed by the human.
I don't have any colleagues like that anymore, but even as far back as the last half of 2025 I was seeing that automated AI review was becoming effective enough that I considered it essential to any project where security was a serious concern.
There days we're generating multiple times more code than we were writing before. That means a similar multiple of opportunities for bugs to be introduced - so the ability to automate security review is more impactful in proportion to that.
> I've been seeing more bugs from their side, while reducing bugs on my side with the help of AI.
You should question your ability to see any bugs on YOUR side.