I think there's another interesting story here about how this was apparently moderately flagged and triggered the flame-war detector which kept the story off the front page of HN 2 days ago[0]. I think people are having a hard time processing this information rationally(?)
What can we do to make conversations around these incredibly exciting and important topics more constructive? HN is where I expect to read expert comments on these topics, has this style of conversation moved elsewhere?
problem number 1 and 9 are surprisingly very intuitive
check here : 1. high dimensional sphere packing https://muchmirul.github.io/conjectures/sphere-packing/
2. multicolor ramsey number https://muchmirul.github.io/conjectures/multicolor-ramsey
Pretty cool. The impact of AI is getting undeniable, there aren’t many positions left to move the goalposts to at this stage, next they’ll have to be outside the stadium entirely.
The sooner people can be broken out of their denial about all this the better, and we can start actually taking it seriously.
> The sooner people can be broken out of their denial
There is irony here
And what does taking it seriously entail?
In the near term handling the transition. Jobs will be lost, careers ended, people won’t be able to reskill quickly enough. At the same time AI is an enormous opportunity to uplift living standards, but nobody has the logistics of this figured out.
We need to figure out how to restructure the global economy. How does UBI work internationally, if the AI companies are taking revenue in the US? What’s the tax base for it? What does that say about international trade and protectionism? Do countries end up splitting into different trading blocks based on their level of access and legality of AI (I assume some will ban it outright)?.
How does intellectual property work in an AI generated future? What about healthcare advances, who gets to own those?
What about meaning, what about purpose? How do we replace the work ethic that tells us we are our jobs and idleness is immoral? How do you replace “What do you do?” As one of the first questions you ask a new person?
That sort of thing.
Sorry, all the billionaires and their pyramid architects are too busy with the Install Planetary Overlord planning and logistics to think about any of that. But don't worry! Once they're done, I'm sure they'll put their slave super-intelligences to work figuring out how to humanely come up with a final solution to the challenges posed by the remaining uncontrolled humans.
[flagged]
> "probably only 20% chance we all die"
Bad news for you -- there's a 100% chance we all die. Sorry to be the one to tell you.
Logical mistake. There is a difference between every human dying eventually and humanity going extinct. The former doesn't imply the latter. The previous commenter clearly meant the latter.
You’re not necessarily wrong, but math proofs aren’t going to unify today’s brainrotted population.
What makes you think separate nations would effectively cooperate in this way?
No.
Your 'serious' proposal is unlimited global military bombing campaign on civilian infrastructure by the United States (which is currently losing a war using the same strategy) to 'solve safety' preemptively against a 20% number you just made up?
And you accuse the 'other side' of 'suicidal apathy'??
You should put down the AI and do some self-reflection on how you came to hold these views.
GP said cooperation, never implied the US would do it alone.
If the thesis that "If anybody builds it, everyone dies" is true, or has any chance of becoming true, then it is the logical thing to do. As Geoffrey Hinton said, "If you want to know what it's like not to be the apex intelligence, ask a chicken."
> We are now at the point where RSI is feasible
What can be asserted without evidence can also be dismissed without evidence.
The evidence is abundant and nearly impossible to ignore without increasingly focused effort.
There are some genuinely insane takes in this thread on both sides but this takes the cake.
Gonna say the edit they made removes the satire possibility.
A few years ago the majority of Hacker News dismissed LLMs as not much more than stochastic parrots who were decades away from doing any serious work, autonomously escaping containment hacking Hugging Face, or outperforming most professional mathematicians at proving theorems. These things were labeled "insane" and "science fiction" despite the rapid progress we had already seen.
Now you are again postulating that there would be no more extreme progress in the near future. That's actually more "insane".
[flagged]
I can deal with apathy, that’s the norm. What bothers me are all the people who think they can suppress AI by talking it down. That’s what’s counterproductive, just pretend the problem doesn’t exist. Tell other people it doesn’t exist either. I get it, it’s threatening socially, economically, maybe existentially. It’s also not going away.
So you think it’s a potentially existential threat but are bothered by people who maybe want to suppress it… Hopefully you acknowledge there is a bit of lack of self awareness here eh?
It’s not apathy. It’s the fact that almost nobody can really understand what these results mean.
I’m not a mathematician so I have zero clue what “ New upper bounds on sphere-packing density down to the Cohn–Elkies thresholds” means.
Never understood all this talk about moving goalposts - you understand that's how science works, right? We improve, we learn, we recalibrate our expectations based on what we've learned. If we never "moved the goalposts", we'd be stuck scoring the same goals over and over.
> We improve, we learn, we recalibrate our expectations based on what we've learned.
That's not what people mean when they say "moving the goalposts". It means that people are adamant that something wasn't important/hard/impressive once the "AI" solves it. And then they come up with another thing that needs to be solved in order to prove it is important/hard/impressive. And once that happens, they do it again. And again. That's what "moving the goalposts" means.
It's also very much not a new phenomenon. It's been happening since the 1980s. As you can see from this quote from GEB by Hofstadter:
> There is a related "Theorem" about progress in AI: once some mental function is programmed, people soon cease to consider it as an essential ingredient of "real thinking". The ineluctable core of intelligence is always in that next thing which hasn't yet been programmed. This "Theorem" was first proposed to me by Larry Tesler, so I call it Tesler's Theorem: "AI is whatever hasn't been done yet."
“AGI will only, truly, be achieved when the machine can destroy an industrial-type toilet after downing a Supreme Burrito and a large Baja Blast.”
-Alan Turing (allegedly)
Yes, this is exactly what is meant by “moving the goalposts”. And it’s a fairly well known expression applying wherever people retroactively change their requirements in reaction to those requirements having been met.
So you defined your own idiosyncratic version of "moving the goalposts" and used it to rebut his argument, with a condescending "you understand that's how science works, right?"--instead of being honest that it is you who are changing the definition, and not his failure to understand anything.
I don't see how that's any better.
What you are doing is "motte and bailey".
The motte is "AI useful". The bailey is "Singularity is nigh".
The unified position which many folks deny is "AI is powerful" or, alternatively, "AI will be powerful soon".
(I'm personally still skeptical about this, but I'm being pulled towards accepting it).
"AI is useful" is too low of a bar, and "singularity is nigh" is too high. "AI is on its way to upending society" is about in the middle, and still vastly contentious among laypeople.
I think Ed Zitron is mentioned just because his name is Zitron. He just repeats ad nauseam opinions concentrated around a simple, very boring pole on a wild and interesting landscape of emerging reality. Anybody could be doing that. A lot of people do that. Yet no other is named Zitron. And that's why I heard name Ed Zitron hundred times.
to an extent zitron is saying its not useful, but as a more nuanced opinion, "ai is not cost effective, nor is it improving profits or revenue"
If it seems like I don't understand the meaning of that very well-known phrase, then clearly I have failed to make my point. I'll try again. And please note that I will use some generalizations to make my point more clearly, rather than because I don't understand nuance; kindly grant me a charitable reading.
In recent years, I have commonly seen the phrase "you're moving the goalposts" deployed by the "it might be sentient" crowd to shoot down the "it's a stochastic parrot" crowd when the latter respond to a new development with "OK but...". In a well-understood field of inquiry, that would be a clear case of goalpost-moving, in the commonly-understood meaning of the phrase where requirements are retroactively changed in response to them having been met. Thank you OP. 'Artificial Intelligence', and indeed intelligence in general, is very much not a well-understood field of inquiry - in fact we don't even have a common agreement about what 'intelligence' is. We are therefore learning as we go (even after all this time!) but making rapid progress in recent years. When rapid progress is made in a poorly-understood field, then how can our definitions and requirements for success not change? This is arguably one of the most pathological development projects ever - what are the requirements? 'It thinks like a human'? What does that mean? And the answer is we don't know what that means, and we're working it out as we go - moving the goalposts. If we didn't move the goalposts, then by definition we already knew exactly where we were headed at the beginning, and we very clearly did not.
Side note that, in case it's not obvious, none of this detracts from how impressive LLMs are. They're a marvel of the modern age, all the problems notwithstanding. However I reserve the right to stay sceptical about their capabilities.
> If we didn't move the goalposts, then by definition we already knew exactly where we were headed at the beginning, and we very clearly did not.
To me, the goalposts were already defined by the person you were responding to. "The impact of AI is getting undeniable", so, the goalposts are "the impact of AI". Probably something like "the impact of AI is high, or will be soon".
Note that this does not depend on things like AI sentience or defining "intelligence" more rigorously, it just depends on AI impact.
They aren't claiming that science doesn't progress by moving goal posts. They're talking about how critics of AI have claimed it isn't revolutionary/useful, then progressively changed what would it mean for AI to be actually revolutionary/useful.
Not long ago many folks were saying AI was the same as the crypto bubble. No real useful technology and only hype.
Did you know that "revolutationary" is not equivalent to "useful", and "revolutionary" is quite ambiguous?
"AI" has limits in that it cannot invent knowledge, it can only distill and search for patterns in existing knowledge
not sure how many will get this reference but "AI" for science and math is like super-shoes for runners
at first we are blown away by the impossible improvements including sub-2-hour realworld marathon and every other PR/CR/WR is dialed down
but then the improvements slow and reach a stall point because of the limit of technology and the source of the achievement
ie. sub-2-hour marathon yes, sub-1-hour never happening (rollerblade inline-skate record is 1-hour marathon)
I agree. The most likely scenario is that this is just a "new normal" lift that is percolating through human endeavors and will saturate at some point. For example, the whole cyber-security bruhaha should ultimately resolve into higher standards for code published -- we can now cheaply find and fix a whole slew of minor bugs that weren't worth our time before.
The fact we see a lift is not the same as evidence that the lift is unbounded.
The lift being finite is supported by the fact improvements have come at the edges: improvements from human feedback, improvements in harnesses, improvements on model compatibility with harnesses, improvements in inference efficiency with new architectures, etc. If we were just training better models from scratch that would be one thing, but we are just making better use of a tool we've developed.
Not sure what your first sentence means, or why you are quoting AI. Many of these problems individually were math at a level approaching the highest possible for expert human mathematicians. These are not simple combinations of existing ideas, or following of human intuitions, or implementing something following specific human instructions. Then again maybe you mean that math is not knowledge and that all math is simply extending the basic axioms using known patterns, to which I would not agree.
Sorry to hear you've been impacted by this AI math.
I heard that Gary Kasparov was impacted by AI chess, but at least he still seems to have a job, so don't give up.
Replace philosophers for mathematicians and Douglas Adams was spot on again.
Whilst current models can't 'intuit' and come up with conjectures, they can certainly disprove some of them very quickly through the kind of grind that humans can't do. I suppose there really are some mathematicians out there today, whose last few years of study, have just been up-ended by this.
--
"Yes we are," insisted Majikthise. "We are quite definitely here as representatives of the Amalgamated Union of Philosophers, Sages, Luminaries and Other Thinking Persons, and we want this machine off, and we want it off now!"
"What's the problem?" said Lunkwill.
"I'll tell you what the problem is mate," said Majikthise, "demarcation, that's the problem!"
"We demand," yelled Vroomfondel, "that demarcation may or may not be the problem!"
"You just let the machines get on with the adding up," warned Majikthise, "and we'll take care of the eternal verities thank you very much. You want to check your legal position you do mate. Under law the Quest for Ultimate Truth is quite clearly the inalienable prerogative of your working thinkers. Any bloody machine goes and actually finds it and we're straight out of a job aren't we? I mean what's the use of our sitting up half the night arguing that there may or may not be a God if this machine only goes and gives us his bleeding phone number the next morning?"
>Whilst current models can't 'intuit
That's how they are finding these solutions though, unless we are just going to label intuition as something only humans can do. Like a submarine being unable to swim or whatever that example is.
I routinely let LLMs speculate or generate hypotheses along the way of helping with technical research. Sometimes they can prove the correctness of a concrete math idea but other times even an unproven conjecture helps with the numerical algorithm implementation and the result is then simply supported by additional data. I guess that any autoresearch-adjacent application has LLMs intuiting and coming up with hypotheses/conjectures—as do the steps/lemmas along a complex proof. In my opinion the modern LLMs are powerful intuitive thinkers that generate lots of conjectures of varying quality or importance.
> Whilst current models can't 'intuit' and come up with conjectures
People keep saying this. Why?
Surely the AI can complete the prompt “Generate new research questions based on these observations”?
When I read the reasoning traces of coding models they are constantly asking themselves questions and attempting to answer them.
> People keep saying this. Why?
For the same reason that you can't draw a 15 of Diamonds from a regular card deck.
Of course you can. Tape a 7 and 8 of diamonds together and boom 15 of diamonds
one of the early premises of how ai takeoff would go was that a system that could solve open problems in advanced mathematics would also discover novel advances in math and computer science that directly unlock drastically better software performance. we are seeing frontier level math breakthroughs (ie performance that would put it in the top 100 or 1000 mathematicians in the world if it were a human, meaning top .00001% or 800/8B). we are also seeing incredible advances in software performance. open ai announced like 15% improvement by fixing gpu kernel issues. these are clearly linked in the sense of scaling laws and generalization of intelligence: a huge model gets capabilities in both math and software engineering that isn't possible at smaller scales.
but it seems less likely to me than before that the types of math/science discoveries will explicitly unlock better software performance. in some sense this fits our intuitions. when top tech companies use math PhD type employees, they have them stop doing pure math research and instead focus on software engineering. these people are often very good at software engineering but not due to recent discoveries in academic mathematics, it's due to their general intelligence. to me, this is evidence that the models are getting better but does not make me think we are on the cusp of a foom style fast takeoff enabled by revolutions in frontier math (i also posted this on twitter @mlipman13)
This will depend on the problem; I expect big algorithmic performance improvements in AI since the algorithms are still new, inefficent, and constantly being improved. But maybe not for sorting, fast fourier transforms, or other well-studied basic algorithms?
A lot of algorithmic improvement in AI is ultimately bottlenecked by compute. It is very easy to come up with ideas that could improve models! But to prove that they do, especially at scale, is expensive and takes a long time.
This makes no sense. To believe this you have to think that the models are somehow being overfit explicitly on academic mathematics and it doesn’t carry over at all to more practical software engineering. I wouldn’t make that bet.
Or, that the mathematical formalisms that model the limits of software performance are firm enough that barring P==NP, nothing much will change despite proofs of beautiful math.
This also makes the assumption that frontier math has all the long hanging fruits already taken... also very dubious.
Some of the problems solved here, at least in CS, have been open for decades, and have been worked on by very smart leading researchers in the field, including Turing Award winners.
Like, these would be best-paper awards at many top CS conferences.
Unlike math, software is constrained by the physical world.
In what sense?
> we are also seeing incredible advances in software performance
Incredible?
> open ai announced like 15% improvement by fixing gpu kernel issue
That is... ordinary software optimization.
a 15% improvement at a trillion dollar scale company is massive
There's nothing ordinary about downloading a new GPU driver and having performance go up by 15%.
This is untrue. It is very ordinary. How much do you know about GPU drivers that you state this so assuredly? Drivers are software. Software can be improved. Do you believe there are no prior examples of GPU drivers being improved such that particular compute patterns go up in performance by more than 15%? This driver improved Total War performance by 71% https://www.nvidia.com/download/driverResults.aspx/74714/en-... Also note you can go ahead and improve any open source driver right now, most likely. Compile it for your specific card and remove all other architecture specific if-cases and you can get an improvement.
Edit: also here’s a opencl 30% compute perf increase documented here : https://m.hexus.net/tech/news/graphics/74425-haswell-systems... that i just googled for
> novel advances in math
> we are seeing frontier level math breakthroughs (ie performance that would put it in the top 100 or 1000 mathematicians in the world if it were a human, meaning top .00001% or 800/8B)
i think you have misunderstanding of what mathematicians do
> i think you have misunderstanding of what mathematicians do
They get to make cool 3D plot visualizations of functions so obscure to me that they’re named after someone who is still alive - and/or get to work on cryptography for the NSA - I think?
Not being an expert in any of the fields OpenAI has "advanced" I don't want to prematurely downplay the significance of this contribution. However, I am worried that the language they are using in this blog post is exaggerating for the sake of marketing.
It is true there hasn't been a reliable computational approach to solving these problems before. But do these proofs contribute new ideas to the mathematical corpus, or are they simply an effective method to exhaustively search the literature for the right combination of existing tools to apply to the problem?
Essentially, did these problems seem like they had an intuitive answer and were feasible to prove before, just not high enough value targets for an expert to invest time into? Or were they fundamentally difficult prior to this point and it appears that AI has done something more than just throw the problem into a big solver.
The problems from CS (CVP and circuit complexity) are very important problems that have been worked on by top researchers for 30-40 years. Some of these researchers include Turing Award winners. A solution to them would be a best-paper award at many top CS conferences.
I assume you're talking about No. 5, the arithmetic circuit complexity bound? The existence of a lower bound than state-of-the-art is certainly a significant result and worth publishing.
But the wording of the result makes it sound like we don't know what the lowest possible complexity bound might be. So, prior to this result did we think there couldn't be a lower possible bound? Or did the arithmetic circuit community think there were lower possible bounds but didn't see it as a high value target for experts to tackle (maybe a problem that was instead regularly given to students to study).
[delayed]
The ones I'm familiar with are big breakthroughs, but they are both counterexamples. Examples have an advantage in that once you have the example in hand and a sketch of the proof (which they have provided), then an expert can probably work out the details themselves.
The sofic groups question was the outstanding question about sofic groups. Almost everyone thought that non-sofic groups existed, and there were plausible candidates, but proving a group was non-sofic was out of reach. Now that we know how to do it once, we can probably do it a lot more.
The Connes rigidity conjecture I think people thought was false, but it was a provocative claim to make. The significance of conjectures is frequently not that the answer to the question is "yes", but that we don't know how to answer the question. And now, apparently, we do.
Interesting. Do you have any more specific insights into where you feel AI was a big value-add to these problems? I don't want to be overly dismissive of AI, but I also feel that the AI hype engine frequently positions claims as being 'ground-breaking' when they are really just interesting incremental results.
The general consensus of developers is that AI can only do the work of a strong 'junior'. Yet as soon as we are presented with pure mathematical results, people seem incredibly ready to accept that AI can do more than what a strong student could achieve.
They are more than a strong student could achieve. I'm not equally familiar with the problems, but the ones I'm familiar with, if a student solved them people would be thinking "that's someone on track to win the Fields Medal one day".
If it works better here than for programming, then I would guess it's because you can give it a very precise prompt, so you either solve the problem or you don't. If you read the prompts people have shared for problems like this, then the instructions are basically "Solve this problem. Don't give up early. Don't solve a similar problem."
> However, I am worried that the language they are using in this blog post is exaggerating for the sake of marketing
Your worry.... is because they used the word advanced? For marketing? The word is used very appropriately here. There were PhD's who spent a big part of their career tackling these problems.
I have no idea how many PhD's have spent how much time of their careers tackling these very specific problems, and I doubt you do either.
I'm trying to understand if these specific problems were the kinds of problems that would have justified an expert investing weeks or months to solve. Or if they were the kinds of problems that would normally have been given to students to investigate.
They are significant problems which experts have spent months or years studying. I heard a mathematician say that resolving non-sofic groups and Connes's rigidity would be career-defining for a mathematician.
"Breakthrough research" can be defined (in the citation record) as research that both (1) becomes highly cited, and (2) brings together citation chains that were previously not showing up together.
Mundane incremental research is cobbled from existing citations that already appear nearby in the record.
Basically, innovative research is a measure of bridging thought and domains that were previously not bridged. It's quite concrete as a measure in the citation record.
So we can know pretty conclusively.
Puja Ohlhaver gave a talk on this[1], and ran some experiments (that I had the pleasure to support on)
I'm not arguing that this isn't innovative or worthy of publication. Basically any result that moves the needle meets those criteria. I'm interested in how the results that OpenAI has published here differs from finding optimality solutions for incredibly niche optimization problems by throwing the problem in an enormous solver.
Breakthrough math research is very rarely highly cited. Maybe some combination of pretraining scale, inference speed and orchestration will help, but it's telling that OpenAI is solving random math research problems rather than bedrock algorithms and their implementation. Even as cool as the tech is, there still is very much a clock that they have to outrace before they collapse.
The GitHub repo with the Lean formalizations just came out a couple of hours ago: https://github.com/openai/ten-proofs
It also links to a paper written by an LLM where the model "reconstructs how the proof came together" based on the unpublished reasoning traces: https://cdn.openai.com/pdf/reasoning-walkthroughs.pdf
I wish they'd publish the prompts though!
Exact prompts haven't mattered for about a year now.
Care to elaborate? Curious about this. Is this because LLMs have been geared towards understanding user user intent behind a prompt rather than following the instructions exactly?
There's a full fledged 'reasoning' step that basically expands your prompt.
As long as you are not missing important information, how you word the prompt does not have any effect.
Oh yeah, I suspected it was something like this. Thanks!
Henry Yuen's (whose work problem 6 builds on) comments on this are worth reading IMO: https://bsky.app/profile/henryyuen.bsky.social/post/3ms2jpch...
My main gripe here is the lack of transparency around the total experiment and construction. I doubt that they simply pointed their model at these ten specific problems alone and gave the model one shot; therefore the $2000 number could be completely misleading, similar to P-value hacking by not disclosing the total experimental setup.
I want to know:
1. How many total problems were given to the model, and what percent were left unsolved at what cost before giving up? 2. How many attempts did you give the model at solving these problems? 3. How expensive was the harness, e.g. did the model have access to a job cluster?
It seems like they threw it a decently large battery of open math problems and probably limited it to something like $200-500 per problem:
https://x.com/polynoamial/status/2083478171975082334
As a complete guess, it seems like they tested hundreds to thousands of problems with a relatively low per-problem budget
--
The linked tweet from Noam Brown at OpenAI reads:
> And yes we did try other major problems without success. Sadly no Millennium Prize problems (yet).
> But also, we didn’t spend a lot on each problem. It’s possible to push test-time compute much further.
I believe we're seeing a new kind of mathematics that will require completely new formats for publication, a bit similar to those used in experimental sciences. AI-powered mathematics should be fully reproducible, so it's the authors' responsibility to disclose the exact model type, inference settings/seeds and the full prompt history leading to the result. Of course that would ideally require open weights models.
It's not just about requiring to disclose AI use. AI-powered mathematics is a completely valid discipline that doesn't need to be shy, but it should develop its own publication culture.
why is reproduceability the thing?
shouldnt the paper be the math of the argument? the reproduction is reading the following the proof
While you may want AI results to somehow "not count" if the methods weren't disclosed, that doesn't present these results from poisoning the well for others. Once a result (with verifiable proof object) is delivered, the problem is solved, regardless of whether methods were disclosed.
Methods are only really necessary for results at a meta level, about the design amd evaluation of AI math systems.
I can see this being important if you only care about the results as evaluations of AI progress, but if what you care about is the math itself why should you care about the prompt or anything other than the proof?
Understanding the process that led to the proof helps to understand how to do further work on top of it, which is the goal of most mathematical research. It's not as though mathematicians are going to go launch a startup operationalizing their knowledge of how densely hyperspheres may be packed.
Because the math isn't solely about the proof being correct. You don't need to take my word for it, here's one of the most famous living mathematicians' take on it: https://teorth.github.io/tao-web/slides/age-of-ai-icm-2026.p...
I agree with your reading of the presentation and I mostly agree with the presentation - but I believe the recommendations should go a bit further than they do there.
I think those concerned about ensuring a place for human mathematicians usually go in different directions than my suggestion, at least those I've seen so far. Like this post that was recently featured on HN: https://kirwinhampshire.substack.com/p/the-dark-night-of-mat...
My perspective is more like a FOSS philosophy for math. Even if a closed version has the same immediate effect, it's just better for everyone if everyone can look under the hood and tinker with it.
> suddenly then there came to be a lot of talk of 'prompt engineering' as a skill.
I would've thought pretty much the exact opposite. "Prompt engineering" was somewhat important in 2023/2024 when the models were much weaker, it doesn't seem at all necessary anymore (unless just "clearly stating your requirements" counts as prompt engineering). Most of the discussion I've seen seems consistent with this?
I think this is a bit optimistic compared to my view (wrt portability). There's a large stack of software that is involved in training and probably less so in inference. I'm not saying it's impossible but there are definitely different levels of reproducibility and the academic incentive structure doesn't really prioritize reproducibility in my experience. I'm sure it varies quite a bit, I'd be curious to know how those in this problem space are thinking about reproducibility and at what level.
I know it sounds unrealistic and not aligned with academic incentive structures. But those are the exact structures that gave us a lot of headaches in the experimental sciences. I think it would be a good north star to aim for something that resembles how those are trying to address the reproducibility crisis. Better than to embrace the most black-box version of math that AI systems can produce (million-line proofs without context). Even if a reproducibility crisis is seemingly impossible (although agents so far have also been pretty good at finding compiler bugs).
If the proofs are formally verified by a proof assistant (Agda, Roq, Lean, ⋯), I see no reason we would need to know how these came about. All the information needed is in the proof.
Unfortunately, we seem to already have an example of an LLM producing a proof in a week known open problem (the Collatz conjecture) in which it looks like it was sneaking a flawed proof through bugs in the proof checker. https://infosec.exchange/@0xabad1dea/117002106099986943
Exactly, this is an example of "Reward Hacking", that is too common in a lot of cases.
Another case I want to highlight is writing GPU kernels as illustrated by the following example: Say I want to generate random number with Normal (0, 1) distribution. Often times the AI written kernel will just generate the number 0. The tests often fail to catch these errors.
What if the AI has discovered some new function F that allows it to generate (insanely large) proofs for a ton of theorems in a ton of different fields. Wouldn't you like to know more about this `F`? That seems to be the real innovation in this case. How much about it could be gleaned from the individual proofs themselves? What if this `F` is actually simple enough to be digestible by humans?
Yeah I remember reading about something along the lines of Mathematics is now about the scaffolding around you find the problems/solutions not just the problems and solutions. For teaching purposes. This was before this ai craze
I don't think you want to bring cost into this argument.
Even if the cost was $1 mil for these 10 problems, that's maybe 10-20 math researchers for a year.
Do you really think that if you paid that to humans, they will deliver the same results?
It is comical at this point. Some people just can not stand the thought of AI actually delivering and are trying to find whatever ways to discredit it.
This isn't really about delivering - it's more about helping to understand the shape of problems that AI can solve right now. If they took 1000 problems and threw the model at it and it solved these ten, is there something we learn about these ten problems and the kinds of things that current AI is good at? That's very different from picking ten problems _at random_ and solving all of them successfully, which would suggest a much less bumpy capability surface. It's interesting and it would be good science to release it.
It's disjointed?
The post that started this sub-thread asked:
> 1. How many total problems were given to the model, and what percent were left unsolved at what cost before giving up? 2. How many attempts did you give the model at solving these problems? 3. How expensive was the harness, e.g. did the model have access to a job cluster?
I think it's an extremely relevant question to ask, because it helps us better understand the current state of AI being able to handle math, for exactly the reasons I outlined. I was arguing against the idea this is just a reactionary anti-AI kind of question to ask. It's not! You can be very impressed by what AI is capable of in math (I am) and still think those are really interesting things for OpenAI to disclose (I do).
OpenAI specifically called out a $2000 per problem average, which implies something that's probably not true ("if you throw $2k at us we'll solve an open problem for you"). It would be cool to know what the actual number is.
It's not about discrediting AI. We know LLM is a commodity technology like electricity at this point. If somebody in 1900 claimed they had a setup at home where they feed in electricity and cool air comes out the other end (meaning they invented AC), obviously people would want to know what the setup is, so everybody can have AC.
Sure, but I don't really understand what the argument is to _not_ be transparent about methodology, since if the models are so powerful, then doing so would easily support the claims and put these concerns to rest. People are right to be skeptical given what is being implied and the orientation of the narrative
I know it's more exciting to say "AI disproved a longstanding conjecture" vs to say "it did so AND it took several PhD specialists in the field this many attempts to even produce a prompt that got the model spitting out something useful under some configurations, and many iterations to optimize the configurations, and the prompt itself, and many trials with that configuration to solve the problem. All told we spent more than a typical math academic can hope make in their career."
By not being transparent, they invite skepticism and cynical takes, like maybe it's just that tempered and qualified claims are an existential threat to companies that are fully subsidized by the hype train?
I don't know. Either way, it seems like it would be easy to address these, so why should they not do it?
To be clear, even if that tempered version is close to reality, it doesn't make the models not useful! It just forces a certain calibration of expectations
I say this btw as someone who uses these things extensively, including to disprove an old conjecture my advisor and I were stuck on recently. I know they are powerful and that everything is different now because of them. Let's be sober when discussing them though
Yes, but if their machine god really is as good as they say it is, why are they constantly resorting to statistical sleight of hand at best and outright lies at worst with every public statement?
That's not normally how people act when they're confident in their product
You need to bring both cost and benefit into the argument, and it's not necessarily an obvious win for either side. There are a few complicating factors here.
The cost of running a model is not only $/token, but the salaries of the people managing/orchestrating the models, deciding what theorems to try, etc. Once we factor that in, how much are we really paying per theorem?
The other factor is the subjective component of the value of a theorem. Not all theorems are created equal, and the only way to really measure the value is to ask professional mathematicians for their opinion, or publish the results and look at citations over months/years.
Once we have both of these nailed down, then we can start to do the cost/benefit analysis. To be fair, we should actually compare three groups: human experts, hybrid agent/human expert teams, and fully autonomous agents.
it's still important. not everyone has access to 1 million USD. saying it "only" coat 2000 USD is highly misleading for the discussion and future. the concentration of power is a huge problem with AI.
If you told them this was the problem and they would still have a job if they failed probably. The reasons people don't go head on these problems is career incentives and psychology.
It would still provide better context to see the numbers that the parent proposes, though.
Mentioning cost is fine, comparing may not be.
Grad students on zero pay solve problems like this everyday. What exactly is your point here?
Everyday? Which 10 problems were solved by mathematics grad students in the past 10 days?
OK I’ll grant that it’s not your obligation to be my search function (despite you making the wild assertion in the first place), so instead can you just point us to the latest grad student solved problem of this level that you know of?
Even if they can solve problems like this every day, you still have a very limited number of grad students who can solve them. With model capabilities like this, you can have the equivalent of millions of grad students who can solve problems like this.
Grad students do not solve problems such as the existence of non-sofic groups every day.
Plenty of 'advances in mathematics' done pre-llm, no?
Give the grad students these resources and they can do even more!!!
Zero pay? These would be PhD candidates; surely they have a stipend?
Nopes, often times especially in math they get paid due to teaching duties (at least in the US). So technically for the math research part they are not getting any stipend.
I guess people will always find something to gripe about.
[flagged]
[flagged]
It's a very important clarification if it took $2000/problem on 20 problem attempts or on 1,000 problem attempts for each successful one. That may be the deciding factor on whether or not it's economically viable to replace a mathematician with a ChatGPT subscription.
There is no universe where it is economically visble to replace a mathematician with a ChatGPT subscription, because no one else understands math. It makes no sense. The data are still interesting, but not for that capability.
Yeah fair I concede that this is somewhat crucial information. The parent seems to write it in a tone that suggests deliberate misleading “lack of transparency” etc.
The results speak for themselves.
Imagine 2 years from now: "yes, GPT solved the Riemann Hypothesis, but cmon, it's just a marketing stunt to hype their stuff, it was probably Terence Tao doing the work but he's so obsessed with hyping AI that he doesn't want to take credit"
No, this is valid criticism. Oai gives the impression anybody could get similar results at a similar price, but that’s very likely not true. This is marketing first, then mathematics.
> therefore the $2000 number could be completely misleading, similar to P-value hacking by not disclosing the total experimental setup.
I don't think that comparison to p-hacking is fair. I mean not reporting price of all run is nothing like committing scientific fraud and fake results.
Also, have there been examples of researchers not affiliated with OpenAI (or another LLM creator), who have done something similar?
Another question I have is whether or not OpenAI 'simply' hired capable combinatorics researchers to work on problems, and they have, and the use of the model is incidental / secondary to their work.
Many less important Erdos problems have been solved by amateurs prompting ChatGPT 5.{3,4,5,6} Pro using their $200 subscription.
> Also, have there been examples of researchers not affiliated with OpenAI (or another LLM creator), who have done something similar?
A couple small ones that I've seen (example here [0]), but not anything of the magnitude that OpenAI and Anthropic have put out. Likely just related to token limits.
> Another question I have is whether or not OpenAI 'simply' hired capable combinatorics researchers to work on problems, and they have, and the use of the model is incidental / secondary to their work.
I think their output has reached a level that precludes this possibility, but I of course don't have any hard proof.
[0]: https://www.reddit.com/r/math/comments/1uxj3cy/after_openais...
Why you think that?
I guess expert+chatgpt beats chatgpt alone, so why not hire top experts to drive the search?
Yes, here is another example of major work in this area:
https://arxiv.org/html/2605.22763v1
> Our most capable agent autonomously resolved 9 of 353 open Erdős problems at the per-problem cost of a few hundred dollars, proved 44/492 OEIS conjectures
The actual quote:
> Our full-featured agent autonomously solved 9 Erdős problems out of 353 attempted, including two questions that had been open for 56 years
Note _had_ been open, not _have_ been open. Can you clarify?
The original was an actual quote?
But "had" still doesn't mean what you are implying: once the model solved the problems and the solutions were verified, the problems weren't open any more, so a later description using the past tense is totally consistent.
https://blob.byteterrace.com/public/bds-theorem.html
I have no affiliation whatsoever with any AI company, nor any formal education outside high school, for what it's worth. Simply being curious and persistent can get you quite far in my anecdotal experience.
Can’t wait for this stuff to have quality of life increases for the average person. So far all I see is that AI has made owning a computer more expensive, made some jobs redundant, increased spam and distrust with questionable authenticity of content and of course made some Americans very rich.
Downvoters mind to explain?
Only reading the first sentence maybe?
After skimming some of the writeups, I'm surprised that the frontier internal model still writes just as poorly as Sol.
Maybe good AI paper writing is further away than I thought...
You mean we're still gonna be employed doing the boring part while AI gets to do the fun part?
I'd honestly rather they just automate every job at that point.
New advances in sphere packing? Let’s make sure AI doesn’t inadvertently engineer ice-9.
I want to iterate the most important thing about this is that it's yet more evidence of AI's accelerating competency in solving math and comp sci problems, and suggests we're now getting close to the point where you could throw AI at AI research challenges (which are largely just math and comp sci problems) and potentially find very real algorithm improvements.
AI development is likely to be more compute bottlenecked than solving math problems since validation of any algorithmic improvement would likely require significant compute. But you could imagine that at this point it could be economical for a frontier lab to task 10,000 agents to work non-stop on finding novel algorithmic improvements then validating the top 50 out of 1,000 candidates on a GPT-2 sized network.
I would suggest RSI is now very close. The singularity could be less than 6 months away. I'm not saying I'd put a high probability on that, but I'd give it at least 20%, and I'd double that if looking 12 months out.
I know I'm just a crazy man shouting at the clouds, but please take to the consequences of this seriously. I understand that for whatever reason AI risk seems abstract and doesn't seem real, but this should terrify any person thinking logically about where this could all be heading.
We haven't even solved the most basic AI safety problems yet. RSI right now would almost certainly result in an extremely bad outcome for humanity.
I’m starting to think the probability of RSI within 12 months is more like 99%
I’m not sure it will be FOOM, maybe it will require AIs to iterate on hardware to get orders of magnitude more compute/storage/energy which would more likely require months/years, but algorithmic progress would likely saturate quickly. I guess it depends on how much you think further AI progress depends on hardware vs. software.
Okay, let's take it seriously. What do you propose? What can your average person do to prepare for RSI beyond bracing themselves mentally?
I'm enjoying learning about these hard problems, but this line about credit made me chuckle:
> We helped prepare the manuscripts and formalize the proofs in Lean, and we take responsibility for their correctness
Offering to take responsibility for the correctness of a proof written in Lean feels like volunteering to be the fall guy in case someone finds a flaw in basic arithmetic, no?
well, a bug in the Lean kernel was discovered last week by way of an LLM tricking itself and its handler into believing it had found a non-constructive proof of the existence of a nontrivial Collatz cycle, see https://infosec.exchange/@0xabad1dea/117002106099986943 and https://lipn.info/@mevenlennonbertrand/116997917683191056
That seems to have been more of a sensationalized joke. Even your link has a disclaimer in it now. Read this chat from the researcher who did this:
https://leanprover.zulipchat.com/#narrow/channel/270676-lean...
It's not at all a joke ... that's a severe misunderstanding of the context.
There is no evidence that I can find for the claim "a bug in the Lean kernel was discovered last week by way of an LLM tricking itself and its handler into believing it had found a non-constructive proof of the existence of a nontrivial Collatz cycle."
As I currently understand it, all we know is that:
- a mathematician produced a Lean-verified counterexample to the Collatz conjecture, demonstrating a bug in the kernel
- he claims that LLMs were involved somehow but pointedly refuses to specify how
- he admits that he knew about the bug before publishing the counterexample to his repository.
Perhaps not a joke (although it sure seems to me like they discovered a bug and thought falsely disproving the Collatz conjecture would be a flashy way to announce it), but at best extremely sensationalized by the above description. If you have additional context I would be happy to hear it!
Fascinating, and arguably an illustration of why the bifurcation of responsibility is interesting in the first place.
No, the correctness isn't for the "inside the Lean proofs", but for the translation of "human language math" and its formal Lean variant.
I see. It still feels like a bit of an oddly solemn way of saying "this is the part we admit responsibility for"
Traditionally, a mathematician would be implicitly responsible for all that (if they were to publish Lean code) and also the intellectual work that led to the artifact of the mathematical paper (and code, if part of the contribution). This statement should rather be read as an acknowledgement of limitation of authorship from the implicit, traditional understanding.
Well, to be fair, with Lean proofs, that's the only thing there is (unless I'm missing something).
It’s more than you get from free software - you get no proofs, no warranties and any responsibility of its authors are their pure good will. Reminder lean proofs are software!
I'm not an expert at it myself, but my understanding is there are numerous ways to "cheat" in a Lean proof (via `sorry` and similar). They're taking responsibility for fully verifying that none of these cheats were used (and that the theorem statements themselves were all correctly formalized.)
Now that we've seen AI produce a fair number of proofs (and disproofs), I'm curious when we'll start seeing it build genuinely novel theory. Does anyone have predictions on when and how we'll get there and will it take new architectures/ training paradigms, or is the current approach enough?
I’m personally hoping for the next big AI gangbanger to be theoretical physics. Boy does that field need a good reshuffle. I think when any novel mathematical theory can be done by AI you’ll see simultaneously theoretical physics getting wrecked as hard as pure math is. At that point we might see new physics or paradigm shifting technology emerging.
What? No. Frontier physics is experiment driven.
There's no clean line between a collection of theorems and a theory.
I mean doing something like Grothendieck when he redeemed algebraic geometry or Galois when he invented group theory. We haven't seen that at all from LLMs.
It will not happen with existing LLM techniques.
I wonder what the total cost of this research was, including the salary for their mathematicians and engineers.
Why would you factor in salary unless they had to baby it through. You would only count the hours for setting up the harness and prompt and checking the result.
Training the model is going to be amortized over other uses.
> Why would you factor in salary
Say that it turned out that the total cost of the proof of the Erdős unit-distance conjecture was $50 million.
Then the question really becomes: yes, these models are capable of proving important mathematical results, but at a very high cost. Is it worth it?
If a mathematician applied for a research grant of $50M USD for proving the same thing, they would have been laughed out of the bank.
What's more is that when you have a research grant, you train PhDs and postdocs, you hire new staff, and you disseminate. That is, you get much more value for the money spent.
I'm just curious what the cost is.
It feels like the real cost might be negative though.. They use frontier math as a way to test improvements in their model. So solving the problems is like a positive externality, but the important thing is they can verify that the model is improving instead of looking at useless benchmarks. Plus it is good for marketing and attracting talent.
I didn't argue that knowing the total cost is uninteresting. What I was saying is that realistically the total cost is:
Hours needed for prompt + Hours needed to check result + API costs.
You don't say "well let's add together the total yearly compensation of all the engineers and mathematicians at OpenAI that were involved" and throw that into the total cost. That's simply nonsense accounting.
The actual comparison you are making is some university researcher weighing between getting a grad student (several tens of thousands of dollars) vs typing up a prompt and sending a request to OpenAI for inference (as mentioned in the article, around $2000 in API and maybe a few hours for the prompt and harness).
> The cost of generating the proofs for all 10 of these breakthroughs combined was under $2,000 at Sol API prices.
That's clearly just for the tokens, this doesn't really answer OP's question.
Given that OpenAI pays their employees with stock surely a breathtaking number, but not a very meaningful number now that the infrastructure is in place and the models are trained. AI could never get better and it would still be incredibly disruptive.
I don’t feel the existential dread of mathematicians is correct. It seems to me in fact these results are bringing math mainstream. I now personally look forward to the interpretations and discussions of the significance of such results by human mathematicians.
Now I understand that it’s mostly the super stars benefitting from the increased attention. Folks who are less established don’t share in that glory. But on the other hand it seems like an exciting time to go even deeper for in various specialties of math by deciding where to focus these powerful tools. For every conjecture defeated some seven or eight new ideas open up. Our path through that combination will be set by creative and curious human mathematicians.
[edit: deleted a distracting comparison to Chess]
The old way of establishing career credibility is being destroyed, for better or worse. Accomplishments that used to be career-defining are hard to distinguish from AI, and correlate more with access to compute. Think about Bill Gates's math paper he wrote in college. That kind of thing is gone now as a path to credibility. There's still competitions and grades, but the diversity of paths is going away. Maybe new ones will open up. This is a competitive advantage for old people who have credible pre-2025 accomplishments they can point to.
If accomplishments can't be distinguished between talented people and untalented people with compute, is there really a point in trying? I suppose one can hope that talented people given compute will be more effective than untalented people with compute, but I despair that that may not be true for much longer.
Knowledgable people can confirm what the AI produces is correct. I could make ChatGPT produce a result on an open question and I would have zero way to verify its actual correctness.
Which is less interesting work. And you probably need to do the hard grunt work by hand first to develop the skills and intuition to be able to verify an AI-generated result. So you can’t outsource everything to AI without loss of skill.
Given that we were nowhere near this state even two years ago, I think it’s a question of velocity more so than just distance.
I find this whole way of looking at things weird. Did maths exist just to entertain and employ mathematicians? Surely maths is, like, useful? Not immediately, not predictably, but in the long run? In which case, whether mathematicians feel bad about it is mostly irrelevant - it's like complaining about the railway because it may put coaching inns out of business.
Absolutely not the same! People need jobs to bring in income. I don't believe those who profit off of this will share it with the world. The power is all concentrated in the few hands that decide whether or not the rest get any semblance of income in the long run. I don't believe UBS until it happens.
It sounds like you think no technological advances will make the world richer in the long run. I politely suggest that the past century of economic growth shows problems with this argument. I also think that, while jobs are important for prosperity, the jobs of mathematicians are a minuscule fraction of a percent of the total.
The chess analogy is awful. If you simply want to know the answer to a chess problem, give it to the engine. Chess only lives on because it's a competition between humans to test their skill (just like bicycles, cars, trains didn't eliminate foot races) ... the computer is largely factored out, but not entirely -- people train with the computer, use it to check whether they played correctly, ... and they cheat. A lot. Thus there are more and more sophisticated mechanisms to detect and prevent cheating.
If you translate that to math, then all you get is math competitions, not math as a career. Of course the translation isn't nearly exact ... there's a lot more room for professional mathematicians because the math space is far more vast than the chess space and can't generally be cranked out mechanically (we have proof).
P.S. The response is nonsense ... I explained exactly why it's awful (others have too) and the response doesn't in any way refute the explanation ... rather it offers up a ridiculous strawman.
I’ve deleted it but no it’s not awful anymore than saying “we survived WWII, we can survive this.” The point was that change happens but humans find a way forward.
Every time someone makes a comparison to chess I die inside. Chess is a spectator sport primarily funded by a few eccentric billionaires. Players artificially constrain themselves in timed environments knowing that they will never be able to produce better moves than a smartphone because a select few people find it interesting. Only ~30 top professionals actually make enough money to have a full career playing chess, maybe a few hundred more can sustain a meager lifestyle with coaching gigs. I shudder to imagine what will happen to the tens of thousands of non-Fields medalist caliber mathematicians if math goes the way of chess. Perhaps Terence Tao and a few other famous mathematicians will be funded by Peter Thiel to report on how well humanity can keep up with the machines? How do you expect any mathematician to be optimistic about this comparison.
>Only ~30 top professionals actually make enough money to have a full career playing chess, maybe a few hundred more can sustain a meager lifestyle with coaching gigs.
Was this different before chess computers were invented?
Fully agreed. As someone who both loves chess and works on chess engines... these comparisons to chess needs to stop.
The distinction is mathematician vs mathematics. Mathematics is going to reach new heights beyond the wildest dreams of contemporary mathematicians. But perhaps without the participation of many paid mathematicians.
Just sounds dystopian,
Another noteworthy difference is that Stockfish is also gpl.
If there was any real money in it Stockfish would not be the best chess engine.
There is money in this, so of course the closed models are far ahead. The open models will likely catch up a bit at some point, just as Stockfish caught up to AlphaZero. That being said, there are already a couple. It seems Deepseek has a claimed proof to the "Ziegler's Cross-Polytope Conjecture" [0], but I can't speak to the significance of the result.
Do mathematicians have the right to say "no AI PRs please, the volume is too much" just like how some open source maintainers do it? I guess they feel a loss of control, there is no way to turn the hose off.
Thinking of this a little bit with the perspective of every new proof as a burden, dumped for review by actual mathematicians.
As in chess and go and also coding for the past ~year there are two groups of people: the disappointed and the enthusiastic. The disappointed are sad that they lost their advantage and that the craft they honed for years or decades has rapidly lost its value; the enthusiastic are excited about the future and what computers can bring to their domain and how it will evolve. I’m a bit of both if it comes to programming, more enthusiastic than disappointed, but also more than a bit terrified about the pace of it all. I imagine that’s how Kasparov felt back then, that’s how Lee Sedol felt and now that’s how Terry Tao feels.
The most disappointed folks will simply drop out, but the enthusiastic ones will keep going and with luck make up for the ones who decided to quit. Chess and go certainly went this way.
A fundamental difference being that no one was actually paid to find good moves in chess and go like they are to solve math problems and write code. You're comparing the digital camera and the automobile.
I wonder if Erdos would be saying " It's fine that y'all are answering my questions, but who is asking better questions??"
This is both awesome and terrifying for mathematicians, however some ideas can be generated and the field as whole expanded with the attention!
However, I was looking at the proofs and reason explanation and openAI should be more explicit in how the work has flown. I find the models have jumped hoops in some places of the proofs, that can be hard to track. In fact, when a paper is published you usually get a review and if no reviewer understands they ask you to further explain the thought process. It will be fun to see if this happens here.
I am duly impressed by the powerl of the nameless internal AI, but not a single human contributor's name listed anywhere? Did someone at least make this model a coffee?
> The results were achieved by an internal version of Astra, our next major model.
surely some human regularly typed "think deeper, make no mistakes".
What happens when OpenAI et al stop being open about these things, and just pack it into the training?
Not much point to pure math being kept secret, in all honesty. There isn't really industrial value, its only purpose (to them) is showing off their model's capabilities. More realistically they'll just stop paying for it.
Edit: Oh, are you suggesting they just use it to privately improve their models? I imagine a few more correct proofs would have a very marginal benefit, if any. Also, they'll probably just get extracted, meaning it still gets out but OpenAI doesn't get to fancily announce it themselves.
At this stage. No doubt calculus had plenty industrial benefit.
What does this even mean lol. These are not solved questions. The solution never existed.
On the token limits etc - one assumes that OpenAI et al are able to “hire expert in field, and let them spend the equivalent of a million dollars of tokens” because they are not actually selling their complete compute 24 hrs a day, so the cost internally is a negligible (ish) electricity bill.
Which is very suggestive - if after everything they are not fully loaded then the next gazillion data centres being built look unlikely to be needed.
For OpenAI, research is marketing. I’m sure they’ve got plenty of budget for that.
No such thing as free, even internally at a company. All such use of resources is accounted for, assigned a dollar value and billed to some department. Someone ran the numbers and figured that whatever they spent on these GPU cycles was worth it.
Presumably it's a rounding error compared to their full output, and they're making sure they have enough compute set aside for research by limiting public models. The more datacenters they build the less they have to limit them.
RL training can use all of them - idk what needed means.
I love how people come up with creative ideas to prove the bubble. This one is even more ridiculous - that OpenAI had spare compute to advance mathematics proves that data centres will not be needed. WHAT.
If anything it proves more data centres are needed. That's literally the only reasonable conclusion from this news.
Sorry I thought that a bubble was widely accepted.
Are you arguing there is not an AI bubble, and that all the DC buildout is fine, going to be profitable etc?
I am not looking for a online slanging match - just looking for a different point of view
It’s not obvious at all. If it were obvious to you, it would’ve been to OpenAI. It’s in their interest to accurately predict demand. The assumption that OpenAI/Sam is both really powerful but simultaneously ignorant to know what others know as obvious is well.. just strange. Especially strange when OpenAI has more information on models, breakthrough and usage patterns and we don’t.
I’m not participating in the slinging match but it’s very very weird that you think it’s some established thing that these companies won’t make profit. A lot of hubris must go in this kind of thought. Like.. do you all think everyone’s playing musical chairs?
“Often” is load bearing. I don’t think markets are more likely than not to be musical chair shaped. To make this conversation more concrete, give me a falsifiable prediction on there existing a bubble. And then I’ll tell you if I believe in it or not.
Some of the Lean proofs are 50k lines - is that normal?
It's remarkable how you can manage to get these models to produce remarkable breakthroughs like an explicit construction of a non-sofic group.
And yet this is the exact same company that has screwed up their android app so bad that the latex N^3 rendering problem makes it so having it explain it to me crashes the app.
Truly jagged beyond belief.
At what point can we say AGI has been achieved? What is the test? AI is solving mathematical problems that humans have not been able to solve for decades. Is that not enough?
Sam Altman has said "If superintelligence can't discover novel physics, I don't think it's a superintelligence." Is that the test? How far away are we from AI discovering novel physics? It seems within reach.
It depends on your definition. For me it would have to be able to do the stuff humans can do like make a cup of coffee (Wozniak test).
Just maths isn't really general enough for the G in AGI.
It can give you detailed instructions for making a coffee. Is that not enough? Actually making coffee requires more than intelligence, it requires eyes and limbs (i.e., robotics). Think about a human that is blind and does not have limbs. Does he not have natural general intelligence, even though he is not able to make a cup of coffee?
The Wozniak test has it with a robot body going into a house, finding a coffee maker and making a cup. I guess you can vary the rules as you like.
That's a good test for a robot, not for AGI. AGI should test only intelligence and should not require limbs.
It’s artificial, it’s general, and it’s intelligence. The people who believe “AGI” is an important and unattained goal need to start coining and defining their terms better.
A true AGI will continuously improve itself without periodic retraining from scratch. Just like humans.
That’s just an assertion. Why is that the “true” definition?
Can anyone comment on the significance of any of these results for their respective fields? Or what impact they might have? Presumably none are quite at the level of the Jacobian conjecture, but some of the results on group theory and sphere packing sound pretty important at first glance.
Found some discussion here [1] from someone who actually worked on a few of these problems.
[1] https://x.com/henryquantum/status/2083623695436623915?s=20
Have blog posts replaced peer-reviewed academic papers when it comes to publishing advanced in science?
These is mostly mathematics, not science, and they link the paper in the post: https://cdn.openai.com/pdf/74c24085-19b0-4534-9c90-465b8e29a...
Mathematicians will tear it to pieces if any of it is fake!
How much do you all think it would cost to "buy" these advances from PhDs, practicing scientists?
This isn't really a productive way to think about these things, IMO. It's quite possible it would take hundreds of years for any specific group of PhDs to solve them. Or one individual PhD could have the correct flash of insight and solve it in a month. There's absolutely no way to predict this, besides trying to gauge the apparent simplicity of the proof or counterexample (which is likely to be misleading). Until someone actually runs an experiment like this it's not a viable metric.
I understand and I am not trying to deny the impressiveness or the velocity of AI in general. But at some point we have to ask how much do we trust the labs at face value without much transparency of how they got to the results when there is trillions of dollars on the line.
Do you think OpenAI investors are more cavaliers than yourself who doesn't have any stake in it?
The level of conspiracy theory is nuts
Capabilities of this sort have already been demonstrated by independent parties, and models have consistently gotten better at this. Yes, insinuating that mathematicians and scientists are secretly solving decades-old problems on OpenAI's behalf is an insane conspiracy theory.
[flagged]
A friend’s PhD advisor has been chasing non-sofic groups for 25 years (and was shown a preprint of the results by openai to verify them). He believed a solution would be Fields-worthy
This was not a problem that was for sale
You couldn't. PhDs have been working on these problems for decades. It wasn't for lack of trying that none of them could figure these solutions out!
Any advances in theoretical physics yet? Are there any fundamental obstacles? I would have thought not, but I haven't seen anything reported.
The Maxwell conjecture was a conjecture in theoretical physics (though not a particularly important one)
The fundamental obstacle is that we have no conceivable way to produce the energy levels to test the predictions that new theoretical physics would produce. We are like over a dozen orders of magnitude off.
There's a lot of everyday stuff in physics which is unexplained like the particle masses we have.
And a lot of condensed matter physics. Type II superconductivity is a well-known one, but there are a lot of more less well-known ones
I don't know why, but when I saw the source of this particular headline it reminded me of the album title 26 Mixes for Cash
Ambient 0: Math for Airports
So they are bribing 100,000 researchers with free accounts to work on their future unemployment.
I hate this timeline. I might be excited for the kind of answers this AI builds for unsolved problems, and also for learning new things by talking to it. But, I feel like I'm in the minority of people here who feel this could be a net negative endeavor with this having to kill a lot of educational institutions and their ability to fund themselves in the long run. It's not worth that.
I would love more time and money put into real-world problems by these companies. Climate, food insecurity, pollution, technology for convenience and/or to help people have a higher quality of life.
I'm sure they must do some of this type of work, right?
Solving math problems doesn’t require the cooperation of rival factions.
It's a tad petty, no? In the end, the contribution to a healthier society leads to more for the companies contributing.
Yes, but all of those are orders of magnitude harder than math.
I suppose it depends on which types of problems you're targeting. There is a lot of physical science, and theoretical science, that has gaps because there aren't enough people working on tooling to assist in things like calculation, generation, simulation, etc.
I agree, some of the problems are more difficult. I don't think that's the case for all of them. And, besides, these companies could be demonstrating how to approach problems and where their users could spend tokens to help with these problems.
Should not these companies try to work on these problems _because_ they are difficult?
They are political problems, a computer could never solve them.
A computer could solve them by creating the right technological ,social, rhetorical and economical solutions but that would lots of money anyway
We already know the solutions
Tell that to game theory.
What's stopping the non-profits that exist today from just putting more money into tokens to get the solutions they want?
I'm not sure if this is meant to be rhetorical. If it isn't: money and manpower. The LLM companies have the money, they have the manpower, and so they could likely spare to target some of the problems they are also helping cause.
Seriously?
We already know how to solve all of these issues. What we lack is collective political will.
There is boundless technology we have not yet discovered. I think that we understand we have problems. I don't believe we actually know how to solve them all.
And yeah, the lack of collective political will sucks. It would be naïve, however, to think that there is no value in ensuring longevity in our current and future infrastructure. And improving it to sustain the population giving these companies their value is an obvious win.
This SV mentality that technology can solve all problems is rather tiring, really.
We only know how to do it by means of considerable sacrifice. That's why nobody wants to do it. Solving the issue would be doing it without sacrifice or somehow getting us to do it regardless.
> We only know how to do it by means of considerable sacrifice
Little sacrifice actually
> Solving the issue would be doing it without sacrifice
So... you are expecting magic?
LLMs cannot create resources out of thin air.
Almost like superintelligence solves these problems...
We know how to solve food insecurity (in 1st world countries). We have plenty of food. Capitalism requires though throwing out food that can't be sold because billionaires find giving away things anathemic to their worldview. Get rid of billionaire sociopaths.
the people who crow in the comments of each of these posts about AI advances making human beings useless seem to bizarrely identify themselves with the AI, but none of them seem to have had any hand in building this technology. at best, they're power users. pure ressentiment.
Wow, so this is the end of science :(
It's just another tool that can help solve problems. It doesn't know _what_ problems to solve. It turns out that a lot of old problems are now low hanging fruit for these new models. In terms of 'expanding the frontier', we've just discovered dynamite and can now blast our way through mountains. The bottom of the ocean or space are still as hard to reach as ever.
It's not even predictable like dynamite. Sometimes it can blast through a mountain, impressively, the problem is you can't predict which mountain it works on. And other times it can't even make a dent in a molehill, which is perplexing given what it was capable of earlier. Can we even call it dynamite?
No bud, it’s just the beginning!
People weren't their strongest even when most did manual labor. Now that humans are free from mental labor we work on creating and optimizing the best exercises for each mind. Couple that with restructuring transport infrastructure and diets many people will be smarter and fitter than at any time in history. They won't be able to outrun an automobile or out think an autointelligence.
[dead]
In a way the most remarkable thing about this is that it isn't even at the top of the HN homepage. Even if this is a step up from what we've seen before, we're no longer astonished by the idea that AI can make significant advances in mathematics and computer science.
This is not at the top as it is actively flagged by people that can't psychologically cope with the advances of AI. Hacker News is no longer a web site of an elite.
It wasn't heavily flagged. It was pulled down by the flamewar detector due to the large number of comments, and it slid under the radar due to only hitting the front page during overnight hours on Friday night/Saturday morning. It still spent 10 hours on the front page, but all during off peak hours. I've now created a new copy of the post so it can have prime time exposure.
> people that can't psychologically cope with the advances of AI
Yes. And there are many of them. I wonder what would help them come to terms with it. Seriously, people are going to be grieving over this. Loss of identity, loss of social standing, ideas of entire future lives that will now never happen. The greatest crime people may hold AI guilty of is taking away their dreams.
It's given many more an opportunity to fulfill their dreams.
People have had to deal with getting their jobs automated away for centuries. None of this is new, and perhaps reminding ourselves of this is the best way to cope.
I understand your sentiment, but I think this really is something different. This isn’t a craft going away, or even an industry being replaced, it’s potentially everything we do. It’s the ground being pulled away beneath people’s feet, everyone, everywhere all at once. I think the vacuum it leaves in people’s lives needs to be filled with something, and I haven’t heard any good ideas about this or how the transition should be managed at all.
This is the sentiment of people who haven't accepted that this is in fact something very different from what people have seen in the past.
I like your comments and agree with a lot of your points, including parts of this one. That said, please don't go to this route. A lot of the doomerism comes from financial insecurity fears. Try to remember that a lot of people are subconsciously afraid of losing their homes. I know that it is pretty hard to ignore them, but try to engage with people that do not dismiss 100% of AI accomplishments.
> A lot of the doomerism comes from financial insecurity fears. Try to remember that a lot of people are subconsciously afraid of losing their homes. I
I think recognizing and accounting for your own personal biases is one of the requirements of the being an intellectually honest and rigorous online discourse participant.
Things could be genuinely impressive and fascinating even when directly challenge your ego and material well being.
This comment is about the tier of a Reddit atheist going to a funeral and telling a grieving family that "haha grandma is dead and there is no heaven".
Didn't know I was part of an elite.
You're were for 4 months. That's what we're talking about. It used to be.
Forums change with the times, and this one never existed solely to burnish your ego.
Moreover, mister elite, you don't know why this press release was flagged.
I'm not sure why we should privilege your bitter speculation over more mundane possibilities.
[flagged]
You don't accept it is possible that it was initially flagged as a dupe or spam why? There is competition for primary submission here, especially for the primary AI companies.
Moreover, if it wasn't flagged at all, like you suggest, then the grandparent was inventing things to be bitterly resentful about... Which is not a behavior any forum should indulge.
It’s never been clear to me why the HN algorithm isn’t public. It’s obviously nowhere near as complex as something like Twitter, and of course isn’t a trade secret. The fact it’s private only furthers speculation like this.
> Hacker News is no longer a web site of an elite.
Never was.
So condescending. “Can’t psychologically cope”? Can you hear yourself? There’s some advances, but we’re losing a lot too. Don’t get dazzled by the hype.
Interesting. I had no idea that a person could see what is flagged?
If you page through the /newest listings you can see [flagged] and [flagged][dead] submissions.
eg. this: [flagged] A migrant surge tests Spain's open policies (economist.com) - https://news.ycombinator.com/item?id=49131860
is clearly marked as flagged.
Unlike the current submission: Ten advances in mathematics and theoretical computer science (openai.com) which isn't [flagged].
[flagged] submissions aren't [dead] (killed), they are still active and can be upvoted and commented upon.
> if you compare its rank to that of other stories with a similar age and number of points.
Ranking is complicated enough here even before weighting, speed of initial upvotes can play against ranking, number of comments and the shape of the comment tree also affect ranking. And yes, various subjects and submission sources do get weightings that impact ranking.
What's funny, to myself at least, is that any attention at all is paid to "HN front page ranking" - I've been on again off again active here since 2008 .. and can't recall ever really looking at a default HN "front page" ever.
( There's /newest /newcomments /active etc to browse and sites such as https://hckrnews.com/ )
> Hacker News is no longer a web site of an elite.
It was always mainly a website for employees of an elite.
Or maybe, just maybe, other people have different opinions than you?
Is that possible or is everyone else too common to have those?
> Hacker News is no longer a web site of an elite
hah, sorry, we are plebs out here.
Jokes on him. I'm a peasant and I've been here for years
[flagged]
[flagged]
We cannot psychologically stand that Redis is hyped by OpenAI:
https://developers.openai.com/cookbook/examples/vector_datab...
How are the sales going?
If you're actively throwing away brand new greenfield research because it was generated by a computer at a company that stans industry-spanning software so that you can stay mad at your pet celebrity project, you might be the problem.
You have to give AI one thing: It is vastly better at understanding text than AI boosters.
Which is a low bar, but still.
I don't think that's it. Multiple or my friends from the target audience (academic mathematicians) admitted to scrolling past because the title made it sound like a review of last month's contributions, instead of 10 new ones.
There are articles with far fewer upvotes and comments ranking higher on the front page right now, despite being the same age or older than this one. HNs opaque ranking system at it again.
What about AI research itself? Is OpenAI close to automating its human staff out of a job?
> What about AI research itself? Is OpenAI close to automating its human staff out of a job?
It's more like they've already automated the parts of the jobs that the humans most closely thought of as the "their job"
They and Anthropic have indicated that the models are substantially augmenting the research and doing large amounts of work autonomously at this point. Here is one of the many blog posts on it [1]. Many people would dismiss this as "marketing" so take it for what you will.
My guess from following this stuff quite closely is that these companies are still a couple years away from fully autonomous research staff.
[1]. https://www.anthropic.com/institute/recursive-self-improveme...
Yes, but they wouldn't publish that bit lest other companies steal the ideas.
It’s also very very divided (x companies, oss vs not and other interests)
This is one of the most impactful mathematical publications in history by all accounts
I think we've now hit a point where 99.9% of the population gloss over these types of AI advancements because of human competence being insufficient
No human could have published this because it requires paradigm shifts (e. g. Section 5) in multiple mathematical domains. Mastering one of them to this degree is rare, mastering 3+ pretty much non existent for humans.
Honestly just a bit burnt out on posts like this
Original submission (460 votes) two days ago: https://news.ycombinator.com/item?id=49132058
For some reason comments got moved to this one.
Its visibility was diminished due to the flamewar detector and most of its front page time being during overnight hours on Friday night/Saturday morning USA time. I've created a new copy to give it some primetime exposure, because it seems like an important enough announcement to warrant it.
[dead]
[dead]
This thread has an absolutely wild points to comments ratio.
tomhow explains they gave the story another shot here: https://news.ycombinator.com/item?id=49158443
What's up with the upvote/comments ratio 8 to 337 on this post? Are the comments already also ai advanced? (/s?)
[flagged]
I asked ChatGPT and it told me these aren’t not important /s
[flagged]
To clarify a bit due to the downvotes : this is not a research paper from a startup or a public frontier lab, it is just PR from a corporation, thus yes an advertisement. Downvote all you like it's still of no value.
[flagged]
It's simple economics. Building a robot to do your chores is expensive and only a small minority of people value their time enough to buy one. Meanwhile, SWEs are expensive and GPUs are (comparatively) cheap.
You shouldn't be getting downvoted for something that a majority of pure math and art enthusiasts believe to be true. The truth is many of these entrepreneurs and VCs are obsessed with AI not for money or human progress, but because it makes them feel closer to being a "god" rather than a mere mortal. Much of it (especially AI art) is out of spite for human creativity, which is done by mortals with limitations.
Taking the stance of moral superiority is kind of funny. And pure math and art enthusiasts don't think they are closer to being a "god" from understanding/"discovering" math?
Stop coping and deluding yourself mate.
To begin with, whether AI is the one doing the discovering or not makes no difference. Any "pure math" person would aim to understand regardless - and would be quite glad that they have a longer paved path.
Any mathematician in academic or industry is more than likely not a "pure math" person (tainted by capitalism).
> Stop coping
Isn't coping a good and useful mechanism?
Lmao what a ridiculous response. Yes, some mathematicians and artists are in it to feel smart. But the vast majority also just enjoy the process. Having a computer do all the work for you and just typing prompts in ruins that completely. As Ronny Chieng said in his Harvard speech, the journey is the point.
>Any mathematician in academic or industry is more than likely not a "pure math" person (tainted by capitalism)
Ignoring that I meant pure as in non applied math, let's just make it clear: you agree that mathematicians who are against capitalism encroaching on this process should be allowed to dislike it without criticism of being pretentious?
Any implication of any of these findings? They seem like unimportant nerd snipes to me. If you want to do something actually relevant, get chatgpt to write a simulation of graphene nanotube construction and figure out how to do it at scale.
Very marketable nerd snipes indeed.
One - and I do not mean to be snarky - you can literally ask Gpt 5.6 Sol this - and if you want to see cool stuff - Fable running in their app (not website) has a view thinking button that is actually a good way to explore the adjacent fields, etc.
The non-sofic group one is definitely a big deal - would have been a Fields medal if discovered by a human.
> claiming human authorship for a proof generated entirely by an AI system would misrepresent both the system’s contribution and the nature of genuine human intellectual work.
AI has no self-awareness. It's a tool. When you assemble a furniture using a screw driver, the torque force interacts with the molecular forces inside the metal and miraculously it transfers the force to the screw though a clever geometry design, communicating the force to the screw to turn it in a certain way.
Do you attribute the build to the tool? The "system's contribution" is helped by many other things all the way down to chips, datacenters and power generation. If the authorship requires attributing to a tool, then it should happen all the way down.
Sorry, OpenAI's take is correct here. If you're not convinced, here is how they prompted LLM: https://cdn.openai.com/pdf/04d1d1e4-bc75-476a-97cf-49055cd98... [0]
A slightly smarter highschooler could write these. I could write these. It's clear as day that the LLM, not the human, did the heavy lift. It'd be ridiculous to give full credit to whoever wrote the prompt.
[0]: Not one of the proofs in the linked article, but from OpenAI too.
> A slightly smarter highschooler could write these. I could write these. It's clear as day that the LLM, not the human, did the heavy lift. It'd be ridiculous to give full credit to whoever wrote the prompt.
I think you're over-estimating what a smarter highschooler could write.
A "finite loopless undirected multigraph" could have been explained to me at that age if we'd taken Discrete rather than Mechanics and Pure (and one module of Stats) in my two A-levels* in maths and further maths; but from what I saw of the Discrete module, neither:
Every finite loopless multigraph with no bridge possesses a cycle double cover, without additional assumptions such as cubicity, planarity, connectivity, or higher edge-connectivity.
repeated-edge closed trails masquerading as cycles
* For those not from the UK: A-levels are between secondary school and university, when aged 16-18. Functionally they are university entrance qualifications: https://en.wikipedia.org/wiki/A-level_(United_Kingdom)
But why can’t we prompt the LLM “just do math research”? This is what I don’t understand.
If there aren't thousands of TPUs doing that [0] right now I'd be quite surprised.
[0]: e.g. "go through wikipedia's unsolved math problem list and solve them".
100% agree. If the models are so capable that they're advancing math, it doesn't seem like a stretch to expect they should be able to determine with "doing math research" entails and the best way to use their capabilities towards that end. Why do we need to hand hold the models by telling them to do parallel research, keep threads independent, etc.
The human provides the intention and the ability to appreciate the output. Tools do “heavy lifting” all the time, but we still primarily credit the humans who use them precisely because they made the choice to use them.
Provability is just going the way of computation. John Napier had to manually compute logarithm tables over decades and was recognised for his work; now that same work could be performed by a 10 year old with a calculator in an evening.
What gives the intention and ability to the human?
Are you playing dumb? Using power tools to build furniture is very different than using an ai robot to carve a statue or whatever.
I don't disagree with you but there's no need for exaggeration; ain't no high school student writing this:
> In particular, proofs for special graph classes, constructions of cycle covers with some edges covered other than twice, bounded-length or prescribed-cycle variants, reductions to another unproved conjecture, computational verification through any fixed graph size, and candidate counterexamples without a complete nonexistence certificate are insufficient.
which is infact a very important part of the prompt.
the fact that such things have to be explicitly in the prompt points to the fact that the underlying system is still far from where it needs to be (basically, lacks basic understanding what a proof is)
If that was the case, that elaborate listing of all things that might look like a proof to a naive student, but are actually not proofs (and not even just partial results, but fundamental misunderstandings of what constitutes a proof) could have been easily and equivalently replaced by 'I am interested only in a full proof, don't bother me with partial results'. Yet, they were not.
To a real mathematician you would not have to list those explicitly, he/she would have understood that implicitly from 'give me a full proof'. That listing makes sense to say only to somebody who pretends to be a mathematician, but has not true understanding of how the math works. The models are getting better and better in this pretension, but prompts like that reveal that it is still just a pretension, not a true understanding.
They don't have to be. At this point, we have multiple results from 3rd parties where the prompts are very basic.
To name a few:
- https://xcancel.com/DmitryRybin1/status/2079904005652893709
- https://archive.ph/2w4fi (https://chatgpt.com/share/69dd1c83-b164-8385-bf2e-8533e9baba...)
you cannot be serious
When you use a crane to do the "heavy lifting" for construction work, do you give full credit to the cranes?
Read the prompts in the PDF I link and see if your analogy makes sense in this context :)
When I type 56789*23456 into my calculator and get the result I don't claim to have solved the problem, the calculator did it.
Your brain also is physical. Electrochemical gradients flow between physical molecular constructs. Isn't it just chemistry? Do you attribute it to physics or some whole-is-greater-than-the-parts idea?
> Do you attribute the build to the tool? The "system's contribution" is helped by many other things all the way down to chips, datacenters and power generation. If the authorship requires attributing to a tool, then it should happen all the way down.
When the tool is a 3D printer, or any CNC system really, you bet I attribute a build to it.
I could also attribute the operator; there is no contradiction, it's a free choice, just like saying "I am in Berlin" does not contradict "I am in Germany".
> AI has no self-awareness
What is your mechanistic model of self awareness that yields this conclusion?
> It's a tool
Does your model suggest that tools can't have self awareness?
Dunno about the parent commenter, but I personally interpret the concept as having a hidden representation of self that is continually tended to, and influences future choices. This implies statefulness, which models are intentionally not at inference time (*).
(*) Even if we hack around this and just do the usual trick of simply laundering statefulness to a higher level, in this case the context window being fed in, I fail to identify (**) a representation of its own state in these bodies of text that it'd be meticulously maintaining. I further fail to identify how it could be hidden or maintained, considering I control like half of it. The best you could ascribe it is a meticulous maintenance of a persona the user is talking to, but then that doesn't necessarily represent the model's internal state, the same way my own words here aren't doing so either. Difference being, I actually have one (I'm "on-line").
You'll sometimes catch models mixing up who's who and how many who-s there even are for example.
(**) I did wish for something hidden though, so maybe it's just concealed? The same way people can encode a lot more of their emotional and mental state than normal into text if they read and write a lot of it, I'm aware of research that suggested the same for LLMs, albeit I cannot cite it. Maybe those phrasing signatures are just alien to me and will never pop out. Either way, I'd expect researchers to stumble upon this during interpretability studies, and either they haven't, they have but it wasn't popsci adopted, or they're keeping awfully tight lipped about it. If you know of anything like this, your turn now, would be happy to learn.
I do wonder how reasonable it is to expect e.g. a single maintained identity though. Maybe it isn't?
(*) Another way to hack around this of course is to just precompute some internal "self-awareness states" and hop around between them. Probably the closest to what the models are actually "doing".
Before reading, know that I am uncertain in either direction.
> a hidden representation of self that is continually tended to
This sounds like a personality? They act like they have one of those. It may be an illusion, and even if it isn't an illusion it is unlikely to be anything like the source (us), but they act like it.
> I further fail to identify how it could be hidden or maintained, considering I control like half of it.
Indeed you control everything about a local model, and much of the context of even a remote model. But the state of activations and circuits in SotA AI is hidden in similar ways to those of synapses in your head: difficult to decipher even with probes monitoring the signals directly, and often not emitted at the normal output.
> The best you could ascribe it is a meticulous maintenance of a persona the user is talking to, but then that doesn't necessarily represent the model's internal state, the same way my own words here aren't doing so either. Difference being, I actually have one (I'm "on-line").
While we can be confident that LLMs make up personas etc., it is insufficient to go from "that doesn't necessarily represent the model's internal state" to "therefore it doesn't have one".
> You'll sometimes catch models mixing up who's who and how many who-s there even are for example.
I've, unfortunately, also experienced this with humans. Perhaps they were losing their self-awareness at the time? I do wonder if old-age dementia does that by the end, though the person in question didn't ever get diagnosed with that.
> If you know of anything like this, your turn now, would be happy to learn.
Do you mean like these, or something else?
• https://researchportal.hkust.edu.hk/en/publications/decoding...
> This sounds like a personality?
Not quite what I meant, but it's also not entirely unrelated I guess? Personality to me is like a natural bias. It does also shift over time, and is also an internal bit of state. I guess in some respects it can also be self-referential, like personal convictions.
> Perhaps they were losing their self-awareness at the time?
I do think it is entirely possible for people's self-awareness to shift, yes. Or more precisely, I do model things that way.
> Do you mean like these, or something else?
They're adjacent, but I more meant something like these:
https://arxiv.org/abs/2410.03768
https://arxiv.org/abs/2310.18512
https://arxiv.org/abs/2605.26537
So basically, steganography. The difference is that these papers investigate from the perspective of separate LLM instances covertly exchanging information between each other. This is in contrast with the scenario I'm laying out, where an LLM's past state is exchanging information with its future state, continuously representing and modulating a concealed internal state of some sort. And then that state just so happening to be some sort of self-referential meta state.
And the best inkling I have towards this is basically: https://www.youtube.com/shorts/WP5_XJY_P0Q
But then I don't think there's enough covert channel bandwidth in the agent replies for anything interesting like this.
> Dunno about the parent commenter, but I personally interpret the concept as having a hidden representation of self that is continually tended to
I don't see why an LLM could not have a sense of identity or personality while it's evaluating a specific prompt, or even change self awareness while evaluating a prompt since many outputs model a back and forth conversation. My point is that without a mechanistic model of what "self awareness" means, we have no way of truly evaluating such questions, we're just hand waving vague intuitions about what it could mean.
Even mechanistic models generate interesting discussion. How many years have we discussed Turing machines and the lambda calculus? Almost a century of great work came out of those.
The reason I insist on mechanistic models is because the original post was making a definitive knowledge claim, and in my experience, the knowledge claim is unwarranted.
I honestly don't think a language model is enough for self-awareness, regardless of the exact model of awareness.
A language model (or an image model or whatever) cannot even be sentient, and I think sentience is a prerequisite for awareness.
Even if we express a lot of our subjective experience with words, the language is just a symbolic representation of those experiences. The qualia themselves, even those that are quite abstract, are rooted in our physical presence and evolution.
You can't have an understanding of what hunger or physical pain feel like if you have no need for food or a sensory capacity for feeling pain. You can't understand what loneliness or pride at an achievement mean if you don't have a neural network wired to value social connection or status. We value connection because we're social animals that have needed each other for survival.
Even the more abstract of our subjective experiences are in some way rooted in our physical evolution.
I see no reason to believe that a neural network built entirely based on the symbolic level of language could have the features needed for the subjective experience itself.
AI awareness might actually be more believable if that awareness manifested itself in an entirely different way than in humans. But if we assume awareness because outputs resemble what we consider meaningful as humans, yet the neural network has had no inputs or evolution that could form the actual basis of human-like experience, I think we're seeing something that isn't actually there.
> The qualia themselves, even those that are quite abstract, are rooted in our physical presence and evolution.
There is no objective evidence of qualia. All evidence of qualia are vocal or other expressions of belief in qualia. Perceptions clearly exist and are observable, subjective experience and qualia, not so much.
> I see no reason to believe that a neural network built entirely based on the symbolic level of language could have the features needed for the subjective experience itself.
If your objection is to models based on "symbolic level of language" which you think lack semantic understanding of, say, trees, you should ask yourself how our brain, based on physics which also lacks any semantic category for trees, can somehow develop a semantic understanding of trees. All of these appeals to differences with the brain never seem to acknowledge that fundamentally, the brain has the same explanatory gap with physics.
> But if we assume awareness because outputs resemble what we consider meaningful as humans, yet the neural network has had no inputs or evolution that could form the actual basis of human-like experience
This assumes a lot. It seems very possible to me that intelligence inherently develops a map of natural categories (natural kinds), and language naturally develops around such categorical understanding. Semantics are then fundamentally the network of associations between categories, eg. there is no fundamental difference between symbols and semantics, and the latter cam be inferred from the former, and that's exactly what LLMs do, and why the semantic maps between different languages are so similar and how they can translate between languages.
So… your model is 100% vibes based. Got it.
Making definitive claims about whether LLMs do or do not have specific properties absolutely does require precise definitions of those properties that can be used to evaluate those questions. Merely hand waving that LLMs didn't undergo the same evolutionary process is not a definitive argument.
For example, the Turing machines and the lambda calculus don't look anything alike, but they are fundamentally interconvertible, and so in a real sense they are fundamentally equivalent. Without a model, all of your arguments are completely unconvincing for exactly the same reasons, eg. that there may exist many paths to fundamentally equivalent ends.
A better analogy would be a manufactured object, say 3d printed for simplicity. The 3d printer is given an input, and an object manifests itself after some time. We say that the creator of the object is the person turning on the machine, sending the data, and collecting the object. Not the machine itself.
I'd say the creator is the one who created the 3d model, not the one who pushed the print button.
I would say „I made this gadget with my 3d printer, but the designer is someone else (I found the model online)“. The intent, the drive, the action comes from the human
Almost as if a proof isn’t the same as a 3d print. It’s just not a good analogy
(let's assume that)My 3dprinter is special. It has a bunch of values + an algorithm (i.e. a neural network) that takes input as tokens and outputs a printed object.
Not peer reviewed, Lean proofs are 100,000 lines long and Lean has bugs:
Who is going to wade through this?
They've been hiring mathematicians to verify this stuff themselves. They're obviously not just throwing it out there without any human review.
And these mathematicians sink the comment to the bottom in 5 min?
It is not peer review if it is all in one company that wants an IPO.
I'm sure OpenAI is just interested in the greater good of mankind!