TimeCapsuleLLM: LLM trained only on data from 1800-1875

You would find things in there that were already close to QM and relativity. The Michelson-Morley experiment was 1887 and Lorentz transformations came along in 1889. The photoelectric effect (which Einstein explained in terms of photons in 1905) was also discovered in 1887. William Clifford (who _died_ in 1889) had notions that foreshadowed general relativity: "Riemann, and more specifically Clifford, conjectured that forces and matter might be local irregularities in the curvature of space, and in this they were strikingly prophetic, though for their pains they were dismissed at the time as visionaries." - Banesh Hoffmann (1973)

Things don't happen all of a sudden, and being able to see all the scientific papers of the era its possible those could have fallen out of the synthesis.

I'm trying to work towards that goal by training a model on mostly German science texts up to 1904 (before the world wars German was the lingua franca of most sciences).

Training data for a base model isn't that hard to come by, even though you have to OCR most of it yourself because the publicly available OCRed versions are commonly unusably bad. But training a model large enough to be useful is a major issue. Training a 700M parameter model at home is very doable (and is what this TimeCapsuleLLM is), but to get that kind of reasoning you need something closer to a 70B model. Also a lot of the "smarts" of a model gets injected in fine tuning and RL, but any of the available fine tuning datasets would obviously contaminate the model with 2026 knowledge.

Yann LeCun spoke explicitly on this idea recently and he asserts definitively that the LLM would not be able to add anything useful in that scenario. My understanding is that other AI researchers generally agree with him, and that it's mostly the hype beasts like Altman that think there is some "magic" in the weights that is actually intelligent. Their payday depends on it, so it is understandable. My opinion is that LeCun is probably correct.

Chemistry would be a great space to explore. The last quarter of the 19th century had a ton of advancements in chemistry. It'd be interesting the see if an LLM could propose fruitful hypotheses, made predictions of the science of thermodynamics.

It's going to be divining tea leaves. It will be 99% wrong and then someone will say 'oh but look at this tea leaf over here! It's almost correct"'

Done few weeks ago: https://github.com/DGoettlich/history-llms (discussed in: https://news.ycombinator.com/item?id=46319826)

At least the model part. Although others made same thought as you afaik none tried it.

A rigorous approach to predicting the future of text was proposed by Li et al 2024, "Evaluating Large Language Models for Generalization and Robustness via Data Compression" (https://ar5iv.labs.arxiv.org/html//2402.00861) and I think that work should get more recognition.

They measure compression (perplexity) on future Wikipedia, news articles, code, arXiv papers, and multi-modal data. Data compression is intimately connected with robustness and generalization.

That’s how p-hacking works (or doesn’t work). This is analogous to shooting an arrow and then drawing a target around where it lands.

We've thought of doing this sort of exercise at work but mostly hit the wall of data becoming a lot more scare the further back in time we go. Particularly high quality science data - even going pre 1970 (and that's already a stretch) you lose a lot of information. There's a triple whammy of data still existing, being accessible in any format, and that format being suitable for training an LLM. Then there's the complications of wanting additional model capabilities that won't leak data causally.

>.If the model comes up with anything even remotely correct it would be quite a strong evidence that LLMs are a path to something bigger if not then I think it is time to go back to the drawing board.

In principle I see your point, in practice my default assumption until proven otherwise here -- is that a little something slipped through post-1900.

A much easier approach would be to just download some model, whatever model, today. Then 5 years from now, whatever interesting discoveries are found - can the model get there.

I think it would be fun to see if an LLM would reframe some scientific terms from the time in a way that would actually fit in our current theories.

I imagine if you explained quantum field theory to a 19th century scientists they might think of it as a more refined understanding of luminiferous aether.

Or if an 18th century scholar learned about positive and negative ions, it could be seen as an expansion/correction of phlogiston theory.

That is a very interesting idea, though I would not dismiss LLMs as a dead end if they failed.

Could be an interesting experiment, but its not conclusive proof one way or another. So much of what makes LLMs so great today (vs gpt 3.5) would not be in that dataset. The training to turn these models into coding savants has generalized to other areas just as one example.

Looking at the training data I don't think it will know anything.[0] Doubt On the Connexion of the Physical Sciences (1834) is going to have much about QM. While the cut-off is 1900, it seems much of the texts a much closer to 1800 than 1900.

[0] https://github.com/haykgrigo3/TimeCapsuleLLM/blob/main/Copy%...

I wonder about something similar - leeches. Apparently they really work to eat away only decaying tissue. Could they make a comeback in modern times with better weighting?

That would be an interesting experiment. It might be more useful to make a model with a cut off close to when copyrights expire to be as modern as possible.

Then, we have a model that knows quite a bit in modern English. We also legally have a data set for everything it knows. Then, there's all kinds of experimentation or copyright-safe training strategies we can do.

Project Gutenberg up to the 1920's seems to be the safest bet on that.

I like this, it would be exciting (and scary) if it deduced QM, and informative if it cannot.

But I also think we can do this with normal LLMs trained on up-to-date text, by asking them to come up with any novel theory that fits the facts. It does not have to be a groundbreaking theory like QM, just original and not (yet) proven wrong ?

It's a base model. It hasn't been instruction tuned to "solve problems" necessarily. All it can do is attempt to complete text given some starting text.

I would love to ask such a model to summarise the handful of theories or theoretical “roads” being eyed at the time and to make a prediction with reasons as to which looks most promising. We might learn something about blind spots in human reasoning, institutions, and organisations that are applicable today in the “future”.

Don't you need to do reinforcement learning through human feedback to get non gibberish results from the models in general?

1900 era humans are not available to do this so I'm not sure how this experiment is supposed to work.

I suppose the vast majority of training data used for cutting edge models was created after 1900.

I think it would raise some interesting questions, but if it did yield anything noteworthy, the biggest question would be why that LLM is capable of pioneering scientific advancements and none of the modern ones are.

The development of QM was so closely connected to experiments that it's highly unlikely, even despite some of the experiments having been performed prior to 1900.

Special relativity however seems possible.

You have to make sure that you make it read an article about a painter falling off a roof with his tools.

That would be possible if LLMs can come up with entirely new words and languages, which I doubt.

Wow, an actual scientific experiment. Does anyone with expertise know if such things have been done?

yeah i was just wondering that. i wonder how much stem material is in the training set...

I like this idea. I think I'd like it more if we didn't have to prompt the LLM in the first place. If it just had all of this information and decided to act upon it. That's what the great minds of history (and even average minds like myself) do. Just think about the facts in our point of view and spontaneously reason something greater out of them.

It's already been done, without the model being aware of it, see https://arxiv.org/abs/2512.09742. They also made it think it was Hitler (not MechaHitler, the other guy), and other craziness.

It's a relief to think that we're not trusting these things for stuff like financial advice, medical advice, mental health counselling, ...

As a reader of a lot of 17th, 18th, and 19th century Christian books, this was my thought exactly.

Can you elaborate on this? After skimming the README, I understand that "Who art Henry" is the prompt. What should be the correct 19th century prompt?

I guess chat-ability would require some chat-like data, so would that mean first coming up with a way to extract chat-like dialogue from the era and then use that to fine-tune the model?

> Could this be an experiment to show how likely LLMs are to lead to AGI, or at least intelligence well beyond our current level?

You'd have to be specific what you mean by AGI: all three letters mean a different thing to different people, and sometimes use the whole means something not present in the letters.

> If you could only give it texts and info and concepts up to Year X, well before Discovery Y, could we then see if it could prompt its way to that discovery?

To a limited degree.

Some developments can come from combining existing ideas and seeing what they imply.

Other things, like everything to do with relativity and quantum mechanics, would have required experiments. I don't think any of the relevant experiments had been done prior to this cut-off date, but I'm not absolutely sure of that.

You might be able to get such an LLM to develop all the maths and geometry for general relativity, and yet find the AI still tells you that the perihelion shift of Mercury is a sign of the planet Vulcan rather than of a curved spacetime: https://en.wikipedia.org/wiki/Vulcan_(hypothetical_planet)

It'd be difficult to prove that you hadn't leaked information to the model. The big gotcha of LLMs is that you train them on BIG corpuses of data, which means it's hard to say "X isn't in this corpus", or "this corpus only contains Y". You could TRY to assemble a set of training data that only contains text from before a certain date, but it'd be tricky as heck to be SURE about it.

Ways data might leak to the model that come to mind: misfiled/mislabled documents, footnotes, annotations, document metadata.

I think not if only for the fact that the quantity of old data isn't enough to train anywhere near a SoTA model, until we change some fundamentals of LLM architecture

I think this would be an awesome experiment. However you would effectively need to train something of a GPT-5.2 equivalent. So you need lot of text, a much larger parameterization (compared to nanoGPT and Phi-1.5), and the 1800s equivalents of supervised finetuning and reinforcement learning with human feedback.

This would be a true test of can LLMs innovate or just regurgitate. I think part of people's amazement of LLMs is they don't realize how much they don't know. So thinking and recalling look the same to the end user.

That is one of the reasons I want it done. We cant tell if AI's are parroting training data without having the whole, training data. Making it old means specific things won't be in it (or will be). We can do more meaningful experiments.

This is fascinating, but the experiment seems to fail in being a fair comparison of how much knowledge can we have from that time in data vs now.

As a thought experiment I find it thrilling.

OF COURSE!

The fact that tech leaders espouse the brilliance of LLMs and don't use this specific test method is infuriating to me. It is deeply unfortunate that there is little transparency or standardization of the datasets available for training/fine tuning.

Having this be advertised will make more interesting and informative benchmarks. OEM models that are always "breaking" the benchmarks are doing so with improved datasets as well as improved methods. Without holding the datasets fixed, progress on benchmarks are very suspect IMO.

I fail to see how the two concepts equate.

LLMs have neither intelligence nor problem-solving abillity (and I won't be relaxing the definition of either so that some AI bro can pretend a glorified chatbot is sentient)

You would, at best, be demonstrating that the sharing of knowledge across multiple disciplines and nations (which is a relatively new concept - at least at the scale of something like the internet) leads to novel ideas.

[dead]

These probability shifts would only account for the final output layer (which may also have some shift), but I expect the largest shift to be in the activations in the intermediate latent space. There are a bunch of papers out there that try to get some offset vector using PCA or similar to tune certain model behaviours like vulgarity or friendlyness. You don't even need much data for this as long as your examples capture the essence of the difference well. I'm pretty certain you could do this with "historicalness" too, but projecting it into the future by turning the "contemporaryness" knob way up probably won't yield an accurate result. There are too many outside influences on language that won't be captured in historical trends.

What if it's just genAlpha slang?

> The answer is a resounding “no”.

This is your assertion made without any supportive data or sources. It's nice to know your subjective opinion on the issue but your voice doesn't hold much weight making such a bold assertion devoid of any evidence/data.

I understand where you are coming from, but not every field is hard science. In many fields we deal with some amount of randomness and attribute causality to correlations even if we do not have as much as a speculative hypothesis for a mechanism of action behind the supposed causality.

LLMs trained on data up to a strictly constrained point are our best vehicle to have a view (however biased) on something, detached from its origins and escape a local minima. The speculation is that such LLMs could help us look at correlational links accepted as truths and help us devise an alternative experimental path or craft arguments for such experiments.

Imagine you have an LLM trained on papers up to some threshold, feed your manuscript with correlational evidence and have an LLM point out uncontrolled confounders or something like that.

I think it's pretty likely the answer is no, but the idea here is that you could actually test that assertion. I'm also pessimistic about it but that doesn't mean it wouldn't be a little interesting to try.

I'm sorry but this is factually incorrect and I'm not sure what experts you are referring to here about there being concensus on this topic. I would love know. Geoffrey Hinton, Demis Hassabis, and Yann LeCun all heavily disagree with what you claim.

I think you might be confusing creation ex nihilo with combinatorial synthesis which LLMs excel at. The proposed scenario is a fantastic testcase for exactly this. This doesn't cover verification of course but that's not the question here. The question is wether an already known valid postulate can be synthesized.

I think the question is more about the concept, rather than the specific LLM architectures of today.

> but that is the answer that is a consensus among experts

Do you have any resources that back up such a big claim?

> relatively small mount of focus & critical thinking on the issue of how LLMs & other categories of “AI” work.

I don't understand this line of thought. Why wouldn't the ability to recognize patterns in existing literature or scientific publications result in potential new understandings? What critical thinking am I not doing?

> postulate “new” scientific ideas

What are you examples of "new" ideas that aren't based on existing ones?

When you say "other categories of AI", you're not including AlphaFold, are you?

Models today will be biased based on what's in their training data. If English, it will be biased heavily toward Western, post-1990's views. Then, they do alignment training that forces them to speak according to the supplier's morals. That was Progressive, atheist, evolutionist, and CRT when I used them years ago.

So, the OP model will accidentally reflect the biases of the time. The current, commercial models intentionally reflect specific biases. Except for uncensored models which accidentally have those in the training data modified by uncensoring set.

> but it definitely has some bias.

to be frank though, I think this a better way than all people's thoughts all of the time.

I think the "crowd" of information makes the end output of an LLM worse rather than better. Specifically in our inability to know really what kind of Bias we're dealing with.

Currently to me it feels really muddy knowing how information is biased, beyond just the hallucination and factual incosistencies.

But as far as I can tell, "correctness of the content aside", sometimes frontier LLMs respond like freshman college students, other times they respond with the rigor of a mathematics PHD canidate, and sometimes like a marketing hit piece.

This dataset has a consistency which I think is actually a really useful feature. I agree that having many perspectives in the dataset is good, but as an end user being able to rely on some level of consistency with an AI model is something I really think is missing.

Maybe more succinctly I want frontier LLM's to have a known and specific response style and bias which I can rely on, because there already is a lot of noise.

Biases exposed through artificial constraints help to make visible the hidden/obscured/forgotten biases of state-of-the-art systems.

Average small language model response

Interesting that it reads a bit like it came from a Markov chain rather than an LLM. Perhaps limited training data?

Just have the models converse with each other?

Take a look at The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text (https://arxiv.org/pdf/2506.05209). They build a reasonable 7B parameter model using only open-licensed data.

I'm reminded of SD models that put vaguely-shaped Patreon logos in the corner.

Currently running it using LM Studio. It can download it from Hugging Face. It generates incoherent text though

===

You:

I pray you, who is this Master Newton?

timecapsulellm-v2-1800-1875-mlx:

TI offer to pay you the very same fee as you did before. It was not in the power of your master to deliver the letter to your master. He did. I will be with you as soon as I can keep my word. It is not at all clear, whether the letter has been sent or not. It is not at all clear: but it is clear also that it was written by the person who gave it. "No," I said, "I cannot give it to you." There, the letter was sent to me. "The letter is yours, I believe," I said. "But, I hope, you will not refuse to give it to me?

Not a direct answer but it looks like v0.5 is a nanoGPT arch and v1 is a Phi 1.5 arch, which should be well supported by quanting utilities for any engine. They are small too and should be able to be done on a potato.

I too have completely forgotten how the adapters library works and would have appreciated a simple inference script

+1

Meanwhile, top comments are having visions about the emergence of Einstein-level insights.

There's a disconnect somewhere that I can't quite put my finger on. Am I just lacking imagination?

That was far more amusing than I thought it'd be. Now we can feed those into an AI image generator to create some "art".

Yup that'd be very interesting. Notably missing from this project's list is the KJV (1611 was in use at the time.) The first random newspaper that I pulled up from a search for "london newspaper 1950" has sermon references on the front page so it seems like an important missing piece.

Somewhat missing the cutoff of 1875 is the revised NT of the KJV. Work on it started in 1870 but likely wasn't used widely before 1881.

The problem is that the 'genius' of Einstein wasn't just synthesizing existing data,but actively rejecting the axioms of that data. The 1875 corpus overwhelmingly 'proves' absolute time and the luminiferous aether. A model optimizing for the most probable continuation will converge on that consensus.

To get Relativity, the model needs to realize the training data isn't just incomplete, but fundamentally wrong. That requires abductive reasoning (the spark of genius) to jump out of the local minimum. Without that AGI-level spark, a 'pure knowledge pile' will just generate a very eloquent, mathematically rigorous defense of Newtonian physics.

> I would have a chance to get a rudimentary insight on what the world was like at that time

Congratulations, you've reinvented the history book (just with more energy consumption and less guarantee of accuracy)

> are taking popular benchmark solutions into their training data

That happened in the past, and the "naive" way of doing it is usually easy to spot. There are, however, many ways in which testing data can leak into models, even without data contamination. However this doesn't matter much, as any model that only does well in benchmarks but is bad in real-world usage will be quickly sussed out by people actually using them. There are also lots and lots of weird, not very popular benchmarks out there, and the outliers are quickly identified.

> perform on SWE-bench if trained only on data prior to 2024.

There's a benchmark called swe-REbench, that takes issues from real-world repos, published ~ monthly. They perform tests and you can select the period and check their performance. This is fool-proof for open models, but a bit unknown for API-based models.

GPT2 at 774m is considered a LLM. I wouldn't say there's much difference between that and 700m, or even 123M.

Having said that, looking up small language model these days returns tons of results calling 7B models small language models.

------

My understanding of small language models is that they're generally intended for specific purposes, like analysis and classification (whatever you'd call the text equivalent of image interrogation with clip models), translation, etc; that there small because they don't need to be big to do their intended functions, not because they're just smaller versions of bigger models.

Doesn’t that seem useful though? Isn’t that why I’m forced to read “This movie was made a time when racial stereotypes were different and not well considered” or whatever on old movies?

I think talking to a legit trained LLM from a different era would be rad. But… this seems the opposite of Gemini making black lady popes and Native American Nazis… that these views wouldn’t really be “allowed” (published by anyone that wants AI funding money).