General-Purpose LLMs Beat Materials Language Models on Alloys, but Only on Terms That Leave the Pretraining Question Open
Saturday edition · November 7, 2026 · 2251 words · 12 min read
A preprint submitted to arXiv on November 2 reports that general-purpose language models predicted alloy properties more accurately than the materials-specific language models built for the job. GatiFlow's arXiv collector first logged the paper at 11:50 UTC on November 4 with four mentions, all from arXiv. The count stayed flat across its first three readings, no other source in GatiFlow's feeds had picked the paper up by the end of November 4, and it is the only materials-science entry among the 40 arXiv papers the collector logged between October 22 and November 4. The other four papers in its cycle cover a drug-discovery agent, automation in healthcare, opinion dynamics and prompting. Every batch of new arXiv papers in that window held exactly five, eight batches in all, so a batch of five is routine rather than a cluster. The footprint is small; the question is not: what does domain-specific pretraining actually buy a materials model, and where does it stop paying?
The authors, YuJie Liu, Dezhen Xue, Jun Sun, Xiangdong Ding, Zhibin Gao and Turab Lookman, predicted strength and elongation for titanium alloys, steels, high-entropy alloys and aluminum alloys. Models from the Gemini and DeepSeek families outscored SciBERT, MatSciBERT and SteelBERT, three encoders pretrained on scientific or materials text. Two further results carry more weight than that headline. In what the authors call guided mode, a cheap prior model (the best-scoring conventional model, such as CatBoost, or SteelBERT for steel) supplies a numerical estimate and the LLM corrects it; the abstract reports that this beats both the prior model and the LLM working from reference examples alone. The domain-specific model does not disappear, it becomes the starting point. And when the LLM's prompt held only the 5 percent of training samples closest to the query by embedding similarity, it did about as well as with the whole training set, which the authors read as quality dominating quantity. They also report that the models' self-declared high confidence lined up with small errors, and they conclude that pairing a general model with a numerical baseline and curated data is cost-effective, "reducing the need for expensive domain-specific pretraining."
The full text narrows each of those claims. The 5 percent result holds in the majority of cases across the alloy systems, not in all of them. Guided mode helped least on steel, where processing routes arrive as free-form prose instead of structured tables. In some comparisons Gemini 2.5 Pro posted a lower R-squared than the conventional machine-learning models while still achieving a lower mean absolute error. The datasets are small: 233 titanium entries, 831 aluminum, 491 high-entropy and 677 steel. And despite its title, version 1 never computes a Wasserstein distance. Every model is scored on a random 8:2 train-test split, and outside the title the phrase out-of-distribution appears only in the discussion. The paper promises a way to grade how far test data sit from the training data; this version does not deliver it.
Four limits keep this from being a verdict. First, it is a first-version preprint with no journal reference, so its numbers have not been replicated. Second, the rivals are BERT-family encoders and tree ensembles such as CatBoost, XGBoost and random forests; the crystal-structure and atomistic models that companies such as CuspAI describe building do not appear. Third, the way the encoders were wired decides the headline. The paper feeds their embeddings into a plain multilayer perceptron. In a January 2026 perspective, Lookman, Liu and Gao reported that this setup reaches an R-squared of only about 0.67 to 0.69 on steel tensile strength, while SteelBERT's original hybrid network, which they said is hard to reproduce without the complete code, reported 0.8256 on steel tensile strength. In the new paper Gemini 2.5 Pro reaches 0.743 to 0.772 on that same property: clearly above the MLP versions, but below the original SteelBERT figure, although the two come from different splits. So the general model beats SteelBERT as the authors could rebuild it, not SteelBERT as first published. Fourth, the paper does not address whether Gemini or DeepSeek saw public alloy datasets during their own pretraining, a leakage question any comparison against web-trained models has to answer.
The closest earlier tests of general-purpose models point the other way. LLM4Mat-Bench, led from Princeton and published in Machine Learning: Science and Technology in 2025, spans about 1.9M crystal structures and concluded that general-purpose chat models struggle at property prediction and that task-specific models are needed. The general models there were Llama, Gemma and Mistral variants of 3 to 9 billion parameters, prompted zero-shot or five-shot without fine-tuning. A September 2026 polymer benchmark from Lawrence Livermore researchers found that graph-based models gave the lowest errors, ahead of GPT-4o and o3-mini queried through the API with no fine-tuning. Both prompted a general model directly. The alloy paper adds a numerical prior and a similarity-selected reference set, and it uses frontier models. That is the point: the answer depends on how the test is built and which model generation sits in it, not on a rule about general versus specialized models. The same paper shows how fast that moves: switching from Gemini 2.5 Flash to Gemini 3.0 Flash raised R-squared and cut mean absolute error on steel for both tensile strength and elongation. Two results would change how much weight this paper deserves: the same guided recipe holding against structure-based models, and a replication on an alloy family held out by a distance measure rather than by a random split.
What several benchmarks do agree on is narrower and better supported: scores on convenient test splits overstate how well specialized models generalize. A 2024 npj Computational Materials study by Sadman Sadeed Omee and colleagues built five categories of out-of-distribution test sets from three MatBench datasets and ran eight graph neural networks on them. On average the models fell well short of their MatBench baselines. CGCNN, ALIGNN and DeeperGATGNN proved the most robust, while coGN and coNGN, the models leading the MatBench leaderboard, held up only on the dielectric targets. A Communications Materials paper led by Kangming Li, published in January 2025, ran more than 700 out-of-distribution tasks and found that even boosted trees did well on most of them, because most heuristic splits leave the test data inside territory the training data already covers; on the genuinely hard tasks, more training data or longer training gave marginal or negative returns. BOOM, from Lawrence Livermore National Laboratory and Binghamton University, tested 15 models on 10 molecular tasks in which out-of-distribution means property values in the sparse tails. No model generalized well everywhere, and even the best had an average out-of-distribution error about three times its in-distribution error. Pretraining ChemBERTa, MoLFormer and Regression Transformer cut their in-distribution error by 12 to 35 percent with negligible change out of distribution, while adding about 4 percent of examples from the out-of-distribution region improved the Chemprop graph model on seven of eight QM9 tasks. The alloy paper's 5 percent result belongs to that family: which data a model sees can matter more than how much.
The tempting reading is that the nine-figure rounds in AI for materials are bets on domain pretraining and that this paper undercuts them. The funding record does not support the first half. New Market Pitch counts $973 million in disclosed AI-materials funding in the 12 months to September 24, 2026, and $850 million of it went to three companies: CuspAI ($450 million Series B, July 2026), Periodic Labs ($300 million seed, September 2025) and Cambium ($100 million Series B, January 2026). Amounts were disclosed for only 17 of the 30 deals, and the median disclosed round was $8.1 million. In that analysis the largest checks fund physical infrastructure: autonomous labs, networks of industrial research partners and factory capacity. CuspAI launched its AI Materials Foundry, a network of more than 45 partners, alongside a round co-led by Kleiner Perkins and NEA. Periodic Labs, founded by Liam Fedus, formerly of OpenAI, and Ekin Dogus Cubuk, formerly of Google Brain and DeepMind, is building autonomous laboratories. Cambium designs and manufactures advanced materials for defense and aerospace. Orbital Industries, formerly Orbital Materials, raised a $50 million Series B in May 2026, led by Plural, to scale a cooling fluid for dense GPU racks and a modular data-center system. What these companies describe building is proprietary data, experiments and integration, and an alloy benchmark tests none of that. The segment closest to the paper is small: deal count in metals and alloys rose from two to six, with $47.8 million disclosed. A round announced after New Market Pitch's September 24 cutoff shows the more typical size: on October 12, IP Group led a £4m pre-Series A for Matter Machine, a University of Liverpool spin-out using AI to find inorganic crystalline materials, exactly the structure-based territory the alloy paper never tested.
CuspAI's research-engineer opening asks for large-scale training and for curation work in the same role. It is hiring three engineers to build foundation models over crystal structures, properties, text and images, trained on terabytes of scientific data across hundreds of GPUs, and it wants each hire to own data curation and evaluation design, work it says will "outlast the modelling decisions." A CuspAI posting dated November 7, 2025, almost a year before this paper, opens its role description with the same need to scale data curation, so curation is a standing priority rather than a hedge against the paper. GatiFlow's hiring collectors logged 922 distinct postings between October 22 and November 4, and none came from CuspAI, Periodic Labs, Orbital Industries or Cambium, so the job-board feeds GatiFlow tracks neither confirm nor contradict a shift in what these companies hire for.
For teams building or buying property-prediction tooling, the paper turns into a cheap experiment. Measure how far your deployment data sits from your training data with a Wasserstein-style distance or any other severity metric, instead of scoring on a random split, which is the step this preprint skipped. Then compare your current domain-specific model, wired the way its authors published it, with a general-purpose LLM that receives a numerical prior and a reference set selected by similarity to each query. Before trusting the LLM's stated confidence, test whether high-confidence answers really carry smaller errors on the deployment set, and run each query several times. In the paper's own steel test, a result also reported in the January perspective, Gemini 2.5 Pro gave identical predictions across three runs for only 34 of 136 samples at temperature zero, against 122 for Gemini 2.5 Flash. If the general model matches yours, you have found a spending decision; if it does not, you hold evidence for the pretraining budget. When a vendor claims a pretrained advantage, ask which accuracy figures come from out-of-distribution splits and how those splits were built.
Two dated events follow. The Data & AI for Materials workshop in Rouen, part of the GDR IAMAT network, runs November 18 to 20; it is free and capped at 40 participants, and its invited program leans toward electron microscopy, multiscale simulation and scientific workflows, with generative and foundation models as one listed topic. AI4MC 2026, the Winter AI for Materials & Chemistry workshop, takes place on December 3 at Microsoft Research New England in Cambridge, Massachusetts, and registration and abstract submission close on November 19 at 6 PM Eastern. Its panel asks how far materials AI is from an AlphaFold moment, with prompts on whether foundation models can work when measurements are scarce, where LLMs help discovery and where they only describe results, and whether physics priors are an essential bias or a ceiling.
Pretraining raises the score on the test you already have. Curation and honest evaluation decide the score on the one you do not.
Sources:
- [2611.03767] Quantifying out-of-distribution data severity with Wasserstein distances: benchmarking large language models for alloy property prediction (https://arxiv.org/pdf/2611.03767)
- Materials Informatics: Emergence To Autonomous Discovery In The Age Of AI (https://arxiv.org/pdf/2601.00742)
- LLM4Mat-bench: benchmarking large language models for materials property prediction (https://arxiv.org/abs/2411.00177)
- LLM4Mat-bench (published version) (https://iopscience.iop.org/article/10.1088/2632-2153/add3bb)
- An open benchmark for machine learning-based polymer property prediction (https://www.researchgate.net/publication/414686493_An_open_benchmark_for_machine_learning-based_polymer_property_prediction)
- Structure-based out-of-distribution (OOD) materials property prediction: a benchmark study (https://www.nature.com/articles/s41524-024-01316-4)
- Probing out-of-distribution generalization in machine learning for materials (https://arxiv.org/abs/2406.06489)
- BOOM: Benchmarking Out-Of-distribution Molecular Property Predictions of Machine Learning Models (https://arxiv.org/pdf/2505.01912)
- AI Materials Discovery funding: where is the money going? (https://newmarketpitch.com/blogs/news/ai-materials-discovery-funding-analysis)
- CuspAI closes $450M Series B at $2.6B valuation, backed by Bezos and UK Sovereign AI Fund (https://dealroom.co/news/139818-cuspai-closes-450m-series-b-at-2-6b-valuation-backed-by-bezos-and-uk-sov/)
- CuspAI Raises 450.0M USD in Series B Funding (https://seedtable.com/companies/cuspai/funding-rounds/series-b-2026-07)
- Former OpenAI and DeepMind researchers raise whopping $300M seed to automate science (https://techcrunch.com/2025/09/30/former-openai-and-deepmind-researchers-raise-whopping-300m-seed-to-automate-science)
- IP Group leads £4m funding round for AI materials spin-out Matter Machine (https://www.hl.co.uk/shares/stock-market-news/company--news/ip-group-leads-A4m-funding-round-for-ai-materials-spin-out-matter-machine)
- Cambium: $100 Million Series B Raised To Scale AI-Driven Advanced Materials Platform (https://pulse2.com/cambium-100-million-series-b-raised-to-scale-ai-driven-advanced-materials-platform)
- Orbital Industries raises $50M Series B for AI-designed data-centre hardware (https://thenextweb.com/news/orbital-industries-50m-series-b-data-center-cooling)
- Research Engineer (Materials Foundation Models) at CuspAI (https://startup.jobs/research-engineer-materials-foundation-models-cuspai-10454804)
- Applied AI/ML Engineer (Materials Foundational Models) at CuspAI (https://opportunities.northzone.com/companies/cuspai/jobs/61752217-applied-ai-ml-engineer-materials-foundational-models)
- Data & AI for Materials — Workflows, data Science and AI for the Discovery of New Materials (https://daim26.sciencesconf.org/)
- AI4MC 2026: Winter AI for Materials & Chemistry Workshop (https://ai4mc26.github.io/)
Disclaimer: This article is generated by GatiFlow Intelligence for informational purposes only. It does not constitute investment advice, recruitment recommendations, or legal guidance. All data is derived from public sources and AI analysis — verify independently before making decisions. Past trends do not guarantee future results.
Where this came from
Every Deep Dive starts from GatiFlow's own pipeline: 13 public developer sources, collected every six hours, with a confidence score and the evidence behind each signal. The same signals, filtered to the topics you follow, are a JSON API.
No credit card required.
Get the next one by email
One article every Saturday morning in your time zone. No account needed, and nothing else is sent to the address.
Double opt-in: you confirm by email first. What we store, and for how long, is in the privacy policy.
Tell me I am wrong
Corrections, the version of this you have lived through, or what you would like covered next. It reaches me directly and is never published. It is kept for two years so it can be read and answered; the privacy policy has the details.
0/2000