SCALAR: quantifying structural hallucination, consistency, and reasoning gaps in materials foundation models
Digital Discovery, 2026 (ESCI, Scopus)
- Yayın Türü: Makale / Tam Makale
- Basım Tarihi: 2026
- Doi Numarası: 10.1039/d6dd00393a
- Dergi Adı: Digital Discovery
- Derginin Tarandığı İndeksler: Emerging Sources Citation Index (ESCI), Scopus, INSPEC, Directory of Open Access Journals
- Ankara Üniversitesi Adresli: Evet
Özet
Large language models are increasingly applied to materials science reasoning, yet their behavior under physically structured distribution shifts remains poorly understood. We introduce SCALAR, a benchmark for evaluating geometric scale generalization and its connection to structural hallucination, consistency, and reasoning in materials foundation models. Starting from canonical crystal representations, models reason about derived nanoparticle structures obtained through supercell expansion and spherical carving from literature-grounded bulk unit cells with documented provenance across length scales spanning 25 to 18 123 atoms: 1743 base structures (83 materials × 21 radii), augmented by split-aware rotations to ≈100 000 structures across training, in-distribution and out-of-distribution evaluation splits. SCALAR defines three tasks: (i) CIF → property prediction; (ii) a chain-of-thought variant with explicit physics-grounded reasoning; and (iii) inverse retrieval identifying crystals from candidates given target properties. Outputs are evaluated via structured metrics capturing numeric error, hallucination, cross-prompt consistency, scale-trend reasoning, output validity, and retrieval fidelity. Experiments across diverse foundation models reveal large, model-dependent shifts under explicit reasoning, often reducing hallucination and error, but frequently destabilizing consistency or validity, demonstrating that scale generalization cannot be inferred from accuracy alone.