Testing LLMs on superconductivity research questions

Google Research scientists tested six large language models on expert-level questions about high-temperature superconductivity. The study, published in the Proceedings of the National Academy of Sciences, compared four models with full web access against two systems that used curated sources. A panel of experts scored the responses on six metrics. NotebookLM and a custom-built retrieval-augmented generation system performed best overall. Both systems drew from a library of 1,726 sources that included experimental papers and 15 review articles selected by 12 international experts in the field. Models relying on open web sources tended to mix established theories with speculative ones. The evaluation used 67 questions on topics such as doping levels in LSCO and evidence for quantum critical points in cuprates. NotebookLM scored highest for providing evidence and for offering comprehensive answers with a balance of perspectives. The custom system ranked next in most categories. The models showed weaknesses in temporal context and in interpreting tables and images from scientific papers. The work was a collaboration with Cornell University and Harvard University.