Georgia Tech Team Traces OLMo 3 Social Reasoning

Georgia Tech doctoral candidate Glenn Matlin and co-author Chandreyi Chakraborty mapped how training data shapes model behavior using the open OLMo 3 ecosystem. By leveraging influence functions, the researchers analyzed how specific documents in the training corpus affected the model's performance on downstream tasks. This deep audit was only possible because the Allen Institute for AI released the entire stack, including the model weights, training data, and evaluation tools.

The study utilized five components of the open stack: the OLMo 3 model, the Dolma 3 dataset, the WebOrganizer labeling system, OlmoEval benchmarks, and the OLMES scoring standard. Because Dolma 3 contains roughly 1.26 billion documents, the team sampled 5.68 million texts across 576 categories. They measured how these samples influenced OLMo 3's performance on four benchmarks: SocialIQA and ARC-Challenge for reasoning, alongside the STEM and social-science subsets of MMLU for factual knowledge.

The researchers discovered that social-science knowledge closely mirrored STEM knowledge in its data dependencies, while social reasoning on SocialIQA was a distinct outlier. Performance on SocialIQA depended heavily on dialogue-rich, interpersonal writing, such as literature, customer support, and Q&A threads. To prove causality, the team forced OLMo 3 to unlearn influential documents from the literature category, which caused its SocialIQA score to drop significantly more than when random documents were removed.

For AI practitioners, these findings offer a blueprint for targeted dataset curation. Instead of blindly scaling training data, developers can use these tracing techniques to strategically enrich, reweight, or prune corpora to cultivate specific capabilities. Furthermore, it highlights how fully open-source models enable independent safety researchers to audit AI behaviors without relying on proprietary labs.