LangChain Fine-Tunes Qwen to Slash Trace Judging Costs

LangChain, the creator of the LangSmith developer platform, has collaborated with Fireworks to develop a cost-effective trace evaluation model. By fine-tuning the open-source Qwen-3.5-35B model, the companies created a specialized judge designed to identify "perceived error" within live application logs. This metric flags instances where a human user signals that an AI assistant made a mistake, such as by rejecting an action, repeating a query, or offering an explicit correction.

To build the evaluator, the team applied supervised fine-tuning with Low-Rank Adaptation (LoRA) on the Fireworks infrastructure. They gathered training data from two distinct internal systems: a technical Q&A assistant called chat-langchain, which had a 24 percent rate of perceived errors, and a no-code agent builder called Fleet, which carried an 18 percent rate. To isolate user reactions, the developers stripped out all intermediate tool calls, training the model exclusively on the raw dialogue between humans and the AI.

The resulting model achieved remarkable benchmarks, matching or exceeding the accuracy of prominent proprietary models like Claude 3 Haiku, Sonnet, Opus, and GPT-5.5. Furthermore, the model demonstrated strong generalizability. When evaluated on the Fleet dataset after being trained solely on chat-langchain data, the fine-tuned Qwen model still beat every frontier model, showing that the signals of user dissatisfaction remain consistent across different applications.

For software engineers, this breakthrough dramatically alters the economics of LLM observability. Depending on the volume of application logs, running this fine-tuned model is 10 to 100 times cheaper than querying commercial frontier APIs. This cost reduction makes it financially viable to analyze every single user interaction in production. LangChain is currently rolling out the new evaluator to a limited group of early testers, with a broader public release planned in the next month or two.