SCAND finds most open-source LLMs skip company knowledge bases in RAG tests

SCAND says only 2 of 15 open-weight large language models reliably searched internal company documents before using outside information in a study released Nov. 12 in Warsaw. The findings raise concerns for enterprises using private AI assistants, especially around traceability, source control and potential data-leak risk.

Why it matters: - Enterprise AI assistants are only as useful as their ability to pull from approved internal sources first. - SCAND's study suggests many open-source models can write good answers while still failing the retrieval workflow that keeps company data grounded and traceable. - That gap can create governance risk when internal documents should take priority over web-based information.

What happened: - SCAND released a study on Nov. 12 evaluating 15 open-weight large language models for private retrieval-augmented generation, or RAG, systems. - The company tested the models in Warsaw on seven questions covering summarization, information retrieval, instruction following and tool calling. - Only 2 of the 15 models reliably retrieved information from a company knowledge base before using external information.

The details: - 14 of 15 models passed both summarization tests. - SCAND said that result shows summarization quality alone is not enough to judge readiness for enterprise RAG deployments. - Qwen 3.8 27B in 4-bit quantization was the strongest compact option in the evaluation. - The Qwen model scored 6 out of 7 and used about 16 GB of memory. - A 120-billion-parameter model performed worse than the 27-billion-parameter Qwen model on key tasks in the test set. - Parameter count did not reliably predict whether a model would work well as a knowledge-base assistant. - SCAND said the study compared fluent answer generation with correct retrieval from an approved source. - The company said a model can summarize well and still fail to follow the intended retrieval workflow.

Between the lines: - The study points to a common enterprise AI failure mode: strong language ability does not guarantee disciplined source selection. - For organizations, that means model choice is only part of the decision. - Retrieval architecture, data integration and source-priority rules matter just as much. - SCAND said enterprises need explicit controls if they want AI systems to ground answers in internal documents rather than outside sources.

What's next: - SCAND recommends testing models in realistic workflows instead of relying only on benchmark scores, parameter counts or summarization results. - The company says evaluation should include retrieval reliability, instruction adherence, tool use, memory needs and source-priority compliance. - The full methodology, model comparison and recommendations are available in the study Best Local LLM for RAG: What 15 Open-Source Models Taught Us About Knowledge-Base Assistants. - SCAND also says its RAG development services help businesses build AI systems that retrieve information from private knowledge bases and respond using approved internal data.

The bottom line: - For enterprise AI, good answers are not enough. The system must also choose the right source first.