Original Summary

Setup: FinanceBench , 150 analyst-written questions over 84 real 10-Ks/10-Qs, about 55k chunks in one index. Everything local: nomic-embed-text + Llama 3.1 8B on Ollama, on a laptop. The metric is whether the exact evidence page lands in the top 5 passages given to the LLM. Retrieval setup Evidence page in top 5 Right filing in top 5 Plain vector search 25% 78% Hybrid (BM25 + vector) 25% 71% + entity graph 25% 71% + cross-encoder reranker 29% 77% + source routing 29% 87% Routing + reranker 39% 93% "Source routing" is simple: if the question names a company that matches a file name, search only that file. With 84 filings full of the same finance vocabulary, plain search keeps landing in the wrong company's 10-K. Routing plus a reranker gets the evidence page about 1.6× as often, with no extra LLM calls. Things that surprised me: Hybrid search didn't help by itself. Long analyst questions match generic finance wording in every filing, so BM25 pulled in look-alike passages from other companies. The entity graph I was proud of changed nothing. The benchmark caught real bugs: 39 of the 84 PDFs were encrypted and being silently skipped, and my RAG-Fusion was literally searching for the model's "Here are three alternative queries:" line. Still running: HyDE, RAG-Fusion, CRAG and the "everything on" config, plus end-to-end answer accuracy compared with a naive baseline. I also found something odd about embeddings differing between two Ollama versions on the same machine, which I want to verify before I say more. I'll post the full results, including the configs that made things worse. The harness is in the repo if you want to run it on your own documents: https://github.com/SaiAkhil066/CORTEX-AI-SUPER-RAG/tree/main/eval What else should I test? A different embedding model and chunk size are next on my list.   submitted by   /u/Overall_Mission_1781 [link]   [comments]

  • 情报分类：商业与市场研究
  • 分类依据：内容涉及商业、投资或市场动态
  • 信息来源：Reddit · SaaS
  • 发布时间：2026/9/27 23:58:55