Original Summary

Context: we build an open-source (MIT) self-hosted database client and added a natural-language-to-SQL assistant that can point at a local runtime like Ollama instead of a hosted API. I want to share what we learned running local models against real databases, because most NL2SQL demos use a big hosted model and quietly hide the failure modes. This is not a pitch, just notes, and I am happy to be told where our setup is wrong. What breaks on a local model: Schema selection is the accuracy. You cannot paste a 400 table schema into a short context window. On a hosted model with a long context, picking the relevant tables is a nicety. On a local 7B or 8B run with a 4k to 8k context to fit the GPU, that selection is the whole ballgame: pack in unrelated tables and the model chases them into wrong SQL. Most of our work was not prompting, it was deciding what not to send. Clean-looking wrong SQL. Smaller models produce SQL that is syntactically perfect but semantically wrong: the right shape, the wrong table or column. It runs without an error and returns rows, so a check that only asks "did it run" would pass it. We keep a human between generation and execution for exactly this, and the smaller the model the more that split earns its place. Read-only has to be the database's job, not a regex over the SQL. We enforce it with a read-only transaction on Postgres, PRAGMA query_only on SQLite, and so on. One thing that surprised us: a read-only transaction does not stop server-side file access (COPY TO, local-file functions), so the assistant needs its own least-privilege role, not the owner. The payoff of a local endpoint is the obvious one: the schema, the sample rows and the question never leave the machine. For anyone whose data cannot cross a boundary, that is the difference between using an assistant and not using one. Questions for this sub: which local models have you found actually usable for SQL or structured output at 7B to 14B, and does anyone quantize specifically for keeping JSON or SQL syntax intact? In our testing syntax holds up better than table-name accuracy, but I would like to hear other setups.   submitted by   /u/One-Explorer-8151 [link]   [comments]

  • 情报分类：工作与职业机会
  • 分类依据：内容涉及招聘、求职或职业发展
  • 信息来源：Reddit · SideProject
  • 发布时间：2026/9/24 22:50:26