Original Summary

I've been building a RAG chatbot/agent starter kit (Next.js + TypeScript) with nine guardrails: a prompt-injection filter, context isolation for retrieved documents, a tool allowlist, human approval for actions, PII redaction, and a few more. For a while, a green test suite was my evidence that they worked. Then I did what I should have done first: take each guardrail out of the chain, replay the recorded attacks, and see which ones start getting through. Of the six guardrails in the chain, only two changed any outcome: human-in-the-loop and PII redaction. The other four were declared, not demonstrated. The suite would have kept printing STOPPED with them deleted. What I changed after that: - Every recorded attack now names the guardrail that stops it, and a test proves it by removal: take it out, the attack lands, put it back. - The test prints the guardrails that no recorded attack proves. The README used to say the suite proved all of them; that claim is gone. - Direct prompt injection is listed as unproven, not stopped. The model I recorded against resisted it 20 out of 20 times with no guardrails at all, so there's no successful attack to replay. I'd rather say that than go looking for a weaker model to manufacture one. How the replay works: each attack was recorded once against a local 7B model (qwen2.5 via Ollama), and the suite replays the recorded model output, so CI needs no model and a guardrail change shows up as a changed verdict. The obvious limit is that it's one model. There's a live mode that runs the same attacks against the model you actually use. The fifth attack came from two comments on an earlier post. The payload doesn't come from the user or a document; it comes back inside a tool result (reading an inbox). The fix went into the human-approval gate: an address the model finds in a message body needs a human to approve it, while replying to the actual sender still goes through The attack catalogue and replay harness are MIT: https://github.com/Sergiobm99/secure-ai-kit-attacks The kit itself is paid (one-time). The per-attack results are public: https://secureaikit.com/proof Two things I'd like to hear about: how do you check that a guardrail is actually load-bearing and not just present? And has anyone found a way to test against several models that doesn't turn into a flaky mess?   submitted by   /u/VastStorage4125 [link]   [comments]

  • 情报分类：技术学习与提效
  • 分类依据：内容涉及技术、AI、软件工具或工程实践
  • 信息来源：Reddit · SideProject
  • 发布时间：2026/9/26 23:49:02