AI Safety Test Kit

Test your AI assistant before each release and add a safety layer that logs, flags and can switch it off.

Your first result: a pass/fail test report for the included example assistant.

For developers~5 min to first test run (est.)

$49

One-time · instant download

Coming soon. Leave your email and we'll tell you the day it opens.

I don't sell your email address. You can unsubscribe at any time using the link in every email. See the Privacy Policy.

AI Safety Test Kit card art

What you get

What's inside the kit

  • A test runner for AI assistants

    Run the same test questions before every release and get JSON, Markdown or JUnit reports.

  • 11 ready-made scoring checks

    Score answers automatically instead of reading them all by hand.

  • Checks for prompt-injection and data leaks

    Helps catch common attack and leak patterns (it can still miss some).

  • A redacted audit log

    Keeps a record of what your AI assistant did, with secrets masked.

  • A kill switch

    Stop your AI assistant instantly without redeploying.

  • A before-and-after comparison

    Fails your build if something that used to pass starts failing.

How to put it to work

How to get started

Follow these steps in order. After you buy, the same steps appear as a checklist in your kit viewer, with links to the exact guide for each step.

  1. 1

    Run the built-in checks

    5 min

    Run npm test, then npm run eval to see a report on the 12 example cases.

  2. 2

    Write your own test suite

    20–30 min

    Copy the example JSON suite and add your assistant's real questions and what should happen.

  3. 3

    Connect your real assistant

    15–30 min

    Put your assistant behind the safety wrapper so risky tool calls wait for a person to approve them.

  4. 4

    Save a baseline

    5 min

    Record today's results, review and commit them. From then on, the test run fails if a passing case breaks.

  5. 5

    Know your kill switch, then add CI

    10 min

    Create the kill-switch file (or set the env var) to stop the assistant. Then copy the read-only GitHub workflow.

Is it right for me?

Who it's for

A great fit if…

  • Developers and agencies delivering AI assistants to clients
  • Node.js projects (your assistant connected through a small JavaScript file)

Probably not for you if…

  • You need a security audit or compliance certification

Questions

Frequently asked questions

Will it stop every attack?

No. The safety checks are pattern-based and can miss paraphrased, non-English or encoded attacks. They help catch common patterns.

Does it call OpenAI or Anthropic?

Not by default. Everything runs offline with a mock model. Live adapters are optional, off by default and untested live.

Do I need to install packages?

No. It has no npm dependencies; Node.js 20+ is enough.

Good to know before you buy

  • The guardrails are pattern-based and can miss attacks (paraphrased, non-English or encoded injection, jailbreaks and leaks). They help catch common patterns; they don't fully detect or prevent anything.
  • Provided as is, without warranty; liability capped at the price you paid.
  • Not a security audit, penetration test or compliance certification.
  • No promise of any outcome, time savings or income.
Technical details (for the technically curious)

Technical words on this page, explained

AI agent:
An AI assistant that can take steps for you (like running a command or calling a tool), not just chat.
LLM / AI model:
The AI "brain" behind tools like ChatGPT or Claude. You usually rent it from a provider with an account key.
Evals:
Repeatable tests for an AI assistant: the same questions every time, so you can see if it got better or worse.
Guardrails:
Safety checks around an AI assistant that flag or block risky requests and answers.
CLI / terminal:
Typing short commands into a text window on your computer instead of clicking buttons.
JSON / Markdown:
Simple text file formats. Markdown is readable notes; JSON is structured data for programs.
CI / GitHub workflow:
Automatic checks that run every time code changes, so problems are caught early.

Formerly / technical name: Agent Eval & Guardrails Harness

Full description

A small, dependency-free Node toolkit to help you test, guard and audit LLM agents: eval runner and CLI, regression gate, redacted audit log, kill switch and release checklist. Runs fully offline by default. Not a security audit or compliance certification.

Full parts list

  • Eval runner + CLI (JSON / Markdown / JUnit reports)
  • 11 scorers
  • Pattern-based injection/leak checks (can miss attacks)
  • Guarded agent wrapper
  • Redacted JSONL audit log
  • Kill switch
  • Regression comparison
  • 12 offline example cases
  • SHA-pinned CI workflow, release checklist

Requirements

  • Node.js 20+
  • Your agent wrapped in a small JS adapter

Tested on

  • Node 20.19.2 / npm 11.21.0
  • Linux 6.12.94+

Tested offline with a mock model and recorded fixtures. OpenAI/Anthropic adapters untested live; CI not run on GitHub. (2026-10-28)

What's in the download (42 files)

  • 📁 .github/ (1 file)
  • 📁 bin/ (1 file)
  • 📁 docs/ (2 files)
  • 📁 evals/ (5 files)
  • 📁 reports/ (4 files)
  • 📁 src/ (15 files)
  • 📁 tests/ (5 files)
  • 📄 .gitignore
  • 📄 .gitleaks.toml
  • 📄 CHANGELOG.md
  • 📄 LICENSE
  • 📄 package.json
  • 📄 README.md
  • 📄 RELEASE_CHECKLIST.md
  • 📄 REVIEW_STATUS.md
  • 📄 SETUP.md

Version and changes

v0.1.0 · tested 2026-10-28

  • 0.1.0 · 2026-10-28: Initial release: eval runner, scorers, guardrails, wrapper, logger, kill switch, reports, adapters. Compliance review: license, disclaimers, live-adapter gate.
AI Safety Test Kit | Your AI Future