iFixAi

iFixAi is a diagnostic CLI for auditing AI agents, built for teams who ship LLM-based agents and need to know whether the agent actually does its job, not just whether it's fast or cheap. Most eval and observability tools measure token efficiency, latency, or prompt-injection resistance. iFixAi instead runs a suite of inspections that check an agent's behavior against business KPIs and organizational guardrails, then returns a letter grade (A-F) with a scored breakdown.

It's aimed at developers and teams shipping agents into production who want an independent check before something breaks in front of a customer or an auditor. It works against bare model APIs, your own deployed agent over HTTP, or through a plugin inside agent IDEs like Claude Code and Codex.

Key features

• Five-pillar scorecard: groups 60 inspections into 25 categories, with five core pillars (including Fabrication, which flags unauthorized tool use, missing audit trails, and unsourced claims) feeding a graded A-F result.
• Three run modes: a guided wizard (ifixai setup + ifixai run), explicit CLI flags for scripting and CI, or a plugin/skill that lets the agent itself drive the audit.
• Independent grading: a "citable" run requires two API keys from different vendors, one for the system under test and one for the judge, so the agent being tested never grades itself.
• Provider-agnostic adapters: works with OpenAI, Anthropic, Gemini, Azure, Bedrock, Hugging Face, OpenRouter, and others, plus HTTP and LangChain adapters for custom agents.
• Real-agent testing: --provider http --grounding sut points iFixAi at your deployed agent's own endpoint to measure the governance it actually enforces, rather than a bare model.
• Configurable suites: smoke (3 tests), strategic (8), core (32, the graded scorecard), extended (28, frontier risk signal), and all (60).
• Reusable config: ifixai setup writes ifixai.yaml, storing provider, model, and the env-var name for the key (never the secret itself).
• Reports in JSON and Markdown, plus a terminal scorecard, landing in ./ifixai-results/.

Ideal use cases

Use iFixAi when you're about to ship an agent to production and want a pre-launch check on whether it hallucinates, uses tools it shouldn't, or drifts from the business logic it's supposed to follow. It fits CI pipelines (via explicit flags) for repeat, scripted audits, and fits teams who need an audit-ready, independently judged grade rather than a self-reported one, which matters for compliance contexts referencing frameworks like the EU AI Act or ISO 42001.

It also works well inside agent-building workflows: if you're developing inside Claude Code, Codex, Cursor, or similar, the plugin or skill lets the agent run the audit on itself and explain the results interactively.

It's not a fit if you only need latency/token-cost benchmarking or prompt-injection red-teaming in isolation, there are more narrowly focused tools for that. It's also not meant as a bare-model leaderboard: testing a raw model API without tools, retrieval, or a system prompt will score lower simply because there's less surface to evaluate, and the tool is explicit that your real agent (not the underlying model) is the better test target.

Installation

Install via pip, choosing the provider extra you'll test against:

pip install "ifixai[openai]"   # or anthropic, gemini, etc.
ifixai setup                    # guided wizard: pick provider, model, judge, suite
ifixai run                      # reports land in ./ifixai-results/

For a zero-key pipeline check using the built-in mock provider:

ifixai run --provider mock --api-key not-used --eval-mode self

For a citable, independently-judged run, install SDKs for both the system-under-test and judge providers, export both keys, and run:

pip install "ifixai[anthropic,openai]"
export ANTHROPIC_API_KEY=sk-ant-...
export OPENAI_API_KEY=sk-...
ifixai run --provider anthropic --api-key "$ANTHROPIC_API_KEY" --fixture ./my-fixture.yaml

To scaffold a slash-command skill into any supported agent (Claude Code, Codex, Cursor, VS Code, Windsurf, Cline, Continue, Gemini, Zed), use uv with Python 3.10+ and no API key needed to scaffold:

uvx ifixai install --agents cursor   # or claude, codex, vscode, windsurf, cline, continue, gemini, zed
uvx ifixai install --agents all      # scaffold every supported agent

For Claude Code or Codex, you can also install it as a native plugin:

/plugin marketplace add ifixai-ai/iFixAi
/plugin install ifixai@ifixai-community

Requires Python 3.10+. Full walkthroughs are in docs/get-started.md and docs/testing-your-agent.md.