All news

Now building: Azathoth 1.5, the editor at full parity

Azathoth 1.5 brings everything the command line can do into the editor: production, model policy, judges and benchmarks, optimization, and spend reporting.

One principle: everything the command line can do, the editor can do. That’s Azathoth 1.5, and we’re building it now. Here’s what it brings to the editor, in order.

1. Production

Promote, roll back, see what’s in production, and invoke it. A Production tab lists every invocation, and each one has its own page: input, result, the reason for any failure, and resume.

2. Sensitivity and model policy

Label inputs public, internal or restricted, and see those labels everywhere. You’re warned before restricted data is shown. Declare a model’s no-retention and no-training policy, with who declared it and when.

3. Judges, feedback and benchmarks

Add judges. Mark runs good or bad, with a reason and a corrected output. Build benchmarks, and turn real runs into cases, production runs included. A case is never added without a person choosing its expected output, and it’s never less sensitive than the run it came from.

4. Optimize

Run optimization, inspect experiments, compare benchmarks, and promote the winner. Then see what it saved: cost per run before and after, at the same score.

5. Spend and reporting

Spend per workflow and per model, by day, week and month, with estimated and billed costs shown apart. Plus quality, reliability, adoption and change history.

No date yet. Follow along on GitHub, or get the release notes below.

Want a hand putting Azathoth to work?

Book a 30-minute workflow review