Every number on this page comes from the code that runs the probes. If the code changes, this page changes with it. If you find a place where the two disagree, the page is wrong — tell us at hello@tempzero.io.
| Probe | What it is | How often |
|---|---|---|
| Latency | One short request per model, 30 s timeout. We record total time, HTTP status and error class. | Every 5 min; every 15 min for larger, costlier models |
| Behaviour battery | Twelve fixed prompts at temperature 0 with a small output cap (24 tokens, 96 for JSON). Every raw answer is stored. | Every 60 min; every 240 min for larger models |
| Catalogue | The provider’s own model list (/models), stored as a snapshot. |
Hourly, per provider |
| Network control | A TLS handshake to each provider’s API host and to a neutral host (Cloudflare), with no request sent. | Every 5 min |
The network control exists to answer one question when latency spikes: did the model slow down, did the route to the provider slow down, or did our own server? If the neutral host is slow too, it was us.
Every response also tells us which model actually served it. We store that name on every call, so when a provider re-points an alias, there is a timestamp.
Each prompt asks for something with one correct answer in one exact format, so a change is visible without anyone’s judgement. The prompt text was written once and is never edited — a changed question would make the history worthless.
We don’t use similarity scores from another model. Every signal below can be checked by reading the two raw answers side by side.
| Signal | Meaning | Weight |
|---|---|---|
| Format broken | The answer used to match the required format and now doesn’t (e.g. JSON wrapped in a code fence). | 0.35 |
| Lost correct answer | The model used to get it right and now doesn’t. | 0.35 |
| Fingerprint shift | The exact text differs from the usual answer; scaled by edit distance. | 0.20 |
| Length shift | The answer got much longer or shorter than usual. | 0.10 |
“Usual” means the most common answer among that prompt’s last 20 successful runs. With fewer than 5 runs there is no baseline and no score — the live page says “First reading” instead. The fingerprint is a SHA-256 of the answer with only whitespace normalised, so a single added sentence or code fence changes it.
A single odd reading is noise. An event opens only when the threshold is crossed three runs in a row, and closes on its own when readings return to normal.
| Event | Threshold | Earliest it can open |
|---|---|---|
| Behaviour drift | Model score (mean of its 12 prompt scores) above 0.22 | 3 to 12 hours |
| Latency | Above 3,000 ms | 15 to 45 minutes |
| Errors | Error rate above 4% | 15 to 45 minutes |
| Removal | Model missing from the provider’s catalogue | About 3 hours |
Providers sometimes list an old alias that they never publish in the catalogue but still serve. Those names are excluded from the removal check; otherwise we would raise a false alarm about a model that is working.