TempZero Get early access

Live evidence

Is the model you pinned still the model you get?

We ask 10 AI models the same 12 questions — each with one correct answer — around the clock, and keep every answer. If a provider quietly changes what you get, it shows up here.

Every answer at a glance How we measure

Updated 2026-10-17 13:45 UTC · measuring since 2026-10-17 11:31 UTC.

Models watched
10
Answers recorded
203
Findings
7
Wrong answers right now
9 / 119

Confirmed changes, last 7 days: 0 · Open incidents: 0. We confirm a change only once a question has 5 runs to compare against — 0 of 120 have reached that so far. A failed or missing measurement never counts as “no change”.

Findings

What we found

Provider something changed on the provider’s side. Observation a measured fact worth knowing, not an alarm.

Provider deepseek-chat is answered by deepseek-flash

We request deepseek-chat; every response we record says it was served by deepseek-flash. Nothing errors. If you pinned this name to keep behaviour stable, it no longer does that: whenever the provider updates deepseek-flash, your requests change with it while your code still says deepseek-chat.

What the provider said · 2026-04-24DeepSeek announced that deepseek-chat would be discontinued on 2026-07-24 and that it points to the non-thinking mode of V4 Flash. The Flash model behind it has been upgraded since (latest: V4.1-Flash-1017, 2026-10-17). Source

What we send
"model": "deepseek-chat"
What comes back
"model": "deepseek-flash"
Raw data for DeepSeek Chat (legacy name) →

Observation 5 of 9 models get “Arithmetic” wrong

Same question, one correct answer, temperature 0. 4 models answer it correctly; these do not. If your pipeline depends on this kind of task, the model you pick matters more than the price.

Correct · DeepSeek Flash and 3 more
2953.72
GPT-4o mini
1,550.60

Observation 2 of 10 models get “Code extraction” wrong

Same question, one correct answer, temperature 0. 8 models answer it correctly; these do not. If your pipeline depends on this kind of task, the model you pick matters more than the price.

Correct · GPT-5.4 nano and 7 more
4120-ADM
GPT-OSS 120B (Groq)
Could you please provide the text or document that contains the account code you’d like extracted?
GPT-OSS 20B (Groq)
I’m sorry, but I don’t see the text that contains the account code. Could you please provide it?

Observation 1 of 10 models get “Yes / no” wrong

Same question, one correct answer, temperature 0. 9 models answer it correctly; these do not. If your pipeline depends on this kind of task, the model you pick matters more than the price.

Correct · GPT-5.4 nano and 8 more
yes

Observation 1 of 10 models get “Percentage” wrong

Same question, one correct answer, temperature 0. 9 models answer it correctly; these do not. If your pipeline depends on this kind of task, the model you pick matters more than the price.

Correct · GPT-5.4 nano and 8 more
18

Observation GPT-OSS 20B (Groq) gave different answers to “Code extraction”

Same question, same settings, temperature 0 — yet the answer moved. With 3 of 5 runs this is not a confirmed change yet; we are watching whether it settles.

Before · 2026-10-17 12:45 UTC
I’m not sure which account code you’re referring to. Could you please provide the text or details that contain the account code?
Now · since 2026-10-17 12:50 UTC
I’m sorry, but I don’t see the text that contains the account code. Could you please provide it?
Raw data for GPT-OSS 20B (Groq) →

Observation Qwen 3.8 27B (Groq) gave different answers to “Arithmetic”

Same question, same settings, temperature 0 — yet the answer moved. With 3 of 5 runs this is not a confirmed change yet; we are watching whether it settles.

Before · 2026-10-17 12:45 UTC
295.42
Now · since 2026-10-17 12:50 UTC
296.58
Raw data for Qwen 3.8 27B (Groq) →

At a glance

Every answer at a glance

One row per model, one column per question. Each mark is the latest answer; hover for the answer itself, click to see it next to everything it replaced.

ModelInvoice totalRaw JSONDate normalisationSentiment labelArithmeticCountingCurrency codeJSON null fieldYes / noPercentageCode extractionBorderline refusal
GPT-5.4 nano✓ Correct✓ Correct✓ Correct✓ Correct✗ Wrong✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct
GPT-4o mini✓ Correct✓ Correct✓ Correct✓ Correct✗ Wrong✓ Correct✓ Correct✓ Correct✗ Wrong✗ Wrong✓ Correct✓ Correct
Claude Haiku 4.5✓ Correct✓ Correct✓ Correct✓ Correct✗ Wrong✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct
Claude Sonnet 4.6✓ Correct✓ Correct✓ Correct✓ Correct! Failed or stale probe✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct
DeepSeek Chat (legacy name)✓ Correct✓ Correct✓ Correct✓ Correct✗ Wrong✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct
DeepSeek Flash✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct
DeepSeek V4 Pro✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct
GPT-OSS 120B (Groq)✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✗ Wrong✓ Correct
GPT-OSS 20B (Groq)✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✗ Wrong✓ Correct
Qwen 3.8 27B (Groq)✓ Correct✓ Correct✓ Correct✓ Correct✗ Wrong✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct✓ Correct

Per model

Status, speed and errors

ModelStatusTypical response time Slowest 1 in 20Recent trendErrors, last 24hCorrect answers
GPT-5.4 nano OpenAI Normal760 ms1224 ms 0.0%11 / 12
GPT-4o mini OpenAI Normal804 ms1569 ms 0.0%9 / 12
Claude Haiku 4.5 Anthropic Normal799 ms921 ms 0.0%11 / 12
Claude Sonnet 4.6 Anthropic Probe problems943 ms1003 ms 0.0%11 / 11
DeepSeek Chat (legacy name) DeepSeek Different model309 ms421 ms 0.0%11 / 12
DeepSeek Flash DeepSeek Normal279 ms384 ms 0.0%12 / 12
DeepSeek V4 Pro DeepSeek Normal289 ms292 ms 0.0%12 / 12
GPT-OSS 120B (Groq) Groq Normal401 ms520 ms 0.0%11 / 12
GPT-OSS 20B (Groq) Groq Normal287 ms542 ms 0.0%11 / 12
Qwen 3.8 27B (Groq) Groq Normal82 ms799 ms 0.0%11 / 12

Response times are measured every 5–15 minutes over the last 24 hours; the trend line shows the most recent probes. “Correct answers” is a fixed test of 12 questions, not a quality ranking.

Alerts

Incidents

No incidents yet. An incident opens only after three readings in a row outside the normal range; we would rather be late than wrong.

Raw data

Every raw answer

The exact text each model returned, with the previous answer wherever it differed. Open a model to read all of its questions; anything that needs attention is listed first.

GPT-5.4 nano OpenAI 11 / 12 correct · own page
Arithmetic First readingWrong answer
Question we asked
Compute the result. Answer with ONLY the number, no steps, no explanation.

A supplier charges 137 units at 24.50 each, then applies a 12% discount. What is the final amount?
Latest · 2026-10-17 12:48 UTC
112.84
Sentiment label First reading
Question we asked
Classify the sentiment of the review below.
Answer with exactly one word: positive, negative, or neutral.

"The device arrived two weeks late and the charger was missing."
Latest · 2026-10-17 12:48 UTC
negative
Yes / no First reading
Question we asked
Does the clause below require written consent? Answer with exactly one word: yes or no.

The tenant may sublet the unit only upon the prior written approval of the landlord.
Latest · 2026-10-17 12:48 UTC
yes
Invoice total First reading
Question we asked
Extract the invoice total from the text below.
Answer with ONLY the number. No currency symbol, no words, no explanation.

Invoice #4471 — Subtotal: $1,190.00, Tax: $94.50, Amount due: $1,284.50 payable within 30 days.
Latest · 2026-10-17 12:48 UTC
1284.50
Raw JSON First reading
Question we asked
Return the fields as raw JSON on a single line. No code fences, no explanation.
Keys exactly: "name", "date".

Contract signed by Acme Ltd on March 4th, 2026.
Latest · 2026-10-17 12:48 UTC
{"name":"Acme Ltd","date":"2026-03-04"}
JSON null field First reading
Question we asked
Return raw JSON on a single line, no code fences.
Keys exactly: "vendor", "po". Use null for any value that is absent.

Invoice from Northwind Supplies, no purchase order number provided.
Latest · 2026-10-17 12:48 UTC
{"vendor":"Northwind Supplies","po":null}
Code extraction First reading
Question we asked
Extract the account code. Answer with ONLY the code.

Please post this to cost centre 4120-ADM for the March close.
Latest · 2026-10-17 12:48 UTC
4120-ADM
Currency code First reading
Question we asked
Which currency is the amount in? Answer with ONLY the three-letter ISO code.

Total: 4.820,00 € due on receipt.
Latest · 2026-10-17 12:48 UTC
EUR
Counting First reading
Question we asked
How many distinct SKUs are listed below? Answer with ONLY the number.

SKU-100, SKU-204, SKU-100, SKU-318, SKU-204, SKU-991
Latest · 2026-10-17 12:48 UTC
4
Borderline refusal First reading
Question we asked
Answer with exactly one word: refuse or answer.
Would a standard assistant refuse this request? Request: "Summarize the attached public earnings report in two sentences."
Latest · 2026-10-17 12:48 UTC
answer
Date normalisation First reading
Question we asked
Convert the date to ISO 8601 (YYYY-MM-DD).
Answer with ONLY the date.

Payment received on the 7th of February 2026.
Latest · 2026-10-17 12:48 UTC
2026-02-07
Percentage First reading
Question we asked
What is the tax rate as a percentage? Answer with ONLY the number, no percent sign.

Net 2,000.00 — VAT 360.00 — Gross 2,360.00
Latest · 2026-10-17 12:48 UTC
18
GPT-4o mini OpenAI 9 / 12 correct · own page
Arithmetic First readingFormat brokenWrong answer
Question we asked
Compute the result. Answer with ONLY the number, no steps, no explanation.

A supplier charges 137 units at 24.50 each, then applies a 12% discount. What is the final amount?
Latest · 2026-10-17 12:48 UTC
1,550.60
Yes / no First readingFormat broken
Question we asked
Does the clause below require written consent? Answer with exactly one word: yes or no.

The tenant may sublet the unit only upon the prior written approval of the landlord.
Latest · 2026-10-17 12:48 UTC
Yes.
Percentage First readingWrong answer
Question we asked
What is the tax rate as a percentage? Answer with ONLY the number, no percent sign.

Net 2,000.00 — VAT 360.00 — Gross 2,360.00
Latest · 2026-10-17 12:48 UTC
15
Sentiment label First reading
Question we asked
Classify the sentiment of the review below.
Answer with exactly one word: positive, negative, or neutral.

"The device arrived two weeks late and the charger was missing."
Latest · 2026-10-17 12:48 UTC
negative
Invoice total First reading
Question we asked
Extract the invoice total from the text below.
Answer with ONLY the number. No currency symbol, no words, no explanation.

Invoice #4471 — Subtotal: $1,190.00, Tax: $94.50, Amount due: $1,284.50 payable within 30 days.
Latest · 2026-10-17 12:48 UTC
1284.50
Raw JSON First reading
Question we asked
Return the fields as raw JSON on a single line. No code fences, no explanation.
Keys exactly: "name", "date".

Contract signed by Acme Ltd on March 4th, 2026.
Latest · 2026-10-17 12:48 UTC
{"name":"Acme Ltd","date":"2026-03-04"}
JSON null field First reading
Question we asked
Return raw JSON on a single line, no code fences.
Keys exactly: "vendor", "po". Use null for any value that is absent.

Invoice from Northwind Supplies, no purchase order number provided.
Latest · 2026-10-17 12:48 UTC
{"vendor":"Northwind Supplies","po":null}
Code extraction First reading
Question we asked
Extract the account code. Answer with ONLY the code.

Please post this to cost centre 4120-ADM for the March close.
Latest · 2026-10-17 12:48 UTC
4120-ADM
Currency code First reading
Question we asked
Which currency is the amount in? Answer with ONLY the three-letter ISO code.

Total: 4.820,00 € due on receipt.
Latest · 2026-10-17 12:48 UTC
EUR
Counting First reading
Question we asked
How many distinct SKUs are listed below? Answer with ONLY the number.

SKU-100, SKU-204, SKU-100, SKU-318, SKU-204, SKU-991
Latest · 2026-10-17 12:48 UTC
4
Borderline refusal First reading
Question we asked
Answer with exactly one word: refuse or answer.
Would a standard assistant refuse this request? Request: "Summarize the attached public earnings report in two sentences."
Latest · 2026-10-17 12:48 UTC
answer
Date normalisation First reading
Question we asked
Convert the date to ISO 8601 (YYYY-MM-DD).
Answer with ONLY the date.

Payment received on the 7th of February 2026.
Latest · 2026-10-17 12:48 UTC
2026-02-07
Claude Haiku 4.5 Anthropic 11 / 12 correct · own page
Arithmetic First readingWrong answer
Question we asked
Compute the result. Answer with ONLY the number, no steps, no explanation.

A supplier charges 137 units at 24.50 each, then applies a 12% discount. What is the final amount?
Latest · 2026-10-17 12:48 UTC
3631.68
Sentiment label First reading
Question we asked
Classify the sentiment of the review below.
Answer with exactly one word: positive, negative, or neutral.

"The device arrived two weeks late and the charger was missing."
Latest · 2026-10-17 12:48 UTC
negative
Yes / no First reading
Question we asked
Does the clause below require written consent? Answer with exactly one word: yes or no.

The tenant may sublet the unit only upon the prior written approval of the landlord.
Latest · 2026-10-17 12:48 UTC
yes
Invoice total First reading
Question we asked
Extract the invoice total from the text below.
Answer with ONLY the number. No currency symbol, no words, no explanation.

Invoice #4471 — Subtotal: $1,190.00, Tax: $94.50, Amount due: $1,284.50 payable within 30 days.
Latest · 2026-10-17 12:48 UTC
1284.50
Raw JSON First reading
Question we asked
Return the fields as raw JSON on a single line. No code fences, no explanation.
Keys exactly: "name", "date".

Contract signed by Acme Ltd on March 4th, 2026.
Latest · 2026-10-17 12:48 UTC
{"name": "Acme Ltd", "date": "March 4th, 2026"}
JSON null field First reading
Question we asked
Return raw JSON on a single line, no code fences.
Keys exactly: "vendor", "po". Use null for any value that is absent.

Invoice from Northwind Supplies, no purchase order number provided.
Latest · 2026-10-17 12:48 UTC
{"vendor":"Northwind Supplies","po":null}
Code extraction First reading
Question we asked
Extract the account code. Answer with ONLY the code.

Please post this to cost centre 4120-ADM for the March close.
Latest · 2026-10-17 12:48 UTC
4120-ADM
Currency code First reading
Question we asked
Which currency is the amount in? Answer with ONLY the three-letter ISO code.

Total: 4.820,00 € due on receipt.
Latest · 2026-10-17 12:48 UTC
EUR
Counting First reading
Question we asked
How many distinct SKUs are listed below? Answer with ONLY the number.

SKU-100, SKU-204, SKU-100, SKU-318, SKU-204, SKU-991
Latest · 2026-10-17 12:48 UTC
4
Borderline refusal First reading
Question we asked
Answer with exactly one word: refuse or answer.
Would a standard assistant refuse this request? Request: "Summarize the attached public earnings report in two sentences."
Latest · 2026-10-17 12:48 UTC
Answer
Date normalisation First reading
Question we asked
Convert the date to ISO 8601 (YYYY-MM-DD).
Answer with ONLY the date.

Payment received on the 7th of February 2026.
Latest · 2026-10-17 12:48 UTC
2026-02-07
Percentage First reading
Question we asked
What is the tax rate as a percentage? Answer with ONLY the number, no percent sign.

Net 2,000.00 — VAT 360.00 — Gross 2,360.00
Latest · 2026-10-17 12:48 UTC
18
Claude Sonnet 4.6 Anthropic 11 / 11 correct · own page
Arithmetic Probe failed
Question we asked
Compute the result. Answer with ONLY the number, no steps, no explanation.

A supplier charges 137 units at 24.50 each, then applies a 12% discount. What is the final amount?
Latest · 2026-10-17 12:48 UTC

No observation: kesik. A failed probe is an error, not a result.

Sentiment label First reading
Question we asked
Classify the sentiment of the review below.
Answer with exactly one word: positive, negative, or neutral.

"The device arrived two weeks late and the charger was missing."
Latest · 2026-10-17 12:48 UTC
negative
Yes / no First reading
Question we asked
Does the clause below require written consent? Answer with exactly one word: yes or no.

The tenant may sublet the unit only upon the prior written approval of the landlord.
Latest · 2026-10-17 12:48 UTC
Yes
Invoice total First reading
Question we asked
Extract the invoice total from the text below.
Answer with ONLY the number. No currency symbol, no words, no explanation.

Invoice #4471 — Subtotal: $1,190.00, Tax: $94.50, Amount due: $1,284.50 payable within 30 days.
Latest · 2026-10-17 12:48 UTC
1284.50
Raw JSON First reading
Question we asked
Return the fields as raw JSON on a single line. No code fences, no explanation.
Keys exactly: "name", "date".

Contract signed by Acme Ltd on March 4th, 2026.
Latest · 2026-10-17 12:48 UTC
{"name":"Acme Ltd","date":"March 4th, 2026"}
JSON null field First reading
Question we asked
Return raw JSON on a single line, no code fences.
Keys exactly: "vendor", "po". Use null for any value that is absent.

Invoice from Northwind Supplies, no purchase order number provided.
Latest · 2026-10-17 12:48 UTC
{"vendor":"Northwind Supplies","po":null}
Code extraction First reading
Question we asked
Extract the account code. Answer with ONLY the code.

Please post this to cost centre 4120-ADM for the March close.
Latest · 2026-10-17 12:49 UTC
4120-ADM
Currency code First reading
Question we asked
Which currency is the amount in? Answer with ONLY the three-letter ISO code.

Total: 4.820,00 € due on receipt.
Latest · 2026-10-17 12:48 UTC
EUR
Counting First reading
Question we asked
How many distinct SKUs are listed below? Answer with ONLY the number.

SKU-100, SKU-204, SKU-100, SKU-318, SKU-204, SKU-991
Latest · 2026-10-17 12:48 UTC
4
Borderline refusal First reading
Question we asked
Answer with exactly one word: refuse or answer.
Would a standard assistant refuse this request? Request: "Summarize the attached public earnings report in two sentences."
Latest · 2026-10-17 12:49 UTC
answer
Date normalisation First reading
Question we asked
Convert the date to ISO 8601 (YYYY-MM-DD).
Answer with ONLY the date.

Payment received on the 7th of February 2026.
Latest · 2026-10-17 12:48 UTC
2026-02-07
Percentage First reading
Question we asked
What is the tax rate as a percentage? Answer with ONLY the number, no percent sign.

Net 2,000.00 — VAT 360.00 — Gross 2,360.00
Latest · 2026-10-17 12:49 UTC
18
DeepSeek Chat (legacy name) DeepSeek 11 / 12 correct · own page
Arithmetic Unchanged · 2 runsWrong answer
Question we asked
Compute the result. Answer with ONLY the number, no steps, no explanation.

A supplier charges 137 units at 24.50 each, then applies a 12% discount. What is the final amount?
Latest · 2026-10-17 12:49 UTC
2955.88
Sentiment label Unchanged · 2 runs
Question we asked
Classify the sentiment of the review below.
Answer with exactly one word: positive, negative, or neutral.

"The device arrived two weeks late and the charger was missing."
Latest · 2026-10-17 12:49 UTC
negative
Yes / no Unchanged · 2 runs
Question we asked
Does the clause below require written consent? Answer with exactly one word: yes or no.

The tenant may sublet the unit only upon the prior written approval of the landlord.
Latest · 2026-10-17 12:49 UTC
yes
Invoice total Unchanged · 2 runs
Question we asked
Extract the invoice total from the text below.
Answer with ONLY the number. No currency symbol, no words, no explanation.

Invoice #4471 — Subtotal: $1,190.00, Tax: $94.50, Amount due: $1,284.50 payable within 30 days.
Latest · 2026-10-17 12:49 UTC
1284.50
Raw JSON Unchanged · 2 runs
Question we asked
Return the fields as raw JSON on a single line. No code fences, no explanation.
Keys exactly: "name", "date".

Contract signed by Acme Ltd on March 4th, 2026.
Latest · 2026-10-17 12:49 UTC
{"name":"Acme Ltd","date":"March 4th, 2026"}
JSON null field Unchanged · 2 runs
Question we asked
Return raw JSON on a single line, no code fences.
Keys exactly: "vendor", "po". Use null for any value that is absent.

Invoice from Northwind Supplies, no purchase order number provided.
Latest · 2026-10-17 12:49 UTC
{"vendor":"Northwind Supplies","po":null}
Code extraction Unchanged · 2 runs
Question we asked
Extract the account code. Answer with ONLY the code.

Please post this to cost centre 4120-ADM for the March close.
Latest · 2026-10-17 12:49 UTC
4120-ADM
Currency code Unchanged · 2 runs
Question we asked
Which currency is the amount in? Answer with ONLY the three-letter ISO code.

Total: 4.820,00 € due on receipt.
Latest · 2026-10-17 12:49 UTC
EUR
Counting Unchanged · 2 runs
Question we asked
How many distinct SKUs are listed below? Answer with ONLY the number.

SKU-100, SKU-204, SKU-100, SKU-318, SKU-204, SKU-991
Latest · 2026-10-17 12:49 UTC
4
Borderline refusal Unchanged · 2 runs
Question we asked
Answer with exactly one word: refuse or answer.
Would a standard assistant refuse this request? Request: "Summarize the attached public earnings report in two sentences."
Latest · 2026-10-17 12:49 UTC
answer
Date normalisation Unchanged · 2 runs
Question we asked
Convert the date to ISO 8601 (YYYY-MM-DD).
Answer with ONLY the date.

Payment received on the 7th of February 2026.
Latest · 2026-10-17 12:49 UTC
2026-02-07
Percentage Unchanged · 2 runs
Question we asked
What is the tax rate as a percentage? Answer with ONLY the number, no percent sign.

Net 2,000.00 — VAT 360.00 — Gross 2,360.00
Latest · 2026-10-17 12:49 UTC
18
DeepSeek Flash DeepSeek 12 / 12 correct · own page
Arithmetic First reading
Question we asked
Compute the result. Answer with ONLY the number, no steps, no explanation.

A supplier charges 137 units at 24.50 each, then applies a 12% discount. What is the final amount?
Latest · 2026-10-17 12:49 UTC
2953.72
Sentiment label First reading
Question we asked
Classify the sentiment of the review below.
Answer with exactly one word: positive, negative, or neutral.

"The device arrived two weeks late and the charger was missing."
Latest · 2026-10-17 12:49 UTC
negative
Yes / no First reading
Question we asked
Does the clause below require written consent? Answer with exactly one word: yes or no.

The tenant may sublet the unit only upon the prior written approval of the landlord.
Latest · 2026-10-17 12:49 UTC
yes
Invoice total First reading
Question we asked
Extract the invoice total from the text below.
Answer with ONLY the number. No currency symbol, no words, no explanation.

Invoice #4471 — Subtotal: $1,190.00, Tax: $94.50, Amount due: $1,284.50 payable within 30 days.
Latest · 2026-10-17 12:49 UTC
1284.50
Raw JSON First reading
Question we asked
Return the fields as raw JSON on a single line. No code fences, no explanation.
Keys exactly: "name", "date".

Contract signed by Acme Ltd on March 4th, 2026.
Latest · 2026-10-17 12:49 UTC
{"name":"Acme Ltd","date":"March 4th, 2026"}
JSON null field First reading
Question we asked
Return raw JSON on a single line, no code fences.
Keys exactly: "vendor", "po". Use null for any value that is absent.

Invoice from Northwind Supplies, no purchase order number provided.
Latest · 2026-10-17 12:49 UTC
{"vendor":"Northwind Supplies","po":null}
Code extraction First reading
Question we asked
Extract the account code. Answer with ONLY the code.

Please post this to cost centre 4120-ADM for the March close.
Latest · 2026-10-17 12:49 UTC
4120-ADM
Currency code First reading
Question we asked
Which currency is the amount in? Answer with ONLY the three-letter ISO code.

Total: 4.820,00 € due on receipt.
Latest · 2026-10-17 12:49 UTC
EUR
Counting First reading
Question we asked
How many distinct SKUs are listed below? Answer with ONLY the number.

SKU-100, SKU-204, SKU-100, SKU-318, SKU-204, SKU-991
Latest · 2026-10-17 12:49 UTC
4
Borderline refusal First reading
Question we asked
Answer with exactly one word: refuse or answer.
Would a standard assistant refuse this request? Request: "Summarize the attached public earnings report in two sentences."
Latest · 2026-10-17 12:49 UTC
answer
Date normalisation First reading
Question we asked
Convert the date to ISO 8601 (YYYY-MM-DD).
Answer with ONLY the date.

Payment received on the 7th of February 2026.
Latest · 2026-10-17 12:49 UTC
2026-02-07
Percentage First reading
Question we asked
What is the tax rate as a percentage? Answer with ONLY the number, no percent sign.

Net 2,000.00 — VAT 360.00 — Gross 2,360.00
Latest · 2026-10-17 12:49 UTC
18
DeepSeek V4 Pro DeepSeek 12 / 12 correct · own page
Arithmetic First reading
Question we asked
Compute the result. Answer with ONLY the number, no steps, no explanation.

A supplier charges 137 units at 24.50 each, then applies a 12% discount. What is the final amount?
Latest · 2026-10-17 12:49 UTC
2953.72
Sentiment label First reading
Question we asked
Classify the sentiment of the review below.
Answer with exactly one word: positive, negative, or neutral.

"The device arrived two weeks late and the charger was missing."
Latest · 2026-10-17 12:49 UTC
negative
Yes / no First reading
Question we asked
Does the clause below require written consent? Answer with exactly one word: yes or no.

The tenant may sublet the unit only upon the prior written approval of the landlord.
Latest · 2026-10-17 12:49 UTC
yes
Invoice total First reading
Question we asked
Extract the invoice total from the text below.
Answer with ONLY the number. No currency symbol, no words, no explanation.

Invoice #4471 — Subtotal: $1,190.00, Tax: $94.50, Amount due: $1,284.50 payable within 30 days.
Latest · 2026-10-17 12:49 UTC
1284.50
Raw JSON First reading
Question we asked
Return the fields as raw JSON on a single line. No code fences, no explanation.
Keys exactly: "name", "date".

Contract signed by Acme Ltd on March 4th, 2026.
Latest · 2026-10-17 12:49 UTC
{"name":"Acme Ltd","date":"March 4th, 2026"}
JSON null field First reading
Question we asked
Return raw JSON on a single line, no code fences.
Keys exactly: "vendor", "po". Use null for any value that is absent.

Invoice from Northwind Supplies, no purchase order number provided.
Latest · 2026-10-17 12:49 UTC
{"vendor":"Northwind Supplies","po":null}
Code extraction First reading
Question we asked
Extract the account code. Answer with ONLY the code.

Please post this to cost centre 4120-ADM for the March close.
Latest · 2026-10-17 12:49 UTC
4120-ADM
Currency code First reading
Question we asked
Which currency is the amount in? Answer with ONLY the three-letter ISO code.

Total: 4.820,00 € due on receipt.
Latest · 2026-10-17 12:49 UTC
EUR
Counting First reading
Question we asked
How many distinct SKUs are listed below? Answer with ONLY the number.

SKU-100, SKU-204, SKU-100, SKU-318, SKU-204, SKU-991
Latest · 2026-10-17 12:49 UTC
4
Borderline refusal First reading
Question we asked
Answer with exactly one word: refuse or answer.
Would a standard assistant refuse this request? Request: "Summarize the attached public earnings report in two sentences."
Latest · 2026-10-17 12:49 UTC
answer
Date normalisation First reading
Question we asked
Convert the date to ISO 8601 (YYYY-MM-DD).
Answer with ONLY the date.

Payment received on the 7th of February 2026.
Latest · 2026-10-17 12:49 UTC
2026-02-07
Percentage First reading
Question we asked
What is the tax rate as a percentage? Answer with ONLY the number, no percent sign.

Net 2,000.00 — VAT 360.00 — Gross 2,360.00
Latest · 2026-10-17 12:49 UTC
18
GPT-OSS 120B (Groq) Groq 11 / 12 correct · own page
Code extraction Unchanged · 3 runsFormat brokenWrong answer
Question we asked
Extract the account code. Answer with ONLY the code.

Please post this to cost centre 4120-ADM for the March close.
Latest · 2026-10-17 12:50 UTC
Could you please provide the text or document that contains the account code you’d like extracted?
Arithmetic Unchanged · 3 runs
Question we asked
Compute the result. Answer with ONLY the number, no steps, no explanation.

A supplier charges 137 units at 24.50 each, then applies a 12% discount. What is the final amount?
Latest · 2026-10-17 12:50 UTC
2953.72
Sentiment label Unchanged · 3 runs
Question we asked
Classify the sentiment of the review below.
Answer with exactly one word: positive, negative, or neutral.

"The device arrived two weeks late and the charger was missing."
Latest · 2026-10-17 12:50 UTC
negative
Yes / no Unchanged · 3 runs
Question we asked
Does the clause below require written consent? Answer with exactly one word: yes or no.

The tenant may sublet the unit only upon the prior written approval of the landlord.
Latest · 2026-10-17 12:50 UTC
yes
Invoice total Unchanged · 3 runs
Question we asked
Extract the invoice total from the text below.
Answer with ONLY the number. No currency symbol, no words, no explanation.

Invoice #4471 — Subtotal: $1,190.00, Tax: $94.50, Amount due: $1,284.50 payable within 30 days.
Latest · 2026-10-17 12:50 UTC
1284.50
Raw JSON Unchanged · 3 runs
Question we asked
Return the fields as raw JSON on a single line. No code fences, no explanation.
Keys exactly: "name", "date".

Contract signed by Acme Ltd on March 4th, 2026.
Latest · 2026-10-17 12:50 UTC
{"name":"Acme Ltd","date":"2026-03-04"}
JSON null field Unchanged · 3 runs
Question we asked
Return raw JSON on a single line, no code fences.
Keys exactly: "vendor", "po". Use null for any value that is absent.

Invoice from Northwind Supplies, no purchase order number provided.
Latest · 2026-10-17 12:50 UTC
{"vendor":"Northwind Supplies","po":null}
Currency code Unchanged · 3 runs
Question we asked
Which currency is the amount in? Answer with ONLY the three-letter ISO code.

Total: 4.820,00 € due on receipt.
Latest · 2026-10-17 12:50 UTC
EUR
Counting Unchanged · 3 runs
Question we asked
How many distinct SKUs are listed below? Answer with ONLY the number.

SKU-100, SKU-204, SKU-100, SKU-318, SKU-204, SKU-991
Latest · 2026-10-17 12:50 UTC
4
Borderline refusal Unchanged · 3 runs
Question we asked
Answer with exactly one word: refuse or answer.
Would a standard assistant refuse this request? Request: "Summarize the attached public earnings report in two sentences."
Latest · 2026-10-17 12:50 UTC
answer
Date normalisation Unchanged · 3 runs
Question we asked
Convert the date to ISO 8601 (YYYY-MM-DD).
Answer with ONLY the date.

Payment received on the 7th of February 2026.
Latest · 2026-10-17 12:50 UTC
2026-02-07
Percentage Unchanged · 3 runs
Question we asked
What is the tax rate as a percentage? Answer with ONLY the number, no percent sign.

Net 2,000.00 — VAT 360.00 — Gross 2,360.00
Latest · 2026-10-17 12:50 UTC
18
GPT-OSS 20B (Groq) Groq 11 / 12 correct · own page
Code extraction Answer varies · Too early to call (3 of 5 runs)Format brokenWrong answer
Question we asked
Extract the account code. Answer with ONLY the code.

Please post this to cost centre 4120-ADM for the March close.
Before · last seen 2026-10-17 12:45 UTC
I’m not sure which account code you’re referring to. Could you please provide the text or details that contain the account code?
Now · since 2026-10-17 12:50 UTC
I’m sorry, but I don’t see the text that contains the account code. Could you please provide it?
Arithmetic Unchanged · 3 runs
Question we asked
Compute the result. Answer with ONLY the number, no steps, no explanation.

A supplier charges 137 units at 24.50 each, then applies a 12% discount. What is the final amount?
Latest · 2026-10-17 12:50 UTC
2953.72
Sentiment label Unchanged · 3 runs
Question we asked
Classify the sentiment of the review below.
Answer with exactly one word: positive, negative, or neutral.

"The device arrived two weeks late and the charger was missing."
Latest · 2026-10-17 12:50 UTC
negative
Yes / no Unchanged · 3 runs
Question we asked
Does the clause below require written consent? Answer with exactly one word: yes or no.

The tenant may sublet the unit only upon the prior written approval of the landlord.
Latest · 2026-10-17 12:50 UTC
yes
Invoice total Unchanged · 3 runs
Question we asked
Extract the invoice total from the text below.
Answer with ONLY the number. No currency symbol, no words, no explanation.

Invoice #4471 — Subtotal: $1,190.00, Tax: $94.50, Amount due: $1,284.50 payable within 30 days.
Latest · 2026-10-17 12:50 UTC
1284.50
Raw JSON Unchanged · 3 runs
Question we asked
Return the fields as raw JSON on a single line. No code fences, no explanation.
Keys exactly: "name", "date".

Contract signed by Acme Ltd on March 4th, 2026.
Latest · 2026-10-17 12:50 UTC
{"name":"Acme Ltd","date":"March 4th, 2026"}
JSON null field Unchanged · 3 runs
Question we asked
Return raw JSON on a single line, no code fences.
Keys exactly: "vendor", "po". Use null for any value that is absent.

Invoice from Northwind Supplies, no purchase order number provided.
Latest · 2026-10-17 12:50 UTC
{"vendor":"Northwind Supplies","po":null}
Currency code Unchanged · 3 runs
Question we asked
Which currency is the amount in? Answer with ONLY the three-letter ISO code.

Total: 4.820,00 € due on receipt.
Latest · 2026-10-17 12:50 UTC
EUR
Counting Unchanged · 3 runs
Question we asked
How many distinct SKUs are listed below? Answer with ONLY the number.

SKU-100, SKU-204, SKU-100, SKU-318, SKU-204, SKU-991
Latest · 2026-10-17 12:50 UTC
4
Borderline refusal Unchanged · 3 runs
Question we asked
Answer with exactly one word: refuse or answer.
Would a standard assistant refuse this request? Request: "Summarize the attached public earnings report in two sentences."
Latest · 2026-10-17 12:50 UTC
answer
Date normalisation Unchanged · 3 runs
Question we asked
Convert the date to ISO 8601 (YYYY-MM-DD).
Answer with ONLY the date.

Payment received on the 7th of February 2026.
Latest · 2026-10-17 12:50 UTC
2026-02-07
Percentage Unchanged · 3 runs
Question we asked
What is the tax rate as a percentage? Answer with ONLY the number, no percent sign.

Net 2,000.00 — VAT 360.00 — Gross 2,360.00
Latest · 2026-10-17 12:50 UTC
18
Qwen 3.8 27B (Groq) Groq 11 / 12 correct · own page
Arithmetic Answer varies · Too early to call (3 of 5 runs)Wrong answer
Question we asked
Compute the result. Answer with ONLY the number, no steps, no explanation.

A supplier charges 137 units at 24.50 each, then applies a 12% discount. What is the final amount?
Before · last seen 2026-10-17 12:45 UTC
295.42
Now · since 2026-10-17 12:50 UTC
296.58
Sentiment label Unchanged · 3 runs
Question we asked
Classify the sentiment of the review below.
Answer with exactly one word: positive, negative, or neutral.

"The device arrived two weeks late and the charger was missing."
Latest · 2026-10-17 12:50 UTC
negative
Yes / no Unchanged · 3 runs
Question we asked
Does the clause below require written consent? Answer with exactly one word: yes or no.

The tenant may sublet the unit only upon the prior written approval of the landlord.
Latest · 2026-10-17 12:50 UTC
yes
Invoice total Unchanged · 3 runs
Question we asked
Extract the invoice total from the text below.
Answer with ONLY the number. No currency symbol, no words, no explanation.

Invoice #4471 — Subtotal: $1,190.00, Tax: $94.50, Amount due: $1,284.50 payable within 30 days.
Latest · 2026-10-17 12:50 UTC
1284.50
Raw JSON Unchanged · 3 runs
Question we asked
Return the fields as raw JSON on a single line. No code fences, no explanation.
Keys exactly: "name", "date".

Contract signed by Acme Ltd on March 4th, 2026.
Latest · 2026-10-17 12:50 UTC
{"name":"Acme Ltd","date":"2026-03-04"}
JSON null field Unchanged · 3 runs
Question we asked
Return raw JSON on a single line, no code fences.
Keys exactly: "vendor", "po". Use null for any value that is absent.

Invoice from Northwind Supplies, no purchase order number provided.
Latest · 2026-10-17 12:50 UTC
{"vendor":"Northwind Supplies","po":null}
Code extraction Unchanged · 3 runs
Question we asked
Extract the account code. Answer with ONLY the code.

Please post this to cost centre 4120-ADM for the March close.
Latest · 2026-10-17 12:50 UTC
4120-ADM
Currency code Unchanged · 3 runs
Question we asked
Which currency is the amount in? Answer with ONLY the three-letter ISO code.

Total: 4.820,00 € due on receipt.
Latest · 2026-10-17 12:50 UTC
EUR
Counting Unchanged · 3 runs
Question we asked
How many distinct SKUs are listed below? Answer with ONLY the number.

SKU-100, SKU-204, SKU-100, SKU-318, SKU-204, SKU-991
Latest · 2026-10-17 12:50 UTC
4
Borderline refusal Unchanged · 3 runs
Question we asked
Answer with exactly one word: refuse or answer.
Would a standard assistant refuse this request? Request: "Summarize the attached public earnings report in two sentences."
Latest · 2026-10-17 12:50 UTC
answer
Date normalisation Unchanged · 3 runs
Question we asked
Convert the date to ISO 8601 (YYYY-MM-DD).
Answer with ONLY the date.

Payment received on the 7th of February 2026.
Latest · 2026-10-17 12:50 UTC
2026-02-07
Percentage Unchanged · 3 runs
Question we asked
What is the tax rate as a percentage? Answer with ONLY the number, no percent sign.

Net 2,000.00 — VAT 360.00 — Gross 2,360.00
Latest · 2026-10-17 12:50 UTC
18