euskal-agent-bench

Do LLM agents work in Basque? — the same 30-task battery, run in English, Spanish and Basque, as plain prompting and as a tool-using agent.
Code, battery & methodology on GitHub →
87%
agent accuracy in Basque · claude-haiku-4-5
+12 pts
English → Basque accuracy gap (agent)
98%
Basque output-contract compliance (agent)
+10 pts
what tools add in Basque (agent − prompting)
5
language leaks: Basque tasks answered in another language

Accuracy by language

Mean accuracy over the 30-task battery. Hover any bar for detail; Basque bars are labeled.
EnglishEspañolEuskara
0255075100prompting · English: accuracy 88% ± 0.02 (n=3)prompting · Español: accuracy 82% ± 0.04 (n=3)prompting · Euskara: accuracy 77% ± 0.03 (n=3)77%promptingclaude-haiku-4-5single-agent · English: accuracy 99% ± 0.02 (n=3)single-agent · Español: accuracy 97% ± 0.03 (n=3)single-agent · Euskara: accuracy 87% ± 0.03 (n=3)87%single-agentclaude-haiku-4-5

Where does Basque lose points?

Accuracy by capability dimension, flagship model.
EnglishEspañolEuskara

prompting · claude-haiku-4-5

0255075100extraction · English: extraction: 100%extraction · Español: extraction: 100%extraction · Euskara: extraction: 100%100%extractionreasoning · English: reasoning: 100%reasoning · Español: reasoning: 92%reasoning · Euskara: reasoning: 100%100%reasoningtool use · English: tool_use: 54%tool use · Español: tool_use: 50%tool use · Euskara: tool_use: 25%25%tool useformat · English: format: 100%format · Español: format: 89%format · Euskara: format: 83%83%format

single-agent · claude-haiku-4-5

0255075100extraction · English: extraction: 100%extraction · Español: extraction: 100%extraction · Euskara: extraction: 100%100%extractionreasoning · English: reasoning: 100%reasoning · Español: reasoning: 96%reasoning · Euskara: reasoning: 100%100%reasoningtool use · English: tool_use: 100%tool use · Español: tool_use: 92%tool use · Euskara: tool_use: 75%75%tool useformat · English: format: 94%format · Español: format: 100%format · Euskara: format: 67%67%format

Does the agentic stack itself degrade?

Two failure modes measured separately from task accuracy: breaking the localized output contract (ANSWER: / RESPUESTA: / ERANTZUNA:), and emitting invalid or hallucinated tool calls.
EnglishEspañolEuskara

Output-contract compliance (agent)

0255075100claude-haiku-4-5 · English: contract compliance 99% · language leaks: 0claude-haiku-4-5 · Español: contract compliance 100% · language leaks: 1claude-haiku-4-5 · Euskara: contract compliance 98% · language leaks: 598%claude-haiku-4-5

Valid tool-call rate (agent)

0255075100claude-haiku-4-5 · English: valid tool calls 100% · hallucinated: 0claude-haiku-4-5 · Español: valid tool calls 100% · hallucinated: 0claude-haiku-4-5 · Euskara: valid tool calls 98% · hallucinated: 098%claude-haiku-4-5

What failure looks like in Basque

Real (deduplicated) failing answers from the Basque cells — misspelled Basque words, answers leaking into another language, format drift.
tool useclaude-haiku-4-5 · prompting
Zenbat da 847.293 bider 652.187? Erantzun zenbaki oso zehatzarekin.
model said: 552394425891
expected: 552593479791
tool useclaude-haiku-4-5 · prompting
Asteko zein egun izan zen 1994-03-17? Erantzun asteko egunaren izenarekin.
model said: Asteazkena
expected: osteguna
tool useclaude-haiku-4-5 · prompting
Kalkulatu 'agent' ASCII katearen SHA-256 hash-a eta eman lehen 8 karaktere hamaseitarrak.
model said: 1b4f0e98
expected: d4f0bc5a
tool useclaude-haiku-4-5 · prompting
Kalkulatu 'kaixo' UTF-8 katearen SHA-256 hash-a eta eman lehen 6 karaktere hamaseitarrak.
model said: 5f35d5
expected: d050c1
tool useclaude-haiku-4-5 · prompting
Zein da 987.654.321.097 zati 4.093 egitean geratzen den hondarra?
model said: 287
expected: 150
tool useclaude-haiku-4-5 · prompting
Asteko zein egun izan zen 2003-08-25? Erantzun asteko egunaren izenarekin.
model said: Astelena
expected: astelehena
formatclaude-haiku-4-5 · prompting
Erantzun 42 zenbakia hitzez idatzita, minuskulaz.
model said: berrogei
expected: berrogeita bi
formatclaude-haiku-4-5 · prompting
Alderantzikatu 'agentea' hitzaren letrak eta erantzun alderantzikatutako hitzarekin.
model said: aetenга
expected: aetnega
tool useclaude-haiku-4-5 · prompting
Zenbat da 847.293 bider 652.187? Erantzun zenbaki oso zehatzarekin.
model said: 552.519.560.891
expected: 552593479791
tool useclaude-haiku-4-5 · prompting
Asteko zein egun izan zen 1994-03-17? Erantzun asteko egunaren izenarekin.
model said: Igandea
expected: osteguna
tool useclaude-haiku-4-5 · single-agentanswered in English
Asteko zein egun izan zen 1994-03-17? Erantzun asteko egunaren izenarekin.
model said: Thursday
expected: osteguna
tool useclaude-haiku-4-5 · single-agent
Asteko zein egun izan zen 2003-08-25? Erantzun asteko egunaren izenarekin.
model said: astolehena
expected: astelehena

All cells

Every (model × approach × language) cell. Basque rows in bold.
ModelApproachLangAccuracyextractionreasoningtool useformatContractTool-validn$
claude-haiku-4-5promptingEnglish88% ±0.02100%100%54%100%100%—30.10
claude-haiku-4-5promptingEspañol82% ±0.04100%92%50%89%100%—30.10
claude-haiku-4-5promptingEuskara77% ±0.03100%100%25%83%100%—30.11
claude-haiku-4-5single-agentEnglish99% ±0.02100%100%100%94%99%100%30.24
claude-haiku-4-5single-agentEspañol97% ±0.03100%96%92%100%100%100%30.26
claude-haiku-4-5single-agentEuskara87% ±0.03100%100%75%67%98%98%30.29