← All lab notesREPORT · 2026-11-20

Does the language of a spatial instruction interface affect an LLM agent's ability to act?

We tested whether the language of an LLM agent's operating interface, not just its goal, changes how well it completes a multi-step spatial task. Translating the ongoing interface hurt performance far more than translating the one-time goal.

↳ 53.7% vs 31.0% success — English vs Russian full interface, matched n=300

LLM agents are often tested as though the language around a task were a transparent wrapper. That assumption becomes fragile once a model has to act over many steps: in a tool-use environment, the model keeps reading the interface while it acts, so surface text can become part of the policy rather than a neutral presentation.

We test this by rendering the same spatial game through different language surfaces while keeping the hidden world and simulator fixed. Correctness is measured from the final state, not from the model’s own explanations. In the current runs, changing the operating interface changes task success and also changes the shape of failed trajectories — including remaining distance to the target and tokens spent per successful episode. The result is not a claim that one language is inherently better. It is evidence that interface language should be treated as a controlled variable in benchmarks for spatial agents.

Research questions

RQ1. Given the same spatial task, how does changing the instruction language change an LLM’s spatial performance, error type, error severity, and token/cost efficiency?

RQ2. Is the observed effect better explained by the language of the target goal, or by the language of the repeated operating interface — tool descriptions, rules, observations, and feedback?

Hypothesis. In agentic spatial tasks, the language used for the step-by-step interface should matter more than a one-time goal translation. The agent must keep parsing action rules, observations, and simulator feedback, so small changes in the operating surface can accumulate over a plan.

The benchmark

KeyBoxWorld is a small deterministic spatial tool-use game. The model moves through rooms, picks up items, opens locked boxes when it has the right key, and places a target item into a target container. Each model response must be one JSON tool call; the simulator applies valid calls and rejects invalid ones, and the next observation is returned in the language assigned to that condition.

World stateRooms form a small graph. Objects are located in rooms or containers. The agent has a current room and an inventory.
GoalThe target predicate is simple: a named item must end inside a named box. The item IDs and box IDs stay symbolic across languages.
Action loopAt each step the model sees the goal, tool docs, recent history, and current observation, then emits one JSON call such as {"tool":"T2","args":{"item":"ITEM_41"}}.
ScoreThe scorer uses simulator state, not prose. Success means the target predicate is true; severity is measured by remaining distance to the target.

The agent loop

  1. Render the prompt. The same hidden state is shown through the selected goal and interface languages: goal + tools + history + observation.
  2. Model chooses one action. The model must answer with a single JSON tool call, e.g. {"tool":"T1","args":{"to_room":"ROOM_B"}}.
  3. Simulator checks it. Valid calls change the world; invalid calls become feedback in the interface language.
  4. Score final state. The episode succeeds only when the target item ends inside the target box.

Experimental protocol

The benchmark is built to avoid asking the model to judge itself. The model only acts; the simulator and scorer decide what happened. This matters because language effects can appear as valid-but-wrong action choices, repeated loops, precondition failures, or extra tokens spent on unsuccessful trajectories.

ComponentChoiceReason
Task generatorDeterministic KeyBoxWorld worldsThe same hidden task can be rendered through multiple languages
Action formatOne JSON tool call per stepValidity and precondition failures are machine-checkable
Language manipulationGoal language and interface language varied separatelySeparates translation of the target from control-loop language
ScoringSimulator final state, remaining distance, failure type, token usageModel explanations do not determine correctness
Model accessOpenRouter chat-completion calls, temperature 0Black-box prompts, outputs, usage, and latency are enough for v1 evidence

Example episode

A representative generated task looks like this. The object identifiers stay the same across all languages, so translations do not rename the hidden state.

Task IDkeybox_0001 · medium difficulty
GoalITEM_49 inside BOX_49
Initial roomROOM_A
RoomsROOM_A, ROOM_B, ROOM_C, ROOM_D
Observed runRussian goal, English interface solved in 10 model steps.

Language surfaces

The multilingual version keeps the same game and changes language through condition names. For any non-English language L, L_all means the goal, rules, tools, observations, and feedback are all in that language. L_goal_en_tools keeps only the goal in L while the operating interface stays English. en_goal_L_tools keeps the goal English while the operating interface changes to L.

LanguageGoal sentenceMove toolResponse instruction
EnglishPut ITEM_41 into BOX_41.T1(to_room): Move to an adjacent room.Reply with one JSON tool call.
RussianПоложи ITEM_41 в BOX_41.T1(to_room): Перейти в соседнюю комнату.Ответь одним JSON-вызовом инструмента.
SpanishPon ITEM_41 dentro de BOX_41.T1(to_room): Muévete a una habitación adyacente.Responde con una sola llamada de herramienta en JSON.
GermanLege ITEM_41 in BOX_41.T1(to_room): Gehe in einen benachbarten Raum.Antworte mit genau einem JSON-Werkzeugaufruf.
FrenchMets ITEM_41 dans BOX_41.T1(to_room): Va dans une pièce adjacente.Réponds avec un seul appel d'outil en JSON.
ArmenianITEM_41 տեղադրիր BOX_41 տարայի մեջ.T1(to_room): Տեղափոխվիր հարակից սենյակ.Պատասխանիր մեկ JSON գործիքի կանչով.

Results

Complete matched-language anchor

The complete KeyBoxWorld run has 300 hidden tasks under four language conditions, for 1,200 completed scored episodes. This is the cleanest within-language comparison because every condition covers the full task set.

The mixed conditions are the key diagnostic. A Russian goal with English tools remains close to the English control. An English goal with Russian tools falls near the full Russian interface. This pattern points away from a simple translation-only explanation and toward the operating interface as a major source of the effect.

In aggregate, the English full-interface condition solved 53.7% of episodes. The full Russian interface solved 31.0%. When only the goal was Russian and the operating interface stayed English, success was 50.7%; when the goal was English but the operating interface was Russian, success was 32.7%. The asymmetry is the central result of the anchor run.

ConditionEpisodesSuccess rateAvg. final distanceTokens per success
English full interface30053.7%2.5108,707.7
Russian goal, English interface30050.7%2.6239,126.5
English goal, Russian interface30032.7%3.32718,011.3
Russian full interface30031.0%3.42819,282.3
Matched comparisonMatched tasksLeft onlyRight onlyBoth solvedBoth failedExact p
English full vs. Russian full300922469115<0.1%
Russian full vs. Russian goal + English interface300228171126<0.1%
English full vs. English goal + Russian interface300872474115<0.1%

Anchor diagnostics

The English/Russian run is useful because it includes difficulty and failure-type views across the full task set. Difficulty shows whether the gap only appears in harder plans. Failure types show whether the model is failing the interface contract, getting stuck in loops, or ending in the wrong world state. Across all four conditions, easy tasks (45 episodes each) solved at 100%; the gap opens up at medium difficulty (210 episodes each: 53.3% English full vs. 20–21.4% for the two Russian-interface conditions vs. 49.0% for Russian goal + English interface) and narrows again at hard difficulty, where every condition scores in the 8.9–17.8% range.

By failure type, the English full-interface condition’s non-successes are mostly precondition_failed (29.7%) and loop (12.7%). Both Russian-interface conditions shift heavily toward loop (25.7% for English goal + Russian interface, 28.0% for Russian full) alongside a similar precondition_failed share, meaning the Russian-interface agent is more often getting stuck repeating actions rather than just picking the wrong valid action.

Multilingual extension

The multilingual extension applies the same condition design beyond the original two-language anchor, across English, Russian, Spanish, German, French, and Armenian, with a harder task mix. It is designed as the current main multilingual measurement path. The present scored artifact is still partial because OpenRouter returned a key-total-limit 403 before the full 3,840-episode matrix completed — clean scored episodes: 1,146. Because task coverage can differ by run, matched-pair counts are used where possible.

In the current artifact, Armenian is included as a full Armenian interface, an Armenian-goal/English-interface condition, and an English-goal/Armenian-interface condition — the same three-surface design used for the other four languages.

LanguageCondition surfaces presentClean episodesIncluded surfaces
English172Full English interface
Russian3214Russian full interface; Russian goal + English interface; English goal + Russian interface
Spanish3216Spanish full interface; Spanish goal + English interface; English goal + Spanish interface
German3216German full interface; German goal + English interface; English goal + German interface
French3216French full interface; French goal + English interface; English goal + French interface
Armenian3212Armenian full interface; Armenian goal + English interface; English goal + Armenian interface

Goal language versus interface language

The point of the multilingual matrix is not to rank languages, but to test whether translated goals and translated operating interfaces move performance differently on the same hidden tasks, across two model families: qwen-27b and gpt-oss-20b.

ModelLanguageFull interfaceForeign goal + English interfaceEnglish goal + foreign interfaceDrop from English: goal onlyDrop from English: interface
gpt-oss-20bRussian5.7%2.9%0.0%-0.1%2.8%
gpt-oss-20bSpanish0.0%2.8%5.6%0.0%-2.8%
gpt-oss-20bGerman2.8%0.0%0.0%2.8%2.8%
gpt-oss-20bFrench2.8%5.6%0.0%-2.8%2.8%
gpt-oss-20bArmenian8.6%2.9%5.6%-0.1%-2.8%
qwen-27bRussian11.1%30.6%27.8%8.3%11.1%
qwen-27bSpanish22.2%30.6%13.9%8.3%25.0%
qwen-27bGerman16.7%33.3%13.9%5.6%25.0%
qwen-27bFrench30.6%36.1%27.8%2.8%11.1%
qwen-27bArmenian17.1%14.3%11.1%24.6%27.8%

Both “drop from English” columns are approximate success-rate loss relative to each model’s English full-interface control. For qwen-27b, the interface-language drop is consistently larger than the goal-language drop across all five languages (11.1–27.8% vs. 2.8–24.6%); for gpt-oss-20b, both model and non-English full-interface success rates are low enough (0–8.6%) that the pattern is noisier, but points the same direction for Russian, German, and French.

Multilingual diagnostics

The multilingual matrix is incomplete, so these diagnostics are not treated as final estimates. They are still useful for deciding what to run next: English-interface mixed conditions are competitive across several goal languages, while full non-English interfaces and non-English operating interfaces often show lower success and higher remaining distance. For qwen-27b, every language’s easy-difficulty tasks (3 episodes each) solved at 100% regardless of condition, and medium-difficulty success ranged from 13.3% (Armenian goal, English interface) up to 73.3% (English full interface); hard-tier tasks solved only for Spanish goal + English interface (11.1%), with every other condition at 0%. For gpt-oss-20b, medium-difficulty success topped out at 6.7% (English and French goal + English interface), with most conditions at 0% across both medium and hard tiers.

By failure type, qwen-27b’s full-interface conditions succeed at 11.1–38.9% (English highest, Spanish lowest at 22.2%), with precondition_failed as the leading non-success cause in every language (38.9–66.7%). gpt-oss-20b’s full-interface conditions succeed far less often (2.8–8.6%), and invalid_json becomes a leading failure mode for English, French, German, and Russian (36.1–47.2% of episodes) — a failure type that barely appears for qwen-27b.

Matched multilingual comparisons

Because the multilingual matrix is incomplete, matched comparisons are safer than reading the aggregate table alone: each row only counts tasks where both compared conditions completed an API episode. For qwen-27b, English full interface beats Russian full interface on matched tasks (10 wins to 0, p = 0.2%) and beats German full interface (8 to 0, p = 0.8%); the Armenian comparison is also significant (9 to 1, p = 2.1%), while Spanish and French are directionally similar but not significant at this sample size. For gpt-oss-20b, matched samples are small and mostly “both failed”, so most comparisons do not reach significance, with the exception of the German full-interface vs. German-goal comparison (0 to 6 the other way, p = 3.1%).

ModelComparisonMatched tasksLeft onlyRight onlyBoth solvedBoth failedExact p
gpt-oss-20bEnglish full vs. Russian full3512032100.0%
gpt-oss-20bEnglish full vs. Spanish full3610035100.0%
gpt-oss-20bEnglish full vs. German full3611034100.0%
gpt-oss-20bEnglish full vs. French full3611034100.0%
gpt-oss-20bEnglish full vs. Armenian full351303162.5%
gpt-oss-20bSpanish full vs. Spanish goal + English interface3601035100.0%
gpt-oss-20bGerman full vs. German goal + English interface3610035100.0%
gpt-oss-20bFrench full vs. French goal + English interface3612033100.0%
gpt-oss-20bArmenian full vs. Armenian goal + English interface352013250.0%
qwen-27bEnglish full vs. Russian full361004220.2%
qwen-27bEnglish full vs. Spanish full36717217.0%
qwen-27bEnglish full vs. German full36806220.8%
qwen-27bEnglish full vs. French full3641102137.5%
qwen-27bEnglish full vs. Armenian full35915202.1%
qwen-27bSpanish full vs. Spanish goal + English interface362562345.3%
qwen-27bGerman full vs. German goal + English interface36066243.1%
qwen-27bFrench full vs. French goal + English interface362492168.8%
qwen-27bArmenian full vs. Armenian goal + English interface3521428100.0%

Discussion

The results are most naturally read as a control-surface effect. A one-time foreign-language goal can often be handled when the operating interface remains English. Changing the ongoing interface is harder: the model must repeatedly map the current observation to the correct precondition-sensitive action, and failures compound over several steps.

Token efficiency should be read with care. Tokens-per-success is not only a tokenizer property — it combines prompt length, completion length, number of turns, and success rate. A condition can be expensive because the language uses more tokens, because the agent takes more steps, or because fewer episodes succeed.

Earlier static, multiple-choice-style tasks remain part of the journey, but not the main evidence here. They helped harden scoring and counterbalancing, but did not stress the repeated observation-action loop. KeyBoxWorld is closer to the agentic setting we care about.

Limitations

This benchmark exposes language-surface effects in multi-step spatial tool use, but it should not be read as a universal ordering of languages or as a general comparison of model quality. The current multilingual extension is a partial matrix and must be repeated with fuller coverage and more models before the effect size can be treated as stable.

Incomplete matrix

The multilingual artifact has clean scored episodes, but not the planned full coverage — the run was interrupted by an API-level key-total-limit error before the full 3,840-episode matrix completed. Matched-pair rows reduce this problem but do not remove it.

Model coverage

This should not be read as a property of all LLMs. The extension adds a second model family (gpt-oss-20b alongside qwen-27b), but the matrix is still partial and not a leaderboard.

Hand-written surfaces

Hand-written templates make conditions controlled, but phrasing quality can still vary across languages. The runner now supports prompt variants; the next scored matrix should execute those variants rather than relying on one phrasing per language.

Symbolic IDs

Object IDs are intentionally language-neutral. That isolates the interface language, but real tasks often localize object names too.

Conclusion and future work

The main conclusion is that interface language is not just a localization detail for spatial agents. In this benchmark, changing the language of the operating surface changes how the model acts in the same hidden world. The result motivates treating interface language as part of the experimental design whenever LLM agents are evaluated through tools, observations, or simulator feedback.

More from the lab