Agent-Computer Interface (ACI)
How AI agents interact with computers — from SWE-Agent to Claude Code shell. Covers browsing, editing, executing, and the LM-centric design paradigm.
Agent-Computer Interface (ACI)
Every AI agent that operates in the real world needs a way to interact with a computer. It needs to browse the web, read files, edit code, run commands, and inspect outputs. The Agent-Computer Interface (ACI) is the layer that defines how an agent performs those actions. It is the bridge between a language model's text output and the actual operations on a machine.
The concept crystallized with the SWE-Agent paper from Princeton NLP in 2024 (arxiv.org/abs/2405.15793), which made a foundational observation: computer interfaces should be designed for the language model, not for humans. That insight reframes everything from command design to error formatting, and it separates functional agents from those that merely hallucinate bash commands.
What is an Agent-Computer Interface?
An ACI is the set of tools, commands, and feedback mechanisms that allow an AI agent to perceive and modify a computing environment. Concretely, an ACI answers three questions:
- What can the agent do? -- The action space: which commands, API calls, and file operations are available.
- What does the agent see? -- The observation space: how results, errors, and environment state are formatted and returned.
- How does the agent learn from mistakes? -- The feedback loop: how errors are surfaced, how state is reset, and how the agent knows to try a different approach.
Without an ACI, a language model generates arbitrary shell commands as text -- and the quality of execution depends entirely on whether those commands happen to be valid. With an ACI, the agent operates through a controlled, structured interface that constrains its actions to things that actually work and returns observations in a format the model can reason about effectively.
┌──────────────────────────────────────────────────────┐
│ Language Model │
│ (reasons, plans, decides) │
└──────────────────┬───────────────────────────────────┘
│ generates action
▼
┌──────────────────────────────────────────────────────┐
│ Agent-Computer Interface │
│ │
│ Action Space Observation Space │
│ ┌─────────────┐ ┌──────────────┐ │
│ │ search_dir │ │ File listing │ │
│ │ find_file │ │ Match results│ │
│ │ edit_file │ │ Diff output │ │
│ │ run_tests │ │ Test results │ │
│ │ submit │ │ Submit status│ │
│ └─────────────┘ └──────────────┘ │
└──────────────────┬───────────────────────────────────┘
│ executes on
▼
┌──────────────────────────────────────────────────────┐
│ Computer System │
│ (filesystem, shell, network, browser) │
└──────────────────────────────────────────────────────┘
The LM-Centric Design Paradigm
The SWE-Agent paper introduced the term LM-centric design to describe the principle of designing interfaces specifically for language models rather than adapting human interfaces for AI use. Per the paper: "We find that specialized interfaces for LM agents significantly improve performance" -- a finding that held across multiple benchmarks and model families.
What does LM-centric actually mean in practice? Consider the difference between how a human and a language model interact with a filesystem:
Human-centric interface (standard bash):
# Human finds a Python test file with "auth" in the name
find . -type f -name "*.py" | xargs grep -l "def test_auth" 2>/dev/null | head -20
LM-centric interface (SWE-Agent ACI):
search_dir "def test_auth"
The LM-centric version is shorter, has a single clear intent, and returns structured results rather than raw text. The language model does not need find, xargs, grep, 2>/dev/null, or head -- those are Unix plumbing that a human learned over years. The model needs a direct command that maps to a single conceptual action.
The Princeton team documented several concrete design decisions that followed from this principle:
- Simplified commands -- Each command does one thing. No chaining, no pipes, no shell metacharacters for the agent to get wrong.
- Structured output -- Results come back in a consistent format the model can parse, not in raw terminal output that varies by system.
- Minimal keystrokes -- Short command names reduce token consumption and lower the chance of a typo the model cannot recover from.
- Error recovery built in -- When a command fails, the interface returns a structured error with a suggested next step, not a stack trace.
Three Major ACI Implementations
SWE-Agent: The Specialized Command Set
SWE-Agent (github.com/princeton-nlp/SWE-Agent) pioneered the LM-centric ACI with a purpose-built command set for software engineering tasks. The agent operates inside a Docker container and interacts with the codebase through a fixed set of commands:
┌─────────────────────────────────────────────────────┐
│ SWE-Agent ACI Commands │
├──────────────────────┬──────────────────────────────┤
│ search_dir <regex> │ Search file contents in dir │
│ find_file <pattern> │ Locate files by name/path │
│ edit <path> │ Apply a diff to a file │
│ ls <path> │ List directory contents │
│ view <path> │ Display file contents │
│ search <regex> │ Search across all files │
│ goto <line> │ Jump to a specific line │
│ submit │ Submit the solution patch │
│ attempt_patch │ Try an edit, get diff back │
│ run_tests │ Execute the test suite │
└──────────────────────┴──────────────────────────────┘
A typical agent interaction looks like this:
AGENT: search_dir "class UserAuth"
SYSTEM:
Found 3 matches in 2 files:
src/auth/handler.py:14 class UserAuth:
src/auth/handler.py:45 class UserAuthError(Exception):
src/models/user.py:8 class UserAuth:
AGENT: view src/auth/handler.py 1:30
SYSTEM:
[1] from datetime import datetime
[2] from typing import Optional
...
[14] class UserAuth:
[15] def __init__(self, token: str):
[16] self.token = token
AGENT: edit src/auth/handler.py
OLD: def __init__(self, token: str):
self.token = token
NEW: def __init__(self, token: str, expiry: int = 3600):
self.token = token
self.expiry = expiry
SYSTEM:
File edited successfully. 2 lines changed.
The key property of this interface: every command is atomic and unambiguous. The model never needs to compose shell pipelines. It never sees a permission error from chmod or a path issue from relative vs. absolute paths. The interface handles the plumbing; the model handles the reasoning.
Per the SWE-Agent paper, this design yielded a 12.5% resolve rate on SWE-bench (full repository-level issues) compared to near-zero for agents using raw bash. The interface was not just convenient -- it was the difference between a functional agent and one that could not operate at all.
Codex CLI: The Containerized Sandbox
OpenAI's Codex CLI (github.com/openai/codex) takes a different approach. Instead of a specialized command set, it gives the agent a full bash shell inside a sandboxed Docker container. The agent can run arbitrary shell commands, but the container restricts what those commands can actually affect.
The sandbox boundary is the key design element:
┌─────────────────────────────────────────────────────┐
│ Host System │
│ │
│ ┌───────────────────────────────────────────────┐ │
│ │ Docker Container (Sandbox) │ │
│ │ │ │
│ │ ┌─────────────────────────────────────────┐ │ │
│ │ │ Codex Agent │ │ │
│ │ │ │ │ │
│ │ │ Can run: bash, python, git, npm... │ │ │
│ │ │ Cannot: access host network, │ │ │
│ │ │ persistent storage, │ │ │
│ │ │ unrestricted filesystem │ │ │
│ │ └─────────────────────────────────────────┘ │ │
│ │ │ │
│ └───────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────┘
A Codex agent interaction:
# The agent runs standard shell commands inside the sandbox
$ ls src/
main.py utils.py tests/
$ grep -n "TODO" src/main.py
12: # TODO: implement rate limiting
$ python -m pytest tests/ -v
==================== test session starts ====================
collected 3 items
tests/test_main.py::test_handler PASSED
tests/test_main.py::test_rate_limit SKIPPED
tests/test_main.py::test_auth FAILED
==================== 1 failed, 1 passed, 1 skipped ====================
$ cat tests/test_auth.py | head -40
Codex's approach trades interface simplicity for generality. The agent can use any standard tool -- curl, jq, sed, awk -- because it is running a real shell. The trade-off is that the agent must know how to use these tools correctly. A malformed sed command or a missing 2>/dev/null can crash the entire turn.
The sandbox is the safety net. Codex uses Docker containers with restricted networking, no access to the host filesystem, and ephemeral storage. Per the Codex CLI documentation (github.com/openai/codex/blob/main/README.md), the sandbox is rebuilt from a clean image for each session, ensuring no state leaks between runs.
Claude Code: The Real Shell
Claude Code (code.claude.com/docs) takes the most direct approach: the agent runs in your actual development environment. It uses your real shell, your real files, your real tools. There is no sandbox and no specialized command set -- the ACI is the standard Unix environment plus structured tool definitions.
┌─────────────────────────────────────────────────────┐
│ Your Development Environment │
│ │
│ ┌───────────────────────────────────────────────┐ │
│ │ Claude Code Agent │ │
│ │ │ │
│ │ Tools: │ │
│ │ • Bash - run any shell command │ │
│ │ • Read - read file contents │ │
│ │ • Edit - targeted string replacement │ │
│ │ • Write - create or overwrite files │ │
│ │ • Glob - find files by pattern │ │
│ │ • Grep - search file contents │ │
│ │ • WebSearch - search the web │ │
│ │ • MCP - extensible via MCP servers │ │
│ │ │ │
│ │ Works with: git, docker, kubectl, npm, │ │
│ │ aws cli, any tool on your PATH │ │
│ └───────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────┘
A Claude Code interaction:
# Claude Code reads a file
→ Read /src/handler.ts
[1] export function handleRequest(req: Request): Response {
[2] const url = new URL(req.url);
[3] return new Response("Hello World");
[4] }
# Claude Code edits using targeted string replacement
→ Edit /src/handler.ts
OLD: return new Response("Hello World");
NEW: return new Response(`Hello from ${url.pathname}`);
# Claude Code runs commands in your shell
→ Bash: npm test
PASS src/handler.test.ts
✓ handles root path (2ms)
✓ handles subpath (1ms)
Second, Claude Code supports MCP (Model Context Protocol) servers, which extend the ACI with custom tools. An MCP server can add domain-specific commands -- database queries, deployment triggers, API calls -- without modifying the agent itself. This makes the ACI modular and extensible.
Design Principles Across All Implementations
Despite their different approaches, all three ACIs follow a shared set of design principles. Understanding these principles lets you evaluate new tools and build custom agent interfaces that actually work.
Principle 1: Simplify the Action Space
Every additional command is a surface for the model to misuse. SWE-Agent's interface has roughly 10 commands. Claude Code's core tool set is similarly constrained. Codex offers a full shell but relies on the model's training on bash to keep actions tractable.
The practical guideline: if your agent interface has more than 20 commands, audit whether any can be merged or removed. Complexity in the interface is complexity the model must navigate on every turn.
Principle 2: Return Structured Observations
Raw terminal output is hostile to language models. A git status that includes Unicode file paths, ANSI escape codes, and branch status in various formats is noisy. The best ACIs normalize output into a consistent, parseable format.
SWE-Agent returns file listings with line numbers. Claude Code returns file contents with cat -n formatting. Codex relies on the agent parsing raw output -- one of its known weaknesses.
# Bad: raw terminal output
$ find . -name "*.py" | wc -l && echo "---" && git branch --show-current
42
---
main
# Good: structured observation
Files found: 42 Python files
Current branch: main
Repository: /project/myapp
Principle 3: Minimize Context per Turn
Every token in the observation is a token the model must process before deciding its next action. Efficient ACIs truncate long outputs, summarize repetitive patterns, and show only what changed.
Claude Code implements this with targeted reads: Read takes an offset and limit parameter, so the agent reads only the section it needs rather than an entire 2000-line file. SWE-Agent uses line ranges in its view command. This is context engineering applied at the interface level.
Principle 4: Handle Errors Gracefully
When an agent command fails, the ACI should return a structured error with actionable information -- not a stack trace that consumes 40 lines of context. The model needs to know what went wrong and what it should try differently.
# Bad error response
Traceback (most recent call last):
File "/usr/lib/python3.11/site-packages/pytest.py", line 3264, in main
...
E ModuleNotFoundError: No module named 'requests'
# Good error response
Error: Module 'requests' is not installed.
Suggestion: Run 'pip install requests' before re-running tests.
Principle 5: Support Undo and Rollback
Agents make mistakes frequently. An ACI without rollback forces the agent to manually reverse its changes -- a task that is itself error-prone. Git integration is the standard solution: every agent action is committed, and git checkout or git reset provides a clean undo.
SWE-Agent wraps changes in a patch system. Claude Code works directly with Git and supports worktree isolation per sub-agent. Codex's ephemeral containers provide rollback by default -- destroy the container, start fresh.
Comparison of the Three Approaches
| Property | SWE-Agent ACI | Codex CLI Sandbox | Claude Code Shell |
|---|---|---|---|
| Interface type | Specialized commands | Full bash shell | Unix tools + structured tools |
| Environment | Docker container | Docker container (sandboxed) | Host machine (real environment) |
| Command count | ~10 fixed commands | Unlimited (any bash) | ~10 core + MCP extensions |
| Network access | None | Restricted | Full |
| Filesystem | Repository checkout only | Ephemeral container | Full host access |
| Undo mechanism | Patch system | Container rebuild | Git + worktrees |
| Error handling | Structured with suggestions | Raw shell output | Structured tool errors |
| Extension model | Modify ACI source | Install packages in container | MCP servers |
| Safety model | Sandboxed, no host access | Sandboxed, restricted network | Permission prompts |
| Context efficiency | High (minimal output) | Medium (raw output) | High (targeted reads) |
| Generality | Low (code tasks only) | High (any bash workflow) | High (any dev workflow) |
| Best for | Benchmark tasks, isolated bugs | Prototyping, scripts | Real development work |
The trade-off is clear: SWE-Agent optimizes for reliability (the model cannot generate invalid commands because the interface only exposes valid ones), Codex optimizes for generality (any bash workflow is possible), and Claude Code optimizes for real-world utility (it works with your actual tools and environment).
The Evolution of ACI: From Browsers to IDEs
The concept of an agent-computer interface predates the current wave of software engineering agents. Understanding the evolution helps predict where the technology is heading.
Phase 1: Browser-Based ACI (2020-2022)
The earliest agent interfaces were built for web browsing. Systems like WebGPT (2021) and ReAct (Yao et al., 2022) gave agents the ability to search the web, follow links, and extract text from web pages. The ACI was the browser DOM: the agent could click elements, fill forms, and read page content.
The limitation of browser-based ACIs: they are slow, noisy, and brittle. A webpage redesign breaks the agent's navigation. Rendering takes seconds. The observation space (HTML or pixels) is enormous relative to the useful signal.
Phase 2: IDE Integration (2022-2024)
The second phase brought agents directly into the development environment. Cursor (cursor.com), GitHub Copilot, and Aider (github.com/paul-gauthier/aider) embedded the agent in the editing workflow. The ACI became the editor API: read a file, edit a region, run tests, see the diff.
This phase introduced the key insight that agents benefit from structured, code-aware interfaces more than from raw terminal access. Cursor's agent mode, Windsurf's Cascade, and Aider's Git-first loop all demonstrated that agents perform better when the interface handles the mechanics of file editing and test execution.
Phase 3: Autonomous Software Engineers (2024-present)
The current phase is full autonomy. SWE-Agent, Claude Code, Codex CLI, and Devin (from Cognition AI) are not assistants that edit a file when you ask -- they are agents that receive a task, plan an approach, execute a multi-step solution, and verify the result. The ACI is no longer just an editing tool; it is the agent's entire operating environment.
Phase 1: Browser ACI Phase 2: IDE ACI Phase 3: Full Environment ACI
┌──────────────┐ ┌──────────────┐ ┌──────────────────────┐
│ Search │ │ Read file │ │ Read / Edit / Run │
│ Navigate │ → │ Edit file │ → │ Browse / Deploy │
│ Extract │ │ Run tests │ │ Debug / Verify │
│ │ │ Git ops │ │ Multi-file, multi- │
│ Web only │ │ Editor only │ │ step, multi-tool │
└──────────────┘ └──────────────┘ └──────────────────────┘
2020-2022 2022-2024 2024-present
Each phase expanded the action space while the core challenge remained the same: designing the interface so that a language model can operate effectively within it.
Real Code Examples: Side-by-Side
Here is the same task -- find a configuration file, read its contents, and update the port number -- expressed in each ACI.
SWE-Agent ACI
AGENT: find_file "config"
SYSTEM:
Found 2 files:
config/settings.yaml
config/settings.test.yaml
AGENT: view config/settings.yaml
SYSTEM:
[1] server:
[2] host: "0.0.0.0"
[3] port: 8080
AGENT: edit config/settings.yaml
OLD: port: 8080
NEW: port: 3000
SYSTEM:
File edited successfully. 1 line changed.
Codex CLI (Bash)
# Agent runs standard bash in sandbox
$ find . -name "settings.yaml"
./config/settings.yaml
./config/settings.test.yaml
$ cat config/settings.yaml
server:
host: "0.0.0.0"
port: 8080
$ sed -i 's/port: 8080/port: 3000/' config/settings.yaml
$ cat config/settings.yaml
server:
host: "0.0.0.0"
port: 3000
Claude Code
→ Glob "**/settings.yaml"
config/settings.yaml
config/settings.test.yaml
→ Read /project/config/settings.yaml
[1] server:
[2] host: "0.0.0.0"
[3] port: 8080
→ Edit /project/config/settings.yaml
OLD: port: 8080
NEW: port: 3000
✓ File updated successfully
The functional outcome is identical across all three. The differences are in verbosity (SWE-Agent and Claude Code are more token-efficient), error surface (bash sed can silently do the wrong thing if the regex is ambiguous), and generality (Codex can do things the other two cannot, but it also makes more mistakes on simple tasks).
The ACI Design Checklist
When evaluating or building an agent-computer interface, these are the properties to verify:
- Atomic commands: Each command performs one clear action with no hidden side effects
- Structured output: Observations are formatted consistently, not raw terminal text
- Bounded context: Long outputs are truncated or paginated; the agent can request specifics
- Error actionability: Failures include a description and a suggested next step
- Undo support: Every mutation can be reversed without manual reconstruction
- Permission model: Dangerous actions (deletes, deployments) require explicit authorization
- State isolation: Parallel agents or retries start from a known-good state
- Token efficiency: The average observation per turn is under 500 tokens for routine operations
- Extension points: New commands or tools can be added without rewriting the interface
The ACI and the Agent Loop
The ACI is not independent from the agent loop -- it is the implementation of the Act and Observe phases. In the agent loop (reason, act, observe, verify, iterate), the ACI defines what "act" and "observe" actually mean at the machine level.
The strongest agent systems treat the ACI as a first-class engineering concern, not as a thin wrapper around existing tools. SWE-Agent's paper demonstrated that the ACI alone accounts for the majority of performance differences between agent systems -- even when the underlying language model is identical.
The implication for loop engineers: if you are building an agent and it is not performing well, the first place to look is the interface. The model may be capable, but if it cannot express its intent through the ACI, capability is irrelevant.