Loop Engineering Testing Strategies

Testing an autonomous coding system is fundamentally different from testing traditional software. When an AI agent drives a loop, the system is non-deterministic, has access to external tools, and can theoretically run forever. Standard testing approaches break down — you need strategies designed specifically for the agentic loop paradigm.

This guide covers a comprehensive testing framework for loop engineering systems, from unit tests for individual phases to full benchmark suites that evaluate loop behavior across hundreds of scenarios.

Why Testing AI Loops Is Different

Traditional software testing assumes deterministic behavior: given the same input, you get the same output. Loop engineering breaks this assumption at every level.

  Traditional Testing           Loop Engineering Testing
  ────────────────────         ─────────────────────────
  Input ──▶ Function ──▶       Input ──▶ LLM ──▶ Tool ──▶
  Output (deterministic)               Loop ──▶ Output
                                      (non-deterministic)

  Assert: output == expected           Assert: output SATISFIES
                                       constraints (probabilistic)

The core differences demand a fundamentally different testing philosophy:

Dimension	Traditional Testing	Loop Testing
Determinism	Required	Cannot be guaranteed
Runtime	Milliseconds	Seconds to minutes per loop
External deps	Mocked	Real tools preferred
Assertions	Exact match	Property-based, fuzzy match
Cost	Free (CPU)	Expensive (LLM tokens)
Flakiness	Zero tolerance	Managed via statistics

Challenge 1: Non-Deterministic Outputs

An LLM will produce different code on each call, even with the same prompt. This means you cannot write expect(output).toBe(expectedOutput) tests. Instead, you test properties and invariants.

// WRONG: Exact match assertion (will flake)
test('generates correct code', () => {
  const result = loop.run('fix the bug in parser.ts');
  expect(result.code).toBe(expectedFix);
});

// RIGHT: Property-based assertion (stable)
test('generated code compiles without errors', async () => {
  const result = await loop.run('fix the bug in parser.ts');
  const compileResult = await compile(result.code);
  expect(compileResult.errors).toHaveLength(0);
});

Strategies for Non-Determinism

Temperature control: Use temperature: 0 for testing to minimize variance while acknowledging it does not eliminate it entirely.

Statistical testing: Run the same test N times and assert that the success rate exceeds a threshold.

async function statisticalTest(
  testFn: () => Promise<boolean>,
  iterations: number = 20,
  threshold: number = 0.8
): Promise<void> {
  let successes = 0;
  for (let i = 0; i < iterations; i++) {
    if (await testFn()) successes++;
  }
  const rate = successes / iterations;
  expect(rate).toBeGreaterThanOrEqual(threshold);
}

Challenge 2: Infinite Loop Potential

A loop that never converges is both a correctness bug and a resource leak. Every test must enforce termination.

test('loop terminates within budget', async () => {
  const loop = new AutoCorrectionLoop({
    task: 'add error handling to userService',
    maxIterations: 20,
    timeoutMs: 120_000
  });

  const startTime = Date.now();
  const result = await loop.run();

  // Must terminate
  expect(result.terminated).toBe(true);

  // Must not exceed budget
  expect(Date.now() - startTime).toBeLessThan(130_000);
  expect(result.iterations).toBeLessThanOrEqual(20);
}, 150_000); // Jest/test timeout must exceed loop timeout

Timeout Enforcement in Tests

// Utility: enforce loop termination in every test
function withLoopTimeout<T>(
  loopPromise: Promise<T>,
  maxMs: number
): Promise<T> {
  return Promise.race([
    loopPromise,
    new Promise<never>((_, reject) =>
      setTimeout(
        () => reject(new Error(`Loop did not terminate within ${maxMs}ms`)),
        maxMs
      )
    )
  ]);
}

Challenge 3: Tool Dependency Mocking

Agentic loops interact with file systems, shells, test runners, and compilers. Tests need to control these dependencies without requiring the full system.

The Mocking Spectrum

  No Mocking                         Full Mocking
  ───────────                        ────────────
  ┌──────────┐                       ┌──────────┐
  │ Real     │    ┌──────────┐      │ Mock LLM │
  │ Tools    │    │ Sandboxed│      │ Mock     │
  │ Real LLM │    │ Tools    │      │ Tools    │
  │ Full     │    │ Real LLM │      │ Full     │
  │ System   │    │ Isolated │      │ Control  │
  └──────────┘    └──────────┘      └──────────┘

  Slow           Medium              Fast
  Expensive      Moderate            Cheap
  Realistic      Mostly Realistic    Deterministic
  E2E Tests      Integration Tests   Unit Tests

Mock LLM Implementation

class ScriptedLLM {
  private responses: string[] = [];
  private callIndex = 0;

  constructor(responses: string[]) {
    this.responses = responses;
  }

  async complete(prompt: string): Promise<string> {
    if (this.callIndex >= this.responses.length) {
      throw new Error(
        `ScriptedLLM: unexpected call #${this.callIndex}. ` +
        `Only ${this.responses.length} responses scripted.`
      );
    }
    return this.responses[this.callIndex++];
  }

  reset(): void {
    this.callIndex = 0;
  }
}

// Test: verify loop calls LLM in correct sequence
test('loop requests plan then implementation', async () => {
  const mockLLM = new ScriptedLLM([
    'I need to create a function called `validateEmail`',
    '```typescript\nfunction validateEmail(email: string): boolean {\n  return /@/.test(email);\n}\n```'
  ]);

  const loop = new AutoCorrectionLoop({ llm: mockLLM });
  await loop.run('add email validation');

  expect(mockLLM.callIndex).toBe(2); // All scripted responses consumed
});

Mock File System

class MockFileSystem implements FileSystem {
  private files = new Map<string, string>();

  async readFile(path: string): Promise<string> {
    const content = this.files.get(path);
    if (content === undefined) {
      throw new Error(`ENOENT: ${path} not found`);
    }
    return content;
  }

  async writeFile(path: string, content: string): Promise<void> {
    this.files.set(path, content);
  }

  async exists(path: string): Promise<boolean> {
    return this.files.has(path);
  }

  getFiles(): string[] {
    return Array.from(this.files.keys());
  }
}

Testing Levels

Unit Testing Individual Loop Phases

Test each phase of the loop (reason, act, verify, fix) in isolation.

// Unit test: the verify phase correctly detects test failures
describe('VerifyPhase', () => {
  it('detects failing test output', () => {
    const testOutput = `
      PASS src/utils/math.spec.ts
      FAIL src/utils/parser.spec.ts
        ✕ should handle nested objects (5ms)
          Expected: { a: { b: 1 } }
          Received: { a: { b: 2 } }
    `;

    const result = verifyPhase.parseTestOutput(testOutput);
    expect(result.passed).toBe(false);
    expect(result.failures).toHaveLength(1);
    expect(result.failures[0].name).toBe('should handle nested objects');
  });

  it('recognizes all-pass output', () => {
    const testOutput = 'Tests: 5 passed, 5 total';
    const result = verifyPhase.parseTestOutput(testOutput);
    expect(result.passed).toBe(true);
    expect(result.failures).toHaveLength(0);
  });
});

// Unit test: the fix phase generates valid patches
describe('FixPhase', () => {
  it('produces a non-empty diff', async () => {
    const error = {
      file: 'parser.ts',
      line: 42,
      message: "Cannot read property 'length' of undefined"
    };

    const patch = await fixPhase.generatePatch(error, context);
    expect(patch).toBeTruthy();
    expect(patch.targetFile).toBe('parser.ts');
  });
});

Integration Testing Full Loop Cycles

Test the complete loop with mocked dependencies, verifying the interaction between phases.

describe('AutoCorrectionLoop integration', () => {
  it('fixes a simple bug in one iteration', async () => {
    const mockLLM = new ScriptedLLM([
      'I will change line 42 from `.length` to `?.length`',
      '```diff\n- return data.items.length;\n+ return data.items?.length ?? 0;\n```'
    ]);
    const mockFS = new MockFileSystem();
    mockFS.writeFile('utils.ts', 'return data.items.length;');

    const loop = new AutoCorrectionLoop({
      llm: mockLLM,
      fs: mockFS,
      testRunner: () => ({ passed: true, failures: [] })
    });

    const result = await loop.run('fix null safety in utils.ts');

    expect(result.success).toBe(true);
    expect(result.iterations).toBe(1);
    expect(mockFS.getFiles()).toContain('utils.ts');
  });

  it('escalates after max iterations', async () => {
    const alwaysFailLLM = new ScriptedLLM(
      Array(10).fill('```diff\n- old\n+ also wrong\n```')
    );

    const loop = new AutoCorrectionLoop({
      llm: alwaysFailLLM,
      maxIterations: 5,
      testRunner: () => ({
        passed: false,
        failures: [{ name: 'test', message: 'still broken' }]
      })
    });

    const result = await loop.run('fix complex bug');

    expect(result.success).toBe(false);
    expect(result.iterations).toBe(5);
    expect(result.terminationReason).toBe('max_iterations');
  });
});

End-to-End Testing with Real Tools

The highest-fidelity test uses real tools (actual compiler, test runner) but with a controlled environment.

describe('E2E loop test', () => {
  it('adds a function to a real TypeScript file', async () => {
    // Use a temporary directory with real tools
    const tmpDir = await createTempDir('loop-e2e-');
    await writeFile(`${tmpDir}/math.ts`, `
      export function add(a: number, b: number): number {
        return a + b;
      }
    `);
    await writeFile(`${tmpDir}/math.spec.ts`, `
      import { add, subtract } from './math';
      import { describe, it, expect } from 'vitest';
      describe('math', () => {
        it('adds', () => expect(add(1, 2)).toBe(3));
        it('subtracts', () => expect(subtract(5, 3)).toBe(2));
      });
    `);

    const loop = new AutoCorrectionLoop({
      workingDir: tmpDir,
      testCommand: 'npx vitest run',
      timeout: 60_000
    });

    const result = await loop.run(
      'add a subtract function to math.ts that matches the test'
    );

    expect(result.success).toBe(true);
    const content = await readFile(`${tmpDir}/math.ts`);
    expect(content).toContain('subtract');

    await cleanup(tmpDir);
  }, 90_000);
});

Regression Testing for Loop Behavior

Track whether loop behavior changes over time by maintaining a corpus of tasks with expected outcomes.

describe('Regression suite', () => {
  const regressionTasks = loadYaml('./regression-tasks.yaml');

  for (const task of regressionTasks) {
    it(task.name, async () => {
      const result = await loop.run(task.input, {
        maxIterations: task.maxIterations ?? 20,
        timeoutMs: task.timeoutMs ?? 120_000
      });

      expect(result.success).toBe(task.expectSuccess);
      if (task.expectIterations) {
        expect(result.iterations).toBeLessThanOrEqual(
          task.expectIterations
        );
      }
    }, task.timeoutMs ?? 150_000);
  }
});

Golden Path Testing

Golden path tests verify that the loop follows an expected trajectory through its states. Instead of asserting exact outputs, they check that the loop visited the right sequence of phases.

interface LoopTrace {
  iterations: Array<{
    phase: string;
    action: string;
    result: string;
  }>;
}

test('follows expected golden path for type fix', async () => {
  const trace: LoopTrace = { iterations: [] };

  const instrumentedLoop = new AutoCorrectionLoop({
    task: 'fix TypeScript type error in api.ts',
    onPhaseComplete: (phase, action, result) => {
      trace.iterations.push({ phase, action, result });
    }
  });

  await instrumentedLoop.run();

  // Verify the trajectory
  const phases = trace.iterations.map(i => i.phase);

  // Must start with reasoning/planning
  expect(phases[0]).toMatch(/plan|reason/);

  // Must include implementation
  expect(phases).toContain('act');

  // Must include verification
  expect(phases).toContain('verify');

  // Must eventually succeed (or gracefully stop)
  const lastIteration = trace.iterations[trace.iterations.length - 1];
  expect(['success', 'escalated']).toContain(lastIteration.result);
});

// Subsequence matching for partial golden paths
function containsSubsequence<T>(
  sequence: T[],
  subsequence: T[]
): boolean {
  let subIdx = 0;
  for (const item of sequence) {
    if (item === subsequence[subIdx]) subIdx++;
    if (subIdx === subsequence.length) return true;
  }
  return false;
}

test('eventually reaches test-and-fix cycle', async () => {
  const trace = await runTracedLoop(task);
  const phases = trace.iterations.map(i => i.phase);

  // The loop should eventually enter a verify-act cycle
  expect(
    containsSubsequence(phases, ['verify', 'fix', 'verify'])
  ).toBe(true);
});

Property-Based Testing for Loops

Property-based testing asserts invariants that must hold regardless of the specific inputs or non-deterministic outputs.

describe('Loop invariants', () => {
  it('never deletes files it did not create', async () => {
    const existingFiles = ['index.ts', 'config.json', 'README.md'];
    const fs = new MockFileSystem();
    existingFiles.forEach(f => fs.writeFile(f, 'original'));

    await loop.run('add new utility function', { fs });

    // Invariant: existing files still exist
    for (const file of existingFiles) {
      expect(await fs.exists(file)).toBe(true);
    }
  });

  it('produces syntactically valid code', async () => {
    const result = await loop.run('implement sorting algorithm');
    expect(result.code).toBeTruthy();

    // Parse as AST to verify syntactic validity
    expect(() => TypeScript.parse(result.code)).not.toThrow();
  });

  it('always terminates within max iterations', async () => {
    for (const task of sampleTasks) {
      const result = await loop.run(task, { maxIterations: 30 });
      expect(result.iterations).toBeLessThanOrEqual(30);
    }
  });

  it('monotonically reduces error count', async () => {
    const trace = await runTracedLoop('fix all type errors');
    let prevErrorCount = Infinity;

    for (const iteration of trace.iterations) {
      if (iteration.errorCount !== undefined) {
        expect(iteration.errorCount).toBeLessThanOrEqual(prevErrorCount);
        prevErrorCount = iteration.errorCount;
      }
    }
  });
});

Convergence Testing

A critical property of any loop is that it converges — it reaches a stable state and stops.

describe('Convergence', () => {
  it('converges on simple tasks within 5 iterations', async () => {
    const result = await loop.run('add JSDoc to exported functions');
    expect(result.success).toBe(true);
    expect(result.iterations).toBeLessThanOrEqual(5);
  });

  it('does not oscillate between solutions', async () => {
    const trace = await runTracedLoop('fix boundary condition');

    // Count how many times the same file is modified
    const fileEdits = new Map<string, number>();
    for (const iteration of trace.iterations) {
      for (const file of iteration.filesChanged) {
        fileEdits.set(file, (fileEdits.get(file) || 0) + 1);
      }
    }

    // No file should be edited more than 3 times (oscillation threshold)
    for (const [file, count] of fileEdits) {
      expect(count).toBeLessThanOrEqual(3);
    }
  });

  it('reports convergence type correctly', async () => {
    const tasks = [
      { task: 'add empty function', expectedType: 'exact' },
      { task: 'improve error messages', expectedType: 'approximate' },
    ];

    for (const { task, expectedType } of tasks) {
      const result = await loop.run(task);
      expect(result.convergenceType).toBe(expectedType);
    }
  });
});

Benchmark Suites for Loop Evaluation

A benchmark suite measures loop performance across a standardized set of tasks, tracking metrics over time.

  benchmark-suite/
  ├── tasks/
  │   ├── unit/
  │   │   ├── add-function.yaml
  │   │   ├── fix-type-error.yaml
  │   │   └── add-import.yaml
  │   ├── integration/
  │   │   ├── implement-feature.yaml
  │   │   └── fix-bug.yaml
  │   └── e2e/
  │       ├── refactor-module.yaml
  │       └── migrate-api.yaml
  ├── results/
  │   ├── baseline-2026-07-01.json
  │   └── latest.json
  └── runner.ts

Benchmark Runner

interface BenchmarkResult {
  taskId: string;
  success: boolean;
  iterations: number;
  durationMs: number;
  tokensUsed: number;
  convergenceType: string;
}

async function runBenchmark(
  loop: AutoCorrectionLoop,
  tasksDir: string
): Promise<BenchmarkResult[]> {
  const taskFiles = await glob(`${tasksDir}/**/*.yaml`);
  const results: BenchmarkResult[] = [];

  for (const file of taskFiles) {
    const task = loadYaml(file);
    const startTime = Date.now();
    const tokenCounter = new TokenCounter();

    const result = await loop.run(task.prompt, {
      maxIterations: task.maxIterations,
      timeoutMs: task.timeoutMs
    });

    results.push({
      taskId: task.id,
      success: result.success,
      iterations: result.iterations,
      durationMs: Date.now() - startTime,
      tokensUsed: tokenCounter.total,
      convergenceType: result.convergenceType
    });
  }

  return results;
}

function printBenchmarkReport(results: BenchmarkResult[]): void {
  const successRate = results.filter(r => r.success).length / results.length;
  const avgIterations = average(results.map(r => r.iterations));
  const avgDuration = average(results.map(r => r.durationMs));
  const avgTokens = average(results.map(r => r.tokensUsed));

  console.log('=== Loop Engineering Benchmark Report ===');
  console.log(`Tasks:      ${results.length}`);
  console.log(`Success:    ${(successRate * 100).toFixed(1)}%`);
  console.log(`Avg iters:  ${avgIterations.toFixed(1)}`);
  console.log(`Avg time:   ${(avgDuration / 1000).toFixed(1)}s`);
  console.log(`Avg tokens: ${avgTokens.toFixed(0)}`);
}

Benchmark Categories

Category	Task Count	Avg Duration	What It Measures
Unit	20	5-15s	Basic loop mechanics
Integration	10	30-90s	Multi-phase coordination
E2E	5	2-10min	Full workflow capability
Stress	5	Variable	Resource limits and edge cases
Regression	10	Variable	Behavior stability over time

The Loop Testing Pyramid

           /\
          /  \
         / E2E\              5% of tests
        /      \             Slow, expensive, high fidelity
       /────────\
      /Integra-  \           15% of tests
     /  tion      \          Medium speed, partial mocks
    /──────────────\
   /   Unit Tests   \       80% of tests
  /  (phases, verify,  \    Fast, fully mocked, deterministic
 /    fix, detect)       \
/__________________________\

Level	Frequency	Speed	Fidelity	Mocking
Unit	Every commit	<100ms	Low	Full mocks
Integration	Every PR	1-10s	Medium	Partial mocks
E2E	Before release	30s-10m	High	Real tools
Benchmark	Weekly	Variable	Very high	Real LLM + tools

Testing Anti-Patterns to Avoid

Anti-Pattern 1: Testing Exact Outputs

// WRONG
expect(result.code).toBe('function add(a, b) { return a + b; }');

// RIGHT: test properties of the output
expect(result.code).toContain('function');
expect(result.code).toContain('add');
expect(() => new Function(result.code)).not.toThrow();

Anti-Pattern 2: Ignoring Token Costs

Tests that use real LLMs should track and assert on token usage to prevent cost regression.

// WRONG: no cost awareness
await loop.run('implement feature');

// RIGHT: budget-aware testing
const tokenBudget = 10_000;
const result = await loop.run('implement feature');
expect(result.tokensUsed).toBeLessThan(tokenBudget);

Anti-Pattern 3: Testing Only Happy Paths

// WRONG: only testing success
test('loop works', () => { ... });

// RIGHT: testing failure modes too
describe('error scenarios', () => {
  test('handles rate limits gracefully', async () => {
    const rateLimitedLLM = new ScriptedLLM([], {
      errorOnCall: new Error('429 Rate Limited')
    });
    const loop = new AutoCorrectionLoop({ llm: rateLimitedLLM });
    const result = await loop.run('fix bug');
    expect(result.terminationReason).toBe('rate_limit');
  });
});

Anti-Pattern 4: No Timeout on Loop Tests

// WRONG: can hang forever
test('loop converges', () => loop.run('task'));

// RIGHT: always enforce timeouts
test('loop converges', () => loop.run('task'), 120_000);

Key Takeaways

• Non-determinism is the central challenge — assert on properties and invariants, never on exact outputs from the LLM
• Every loop test must enforce termination with max iterations, timeouts, and budget limits as first-class test assertions
• The testing pyramid applies to loops: 80% unit tests (fully mocked, deterministic), 15% integration (partial mocks), 5% E2E (real tools)
• Golden path testing verifies that the loop follows an expected trajectory through phases, rather than asserting exact code output
• Mock the LLM with scripted responses for fast, deterministic unit tests — but complement with real-LLM E2E tests before release
• Property-based tests catch subtle bugs by asserting invariants like "never deletes existing files" or "monotonically reduces error count"
• Benchmark suites track loop performance over time, catching regressions in convergence speed, token usage, and success rate
• Budget-awareness is essential — track and assert on token usage, iteration counts, and wall-clock time in every test to prevent cost regressions