Execution¶

Added in v0.2.

By default, teststop run generates adversarial scenarios and validates them structurally. Point it at a running system with --target and teststop will also execute those scenarios and feed the real pass/fail outcomes into confidence memory.

This is the jump from a scenario generator to a scenario runner.

teststop tests what's running — it doesn't run your app¶

teststop never starts, builds, or manages the system under test. You run your app however you like — a local process, a container, a staging deployment — and point --target at it:

teststop run --target http://localhost:8080

This is deliberate. teststop is a thin trigger; managing your app's lifecycle would make it "the new loop that needs maintaining," which the project explicitly avoids. The upside: you can target any environment, including a production-like instance, and teststop adapts to it.

Hybrid execution¶

teststop chooses how to execute each scenario individually:

flowchart TD
    A[Scenario] --> B{exec block<br/>and --target set?}
    B -- yes --> C[HTTP executor<br/>deterministic net/http]
    B -- no --> D{--target set?}
    D -- yes --> E[AI executor<br/>AI performs the steps]
    D -- no --> F[Static executor<br/>structural validation]

The HTTP path is deterministic and fast (no AI call at execution time). The AI path covers open-ended, chaos-heavy scenarios that can't be reduced to one request. See the exec field for the structured contract.

Predicted vs executed¶

This distinction matters when you read the results:

• Predicted (no --target) — teststop generates scenarios and validates them structurally. Nothing is sent to a system. The report is a risk surface, and the confidence is labelled PREDICTED — it is not evidence of correctness.
• Executed (with --target) — scenarios are run and judged on real responses. Failures are observed, not guessed, and confidence reflects verified behavior.

The JSON output makes this explicit via exec_summary.executed. Treat a predicted run as a checklist of "things that could break a system like this," not a defect count — run with --target to verify which (if any) actually break.

Concurrency races¶

A single request can't prove a system is safe against double-submit or claim-the-last-item races. Set concurrency in the exec block and teststop fires N identical requests simultaneously, asserting the guard lets at most one winner through:

{
  "exec": {
    "mode": "http",
    "method": "POST",
    "path": "/actions/42/approve",
    "expected_status": 200,
    "concurrency": 10
  }
}

A winner is any 2xx response (a request that actually succeeded). The bug teststop detects is more than one winner — the guard let concurrent duplicates mutate state.

• Pass — at most one request wins (a 2xx) and the rest are cleanly rejected with a 4xx (e.g. 409). Zero winners is also safe (nothing was mutated — e.g. an auth endpoint that correctly rejects every concurrent attempt).
• Fail — more than one request wins (the race is not guarded — the real bug), any request returns a 5xx, or a request fails to complete (transport error).

expected_status should be the winner's 2xx code; it is not used to classify winners, so a rejection code never counts as a success. actual_behavior reports the histogram, e.g. 10 concurrent POST …: 1×200, 9×409.

Flags¶

Execution runs concurrently with a bounded worker pool; results are returned in scenario order.

Sandbox and localhost¶

teststop runs the AI CLI inside an Apple Container sandbox when available. A sandboxed container cannot reach your host's localhost, so:

# Local target → run the AI tester directly
TESTSTOP_SANDBOX=none teststop run --target http://localhost:8080

# Remote / staging target → sandbox works fine
TESTSTOP_SANDBOX=required teststop run --target https://staging.example.com

The deterministic HTTP executor always runs host-side, so the sandbox setting does not affect HTTP-mode scenarios.

What execution looks like¶

Running against a small Go API:

TESTSTOP_SANDBOX=none teststop run --path ./sample-api \
  --target http://localhost:8099 --depth light --output json

{
  "exec_summary": { "executed": true, "count": 5, "passed": 4, "failed": 1, "target": "http://localhost:8099" },
  "executions": [
    {
      "scenario_id": "login-empty-whitespace-username",
      "area": "api/login",
      "mode": "http",
      "passed": false,
      "actual_behavior": "HTTP 401 in 5ms",
      "failure_reason": "expected status 400, got 401",
      "priority": "medium",
      "duration_ms": 5
    }
  ]
}

Here teststop found a real issue: a login with a whitespace-only username was authenticated (401) instead of being rejected as invalid input (400). That failure lowers confidence for the api/login area and is reflected in the exit code.

Effect on confidence and exit codes¶

• Each executed scenario updates its confidence_area: a pass raises confidence, a failure drops it (see Memory).
• A failed critical scenario sets exit code 2 (do not deploy).
• Below-threshold average confidence sets exit code 1 (review).

Skipped scenarios (infrastructure errors)¶

If the AI driver can't return a verdict for an infrastructure reason — the AI CLI exits with an error, hits a rate limit, or returns unparseable output — the scenario is marked skipped, not failed. Skipped scenarios:

• do not move confidence in either direction,
• are not counted as failures and never trigger exit code 2,
• are surfaced separately in the report and as exec_summary.skipped / executions[].skipped so you can see them without them polluting the signal.

This keeps a rate-limited run from fabricating "failures" that say nothing about the system under test. The report shows e.g. Results: 19 passed, 7 failed, 17 skipped (infrastructure error — not counted).

Without --target, well-formed scenarios pass structural validation — static-only mode.