Validation and testing

ArcFlow validates workflow definitions before execution and supports deterministic test mode for CI without live LLM calls. Validation catches schema errors, broken graph references, and missing agent ids early. Test mode simulates step outputs and controlled failures so you can verify retry, fallback, and trace shape cheaply.

Normative schema: workflow schema. Conceptual overview: Workflow specification.

Engine validation

Before a run starts, the engine calls:

Failure surfaces as WorkflowValidationFailed in trace and InvalidWorkflowDefinition error code (HTTP 400 on server).

Common failures:

• Step agent_id not present in run agents list
• Graph step_ref pointing at missing step id
• Join node wait_for referencing unknown node ids
• execution_mode: "graph" without graph object

CLI validate (CLI validate command)

arcflow validate workflow.json

Status: stub. Full schema validation in the CLI is deferred (CLI validate command, ). Today:

• Use JSON Schema validation in CI against v1.schema.json
• Rely on engine validation at run time
• Use arcflow run with test mode for integration checks

Do not document arcflow validate as production-complete until CLI validate command ships.

JSON Schema in CI

Example with a generic validator (adapt to your toolchain):

npx ajv validate -s contracts/normative/rcs/v1.schema.json -d my-workflow.json

Validate both workflow and agent bundles before PUT registry publish. See Workflow registry.

Test mode (exec_config.test)

ExecutionConfig.test drives per-step mock behavior without calling providers.

{
 "recovery_enabled": false,
 "test": {
 "steps": {
 "00000000-0000-4000-8000-000000000010": {
 "output": "mock step 1",
 "fail_times": 0
 },
 "00000000-0000-4000-8000-000000000011": {
 "fail_times": 1,
 "output": "fail first",
 "then_output": "success second attempt"
 }
 }
 }
}

Use fail_times to exercise Retry and backoff traces (RetryAttempted, RetryExhausted). Combine with Step fallbacks to assert StepFallbackActivated.

Stub provider

For runs that still invoke the agent loop but need deterministic LLM behavior, set agent provider to stub:

{
 "provider": {
 "provider_id": "stub",
 "model": "stub-v1",
 "api_key_env": ""
 }
}

Stub is for tests only. First workflow in five minutes uses stub by default.

Graph validation checklist

For Graph workflows:

1. Every step_ref in nodes matches a step id
2. entry_node exists in nodes
3. All edges.from and edges.to reference valid node ids (or to: null)
4. join_nodes[].wait_for lists nodes reachable on parallel paths
5. max_iterations is set to a sane bound

Test routing with test mode outputs that match edge conditions:

{
 "test": {
 "steps": {
 "s-classify": { "output": "billing" },
 "s-billing": { "output": "billing resolution" },
 "s-merge": { "output": "merged summary" }
 }
 }
}

Server test run

curl -s -X POST http://localhost:8080/v1/runs \
 -H "Authorization: Bearer $ARCFLOW_SERVER_API_KEY" \
 -H "Content-Type: application/json" \
 -d @test-run.json

Poll GET /v1/runs/{run_id}/trace and assert event kinds. Traces are trace data policy safe; see Trace data policy.

Recovery testing

Linear recovery with test mode:

{
 "recovery_enabled": true,
 "test": {
 "steps": {
 "00000000-0000-4000-8000-000000000010": { "output": "step one" },
 "00000000-0000-4000-8000-000000000011": { "fail_times": 999, "output": "never" }
 }
 }
}

Simulate interrupt mid-run, then resume per Recovery and resume. Graph checkpoint resume is partial (Graph recovery resume); do not treat graph resume tests as release gates until Graph recovery resume closes.

Verification matrix

Related pages

• Linear workflows
• Graph workflows
• Maturity and known gaps (CLI validate command, Graph recovery resume)
• Install and build