Test suites

A test in TaskSultan is not a unit test. It is an automation run with expectations attached, executed through the same Robot CLI the Studio's Run button uses. So "the suite passed" means the automations worked, not that a mock agreed with itself. A suite is a named list of tests, stored as JSON under the state root.

#What a suite contains

{
  "id": "platform-smoke",
  "name": "Platform smoke",
  "maxDurationMs": 900000,
  "tests": [
    {
      "id": "process-invoices-smoke",
      "automation": "process-invoices",
      "purpose": "The invoice automation processes the sample workbook without a system failure.",
      "labels": ["smoke"],
      "timeoutMs": 180000,
      "expect": {
        "status": ["success"],
        "totalsAtLeast": { "transactions": 1, "systemExceptions": 0 }
      }
    }
  ]
}

Each test needs id, automation and purpose. automation names automations/<automation>.automation.ts.

expect holds the assertions, all read from the run's own evidence (state.json and logs.txt) with the platform's reader:

• status: accepted terminal statuses. Default ['success'].
• maxDurationMs: the run must finish within this many milliseconds of its own start.
• totals: exact match against the run's totals.
• totalsAtLeast: lower bounds, for stable assertions like "at least one transaction".
• logContains and logAbsent: substrings that must, or must not, appear in logs.txt.

timeoutMs is the wall-clock budget for one test. On expiry the child is killed and the test fails, so a hung test cannot hang the suite. maxDurationMs on the suite is a ceiling for the whole run: tests that never start are recorded as cancelled. labels enable subset runs. credentials names assets the automation resolves, injected through the vault into the Robot's environment. params sets data-driven environment values for the test.

#Managing and running suites

npx tsx orchestrator/cli.ts suite add suites/platform-smoke.json
npx tsx orchestrator/cli.ts suite list
npx tsx orchestrator/cli.ts suite show platform-smoke
npx tsx orchestrator/cli.ts suite run platform-smoke
npx tsx orchestrator/cli.ts suite run platform-smoke --only smoke --artifacts artifacts
npx tsx orchestrator/cli.ts suite runs --suite platform-smoke --limit 20
npx tsx orchestrator/cli.ts suite result <suiteRunId>

suite run prints a PASS, FAIL, SKIP or CANCEL line as each test finishes, then the totals and the evidence path for each failure. --only tag,tag selects tests by label. --headed shows the browser. --trigger on-demand|scheduled records how the run was started. --artifacts d names the artifact root. The exit code is the verdict: 0 when the suite passed, 1 otherwise. A scheduler or a CI job can act on that without parsing output.

There are four outcomes, kept apart. passed and failed are the tests. skipped covers a disabled test or one not selected by --only, always with the reason, so a green subset can never be read as "everything was checked". cancelled means the suite ceiling was reached before the test started, which is its own result: "we stopped asking" is not "it broke".

A suite reports every failure, not the first. Test definitions and run records are JSON files under the state root in suites/ and suite-runs/, written with the same atomic writer every other store uses. The Studio reads them and never runs a suite in-process.

#Environment hygiene

A suite spawns the Robot with a clean environment. The platform's own override variables are stripped from the ambient environment: TS_TARGET_URL, TS_DEMO_URL and TS_DATA_DIR. A stale TS_TARGET_URL in the shell that launched the run must not be able to redirect a test. To target a fixture, declare it in the test's params, which makes the choice visible in the suite file instead of invisible in whoever happened to launch it. The runner records, on passes and failures alike, which overrides it stripped and which params it applied.

Failure evidence is handed over, not hidden: a failing test carries its screenshots, its trace when there is one, plus paths to state.json, logs.txt and summary.json.