Runner & results

Programmatic entry points and the structured result they return. All exports come from "@blopai/cli".

runBlopTest(options)

function runBlopTest(options: BlopRunOptions): Promise<BlopRunResult>

Convenience wrapper for running a single spec file. Requires options.specFile.

const result = await runBlopTest({
  specFile: "tests/homepage.blop.ts",
  baseUrl: "http://localhost:3000",
});

runBlopTests(options)

function runBlopTests(options: BlopRunOptions): Promise<BlopRunResult>

The core runner. Launches a Playwright browser, loads spec files, runs each test against a live browser context, streams agent events, records actions / screenshots / events, writes reports, and optionally uploads to Blop Platform.

const result = await runBlopTests({
  specFiles: ["tests/homepage.blop.ts", "tests/login.blop.ts"],
  baseUrl: "http://localhost:3000",
  browser: "chromium",
  reporter: "all",
});

BlopRunOptions

type BlopRunOptions = {
  // What to run
  specFile?: string;        // Single spec (for runBlopTest)
  specFiles?: string[];     // Multiple specs (for runBlopTests)

  // App under test
  baseUrl?: string;
  browserContext?: BrowserContextOptions;

  // Browser
  browser?: BlopBrowserName;
  viewport?: { width: number; height: number };
  headed?: boolean;

  // Agent
  provider?: string;
  model?: string;
  apiKey?: string;
  cwd?: string;
  maxSteps?: number;
  timeoutMs?: number;
  retries?: number;

  // Reporting
  reportDir?: string;
  reporter?: BlopReporter;
  verbose?: boolean;

  // Platform
  platformUrl?: string;
  platformApiKey?: string;

  // Testing the runner itself
  agentStream?: BlopAgentStreamRunner;
};

BlopRunResult

The top-level result returned by runBlopTest / runBlopTests.

type BlopRunResult = {
  runId: string;              // Unique run identifier
  status: BlopTestStatus;     // Aggregate status
  startedAt: string;          // ISO 8601
  finishedAt: string;         // ISO 8601
  durationMs: number;
  results: BlopTestResult[];  // Per-test results
};

The aggregate status is computed as:

• "error" if any test errored
• "failed" if any test failed (but none errored)
• "passed" otherwise

BlopTestResult

Per-test result with full execution detail.

type BlopTestResult = {
  id: string;                 // Unique test identifier
  name: string;
  status: BlopTestStatus;     // "passed" | "failed" | "error"
  reason: string;             // Human-readable explanation
  startedAt: string;          // ISO 8601
  finishedAt: string;         // ISO 8601
  durationMs: number;
  attempts: number;           // 1 + retries used
  baseUrl: string | null;
  provider: string | null;
  model: string | null;
  ci: BlopCiMetadata;
  screenshots: string[];      // Paths to screenshots
  actions: BlopAction[];      // Recorded browser actions
  events: BlopAgentEvent[];   // Full agent event stream
};

BlopTestStatus

type BlopTestStatus = "passed" | "failed" | "error";

Status	Meaning
passed	Agent called finish_test with status "passed"
failed	Agent called finish_test with status "failed" (app-level failure)
error	Runtime error: timeout, crash, no agent output, or invalid spec

BlopAction

A recorded browser tool call.

type BlopAction = {
  name: string;                       // e.g. "browser_click"
  input: Record<string, unknown>;     // Tool input parameters
  output: string;                     // Tool output (typically the agent-visible string)
  metadata?: Record<string, unknown>;
  timestamp: string;                  // ISO 8601
};

BlopAgentEvent

A streaming agent event recorded during the run.

type BlopAgentEvent = {
  event_type: string;                          // "step_start" | "step_finish" | "tool_result" | "text_delta" | ...
  content: string | null;
  metadata: Record<string, unknown> | null;
  workspace_id?: string | null;
  session_id?: string | null;
  timestamp: string;                           // ISO 8601
};

BlopCiMetadata

type BlopCiMetadata = {
  provider: string | null;    // "github-actions" or null
  runId: string | null;
  jobId: string | null;
  branch: string | null;
  commitSha: string | null;
  pullRequest: string | null;       // Raw ref, e.g. "refs/pull/42/merge"
  pullRequestNumber: string | null; // Just the number, e.g. "42"
  repositoryUrl: string | null;
  workflowName: string | null;
  runAttempt: string | null;
  runUrl: string | null;
  refType: "branch" | "tag" | null;
};

Auto-detected on GitHub Actions; other providers fall back to null fields.

BlopAgentStreamRunner

For testing the runner without a real model — see Guides → Writing tests.

type BlopAgentStreamRunner = (options: {
  prompt: string;
  provider?: string;
  model?: string;
  apiKey?: string;
  cwd?: string;
  nativeTools: unknown[];
  signal?: AbortSignal;
}) => AsyncIterable<BlopAgentStreamEvent>;

type BlopAgentStreamEvent = {
  event_type: string;
  content?: string | null;
  metadata?: Record<string, unknown> | null;
  workspace_id?: string | null;
  session_id?: string | null;
};

Platform upload

The runner uploads its CloudEvents and report bundle when BLOP_INGEST_URL, BLOP_INGEST_SECRET, and BLOP_PROJECT_ID are set. Configuration fields platformUrl and platformApiKey can supply the first two values. Local monitoring does not use these settings or upload its SQLite data.