Runner, reporters and types

The programmatic runner behind blop test run, the files it writes, and the types @blopai/test exports. Import from @blopai/test, never @blopai/cli or files under dist/; coverage helpers come from @blopai/test/coverage.

Runner functions

Function	Use
runBlopTest(options)	Run one spec file (options.specFile)
runBlopTests(options)	Run several spec files (options.specFiles) in a browser, write reports and optionally upload

Both return Promise<BlopRunResult>.

import { runBlopTests } from "@blopai/test";

const result = await runBlopTests({
  specFiles: ["tests/homepage.blop.ts"],
  baseUrl: "http://localhost:3000",
  reporter: "all",
});
if (result.status !== "passed") process.exit(1);

Runner options

BlopRunOptions fields. Every field is optional; the blop.config.ts fields in Configuration use the same names and defaults.

Field	Type	Purpose
specFile, specFiles	string, string[]	What to run
baseUrl	string	App under test; relative agent.goto() paths resolve against it
browserContext	Playwright BrowserContextOptions	Extra context options
browser, viewport, headed, containerized	See Configuration	Browser choice and display
provider, model, apiKey, reasoningEffort	string; "none" | "low" | "medium" | "high" | "max"	Provider settings; the effort applies when the API supports it
cwd	string	Working directory for the agent
maxSteps, timeoutMs, retries, workers	number	Step cap, per-check time cap, retries, concurrency
reportDir, reporter, verbose	string, BlopReporter, boolean	Reports
progressFile	string	Append live NDJSON progress (test_start, action, frame, test_finish)
captureStepScreenshots, streamFrames, frameIntervalMs	boolean, boolean, number	Per-action screenshots and live Chromium frames
coverage, coverageEndpoint	boolean, string	Chromium JavaScript coverage and the collector that receives it
platformUrl, platformApiKey	string	Blop Cloud upload; also needs BLOP_PROJECT_ID
otelEndpoint, otelHeaders, otelServiceName, otelEnvironment, otelPropagateToApp, otelPropagateAllowlist	See Configuration	OpenTelemetry export
evidence	BlopEvidence	Launch provenance from a trace or finding; never proof of a fix
agentStream	BlopAgentStreamRunner	Replace the model to test the runner; see Run browser checks without a provider

Result fields

BlopRunResult has runId, status, startedAt, finishedAt (ISO 8601), durationMs and results: BlopTestResult[]. Its status is error if any check errored, else failed if any failed, else passed.

BlopTestResult field	Meaning
id, name, specFile	Check identity; specFile is repo-relative or null
status, reason	Outcome and the agent’s explanation
startedAt, finishedAt, durationMs	Timing
attempts, firstAttemptStatus	1 plus retries used; the first attempt’s status when retried, else null
resumes	Agent recoveries in the same live browser; not retries
synthetic	true for a record the runner made for a spec load error or aborted run
baseUrl, provider, model	What ran, or null
ci	BlopCiMetadata
screenshots, screenshotArtifacts	Screenshot paths and their details
criticalPoints	Requirements the agent recorded with record_critical_point
browserLogs	Console messages, page errors and failed requests
actions	BlopAction: name, input, output, metadata, timestamp, durationMs for each tool call
events	BlopAgentEvent: event_type, content, metadata, timestamp

Status	Meaning
passed	The agent called finish_test with passed
failed	The agent finished with an unmet goal; check the assertions and evidence before blaming the app
error	The run did not finish normally: timeout, browser crash, provider failure or invalid spec

A final passed with a non-passing firstAttemptStatus is flaky.

BlopCiMetadata has provider, runId, jobId, branch, commitSha, pullRequest (raw ref such as refs/pull/42/merge), pullRequestNumber, repositoryUrl, workflowName, runAttempt, runUrl and refType (branch or tag). It is filled on GitHub Actions (provider: "github-actions") and null elsewhere.

Report files

Every run writes to reportDir (default .blop/), whether started from the CLI, watch mode or runBlopTests.

File	Written
results.json	Always; the BlopRunResult
events.jsonl	Always; one BlopAgentEvent per line
report.xml	With reporter junit or all
screenshots/	When the check takes screenshots or per-action capture is on
coverage.json	With coverage; the Inspector shows the latest report at /inspector?view=coverage

Reporter	Files
all (default)	results.json, events.jsonl, report.xml
junit	Same as all
json, basic	results.json, events.jsonl

event_type	Tells you
step_start, step_finish	One reasoning step
tool_result	What a browser tool returned
text_delta, text	The agent’s narration

report.xml is one flat <testsuite> with a <testcase> per check. Both failed and error become <failure> with the reason as its body, and time is in seconds.

Use a fresh --report-dir per run, such as .blop/reports/$GITHUB_RUN_ID, because old screenshots stay. Never delete all of .blop/: it can also hold the project’s collector store and configuration. Add .blop/ to .gitignore.

Exported types

Types	Reference
BlopAgent, BlopAgentTest, BlopAgentTestContext, BlopAgentTestHandler, BlopAgentStep	Test DSL
BlopRunOptions, BlopRunResult, BlopTestResult, BlopTestStatus, BlopAction, BlopAgentEvent, BlopCiMetadata, BlopAgentStreamRunner, BlopAgentStreamEvent	This page
BlopConfig, BlopBrowserName, BlopReporter	Configuration

Exported functions include agentTest, describe, defineAgentTest, loadAgentTests, runBlopTest, runBlopTests, runBrowserAgentStream, runNativeAgentStream and loadBlopConfig; BLOP_BROWSER_NAMES lists the browser names.