Reverify a fix with a browser check

Have a coding agent write a browser check for a reported problem, read the evidence, and rerun the check after the fix.

Prerequisites

@blopai/cli, @blopai/test, Chromium and a provider set up as in Browser checks.
The app running with its seed data loaded.

Write a browser check with explicit assertions

Start from a reported user journey, not “test the site”. State the starting state, allowed actions, expected outcomes and forbidden side effects.
Ask for assertion tools by name. browser_expect_url, browser_expect_text, browser_expect_visible and browser_expect_count check real browser state; a click or the agent’s own summary does not.
Ask for named screenshots at checkpoints, and record_critical_point for each stage of a long flow. A screenshot explains what happened; it does not replace an assertion.
Commit the check with the change so another agent can rerun it.

import { agentTest, describe } from "@blopai/test";

describe("catalog", () => {
  agentTest("opens products from the homepage", async ({ agent }) => {
    await agent.goto("/");
    await agent.goal(`
      Find and click the Products navigation link.
      Use browser_expect_url to assert the pathname is /products.
      Use browser_expect_text to assert the Products heading is present.
      Use browser_expect_count on product cards to assert at least one exists.
      Take an explicit screenshot named products-listing as evidence.
      Do not add items to a cart, buy anything, or create an account.
      Finish as passed only if every assertion succeeds; otherwise finish
      as failed and name the unmet outcome and observed evidence.
    `);
  });
});

The blop-test repository has more real checks. elusive-signup.blop.ts creates an account on a live site: do not run it just to try the package.

Run the browser check and read the evidence

Run the check into its own report directory:

pnpm exec blop test run tests/catalog.blop.ts --base-url http://localhost:3000 \
  --max-steps 50 --report-dir .blop/catalog-before

Open results.json for each check’s status, reason, critical points, attempts, actions and artifact paths.
Read events.jsonl for the tool calls and the feedback the agent got.
Open the screenshots it references; the last frame does not prove the whole journey.

Add --verbose to stream agent events, or --headed to watch the browser. A separate report directory keeps old artifacts from passing as current evidence.

failed: the agent finished with a goal unmet. Ambiguous goals, missing seed data or a wrong target can cause this too, so read the assertions before blaming the app.
error: the check could not finish, for example a browser crash, a timeout or a provider authentication failure. Fix the cause and rerun; it says nothing about the feature.

Check reports, local traces and Blop Cloud results are separate stores, with no automatic link between a check and a trace. Query related app telemetry with the investigate commands.

Launch a browser check from a trace or finding

pnpm exec blop test run tests/checkout.blop.ts --from trace 4bf92f3577b34da6a3ce929d0e0e4736 --json
pnpm exec blop test run tests/checkout.blop.ts --from finding app_error --since 1h --json

Take the trace ID from blop traces or the detector code from blop issues. --from trace <id> reads that exact trace; --from finding <code> uses the latest recorded occurrence of that code, and never falls back to another finding or nearby telemetry.

Bounds:

Blop reads the local database (--db PATH, BLOP_MONITOR_DB_PATH or project configuration). It makes no collector request, cloud lookup or model call.
The lookback defaults to 15 minutes, is at most 24 hours, and never exceeds configured retention. --db and --since require --from.
Context is capped at 500 spans, 500 logs, 1 MiB in total and 1,024 bytes per displayed field.
Missing, stale, deleted and over-cap evidence exits 2 before any check loads. Blop never launches from partial or guessed context.

Provenance:

data.evidence in the JSON output holds the recorded service, the sanitized recorded base_url, the failing operation and the error signature, with null for absent fields.
The recorded URL is context only. The target still comes from the reviewed check, blop.config.ts or --base-url.
Each check’s spans get blop.evidence.source, blop.evidence.trace_id and, for findings, blop.evidence.finding_code; without --from there is no evidence metadata. They say why the check started, not that the trace caused anything or is fixed. Exporting them still needs an OTLP endpoint; see OpenTelemetry export.
Evidence never generates a check. Without a pattern, normal discovery applies.

Rerun the browser check after the fix

Rerun the exact check with the same target, seed data, model and step cap into .blop/catalog-after.
Compare its assertions and evidence with the first report.
Run the neighbouring checks for the same feature.
Report the command, target, outcome and evidence. A passing list, written code or an error result is not success.

Retries and flaky browser checks

retries in blop.config.ts defaults to 0. A retry is a new attempt; an agent resume within the same browser is recovery, not a retry. Check attempts, firstAttemptStatus and resumes in results.json: a pass after a failed first attempt is flaky, not a pass. Keep the failed evidence and fix unstable data, timing or goals instead of raising retries or weakening the acceptance criterion.

Chromium JavaScript coverage

blop up --detach --no-open
pnpm exec blop test run tests/catalog.blop.ts --base-url http://localhost:3000 \
  --coverage --coverage-endpoint http://127.0.0.1:27811

--coverage works in Chromium only and writes coverage.json to the report directory. --coverage-endpoint is optional and takes the collector’s base URL, not the app’s or the OTLP URL; use the address blop up prints. Blop posts the report to /api/test-coverage but does not start the collector. Open /inspector?view=coverage for the latest report by file, with annotated source and warnings. It is stored locally, apart from traces, with no history and no link to a trace.

What it measures: executed UTF-16 source characters of named scripts loaded in the collected pages, merged across attempts. Not statement, branch or backend coverage, not source-mapped TypeScript, not unloaded files. No collected scripts means unavailable, not 100%. Limits: 200 script versions, 1 Mi characters per source, 20,000 ranges per script, 100,000 ranges in total and 5 MiB of file evidence (8 MiB per report). Oversized sources are dropped with a warning; popups, closed pages and missing source also leave gaps. Coverage is not a pass condition. Reports contain your JavaScript source and URLs: review privacy before you share them.