Browser checks in CI

Run browser checks on every pull request, keep their evidence, and send test results to Blop Cloud.

Prerequisites

@blopai/cli and @blopai/test committed as dev dependencies with their lockfile (Browser checks).
A preview or staging URL for the app, stored as the repository variable APP_URL.
The provider key stored as the repository secret BLOP_AGENT_API_KEY.
Node.js 22.18 or newer; the workflow below uses the latest Node.js 22.
To upload results to Blop Cloud: the ingest secret from your project’s Settings → Hosted evidence ingest, the project ID, and your Blop Cloud address followed by /api/ingest. Store them as BLOP_INGEST_SECRET (secret), BLOP_PROJECT_ID and BLOP_INGEST_URL (variables).

Run browser checks in GitHub Actions

Add this workflow:

name: Browser checks
on:
  pull_request:
  push:
    branches: [main]

permissions:
  contents: read
  checks: write

jobs:
  checks:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: pnpm/action-setup@v4
        with:
          version: 9
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: pnpm install --frozen-lockfile
      - run: pnpm dlx playwright@1.61.1 install --with-deps chromium
      - name: Run browser checks
        run: pnpm exec blop test run --base-url ${{ vars.APP_URL }} --max-steps 75
        env:
          BLOP_AGENT_PROVIDER: openai
          BLOP_AGENT_MODEL: gpt-5
          BLOP_AGENT_API_KEY: ${{ secrets.BLOP_AGENT_API_KEY }}
      - name: Upload reports
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: blop-results
          path: .blop/
      - name: Publish JUnit report
        if: always()
        uses: dorny/test-reporter@v1
        with:
          name: Browser checks
          path: .blop/report.xml
          reporter: java-junit

Push a branch and open a pull request.

dorny/test-reporter needs the checks: write permission. Pull requests from forks get no secrets or variables, so checks there cannot reach the provider.

The step fails when any check fails or errors, so branch protection needs no extra wiring. if: always() keeps the reports when checks fail, which is when you need them. Install Chromium on every fresh runner, keep keys in secrets, and pin the model: a new model version can change what the agent does. Seed isolated test data before the checks start.

On GitHub Actions, each result in results.json carries CI metadata: branch, commit, pull request, workflow and run URL (BlopCiMetadata). Other CI systems get null for these fields.

Other CI systems read the same JUnit file: GitLab with artifacts.reports.junit: .blop/report.xml, CircleCI with store_test_results on .blop, Buildkite with buildkite-agent artifact upload .blop/**/*.

Shard browser checks and control cost

strategy:
  matrix:
    shard: [1, 2, 3, 4]
steps:
  - run: pnpm exec blop test run "tests/shard-${{ matrix.shard }}/**/*.blop.ts" --report-dir .blop/${{ matrix.shard }}

--workers <n> runs up to n checks at once in one job; shards give separate failures and artifacts. Give each shard its own report directory and merge the JUnit files afterwards. A pass on retry is flaky, not a pass; see retries. --coverage adds coverage.json to the reports but is not a CI gate.

--max-steps caps agent tool steps, not money, and does not count retries or resumes. There is no default cap. Set one that is tight but realistic, run the full suite on main and a smoke subset on pull requests, and cache ~/.cache/ms-playwright.

Export browser check telemetry with OpenTelemetry

      - run: pnpm exec blop test run --base-url ${{ vars.APP_URL }}
        env:
          OTEL_EXPORTER_OTLP_ENDPOINT: http://collector.internal:4318
          OTEL_SERVICE_NAME: blop-runner

The CLI exports straight from the job to your OTLP/HTTP collector, and nothing goes to Blop Cloud. It supports http/protobuf (the default) and http/json, not gRPC. Without an endpoint nothing is exported. Export failures are logged and never fail the check, and shutdown waits at most five seconds for an unreachable collector. All variables are in Environment.

Each check execution is one trace: a run span, a scenario span per check, and a step span per tool call. Spans carry the OpenTelemetry test, CI/CD and VCS attributes, plus blop.journey.id (the root describe), blop.scenario.path, blop.failure.category and blop.step.tool. They never carry screenshots, DOM snapshots, typed text or tool output, and URL credentials and token query parameters are redacted.

Three metrics, without execution or scenario IDs to keep cardinality low: blop.scenario.duration (histogram, seconds, by journey, status and failure category), blop.agent.recoveries (counter, by journey and resume or retry) and blop.agent.tokens (counter, by journey, kind and team). Tokens are not a bill. The events qa.run.started.v1, qa.run.step.finished.v1 and qa.run.finished.v1 are also sent as log records in the trace.

To link a failed step to the backend span behind it, send W3C traceparent to the app:

        env:
          OTEL_EXPORTER_OTLP_ENDPOINT: http://collector.internal:4318
          BLOP_OTEL_PROPAGATE_TO_APP: "true"
          BLOP_OTEL_PROPAGATE_ALLOWLIST: staging.example.com

Propagation is off by default and has no wildcard: only listed hosts and their subdomains get the header. Use non-production targets; a production target needs the production safeguards. Service worker requests carry no header, and with Camoufox the header is a fingerprinting signal, which Blop warns about.

Upload browser check results to Blop Cloud

        env:
          BLOP_INGEST_URL: ${{ vars.BLOP_INGEST_URL }}
          BLOP_INGEST_SECRET: ${{ secrets.BLOP_INGEST_SECRET }}
          BLOP_PROJECT_ID: ${{ vars.BLOP_PROJECT_ID }}

With all three set on the blop test step, each execution becomes a check run in your Blop Cloud project, with a zipped bundle of results.json, events.jsonl and screenshots. Upload failures do not fail the step.

Upload Vitest results to Blop Cloud

Install the reporter:
Terminal window
```
pnpm add -D @blopai/ingest
```

Add it to vitest.config.ts:

import { defineConfig } from "vitest/config";

export default defineConfig({
  test: {
    reporters: ["default", "@blopai/ingest/vitest"],
  },
});

Set the three ingest variables on the CI step that runs Vitest:

      - run: pnpm test
        env:
          BLOP_INGEST_URL: ${{ vars.BLOP_INGEST_URL }}
          BLOP_INGEST_SECRET: ${{ secrets.BLOP_INGEST_SECRET }}
          BLOP_PROJECT_ID: ${{ vars.BLOP_PROJECT_ID }}

The reporter sends counts, up to 25 failures (2,000 characters each), CI metadata and, when artifact storage is configured, vitest-report.json. If any of the three variables is unset, it does nothing and never fails Vitest. In watch mode, each rerun is uploaded as a separate check run in Blop Cloud unless BLOP_RUN_ID is set. Optional: BLOP_RUN_ID (generated if unset) and BLOP_TRIGGER (ci, manual, schedule or healer_verify).

Upload JUnit XML from any framework to Blop Cloud

Make your framework write JUnit XML, for example pytest --junitxml=report.xml, gotestsum --junitfile report.xml -- ./... or phpunit --log-junit report.xml.

Add an upload step after the tests:

      - name: Upload to Blop
        if: always()
        run: npx @blopai/ingest upload --junit report.xml
        env:
          BLOP_INGEST_URL: ${{ vars.BLOP_INGEST_URL }}
          BLOP_INGEST_SECRET: ${{ secrets.BLOP_INGEST_SECRET }}
          BLOP_PROJECT_ID: ${{ vars.BLOP_PROJECT_ID }}

Without the three variables, or --ingest-url, --secret and --project-id, the upload exits 1. --junit takes a glob and repeats; --artifact <file> uploads extra files, not directories. BLOP_TRIGGER defaults to ci in CI and manual elsewhere. Run npx @blopai/ingest upload --help for every flag.

Each test gets a stable identity from suite, class and name; a test that failed then passed on rerun is recorded as flaky; failures with the same signature are grouped. Results are filed under their branch, so a feature branch never affects the default branch’s pass rate.

Check it worked

The pull request shows a Browser checks status and a JUnit summary, and the blop-results artifact holds results.json, events.jsonl, report.xml and screenshots.