Active checks in CI

Blop is designed CI-first. The default reporter writes JUnit XML and a JSON result that any pipeline can pick up.

GitHub Actions (recommended)

.github/workflows/blop.yml

name: Blop E2E
on:
  pull_request:
  push:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: oven-sh/setup-bun@v1

      - run: bun install

      - run: bunx playwright install --with-deps chromium

      - name: Run Blop
        run: bunx @blopai/cli test --base-url ${{ vars.APP_URL }} --report-dir .blop --reporter all
        env:
          BLOP_AGENT_PROVIDER: openai
          BLOP_AGENT_MODEL: gpt-5
          BLOP_AGENT_API_KEY: ${{ secrets.BLOP_AGENT_API_KEY }}

      - name: Upload artifacts
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: blop-results
          path: .blop/

      - name: Publish JUnit report
        if: always()
        uses: dorny/test-reporter@v1
        with:
          name: Blop tests
          path: .blop/report.xml
          reporter: java-junit

Key points:

• Install browsers explicitly. bunx playwright install is required on fresh runners.
• Use repository secrets for API keys. Never inline them.
• if: always() on artifact upload. You want the bundle even when tests fail; that’s when it matters most.
• Pin the model. Floating model versions can change agent behavior between runs.

CI metadata captured automatically

When running on GitHub Actions, Blop populates BlopCiMetadata on every test result:

{
  "provider": "github-actions",
  "runId": "1234567890",
  "jobId": "test",
  "branch": "feat/checkout-flow",
  "commitSha": "deadbeef...",
  "pullRequest": "refs/pull/42/merge",
  "pullRequestNumber": "42",
  "repositoryUrl": "https://github.com/acme/storefront",
  "workflowName": "QA",
  "runAttempt": "1",
  "runUrl": "https://github.com/acme/storefront/actions/runs/1234567890",
  "refType": "branch"
}

This is included in results.json, uploaded to Blop Platform if configured, and mapped onto the cicd.* and vcs.* OpenTelemetry conventions when trace export is on.

OpenTelemetry

If you already run an OpenTelemetry collector, point Blop at it and test runs show up beside everything else you monitor. Nothing leaves your network: the CLI exports straight from your CI job to your collector.

      - run: npx @blopai/cli test
        env:
          OTEL_EXPORTER_OTLP_ENDPOINT: http://collector.internal:4318
          OTEL_SERVICE_NAME: blop-runner

That is the whole setup. Blop reads the standard OTLP/HTTP variables. It supports http/protobuf (the default) and http/json; it doesn’t support OTLP/gRPC. With no signal endpoint set, the OpenTelemetry SDK is never loaded.

Each run is one trace, nested to the step:

run  (INTERNAL)                    blop run <suite>
└── scenario  (INTERNAL)           test.case.name
    ├── step  (INTERNAL)           browser_goto / browser_click / browser_expect_*
    └── step  (INTERNAL)

Spans carry the standard test, CI/CD and VCS attributes, plus a blop.* namespace for blop.journey.id (the root describe block), blop.scenario.path, blop.failure.category and blop.step.tool.

Spans carry pointers, never payloads. Screenshots, DOM snapshots, typed text and tool output are deliberately excluded, since tool inputs can hold credentials, and URL credentials plus token-bearing query parameters are redacted.

Linking a failed step to the backend span that caused it

This is the part worth setting up. With propagation on, Blop injects a W3C traceparent into requests the browser makes, so an OpenTelemetry-instrumented app under test parents its own spans under the Blop step span. A failing checkout step then expands into the actual 500 from your payments service, in one trace, with no correlation work.

      - run: npx @blopai/cli test
        env:
          OTEL_EXPORTER_OTLP_ENDPOINT: http://collector.internal:4318
          BLOP_OTEL_PROPAGATE_TO_APP: "true"
          BLOP_OTEL_PROPAGATE_ALLOWLIST: staging.example.com

It is default-deny and there is no wildcard. Only the hosts you list, and their subdomains, ever receive trace context; everything else is not intercepted at all. Point it at non-production targets by default. A production target is a deliberate synthetic check and requires the safeguards in Production-target browser checks. Two caveats: requests made by a ServiceWorker bypass interception and so carry no header, and with the Camoufox browser the extra header is a fingerprinting signal, which Blop warns about rather than silently overriding your choice.

Metrics

Three instruments, deliberately free of run and scenario ids so they cannot inflate your cardinality:

Instrument	Type	Unit	Dimensions
blop.scenario.duration	histogram	s	journey, result status, failure category
blop.agent.recoveries	counter	{recovery}	journey, kind (resume or retry)
blop.agent.tokens	counter	{token}	journey, kind, team

There is no separate “scenario results” counter because the histogram’s own count already provides it per journey and status (blop_scenario_duration_seconds_count in Prometheus).

blop.agent.tokens counts tokens, not currency. Converting to spend needs a price table Blop does not have.

Events

The existing CloudEvent taxonomy is mirrored onto OTel log records carrying trace context, so qa.run.started.v1, qa.run.step.finished.v1 and qa.run.finished.v1 land in your log backend correlated to the run trace. This is a second transport for the taxonomy the platform already speaks, not a new one.

When Blop Platform upload is configured, the same run context is also attached to the CloudEvents sent to /api/ingest. The ingest service extracts that context and traces authentication, rate limiting, deduplication, persistence, and domain processing under the runner trace.

Export failures are logged and never fail a run, and shutdown will not wait more than five seconds in total for an unreachable collector.

Other CI providers

The same shape works on any provider that supports JUnit ingestion. Fill in the equivalent steps:

• GitLab CI → artifacts.reports.junit: .blop/report.xml
• CircleCI → store_test_results: path: .blop
• Buildkite → buildkite-agent artifact upload .blop/**/*

CI metadata auto-detection currently covers GitHub Actions only; on other providers BlopCiMetadata.provider will be null.

Parallelization

Use --workers <n> for bounded concurrency within one process. If you need separate failure domains or independently retained artifacts, shard at the workflow level instead:

strategy:
  matrix:
    shard: [1, 2, 3, 4]
steps:
  - run: bunx @blopai/cli test "tests/shard-${{ matrix.shard }}/**/*.blop.ts" --report-dir .blop/${{ matrix.shard }}

Pair with actions/upload-artifact and merge JUnit reports downstream.

Cost control

Each test costs up to maxSteps model calls. To keep CI bills sane:

• Pin a --max-steps budget that’s tight but realistic.
• Run the full suite on main only; on PRs, run a smoke subset.
• Cache ~/.cache/ms-playwright to skip re-downloading browsers.

Uploading runs to Blop Platform

Blop uploads results to the platform using CloudEvents (qa.run.started.v1 + qa.run.finished.v1) via the @blopai/ingest client. Set:

env:
  BLOP_INGEST_URL: ${{ vars.BLOP_INGEST_URL }}
  BLOP_INGEST_SECRET: ${{ secrets.BLOP_INGEST_SECRET }}
  BLOP_PROJECT_ID: ${{ vars.BLOP_PROJECT_ID }}

The runner emits started + finished events automatically when BLOP_INGEST_URL, BLOP_INGEST_SECRET, and BLOP_PROJECT_ID are present, and uploads a zipped report bundle (results.json, events.jsonl, screenshots) as an artifact. CI metadata (branch, commit, run URL) is attached automatically on GitHub Actions.

This is the same wire protocol used by every other test runner adapter — see Vitest integration and any framework via JUnit XML.

Failing the build

blop test exits non-zero if any test failed or errored. No extra wiring needed — let the step fail and your branch protection / required checks kick in.