Investigate

Narrow a failure from the collector’s status to one trace, its logs and the audit findings that cite it.

Prerequisites

• A running collector with telemetry in it. The Quickstart records sample data with blop demo.
• Add --json to any command below when an agent reads the output; How it works describes the document and exit codes.

If queries come back empty, run blop setup doctor --json first. It audits the collector, store, agent recipes and Git state without changing anything, and exits 1 with a fix command for each failing check.

Drill down from status to findings

1. Check the collector and what it stores:

   blop status --json

2. List the services seen in the window:

   blop services --since 1h --json

3. Find the failing traces of one service:

   blop traces --service checkout --status error --since 15m --json

4. Print one trace as a span tree with its correlated logs:

   blop trace <trace-id> --include logs --json

5. Read the logs around it:

   blop logs --trace-id <trace-id> --severity WARN --json

6. Audit the window against a baseline:

   blop audit --service checkout --since 15m --baseline 1h --json

Every query is read-only and bounded by --limit. blop spans and blop span <span-id> search spans directly when a trace is too broad. blop open <trace-id> shows the same trace in the Inspector.

Find the slowest spans

No command sorts by duration, so sort the JSON:

blop spans --since 1h --limit 500 --json | jq -r '.data | sort_by(-.span.duration_ms) | .[:5][] | "\(.span.duration_ms) ms \(.trace_id) \(.span.operation_name)"'

Root spans last as long as their trace, so they lead. The sort sees only --limit spans: if .page.truncated is true, raise it. To rank one span’s children, add --parent-operation <name>.

Fail on a regression

blop audit --service checkout --since 15m --fail-on regression --json

The audit is deterministic and calls no model. Its findings are error_trace, new_error_signature, latency_regression (from a baseline delta or an explicit --duration-min), log_burst, incomplete_trace and no_telemetry (a service that stopped sending). Each one carries trace, span or log IDs, current and baseline values, and follow-up commands. The baseline defaults to the window before --since, of equal length.

--fail-on error exits 1 on error-severity findings; --fail-on regression also exits 1 on new error signatures and latency regressions. The default, never, always exits 0. Missing evidence is not a measured zero: a counter nothing reported is unknown.

Follow issues and live events

The collector groups failures into issues, each open, muted or fixed.

blop issues --json                          # open issues, current branch first
blop issue <issue-id> --fix-context         # Markdown brief to hand to an agent
blop why --last                             # explain the latest failure
blop tail -o ndjson                         # stream new issues, runs and sessions

blop issues takes --state open|muted|fixed|all and --branch <name>. blop tail is a stream, not one JSON document; when no collector runs it waits and reconnects instead of exiting 3.

The Inspector also has an optional chat over the same evidence. It is off until the collector is started with BLOP_AGENT_API_KEY set; queries and audits never need a model.

Filter, aggregate and count

Filter or command	Applies to	Example
--service, --operation	traces, spans, logs (service)	--service checkout
--status ok|error	traces, spans	--status error
--duration-min <ms>	traces, audit	--duration-min 500
--severity <level> (minimum)	logs	--severity WARN
--contains <text>	logs	--contains timeout
--trace-id	spans, logs	--trace-id <trace-id>
--span-id	logs (for one span, blop span <span-id>)	--span-id <span-id>
attr.KEY=VALUE (exact, indexed)	traces, spans	attr.http.route=/api/pay
attrContains.KEY=TEXT (substring)	spans, logs	attrContains.exception.message=timeout
--since <duration> (default 15m)	every query	--since 2h
--limit default and cap	traces 20, at most 100; spans and logs 100, at most 500	--limit 50
--cursor <token>	traces, logs: continue a JSON page	--cursor <next>
blop trace-stats <group> <agg>	traces: count, avg_duration, p95_duration, error_rate	blop trace-stats service error_rate
blop log-stats <group>	log counts by service, severity, scope or attr.KEY	blop log-stats severity
blop facets <traces|logs> <field>	distinct values	blop facets traces attribute_values --key http.route
blop usage	token counts per service, agent or model	blop usage --since all --group-by model

blop usage sums token counters already on spans; input includes cache reads and writes, output includes reasoning, and cost stays unknown.

Control what applications send

Application capture is opt-in per source: without consent the collector keeps metadata but omits log bodies and custom attributes. Credential keys, authorization headers and cookies are redacted before storage; pattern redaction is best effort, not a guarantee.

1. Preview consent for one source; nothing is saved:

   blop sources configure --id checkout-api --kind application --capture-log-body --capture-attributes --enable

2. Apply the same consent:

   blop sources configure --id checkout-api --kind application --capture-log-body --capture-attributes --enable --apply

3. Print the exporter variables for the app:

   blop sources env --id checkout-api --service checkout

   They set blop.source.id and blop.source.kind. Declare these yourself if you configure the exporter by hand: service.name alone does not identify a source.

4. Check intake:

   blop sources check --id checkout-api

   It reports reachability, configuration and accepted counts; applicationOutcome stays unknown.

GenAI prompt and output content needs two opt-ins: --capture-gen-ai-content here and the producer’s own switch; --capture-attributes does not grant it. blop sources disable --id checkout-api stops intake and keeps history, but does not stop the app’s SDK; set OTEL_SDK_DISABLED=true in the app for that.

Keep one collector per project

blop monitor init --port 27812
blop up --detach

blop monitor init writes .blop/monitor.json with a loopback address and a fixed port for this project. Pick a port other than 27811; run blop up from the same directory to start that collector. Without --url or --port, query commands use the running collector whose working directory is the closest parent of yours, and refuse an ambiguous choice.

Delete local evidence

blop delete --scope trace --id <trace-id>          # preview, removes nothing
blop delete --scope trace --id <trace-id> --yes    # delete

Select one exact trace, session or source ID, never a pattern. The preview shows what would be removed and how many bytes; --yes checks the selection again and deletes it all or nothing. One delete removes at most 250,000 rows: a larger scope exits 2 (scope_too_large), so narrow it with --since 1h. Deletion is logical: database pages, WAL files, exports and backups can still hold the data until they are overwritten or expire.

Check it worked

blop audit --fail-on regression --json exits 0 when the window has no error findings or regressions, and 1 with error.code audit_failed when it does. After a fix, reproduce the behaviour and run the same audit again.

Next

• Connect a coding agent: capture the agent’s side of a failure.
• Browser checks: verify a user flow end to end.
• Query command reference.