# Investigate

> Drill from collector status to services, traces, logs and audit findings, follow issues live, and filter or delete local evidence.

Narrow a failure from the collector's status to one trace, its logs and the audit findings that cite it.

## Prerequisites

- A running collector with telemetry in it. The [Quickstart](/quickstart/) records sample data with `blop demo`.
- Add `--json` to any command below when an agent reads the output; [How it works](/how-it-works/#json-output-contract) describes the document and exit codes.

If queries come back empty, run `blop setup doctor --json` first. It audits the collector, store, agent recipes and Git state without changing anything, and exits 1 with a fix command for each failing check.

## Drill down from status to findings

1. Check the collector and what it stores:

   ```bash
   blop status --json
   ```

2. List the services seen in the window:

   ```bash
   blop services --since 1h --json
   ```

3. Find the failing traces of one service:

   ```bash
   blop traces --service checkout --status error --since 15m --json
   ```

4. Print one trace as a span tree with its correlated logs:

   ```bash
   blop trace <trace-id> --include logs --json
   ```

5. Read the logs around it:

   ```bash
   blop logs --trace-id <trace-id> --severity WARN --json
   ```

6. Audit the window against a baseline:

   ```bash
   blop audit --service checkout --since 15m --baseline 1h --json
   ```

Every query is read-only and bounded by `--limit`. `blop spans` and `blop span <span-id>` search spans directly when a trace is too broad. `blop open <trace-id>` shows the same trace in the Inspector.

## Find the slowest spans

No command sorts by duration, so sort the JSON:

```bash
blop spans --since 1h --limit 500 --json | jq -r '.data | sort_by(-.span.duration_ms) | .[:5][] | "\(.span.duration_ms) ms \(.trace_id) \(.span.operation_name)"'
```

Root spans last as long as their trace, so they lead. The sort sees only `--limit` spans: if `.page.truncated` is `true`, raise it. To rank one span's children, add `--parent-operation <name>`.

## Fail on a regression

```bash
blop audit --service checkout --since 15m --fail-on regression --json
```

The audit is deterministic and calls no model. Its findings are `error_trace`, `new_error_signature`, `latency_regression` (from a baseline delta or an explicit `--duration-min`), `log_burst`, `incomplete_trace` and `no_telemetry` (a service that stopped sending). Each one carries trace, span or log IDs, current and baseline values, and follow-up commands. The baseline defaults to the window before `--since`, of equal length.

`--fail-on error` exits 1 on error-severity findings; `--fail-on regression` also exits 1 on new error signatures and latency regressions. The default, `never`, always exits 0. Missing evidence is not a measured zero: a counter nothing reported is `unknown`.

## Follow issues and live events

The collector groups failures into [issues](/reference/glossary/#issue), each open, muted or fixed.

```bash
blop issues --json                          # open issues, current branch first
blop issue <issue-id> --fix-context         # Markdown brief to hand to an agent
blop why --last                             # explain the latest failure
blop tail -o ndjson                         # stream new issues, runs and sessions
```

`blop issues` takes `--state open|muted|fixed|all` and `--branch <name>`. `blop tail` is a stream, not one JSON document; when no collector runs it waits and reconnects instead of exiting 3.

The Inspector also has an optional chat over the same evidence. It is off until the collector is started with `BLOP_AGENT_API_KEY` set; queries and audits never need a model.

## Filter, aggregate and count

| Filter or command | Applies to | Example |
| --- | --- | --- |
| `--service`, `--operation` | traces, spans, logs (service) | `--service checkout` |
| `--status ok\|error` | traces, spans | `--status error` |
| `--duration-min <ms>` | traces, audit | `--duration-min 500` |
| `--severity <level>` (minimum) | logs | `--severity WARN` |
| `--contains <text>` | logs | `--contains timeout` |
| `--trace-id` | spans, logs | `--trace-id <trace-id>` |
| `--span-id` | logs (for one span, `blop span <span-id>`) | `--span-id <span-id>` |
| `attr.KEY=VALUE` (exact, indexed) | traces, spans | `attr.http.route=/api/pay` |
| `attrContains.KEY=TEXT` (substring) | spans, logs | `attrContains.exception.message=timeout` |
| `--since <duration>` (default `15m`) | every query | `--since 2h` |
| `--limit` default and cap | traces 20, at most 100; spans and logs 100, at most 500 | `--limit 50` |
| `--cursor <token>` | traces, logs: continue a JSON page | `--cursor <next>` |
| `blop trace-stats <group> <agg>` | traces: `count`, `avg_duration`, `p95_duration`, `error_rate` | `blop trace-stats service error_rate` |
| `blop log-stats <group>` | log counts by `service`, `severity`, `scope` or `attr.KEY` | `blop log-stats severity` |
| `blop facets <traces\|logs> <field>` | distinct values | `blop facets traces attribute_values --key http.route` |
| `blop usage` | token counts per service, agent or model | `blop usage --since all --group-by model` |

`blop usage` sums token counters already on spans; input includes cache reads and writes, output includes reasoning, and cost stays unknown.

## Control what applications send

Application capture is opt-in per source: without consent the collector keeps metadata but omits log bodies and custom attributes. Credential keys, authorization headers and cookies are redacted before storage; pattern redaction is best effort, not a guarantee.

1. Preview consent for one source; nothing is saved:

   ```bash
   blop sources configure --id checkout-api --kind application --capture-log-body --capture-attributes --enable
   ```

2. Apply the same consent:

   ```bash
   blop sources configure --id checkout-api --kind application --capture-log-body --capture-attributes --enable --apply
   ```

3. Print the exporter variables for the app:

   ```bash
   blop sources env --id checkout-api --service checkout
   ```

   They set `blop.source.id` and `blop.source.kind`. Declare these yourself if you configure the exporter by hand: `service.name` alone does not identify a source.

4. Check intake:

   ```bash
   blop sources check --id checkout-api
   ```

   It reports reachability, configuration and accepted counts; `applicationOutcome` stays `unknown`.

GenAI prompt and output content needs two opt-ins: `--capture-gen-ai-content` here and the producer's own switch; `--capture-attributes` does not grant it. `blop sources disable --id checkout-api` stops intake and keeps history, but does not stop the app's SDK; set `OTEL_SDK_DISABLED=true` in the app for that.

## Keep one collector per project

```bash
blop monitor init --port 27812
blop up --detach
```

`blop monitor init` writes `.blop/monitor.json` with a loopback address and a fixed port for this project. Pick a port other than 27811; run `blop up` from the same directory to start that collector. Without `--url` or `--port`, query commands use the running collector whose working directory is the closest parent of yours, and refuse an ambiguous choice.

## Delete local evidence

```bash
blop delete --scope trace --id <trace-id>          # preview, removes nothing
blop delete --scope trace --id <trace-id> --yes    # delete
```

Select one exact `trace`, `session` or `source` ID, never a pattern. The preview shows what would be removed and how many bytes; `--yes` checks the selection again and deletes it all or nothing. One delete removes at most 250,000 rows: a larger scope exits 2 (`scope_too_large`), so narrow it with `--since 1h`. Deletion is logical: database pages, WAL files, exports and backups can still hold the data until they are overwritten or expire.

## Check it worked

`blop audit --fail-on regression --json` exits 0 when the window has no error findings or regressions, and 1 with `error.code` `audit_failed` when it does. After a fix, reproduce the behaviour and run the same audit again.

## Next

- [Connect a coding agent](/guides/connect-agent/): capture the agent's side of a failure.
- [Browser checks](/checks/): verify a user flow end to end.
- [Query command reference](/reference/cli/#blop-traces).