Skip to content
GitHub
Start a 14-day trial

Find in trace

A checkout trace can run to hundreds of spans. Type what you’re after into Search spans and the trace narrows to the spans that match, with the path from the root down to each one kept, so you see where every match sits in the request.

The search box sits above the span list in every trace you open. As you type, the span list and the waterfall narrow together to the matching spans and their parents. Clear the box, or press Esc inside it, and the whole trace comes back.

Type any part of what you’re looking for — no query syntax, and case doesn’t matter. A span matches when your text appears in:

  • its service — payment;
  • its operation — PlaceOrder;
  • its span ID;
  • one of its attributes — a key, a value, or both as key:value, so app.payment, Invalid token and rpc.method:Charge all work.

Matched text in operation names is highlighted in the span list, and matching attributes are highlighted in the open span’s details.

A failed checkout trace searched for app.payment. The counter beside the search box reads 1 of 1. The span list and waterfall show only the payment service's Charge span, which is selected, and the chain of parents above it, from user_checkout_single through load-generator, frontend-proxy, frontend and checkout, every one but the root outlined red as failing.

Beside the box, a counter reads N of M: M is the number of matching spans in the trace, and N is the one you’re on. Each span counts once, however many of its fields match. Press Enter for the next match and Shift+Enter for the previous one, or use the up and down arrows beside the box; both wrap around at the ends. The current match opens with its details, and any collapsed parents above it expand so it shows in the list. When nothing matches, the counter reads No matches and the whole trace stays in view.

Status — All, Ok, Error or Unset — sits just below the search box and narrows the trace on its own or together with a search. With both set, Status applies to what the search left: you see the spans there that have that status, with their parents — search PaymentService and pick Error to keep the failing calls to the payment service and the failing path above them. The counter still counts every match in the trace, so it can read higher than the matches in view; pick All to bring the rest back.

Or skip the typing. Open a span, expand its Attributes, and each value has a magnifier beside it: Find in this trace. Click it and the search box fills with that attribute as key:value, the trace narrows to every span with an attribute containing it, and you stay on the span you started from — the counter shows where it falls among the matches.

Find in this trace run from the payment span's rpc.service attribute. The search box holds rpc.service:oteldemo.PaymentService and the counter reads 2 of 2. The waterfall is narrowed to the two spans that carry it — checkout's call to the payment service and the payment service's own Charge span — with their parents above. The payment span's details are open below, its Attributes expanded and the rpc.service row highlighted, with a magnifier beside every value.

The match you stop on opens with everything the span recorded. Under its attributes, Events lists what happened inside the span and when; open Events and an exception shows its message, with the stack trace a click away. The footer counts the span’s events and its logs.

The payment span's Events section, expanded: one exception event at +90.955 ms reading Error: Payment request failed. Invalid token. app.loyalty.level=gold, with an exception.stacktrace row collapsed beneath it. The footer below reads OTel, Kind: SPAN_KIND_SERVER, 1 events, 2 logs.

Click the logs count and Logs for this span opens beneath the waterfall with the lines written under that span, newest first, each with its level. Before you pick a span, the same panel at the bottom of the trace reads Logs for this trace and lists the log lines from the whole request.

The Logs for this span panel, open with two rows from the payment service: a WARN line reading Payment request failed. Invalid token. app.loyalty.level=gold, and the INFO line Charge request received. that preceded it, each with its timestamp and span ID.

The Find in this trace magnifier searches this trace; the attribute’s value itself takes you out of it. Click the value and the trace closes, and Traces adds that attribute and value to its current query as a filter and runs it, so you see the other traces with a span that carries it. Hover the value first to see the filter it will add. From an analysis, Find similar traces does the same for a whole pattern.