Skip to content
GitHub
Start a 14-day trial

Diagnose a fault

Break something in the OTel Demo and I find it for you: the request path it hurts is at the top of Critical Paths, one press of Analyze names the service the failure started in, and Copy for agent gives you the whole diagnosis to paste into your coding agent. This page walks three real faults that way, from the symptom you would notice to the cause worth fixing.

  1. Turn the fault on in the demo’s flag page (http://localhost:8080/feature) and leave it running. Each example says how soon to expect it.

  2. Open Critical Paths. I have already ranked the path the fault runs through first. Select it, and the services it crosses are listed above the detail panel; View full topology lights them on the map and dims the rest.

  3. Press Analyze on that path. I read a handful of its example traces, the failing ones first, and lay out where their errors went.

  4. Read the Root Cause card. It names the service the failure started in, what went wrong there and how confident I am, with the evidence under it.

  5. Press Copy for agent on the analysis panel’s Summary card — the one that opens after Analyze — and paste the analysis into Claude Code, Cursor or any coding agent.

The same fault usually opens an incident in Incidents too, so it is waiting for you there even when you aren’t on Critical Paths.

When the symptom isn’t the cause — recommendationCacheFailure

Section titled “When the symptom isn’t the cause — recommendationCacheFailure”

The symptom. Recommendations fail to load for a share of storefront requests, and the errors come back from product-catalog: it is answering requests with Product Not Found. Expect it on Critical Paths and in Incidents usually within about 7 minutes.

Critical Paths. A path through recommendation is ranked first, and recommendation is the highlighted hop in the selected path. The card below it, a shorter path through recommendation, shows errors on its recommendation → product-catalog hop.

Critical Paths with the End-to-End Browse and Checkout Journey ranked first at critical severity, 9 services in the path and 25.3% errors. In the selected-path strip, recommendation is the highlighted hop after load-generator, frontend-proxy, frontend, ad, cart, checkout, product-catalog and product-reviews. The second path, Feature-Flagged Recommendations, is also critical and reads: also affects frontend → recommendation → product-catalog → postgresql (own fault: recommendation → product-catalog 3.8%). The trace list on the right shows the path's ListRecommendations requests, several of them marked Error.

Analyze and the Root Cause card. Press Analyze on that path and I name recommendation. Product-catalog refused the request because of what recommendation sent it — a product lookup with no product id in it — so the fault is recommendation’s, not product-catalog’s. The evidence points at the file and line of the call that sent the empty request.

The Root Cause card at 85% confidence names recommendation: its get_product_list is the fault origin; product-catalog rejected recommendation's GetProduct call with NOT_FOUND, a caller-fault status, so the fault is recommendation's, not product-catalog's, and 3 of 3 spans errored. The Evidence list is open: get_product_list is the only originating error span, the NOT_FOUND exception raised on recommendation, and the stack trace showing the failing call in recommendation_server.py passing an empty request to GetProduct.

Product-catalog is healthy; the broken code is in recommendation, so that is where your fix goes.

Copy for agent on the Summary card gives your coding agent the whole analysis, including that stack-trace evidence. In Incidents, the same fault opens an incident on recommendation.

A failure that surfaces in a caller — paymentFailure

Section titled “A failure that surfaces in a caller — paymentFailure”

The symptom. Some or all checkouts fail, depending on the percentage you pick for the flag. The shopper sees an error from the checkout page, and checkout, the frontend and the proxy in front of it all report the request as failed. Expect it on Critical Paths and in Incidents usually within about 8 minutes.

Critical Paths. The checkout payment path is ranked first, and payment is the highlighted hop in its selected path. Its trace list is the payment service’s Charge calls, with the failed ones marked.

Critical Paths with Checkout Payment Processing ranked first at medium severity, 4 services in the path and 12.5% errors. In the selected-path strip, payment is the highlighted hop between checkout and flagd. The trace list on the right shows the payment service's Charge calls, several of them marked Error.

Analyze and the Root Cause card. The card names payment: its Charge handler rejected the charge and logged why, and every other service in the chain only passed that error back up. The evidence lists the exception payment raised, the warning it logged, and checkout reporting that it could not charge the card.

The Root Cause card at 85% confidence names payment: the payment service's Charge handler originated the failure in all 3 error traces, logging a warning and throwing an error that the payment request failed because of an invalid token; checkout's PlaceOrder call to Charge therefore failed and checkout surfaced 'could not charge the card' up the call chain. The Evidence list is open: the payment span is the only operation where the error originated rather than propagated, the exception thrown on payment, the warning log, checkout's report, and the frontend and proxy spans erroring with the same message.

I separate the service that caused the failure from the ones that only passed it along, so you start in payment rather than in checkout, where the error first became visible.

Copy for agent sits on the Summary card, beside Find similar traces. It gives your coding agent the failing handler, the error it raised and the chain it travelled.

The Summary card of the analysis, with Find similar traces and Copy for agent in its header. It reads: checkout requests across three traces failed because the payment service's Charge call rejected the card with an invalid-token error, and the error propagated up through checkout, frontend, frontend-proxy and load-generator. Tags show high severity, 124 spans, 11 services and 1.6s duration.

In Incidents, the same fault opens an incident on payment.

One operation out of three — cartFailure

Section titled “One operation out of three — cartFailure”

The symptom. Errors show up in checkout, on its call to the cart service once an order is placed. Shoppers can still add items and view their cart (AddItem and GetCart keep working); what fails is emptying it (EmptyCart). Expect it on Critical Paths and in Incidents usually within about 9 minutes.

Critical Paths. The checkout path that goes through the cart is ranked first, at critical severity, with cart the highlighted hop.

Critical Paths with Checkout with Cart Validation ranked first at critical severity, 4 services in the path and 26% errors. In the selected-path strip, cart is the highlighted hop between checkout and the cart's Redis store.

Analyze and the Root Cause card. The card names cart, and narrows it to one operation: the EmptyCart handler failed in every trace, unable to reach its cart storage. Checkout’s call to EmptyCart errored only because cart’s handler did, so checkout is not where it started.

The Root Cause card at 90% confidence names cart: its EmptyCart handler failed in 3 of 3 traces with a 'Can't access cart storage' error after it could not connect to Redis, and checkout is a propagation point rather than the origin. The Evidence list is open beneath it.

Because I narrow the fault to EmptyCart, your fix goes to that handler, not to the whole cart service.

Copy for agent on the Summary card hands your coding agent the failing operation and the error behind it. In Incidents, the same fault opens an incident on cart.

The same route works on any path in your own workspace, the moment your services send OpenTelemetry.