SIPI

Methodology / 05

SI/PI Debugging: From Symptom to Cause

By Geetansh Arora — Principal SI/PI Engineer9 min

Every SI symptom is consistent with several causes, so the useful move is never to guess better — it is to pick the measurement whose outcome differs between the candidates. This page is a set of hypotheses and the experiments that separate them, not a list of answers.

Four questions that eliminate most of the table
narrow the space before you reach for the symptom table One instance, or all of them? oneall a physical defect Data dependent? yesno ISI · crosstalk · SSN Moves with corner? yesno timing drift · training Eye closed? yesno channel problem tails — noise change ONE thing changing three at once is almost always slower: if it improves you have learned nothing about which change did it and keep the log — the log is what makes the next occurrence take an hour instead of a week
Most high-speed debugging is pattern matching, and the table on this page is the shortcut. But before reaching for it, four questions eliminate most of the possibilities, and they are ordered by how cheap they are to answer. One instance or all of them? One board or one lane means a defect; every board means a design margin problem — and these two have almost no overlap in cause. Is it data dependent? If a pattern change moves it, the mechanism is ISI, crosstalk or SSN. Does it move with corner? Voltage and temperature dependence points at timing, drift and training. Is the eye closed, or clean with occasional errors? A visibly closed eye is a channel problem. A clean eye with rare errors is a tail problem — noise, supply, or something rare — and no amount of channel work will touch it. The strongest temptation in a bring-up lab is to change three things at once because the schedule is tight. It is almost always slower, and you carry two modifications of unknown value into production.

When a brand-new prototype board arrives in the bring-up lab and refuses to work — an LPDDR5X bus fails training, or a PCIe Gen 5 link drops packets every few minutes — the immediate human impulse is panic. Under intense schedule pressure, engineers instinctively reach for the soldering iron: tacking extra decoupling capacitors onto voltage rails, tweaking transmitter drive strength registers, changing termination resistors, and cutting traces with an X-Acto knife.

This chaotic approach, known on the bench as “shotgun debugging,” almost never works. If you change three physical variables simultaneously and the board starts working, you have not solved the problem — you have merely masked it. You now have no idea which change fixed the fault, and you have introduced uncharacterized modifications that will inevitably cause intermittent failures across high-volume production.

Debugging high-speed signal and power integrity is not an exercise in trial-and-error; it is an exercise in ruthless physical isolation. A symptom on an oscilloscope (such as a closed eye or excessive jitter) is rarely unique — it can be caused by excessive dielectric loss, via stub reflections, power supply ground bounce, or simply an inductive oscilloscope ground lead. The professional debug process centers on designing discriminating experiments that bisect the failure space, eliminating entire categories of physics before a single component is touched.

Why a symptom is not a diagnosis

A closed eye is consistent with loss, reflections, crosstalk, supply noise and a bad measurement. So is jitter. So is a bit error rate that is worse than predicted. Almost nothing you observe points at one cause.

That means the skill being exercised is not pattern-matching a symptom to a fix — it is designing a discriminating experiment. A measurement that would look the same whichever hypothesis is true has told you nothing, however much effort it took.

So the structure below is deliberately not symptom → cause. It is symptom → leading hypothesis → the experiment that would rule it out. The third column is the one that matters.

The table

These are hypotheses, not diagnoses Every row names the most likely cause and an experiment that would separate it from the alternatives. That second column is the one that matters: a symptom is consistent with several mechanisms, and the value of the table is in narrowing quickly, not in being right immediately. A measurement that would look the same whichever hypothesis is true has not told you anything — pick the one whose outcome differs between them.
SymptomLeading hypothesisExperiment that discriminates it
Fails at fast-strong corner, passes at typicalreflections — lower Rs and faster edge togetherTDR the net; sweep driver strength
Periodic jitter at a specific frequencyPDN resonance, or a coupled tonePDN impedance plot; look for a peak at that frequency
Fails only with neighbours activecrosstalkrun the aggressors quiet, then at worst-case pattern
Fails only on specific data patternsISI, or a termination that is pattern-dependentcompare isolated bit vs long run amplitude
Deep narrow notch in S21via stub or cavity resonancecompute λ/4 for the stub; check plane modes
Eye fine, occasional bit errorsSSN, supply droop, or a rare patternscope the rail during the failing traffic
Fails after the board warms uptiming drift, or training that has not repeatedretrain at temperature; check corner assumptions
One lane of many failsa physical defect on that lane — via, escape, solderTDR that lane against a working one
Failure moves when you probe itthe probe, not the boardshorten the ground lead; probe elsewhere
Emissions peak with no matching clock harmonicplane cavity mode or common-mode on a cablecompute plane modes; measure SCD21

The ordering rule

Before reaching for the table, narrow the space. Each of these questions eliminates most of the remaining possibilities, and they are ordered by how cheap they are to answer:

  1. Is it one instance or all of them? One board or one lane means a defect; every board means a design margin problem. These have almost no overlap in cause.
  2. Is it data-dependent? If a pattern change moves it, the mechanism is ISI, crosstalk or SSN. If it does not, look at noise, supply and clocking.
  3. Does it move with corner? Voltage and temperature dependence points at timing, drift and training. Independence points at the channel.
  4. Is it in the eye or outside it? A visibly closed eye is a channel problem. A clean eye with occasional errors is a tail problem — noise, supply, or a rare event — and no amount of channel work will touch it.
Change one thing The strongest temptation in a bring-up lab is to change three things at once because the schedule is tight. It is almost always slower: if the symptom improves you have learned nothing about which change did it, and you now carry two modifications of unknown value into production. Change one thing, record what happened, and keep the log — the log is what makes the next occurrence take an hour instead of a week.

Why this decides how your board behaves

What to do about it

Write the hypotheses down before measuring. Three or four, ranked. It takes two minutes and it prevents the commonest failure mode, which is pursuing the first idea until it is disproven rather than choosing between ideas.

Choose the experiment by how much it separates, not by how easy it is. A difficult measurement that eliminates two hypotheses beats an easy one that eliminates none.

Change one thing. Always.

Prove the measurement before trusting it. Probe a known-good net. Check that removing the probe changes the failure. Verify the instrument against a reference.

The ordering rule Work from cheap and broad to expensive and narrow. In practice that means: confirm the symptom is real and reproducible; rule out the measurement; separate channel from supply; separate this net from its neighbours; and only then start characterising structures. Every one of those steps eliminates a class of cause, and each is cheaper than the one after it.
Go deeper — the discriminators worth memorising

A handful of experiments separate most of the common confusions, and they are worth knowing by reflex:

  • Turn the neighbours off. Separates crosstalk from everything else, conclusively, in one measurement.
  • Change the data pattern. Separates ISI and data-dependent jitter (which track the pattern) from noise and supply effects (which do not).
  • Change the workload without changing the link. Separates supply-induced effects from channel effects.
  • Measure how long, not how big. On a PDN, the droop's duration names the responsible stage. See the three droops.
  • Capture for longer. Separates random jitter (keeps growing) from deterministic (plateaus).
  • Compare the two walls of a bathtub. Unequal slopes mean asymmetric jitter — duty-cycle distortion or a threshold offset.
  • Look at ripple period in S21. Δf = 1/(2Td) gives the distance between the two reflecting structures, which usually identifies them.

What these have in common is that each one's outcome differs between the hypotheses it is meant to separate. That is the property to look for when inventing a new one: before running a measurement, ask what each candidate cause predicts it will show. If they all predict the same thing, do something else.

In the real world

The most expensive debugging mistake is not a wrong conclusion — it is a right conclusion reached without evidence, which then cannot be defended when it matters. If the mechanism behind a successful fix is not established, the next similar failure may require the investigation to start again.

So the deliverable from a debug is not the fix. It is the evidence chain: what was observed, which hypotheses it was consistent with, which experiment eliminated which, and what remained. That is what makes the conclusion survive a review.

Related