Transmitter Compliance Masks and Receiver Eye Masks

When an engineering team prepares a high-speed serial product for commercial release — whether it is a PCIe add-in card, an enterprise server backplane, or a consumer USB-C peripheral — it must pass formal industry compliance testing to earn certification. The centerpiece of physical-layer certification is the eye diagram compliance mask: a standardized geometric keep-out polygon drawn directly over the center of the eye. If even a single measured waveform trace or statistical contour penetrates this boundary, the link fails compliance and cannot be shipped.

At first glance, placing a polygon in the center of an oscilloscope screen looks like a simple pass/fail visual inspection. In reality, an eye mask is a legally binding engineering contract between independent silicon vendors who will never meet. The transmitter vendor guarantees that their launched waveform will never encroach on the TX mask; the receiver vendor guarantees that their equalizer and clock recovery can reliably decode any incoming signal that clears the RX mask. The space between these two boundaries is the allowable channel loss, reflection, and crosstalk budget owned by the PCB system designer.

However, evaluating an eye mask on a modern real-time oscilloscope requires looking beyond the raw display. Because oscilloscopes do not have physical access to the transmitter's internal clock, they reconstruct timing using software-emulated Clock and Data Recovery (CDR) loops, apply mathematical reference equalizers, and extrapolate millions of captured edges into extreme Bit Error Rate (BER) contours. Understanding how these digital post-processing steps create the eye is essential to passing compliance on the first revision.

Why not just two numbers?

A receiver needs two things at once: enough voltage to resolve the level, and enough time to sample it. The obvious specification is therefore two numbers — a minimum eye height and a minimum eye width.

That does not work, because the two trade against each other. A signal with generous amplitude can tolerate worse timing; one arriving exactly on time can tolerate less voltage. Specifying them independently is either too lenient — passing a signal that is marginal in both at once — or too strict, failing signals that are genuinely fine.

A polygon captures the trade directly. Its width at the vertical centre is the timing requirement, its height at the horizontal centre is the amplitude requirement, and its sloped corners encode the acceptable exchange between them. One shape, one test, pass or fail.

Why a polygon

A receiver needs enough voltage to resolve a level and enough time to sample it, and the two trade against each other: a bigger signal can tolerate worse timing and vice versa. Specifying them separately as minimum eye height and minimum eye width would either be too lenient — allowing a signal that is marginal in both at once — or too strict.

Transmitter and receiver masks are opposites

A transmitter mask is a floor. Measured at the transmitter's compliance point, usually through a defined reference channel, it asserts the signal is at least this good.
A receiver mask is a floor on what the receiver must tolerate. It is tested by deliberately generating a signal that only just clears the mask — stressed with calibrated jitter, ISI and noise — and requiring the receiver to hold its error rate. The test equipment is producing the worst legal signal, not a good one.

The measurement point is half the spec “Does it meet the mask?” is not answerable without saying where, and through what. At the transmitter pin, after a reference channel, at the receiver pad, or after the receiver's equaliser are four different pictures of the same link — and modern serial standards deliberately permit a fully closed eye at the pad, because the DFE is expected to open it. A mask quoted without its reference point and reference equaliser is not a specification.

Masks and BER

An eye drawn from a finite capture shows the common cases, not the tails. A mask evaluated against a million unit intervals is being tested at roughly 10⁻⁶, while the requirement is usually written at 10⁻¹². So compliance masks are normally evaluated against a statistical eye — contours of constant probability rather than overlaid traces — where the mask is checked against the 10⁻¹² contour specifically.

This is why a link can visibly clear the mask on a scope and still fail compliance: the visible eye is the 10⁻⁶ contour, and the one being judged is six decades further out.

Why this decides how your board behaves

Interpret a mask at its specified reference plane. The same signal passes or fails depending on where it is measured, and modern specifications deliberately place transmitter and receiver test points where the signal looks very different.
Passing a mask is not the same as having margin. A mask is a threshold test. It tells you that you cleared it, not by how much, and two designs that both pass can have very different robustness.
An illustrative mask is not a specification mask. Masks drawn for teaching — including on this site — are shaped like the real thing and are not the real thing. Only the published document governs.
Receiver masks work backwards from transmitter masks. Confusing the two leads to testing the wrong thing entirely.

What to do about it

Read the test setup, not just the polygon. A mask comes with a reference channel, a reference receiver, a clock recovery definition and a measurement plane. All four change the answer, and the polygon is the least of them.

Measure margin as well as pass/fail. Scaling the mask until it just touches gives you a margin number, which is far more useful in a review than a binary result — and it is what tells you whether you are close to an edge.

Do not use a transmitter mask to judge a receiver's input. They are opposite claims. A transmitter mask says the signal is at least this good; a receiver mask says the part must work with a signal only just this good.

Treat a closed eye at the receiver pad as normal on a modern link. The mask that matters there is applied after the reference equaliser, not before it.

Using this to find a fault

A signal that fails the mask at the corners but not the centre. Jitter, not amplitude — the corners are where the timing requirement lives.
Fails the centre but clears the corners. Amplitude: loss or an attenuating fixture rather than a timing problem.
Passes on one instrument and fails on another. Almost always clock recovery. Compare the loop bandwidths before suspecting anything physical.
Passes the mask and errors in traffic. The mask tested the common cases. Errors come from the tails — go to a bathtub.

Go deeper — Golden PLL emulation, BER contour scaling, and compliance reference planes

Why the corners slope. The polygon is, in effect, a contour of constant error probability. An error happens when noise carries the voltage across the threshold at the sampling instant, and the amount of noise required depends on both how far the signal is from the threshold and how far the sample is from the crossing. Lines of equal risk in that space are not rectangular — they trade voltage against time — and the mask's sloped corners are a piecewise-linear approximation of that contour.

What it is standing in for. The real requirement is a bit error rate, which cannot be tested quickly. A mask is a fast, repeatable proxy: if the signal clears this shape under these conditions, the specification's authors believe the error rate will be met by a conforming receiver. Everything uncomfortable about masks follows from that — the proxy is only as good as the conditions it was defined under, and it says nothing about a non-conforming receiver or an impairment outside the modeled set.

Software CDR (The Golden PLL). When an oscilloscope captures a multi-gigabit waveform, it does not have an external clock reference. To slice the continuous waveform into unit intervals (UI) and overlay them into an eye diagram, the scope executes a software Phase-Locked Loop (PLL), universally referred to as the "Golden PLL". Standards specify the exact loop architecture — typically a Type-II PLL with a precisely calibrated loop bandwidth (e.g. f_Baud / 1667 or f_Baud / 2500, corresponding to 10 to 20 MHz) and a 20 dB/decade roll-off.

This software emulation replicates what a conforming silicon receiver does in hardware: it tracks and cancels low-frequency phase wander (such as low-frequency spread-spectrum clock modulation or thermal drift), removing it from the measured eye. Conversely, high-frequency jitter above the loop bandwidth cannot be tracked; it manifests as horizontal eye closure against the mask. If you change the oscilloscope's software CDR bandwidth or damping factor, the eye width changes instantly. Two instruments testing the exact same physical board will disagree if their software CDR settings do not match the standard to the letter.

BER contour scaling (10^-6 vs. 10^-12). An oscilloscope running on the bench typically acquires 10⁵ to 10⁶ individual bits within a few seconds. The visible boundaries on screen therefore represent an eye opening at a confidence level of roughly 10^-6. But industry standards mandate compliance at 10^-12 or 10^-15.

To bridge this gap without waiting hours for a trillion bits to trigger, compliance test software fits a Dual-Dirac model to the measured jitter distributions, decomposing edge timing into Deterministic Jitter (DJ) and Random Jitter (RJ). The software then mathematically projects the Gaussian tails outward by multiplying the random jitter standard deviation: TJ(10^-12) = DJ + 14.069 · σ_RJ. The resulting inner boundary — the statistical contour — is plotted against the mask. This is why links frequently fail compliance while looking completely open on the scope screen: the visible persistence eye clears the mask easily, but the heavy Gaussian tails extrapolated to 10^-12 penetrate deep into the keep-out zone.

Transmitter vs. Channel compliance reference planes. Compliance specifications never judge a signal in isolation; they evaluate it at defined physical and mathematical reference planes:

Transmitter Compliance (Near-End / Far-End): For plug-in cards, the transmitter is tested through a standard compliance test fixture (such as a PCIe Compliance Base Board, CBB). The signal is captured at the connector boundary and passed through a software-emulated reference equalizer (a standardized CTLE and DFE) defined in the specification before being judged against the TX mask.
Channel Compliance (Interconnect S-Parameters): Rather than measuring an eye, the physical trace and connector are measured on a VNA. Tools like Channel Operating Margin (COM) mathematically drive a synthetic worst-case transmitter through the measured S-parameter matrix and verify that the calculated eye opening meets the required margin.
Receiver Tolerance (Stressed Eye): The receiver is tested under worst-case conditions. A calibrated bit error rate tester (BERT) synthesizes an intentionally degraded signal — dialed in with maximum allowable ISI, sinusoidal jitter, and random noise so that it just touches the compliance mask — and verifies that the receiver's physical silicon still achieves the target BER without dropping packets.

In the real world

Masks exist for interoperability, not for design. They are the minimum a specification body could agree on that still guarantees two conforming parts will talk to each other, which means passing one is a floor rather than a target.

Designing to just clear a mask is therefore a reasonable commercial decision and a poor engineering one, because the mask's conditions are not your conditions. Your channel, your neighbours and your supply noise are not in the reference setup, and the margin between clearing the polygon and clearing it comfortably is where all of those have to fit.