From Raw Sensor Data to Final Image Output

Hello. In the previous lesson, you reconstructed a full-color image from a Bayer mosaic. Its three channels are still camera measurements, not yet the red, green, and blue values of a finished photograph. A raw converter must establish what the recorded numbers mean, choose how to interpret the scene’s illumination, translate the camera’s color responses into an output color space, and fit the result into a displayable range.

This lesson follows one measurement through those operations. The distinction to keep in view is between calibration, which corrects how the instrument records light, and rendering, which chooses how that light becomes a photograph. Allow about 40–45 minutes, including the focused readings.

Establishing a linear measurement

A raw file contains digitized sensor signals and metadata needed to interpret them. Its numbers are not generally photon counts: photoelectrons have been converted to voltage, amplified, digitized, and often stored with an offset. That offset permits noise fluctuations around a zero-light signal without requiring negative integer codes.

The Digital Negative (DNG) Specification gives a precise processing model for this first stage. Read Chapter 5 before treating any raw code as an intensity.

In a simplified case, let $d$ be a stored pixel code, $b$ its black level, and $w$ its white level. If digitization is already linear, its normalized value is approximately

For example, with $d=1064$, $b=64$, and $w=4095$, the light-dependent value is $1000/4031$, or about $0.248$. Dividing $1064$ by the maximum integer code instead would incorrectly count the offset as light. Nor must a file’s white level equal its largest representable integer: it denotes the maximum useful signal under that format’s calibration.

The DNG model permits a linearization lookup table before black subtraction when stored codes do not vary linearly with sensor signal. Black levels can also vary across channels, rows, or columns. In a fuller calibration, defective pixels may be replaced and flat-field or lens-shading corrections may compensate for spatial sensitivity variation. Such corrections improve consistency but do not undo shot noise; multiplying a weakly illuminated corner also multiplies its noise.

At this point the values represent light on a camera-relative linear scale. Doubling an unsaturated value corresponds approximately to doubling the underlying light-dependent signal. The preceding lesson’s demosaicing can supply three estimated camera-channel values at each location. Some pipelines apply per-channel gains to mosaic sites before demosaicing; others incorporate them later. Either way, which samples were measured and which were interpolated remains important.

White balance: choosing a neutral under the illumination

Consider a neutral surface photographed under a particular light. Its spectrum need not produce equal readings in the camera’s red-, green-, and blue-filtered photosites. Each channel responds to a broad, overlapping range of wavelengths, and the illuminant changes the light available in those ranges.

For a patch that the renderer intends to appear neutral, suppose its calibrated camera-channel vector is proportional to

Taking green as the reference, diagonal gains of $2$, $1$, and $4$ would make the three values equal. More generally, for a selected camera neutral $\mathbf{n}=(n_R,n_G,n_B{)}^{\mathsf{T}}$, a simple white-balance model is

This is not a discovery of the surface’s complete reflectance spectrum. It is a rendering choice about which illumination-relative camera response should appear neutral. An automatic estimate, a selected reference surface, and a deliberate warmer rendering can therefore produce different legitimate white balances from the same raw data.

Gains have physical consequences. A blue-channel gain of $4$ also scales that channel’s noise by $4$ in signal units; it does not create four times as many detected photons. A strong gain can push a highlight beyond the output range even when the corresponding raw photosite was not saturated. Conversely, once a photosite has saturated, changing white balance cannot reconstruct its missing measurement.

DNG can record an as-shot neutral while leaving the stored raw values largely unbalanced. This preserves flexibility for a later rendering decision. Its color-transform model may also incorporate white-balance scaling into a combined matrix. When reading a real pipeline, check whether its input values have already been balanced: applying the same gains twice is an easy way to manufacture a color cast.

Color transformation: from camera responses to output primaries

White balance aligns a chosen neutral, but it does not make camera RGB identical to display RGB. The sensor’s filters and a display’s primaries have different spectral definitions. In a simplified linear model, a camera channel measures a weighted integral of incident spectrum:

where $S_k(\lambda )$ includes that channel’s spectral sensitivity and $L(\lambda )$ is the light reaching the pixel. Standard color coordinates instead use specified color-matching functions. A calibrated matrix approximates the relationship between these different sets of three responses.

Return to the DNG specification for a concrete color-management model. It is deliberately more detailed than the single “camera-to-sRGB matrix” often used to sketch an image pipeline.

Conceptually, after calibration, demosaicing, and the chosen white balance, one route to a display color space is

Both matrices act on linear values. The first relates the camera’s responses to standardized color coordinates; the second expresses those coordinates using the chosen output primaries. White-point adaptation must be handled where needed: the DNG transform discussed above targets XYZ with a D50 white point, whereas sRGB uses a D65 white point. A production pipeline may combine these operations into fewer matrices, but their meanings should not disappear.

Why can’t one matrix be perfect for every photograph? Two different spectra can generate the same three camera readings, and camera sensitivities do not generally reproduce the standard color-matching functions exactly. A fixed transform therefore cannot recover every surface color under every illuminant. DNG’s provision for calibrations at different illuminants is one engineering response to that limitation. Profiles may add further color adjustments beyond a basic matrix.

The transform can also produce negative components or values above $1$. These are not necessarily failed arithmetic: the measured color may fall outside the chosen output primaries or exposure range. Simply clamping each channel independently is possible but can alter hue. Output-gamut handling and highlight treatment are rendering decisions, not sensor calibrations.

Tone mapping is not output encoding

Even after color transformation, linear output RGB is not usually ready to store as a conventional photograph. A scene may span more brightness than the intended output can convey. Tone mapping determines how scene-referred brightness is rendered into the chosen output range: it may set overall exposure, preserve midtone contrast, raise shadows, or compress highlights. A simple global curve applies the same rule everywhere; many cameras also make spatially varying adjustments.

A research article on reversing image processing provides a useful simplified forward model. Its inverse pipeline is not evidence that every camera uses its particular curves; read it for the separation it makes between contrast rendering and nonlinear encoding.

For values already confined to $0$ through $1$, the article’s illustrative S-curve can be written

It maps $0.25$ to $0.15625$ and $0.75$ to $0.84375$: shadows darken, highlights brighten, and midtone contrast increases. That particular curve does not, by itself, fit a high-dynamic-range scene into a narrower range; its input is already bounded. A highlight-compressing rendering needs an appropriate mapping from the scene’s wider brightness range before final output. The tone curve is a choice about appearance, not a calibration that reveals more light than the sensor captured.

Output encoding serves a different purpose. Once linear output RGB has been rendered into range, an encoding curve assigns stored code values. Nonlinear encoding gives relatively more code values to darker linear-light intervals than uniform linear quantization would. It must not be confused with applying a photographic contrast curve, nor is it accurate to say that the sensor itself responds “with gamma.”

For sRGB, each normalized linear component $u$ is encoded by the specified piecewise function

The article’s power law with an approximate exponent of $1/2.2$ is useful for intuition, but it is not the exact sRGB encoding rule. After encoding, values can be quantized into an output representation—for example, integer channel codes in an 8-bit sRGB image. Adding those encoded codes is no longer equivalent to adding light.

Following one neutral patch all the way through

Suppose nearby photosites sampling a neutral patch give representative raw codes of $1064$, $2064$, and $564$ in the red, green, and blue CFA bands. Assume a common black level of $64$, a white level of $4095$, and no additional linearization. After black subtraction and rescaling, the representative channel values are

These components originated at different CFA locations. Demosaicing, as studied previously, is what supplies an estimated triplet at each output location. For this uniform patch, suppose that estimate retains the ratios above.

Choose this patch as neutral. Applying white-balance gains $(2,1,4)$ gives approximately $(0.496,0.496,0.496)$. Now suppose a calibrated color transform maps this balanced neutral to equal components in linear sRGB. That assumption is appropriate for this illustrative neutral; it says nothing about how the same transform maps other colors.

If the renderer applies the illustrative S-curve, each component becomes about $T(0.496)\approx 0.494$. Applying the exact sRGB encoding gives approximately $0.732$, which rounds to about 187 on an 8-bit scale. The stored value 187 is neither a sensor code nor a quantity proportional to incident photons. It is the result of a selected neutral, a selected rendering curve, an output color space, encoding, and quantization.

The order explains several common failures. Subtracting the wrong black level can tint shadows after strong channel gains. Applying a color matrix to gamma-encoded values changes its intended linear relationship between colors. Encoding before tone mapping makes a curve designed for linear light behave differently. And if the starting photosite clipped, no later stage can restore the spectral measurement that clipping removed.

Conclusion

A raw measurement becomes a finished pixel only after several distinct interpretations. Calibration establishes a linear light-dependent value; demosaicing estimates missing camera-channel measurements; white balance selects a neutral for the illumination; and a color transform relates camera responses to chosen output primaries. Tone mapping chooses how brightness will appear within the output range, while encoding and quantization determine the stored codes.

This division matters because some errors belong to capture, while others can be changed by re-rendering the same raw file. The next lesson turns to compact camera engineering, where sensor architecture and restricted optical space place new constraints on the measurements available to this pipeline.