PSNR ceiling at high bitrates caused by the non-orthogonal transform pair, with an encoder-only fix

[kpchoi]

Summary

PSNR saturates at high bitrates: beyond a certain point, spending more bits does not raise PSNR, unlike other codecs. The cause is not the internal bit depth, the transform shifts, or the quantizer — it is the non-orthogonality of the integer transform matrix pair. An encoder-only change that is fully bitstream-compatible removes the ceiling; measurements below.

Analysis

Ruled out by measurement (8x8 kernel harness, 20k random full-range blocks per case):

hypothesis	measurement	verdict
coefficient grid (s16) too coarse	grid-only floor is ~83 dB	not the cause
forward-transform intermediate rounding	single-rounding forward changes nothing (0.4 dB)	not the cause
quantizer step / level clipping	at QP 0 max level 8953, zero clipped levels	not the cause
implementation bug	exact-arithmetic ITX reproduces the same floor	not the cause

The floor comes from the matrix pair itself. For the 8x8 base matrix T, T*T^T has diagonal 32768, 32740, 33124, 32740, ... and off-diagonal entries up to 50 — rows 2 and 6 (84, 35, ...) have a squared norm 1.1% larger than the others. The decoder applies T^T as the inverse, but T^T != T^-1 * 32768, so the reconstruction error is proportional to the signal amplitude: with full-range random blocks the round-trip is ~52 dB at any bit depth; at 1/8 amplitude it is ~69 dB; at 1/64 amplitude it is lossless.

This also explains why codecs sharing this matrix family do not show the ceiling in practice: they transform small prediction residuals, so the multiplicative error is proportionally small. APV has no pixel-domain prediction — full-amplitude samples enter the transform, exposing the floor in the working range.

Encoder-only fix (bitstream compatible)

The forward transform is an encoder choice; only the inverse is normative. Replacing the forward matrix with the least-squares pseudo-inverse of the normative inverse transform, G = (T*T^T)^-1 * T, makes the encoder produce the coefficients that the fixed decoder reconstructs best. Scaled to s16 constants (2^19, i.e. 16x the base scale; entries are the base values x16 with small corrections, e.g. row 2: 1344 -> 1330) the matrix keeps its even/odd symmetry, so the same butterfly structure applies with a generic 4-multiply stage and +4 on both forward shifts.

Branch: https://github.com/AcademySoftwareFoundation/openapv/tree/improve_forward_transform

Measurements

Kernel round-trip (transform + spec integer ITX): 52.0 -> 100.7 dB (10-bit), 52.0 -> 76.6 dB (12-bit, coefficient-grid bound).

End-to-end, pattern y4m 320x240 10-bit, medium preset, 3 frames:

QP	size (base -> new)	PSNR (base -> new)
0	88230 -> 88125	67.33 -> 87.46
2	83847 -> 83776	67.35 -> 83.96
6	76751 -> 76671	67.05 -> 78.15
12	66781 -> 66746	66.07 -> 71.87
18	58195 -> 58172	63.90 -> 66.30
24	50514 -> 50487	59.52 -> 60.16
30	43228 -> 43209	54.38 -> 54.52
40	32373 -> 32366	45.03 -> 45.07

The bitrate is unchanged (marginally smaller), the gain appears exactly where the ceiling used to bind, and mid/high QPs are untouched. 12-bit behaves the same (QP 0: 64.7 -> 79.7 dB, measured with the fix from #254 applied to the decoder). All ctest cases pass, including the conformance hash checks, since the decoder is not modified.

Remaining work

• The branch forces the C forward transform; the AVX2/NEON txb kernels need the new constants (s16, madd-friendly) and the +4 shifts before this can be a PR
• BD-rate measurement on natural content to confirm the mid-QP neutrality
• The RDOQ distortion model keeps working unchanged since the coefficient scale is identical