Getting 50 GB/S Back from the Apple Neural Engine

The Apple M3 Neural Engine ( ANE ) suffers an RTL performance erratum that caps DRAM weight‑streaming throughput at 17–19 GB/s instead of the nominal 45–60 GB/s whenever the total weight size is an integer multiple of 1 MiB. The issue touches 7 of 15 models in ANEMLL, but a kernel‑DMA tweak can lift token throughput from 10.0 to 24.3 tokens/s for Llama 3.2 1B and from 1.36 to 2.97 tokens/s for Qwen3‑8B.

Discovery

Profiling a single‑token decode ( X[1, D] × W[D, N] = Y[1, N] ) on an M3 Air revealed that D = 1536 ran almost three times faster than the default D = 2048. The static kernel‑DMA median latency (µs per replica, N = 4096) is shown below; all numbers are bolded as required.

| D | 576 | 768 | 1024 | 1280 | 1536 | 2048 | |------|--------|--------|----------|----------|----------|----------| | rep a | 150.4 | 196.8 | 238.1 | 293.8 | 310.9 | 997.6 | | rep b | 157.8 | 190.2 | 250.1 | 288.3 | 326.8 | 995.0 | | rep c | 148.7 | 189.6 | 249.4 | 275.2 | 316.5 | 995.4 |

Sweeping D around 2048 exposed a dramatic dip: at D = 2048 the measured bandwidth was 16.93 GB/s, while at D = 2016 it rose to 44.5 GB/s, a loss of 27.57 GB/s (61.96% lower). The experiment ran forty times under identical thermal conditions, varying only the DMA size and address registers.

The relevant register snapshot (hex) highlights the precise change in the DMA size field:

• TD+0x004 estimated cycles: 0x000001ea → 0x000001eb → 0x000001ec

• TD+0x078 core 1 base: 0x000ff800 → 0x00100000 → 0x00100800

• TD+0x07c core 2 base: 0x001ff000 → 0x00200000 → 0x00201000

• …

• TD+0x0b0 core 15 base: 0x00ef8800 → 0x00f00000 → 0x00f07800

• TD+0x0b4–0x0f0 core sizes × 16: 0x000ff800 → 0x00100000 → 0x00100800

• TD+0x134 Common.Cin: 0x000007fc → 0x00000800 → 0x00000804

• TD+0x1f0 L2 source stride: 0x00007fc0 → 0x00008000 → 0x00008040

• TD+0x1f4 unknown stride mirror: 0x00007fc0 → 0x00008000 → 0x00008040

• TD+0x214 L2 result base: 0x00008fc0 → 0x00009000 → 0x00009050

A full‑aperture sweep of D produced a clear resonance at 2048, resembling an FFT of bandwidth versus tensor dimension. The memory controller exhibits a dominant harmonic with a wavelength of 2048 in the D‑space, and every multiple of 2048 suffers the same throttling floor of 17–19 GB/s, recovering only about 256 lines away.

Hypothesis 1 – DRAM Spatial Correlation

The LPDDR‑6400 interface on the M3 delivers a theoretical peak of 102.4 GB/s ( 128 bit × 6.4 GT/s ), matching Apple’s advertised 100 GB/s. DRAM bandwidth depends on spreading parallel requests across independent banks; if all 16 cores target the same bank, the effective throughput collapses to the observed floor. A power‑of‑two stride such as 2048 may cause lower address bits to remain constant, forcing all cores onto a single bank.

Core‑level parallelism in the ANE involves 16 cores fetching disjoint slices of the weight buffer simultaneously. Experiments varying the active core count from 1 to 16 showed identical latency for both D = 2016 and D = 2048, indicating that the bottleneck exists even with a single core and is therefore replicated across cores rather than caused by inter‑core contention.

Address Contention and Random Scrambling

To probe address‑level aliasing, the weight buffer’s physical address was scrambled across the full ~64 MiB IOVA arena ( 59.90 MiB span), both within pages and across page boundaries. Baseline runs interleaved with scrambled runs eliminated thermal bias on the fanless M3 Air. The median throughput rose from 31.37 GB/s (baseline) to 32.29 GB/s (scrambled), a modest 1 GB/s gain that suggests spatial correlation plays a minor role but cannot explain the 200% throughput collapse.

Hypothesis 2 – RTL Integer Wraparound

The notch recurs at every integer multiple of 2048, hinting at a fixed‑width counter overflow. A simple 14‑bit line counter illustrates the pattern:

In the kernel‑DMA operation, each weight tensor contributes D FP16 weights ( 2 bytes each). With N = 4096 output channels, each core processes N/16 kernels, yielding:

bytes/core = (N/16) × D × 2.

When D and N are chosen so that bytes/core equals exactly 1 MiB, the transfer hits the throttled bandwidth of 17–19 GB/s regardless of the exact D value. The ANE’s L1 “KMem” per core is 64 KiB, far smaller than the 1 MiB transfer, indicating that a speculative prefetch ring operates on the larger window.

The kernel‑DMA line granularity is 64 bytes ( 2⁶ alignment). A 1 MiB transfer therefore comprises 0x4000 lines (16 384 lines). This matches a 14‑bit counter limit, and the lower 14 bits of the address correspond to the 16 KiB virtual‑memory page offset, which remains unchanged by address translation. Consequently, a prefetch engine that only tracks the low 14 bits will treat a full‑lap transfer as “empty” when the counter wraps, starving its own credit pool.

The prefetch ring is limited to 256 lines (16 KiB) of look‑ahead credit. The effective transfer size per core can be expressed as:

S(k,x) = 64 × (k · 0x4000 + x) = k · 1 MiB + 64x bytes,

where k is the number of full 0x4000‑line laps and x is the offset from the notch. Measured latency scales linearly with k, confirming that each lap repeats the same bandwidth profile B(x), while x determines the instantaneous rate. The slope of the latency curve is approximately 3.18 µs/line per lap, matching the observed median of 900 µs for a full lap at the floor bandwidth.

Likely RTL Bug in the Prefetch Ring

The evidence points to a speculative prefetch ring whose 14‑bit head/tail arithmetic omits an epoch bit. When a transfer size is an exact multiple of 0x4000 lines, the ring mistakenly believes it has no remaining lines, halting further prefetch issuance. The DMA still completes correctly, but the pipeline starves, converting a steady‑state stream into a stop‑and‑go pattern.

By avoiding exact 1 MiB per‑core DMA sizes—e.g., splitting kernels that would hit the 0x4000‑line boundary—software can sidestep the bottleneck without hardware changes. This workaround already yields a 2.4× token‑throughput increase for Llama 3.2 1B and a 2.2× boost for Qwen3‑8B.

Conclusion

The Apple M3 ANE performance erratum stems from an RTL integer wraparound in the kernel‑DMA speculative prefetch ring, which throttles any transfer sized at an exact multiple of 1 MiB ( 0x4000 lines). The phenomenon manifests as a bandwidth floor of 17–19 GB/s versus the expected 45–60 GB/s, with a harmonic dip every 2048 tensor dimension. Mitigation is straightforward: ensure that per‑core weight DMA sizes avoid the 0x4000‑line alignment, or patch the firmware to include an epoch bit in the prefetch counters. Until Apple releases a silicon fix, compiler‑level splitting remains the most reliable path to reclaim the 50 GB/s streaming potential of the M3 Neural Engine.