Meta GEM Training Efficiency: Why LLM-Scale Recommenders Need a Different Stack

Meta GEM training efficiency reached 20% to 25% end-to-end Model FLOPs Utilization while Meta increased total training FLOPs fourfold in 12 months. The reusable lesson is a measurement method: identify the current bottleneck, optimize the layer that owns it, validate model quality, and measure again. LLM-scale recommendation models resemble language models in compute volume, but their jagged data, sparse parameters, asymmetric attention, and communication patterns require a different efficiency stack.

Start with the denominator

Meta reports that GEM trains on several thousand recent-generation GPUs and combines trillions of sparse embedding parameters with billions of dense parameters. It decomposes training efficiency as:

End-to-end MFU = Local MFU × Scaling Ratio

Local MFU measures how well one GPU executes the workload. Kernel design, numerical precision, tensor shapes, and memory access determine this part. Scaling ratio measures how much of that local performance survives distribution across many GPUs. Communication, topology, load imbalance, recomputation, and stragglers determine this part.

This split prevents a common measurement error. A kernel can run twice as fast while the training job barely changes because the step is waiting on communication. A collective can move fewer bytes while consuming the same wall-clock time because latency or synchronization has become dominant. Local wins matter only when they move an end-to-end metric.

MFU also needs a precise denominator. Hardware peak FLOPs depend on device generation and numerical format. An MFU of 25% does not mean that the remaining 75% is recoverable waste. Some work is memory-bound, some operations do not map to tensor cores, and production training includes communication and synchronization that a peak-FLOPs denominator does not represent.

Why the standard LLM training recipe misses the workload

GEM uses sequence features such as user activity history and non-sequence features such as location and ad representations. The resulting workload breaks several assumptions behind dense LLM infrastructure.

The point is workload fit. A mature LLM stack remains useful infrastructure, but every major optimization needs to be re-profiled against recommendation data and production tensor shapes.

Bottleneck move 1: dense-sequence kernels to jagged execution

Padding GEM's variable-length sequences to the maximum length could waste up to 50% of compute, according to Meta. Avoiding padding introduces another problem: short and long samples finish at different times, creating irregular work within and across GPU ranks.

Meta built several recommendation-specific kernels:

• Jagged Flash Attention operates directly on variable-length tensors and supports custom biases and asymmetric query/key-value lengths.
• Generalized Dot-Product Attention handles attention-like modules that use activations such as GELU or SiLU instead of softmax.
• BlockAttention converts eligible long-sequence attention into independent fixed-size blocks, reducing unnecessary work.

The PyTorch GDPA engineering report shows why synthetic benchmarks are weak evidence here. Under disclosed production shapes, the optimized kernel achieved up to 2x forward and 1.6x backward speedups over its Triton baseline, and more than 30% full-model training throughput improvement. Under some short key/value settings it reached up to 3.5x forward speedup over FlashAttention 4. These numbers belong to those shapes and baselines. Their broader lesson is to benchmark the distribution that production actually generates.

The bottleneck then moves. Removing padding exposes tile scheduling, backward accumulation, memory reordering, and rank imbalance. Faster matrix multiplication alone does not close that chain.

Bottleneck move 2: peak low-precision FLOPs to a numerical pipeline

Recent GPUs advertise much higher FP8 and FP4 peak throughput than FP16. GEM's results show that converting a model to a smaller datatype is only the first step.

Scale-factor generation, casting, and extra high-bandwidth-memory traffic can consume the theoretical gain. Meta fused activation quantization into upstream normalization and projection kernels, quantized local FSDP shards before all-gather, and communicated low-precision payloads. It also used mixed precision, stochastic rounding, outlier mitigation, and selective higher precision for sensitive weight-gradient paths and later layers.

This creates two release gates for low precision:

1. The quantized path must reduce module and end-to-end time after conversion overhead.
2. Training stability and recommendation quality must remain within a predefined acceptance range.

A faster kernel that misses the quality gate is a failed optimization. A numerically stable conversion that adds enough data movement to erase the speedup is also a failed optimization.

Bottleneck move 3: GPU count to topology-aware parallelism

GEM's dense and sparse parameters need different distribution strategies. Meta uses 2D Fully Sharded Data Parallelism plus Expert Parallelism for dense modules, and fully sharded 2D model parallelism for sparse tables. Together these form a topology-aware five-dimensional parallel design.

The important principle is message placement. Large, latency-sensitive communication should stay inside the fastest topology tier when possible. GEM's disclosed hierarchy includes NVLink within a host, RoCE within an AI zone, and lower-bandwidth oversubscribed links across zones. Adding a parallel dimension can reduce message size or collective group size on the constrained tier.

PyTorch's account of 2D sparse parallelism describes the same underlying trade-off: smaller model-parallel groups improve load balance and communication, while naive replication creates an unacceptable memory bill for very large embedding tables. Fully sharding the replica dimension trades extra fast-link communication for lower HBM use.

Communication overlap has its own trap. A collective may run at the same time as compute while still occupying streaming multiprocessors and slowing the compute kernel. Meta reports that its NCCLX all-gather path reduced SM use from about 24 to 1 and produced roughly 5% end-to-end QPS improvement at full training scale. The NCCLX paper provides the broader collective-communication context. Visible overlap and resource-independent overlap are different conditions.

Bottleneck move 4: memory relief to load balance

Once dense and sparse parameters are sharded, activations can dominate per-GPU memory. GEM combines compiler-based automatic activation checkpointing with region-specific memory budgets and activation quantization. The objective is to keep a large local batch without paying excessive recomputation cost.

Larger batches and jagged execution then expose rank imbalance. Meta observed the heaviest rank running about 15% above the average workload on each iteration. A globally optimal rebalance would add an all-to-all collective every step, so its coordination cost could erase the benefit. Meta instead used Base Batch Shuffling, which sorts small sub-batches by total sequence length and interleaves heavy and light groups locally. The company reports 4% higher QPS and 4% lower peak memory.

This is a useful systems pattern: the most balanced schedule can produce lower throughput than a slightly imperfect schedule with almost no coordination overhead.

Build an evidence ladder, not a pile of speedups

Performance numbers from different layers cannot be added. A useful training-efficiency report keeps the evidence levels separate.

Meta's public material is strongest from the kernel through steady-state distributed-step levels. It provides less detail on total training cost, energy, startup, checkpoint recovery, failure rate, and cost per accepted model. Those missing layers should remain visible rather than being silently inferred from MFU.

A reusable optimization sequence

Teams can copy the diagnostic process without copying GEM's exact kernels or parallel dimensions.

1. Record the production distribution. Capture sequence-length histograms, attention shapes, sparse-table sizes, numerical outliers, and rank-level work.
2. Create a metric tree. Connect kernel time to module time, Local MFU, scaling ratio, end-to-end step time, effective training time, and model-quality gates.
3. Classify the active constraint. Use at least compute, memory, communication, synchronization, load imbalance, and input-pipeline categories.
4. Change one bottleneck owner. A custom kernel, precision recipe, sharding dimension, or batching policy should have an explicit target metric.
5. Verify the whole chain. Re-run numerical tests, quality metrics, distributed profiles, checkpoint recovery, and wall-clock measurements.
6. Profile again. A successful optimization changes the constraint map, so the previous priority becomes stale.

Small teams can start with production-shape benchmarks, profiler traces, sequence-length-aware batching, selective mixed precision, and a clear metric tree. Custom kernel libraries and five-dimensional parallelism become justified only when measurements show that standard implementations own a material share of end-to-end time.

The same logic appears at other infrastructure layers. OpenAI's MRC networking design treats the constrained network tier as the unit of optimization, while token-budget engineering shows why resource ceilings should be connected to delivered capability rather than tracked as isolated expense lines.

What Meta's evidence does and does not establish

The primary report provides unusually detailed production mechanisms, but it remains a company-authored case study. The exact GPU model, absolute GPU count, power envelope, complete baselines, total cost, and confidence intervals for quality claims are not fully disclosed. Most local gains use different baselines and cannot be summed into the reported twofold end-to-end improvement.

The 20% to 25% MFU result establishes Meta's stated outcome under its internal workload and measurement method. Independent replication would require the model configuration, data-length distribution, hardware topology, precision denominator, training-quality gates, and full profiler traces.

FAQ

What is Meta GEM?

GEM is Meta's generative ads recommendation foundation model. Meta describes it as a hybrid architecture with trillion-scale sparse embedding parameters and billion-scale dense parameters, trained on ad content and user-engagement data.

What is Model FLOPs Utilization?

MFU compares achieved model computation with a hardware peak-FLOPs denominator. It is useful for tracking one defined workload and hardware setup. Comparisons across devices, precision formats, or model definitions require aligned denominators.

Why does standard FlashAttention fit recommendation workloads poorly?

Many production recommendation tensors have variable lengths, large batches, asymmetric query and key/value dimensions, custom biases, and non-softmax activations. Kernels optimized for dense, regular LLM sequences can leave compute units idle or spend work on padding.

What is a jagged tensor?

A jagged tensor stores variable-length samples compactly, usually with values plus offsets or length metadata. It avoids dense padding, while introducing irregular scheduling and communication requirements.

Should every recommendation team build custom GPU kernels?

Profiling should decide. Custom kernels make sense when a stable production shape consumes a material share of end-to-end time and existing implementations miss the hardware roofline. Data, batching, communication, or recovery may offer a larger gain with lower engineering cost.

Can the reported speedups be added together?

No. They use different scopes and baselines, including kernel, layer, single-GPU, and distributed-job measurements. Use an end-to-end ablation or waterfall to attribute the combined result.

References

• Meta Engineering: GEM Training and the LLM-scale ads foundation model
• Meta Engineering: Meta's Generative Ads Model
• RecSys 2025: Scaling Generative Recommendations with Context Parallelism on HSTU
• PyTorch: Generalized Dot-Product Attention
• PyTorch: Scaling Recommendation Systems Training with 2D Sparse Parallelism
• Meta researchers: Collective Communication for 100k+ GPUs
• Meta AI Research: Quantized Collective Communications for DLRM