The large-scale model industry has undergone a cognitive shift: training is a one-time investment; the real continuous drain on computing power and the key to user experience lies in inference. Especially with the rise of AI agents, workflows require thousands of consecutive token generation steps. The latency of the first token, generation stability, and the speed of long-context decoding directly determine whether the agent can complete complex tasks. Traditional GPU architectures, which handle both training and inference, have inherent bottlenecks in continuous decoding scenarios.
The Groq 3 LPX, a rack-mounted LPU (Language Processing Unit) inference hardware system launched after acquiring the Groq team, deviates from the general-purpose GPU route. All hardware resources are dedicated to Transformer inference and decoding. With massive SRAM + deterministic dataflow architecture, it creates a dedicated inference rack for the Agent era. It’s not intended to replace GPUs, but rather to form a heterogeneous computing cluster with Vera Rubin GPUs, reshaping the computing power division of the AI factory.
36Kr ## Hardware System Architecture: Three-Tier Design of Rack, Tray, and LPU Chip
The Groq 3 LPX is not a single card, but a complete liquid-cooled standard rack system. A single rack integrates 256 Groq3 LP30 LPU chips, comprised of 32 compute trays. Each tray carries 8 LPU chips + 1 FPGA scheduling chip. The FPGA handles high-speed interconnection between chips and task scheduling bypass, solving the challenge of large-scale multi-chip collaborative scheduling.
Groq3 LPU Chip Core Hardware Specifications
- Each LPU has a built-in 500MB compiler-managed SRAM, with a single-chip SRAM bandwidth of 150TB/s, far exceeding the 22TB/s bandwidth of HBM4 memory. Data access achieves nanosecond-level latency, which is the foundation of the entire system’s performance.
- FP8 computing power: 1.2 PFLOPS; 98 billion transistors; native vector SIMD architecture, with the matrix execution module MXM and the vector execution module VXM handling matrix multiplication, activation functions, and MoE expert operators respectively; native hardware support for Transformer operators.
- 96-way 112Gbps direct chip-to-chip links, enabling high-speed tensor parallelism between chips without relying on the host CPU for data relay.
Key Rack Parameters
- Total SRAM capacity: 128GB; On-chip SRAM bandwidth: 40 PB/s; Inference power: 315 PFLOPS; Inter-chip extended bandwidth: 640TB/s NVIDIA.
- Liquid-cooled system, compatible with MGX ETL rack standards, and can be deployed in the same rack as Vera Rubin NVL72 GPU racks.
- At the software level, it relies on the NVIDIA Dynamo scheduling framework to achieve heterogeneous decoupled inference: the Vera Rubin GPU is responsible for prefilling and attention computation; the Groq3 LPX handles the FFN feedforward network, the MoE expert layer, and continuous token decoding and generation, separating the inference pipeline so that each component performs its specific NVIDIA function.
Core architectural differences: The GPU relies on large-capacity HBM memory and dynamically schedules resources at runtime; the Groq LPU relies on the compiler to pre-complete the entire scheduling plan, belonging to a deterministic execution architecture. The computation timing and data flow path are all determined during the compilation stage, eliminating resource contention at runtime, resulting in extremely low jitter and ensuring stable output speed for long-sequence tokens, without sudden changes in speed.
Core Performance: For Long-Context Agent Workloads
Third-party Artificial Analysis benchmark: With the Gemma4-31B model and an ultra-long context of 100,000 tokens, the Groq3 LPX rack achieves an output rate of 3431 tokens/s, a 4x performance improvement compared to competing platforms in the same scenario, according to official data. In a trillion-parameter model scenario, the inference throughput per megawatt of power consumption is up to 35x higher than that of the GB200 NVL72 (NVIDIA)…
Typical Business Scenario Experience
- Multi-Step AI Agent Tasks
The agent needs to iterate through processes, call tools, and generate text, with decoding required in each iteration. Under high concurrency with traditional GPUs, the token rate fluctuates drastically, continuously amplifying task time. LPX’s deterministic execution ensures stable generation speed at each step, even with ultra-long contexts, compressing multi-step agent tasks from hours to minutes.
- Long Document RAG and Codebase Parsing With 100,000-200,000 tokens as input context, the system retrieves information from the knowledge base and generates answers. Traditional hardware decoding suffers from significant bandwidth bottlenecks, resulting in output stuttering. SRAM’s ultra-high bandwidth greatly alleviates the pressure on KV cache, with the most significant benefits in long context scenarios.
- High-Concurrency Interactive Dialogue First token latency is significantly reduced; under high concurrency, latency jitter is far less than that of general-purpose GPU inference clusters, preventing users from experiencing “instant responses followed by stuttering.”
Performance Boundary Reminder: Official benchmark scores are from a laboratory environment; in real public cloud production environments, network overhead, request complexity, and scheduling overhead will consume some of the theoretical peak. The publicly reported 3431 tokens/s is a benchmark result under optimal conditions and does not equate to consistently achieving full performance in production environments.
Core Advantages
- SRAM Dataflow Architecture, Directly Addressing the Storage Limits of Inference and Decoding By abandoning the pursuit of ultra-large external GPU memory, a huge number of transistors are placed in on-chip SRAM, achieving memory bandwidth that far surpasses HBM. The decoding phase involves extensive repetitive read/write operations to the KV cache, a critical area for bandwidth-sensitive scenarios. This architecture precisely addresses the biggest pain point in Transformer inference (36Kr).
- Deterministic Execution, Predictable Latency The compiler pre-arranges all computations and data flow, eliminating dynamic resource preemption at runtime. This is invaluable for Agent systems: Agent programs need to estimate the time required for each step, and unstable latency can cause the entire business logic to time out and fail—a problem that general-purpose GPUs struggle to completely solve.
- Heterogeneous Collaboration, Maximizing the Value of Computing Assets It doesn’t require users to replace all their GPU hardware. The existing Vera Rubin GPU cluster serves as prefill computing power, while LPX is dedicated to offloading decoding load. Prefill and decoding inherently have completely different load characteristics; this heterogeneous division of labor significantly improves overall data center energy efficiency and dramatically enhances economic benefits for trillion-parameter model scenarios (NVIDIA…).
- Rack-Level Native Scalability
256 LPUs are interconnected via high-speed direct connections, functioning as a giant inference engine; tensor parallelism can be deployed across a large number of chips, eliminating the need for excessive fragmentation of ultra-large models and reducing the overhead of distributed inference software. - Outstanding Energy Efficiency
Official data shows that the Llama4-405B workload achieves a 3.2x improvement in inference throughput per watt compared to the H100. For large-scale AI factories, electricity costs are a major operating expense, and energy efficiency directly determines business profit margins.
Existing Shortcomings and Real-World Constraints
- Only Proficient in Inference Decoding, Completely Lacks Training Capabilities
The hardware does not support backpropagation training and fine-tuning. Prefilling computationally intensive tasks still heavily rely on the accompanying Vera Rubin GPU. A standalone LPX rack cannot be deployed; a GPU cluster must be used to form a complete inference service, resulting in a high overall hardware procurement threshold. - High Software Stack Ecosystem Threshold
It relies on a dedicated compiler for static scheduling, unlike the flexible CUDA ecosystem. Models require specialized compilation and adaptation; adaptation costs are high for non-Transformer models and niche operators. MoE models offer significant benefits, but customized models require substantial adaptation work. - SRAM Physical Capacity Ceiling A single chip has only 500MB of SRAM. SRAM density is far lower than DRAM/HBM. Even in ultra-large KV cache scenarios, collaboration with external storage is still necessary. In extremely large-scale context scenarios, hardware is inherently constrained by SRAM capacity.
- Extremely High Cost Barrier, Targeting Large AI Factories The entire system is positioned for cloud vendors and ultra-large-scale AI service providers. The investment in single-rack hardware is enormous, making it unsuitable for SMEs and private small-model deployment scenarios. The target customers are “Token Factories,” not ordinary enterprise privatization.
- Increased Operational Complexity Due to Heterogeneous Architecture A single inference request spans two hardware pools: GPUs and LPUs. The Dynamo scheduling framework needs to manage cross-hardware task distribution, load balancing, and failover. The operations team needs to master two hardware systems, resulting in a larger system failure surface compared to a single GPU cluster.
Competitive Product Comparison
Table
| System | Groq 3 LPX Rack | GB200 NVL72 | Traditional H100 Inference Cluster |
| Core Positioning | Agent low-latency decoding accelerator, heterogeneous collaboration | General-purpose, integrated training + inference | General-purpose training and inference |
| Storage Core | On-chip SRAM, ultra-high bandwidth, limited capacity | Large-capacity HBM high-performance memory | HBM memory |
| Execution Mode | Compiler static deterministic scheduling | Runtime dynamic scheduling | Runtime dynamic scheduling |
| Strengths | Long context token generation, Agent decoding | Model training, Prefill, general-purpose inference | General-purpose inference, mature ecosystem |
| Weaknesses | No training capability, dependent on matching GPU, software adaptation costs | Decoding stage latency jitter, average energy efficiency | Long context decoding bandwidth bottleneck |
| Target Customers | Large cloud vendors’ AI factories | General-purpose supercomputing, large model training | Various enterprises, cloud service providers |
Summary: Inference Hardware Moving Towards Specialization, an Infrastructure Example for the Agent Era
The biggest takeaway from the Groq3 LPX rack is that AI computing hardware is no longer pursuing a “one-size-fits-all” approach. The training, prefill, and decoding stages have vastly different load characteristics; using the same GPU hardware to run the entire process inherently leads to efficiency losses. Groq3 LPX represents an industry move towards specialization: GPUs continue to handle training and prefilling, while dedicated LPU hardware takes over the high-pressure token decoding and generation.
It’s not designed to replace GPUs, but rather to address the shortcomings of GPUs in agent inference. As applications shift from simple question-and-answer to numerous multi-step agent intelligences, placing stringent demands on generation speed and latency stability, Groq3 LPX is a hardware product designed for this future.
However, it must be viewed objectively: this system is a heavy-duty data center system with high investment, maintenance, and model adaptation costs. In the short term, it will only appear in the AI factories of leading cloud vendors; it will be difficult for small and medium-sized enterprises to implement directly. Meanwhile, benchmark scores are ideal laboratory results. The stability, fault performance, and performance under complex mixed traffic conditions in real production environments still need to be verified by more large-scale commercial cases.