⚡ Major Hardware Breakthroughs in AI (2021-2026)
From NVIDIA's Blackwell architecture to Cerebras' wafer-scale engines, Groq's deterministic LPUs, and the rise of chiplets, HBM, and co-packaged optics - exploring the technologies redefining AI compute.
1. NVIDIA - Grace Blackwell, Vera Rubin & DGX Spark
1.1 Blackwell Architecture (GB200/B200 Series)
Announcement: March 18, 2024 at GTC | Production: 2024-2025
Technical Capabilities
- Dual-chiplet design: 208 billion transistors on two reticle-limited dies connected by 10 TB/s chip-to-chip interconnect
- Process: TSMC 4NP (custom 4nm variant)
- Second-generation Transformer Engine: Accelerated inference for large language models
- NVLink: 1.8 TB/s bidirectional per GPU (18 links x 100 GB/s), up to 576 GPUs in non-blocking fabric
- Memory: HBM3E support with massive capacity
- Precision: Strong support for FP4/NVFP4, FP6, FP8, FP16 with mixed precision
- Decompression Engine: Accelerates database queries with LZ4, Snappy, Deflate
Key Products
| Product | Configuration | Notes |
|---|---|---|
| B200 GPU | Single Blackwell GPU | Consumer/datacenter variant |
| GB200 Grace Blackwell Superchip | 2 Blackwell GPUs + Grace CPU via NVLink-C2C (900 GB/s) | Co-packaged CPU-GPU |
| GB200 NVL72 | 36 Grace CPUs + 72 Blackwell GPUs in rack-scale | Liquid-cooled, acts as single massive GPU, 30X faster real-time inference for trillion-parameter LLMs |
| DGX Spark (Project DIGITS) | Blackwell-based desktop supercomputer | $3,000 (May 2025), connects two units for up to 405B parameter models, runs from standard outlet |
Educational Relevance
- Chiplet architecture: Excellent example of modular design overcoming reticle limits
- CPU-GPU co-design: NVLink-C2C demonstrates cache-coherent heterogeneous computing
- Rack-scale computing: NVL72 shows disaggregated/supercomputer-scale integration
- Transformer optimization: Second-gen Transformer Engine is perfect case study for workload-specific acceleration
- Desktop AI: DGX Spark democratizes large model experimentation
Sources: NVIDIA Blackwell Architecture, NVIDIA Blackwell Architecture Technical Overview, Inside NVIDIA Blackwell Ultra
1.2 Vera Rubin Roadmap
Expected: 2026 (following Blackwell Ultra)
- Next-generation platform building on Blackwell Ultra
- Tighter integration across compute, networking, and memory
- Focus on "AI factories" at scale
- Expected to leverage next-gen HBM (HBM4) and advanced networking
Note: Full technical details still emerging as of late 2025/early 2026.
2. Cerebras Systems - Wafer-Scale Engines
Cerebras pioneered wafer-scale integration for AI, placing an entire wafer as one processor. Their third and fourth generation systems are pushing boundaries.
2.1 Wafer-Scale Engine 3 (WSE-3)
Announced: March 13, 2024 | Process: TSMC 5nm
Technical Capabilities
- Size: 46,225 mm² (largest chip ever produced)
- Transistors: 4 trillion
- Cores: 900,000 AI-optimized cores
- On-chip memory: 44 GB SRAM
- Compute: 125 PFLOPS AI compute (WSE-3)
- Memory bandwidth: 21 PB/s on-chip
- Fabric bandwidth: 214 PB/s core-to-core
- TDP/Power: ~23 kW per CS-3 system
WSE-3 Turbo (CS-4)
- Up to 30X faster inference than GPU systems
- Powers Cerebras CS-4 rack-scale systems
Educational Relevance
- Wafer-scale integration: Extreme case of scale-integration, yield management via defect tolerance
- On-chip memory hierarchy: 21 PB/s bandwidth vs off-chip - demonstrates memory wall solutions
- Distributed computing on-chip: 900K cores with mesh interconnect
- Yield engineering: Overprovisioning + rerouting around defects
- No-cluster simplicity: Entire large models fit on single wafer/system
Sources: Cerebras WSE-3 Announcement, Cerebras WSE-3 Chip, WSE-3 Datasheet
3. Groq - Language Processing Units (LPUs)
Groq's LPU (originally Tensor Streaming Processor) takes a deterministic, compiler-first approach to AI inference, offering extremely low latency.
Technical Capabilities
- Architecture: Tensor Streaming Processor (TSP) / Language Processing Unit (LPU)
- Execution: Deterministic, software-scheduled (compiler controls all timing)
- Memory: Large on-chip SRAM (e.g., 500 MB per LPU in Groq 3), ~150 TB/s SRAM bandwidth
- Interconnect: No top-rack switches, chips connect directly in mesh (scale-up without traditional switch fabric complexity)
- Focus: Inference-optimized (vs training)
- Groq 3 LPX Rack: 256 LPUs per rack, 128 GB SRAM + 12 TB DDR5 in rack, 40 PB/s SRAM bandwidth per rack
Recent Developments
- December 2025: NVIDIA announced non-exclusive licensing deal worth ~$20B for Groq's LPU architecture, plus hiring of founder Jonathan Ross and president Sunny Madra. Groq continues operating independently under new CEO.
Educational Relevance
- Deterministic computing: Compiler-controlled scheduling vs dynamic hardware scheduling
- Software-hardware co-design: Extreme compiler reliance
- SRAM-centric design: Trade-off between capacity and bandwidth
- Scale-up networking: Direct chip-to-chip vs switched fabrics
- Inference specialization: Contrast with general-purpose training accelerators
Sources: Groq LPU Explained, Architecture of Groq's LPU, NVIDIA Groq 3 LPX
4. AMD Instinct MI Series (CDNA Architecture)
4.1 MI350X / MI355X (CDNA 4)
Announced: June 12, 2025 | Process: TSMC 3nm (XCD) + 6nm (IOD)
Technical Capabilities
- Architecture: CDNA 4, chiplet design (10 chiplets, 185B transistors)
- Memory: 288 GB HBM3E per GPU, 8 TB/s bandwidth
- Form factors: MI350X (air-cooled, 1000W), MI355X (liquid-cooled, up to 1400W)
- Precision support: FP64, FP32, FP16, BF16, FP8/OCP-FP8, MXFP6, MXFP4, INT8
- Matrix performance: Up to 10.1 PFLOPS MXFP4/MXFP6 (MI355X)
- Platforms: 8x GPU platforms deliver 2.3 TB memory, 64 TB/s aggregate bandwidth; scales to 128x in hyperscale
- Infinity Cache: 256 MB
Competitive Position
- Targets NVIDIA Blackwell in performance/capability
- Emphasizes memory capacity (288 GB) for large models
- Strong focus on open software stack (ROCm 7)
Educational Relevance
- Chiplet economics: 3nm + 6nm mix optimizes cost/performance
- Microscaling formats: MXFP4/MXFP6 adoption shows industry-wide low-precision trend
- Large memory footprint: Enables very large models on fewer GPUs
- Open ecosystem: ROCm vs CUDA - important for understanding software moats
Sources: AMD Instinct MI350 Series, MI355X Product Page
5. Intel Gaudi Accelerators
5.1 Gaudi 3
Announced: April 2024 (Intel Vision) | GA: October 2024 (OAM), PCIe H2 2025
Technical Capabilities
- Process: 5nm
- Architecture: Purpose-built for training and inference
- Compute: ~1.678 PFLOPS BF16/FP8 (revised spec)
- Networking: 24x 200 GbE RoCE v2 ports per accelerator (Ethernet-based, open)
- Memory: HBM (high bandwidth)
- Engines: Matrix Multiplication Engine (MME), Tensor Processor Cores (TPCs), NICs working in parallel
- Form factors: OAM (900W) and PCIe cards
- Scale: Reference designs for up to 64 accelerators per rack
Software & Ecosystem
- SynapseAI software stack, PyTorch support, vLLM integration
- Focus on open standards (Ethernet vs proprietary fabrics)
- Available on IBM Cloud, Dell, HPE, Supermicro systems
Educational Relevance
- Open networking: RoCE/Ethernet vs NVLink/InfiniBand - tradeoffs in cost/flexibility
- Specialized engines: MME + TPC + NIC co-design
- Price/performance focus: Alternative to NVIDIA's proprietary stack
- Cloud availability: Shows multi-vendor ecosystem development
Sources: Intel Gaudi 3, Gaudi 3 White Paper
6. Tenstorrent - RISC-V Based AI Accelerators
| Chip | Generation | Status | Notes |
|---|---|---|---|
| Grayskull | Gen 1 | Developer hardware | Entry point, e75/e150 cards |
| Wormhole | Gen 2 | In production/deployment | n150/n300 cards, adds Ethernet, 16 RISC-V cores |
| Blackhole | Gen 3 | Developer/early production (2025) | 140+ Tensix cores, 16 RISC-V CPU cores (4 clusters of 4), PCIe 5.0, 10x400G Ethernet, can boot Linux standalone |
| Galaxy | Rack-scale | Available | Multi-chip systems (e.g., 32 processors) |
Technical Capabilities
- Architecture: Tensix cores + RISC-V CPUs, Ethernet mesh interconnect (no NVLink)
- Programming: TT-Metalium (low-level), open-source toolchain
- Blackhole highlights: Standalone AI computer capability (Linux-capable), 745 TOPS FP8 per chip
- Interconnect: Scalable Ethernet mesh vs proprietary fabrics
- Philosophy: Open source, RISC-V based, software-defined
Educational Relevance
- RISC-V in AI: Shows open ISA entering accelerator space
- Dataflow architecture: Tensix cores demonstrate different paradigm than SIMD GPUs
- Mesh networking: Ethernet-based scaling economics
- Open ecosystem: Contrasts with closed CUDA ecosystem
- HPC applicability: Research into stencil computations, FFTs on these architectures
Sources: Tenstorrent, tt-awesome (GitHub), Tenstorrent Newsroom
7. Other Notable Architectures
7.1 Graphcore (IPU) - Wafer-on-Wafer
- Bow IPU: First 3D Wafer-on-Wafer processor (bonding two wafers)
- Benefits: 40% higher performance, 16% better power efficiency vs predecessor
- Architecture: Colossus architecture with 1000s of independent IPU cores, large in-processor memory
7.2 SambaNova (Reconfigurable Dataflow Units - RDUs)
- Approach: Reconfigurable dataflow architecture, spatial computing
- SN40L/SN50: Next-gen systems
- Partnership: Working with Intel (Xeon 6 as host) for heterogeneous inference (2026)
- Focus: Full-stack platform, inference optimization
7.3 d-Matrix
- Approach: Digital in-SRAM compute, focuses on inference efficiency
8. Critical Enabling Technologies
8.1 High Bandwidth Memory (HBM)
The memory wall remains critical. HBM evolution is key to AI performance.
| Generation | Data Rate | Interface | Bandwidth/Stack | Max Capacity | Status |
|---|---|---|---|---|---|
| HBM3E | 9.8 Gbps | 1024-bit | ~1.23 TB/s | 48-64 GB (12-high) | In production (2024-2025) |
| HBM4 | 6.4-8+ Gbps | 2048-bit (doubled) | Up to ~2.0 TB/s | Up to 64 GB | JEDEC spec April 2025, production ramping 2026 |
| HBM4E | Up to 16 Gbps | 2048-bit | Up to ~4.0 TB/s | Up to 64 GB | Future |
Market (Q2 2025): SK Hynix 62%, Micron 21%, Samsung 17%. HBM market growing rapidly ($38B in 2025 → $58B in 2026).
Educational Relevance: 3D stacking, TSVs, memory hierarchy, bandwidth vs latency tradeoffs.
Sources: HBM Evolution, JEDEC HBM4 Spec
8.2 Optical Interconnects & Co-Packaged Optics (CPO)
Copper interconnects are hitting limits at AI cluster scales; optics are becoming essential.
Technologies
- Silicon Photonics: Integrating optical components on silicon
- Co-Packaged Optics (CPO): Moving optical engines next to switch ASICs/GPUs, eliminating long electrical traces
- Near-Packaged Optics (NPO): Intermediate step
- Linear Pluggable Optics (LPO): Simplified pluggable optics
Key Players & Developments
- Broadcom: Bailly 51.2 Tbps CPO switch (8x 6.4 Tbps silicon photonics engines)
- NVIDIA: Investing heavily in silicon photonics for next-gen switches/clusters
- Lightmatter: $400M Series D, Passage photonic interconnect
- Celestial AI: Photonic Fabric, acquired by Marvell for $3.25B (Dec 2025)
- Ranovus/Cerebras: DARPA-funded wafer-scale CPO for Cerebras systems
Educational Relevance: Photonics fundamentals, signaling, SERDES limits, energy-per-bit, rack-scale topology design.
Sources: CPO Market & Technologies, Silicon Photonics & CPO Evolution
8.3 Chiplet Architectures
Now mainstream across NVIDIA Blackwell, AMD MI350, Intel products:
- Advantages: Better yields, mix of process nodes, modular design, cost optimization
- Challenges: Die-to-die interconnects (UCIe emerging standard), thermal, testing
- Examples: AMD uses 3nm XCD + 6nm IOD; NVIDIA dual-die Blackwell
8.4 Advanced Packaging
- 2.5D/3D packaging: CoWoS (NVIDIA), EMIB, Foveros, Wafer-on-Wafer (Graphcore)
- Importance: Enables higher bandwidth, lower power between dies
9. Educational Takeaways for Computing/Electronics
| Technology | Key Learning Concepts | Suggested Exercises/Projects |
|---|---|---|
| Blackwell/Chiplets | Modular SoC design, reticle limits, die-to-die I/O | Study UCIe specs, analyze multi-die tradeoffs |
| Wafer-Scale (Cerebras) | Yield engineering, defect tolerance, on-chip networks | Compare cluster vs wafer-scale for different workloads |
| LPUs/Groq | Deterministic execution, compiler-driven design, VLIW-like scheduling | Contrast static vs dynamic scheduling |
| RISC-V Accelerators (Tenstorrent) | Open ISA, domain-specific accelerators, mesh NoCs | Explore TT-Metal, write simple kernels |
| HBM | 3D memory, TSVs, bandwidth wall, memory hierarchy | Model bandwidth bottlenecks |
| CPO/Silicon Photonics | Optoelectronics, energy-per-bit, scale-up fabrics | Study power budgets for large clusters |
| Low Precision (MXFP4/FP6) | Quantization, numerical formats, accuracy/performance tradeoffs | Experiment with quantization effects |
| CPU-GPU Co-design | Heterogeneous computing, coherence, unified memory | NVLink-C2C architecture analysis |
10. Sources & Further Reading
Primary Vendor Sources
- NVIDIA - Blackwell, Grace, DGX
- Cerebras - Wafer-Scale Engines
- Groq - LPUs
- AMD - Instinct MI Series
- Intel - Gaudi
- Tenstorrent - RISC-V Accelerators
Technical Analysis
- IntuitionLabs AI Hardware Comparisons
- Hot Chips Symposium Proceedings / Hot Chips 2024
- MLCommons MLPerf Results - Real performance data
- SemiAnalysis - Industry analysis
- AnandTech - Deep technical reviews
Research Papers
- "A Comparison of the Cerebras Wafer-Scale Integration Technology with Nvidia GPU-based Systems" (arXiv:2503.11698)
- "Dissecting the NVIDIA Blackwell Architecture with Microbenchmarks" (arXiv:2507.10789)
- Various Hot Chips presentations from vendors