SambaNova Pits Its Engineering Against Nvidia For Agentic AI

Agentic caching is a memory management technique that enables two key capabilities: first, the ability to cache inference context for multiple models and rapidly load it into SRAM for processing, and second, the ability to hot-swap models by moving them between different memory tiers (DRAM, HBM, and SRAM) as needed based on changing workload demands. This is particularly important for agentic AI systems that orchestrate multiple models and process requests in near real-time, as it reduces latency and improves efficiency compared to traditional GPU-centric systems that struggle with production latency and cost requirements for such workloads.

SambaNova uses a three-tier memory architecture with SRAM inside the chip (positioned very close to compute cores for high bandwidth and locality), HBM on the chip package, and DRAM on the accelerator card rather than the host server. This design allows SambaNova to overlap computation and communication, reducing the need for the latest HBM specifications. In contrast, Nvidia's Blackwell GPUs top out at 72 GPU sockets in a rackscale system and rely on NVSwitch for shared memory. SambaNova's approach enables the company to use HBM2E instead of the supply-chain-constrained HBM3E while still achieving superior performance, particularly for inference workloads where throughput and efficiency are critical.