
“It’s like adding more lanes to the highway,” he observed. “We’re not necessarily making each car travel faster, we’re creating more capacity so more traffic can move through the system at the same time.”
Handling bursts, idling
However, an agentic workload doesn’t behave exactly like a traditional inference workload. “It is bursty, long running, but often idle,” Lohmeyer noted. While it might generate outputs and call tools, it still has to wait for a response, maybe from a human, “literally for days.” During that time, it doesn’t need to do anything; it just needs to preserve context. But if resources are tied up for that entire multi-day period, they’re being wasted “90% plus of the time,” he said.
Google Cloud’s open source TPU Sync is designed to address these issues by unifying memory, storage, and networking in software. The model-state layer for TPUs turns chip high bandwidth memory (HBM), host DRAM, shared storage, and the data center network into a “single, co-designed memory hierarchy,” the company said. It automatically tiers KV caches off the chip in milliseconds, streams context across nodes, and synchronizes RL weights to cut transfer latency and Time to First Token (TTFT) by more than 90%.




















