Why AI Training Clusters Are Pushing Peering Into the Data Center
Most of the time, a GPU node in a training cluster is busy talking to the switches right above it and swapping gradients with the rest of its compute pod over the fabric. But it may need to get outside the Data Center. It grabs dataset shards from an object store and pushes checkpoints back every few minutes. More and more, it also has to exchange parameters with a second data center running the other half of the job. All of this used to go over the same transit link that handled the usual outbound traffic, monitoring, remote access, software updates, you name it.
A checkpoint write dumps several gigabytes onto the fabric at once, and it does this on a set schedule for as long as the run is active. This isn't like a one-off spike from a user request, the traffic keeps coming. It can keep a transit port close to maxed out for the whole job.
Some jobs just won't fit in one data center. If a cluster needs more power or space than a single room can handle, you end up splitting it across two sites and running the job across both. GPUs on both sides have to exchange gradients before moving to the next step. If any of that traffic has to cross between buildings, that link is now in the critical path for every step.
A transit circuit that had capacity before the training job started can reach its capacity immediately. At that point, the operator is left choosing between a bigger port, a second circuit, or a different path for this traffic.
Most of the outbound traffic in an AI job is headed to a known place. The object storage provider with the datasets is usually already reachable over a regional exchange like FD-IX, on the same fabric as your border router. Same goes for the cloud region providing extra compute, and in a multi-site setup, the second data center. If you send this traffic over transit, you pay for it twice, once to leave your network, and again as it takes an upstream path that isn't built for this specific route. Setting up a direct BGP session to that ASN puts the traffic on the shortest path and keeps it off your transit port.
Bringing that session in-house usually means new border hardware. The border router needs another port. In a lot of setups, you need a 400G or 800G optic where a 100G port used to be enough for general internet traffic. That optic and its cross-connect to the exchange fabric end up in the same cabinet as the training network's uplinks. Now you've got more power draw and cooling to deal with, since optics at that speed add a real load to the row. That's rack space and power you could have used for another GPU.
On the routing side, you make some policy changes. Set local preference so the new peer route wins over transit. Use a prefix list to keep only the subnets you want, maybe the storage provider's or the partner data center's, on that path. Everything else still goes out over transit. You monitor the session like any other BGP session, since the transit path is still your fallback if the peer drops.
If the peering session to storage flaps during a checkpoint write, the write can time out before the router switches back to transit. The job either retries or stalls. Moving this traffic off transit doesn't mean you can skip having a working backup path.
