With power, datacenter capacity, and other supply chain constraints, AI infrastructure is in short supply, and service providers will rent all the compute they can get, even if it means working with competing cloud providers.
Together AI is the latest example. This week the service provider announced a $240 million deal to run its open weights inference platform on a “large cluster” of Nvidia GPUs housed in IBM Cloud.
“Together AI selected IBM with Nvidia because of their innovative product roadmaps and their ability to deliver GPU capacity at the pace required for rapid AI scaling and lowest token cost,” IBM's announcement says.
Translation: Together AI tapped IBM because Big Blue had the capacity it needed when it needed it.
Together AI sits toward the top of the AI inference ecosystem. Its business model largely revolves around renting compute from cloud or neocloud providers, many of which themselves rent floorspace and capacity from bit barn operators and operate competing services.
More recently, the company has begun deploying GPU compute in datacenters in Maryland, Memphis, and Sweden, but like most AI service providers, its main value proposition remains making it easier for users to consume the hardware for applications like inference, fine-tuning, and training. For inference, this largely boils down to an OpenAI-compatible API endpoint.
The service provider isn’t particularly picky about what hardware that API endpoint runs on top of either, so long as price performance is favorable. For example, Together AI will be deploying services atop SambaNova’s heterogeneous compute platform built in collaboration with Intel and leveraging Nvidia GPUs for prefill processing. Those systems went live in Vector Core Compute’s new AI bit barn earlier this year.
MORE CONTEXT
In this case, Together AI is getting Nvidia B300s, no doubt because that’s what IBM had to offer them. Announced in early 2025, the HGX B300 platform isn’t Nvidia’s most powerful offering. Unlike the 72-GPU rack systems that CEO Jensen Huang likes to show off every opportunity he gets, the HGX platform is much more conventional and can be deployed in traditional air-cooled datacenters.
The 14-15 kW boxes each contain eight B300 GPUs that are interconnected by a combination of NVLink in the box, and Nvidia’s Spectrum-X Ethernet fabrics between them. The deployment, IBM says, is its first large scale use of B300s for inference applications.
The compute capacity is expected to come online in the first quarter of 2027. ®
