Incident sheet · /infrastructure

Filed
Read
4 min · 821 words
Severity
Routine

Lambda Puts Nvidia Co-Packaged Optics Into 10,000-GPU Production Cluster

Lambda deployed Nvidia co-packaged optics across 10,000+ Blackwell Ultra GPUs in Quebec, reporting 10.1% pod-scale latency gains and a modeled 658 kW of freed power for large clusters.

Lambda puts Nvidia photonics to work in 10,000-GPU AI cluster
FeedLambda puts Nvidia photonics to work in 10,000-GPU AI cluster — AI-generated

Event log

  1. Lambda connected more than 10,000 Blackwell Ultra GPUs using Nvidia Photonics co-packaged optics switches in Quebec.
  2. Lambda reported 3.8% rack-scale and 10.1% pod-scale latency improvements versus pluggable optics at the same throughput.
  3. Lambda's June analysis modeled 658 kW of freed power in a 10,368-GPU cluster, enough for about 470 additional GPUs.
  4. Nvidia claims up to 5x power efficiency, 15x reliability and 4x fewer lasers than conventional pluggable optics.
  5. The switches have 144 ports at 800 Gb/s, liquid cooling and 18 field-replaceable external light sources.

By Rebecca Stone

Lambda has deployed Nvidia's co-packaged optics technology in a production AI cluster connecting more than 10,000 Blackwell Ultra GPUs at its Quebec facility. The installation pairs Nvidia Photonics switches with GB300 NVL72 systems and marks the first move of the architecture beyond demonstrations into a large-scale production environment.

Co-packaged optics (CPO) integrates optical engines directly alongside switching silicon, shortening the electrical path between components and cutting the power and heat associated with conventional pluggable transceivers. Lambda reported a 3.8% improvement in rack-scale latency and a 10.1% improvement in pod-scale latency versus conventional pluggable optics, at the same throughput. Nvidia confirmed the figures reflect network latency measured with a microbenchmark, not end-to-end application performance.

Why does networking power matter at this scale?

As AI clusters grow, the networks connecting accelerators consume more electricity, generate more heat and add failure points. In power-constrained data centers, that overhead competes directly with compute capacity. Lambda estimates a 128,000-GPU data center using conventional pluggable optics would require approximately 655,000 discrete transceiver modules across its switching fabric.

"When the facility is power-capped (and nearly all are), every watt the network uses is a watt that isn't going to compute," said Matt Kimball, vice president and principal analyst of data center technologies at Moor Insights & Strategy. "So network efficiency is no longer a networking problem – it's also a capacity and revenue problem."

What are the claimed efficiency gains?

Nvidia claims Photonics delivers up to five times better power efficiency and up to 15 times higher reliability than conventional pluggable optics, with four times fewer lasers. An Nvidia spokesperson said the reliability figure reflects field-performance testing of signal-path reliability.

At the component level, Nvidia said each CPO switch replaces 25-watt, 1.6-Tb/s transceivers with optical components consuming five watts for the same bandwidth. Across 72 transceivers, that amounts to a potential 1.44-kilowatt reduction per switch in optical-component power, not total switch power.

Lambda separately modeled larger installations in a June technical analysis. The company estimated a 10,368-GPU GB300 cluster could free 658 kW by using co-packaged optics instead of conventional pluggable optics — enough power for approximately 470 additional Blackwell Ultra GPUs. That figure is a modeled cluster-level estimate, not a measurement from the Quebec deployment. Lambda's analysis also compared approximately 3.95 kW per CPO switch against 7 kW for a conventional switch configuration; those figures cover a broader switch-power comparison and are not directly comparable with Nvidia's optical-component estimate.

The Quebec switches carry 144 ports running at 800 Gb/s, liquid cooling and 18 removable external light sources. Technicians can replace the light sources in the field, while the optical engines remain integrated into the switch package.

How serviceable is the technology?

Co-packaged optics changes maintenance practice. Conventional pluggable transceivers can be removed and replaced individually; with CPO, the optical engines are part of the switch assembly. Nvidia said it and its ecosystem partners test components and assemble switches before shipment so equipment arrives in a known-good state. The company did not detail how operators would repair or replace an integrated optical engine after a failure.

Ron Westfall, vice president and practice lead for networking and infrastructure at HyperFrame Research, cautioned that localized optical or thermal failures could have broader consequences for the switch assembly. He cited manufacturing yields, supply-chain constraints and limited interoperability as potential adoption obstacles.

"Operators must weigh these significant day-to-day energy and uptime benefits against stricter serviceability requirements and deeper architectural lock-in to Nvidia's end-to-end networking stack," Westfall said.

What does this mean for large-scale AI networks?

Kimball said reducing networking power frees capacity for revenue-generating compute, particularly where electricity availability limits expansion. He expects co-packaged optics to become the default architecture for the largest scale-out AI networks over the next several product generations, though technical and operational challenges will take time to resolve.

"In an AI factory, the network is a business accelerator. Every interruption stalls the job, and every watt counts," said Gilad Shainer, senior vice president of networking at Nvidia, in a statement. "Nvidia Photonics cuts network power, improves resiliency and delivers up to 15x higher reliability, so Lambda's customers get more productive GPU hours from their investment."

"Networking has to keep pace with the demands of increasingly large AI clusters," said Dave Ward, president of cloud services at Lambda, in a statement. "Bringing co-packaged optics into a deployment of this scale is an important step toward building the efficient and resilient infrastructure needed to support the next generation of AI."

One production installation does not establish a market. Operators must weigh efficiency and reliability gains against serviceability requirements, manufacturing challenges and tighter ties to Nvidia's networking ecosystem.

via techtarget.com (Source)

Filed under

  • co-packaged-optics
  • nvidia-photonics
  • ai-infrastructure
  • gpu-clusters
  • data-center-networking
Share this article:

More from Rebecca Stone

Rebecca Stone

Show full bio

Senior reporter covering money, funding and market shifts around enterprise information technology and the leaders who run at CIO Monitor.

199 articles

The newsroom digest

One email a day with the stories that matter. No noise.

By subscribing you agree to the privacy policy.

Cross-references

‹ Previous articleNext article ›