• AI chip performance depends on more than NPU counts and FLOPS.
• Memory, interconnects, storage, software, and power also shape performance.

Adding more accelerators does not guarantee a proportional increase in AI performance. As systems scale across hundreds or thousands of processors, compute has to be supported by enough memory bandwidth, interconnect capacity, storage throughput, and power to keep those processors working.

That complicates the way AI infrastructure is compared. Peak FLOPS and accelerator counts remain useful specifications, but neither shows how effectively a complete system trains or serves a model.

Huawei put that issue into focus at HUAWEI CONNECT 2026 with its latest Ascend roadmap and Atlas SuperPoD systems. The company also introduced changes to its optical interconnect and storage architecture, extending the performance discussion beyond the NPU itself.

Huawei has said intra-cluster communications account for more than 40% of total training time in traditional 100,000-NPU clusters. Adding theoretical compute therefore does not necessarily produce the same increase in workload performance.

Comparing AI systems increasingly requires looking at compute, memory, accelerator utilisation, interconnect performance, storage, software, and power rather than processor specifications alone.

From chip performance to system performance

Huawei’s own Ascend generations show why processor specifications still matter, but cannot be considered in isolation.

The Atlas 900 A3 supports up to 384 Ascend 910C processors operating within a SuperPoD. Huawei lists roughly 300 PFLOPS of compute, 784 GB/s of bidirectional device-to-device bandwidth, and 48 TB of aggregate on-chip memory for the system.

The Ascend 950 generation increased several specifications beyond compute. Huawei said the processor provides 2 TB/s of interconnect bandwidth, about 2.5 times that of the Ascend 910C, alongside higher memory capacity and memory-access bandwidth.

Huawei says Ascend 960 is designed to double compute performance, memory capacity, memory-access bandwidth, and the number of interconnect ports compared with Ascend 950.

Those gains are linked: higher processor performance also requires memory and communication bandwidth that can keep pace with the additional compute.

Huawei’s full Atlas 950 configuration supports up to 8,192 Ascend 950 processors and is rated at 8 EFLOPS at FP8 and 16 EFLOPS at FP4. Huawei has also published training and inference throughput figures for the system, although those are company-reported measurements rather than results from a common cross-vendor workload benchmark.

The Atlas 960E announced at this year’s HUAWEI CONNECT supports 4,096 Ascend 960 processors and is also rated at 8 EFLOPS at FP8 and 16 EFLOPS at FP4.

The systems reach the same claimed FP8 performance with different processor counts. Huawei rates each Ascend 960 at twice the FP8 performance of Ascend 950, allowing Atlas 960E to reach the same theoretical total with half as many NPUs.

That means NPU count alone does not provide a complete system-performance comparison. Peak FLOPS also has to be considered alongside processor architecture, memory, interconnect design, and workload conditions.

The figures cannot be directly translated into real-world model performance. Numerical precision, model architecture, workload size, software optimisation, memory configuration, and communications overhead all affect how much theoretical compute is used.

Accelerator utilisation adds another measure once workloads span large numbers of processors. Processors that are waiting for data or synchronisation cannot use their full theoretical capacity.

Huawei’s UnifiedBus architecture is designed to address that issue by connecting processors, memory, networking, and storage through a common interconnect architecture and supporting communication across physical servers.

Huawei had previously said existing inter-cabinet NPU bandwidth was about five times below the requirement it had set for its newer Atlas systems. It also identified inter-cabinet latency of about three microseconds as being above its design target.

Distributed training requires processors to exchange parameters and intermediate results throughout a workload. As those systems scale, interconnect bandwidth and latency become part of the overall performance equation.

The Atlas 960E also introduces Huawei’s Hi-ONE near-packaged optical, or NPO, technology. Huawei says the system uses 5,500 Hi-ONE units instead of the 48,000 800G optical modules that would otherwise be required.

The company claims that configuration reduces power consumption by more than 550 kW and supports 99.8% system availability. These are Huawei-reported figures and have not been established through a common cross-vendor benchmark.

The interconnect design adds further system-level measures. Bandwidth, latency, power consumption, component count, and availability can change even when the processor’s headline performance figure does not.

Storage and software enter the benchmark

Training and inference place different demands on storage, making bandwidth alone an incomplete measure.

A storage platform supporting model training has to deliver data quickly enough to prevent accelerators from waiting for input. MLPerf Storage provides standardised tests for that relationship. Version 3.0, released in early September, also added a KV-cache benchmark for storage used in LLM inference.

Huawei’s earlier OceanStor A800 participated in MLPerf Storage testing. Huawei reported that an 8U dual-node configuration sustained 698 GiB/s in the 3D U-Net workload while supporting 255 simulated H100 accelerators at more than 90% accelerator utilisation.

The result links storage throughput directly to accelerator utilisation, measuring whether the storage system can supply data quickly enough to keep the processors working.

Huawei’s OceanStor M900, introduced at HUAWEI CONNECT 2026, addresses a different workload. It is designed around the KV cache generated during inference rather than primarily supplying training data.

M900 extends KV-cache capacity from memory close to the accelerator to an SSD-based storage tier. Huawei says a single cluster supports up to 64 PB of pooled KV-cache capacity, with 60-microsecond access latency and 40 TB/s of aggregate bandwidth.

Huawei also claims the system doubled inference-cluster token throughput and reduced time to first token by half in its AI programming workload tests. Those results are based on Huawei’s own configurations.

MLPerf Storage 3.0’s KV-cache test provides an independent framework for measuring this type of workload, although no published M900 result was included with Huawei’s announcement.

A800 and M900 therefore need to be assessed against different measures. Training storage can be evaluated through sustained data throughput and accelerator utilisation, while inference adds metrics such as time to first token, token throughput, latency, concurrency, and cache capacity.

Huawei is not alone in moving AI infrastructure towards system-level design. Nvidia’s rack-scale platforms integrate GPUs with CPUs, NVLink, networking, and data processing units, while AMD’s Helios architecture combines Instinct accelerators, EPYC processors, Pensando networking, and ROCm software.

The architectures use different processors, interconnects, memory designs, and software stacks. Comparing them only through accelerator counts or theoretical FLOPS would leave out several parts of the system that affect workload performance.

Equivalent workload testing would need to control for factors such as model, precision, batch size, software environment, and operating conditions.

Software adds another variable. Huawei’s Ascend platform uses CANN, Nvidia’s GPU platform uses CUDA, and AMD’s Instinct platform uses ROCm.

Framework support, libraries, and optimisation determine how efficiently workloads use the underlying hardware. Higher theoretical performance does not by itself establish higher application performance when workloads require additional engineering or optimisation for another platform.

What enterprises actually need to measure

Workload-level benchmarks can show how effectively the different parts of an AI system work together. The relevant measurements differ between training and inference.

Training can be measured through time to train, tokens processed per second, accelerator utilisation, scaling efficiency, and the amount of compute required to reach a specified result.

Inference introduces different measures, including tokens per second, time to first token, concurrent requests, memory consumption, and response latency. Long-context workloads also place more emphasis on KV-cache capacity and access performance.

Scaling efficiency becomes important as processor counts increase. If doubling the number of accelerators produces substantially less than double the workload throughput, the bottleneck can lie in communications, memory, storage, software, or another part of the system.

Huawei’s comparison between Atlas generations provides one example. Atlas 950 and Atlas 960E have the same claimed peak FP8 and FP4 totals despite using different processor counts and interconnect implementations.

A common workload benchmark would be needed to determine whether those architectural differences produce higher utilisation, lower communications overhead, or better training and inference throughput.

Power consumption adds another system-level measure. Two platforms delivering similar workload throughput can have different operating profiles if their electricity requirements differ.

Huawei’s claimed reduction of more than 550 kW through the Hi-ONE design provides one example, although equivalent measurements would be required before that figure could be compared with competing architectures.

Performance per watt relates energy consumption to the amount of work completed, while total power measures the system’s overall electricity use.

Power also contributes to another operational measure for inference: cost per token.

The specifications disclosed at HUAWEI CONNECT 2026 are not enough to calculate a comparable cost-per-token figure. That would require hardware acquisition costs, electricity prices, utilisation, software overhead, memory and storage requirements, system lifetime, and measured workload throughput under equivalent conditions.

Cost per token therefore depends on several parts of the infrastructure rather than the processor alone. Utilisation, memory requirements, networking, storage, and power consumption can all affect operating costs even when headline compute performance is higher.

Huawei’s Atlas 950 and Atlas 960E illustrate the wider benchmarking problem. Both carry Huawei’s claimed 8 EFLOPS FP8 rating, but they reach that figure with different processor counts and system architectures.

Determining whether one delivers better workload performance requires equivalent workload testing under comparable conditions, alongside measurements of utilisation, memory, interconnect performance, software, and power.

Want to learn more about AI and big data from industry leaders? Check out AI & Big Data Expo taking place in Amsterdam, California, and London. The comprehensive event is part of TechEx and is co-located with other leading technology events, click here for more information.

Tech Wire Asia is powered by TechForge Media. Explore other upcoming enterprise technology events and webinars here.