Huawei Unveils CloudMatrix System for Large-Scale AI Model Inference
Huawei has released technical documentation detailing its CloudMatrix384 system, a specialized computing platform built around 384 Ascend-910C neural processing units designed for inference of large AI models. The system includes CloudMatrix-Infer, a serving stack optimized for very large Mixture-of-Experts language models such as DeepSeek-R1 with 671 billion parameters.
Technical Architecture and Innovations
The CloudMatrix system implements several key technical innovations in its architecture. Central to the design is a Unified-Bus mesh network that enables direct communication between all NPUs and CPUs with minimal latency overhead. According to the technical report, this interconnect adds less than 3% bandwidth loss and under 1 microsecond of latency during inter-node communication.
The system employs a disaggregated approach to AI inference, separating the process into three distinct resource pools: prefill NPUs, decode NPUs, and a DRAM-backed caching cluster. These components access each other directly over the Unified-Bus, eliminating the need for locality-aware schedulers and allowing for more efficient resource utilization.
"This approach effectively flattens the memory hierarchy, enabling simpler stateless schedulers and pooled caches," explains the technical documentation, which notes that the design allows idle DRAM anywhere in the system to become available for key-value or model caching.
Performance Metrics and Comparisons
Performance data included in the report indicates the CloudMatrix system achieves 6,688 tokens per second per NPU during prefill operations with 4K token prompts, which translates to 4.45 tokens per second per TFLOPS. For decode operations, the system delivers 1,943 tokens per second per NPU with total processing overhead time under 50 milliseconds when running with batch size 96.
The report states these efficiency metrics exceed published figures for comparable systems running on NVIDIA H100 and H800 hardware. When constrained to stricter latency requirements of 15 milliseconds, the system maintains throughput at 538 tokens per second by automatically adjusting batch sizes.
Quantization techniques implemented in the system allow for INT8 precision where appropriate while maintaining BF16/FP32 precision for sensitive operations. The report indicates this mixed-precision approach preserves model accuracy while substantially reducing memory requirements, with benchmark results showing the INT8 model matching or nearly matching FP-baseline performance across English, Code, Math, and Chinese evaluation suites.
Economic Considerations
The technical documentation includes approximate cost estimates for the system. A complete CloudMatrix384 rack is estimated at approximately ¥50 million (around $6.9 million), with individual Ascend 910C chips priced at roughly ¥150,000 . Based on throughput measurements, this translates to an estimated cost of approximately ¥2 per million tokens generated.
The system's Elastic Memory Service provides additional operational efficiency benefits, enabling model switching in approximately 5 seconds versus 281 seconds with traditional node-local DRAM approaches, while requiring only 1x the DRAM footprint compared to 8x for conventional approaches.
Technical Implementation Details
Several specialized techniques contribute to the system's performance characteristics:
-
Large-scale Expert Parallelism - The system implements expert parallelism at up to 320 experts , distributing exactly one expert per NPU die, while using data parallelism for dense layers.
-
Hybrid and Micro-batch Pipelining - For prefill operations, the system uses hybrid tensor parallelism and expert parallelism for Multi-Head Latent Attention with two-micro-batch pipelining. Decode operations use overlapped attention and MoE streams per micro-batch.
-
Multi-Token Prediction - The system generates two tokens per step with speculative validation, providing between 6-49% additional decode throughput depending on batch size.
-
Ascend-native Kernels - Custom INT8 GEMM implementations and fused operators specifically designed for the Ascend architecture's cube/vector/DMA engines help maintain performance with the chosen parallelism strategy.
Market Implications
The CloudMatrix system represents a fully domestic Chinese AI compute solution, from NPU silicon to system software. The technical report notes that with China's manufacturing capabilities, the system has potential for mass production.
The DeepSeek model's strategy of providing open weights for community adoption while offering commercial add-ons for private deployments aligns with what the report describes as Beijing's policy focus on application value over raw parameter counts.
For organizations working with large-scale AI models, the system potentially offers a cost-effective approach to inference, particularly for data-intensive tasks such as model distillation or self-training without extensive labeling operations.
Future Developments
The technical documentation outlines several areas for future development, including denser optical interconnects and more sophisticated scheduling algorithms. The authors suggest the current architecture could scale to even larger configurations than the 384-NPU implementation detailed in the report.
Note: This analysis is based on technical information provided in the documentation. Potential investors should conduct their own research and consult financial advisors before making investment decisions.