Our team is at the ready to create a customized plan for you to optimize and scale your business. Contact us today to set up a call.

Curated content and latest news for MangoBoost solutions

MangoBoost continues to define the cutting edge of AI scalability with our latest 

. Building on our track record, this round establishes several new industry firsts, including the first MLPerf multi-node results for the AMD Instinct™ MI325X GPU.

With a close partnership with AMD, MangoBoost submission this round features the multi-node results on AMD InstinctTM MI300X and the AMD InstinctTM MI325X GPUs, our submission demonstrates significant performance gains and the power of our 

 MangoBoost submissions in this round feature the new AMD Instinct™ MI325X multi-node systems showing strong scalability and proving our stack is ready for next-generation AI accelerators.

 Continuing our strong multi-node scaling performance on AMD GPUs from MLPerf Training 5.0, MangoBoost’s multi-node results on AMD GPUs in this round proves our unique scalability expertise—an achievement driven by software intelligence, optimized ROCm™ integration, and deep collaborations with AMD to push the frontiers of large-scale LLM training.

 We showcased our platform's continuous improvement, achieving a 

 Expanding the Open Ecosystem: This round marks our first-ever joint Training submission on AMD GPUs with our partner, Supermicro, demonstrating the growing momentum and collaborative power of AMD's open, partner-driven mission.

 Our MI300X multi-node setup was accelerated by our 

 solution, enabling highly efficient, large-scale training over RoCEv2 networks.

Advancing Multi-Node Scalability on AMD Instinct™ MI325X

In MLPerf Training v5.1, MangoBoost has set a critical new milestone. Our submission represents the first-ever 

public demonstration of multi-node scalability for the AMD Instinct™ MI325X GPU

performance, scalability, and cost efficiency

 for large-scale LLM training. This achievement validates that our Mango LLMBoost™ software stack is optimized and ready to harness the power of next-generation hardware from day one.

In addition to this milestone, we improved performance across all submitted AMD GPUs, delivering a 

 compared to our own 5.0 submission and a 

 compared to AMD's 5.0 submission. This demonstrates the continuous optimization and maturity of our LLMBoost platform.

This milestone was also recognized by industry leaders:

Mangoboost has collaborated with us at AMD across several rounds of competitive MI Instinct MLPerf submissions for both training and inference workloads. Through 

 LLMBoost stack — with rich features such as autotuning, parallelism, memory management, and optimal scheduling — Mangoboost has extended the capabilities of our ROCm™ AI software stack and system solutions for LLM and GenAI training and inference. In this MLPerf 5.1 round, using GPUBoost and LLMBoost, the team successfully delivered competitive AMD Instinct™ MI325X and MI300X submissions. Congratulations to Team Mangoboost! — 

Meena Arunachalam, Fellow and Director, AI Performance Design Engineering, AMD

Powering these record-setting results is MangoBoost’s end-to-end AI platform, which co-designs software and hardware for maximum efficiency.

Mango LLMBoost™ AI Enterprise MLOps Software:

 Our core software solution for AI training and finetuning. It features patent-pending technology for auto-tuning, model parallelism, batch scheduling, and memory management.

 Our DPU-based hardware acceleration, which enables customizable, low-latency, and high-throughput communication across nodes.

Figure 1: MangoBoost's end-to-end stack, showing how the LLMBoost software platform and the GPUBoost hardware acceleration work together over AMD Instinct™ hardware with RDMA-enabled interconnects to accelerate training

Spotlight on Mango GPUBoost™: Advanced RDMA for AI

Multi-node training isn’t just about adding more GPUs—it's about ensuring those GPUs communicate efficiently and reliably across nodes. That’s where MangoBoost’s GPUBoost™ RDMA comes in, as depicted in Figure 2.

Built on a custom RoCEv2-enabled DPU architecture, Mango GPUBoost™ delivers line-rate performance and scalability. It is 

 and offers significant advantages over standard RNICs. Its 

customized congestion control and load balancing

, which ensures stable, high-throughput communication even under heavy, large-scale AI workloads.

, up to 3.2Tbps RDMA (RoCEv2) throughput in total (up to 400Gbps per NIC)

Packet spray, programmable congestion control, selective re-transmission.

 Enables large-scale RoCEv2 deployment without requiring switch configuration or additional congestion control setup

Peer-to-peer communication between RNIC and GPU (i.e., ROCm™ – RDMA support)

Workload-specific optimization hooks using RoCEv2 extended headers

High connection scalability: Maintains line-rate throughput at thousands of QPs—3.4× higher than standard RNICs under heavy load

MangoBoost’s RDMA stack ensures that your training isn’t bottlenecked by the network—one of the biggest pain points in distributed deep learning today.

The Mango LLMBoost™ Platform: Train Anywhere, Any Model

Mango LLMBoost™ is designed to be seamless and flexible, freeing ML developers from hardware or model lock-in.

 Our platform is rigorously tested to support popular open models and multi-modal models right out of the box, including:
✅ Dense models like 


✅ Sparse Mixture-of-Experts models like Mixtral 
✅ 

Multi-modal models like Whisper and Llava


✅ LoRA and PEFT-style fine-tuning variants

 LLMBoost is optimized for a diverse range of GPUs from both NVIDIA and AMD, allowing you to make the best hardware choices based on price, performance, or availability.

 LLMBoost enables developers to effortlessly scale from a single-GPU proof of concept to large-scale, multi-GPU deployments with one-line deployment and robust API integration.

 Full support for Kubernetes & SLURM-compatible environments

Beyond Training: A Unified Platform for GenAI Inference

MangoBoost’s LLMBoost platform is a comprehensive GenAI solution that excels at both training and inference. Our leadership was also highlighted in the recent 

 round, where we achieved record-breaking performance and several industry firsts.

Read more about our groundbreaking inference results in our 

 Achieved the highest MLPerf inference performance for Llama2-70B (

 Demonstrated near-linear performance scaling across multi-architecture clusters combining 

 validated LLMBoost across diverse hardware platforms.

Beyond MLPerf, LLMBoost delivers breakthrough results on diverse workloads, proving up to 

 on models like Llama4-Scout MoE, and up to 

MangoBoost AI Infrastructure Hardware Solutions

Beyond our LLMBoost software, MangoBoost offers a suite of DPU-powered hardware acceleration solutions for AI and cloud infrastructure:

 Provides RDMA acceleration for multi-node training and inference via RoCEv2, as showcased in this round.

 Offloads web-serving and TCP stacks to dramatically reduce CPU utilization.

 Delivers high-performance NVMe initiator and target solutions for scalable AI storage systems.

This is just the beginning. Our R&D team is already implementing next-generation communication optimizations, hybrid parallelism strategies, and topology-aware scheduling to push multi-node performance even further.

anyone can replicate our results with ease

If your organization is building large-scale AI models or exploring alternative GPU architectures, MangoBoost is your partner in unlocking full-stack efficiency and scalability.

MangoBoost Pushes AI Scalability in MLPerf Training v5.1: First-Ever Multi-Node MI325X Results Highlight AMD's Scaling Capabilities

 Highest MLPerf inference performance for 

 Demonstrated near linear performance scaling across multi-architecture clusters combining AMD 

 First and only third-party submission showcasing AMD Instinct™ 

 across diverse hardware platforms from various server vendors.

breakthrough results across diverse workloads

Interested in trying our LLMBoost software

In each MLPerf round, a single new system is selected for a performance audit via a vote by all submitters. This selection process is designed to promote transparency and trust in reported metrics, and it often highlights systems with the most novel architectures. 

In the v5.1 round, MangoBoost’s heterogeneous system (4x MI300X + 2x MI325X) was chosen as the audit candidate

A third-party audit team thoroughly reviewed our submission, and we 

successfully passed MLCommons’ auditing process

. This recognition validates not only the record-breaking performance of our architecture, but also its 

reliability and credibility in real-world scenarios

unlocks the full potential of heterogeneous and multi-node GPU clusters

The scale and innovation of MangoBoost’s submission have been recognized across the industry:

David Kanter – Founder & Head of MLPerf, MLCommons: 

“I am thrilled to see MangoBoost’s creativity in pushing the limits of LLM serving with the first-ever heterogeneous results for MLPerf Inference. This submission integrates multiple generations of AMD GPUs with MangoBoost's LLMBoost platform to deliver impressive performance for Llama2-70B at 160K tokens/s, underscoring the power of software and system architecture in inference serving.”

“We are thrilled with our MLPerf Inference v5.1 co-submission with MangoBoost. Together, we set a new performance record for Llama2-70B using AMD Instinct™ MI355X GPUs, scaling to 4-node and 8-node configurations. MangoBoost also delivered the first heterogeneous multi-node submission showcasing the combined power of AMD Instinct MI300X and MI325X GPUs, along with strong multi-node results on Dell and Supermicro servers. With nine high-performance MLPerf inference submissions, MangoBoost’s LLMBoost GenAI, powered by AMD ROCm™, provides seamless scalability and easy deployment for enterprise AI workloads.”

Frank Han – Distinguished Member of Technical Staff, Dell: 

“Dell's collaboration with MangoBoost is built on a foundation that began with MLPerf Training v5.0. In our official MLPerf Inference v5.1 co-submission (Dell_MangoBoost), MangoBoost’s LLMBoost software demonstrated impressive linear scaling across multi-node Dell PowerEdge server configurations with AMD accelerators.”


LLMBoost™: GenAI Software for Any GPU, Any Scale

turn-key, high-performance inference stack

 with broad compatibility and advanced optimizations:

 Launch models instantly using Docker with 

 for maximum control, security, and performance.

99.5% lower cost per million tokens vs. Ollama

 for efficient distributed inference on homogeneous or heterogeneous GPU clusters.

 across the AI stack for real-time visibility into GPU utilization, memory usage, latency, throughput, and queue states.Exports metrics in standard formats for 

Full-Stack Auto-Tuning (Patent Pending) : 

Dynamically adjusts runtime configurations and model deployment strategies based on 

model architecture, hardware topology, and live traffic patterns

dynamic scaling, load balancing, straggler mitigation, and automatic failover

 to ensure high availability and consistent latency during heavy workloads.

Deep Collaborations with Industry Leaders

 is built on deep, strategic collaborations with industry leaders such as 

(listed alphabetically), alongside validation on Gigabyte platforms. These partnerships ensure 

 across a wide range of hardware ecosystems to 

 requires more than just software innovation — it depends on 

 and validation across diverse platforms. By working directly with leading technology partners, MangoBoost ensures that 

 is not only optimized for the latest AMD Instinct™ GPUs, but also proven on a variety of server architectures from major vendors. These collaborations demonstrate that enterprises can 

trust LLMBoost™ to deliver consistent, high-performance results

 on their infrastructure of choice, whether in 

single-node systems, multi-node clusters, or heterogeneous GPU deployments

Through co-engineering efforts with AMD, MangoBoost gained early access to next-generation AMD Instinct™ GPUs such as the MI355X, enabling us to 

tightly integrate LLMBoost™ with the ROCm™ software stack

, achieving 648K tok/s on 64 GPUs and delivering 

record-setting performance in the open division with 8-node MI355X clusters

Read more on AMD’s perspective in their 

In collaboration with Dell Technologies, MangoBoost validated 

, demonstrating reliable performance across enterprise-scale deployments. This collaboration included validation on 

multi-node MI300X clusters with near-linear scalability

LLMBoost™’s enterprise-ready performance

MangoBoost Collaboration with Supermicro:


Working with Supermicro, MangoBoost validated 

 across a wide range of single-node, multi-node, and heterogeneous GPU deployments. This included the 

MI300X nodes (Supermicro AS-8125GS-TNMR2)

MI325X nodes (Supermicro AS-8126GS-TNMR) 

LLMBoost™ can seamlessly orchestrate workloads across different GPU generations

 while maintaining consistent performance and efficiency. 

 achieved by MangoBoost and our partners across 

, demonstrating LLMBoost™’s scalability across diverse hardware platforms.

 Consistent near-linear performance gains were observed as the number of nodes increased, even in heterogeneous setups combining 

, validating LLMBoost™’s optimizations with AMD’s latest GPUs.

servers (listed alphabetically), ensuring consistent performance across hardware ecosystems.

Below are some of the key highlights from this round:

As a result of deep collaboration with AMD and full integration of the optimized ROCm software stack, our submission delivered 

, outperforming the next-best NVIDIA result by 

 higher performance on average across all submissions by primary metric.

In the open division, our joint submission with AMD scaled an 

 of LLMBoost™ on the latest GPU architectures.

2. First-Ever Heterogeneous GPU Deployment

For the first time in MLPerf history, MangoBoost submitted results using 

, proving LLMBoost™’s ability to efficiently orchestrate workloads across multiple GPU generations.

, allowing them to integrate newer GPUs into their infrastructure 

intelligent load balancing, optimized communication libraries, and full-stack auto-config tuning

 across a wide range of workloads — from 

highly complex vision-text (multi-modal) applications

. Its ability to scale across different models and workloads ensures 

enterprise-grade reliability and efficiency

1. Text-Only Models: Superior Throughput and Cost Efficiency

, LLMBoost™ consistently outperforms competing solutions like 

LLMBoost™ achieves up to 186× faster performance than Ollama and 4× faster than vLLM

, setting a new benchmark for throughput and cost efficiency.

This unmatched performance translates into 

scale inference workloads at a fraction of the cost

2. High-Performance Vision-Text Multi-Modal Deployments

 remain minimal, even as the number of images per prompt increases from 

LLMBoost™ maintains stable throughput and low latency

LLMBoost™ as the preferred choice for vision-text workloads

high-performance, scalable, and cost-efficient solution

Getting started with LLMBoost™ is as simple as selecting the model you want on HuggingFace and then running one command.

Beyond Software: MangoBoost DPU Solutions

Alongside LLMBoost™, MangoBoost accelerates infrastructure with 

 RDMA acceleration for multi-node training and inference.

 TCP/IP stack offloading for optimized CPU utilization.

 High-performance NVMe stack for AI workloads.

To experience the record-setting performance of 

Pioneering Multinode Heterogeneous Inference: MangoBoost and AMD, in Collaboration with OEM Partners, Sets Record Llama2-70B Performance in MLPerf Inference v5.1

Record Breaking Performance for DPU-accelerated NVMe/TCP system submission for MLPerf™ Storage.

Best-performing ethernet-based systems result in MLPerf Storage v2.0.

Near-local SSD performance by accelerating the entire NVMe/TCP stack fully on hardware on our DPU solutions and achieving line-rate performance on 400G network.

Lower TCO and hardware cost for high-performance storage.

Easy to deploy on standard systems. Our DPU solutions provide the standard interface in both hardware and software without modification to the existing infrastructure.

Try it now. If interested in the MangoBoost AI solutions, please contact us at contact@mangoboost.io

In this MLPerf Storage 2.0 round, MangoBoost is proud to announce how our Mango StorageBoostTM addresses the scalability and performance challenges with its patent-pending DPU technology. StorageBoostTM technology offloads infrastructure and data-processing tasks from the host to datacenter-deployable FPGA PCIe cards. By deploying our best-in-class NVMe/TCP Initiator on the host and NVMe/TCP Target on the storage server, we further push the performance of 3D-UNet training on both A100 and H100 configuration making it the leading Ethernet-based storage system for AI training.

DPU-accelerated NVMe/TCP submission for MLPerf™ Storage Solutions

Modern high-performance computing applications, including large-language models, are growing in complexity and size. This trend places significant demands not only on compute and memory, but also on storage systems. Storage I/O bottlenecks can lead to underutilized GPUs. As models increase in size, the need for high-performance, high-capacity storage becomes even more critical. While technologies like NVMe/TCP enable remote block devices for ML systems and help meet capacity demands, these remote storage solutions often experience performance degradation due to the CPU overhead associated with NVMe/TCP software stack.

To address this performance bottleneck, DPU-accelerated Mango StorageBoostTM Initiator and Mango StorageBoostTM Target fully offload NVMe/TCP to the hardware, delivering line-rate performance and allows remote storage to operate at the speed similar to a local high-performance NVMe.

In this round, we have set up our evaluation for MLPerf Storage 2.0 Training workload by connecting the host with Mango StorageBoostTM Initiator DPU with the storage server with Mango StorageBoostTM Target through ethernet switch with a 400G network connection between the host and the storage server as shown in Figure 1. The host then emulates the demand of ML workloads by running multiple copies of emulated A100 and H100 3D-UNet traces on the storage system with the goal of testing the maximum number of GPUs that can be connected to the storage system.

MangoBoost shows the best-performing Ethernet-based system result in MLPerf Storage v2.0 and provides near-local SSD performance.

Measuring the performance efficiency of a remote storage system targeting ML workloads can be done through two key metrics: number of GPUs that the storage system can handle normalized by the number of clients, and the number of GPUs normalized by the network bandwidth. Figure 2 shows the overall GPU scalability per unit relative to other submitters in the Fabric-attached Block Storage category (both NVMe/TCP and NVMe/RDMA).These results show how MangoBoost’s DPU solution delivers the best performance: 6.2x for 3D-UNET A100 and 1.25x-7.5x for 3D-UNET H100, and how offloading NVMe/TCP to hardware allows more GPUs to be connected to the storage system.

Figure 3 shows the performance measured as the number of GPUs that can be connected normalized by the network bandwidth for both NVMe/RDMA and NVMe/TCP submitters. This result highlights the efficiency of our solution in utilizing the available network bandwidth as MangoBoost solutions provide the best performance: 1.57x for 3D-UNET A100 and 1.27x-2.05x for 3D-UNET H100.

Figure 4 also provides further performance breakdown across two of our MLPerf Storage submissions (MLPerf Storage 1.0 and MLPerf Storage 2.0), showing that the DPUs can achieve near line-rate performance regardless of GPU architectures (H100 and A100 architectures).

The combination of the Mango StorageBoostTM Initiator DPU on the host and the Mango StorageBoostTM Target DPU on the target also allows ML workloads to operate remote block storage at performance near that of local storage. This is highlighted in Figure 5 through the performance comparison between MangoBoost solutions against local SSD baselines. Our system deployed four Solidigm D7-PS1030s on the target. We normalized the score by the number of SSDs to compare with other non-Fabric attached submissions using a single SSD.

Industry-leading DPU storage solutions, delivering higher performance and lower TCO

While MangoBoost is the only submitter focusing on DPU solutions, this section now focuses on the performance comparison of Mango StorageBoost DPUs against the most apple-to-apple DPU solution available in the market – the NVIDIA BlueField-3 400G DPU, using the same MLPerf Storage 2.0 methodology but replacing MangoBoost DPU with NVIDIA BlueField-3 as shown in Figure 6.

To highlight our performance and cost-efficiency, we rerun MLPerf Storage 2.0 on 3D-UNET benchmarks using the same methodology as the official MLPerf Storage 2.0. Figure 7 compares the scores of BlueField-3 running NVMe/TCP and NVMe/RDMA with host CPU and MangoBoost NVMe/TCP full offloading. The results demonstrate that Mango StorageBoost significantly outperforms BlueField-3 NVMe/TCP and even shows higher performance than BlueField-3 NVMe/RDMA.

Additionally, we also ran the Llama3-8B checkpointing workload on the same setup (this result is not included in v2.0, we will submit the checkpointing results in the next round). Figure 8 shows the comparison of checkpointing performance for different storage configurations. The setup used only one client and did not fully saturate the given bandwidth. The I/O pattern of MLPerf Storage v2.0 checkpointing is not optimized for throughput; shorter storage system latency is crucial for achieving higher throughput on a single client. The results demonstrate that Mango StorageBoost significantly outperformed BlueFIeld-3 (NVMe/TCP) and achieved near local SSD performance.

The superior performance advantage directly translates into TCO reduction. Figure 9 shows the TCO reduction of MangoBoost DPU compared to BlueField-3; as more storage is deployed, the TCO gap widens. For more details, please check our 

, which provides further analysis on throughput and cost breakdown.

Mango StorageBoost™ Traditional Storage, JBoF, JBoD and EBoF Solutions.

Mango StorageBoost™ is a high-performance vendor-agnostic easy-to-use solution that allows AI systems running on any host and GPUs to scale their ML workloads.

Solution 1: Mango StorageBoost™- NVMe/TCP Initiator (NTI).

Mango StorageBoost™- NVMe/TCP Initiator (NTI) is a high-performance NVMe-oF initiator solution that unlocks the full potential of storage disaggregation. By offloading the entire NVMe/TCP stack into hardware, NTI delivers unmatched full-duplex line-rate performance with zero CPU consumption.

Mango StorageBoost™- NVMe/TCP Initiator (NTI) integrates seamlessly into NVMe/TCP initiator servers. By exposing the DPU as a standard NVMe PCIe device, existing storage systems can leverage NTI without the need for any SW modifications. Furthermore, NTI can connect to any NVMe/TCP target servers over a standard TCP/IP network.

Solution 2: Mango StorageBoost™- NVMe/TCP Target (NTT).

Mango StorageBoost™ - NVMe/TCP Target is a groundbreaking NVMe-oF solution, powering the storage system with outstanding performance. It enables disaggregated storage servers to be connected to the network through the standard TCP/IP protocol stack. By fully offloading and accelerating both the TCP/IP and NVMe-oF processing, it entirely eliminates the CPU intervention from the I/O path.

On top of Mango NetworkBoost™ - TCP/IP, NTT decapsulates the TCP packets into NVMe-oF commands and translates them into standard NVMe commands. By utilizing PCIe peer-to-peer technology, NTT directly sends those commands to the peer NVMe SSDs without any CPU consumption. Additionally, its modular and flexible architecture also enables more dynamic customer-driven storage systems such as data reduction and reliability.

Solution 3: Mango StorageBoost™- GPU Storage Boost (GSB).

Mango StorageBoost™- GPU Storage Boost (GSB) is an essential software solution designed to dramatically enhance the efficiency of GPU data movement, enabling unprecedented performance capabilities. By establishing a direct data path for DMA (Direct Memory Access) transfers between GPU memory and storage, GSB eliminates the need for bounce buffers through the CPU, and bypasses CPU involvement entirely. It facilitates peer-to-peer communication within GPU systems, regardless of whether the storage is local or remote. Furthermore, GSB offers a nearly identical syntax and semantic structure to traditional POSIX File APIs, making integration straightforward. This allows for efficient, filesystem-based data management with minimal adaptation, enabling users to harness the full power of the GPU systems without sacrificing ease of use.

When combined with Mango StorageBoost™- NVMe/TCP Initiator (NTI), GSB enables highly efficient GPU data movement, even over remote storage systems based on the TCP/IP transport layer, without any CPU involvement. Typically, handling TCP/IP protocols and the NVMe/TCP stack requires CPU intervention and memory copying, which introduces latency and reduces overall efficiency. NTI addresses these challenges by offloading these tasks to dedicated hardware, allowing the host to leverage the DMA engine as if it were directly connected to an NVMe PCI device. NTI effectively removes the performance and efficiency constraints associated with CPU resources during remote storage access, while GSB eliminates similar limitations for GPU data movement. By utilizing both solutions together, users can unlock unparalleled synergies, dramatically improving data flow efficiency and reducing system bottlenecks.

With the three solutions combined, MangoBoost DPU not only can be used for a traditional remote block storage, but can be used to manage headless storage network including Just-a-bunch-of-Flash (JBoF), Just-a-bunch-of-Disk (JBoD), and Ethernet-bunch-of-Flash (EBoD) to further lower the TCO of high-performance storage system.

Interested in how our DPU solution can deliver higher performance and lower your TCO, try our solutions today by contacting us at 

Blog

Ready for
a boost?

Blog

MangoBoost Pushes AI Scalability in MLPerf Training v5.1: First-Ever Multi-Node MI325X Results Highlight AMD's Scaling Capabilities

Pioneering Multinode Heterogeneous Inference: MangoBoost and AMD, in Collaboration with OEM Partners, Sets Record Llama2-70B Performance in MLPerf Inference v5.1

MangoBoost Demonstrates Record-Breaking Performance in AI Training Storage Solution with MLPerf® Storage v2.0

MangoBoost Sets a New MLPerf Training v5.0 Standard for Multi-nodes Llama2-70B-LoRA on AMD Instinct™ MI300X GPU!

MangoBoost Sets The Highest MLPerf Inference v5.0 Result in History for Llama2-70B Offline!

MangoBoost Demonstrates Mango GPUBoost RDMA™ with MLPerf Training v4.1 on a Multi-Node System Powered by AMD MI300X GPUs

MangoBoost Demonstrates Turn-Key Storage Solution for AI Training with MLPerf® Storage v1.0

MangoBoost Demonstrates a Turn-Key LLM Inference Serving Solution on AMD Instinct™ MI300X GPUs with the MLPerf® Inference v4.1 Llama Benchmark

Ready for a boost?

Ready for
a boost?