Our team is at the ready to create a customized plan for you to optimize and scale your business. Contact us today to set up a call.

Curated content and latest news for MangoBoost solutions

 unlocks the power of state-of-the-art GPUs for AI training systems with multi-GPUs. As an example, MangoBoost demonstrates linear scaling using our RDMA solution, for MLperf Training v4.1 Llama2-70B-LoRA workload (BF16), using 24x AMD MI300X GPUs across multiple nodes.

 to allow direct communication across GPUs to scale AI systems. Furthermore, MangoBoost HW-based 

in-line customizable congestion control and custom headers

 enable even more optimal solutions at scale.

Mango GPUBoost RDMA™ is ready to ship. Pre-orders are open for customers who would like to accelerate their AI infrastructure.

 and plans to offer an Ultra Ethernet solution in the near future.

Efficient GPU-to-GPU communication across multiple nodes is essential for large-scale AI training workloads, where dozens to thousands of GPUs work together to train complex models. 

 has emerged as a key technology for enabling this scalability, as it allows a direct data transfer between GPU memory on different nodes without the help of the CPU host significantly reducing the latency of GPU-to-GPU data transfers. Among various RDMA solutions,

has been shown to be a promising solution that offers unique advantages over traditional Infiniband for large-scale AI training environments[1].

One of RoCEv2’s primary advantages is its 

. Because it uses an IP header for routing, RoCEv2 can scale over larger, more complex network topologies than Infiniband. This IP-based routing capability makes RoCEv2 especially appealing in hyperscale AI infrastructures, where scalability is crucial for managing massive AI models and multi-node GPU configurations.

, which directly impacts the physical layout of the AI networking infrastructure. Ethernet switches typically support more ports than Infiniband switches of the same generation, reducing the level of switches that introduces latency and potential oversubscription. With RoCEv2’s use of high-port-count Ethernet switches, system architects can create

 streamlined, low-latency network designs

 that improve data flow across the entire GPU cluster.

a broad ecosystem, with multiple vendors supporting Ethernet-based solutions

. Unlike Infiniband, which is primarily produced by NVIDIA, RoCEv2 gives hyperscalers and enterprises the flexibility to choose from a variety of vendors, helping them to avoid vendor lock-in and maintain more control over their supply chain.

Figure 1 shows how MangoBoost set up a large-scale AI Training system using MangoBoost DPUs. Each server is equipped with eight MangoBoost DPUs and eight GPUs, linked via PCIe for GPU-NIC communication. In this system, high-bandwidth interconnects like AMD's Infinity Fabric or NVIDIA's NVLink manage intra-node communication. For inter-node communication, MangoBoost DPUs enable RoCEv2, connecting multiple nodes over Ethernet switches and creating a high-performance and scalable setup optimized for large-scale AI workloads.

Mango GPUBoost RDMA™ enables linear performance scale-up of MLPerf Training v4.1 Llama-70B-LoRA with AMD MI300X GPUs

Our Llama-70B LoRA training leveraged from 8 to 24 MI300X GPUs with MLPerf datasets and was conducted in full compliance with the MLPerf Training v4.1 standards using BF16 precision (code provided by MLPerf). The setup focused on 

maximizing GPU-to-GPU communication efficiency to boost throughput

. We used AMD’s Infinity Fabric for intra-node communication, while inter-node communication was powered by RoCEv2 RDMA, which is crucial for seamless scaling across multiple servers.

To ensure high-efficiency RoCEv2 performance, we installed MangoBoost DPUs, assigning one DPU per GPU in each server. This 

1:1 mapping enabled optimized data transfer across nodes

, taking advantage of RDMA’s capabilities to support 

. Each server used an identical configuration, providing a robust and scalable infrastructure capable of handling the intense demands of large language model (LLM) training.

 as we increased the number of servers, which is paramount in large-model training where hardware limitations and network overhead often constrain scalability. With our setup, we maintained nearly optimal performance even as additional resources were added.

we ensured strict adherence to the MLPerf v4.1 validation requirements

. This included following guidelines for hyperparameters, evaluation methodology, and quality targets, such as achieving a 0.925 evaluation loss threshold. We validated our setup against the Reference Convergence Point (RCP) rules, ensuring accuracy and reliability in producing high-quality training results for large language models.

Mango GPUBoost RDMA™ powers accelerated AI with industry-standard and customizable features

high-performance RoCEv2-based RNIC solution 

that unleashes the full potential of ethernet-based large-scale AI workloads. It provides greater network scalability alongside peer-to-peer communication between GPU and RNIC (e.g., GPUDirect, ROCmRDMA) to enable optimized connectivity among GPUs at scale.

 achieves impressive performance, sustaining 90% of line-rate throughput

 even as the number of QP (Queue Pair) connections increases. It also delivers 

compared to other standard RDMA network interface cards (RNICs) available on the market. Figure 3 demonstrates these advantages using results from Perftest, a widely-used RDMA benchmarking tool, where Mango GPUBoost RDMA™ achieves 3.4 times higher throughput and reduces 99.9% latency by 52%.

two important features (multipath and custom congestion control) 

to resolve well-known problems within RoCEv2.

First, due to the nature of the RoCEv2 protocol — which enforces packets to be delivered in the order they were sent — network hotspots often occur because of path collision [2]. To alleviate these hotspots and take advantage of multiple rich paths, Mango GPUBoost RDMA™ supports a 

unique out-of-order packet handling mechanism and provides multipath extension

Moreover, no single congestion control solution fits all use cases for RoCEv2. Although DCQCN has become the de-facto standard for RoCEv2, it is hard to configure in practice [3] and has been shown to be less effective for AI workloads [4]. Instead, each workload demands a tailored congestion control algorithm depending on its unique characteristics [5] (e.g., traffic pattern, incast, etc). To maximize workload performance, Mango GPUBoost RDMA™ offers a 

placeholder for custom congestion control

, allowing users to deploy their own algorithms directly on our RNIC.

Please check the product specification and more details 

Try our Mango GPUBoost RDMA™ - Pre-Order is now open!

We are ready for customers who want to innovate their AI infrastructure by accelerating network communication with Mango GPUBoost RDMA™. Our DPUs have been tested on various AI training and inference workloads with top-of-the-line GPUs, demonstrating its flexibility along with industry-leading performance. We support standard RoCEv2 with customizable features and compatibility with common server products.

We continue to innovate performance and scalability in AI infrastructure. As a member of the UltraEthernet Consortium, we also plan to showcase our UltraEthernet DPU in the near future.

References
[1] RDMA over Ethernet for Distributed Training at Meta Scale (Meta, SIGCOMM’24)
[2] Multi-Path Transport for RDMA in Datacenters (Microsoft, NSDI’18)
[3] HPCC: High Precision Congestion Control (Alibaba, SIGCOMM’19)
[4] Impact of RoCE Congestion Control Policies on Distributed Training of DNNs (Meta, HOTI’22)
[5] Datacenter Ethernet and RDMA: Issues at Hyperscale (Microsoft/Broadcom/HPE/Google, IEEE Computer’23)


The performance claims in this document are based on the internal cluster environment. Actual performance may vary depending on the server configuration. Software and workloads used in performance tests may have been optimized for performance only on MangoBoost products. Performance results are based on testing as of dates shown in configurations and may not reflect all publicly available updates. Results that are based on pre-production systems and components as well as results that have been estimated or simulated using MangoBoost reference platform for informational purposes only. Results may vary based on future changes to any systems, components, specifications, or configurations. Statements in this document that refer to future plans or expectations are forward-looking statements. These statements are based on current expectations and involve many risks and uncertainties that could cause actual results to differ materially from those expressed or implied in such statements. MangoBoost does not guarantee any specific outcome. Nothing contained herein is, or shall be relied upon as, a promise or representation or warranty as to future performance of MangoBoost or any MangoBoost product. The information contained herein shall not be deemed to expand in any way the scope or effect of any representations or warranties contained in the definitive agreement for MangoBoost products.

The information contained herein may not be reproduced in whole or in part without prior written consent of MangoBoost. The information presented in this document is for informational purposes only and may contain technical inaccuracies, omissions and typographical errors. The information contained herein is subject to change and may be rendered inaccurate for many reasons, including but not limited to product and roadmap changes, component and motherboard version changes, new model and/or product releases, product differences between differing manufacturers, software changes, BIOS flashes, firmware upgrades, or the like. MangoBoost assumes no obligation to update or otherwise correct or revise this information and MangoBoost reserves the right to make changes to the content hereof from time to time without any notice. Nothing contained herein is intended by MangoBoost, nor should it be relied upon, as a promise or a representation as to the future.

MANGOBOOST MAKES NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE CONTENTS HEREOF AND ASSUMES NO RESPONSIBILITY FOR ANY INACCURACIES, ERRORS OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION.

MangoBoost Demonstrates Mango GPUBoost RDMA™ with MLPerf Training v4.1 on a Multi-Node System Powered by AMD MI300X GPUs

World's First DPU-accelerated System Submission for MLPerf™ Storage. 

Mango StorageBoost™ showed impressive results in the MLPerf™ Storage v1.0 

, setting the precedent for full hardware acceleration for accessing disaggregated remote storage systems for large data centers using patent-pending Data Processing Unit (DPU) solutions. MangoBoost solutions are demonstrated across all MLPerf™ Storage v1.0 workloads, which include large language models (LLMs), scientific computing, and image segmentation.

MangoBoost Showed the Best-Performing Ethernet-Based System Result in MLPerf

In the single-host system category, MangoBoost's solution offers a 1.2x to 4.7x speedup (avg of 2.7x) over the 5 other single-host systems submitted.

In the multi-host system category, when linearly scaled to the same number of hosts, MangoBoost’s solution offers an estimated 1.4x to 7.8x speedup (avg of 4.3x) over the 15 other multi-host systems submitted.

MangoBoost Provides Near-Local SSD Performance by Accelerating the Entire NVMe/TCP Stack Fully on Hardware on Our DPU Solutions.

Our DPU solution accelerates the entire NVMe/TCP protocol stack, including TCP packet processing, by using specialized hardware in the DPU rather than relying on software running on CPUs (either on the host or on the DPU card itself).

MangoBoost’s DPUs are Easy to Deploy on Standard Ethernet-Based Systems.

Our DPUs work “out-of-the-box,” providing standard hardware and software interfaces that integrate into existing infrastructure without modification.

We chose to submit results on two different server configurations, with CPUs and SSDs from various major vendors for all GPU types and models in the MLPerf™ Storage Benchmark. This demonstrates the generality and robustness of our DPUs.

If interested in the MangoBoost AI solutions, please contact us at contact@mangoboost.io

Modern large-language models and other high-performance computing applications have scaled up in the complexity of their operations and sizes. This increase often leads to additional pressure not only on the compute and memory elements but also on the storage systems designed to support these large-scale applications. MLPerf™ Storage is created to benchmark modern storage systems and is designed to show performance in supporting AI/ML tasks.

To address the scalability and performance challenges, MangoBoost’s patent-pending DPU technology allows offloading of infrastructure and data-processing tasks from the host to datacenter-deployable FPGA PCIe cards. With our best-in-class NVMe/TCP Initiator(s) deployed on AI server(s) connected via standard ethernet to NVMe/TCP Target(s) deployed on storage server(s), we are able to deliver the best performance ethernet-based storage system across all evaluated benchmarks on top-of-the-line NVIDIA H100 and A100 GPUs as measured and published in MLPerf™ Storage v1.0 results.

shows an overview of AI infrastructure using MangoBoost’s DPU-accelerated storage. MangoBoost DPUs are deployed on GPU servers and storage servers to accelerate the feeding of training data to GPU. We introduce the three turn-key solutions for accelerating storage for AI training.

Mango StorageBoost™- NVMe/TCP Initiator (NTI) provides local NVMe SSD illusion of remote NVMe SSDs. The transport layer (TCP/IP) for remote NVMe access is fully offloaded onto the DPU and GPU server can save CPU cycles and improve training performance.

Mango StorageBoost™- NVMe/TCP Target (NTT) provides the entire NVMe/TCP target offloading on the storage server. NVMe/TCP data path is fully offloaded on NTT and Target only utilizes a single CPU core to handle control path operations.

Mango StorageBoost™- GPU Storage Boost (GSB) provides direct data transfer between DPU and GPUs via peer-to-peer PCIe communications. It eliminates the overhead of using a CPU bounce buffer for the data transfer with the GPU and improves GPU utilization.

In the following sections, we showcase MangoBoost’s MLPerf™ Storage v1.0 submission and introduce details of the above turn-key solutions applied to our systems. Then, we offer further details on our solutions.

World first DPU-accelerated NVMe/TCP submission for MLPerf™ Storage 

Storage systems play a crucial role in AI training. A bottleneck in the storage system results in the under-utilization of GPUs. This arises when GPUs are idle, waiting for the completion of I/O to read training data from storage. GPUs are an expensive resource, and we should keep GPU utilization high to minimize the completion time of AI model training.

The MLPerf™ Storage benchmark is currently the best environment for storage performance evaluation for AI workloads. Fig. 2 shows an overview of MLPerf™ Storage. One of the notable features in MLPerf™ Storage is "GPU emulation." This means from the perspective of the storage system evaluation; the GPU is only an external processor that ingests fetched data from storage. Since the real computation of the GPU is not necessary, MLPerf™ Storage emulates GPU computation with "sleep."

Additionally, MLPerf™ Storage provides many parameters that are used for emulating a variety of AI training workloads. Users can easily generate datasets with different size distributions and simulate various I/O patterns.

Fig. 3 presents our evaluation setup for MLPerf™ Storage. We utilize a single GPU server that accesses training data in a remote SSD over NVMe/TCP. NVMe/TCP is fully offloaded to NTI to save CPU cycles and maximize training throughput on GPU servers. The storage servers deploy NTT as a counterpart to NTI. The storage server offloads the entire data path processing onto the DPU, utilizing only a single CPU core to handle control path operations. NTT can saturate a 200GbE line with a single card while significantly reducing power consumption for CPU utilization.

To showcase the wide compatibility with standard servers and SSDs, we demonstrate our solution with two different types of storage servers. Each storage server is powered by either an Intel or AMD CPU and equipped with two SSD models from Samsung and Kioxia. Since both storage servers show almost identical performance, for simplicity, we report the results with one of the Storage Servers in this blog.

In the following section, we showcase our 

 result, highlighting our top-ranked Ethernet category scores.

MangoBoost shows the best-performing ethernet-based system result in MLPerf Storage v1.0.

Fig. 4 shows a direct comparison of our MLPerf™ Storage score against other Ethernet-based single-host submissions. The results show that the MangoBoost DPU-accelerated NVMe/TCP storage significantly outperforms other systems in AI training workloads. Our system significantly outperforms other systems on ResNet50 and UNET3D workloads, with an average performance increase of 2.7x, ranging from 1.2x to 4.7x, respectively.The Cosmoflow result is omitted because Nutanix and Hammerspace did not submit results for this workload.

Fig. 5 shows comparisons to Ethernet-based multi-host submissions. To estimate the relative performance of our single-host result, we normalized competitors' numbers by the number of hosts used in the evaluation. As the figure shows, our system outperforms others by 1.4x to 7.8x and 4.3x on average for ResNet50 and UNET3D workloads, respectively.

MangoBoost DPU solution provides near-local SSD performance for NVMe/TCP-attached storage

Finally, we demonstrate the superior performance of our NVMe/TCP solutions compared to systems using software NVMe/TCP, achieving performance on par with local NVMe. Local NVMe is an ideal system with minimal overhead for storage access, and software NVMe/TCP is the existing available solution (albeit slow). Our DPU solution is the first to unlock the full power of NVMe/TCP.

Fig. 6 shows a performance comparison of our solution versus "Local NVMe" and "SW NVMe/TCP" baselines. We show a 2.5x and 1.5x performance improvement in UNET3D and ResNet50, respectively, compared to SW NVMe/TCP. The limited computation resources severely limit the performance of SW NVMe/TCP since many cycles are spent processing the TCP/IP stack for remote NVMe access.

We also achieved the same score in UNET3D and ResNet50 compared to "Local NVMe." This is a great achievement since closing the performance gap completely is impossible to do without the state-of-the-art full hardware acceleration of NVMe/TCP with our DPU. It also shows that MangoBoost DPU allows true storage disaggregation with high-performance data center-grade SSDs.

NVMe/TCP is widely recognized for its scalability and ease of deployment compared to RDMA-based protocols like RoCEv2, which require specialized network hardware such as RDMA network cards and switches [

]. This makes NVMe/TCP a more accessible and flexible solution for many organizations. However, despite these advantages, NVMe/TCP has traditionally faced performance challenges due to higher overheads, making it less efficient for high-performance workloads like AI and machine learning.

MangoBoost addresses these limitations with a fully hardware-accelerated NVMe/TCP solution on both initiator and target-side servers. By leveraging our Data Processing Units (DPUs), MangoBoost is the first to overcome the inherent performance bottlenecks of NVMe/TCP, significantly enhancing both throughput and efficiency. This makes our NVMe/TCP solution an ideal choice for AI-driven applications, where high-performance storage is critical.

Our DPU solutions are designed to be deployed seamlessly into existing infrastructure. From a hardware perspective, MangoBoost’s DPU installs seamlessly into any standard PCIe slot and connects via standard Ethernet QSFP ports, similar to a typical network interface card (NIC). On the software side, the DPU is fully compatible with NVMe/PCIe and NVMe/TCP protocols, requiring no modifications to the user’s software environment.

Additionally, we offer a GPU Storage Boost solution. We co-presented this solution with AMD at SDC 2024 

, where we showed end-to-end training workloads running on AMD MI300X GPUs accelerated by MangoBoost DPUs, dramatically improving overall system performance. This solution further enables peer-to-peer direct data communication between GPUs and remote storage (on top of NTI). However, this was not included in our submission since the MLPerf Storage setup only emulates the GPU hardware, so there is no real GPU hardware to communicate peer-to-peer within this setup.

Solution 1: Mango StorageBoost™- NVMe/TCP Initiator (NTI).

Mango StorageBoost™- NVMe/TCP Initiator (NTI) is a high-performance NVMe-oF initiator solution that unlocks the full potential of storage disaggregation. By offloading the entire NVMe/TCP stack into hardware, NTI delivers unmatched full-duplex line-rate performance while significantly reducing CPU consumption.

NTI integrates seamlessly into any initiator servers that connect using standard ethernet (including AI GPU servers). By exposing the DPU as a standard NVMe PCIe device, existing storage systems can leverage NTI without the need for any software modifications. Furthermore, NTI can connect to any NVMe/TCP target servers over a standard TCP/IP network.

Solution 2: Mango StorageBoost™- NVMe/TCP Target (NTT).

Mango StorageBoost™ - NVMe/TCP Target is a groundbreaking NVMe-oF target solution, powering the storage system with outstanding performance. It enables disaggregated storage servers to be connected to the network through the standard TCP/IP protocol stack. By fully offloading and accelerating both the TCP/IP and NVMe-oF processing, it entirely eliminates the CPU intervention from the I/O path.

NTT decapsulates the TCP packets into NVMe-oF commands and translates them into standard NVMe commands. By utilizing PCIe peer-to-peer technology, NTT directly sends those commands to the peer NVMe SSDs without any CPU consumption. Additionally, its modular and flexible architecture also enables more dynamic customer-driven storage systems such as data reduction and reliability.

Solution 3: Mango StorageBoost™- GPU Storage Boost (GSB).

Mango StorageBoost™- GPU Storage Boost (GSB) is an essential software solution designed to dramatically enhance the efficiency of GPU data movement, enabling unprecedented performance capabilities. By establishing a direct data path for Direct Memory Access (DMA) transfers between GPU memory and storage, GSB eliminates the need for bounce buffers through the CPU, and bypasses CPU involvement entirely. It facilitates peer-to-peer communication within GPU systems, regardless of whether the storage is local or remote. Furthermore, GSB offers a nearly identical syntax and semantic structure to traditional POSIX File APIs, making integration straightforward. This allows for efficient, filesystem-based data management with minimal adaptation, enabling users to harness the full power of the GPU systems without sacrificing ease of use.

When combined with Mango StorageBoost™- NVMe/TCP Initiator (NTI), GSB enables highly efficient GPU data movement, even over remote storage systems based on the TCP/IP transport layer, without any CPU involvement. Typically, handling TCP/IP protocols and the NVMe/TCP stack requires CPU intervention and memory copying, which introduces latency and reduces overall efficiency. NTI addresses these challenges by offloading these tasks to dedicated hardware, allowing the host to leverage the DMA engine as if it were directly connected to an NVMe PCI device. NTI effectively removes the performance and efficiency constraints associated with CPU resources during remote storage access, while GSB eliminates similar limitations for GPU data movement. By utilizing both solutions together, users can unlock unparalleled synergies, dramatically improving data flow efficiency and reducing system bottlenecks.

If you are interested in NTI, NTT and GPU Storage Boost, please email contact@mangoboost.io

https://mlcommons.org/2024/09/mlperf-storage-v1-0-benchmark-results/

https://mlcommons.org/2023/06/introducing-the-mlperf-storage-benchmark-suite/


[3] Answering Your Questions: NVMe™/TCP: What You Need to Know About the Specification, https://nvmexpress.org/answering-your-questions-nvme-tcp-what-you-need-to-know-about-the-specification-webcast-qa/
[4] Pavilion compares RoCE and TCP NVMe over Fabrics performance, https://blocksandfiles.com/2018/08/16/pavilion-compares-roce-and-tcp-nvme-over-fabrics-performance/
[5] SRNIC: A Scalable Architecture for RDMA NICs, NSDI’23, https://www.usenix.org/conference/nsdi23/presentation/wang-zilong
[6] RDMA over Ethernet for Distributed Training at Meta Scale, SIGCOMM’24, https://dl.acm.org/doi/abs/10.1145/3651890.3672233
[7] Accelerating GPU Server Access to Network-Attached Disaggregated Storage using Data Processing Unit (DPU),

https://www.sniadeveloper.org/events/agenda/session/666

The performance claims in this document are based on the internal cluster environment. Actual performance may vary depending on the server configuration. Software and workloads used in performance tests may have been optimized for performance only on MangoBoost products. Performance results are based on testing as of dates shown in configurations and may not reflect all publicly available updates. Results that are based on pre-production systems and components as well as results that have been estimated or simulated using MangoBoost reference platform for informational purposes only. Results may vary based on future changes to any systems, components, specifications, or configurations. Statements in this document that refer to future plans or expectations are forward-looking statements. These statements are based on current expectations and involve many risks and uncertainties that could cause actual results to differ materially from those expressed or implied in such statements. MangoBoost does not guarantee any specific outcome. Nothing contained herein is, or shall be relied upon as, a promise or representation or warranty as to future performance of MangoBoost or any MangoBoost product. The information contained herein shall not be deemed to expand in any way the scope or effect of any representations or warranties contained in the definitive agreement for MangoBoost products.

MangoBoost Demonstrates Turn-Key Storage Solution for AI Training with MLPerf® Storage v1.0 

New MLPerf® result on AMD Instinct™ MI300X GPUs.

 Our first MLPerf Inference v4.1 results on AMD MI300X GPUs achieve 23.5K tokens per second on the Llama-70B benchmark! 

 The productization challenge: Going from the MLPerf inference benchmark to production deployment is no easy task. It demands a significant effort, including bring-up, co-optimization, and validation of the full-stack AI inference serving software.

— a ready-to-deploy, full-stack AI inference serving container. It includes MangoBoost’s inference serving software, and comes pre-packaged with everything needed to run optimized Large Language Models on any GPU, including the AMD MI300X.
-  

 Easily reproduce valid FP8 MLPerf Inference v4.1 Llama2-70B results on AMD MI300X systems. 

 The Mango LLMBoost™ package includes optimized server software for LLM inference, along with various industry-standard APIs for easy integration into your application.
-  

MangoBoost demonstrates cutting-edge performance on AMD MI300X GPUs with various LLMs, including Llama-70B and Llama Guard-7B.

 — a hardware-accelerated frontend gateway web server designed for GPU-accelerated AI inference clusters, delivering unparalleled throughput and ultra-low latency.
-  

 Mango WebBoost utilizes the MangoBoost Hardware Data Processing Unit (DPU) solution on the AMD Alveo™ U45N FPGA card to offload complete TCP/IP networking tasks with stateful processing, ensuring consistently low average and tail latency for reliable performance.
-  

 Our solution has been successfully integrated and demonstrated with Nginx, one of the most widely used web servers. It can also be seamlessly integrated into other web servers without requiring modification.

If you're interested in the above solutions, please contact us at contact@mangoboost.io

regarding MLPerf Inference v4.1 have included the first published results of systems with AMD’s MI300X GPUs on the Llama-70B benchmark. Our solution delivers state-of-the-art performance, computing 23.5K tokens per second using 8 MI300X GPUs on our custom servers. 

Beyond the MLPerf benchmark, deploying the MI300X in a production-grade AI inference serving solution requires additional capabilities, including support for diverse web API endpoints, multi-model serving, and integration hooks for cluster management. Additionally, a production inference serving cluster requires a front-end web server to efficiently manage multiple back-end GPU servers.

MangoBoost introduces two turn-key solutions to address above said challenges.

 — a ready-to-deploy, full-stack AI inference serving container. It includes MangoBoost’s inference serving software, and comes pre-packaged with everything needed to run optimized Large Language Models on any GPU, including the AMD MI300X.

 — a hardware-accelerated frontend gateway web server designed for GPU-accelerated AI inference clusters, delivering unparalleled throughput and ultra-low latency.

We detail these solutions in the next sections, and report case studies that quantify their benefits.


LLMBoost - — Ready-to-deploy LLM inference software demonstrating industry-leading performance on AMD MI300X GPUs.

MangoBoost’s LLMBoost AI inference server software is a fully optimized, turn-key solution, ready for deployment as a containerized software package. It delivers exceptional LLM inference performance on AI servers equipped with any GPU, including AMD’s MI300X accelerators. Purpose-built for the next generation of resource-hungry LLMs and the GPUs required to run them, LLMBoost leverages advanced parallelism strategies to accelerate a wide range of models — from Llama-70B to Deepseek-R1.

Figure 1 highlights two key areas of optimization within LLMBoost: the inference engine and the model deployment. The inference engine leverages advanced GPU kernels to enable multidimensional parallelism, complemented by tuning of the ROCm software stack for expensive matrix multiplication (GEMM) kernels. Alongside these kernel optimizations, the inference engine utilizes advanced memory management techniques to reduce the cost of data transfer and eliminate memory fragmentation within the GPU. Additionally, LLMBoost offers automatic deployment optimization, in which it finds the optimal GPU deployment configuration to maximize performance without the need for manual user tuning.

Our inference server offers multiple endpoint options to ensure seamless integration with existing solutions. Synchronous and asynchronous REST APIs provide flexible request management, while WebSockets enable an efficient interface for streaming applications. Advanced dynamic scheduling algorithms intelligently allocate incoming requests to the most suitable GPU, maximizing efficiency across diverse deployment scenarios. Customers can fine-tune their deployments to prioritize throughput, latency, or a balanced hybrid approach, delivering a fully customizable inference server. This ensures consistently state-of-the-art performance on cutting-edge GPU hardware, meeting the diverse needs of a wide range of applications.

Our containers also expose standard Prometheus endpoints for performance monitoring, enabling seamless integration with existing Kubernetes deployments and supporting Horizontal Pod Autoscaling. Additionally, a single container can serve multiple models, offering users a high degree of flexibility to tailor the inference server to their specific requirements.

Case Study : MLPerf Inference v4.1 Llama-70B and Llama Guard-7B.

 We compared LLMboost’s performance to vLLM, a popular open-source inference engine for LLMs [

]. For this evaluation, we selected two prominent LLMs: Llama2-70B and Llama Guard-7B. The tests were conducted on a high-performance Supermicro server featuring 8 AMD MI300X GPUs. Detailed server configurations are provided in the table below.

For the Llama2-70B model, we strictly followed the MLPerf Inference v4.1 Closed Division rules to ensure the validity and integrity of the results on the LLMBoost configuration with all 8 GPUs active. This setup established a consistent and rigorous foundation for comparison. To maintain fairness, identical conditions were applied to all subsequent experiments.

In the MLPerf Inference v4.1 Llama-70B test case, we achieved a performance of 22.8K tokens per second, which is within 3% of AMD's published MLPerf result of 23.5K tokens per second. It's worth noting that our system uses a different CPU than AMD's official MLPerf submission, which may explain the slight performance difference. As the MLPerf Inference Benchmark does not yet provide validation rules for Llama Guard, we applied the same configuration used for Llama2-70B to maintain consistency in our testing methodology.

As illustrated in Figure 2, LLMBoost achieved 5.2x to 6.0x higher throughput over vLLM, driven by three key features: (1) Multi-dimensional parallelism, (2) dynamic scheduling across the 8 different GPUs, and (3) a lightweight, streamlined interface. This significant enhancement highlights LLMBoost's capability to accelerate inference workloads, establishing it as an essential tool for optimizing large-scale AI deployments.


WebBoost — Accelerating the Nginx Web Server with MangoBoost's Stateful TCP/IP Hardware Acceleration on the AMD U45N FPGA Card

WebBoost is a web-serving container designed to run Nginx on front-end servers equipped with hardware acceleration powered by MangoBoost's DPU (Data Processing Unit) solution. This solution supports multiple FPGA cards, and in this paper, we demonstrate it on the AMD U45N. The LLMBoost and WebBoost containers are engineered to work seamlessly together, enabling coordinated operations to stream input data to GPUs and return generated output tokens to clients.

Figure 3 illustrates the network processing layers between incoming LLM requests and the GPU of an LLM server. Clients submit their LLM requests via a REST API, which are then sent to the Nginx front-end gateway. Nginx collects incoming requests and distributes them to back-end GPU computation containers.

On a front-end server equipped with WebBoost, MangoBoost’s DPU solution provides stateful hardware acceleration of the TCP/IP networking stack, significantly enhancing the performance of web server software like Nginx. WebBoost delivers consistently low average and tail latency while boosting throughput—key factors for optimizing LLM inference performance. Furthermore, WebBoost seamlessly integrates with existing systems by hooking into standard POSIX socket API calls, requiring no modifications to user applications. This makes it a reliable, high-performance, and easy-to-use solution for modern AI deployments.

To quantify the performance benefits of MangoBoost’s WebBoost when integrated with Nginx, we conducted a series of benchmarks comparing our solution against a setup using Linux’s standard TCP kernel software stack. The TCP kernel software stack relies on the NVIDIA® ConnectX®-6 network card, which is widely used in typical data center environments.

We utilized the wrk benchmarking tool to generate network requests to the Nginx web server. This tool is designed for measuring HTTP performance, making it ideal for evaluating the throughput and latency of our solution. In this evaluation, wrk sends HTTP POST requests to the Nginx web server. The requests contain sentences typically input by users of LLM AI inference serving. The Nginx then responds with sentences as part of the HTTP responses. The configuration details of our server hardware and software are in the table below.

Web-Serving Front-End Server Configuration

The evaluation results are summarized in the graphs in Fig. 4.

Throughput: WebBoost outperforms the Linux TCP stack significantly, achieving a 2x improvement in throughput. This means that Nginx can handle 100% more requests per second when using the WebBoost solution compared to the standard TCP stack.

Latency: WebBoost also demonstrates 0.53x and 0.54x reductions on the 50th and 90th percentile latency, respectively.

These results clearly illustrate the efficiency gains from offloading TCP packet processing to our DPU. By handling TCP connections more effectively, the WebBoost solution not only increases throughput but also significantly reduces the time it takes to process requests, leading to a more responsive and scalable Nginx deployment.

Nginx is a cornerstone of modern web infrastructure, and optimizing its performance is crucial for maintaining the efficiency and reliability of web services. By offloading TCP packet processing to a DPU, we can significantly enhance Nginx’s capabilities, allowing it to handle more traffic with lower latency. Our solution represents a leap forward in web server performance, setting a new standard for what is possible in high-demand environments.

If you are interested in LLMBoost and WebBoost, please email contact@mangoboost.io.

MLPerf Inference: Datacenter Benchmark Suite Results

Unveiling MLPerf® Results on AMD Instinct™ MI300X Accelerators

Smith, Alan, et al. "Realizing the AMD Exascale Heterogeneous Processor Vision: Industry Product." 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 2024.

vLLM in AMD MLPerf Inference v4.1 submission


[6] Result not verified by MLCommons Association.

Blog

Ready for
a boost?

Blog

MangoBoost Demonstrates Mango GPUBoost RDMA™ with MLPerf Training v4.1 on a Multi-Node System Powered by AMD MI300X GPUs

MangoBoost Demonstrates Turn-Key Storage Solution for AI Training with MLPerf® Storage v1.0

MangoBoost Demonstrates a Turn-Key LLM Inference Serving Solution on AMD Instinct™ MI300X GPUs with the MLPerf® Inference v4.1 Llama Benchmark

Ready for a boost?

Ready for
a boost?