Our team is at the ready to create a customized plan for you to optimize and scale your business. Contact us today to set up a call.

Curated content and latest news for MangoBoost solutions

 unlocks the power of state-of-the-art GPUs for AI training systems with multi-GPUs. As an example, MangoBoost demonstrates linear scaling using our RDMA solution, for MLperf Training v4.1 Llama2-70B-LoRA workload (BF16), using 24x AMD MI300X GPUs across multiple nodes.

 to allow direct communication across GPUs to scale AI systems. Furthermore, MangoBoost HW-based 

in-line customizable congestion control and custom headers

 enable even more optimal solutions at scale.

Mango GPUBoost RDMA™ is ready to ship. Pre-orders are open for customers who would like to accelerate their AI infrastructure.

 and plans to offer an Ultra Ethernet solution in the near future.

Efficient GPU-to-GPU communication across multiple nodes is essential for large-scale AI training workloads, where dozens to thousands of GPUs work together to train complex models. 

 has emerged as a key technology for enabling this scalability, as it allows a direct data transfer between GPU memory on different nodes without the help of the CPU host significantly reducing the latency of GPU-to-GPU data transfers. Among various RDMA solutions,

has been shown to be a promising solution that offers unique advantages over traditional Infiniband for large-scale AI training environments[1].

One of RoCEv2’s primary advantages is its 

. Because it uses an IP header for routing, RoCEv2 can scale over larger, more complex network topologies than Infiniband. This IP-based routing capability makes RoCEv2 especially appealing in hyperscale AI infrastructures, where scalability is crucial for managing massive AI models and multi-node GPU configurations.

, which directly impacts the physical layout of the AI networking infrastructure. Ethernet switches typically support more ports than Infiniband switches of the same generation, reducing the level of switches that introduces latency and potential oversubscription. With RoCEv2’s use of high-port-count Ethernet switches, system architects can create

 streamlined, low-latency network designs

 that improve data flow across the entire GPU cluster.

a broad ecosystem, with multiple vendors supporting Ethernet-based solutions

. Unlike Infiniband, which is primarily produced by NVIDIA, RoCEv2 gives hyperscalers and enterprises the flexibility to choose from a variety of vendors, helping them to avoid vendor lock-in and maintain more control over their supply chain.

Figure 1 shows how MangoBoost set up a large-scale AI Training system using MangoBoost DPUs. Each server is equipped with eight MangoBoost DPUs and eight GPUs, linked via PCIe for GPU-NIC communication. In this system, high-bandwidth interconnects like AMD's Infinity Fabric or NVIDIA's NVLink manage intra-node communication. For inter-node communication, MangoBoost DPUs enable RoCEv2, connecting multiple nodes over Ethernet switches and creating a high-performance and scalable setup optimized for large-scale AI workloads.

Mango GPUBoost RDMA™ enables linear performance scale-up of MLPerf Training v4.1 Llama-70B-LoRA with AMD MI300X GPUs

Our Llama-70B LoRA training leveraged from 8 to 24 MI300X GPUs with MLPerf datasets and was conducted in full compliance with the MLPerf Training v4.1 standards using BF16 precision (code provided by MLPerf). The setup focused on 

maximizing GPU-to-GPU communication efficiency to boost throughput

. We used AMD’s Infinity Fabric for intra-node communication, while inter-node communication was powered by RoCEv2 RDMA, which is crucial for seamless scaling across multiple servers.

To ensure high-efficiency RoCEv2 performance, we installed MangoBoost DPUs, assigning one DPU per GPU in each server. This 

1:1 mapping enabled optimized data transfer across nodes

, taking advantage of RDMA’s capabilities to support 

. Each server used an identical configuration, providing a robust and scalable infrastructure capable of handling the intense demands of large language model (LLM) training.

 as we increased the number of servers, which is paramount in large-model training where hardware limitations and network overhead often constrain scalability. With our setup, we maintained nearly optimal performance even as additional resources were added.

we ensured strict adherence to the MLPerf v4.1 validation requirements

. This included following guidelines for hyperparameters, evaluation methodology, and quality targets, such as achieving a 0.925 evaluation loss threshold. We validated our setup against the Reference Convergence Point (RCP) rules, ensuring accuracy and reliability in producing high-quality training results for large language models.

Mango GPUBoost RDMA™ powers accelerated AI with industry-standard and customizable features

high-performance RoCEv2-based RNIC solution 

that unleashes the full potential of ethernet-based large-scale AI workloads. It provides greater network scalability alongside peer-to-peer communication between GPU and RNIC (e.g., GPUDirect, ROCmRDMA) to enable optimized connectivity among GPUs at scale.

 achieves impressive performance, sustaining 90% of line-rate throughput

 even as the number of QP (Queue Pair) connections increases. It also delivers 

compared to other standard RDMA network interface cards (RNICs) available on the market. Figure 3 demonstrates these advantages using results from Perftest, a widely-used RDMA benchmarking tool, where Mango GPUBoost RDMA™ achieves 3.4 times higher throughput and reduces 99.9% latency by 52%.

two important features (multipath and custom congestion control) 

to resolve well-known problems within RoCEv2.

First, due to the nature of the RoCEv2 protocol — which enforces packets to be delivered in the order they were sent — network hotspots often occur because of path collision [2]. To alleviate these hotspots and take advantage of multiple rich paths, Mango GPUBoost RDMA™ supports a 

unique out-of-order packet handling mechanism and provides multipath extension

Moreover, no single congestion control solution fits all use cases for RoCEv2. Although DCQCN has become the de-facto standard for RoCEv2, it is hard to configure in practice [3] and has been shown to be less effective for AI workloads [4]. Instead, each workload demands a tailored congestion control algorithm depending on its unique characteristics [5] (e.g., traffic pattern, incast, etc). To maximize workload performance, Mango GPUBoost RDMA™ offers a 

placeholder for custom congestion control

, allowing users to deploy their own algorithms directly on our RNIC.

Please check the product specification and more details 

Try our Mango GPUBoost RDMA™ - Pre-Order is now open!

We are ready for customers who want to innovate their AI infrastructure by accelerating network communication with Mango GPUBoost RDMA™. Our DPUs have been tested on various AI training and inference workloads with top-of-the-line GPUs, demonstrating its flexibility along with industry-leading performance. We support standard RoCEv2 with customizable features and compatibility with common server products.

We continue to innovate performance and scalability in AI infrastructure. As a member of the UltraEthernet Consortium, we also plan to showcase our UltraEthernet DPU in the near future.

References
[1] RDMA over Ethernet for Distributed Training at Meta Scale (Meta, SIGCOMM’24)
[2] Multi-Path Transport for RDMA in Datacenters (Microsoft, NSDI’18)
[3] HPCC: High Precision Congestion Control (Alibaba, SIGCOMM’19)
[4] Impact of RoCE Congestion Control Policies on Distributed Training of DNNs (Meta, HOTI’22)
[5] Datacenter Ethernet and RDMA: Issues at Hyperscale (Microsoft/Broadcom/HPE/Google, IEEE Computer’23)


The performance claims in this document are based on the internal cluster environment. Actual performance may vary depending on the server configuration. Software and workloads used in performance tests may have been optimized for performance only on MangoBoost products. Performance results are based on testing as of dates shown in configurations and may not reflect all publicly available updates. Results that are based on pre-production systems and components as well as results that have been estimated or simulated using MangoBoost reference platform for informational purposes only. Results may vary based on future changes to any systems, components, specifications, or configurations. Statements in this document that refer to future plans or expectations are forward-looking statements. These statements are based on current expectations and involve many risks and uncertainties that could cause actual results to differ materially from those expressed or implied in such statements. MangoBoost does not guarantee any specific outcome. Nothing contained herein is, or shall be relied upon as, a promise or representation or warranty as to future performance of MangoBoost or any MangoBoost product. The information contained herein shall not be deemed to expand in any way the scope or effect of any representations or warranties contained in the definitive agreement for MangoBoost products.

The information contained herein may not be reproduced in whole or in part without prior written consent of MangoBoost. The information presented in this document is for informational purposes only and may contain technical inaccuracies, omissions and typographical errors. The information contained herein is subject to change and may be rendered inaccurate for many reasons, including but not limited to product and roadmap changes, component and motherboard version changes, new model and/or product releases, product differences between differing manufacturers, software changes, BIOS flashes, firmware upgrades, or the like. MangoBoost assumes no obligation to update or otherwise correct or revise this information and MangoBoost reserves the right to make changes to the content hereof from time to time without any notice. Nothing contained herein is intended by MangoBoost, nor should it be relied upon, as a promise or a representation as to the future.

MANGOBOOST MAKES NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE CONTENTS HEREOF AND ASSUMES NO RESPONSIBILITY FOR ANY INACCURACIES, ERRORS OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION.

MangoBoost Demonstrates Mango GPUBoost RDMA™ with MLPerf Training v4.1 on a Multi-Node System Powered by AMD MI300X GPUs

World's First DPU-accelerated System Submission for MLPerf™ Storage. 

Mango StorageBoost™ showed impressive results in the MLPerf™ Storage v1.0 

, setting the precedent for full hardware acceleration for accessing disaggregated remote storage systems for large data centers using patent-pending Data Processing Unit (DPU) solutions. MangoBoost solutions are demonstrated across all MLPerf™ Storage v1.0 workloads, which include large language models (LLMs), scientific computing, and image segmentation.

MangoBoost Showed the Best-Performing Ethernet-Based System Result in MLPerf

In the single-host system category, MangoBoost's solution offers a 1.2x to 4.7x speedup (avg of 2.7x) over the 5 other single-host systems submitted.

In the multi-host system category, when linearly scaled to the same number of hosts, MangoBoost’s solution offers an estimated 1.4x to 7.8x speedup (avg of 4.3x) over the 15 other multi-host systems submitted.

MangoBoost Provides Near-Local SSD Performance by Accelerating the Entire NVMe/TCP Stack Fully on Hardware on Our DPU Solutions.

Our DPU solution accelerates the entire NVMe/TCP protocol stack, including TCP packet processing, by using specialized hardware in the DPU rather than relying on software running on CPUs (either on the host or on the DPU card itself).

MangoBoost’s DPUs are Easy to Deploy on Standard Ethernet-Based Systems.

Our DPUs work “out-of-the-box,” providing standard hardware and software interfaces that integrate into existing infrastructure without modification.

We chose to submit results on two different server configurations, with CPUs and SSDs from various major vendors for all GPU types and models in the MLPerf™ Storage Benchmark. This demonstrates the generality and robustness of our DPUs.

If interested in the MangoBoost AI solutions, please contact us at contact@mangoboost.io

Modern large-language models and other high-performance computing applications have scaled up in the complexity of their operations and sizes. This increase often leads to additional pressure not only on the compute and memory elements but also on the storage systems designed to support these large-scale applications. MLPerf™ Storage is created to benchmark modern storage systems and is designed to show performance in supporting AI/ML tasks.

To address the scalability and performance challenges, MangoBoost’s patent-pending DPU technology allows offloading of infrastructure and data-processing tasks from the host to datacenter-deployable FPGA PCIe cards. With our best-in-class NVMe/TCP Initiator(s) deployed on AI server(s) connected via standard ethernet to NVMe/TCP Target(s) deployed on storage server(s), we are able to deliver the best performance ethernet-based storage system across all evaluated benchmarks on top-of-the-line NVIDIA H100 and A100 GPUs as measured and published in MLPerf™ Storage v1.0 results.

shows an overview of AI infrastructure using MangoBoost’s DPU-accelerated storage. MangoBoost DPUs are deployed on GPU servers and storage servers to accelerate the feeding of training data to GPU. We introduce the three turn-key solutions for accelerating storage for AI training.

Mango StorageBoost™- NVMe/TCP Initiator (NTI) provides local NVMe SSD illusion of remote NVMe SSDs. The transport layer (TCP/IP) for remote NVMe access is fully offloaded onto the DPU and GPU server can save CPU cycles and improve training performance.

Mango StorageBoost™- NVMe/TCP Target (NTT) provides the entire NVMe/TCP target offloading on the storage server. NVMe/TCP data path is fully offloaded on NTT and Target only utilizes a single CPU core to handle control path operations.

Mango StorageBoost™- GPU Storage Boost (GSB) provides direct data transfer between DPU and GPUs via peer-to-peer PCIe communications. It eliminates the overhead of using a CPU bounce buffer for the data transfer with the GPU and improves GPU utilization.

In the following sections, we showcase MangoBoost’s MLPerf™ Storage v1.0 submission and introduce details of the above turn-key solutions applied to our systems. Then, we offer further details on our solutions.

World first DPU-accelerated NVMe/TCP submission for MLPerf™ Storage 

Storage systems play a crucial role in AI training. A bottleneck in the storage system results in the under-utilization of GPUs. This arises when GPUs are idle, waiting for the completion of I/O to read training data from storage. GPUs are an expensive resource, and we should keep GPU utilization high to minimize the completion time of AI model training.

The MLPerf™ Storage benchmark is currently the best environment for storage performance evaluation for AI workloads. Fig. 2 shows an overview of MLPerf™ Storage. One of the notable features in MLPerf™ Storage is "GPU emulation." This means from the perspective of the storage system evaluation; the GPU is only an external processor that ingests fetched data from storage. Since the real computation of the GPU is not necessary, MLPerf™ Storage emulates GPU computation with "sleep."

Additionally, MLPerf™ Storage provides many parameters that are used for emulating a variety of AI training workloads. Users can easily generate datasets with different size distributions and simulate various I/O patterns.

Fig. 3 presents our evaluation setup for MLPerf™ Storage. We utilize a single GPU server that accesses training data in a remote SSD over NVMe/TCP. NVMe/TCP is fully offloaded to NTI to save CPU cycles and maximize training throughput on GPU servers. The storage servers deploy NTT as a counterpart to NTI. The storage server offloads the entire data path processing onto the DPU, utilizing only a single CPU core to handle control path operations. NTT can saturate a 200GbE line with a single card while significantly reducing power consumption for CPU utilization.

To showcase the wide compatibility with standard servers and SSDs, we demonstrate our solution with two different types of storage servers. Each storage server is powered by either an Intel or AMD CPU and equipped with two SSD models from Samsung and Kioxia. Since both storage servers show almost identical performance, for simplicity, we report the results with one of the Storage Servers in this blog.

In the following section, we showcase our 

 result, highlighting our top-ranked Ethernet category scores.

MangoBoost shows the best-performing ethernet-based system result in MLPerf Storage v1.0.

Fig. 4 shows a direct comparison of our MLPerf™ Storage score against other Ethernet-based single-host submissions. The results show that the MangoBoost DPU-accelerated NVMe/TCP storage significantly outperforms other systems in AI training workloads. Our system significantly outperforms other systems on ResNet50 and UNET3D workloads, with an average performance increase of 2.7x, ranging from 1.2x to 4.7x, respectively.The Cosmoflow result is omitted because Nutanix and Hammerspace did not submit results for this workload.

Fig. 5 shows comparisons to Ethernet-based multi-host submissions. To estimate the relative performance of our single-host result, we normalized competitors' numbers by the number of hosts used in the evaluation. As the figure shows, our system outperforms others by 1.4x to 7.8x and 4.3x on average for ResNet50 and UNET3D workloads, respectively.

MangoBoost DPU solution provides near-local SSD performance for NVMe/TCP-attached storage

Finally, we demonstrate the superior performance of our NVMe/TCP solutions compared to systems using software NVMe/TCP, achieving performance on par with local NVMe. Local NVMe is an ideal system with minimal overhead for storage access, and software NVMe/TCP is the existing available solution (albeit slow). Our DPU solution is the first to unlock the full power of NVMe/TCP.

Fig. 6 shows a performance comparison of our solution versus "Local NVMe" and "SW NVMe/TCP" baselines. We show a 2.5x and 1.5x performance improvement in UNET3D and ResNet50, respectively, compared to SW NVMe/TCP. The limited computation resources severely limit the performance of SW NVMe/TCP since many cycles are spent processing the TCP/IP stack for remote NVMe access.

We also achieved the same score in UNET3D and ResNet50 compared to "Local NVMe." This is a great achievement since closing the performance gap completely is impossible to do without the state-of-the-art full hardware acceleration of NVMe/TCP with our DPU. It also shows that MangoBoost DPU allows true storage disaggregation with high-performance data center-grade SSDs.

NVMe/TCP is widely recognized for its scalability and ease of deployment compared to RDMA-based protocols like RoCEv2, which require specialized network hardware such as RDMA network cards and switches [

]. This makes NVMe/TCP a more accessible and flexible solution for many organizations. However, despite these advantages, NVMe/TCP has traditionally faced performance challenges due to higher overheads, making it less efficient for high-performance workloads like AI and machine learning.

MangoBoost addresses these limitations with a fully hardware-accelerated NVMe/TCP solution on both initiator and target-side servers. By leveraging our Data Processing Units (DPUs), MangoBoost is the first to overcome the inherent performance bottlenecks of NVMe/TCP, significantly enhancing both throughput and efficiency. This makes our NVMe/TCP solution an ideal choice for AI-driven applications, where high-performance storage is critical.

Our DPU solutions are designed to be deployed seamlessly into existing infrastructure. From a hardware perspective, MangoBoost’s DPU installs seamlessly into any standard PCIe slot and connects via standard Ethernet QSFP ports, similar to a typical network interface card (NIC). On the software side, the DPU is fully compatible with NVMe/PCIe and NVMe/TCP protocols, requiring no modifications to the user’s software environment.

Additionally, we offer a GPU Storage Boost solution. We co-presented this solution with AMD at SDC 2024 

, where we showed end-to-end training workloads running on AMD MI300X GPUs accelerated by MangoBoost DPUs, dramatically improving overall system performance. This solution further enables peer-to-peer direct data communication between GPUs and remote storage (on top of NTI). However, this was not included in our submission since the MLPerf Storage setup only emulates the GPU hardware, so there is no real GPU hardware to communicate peer-to-peer within this setup.

Solution 1: Mango StorageBoost™- NVMe/TCP Initiator (NTI).

Mango StorageBoost™- NVMe/TCP Initiator (NTI) is a high-performance NVMe-oF initiator solution that unlocks the full potential of storage disaggregation. By offloading the entire NVMe/TCP stack into hardware, NTI delivers unmatched full-duplex line-rate performance while significantly reducing CPU consumption.

NTI integrates seamlessly into any initiator servers that connect using standard ethernet (including AI GPU servers). By exposing the DPU as a standard NVMe PCIe device, existing storage systems can leverage NTI without the need for any software modifications. Furthermore, NTI can connect to any NVMe/TCP target servers over a standard TCP/IP network.

Solution 2: Mango StorageBoost™- NVMe/TCP Target (NTT).

Mango StorageBoost™ - NVMe/TCP Target is a groundbreaking NVMe-oF target solution, powering the storage system with outstanding performance. It enables disaggregated storage servers to be connected to the network through the standard TCP/IP protocol stack. By fully offloading and accelerating both the TCP/IP and NVMe-oF processing, it entirely eliminates the CPU intervention from the I/O path.

NTT decapsulates the TCP packets into NVMe-oF commands and translates them into standard NVMe commands. By utilizing PCIe peer-to-peer technology, NTT directly sends those commands to the peer NVMe SSDs without any CPU consumption. Additionally, its modular and flexible architecture also enables more dynamic customer-driven storage systems such as data reduction and reliability.

Solution 3: Mango StorageBoost™- GPU Storage Boost (GSB).

Mango StorageBoost™- GPU Storage Boost (GSB) is an essential software solution designed to dramatically enhance the efficiency of GPU data movement, enabling unprecedented performance capabilities. By establishing a direct data path for Direct Memory Access (DMA) transfers between GPU memory and storage, GSB eliminates the need for bounce buffers through the CPU, and bypasses CPU involvement entirely. It facilitates peer-to-peer communication within GPU systems, regardless of whether the storage is local or remote. Furthermore, GSB offers a nearly identical syntax and semantic structure to traditional POSIX File APIs, making integration straightforward. This allows for efficient, filesystem-based data management with minimal adaptation, enabling users to harness the full power of the GPU systems without sacrificing ease of use.

When combined with Mango StorageBoost™- NVMe/TCP Initiator (NTI), GSB enables highly efficient GPU data movement, even over remote storage systems based on the TCP/IP transport layer, without any CPU involvement. Typically, handling TCP/IP protocols and the NVMe/TCP stack requires CPU intervention and memory copying, which introduces latency and reduces overall efficiency. NTI addresses these challenges by offloading these tasks to dedicated hardware, allowing the host to leverage the DMA engine as if it were directly connected to an NVMe PCI device. NTI effectively removes the performance and efficiency constraints associated with CPU resources during remote storage access, while GSB eliminates similar limitations for GPU data movement. By utilizing both solutions together, users can unlock unparalleled synergies, dramatically improving data flow efficiency and reducing system bottlenecks.

If you are interested in NTI, NTT and GPU Storage Boost, please email contact@mangoboost.io

https://mlcommons.org/2024/09/mlperf-storage-v1-0-benchmark-results/

https://mlcommons.org/2023/06/introducing-the-mlperf-storage-benchmark-suite/


[3] Answering Your Questions: NVMe™/TCP: What You Need to Know About the Specification, https://nvmexpress.org/answering-your-questions-nvme-tcp-what-you-need-to-know-about-the-specification-webcast-qa/
[4] Pavilion compares RoCE and TCP NVMe over Fabrics performance, https://blocksandfiles.com/2018/08/16/pavilion-compares-roce-and-tcp-nvme-over-fabrics-performance/
[5] SRNIC: A Scalable Architecture for RDMA NICs, NSDI’23, https://www.usenix.org/conference/nsdi23/presentation/wang-zilong
[6] RDMA over Ethernet for Distributed Training at Meta Scale, SIGCOMM’24, https://dl.acm.org/doi/abs/10.1145/3651890.3672233
[7] Accelerating GPU Server Access to Network-Attached Disaggregated Storage using Data Processing Unit (DPU),

https://www.sniadeveloper.org/events/agenda/session/666

The performance claims in this document are based on the internal cluster environment. Actual performance may vary depending on the server configuration. Software and workloads used in performance tests may have been optimized for performance only on MangoBoost products. Performance results are based on testing as of dates shown in configurations and may not reflect all publicly available updates. Results that are based on pre-production systems and components as well as results that have been estimated or simulated using MangoBoost reference platform for informational purposes only. Results may vary based on future changes to any systems, components, specifications, or configurations. Statements in this document that refer to future plans or expectations are forward-looking statements. These statements are based on current expectations and involve many risks and uncertainties that could cause actual results to differ materially from those expressed or implied in such statements. MangoBoost does not guarantee any specific outcome. Nothing contained herein is, or shall be relied upon as, a promise or representation or warranty as to future performance of MangoBoost or any MangoBoost product. The information contained herein shall not be deemed to expand in any way the scope or effect of any representations or warranties contained in the definitive agreement for MangoBoost products.

MangoBoost Demonstrates Turn-Key Storage Solution for AI Training with MLPerf® Storage v1.0 

New MLPerf® result on AMD Instinct™ MI300X GPU.

 First MLPerf Inference v4.1 results on AMD MI300X GPU shows 23.5K token/second on Llama-70B benchmark! 

 Going from MLPerf inference benchmark to production deployment is not easy, as it requires a tremendous effort (e.g., bring-up, co-optimize, and validate of full-stack AI inference serving software).

. Ready-to-deploy full-stack AI inference serving container, containing MangoBoost's inference serving software that complements popular vLLM, packaged with everything needed to run optimized FP8 Llama on AMD MI300X.
-  

 Out of the box, easily reproduce valid MLPerf Inference v4.1 Llama2-70B results with FP8 on AMD MI300X systems. 

 Mango LLMBoost container includes AI-inference server software optimized with vLLM, with added features for production (e.g., data parallelism and tensor parallelism, web API).
-  

 MangoBoost reports results with Llama-70B from MLPerf inference and Llama Guard-7B from Meta, measured on MangoBoost’s server with AMD MI300X GPUs.

. A hardware-accelerated web server for frontend gateway server for AI GPU inference clusters, which offers unparalleled throughput and reduced latency.
-  

 Mango WebBoost leverages MangoBoost Hardware Data Processing Unit (DPU) solution on an AMD Alveo™ U45N FPGA card to offer stateful offload of complete TCP/IP networking tasks, to achieve reliable low average latency and tail latency.
-  

 Integrated and demonstrated with Nginx, one of the most widely used web servers. Can be also integrated into other web servers without modification.

If interested in the above solutions, please contact us at contact@mangoboost.io

The recent MLPerf Inference v4.1 publications 

have reported the first published result of a system with 8x AMD’s MI300X GPUs on Llama-70B benchmark delivering state-of-the-art performance of 23.5K tokens/second. As reported by AMD, this result is by leveraging vLLM, which has been optimized for AMD GPUs 

However, if one were to download vLLM, and run it out-of-the-box on MI300X GPUs, the AMD reported MLPerf performance level can be difficult to achieve. One needs to tune vLLM and ROCm, as well as quantize the Llama-70B model properly to FP8. Furthermore, vLLM does not support certain key features out-of-the-box, such as data parallelism to scale inference across multiple GPUs.

Beyond the MLPerf benchmark, deploying MI300X in a production AI inference serving solution needs additional features, such as support for various web API endpoints, multi-model serving, hooks for cluster management, etc. Within the inference serving production cluster, one also needs a front-end web server to manage multiple back-end GPU servers.

MangoBoost introduces two turn-key solutions to address above said challenges.

 - Ready-to-deploy full-stack AI inference serving container, containing MangoBoost's inference serving software that complements vLLM, packaged with everything needed to run optimized FP8 Llama on MI300X.

 - a A hardware-accelerated web server for frontend gateway server for AI GPU inference clusters, which offers unparalleled throughput and reduced latency.

We detail these solutions in the next sections, and report case studies that quantify their benefits.


LLMBoost - Ready-to-deploy AI inference software, demonstrated on AMD MI300X GPUs with MLPerf Inference v4.1 Llama-70B and Meta Llama Guard-7B.

MangoBoost's LLMBoost AI inference server software is an optimized turn-key solution, ready-to-deploy as a container that is packaged and optimized with open-source vLLM and ROCm. It offers unparalleled LLM inference performance on AI servers with GPUs, such as AMD’s MI300X. Designed for the next generation of high-memory capacity GPUs and LLMs, LLMBoost employs tensor and data parallelism schemes optimized for accelerating a variety of models, from larger models such Llama-70B, to more compact ones like Meta Llama Guard-7B.

Fig. 1 highlights two key areas of optimizations that are applied in an LLMBoost container, namely the inference engine and model optimization. Our inference engine employs advanced GPU Kernel techniques allowing for multidimensional parallelism. This is coupled with domain-specific tuning of the ROCm software stack for low-precision computation. This approach enhances the efficiency of generalized matrix multiplication (GEMM) kernels and further economizes GPU shared memory usage and minimizes costly data transfers between the on-chip shared memory and the off-chip global memory. Asynchronous data transfer, along with data prefetching, PagedAttention, and KV-caching, has been optimized in vLLM to enhance memory efficiency and boost performance.

Our inference server provides multiple endpoint options for seamless integration with existing solutions. Synchronous and asynchronous REST APIs offer comprehensive flexibility in request management, while WebSockets provide a streamlined interface for streaming applications. Our adaptable dynamic scheduling algorithms ensure peak GPU efficiency across diverse deployment scenarios by intelligently assigning incoming requests to the most suitable GPU. Customers can fine-tune their deployments to prioritize throughput, latency, or a hybrid approach, creating a fully customizable inference server that consistently delivers state-of-the-art performance on cutting-edge GPU hardware to meet a broad spectrum of customer needs.

Furthermore, our containers expose standard Prometheus endpoints for monitoring performance, allowing seamless integration with existing Kubernetes deployments and enabling Horizontal Pod Autoscaling. Multiple models can be served in a single container with various degrees of flexibility offered to the user, tailoring the inference server to their specific needs.

Case Study with MLPerf Inference v4.1 Llama-70B and Meta Llama Guard-7B.

 We integrated LLMBoost with vLLM and compared it with vLLM-only 

]. We selected two prominent LLMs for this assessment: Llama2-70B from the MLPerf Inference v4.1 Benchmark and Llama Guard-7B from Meta. Our testing environment featured a high-performance Supermicro server equipped with 8x AMD MI300X GPUs. The details are in the table below.

For the Llama2-70B model, we adhered to the MLPerf Inference v4.1 Closed Division rules, ensuring the validity and integrity of the results on the LLMBoost + vLLM configuration with all 8 GPUs active. This setup provided a consistent and rigorous foundation for comparison. Subsequently, identical conditions were applied across all other experiments to maintain fairness in our evaluation.

For the MLPerf Inference v4.1 Llama-70B test case, we are able to achieve 22.8K tokens per second, which is within 3% of AMD’s published MLPerf results of 23.5K tokens per second. Note that our system uses a different CPU than what AMD uses in their MLPerf official submission, which may account for the slight difference in performance.
Since the MLPerf Inference Benchmark does not yet include validation rules for Llama Guard, we applied the same configuration used for Llama2-70B to ensure consistency in testing.

As shown in Fig. 2, LLMBoost achieved a 5.2x to 6.0x improvement over the native vLLM thanks to three features of LLMBoost; (1) enable data parallelism, (2) dynamic scheduling across 8 GPUs, and (3) lightweight streamlined interface. This remarkable enhancement underscores the potential of LLMBoost to accelerate inference workloads, making it a crucial tool for optimizing large-scale AI deployments.


WebBoost - Accelerating Nginx web server with MangoBoost stateful TCP/IP hardware acceleration on an AMD U45N FPGA card

WebBoost is a web-serving container for running Nginx on a front-end server equipped with hardware acceleration using MangoBoost DPU (data processing units) solution. Our solution can support multiple FPGA cards, and in this paper we demonstrate it on AMD U45N. The LLMBoost container and the WebBoost container are designed to work together seamlessly in a coordinated operation to stream input data to the GPUs and to return generated output tokens back to the clients. Fig. 3 below illustrates the network processing layers between arriving LLM requests and an LLM server’s GPU. Clients submit their LLM requests into a REST API and then send them to the Nginx front-end gateway. Nginx collects all incoming requests and then distributes the request to the back-end GPU computation containers. In a front-end server with WebBoost, MangoBoost DPUs solution offers stateful hardware acceleration of the TCP/IP networking stack, to boost web-server software such as Nginx. WebBoost offers reliable low average latency and tail latency, as well as boosting throughput; these are all important for LLM inference. Additionally, WebBoost hooks the standard POSIX socket API calls, ensuring that it requires no modifications to the user application, making it an easy-to-use solution for seamless integration into existing systems.

To quantify the performance benefits of MangoBoost’s WebBoost when integrated with Nginx, we conducted a series of benchmarks comparing our solution against a setup using Linux’s standard TCP kernel software stack. TCP kernel software stack relies on the NVIDIA® ConnectX®-6 network card, which is widely used in typical data center environments.

We utilized the wrk benchmarking tool to generate network requests to the Nginx web server. This tool is designed for measuring HTTP performance, making it ideal for evaluating the throughput and latency of our solution. In this evaluation, wrk sends HTTP POST requests to the Nginx web server. The requests contain sentences typically input by users of LLM AI inference serving. The Nginx then responds with sentences as part of the HTTP responses. The configuration details of our server hardware and software are in the table below.

Web-Serving Front-End Server Configuration

The evaluation results are summarized in the graphs in Fig. 4.

Throughput: WebBoost outperforms the Linux TCP stack significantly, achieving a 2x improvement in throughput. This means that Nginx can handle 100% more requests per second when using the WebBoost solution compared to the standard TCP stack.

Latency: WebBoost also demonstrates 0.53x and 0.54x reductions on the 50th and 90th percentile latency, respectively.

These results clearly illustrate the efficiency gains from offloading TCP packet processing to our DPU. By handling TCP connections more effectively, the WebBoost solution not only increases throughput but also significantly reduces the time it takes to process requests, leading to a more responsive and scalable Nginx deployment.

Nginx is a cornerstone of modern web infrastructure, and optimizing its performance is crucial for maintaining the efficiency and reliability of web services. By offloading TCP packet processing to a DPU, we can significantly enhance Nginx’s capabilities, allowing it to handle more traffic with lower latency. Our solution represents a leap forward in web server performance, setting a new standard for what is possible in high-demand environments.

If you are interested in LLMBoost and WebBoost, please email contact@mangoboost.io.

MLPerf Inference: Datacenter Benchmark Suite Results

Unveiling MLPerf® Results on AMD Instinct™ MI300X Accelerators

Smith, Alan, et al. "Realizing the AMD Exascale Heterogeneous Processor Vision: Industry Product." 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 2024.

vLLM in AMD MLPerf Inference v4.1 submission


[6] Result not verified by MLCommons Association.

Blog

Ready for
a boost?

Blog

MangoBoost Demonstrates Mango GPUBoost RDMA™ with MLPerf Training v4.1 on a Multi-Node System Powered by AMD MI300X GPUs

MangoBoost Demonstrates Turn-Key Storage Solution for AI Training with MLPerf® Storage v1.0

MangoBoost Demonstrates Turn-Key AI LLM Inference Serving solution on AMD Instinct™ MI300X GPUs with MLPerf® Inference v4.1 Llama Benchmark

Ready for a boost?

Ready for
a boost?