NVIDIA HPC Benchmarks

Benchmarking NVIDIA HPC Workloads on the Kempner AI Cluster

Overview

This benchmarking study aims to evaluate the floating-point compute performance, memory bandwidth, and system scalability of the Kempner AI Cluster using standard NVIDIA HPC Benchmark containers. The benchmarks cover four key workloads: HPL (High-Performance Linpack), HPL-MxP (mixed-precision Linpack), HPCG (High-Performance Conjugate Gradient), and STREAM (memory bandwidth test). Tests are conducted on NVIDIA Hopper GPUs (H100) in configurations ranging from single-GPU up to multi-GPU (1–8 GPUs) and multi-node scenarios. Results will provide a robust baseline for performance tuning, user expectations, and future hardware procurement decisions. Additional tests using Grace Hopper (H200) nodes may be added to compare next-generation CPU-GPU memory architectures.

Prerequisites

The following prerequisites must be met before preparing for benchmarking:

• Access to the Kempner AI Cluster with valid user credentials and appropriate SLURM job submission permissions.
• At least one allocated NVIDIA Hopper (H100) GPU per test; multi-GPU and multi-node runs require correct GPU partition and Infiniband network access.
• Podman installed locally (or on a build node) to pull official NVIDIA HPC benchmark container images.
• Singularity (or Apptainer) installed and configured for container execution in the cluster environment.
• Basic familiarity with SLURM job submission (srun, sbatch), resource allocation flags (–gres, –ntasks), and container runtime options.
• Sufficient scratch storage or a working directory to store benchmark input files, result logs, and Singularity image files (.sif).
• Access to NVIDIA NGC account for pulling the most up-to-date benchmark containers. (A copy of the container images can also be provided by the cluster administrators.)
• Recommended: Familiarity with HPL benchmarking concepts, including problem size (N), process grid (P x Q), and performance metrics (GFLOPS, efficiency).

NVIDIA HPC Container

For this benchmarking effort, we use the official NVIDIA HPC Benchmark container, which conveniently packages industry-standard workloads, including HPL, HPL-MxP, HPCG, and STREAM, with tuned libraries and drivers to deliver consistent, reproducible performance. This container ensures compatibility with the latest NVIDIA architectures and provides an easy way to run demanding HPC tests without building complex software stacks from scratch. See the link below for details on how we pull, convert, and deploy this container on the Kempner AI Cluster.

• NVIDIA HPC Benchmark
• Convert NVIDIA HPC Benchmark Container to Singularity

NVIDIA HPL Benchmark

HPL (High-Performance Linpack) is a widely used benchmark for measuring the floating-point compute performance of supercomputers. It solves a dense system of linear equations and is the basis for the TOP500 list of supercomputers. In this section, we will run the HPL benchmark using the NVIDIA HPC container on the Kempner AI Cluster.

• NVIDIA HPL Benchmark on Kempner AI Cluster

Here is the summary of the HPL benchmark runs on the Kempner AI Cluster:

NVIDIA HPL-MxP Benchmark

HPL-MxP (High-Performance Linpack - Mixed Precision) is an enhanced version of the traditional HPL benchmark that leverages NVIDIA’s Tensor Core acceleration for mixed-precision matrix operations, delivering significantly higher performance on supported GPUs. HPL-MxP demonstrates how modern hardware can use mixed precision arithmetic combined with iterative refinement to solve dense linear systems faster while maintaining double-precision accuracy. In this section, we run the HPL-MxP benchmark using the NVIDIA HPC container on the Kempner AI Cluster.

• NVIDIA HPL-MxP Benchmark on Kempner AI Cluster

Here is a summary of MxP benchmark (using FP16 for LU factorization) runs on the Kempner AI Cluster using NVIDIA Hopper GPUs (H100):

NVIDIA HPCG Benchmark

HPCG (High Performance Conjugate Gradients) is designed to complement HPL by providing a more realistic benchmark for modern HPC systems. Unlike HPL, which is compute-bound, HPCG stresses memory system performance, data movement, and interconnect efficiency, resembling the sparse linear algebra workloads found in many real-world scientific applications.

• NVIDIA HPCG Benchmark on Kempner AI Cluster

Here is a summary of the HPCG benchmark runs on the Kempner AI Cluster using NVIDIA Hopper GPUs (H100):

NVIDIA STREAM Benchmark

The STREAM benchmark is a simple yet powerful memory bandwidth test originally designed to measure sustainable memory throughput for CPUs. NVIDIA’s GPU-accelerated version extends this to modern GPUs, measuring the achievable bandwidth of on-device memory (HBM) by performing simple vector operations like COPY, SCALE, ADD, and TRIAD. It does not include PCIe or interconnect transfers — it focuses purely on how fast the GPU can read and write data within its own high-bandwidth memory. For more details on the STREAM benchmark, see the following resources:

• NVIDIA STREAM Benchmark on Kempner AI Cluster

Here is the summary of the STREAM benchmark runs on the Kempner AI Cluster using NVIDIA Hopper GPUs (H100):

Done!