mahout

NVTX Profiling Guide

Overview

NVTX (NVIDIA Tools Extension) provides performance markers visible in Nsight Systems. This project uses zero-cost macros that compile to no-ops when the observability feature is disabled.

Build with NVTX

Default builds exclude NVTX for zero overhead. Enable profiling with:

cd mahout/qdp
cargo build -p qdp-core --example nvtx_profile --features observability --release

Run Example

./target/release/examples/nvtx_profile

Expected output:

=== NVTX Profiling Example ===

✓ Engine initialized
✓ Created test data: 1024 elements

Starting encoding (NVTX markers will appear in Nsight Systems)...
Expected NVTX markers:
  - Mahout::Encode
  - CPU::L2Norm
  - GPU::Alloc
  - GPU::H2DCopy
  - GPU::KernelLaunch
  - GPU::Synchronize
  - DLPack::Wrap

✓ Encoding succeeded
✓ DLPack pointer: 0x558114be6250
✓ Memory freed

=== Test Complete ===

Profile with Nsight Systems

nsys profile --trace=cuda,nvtx -o report ./target/release/examples/nvtx_profile

This generates report.nsys-rep and report.sqlite.

Viewing Results

GUI View (Nsight Systems)

Open the report in Nsight Systems GUI:

nsys-ui report.nsys-rep

In the GUI timeline view, you will see:

• Colored blocks for each NVTX marker
• CPU timeline showing CPU::L2Norm
• GPU timeline showing GPU::Alloc, GPU::H2DCopy, GPU::Kernel
• Overall workflow covered by Mahout::Encode

Command Line Statistics

View summary statistics:

nsys stats report.nsys-rep

Example NVTX Range Summary output:

Time (%)  Total Time (ns)  Instances    Avg (ns)      Med (ns)     Min (ns)    Max (ns)   StdDev (ns)   Style        Range
--------  ---------------  ---------  ------------  ------------  ----------  ----------  -----------  --------  --------------
   50.0       11,207,505          1  11,207,505.0  11,207,505.0  11,207,505  11,207,505          0.0  StartEnd  Mahout::Encode
   48.0       10,759,758          1  10,759,758.0  10,759,758.0  10,759,758  10,759,758          0.0  StartEnd  GPU::Alloc
    1.8          413,753          1     413,753.0     413,753.0     413,753     413,753          0.0  StartEnd  CPU::L2Norm
    0.1           15,873          1      15,873.0      15,873.0      15,873      15,873          0.0  StartEnd  GPU::H2DCopy
    0.0              317          1         317.0         317.0         317         317          0.0  StartEnd  GPU::KernelLaunch

The output shows:

• Time percentage for each operation
• Total time in nanoseconds
• Number of instances
• Average, median, min, max execution times

CUDA API Summary shows detailed CUDA call statistics:

Time (%) Total Time (ns) Num Calls Avg (ns) Med (ns) Min (ns) Max (ns) StdDev (ns) Name ——– ————— ——— ———– ———– ——– ———- ———– ——————– 99.2 11,760,277 2 5,880,138.5 5,880,138.5 2,913 11,757,364 8,311,652.0 cuMemAllocAsync 0.4 45,979 2 22,989.5 22,989.5 7,938 38,041 21,286.0 cuMemcpyHtoDAsync_v2 0.1 14,722 1 14,722.0 14,722.0 14,722 14,722 0.0 cuEventCreate 0.1 13,100 3 4,366.7 3,512.0 861 8,727 4,002.0 cuStreamSynchronize 0.1 9,468 11 860.7 250.0 114 4,671 1,453.3 cuCtxSetCurrent 0.1 6,479 1 6,479.0 6,479.0 6,479 6,479 0.0 cuEventDestroy_v2 0.0 4,599 2 2,299.5 2,299.5 1,773 2,826 744.6 cuMemFreeAsync

• Memory allocation (cuMemAllocAsync)
• Memory copies (cuMemcpyHtoDAsync_v2)
• Stream synchronization (cuStreamSynchronize)

NVTX Markers

The following markers are tracked:

• Mahout::Encode - Complete encoding workflow
• CPU::L2Norm - L2 normalization on CPU
• GPU::Alloc - GPU memory allocation
• GPU::H2DCopy - Host-to-device memory copy
• GPU::KernelLaunch - CPU-side kernel launch
• GPU::Synchronize - CPU waiting for GPU completion
• DLPack::Wrap - Conversion to DLPack pointer

Using Profiling Macros

The project provides zero-cost macros in qdp-core/src/profiling.rs:

// Profile a scope (automatically pops on exit)
crate::profile_scope!("MyOperation");

// Mark a point in time
crate::profile_mark!("Checkpoint");

When observability feature is disabled, these macros compile to no-ops with zero runtime cost.

Example Location

Source code: qdp-core/examples/nvtx_profile.rs

Troubleshooting

NVTX markers not appearing:

• Ensure --features observability is used during build
• Verify CUDA device is available
• Check that encoding actually executes

nsys warnings: Warnings about CPU sampling are normal and can be ignored. They do not affect NVTX marker recording.

This site is open source. Improve this page.