GPUs

Graphical Processing Units (GPUs) are specialized computing devices originally developed for rendering graphics. They typically have a large number of compute cores running many independent processing streams with low communication between cores and numerical precision. These attributes have made them appealing for solving certain numerical problems with much greater speed than CPUs, particularly in the field of machine learning.

Nvidia is a popular manufacturer of GPUs, with their CUDA lanuage and ecosystem commonly used for building GPU accelerated applications.

The nvidia-smi command line application is useful for quering GPU info. Via a flag it can produce output in machine readable XMl format.

nvidia-ml-py is a Python wrapper for the Nvidia Management Library which can be useful for monitoring and managing the state of NVIDIA GPUs.

pip install nvidia-ml-py

Use NVIDIA Nsight Systems for GPU tracing and CPU sampling and NVIDIA Nsight Compute for GPU profiling with:

nsys profile -t cuda -o ./PROF_GPU --stats=true --force-overwrite true python sample_batching.py