GPU Drivers and SDKs

MLC LLM is a universal deployment solution that allows efficient CPU/GPU code generation without AutoTVM-based performance tuning. This section focuses on generic GPU environment setup and troubleshooting.

CUDA

CUDA is required to compile and run models with CUDA backend.

Installation

If you have a NVIDIA GPU and you want to use models compiled with CUDA backend, you should install CUDA, which can be downloaded from here.

Validate Installation

To verify you have correctly installed CUDA runtime and NVIDIA driver, run nvidia-smi in command line and see if you can get the GPU information.

ROCm

ROCm is required to compile and run models with ROCm backend.

Installation

Right now MLC LLM only supports ROCm 6.1/6.2. If you have AMD GPU and you want to use models compiled with ROCm backend, you should install ROCm from here.

Validate Installation

To verify you have correctly installed ROCm, run rocm-smi in command line. If you see the list of AMD devices printed out in a table, it means the ROCm is correctly installed.

Vulkan Driver

Installation

To run pre-trained models (e.g. pulled from MLC-AI’s Hugging Face repository) compiled with Vulkan backend, you are expected to install Vulkan driver on your machine.

Please check this page and find the Vulkan driver according to your GPU vendor.

AMD Radeon and Radeon PRO

For AMD Radeon and Radeon PRO users, please download AMD’s drivers from official website (Linux / Windows). For Linux users, after you installed the amdgpu-install package, you can follow the instructions in its documentation to install the driver. We recommend you installing ROCr OpenCL and PRO Vulkan (proprietary) for best performance, which can be done by running the following command:

amdgpu-install --usecase=graphics,opencl --opencl=rocr --vulkan=pro --no-32

Validate Installation

To verify whether Vulkan installation is successful or not, you are encouraged to install vulkaninfo, below are the instructions to install vulkaninfo on different platforms:

sudo apt-get update
sudo apt-get install vulkan-tools

After installation, you can run vulkaninfo in command line and see if you can get the GPU information.

Note

WSL support for Windows is work-in-progress at the moment. Please do not use WSL on Windows to run Vulkan.

Vulkan SDK

Vulkan SDK is required for compiling models to Vulkan backend. To build TVM Unity compiler from source, you will need to install Vulkan SDK as a dependency, but our pre-built wheels already ships with Vulkan SDK.

Check Vulkan SDK installation guide according to your platform:

Please refer to installation and setup page for next steps to build TVM-Unity from source.

OpenCL SDK

OpenCL SDK is only required when you want to build your own models for OpenCL backend. Please refer to OpenCL’s Github Repository for installation guide of OpenCL-SDK.

Orange Pi 5 (RK3588 based SBC)

OpenCL SDK and Mali GPU driver is required to compile and run models for OpenCL backend.

Installation

  • Download and install the Ubuntu 22.04 for your board from here

  • Download and install libmali-g610.so

cd /usr/lib && sudo wget https://github.com/JeffyCN/mirrors/raw/libmali/lib/aarch64-linux-gnu/libmali-valhall-g610-g6p0-x11-wayland-gbm.so

  • Check if file mali_csffw.bin exist under path /lib/firmware, if not download it with command:

cd /lib/firmware && sudo wget https://github.com/JeffyCN/mirrors/raw/libmali/firmware/g610/mali_csffw.bin

  • Download OpenCL ICD loader and manually add libmali to ICD

sudo apt update
sudo apt install mesa-opencl-icd
sudo mkdir -p /etc/OpenCL/vendors
echo "/usr/lib/libmali-valhall-g610-g6p0-x11-wayland-gbm.so" | sudo tee /etc/OpenCL/vendors/mali.icd

  • Download and install libOpenCL

sudo apt install ocl-icd-opencl-dev

  • Download and install dependencies for Mali OpenCL

sudo apt install libxcb-dri2-0 libxcb-dri3-0 libwayland-client0 libwayland-server0 libx11-xcb1

  • Download and install clinfo to check if OpenCL successfully installed

sudo apt install clinfo

Validate Installation

To verify you have correctly installed OpenCL runtime and Mali GPU driver, run clinfo in command line and see if you can get the GPU information. You are expect to see the following information:

$ clinfo
arm_release_ver: g13p0-01eac0, rk_so_ver: 3
Number of platforms                               2
   Platform Name                                   ARM Platform
   Platform Vendor                                 ARM
   Platform Version                                OpenCL 2.1 v1.g6p0-01eac0.2819f9d4dbe0b5a2f89c835d8484f9cd
   Platform Profile                                FULL_PROFILE
   ...