GPU Drivers and SDKs¶
MLC LLM is a universal deployment solution that allows efficient CPU/GPU code generation without AutoTVM-based performance tuning. This section focuses on generic GPU environment setup and troubleshooting.
CUDA¶
CUDA is required to compile and run models with CUDA backend.
Installation¶
If you have a NVIDIA GPU and you want to use models compiled with CUDA backend, you should install CUDA, which can be downloaded from here.
Validate Installation¶
To verify you have correctly installed CUDA runtime and NVIDIA driver, run nvidia-smi
in command line and see if you can get the GPU information.
ROCm¶
ROCm is required to compile and run models with ROCm backend.
Installation¶
Right now MLC LLM only supports ROCm 6.1/6.2. If you have AMD GPU and you want to use models compiled with ROCm backend, you should install ROCm from here.
Validate Installation¶
To verify you have correctly installed ROCm, run rocm-smi
in command line.
If you see the list of AMD devices printed out in a table, it means the ROCm is correctly installed.
Vulkan Driver¶
Installation¶
To run pre-trained models (e.g. pulled from MLC-AI’s Hugging Face repository) compiled with Vulkan backend, you are expected to install Vulkan driver on your machine.
Please check this page and find the Vulkan driver according to your GPU vendor.
AMD Radeon and Radeon PRO¶
For AMD Radeon and Radeon PRO users, please download AMD’s drivers from official website (Linux / Windows).
For Linux users, after you installed the amdgpu-install
package, you can follow the instructions in its documentation to install
the driver. We recommend you installing ROCr OpenCL and PRO Vulkan (proprietary) for best performance, which can be done by running the following command:
amdgpu-install --usecase=graphics,opencl --opencl=rocr --vulkan=pro --no-32
Validate Installation¶
To verify whether Vulkan installation is successful or not, you are encouraged to install vulkaninfo
, below are the instructions to install vulkaninfo
on different platforms:
sudo apt-get update
sudo apt-get install vulkan-tools
# It comes with your GPU driver
sudo dnf install vulkan-tools
sudo pacman -S vulkan-tools
# Arch Linux has maintained an awesome wiki page for Vulkan which you can refer to for troubleshooting: https://wiki.archlinux.org/title/Vulkan
# Please install Vulkan SDK for your platform
# https://vulkan.lunarg.com/sdk/home
After installation, you can run vulkaninfo
in command line and see if you can get the GPU information.
Note
WSL support for Windows is work-in-progress at the moment. Please do not use WSL on Windows to run Vulkan.
Vulkan SDK¶
Vulkan SDK is required for compiling models to Vulkan backend. To build TVM Unity compiler from source, you will need to install Vulkan SDK as a dependency, but our pre-built wheels already ships with Vulkan SDK.
Check Vulkan SDK installation guide according to your platform:
For Ubuntu user, please check Getting Started with the Ubuntu Vulkan SDK
For other Linux distributions, please check Getting Started with the Linux Tarball Vulkan SDK
Please refer to installation and setup page for next steps to build TVM-Unity from source.
OpenCL SDK¶
OpenCL SDK is only required when you want to build your own models for OpenCL backend. Please refer to OpenCL’s Github Repository for installation guide of OpenCL-SDK.
Orange Pi 5 (RK3588 based SBC)¶
OpenCL SDK and Mali GPU driver is required to compile and run models for OpenCL backend.
Installation¶
Download and install the Ubuntu 22.04 for your board from here
Download and install
libmali-g610.so
cd /usr/lib && sudo wget https://github.com/JeffyCN/mirrors/raw/libmali/lib/aarch64-linux-gnu/libmali-valhall-g610-g6p0-x11-wayland-gbm.so
Check if file
mali_csffw.bin
exist under path/lib/firmware
, if not download it with command:
cd /lib/firmware && sudo wget https://github.com/JeffyCN/mirrors/raw/libmali/firmware/g610/mali_csffw.bin
Download OpenCL ICD loader and manually add libmali to ICD
sudo apt update
sudo apt install mesa-opencl-icd
sudo mkdir -p /etc/OpenCL/vendors
echo "/usr/lib/libmali-valhall-g610-g6p0-x11-wayland-gbm.so" | sudo tee /etc/OpenCL/vendors/mali.icd
Download and install
libOpenCL
sudo apt install ocl-icd-opencl-dev
Download and install dependencies for Mali OpenCL
sudo apt install libxcb-dri2-0 libxcb-dri3-0 libwayland-client0 libwayland-server0 libx11-xcb1
Download and install clinfo to check if OpenCL successfully installed
sudo apt install clinfo
Validate Installation¶
To verify you have correctly installed OpenCL runtime and Mali GPU driver, run clinfo
in command line and see if you can get the GPU information.
You are expect to see the following information:
$ clinfo
arm_release_ver: g13p0-01eac0, rk_so_ver: 3
Number of platforms 2
Platform Name ARM Platform
Platform Vendor ARM
Platform Version OpenCL 2.1 v1.g6p0-01eac0.2819f9d4dbe0b5a2f89c835d8484f9cd
Platform Profile FULL_PROFILE
...