Install Cuda

Install CUDA Toolkit

Note

This section is only required if you’re running LLMs locally on your physical machine.

For Google Colab users:

  • Skip this installation (Colab provides pre-installed CUDA environments)

  • Proceed directly to Get Model API keys

Install Cuda

  1. Cuda Install

    Determine which version of cuda you should download

    In the terminal, type

    nvidia-smi   //This will give information of your graphics driver
// Find this line
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 566.36                 Driver Version: 566.36         CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+

    The following guide will use CUDA 12.8 as an example

    1. Visit CUDA Toolkit Archive

    2. Select version matching your framework requirements:

    ../_images/Archive.png

    1. Follow the installation instruction and choose default setups

    // Verify installations
nvcc -v

  2. cuDNN Version a. Visit cuDNN Toolkit Archive

    Download
    ../_images/cudnn_archive.png ../_images/cudnn_zip.png

    Unzip everything from cuDNN zip to your local CUDA directory

    ../_images/cudnn_files.png

    Create Environment Variables

    ../_images/env_var.png 
    C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.8\bin
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.8\lib
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.8\include
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.8\libnvvp