In order to see if you can run any LLM model on your system, refer to the information below to see our recommendations for running LLMs on your system.
As of the time of writing, this list only covers systems running NVIDIA and AMD GPUs with ROCm support. If you are using a Intel GPU, Apple Silicon, or a AMD GPU that does not support ROCm, our recommendation is to either use KoboldCPP or use cloud hosting.
For AMD Users with ROCm support
If you have a AMD GPU that supports ROCm, you are limited to running YellowRoseCx's fork of KoboldCPP or using cloud hosting.
¶ Find How Much VRAM You Have
¶ Windows
¶ Windows 10/11
Open Task Manager, go to Performance, select your GPU and see Dedicated GPU memory.
¶ Windows 8.1 and lower
- Press Win+R on your keyboard, type
dxdiagand hit Enter. You should see something similar to this screen below.
- Click on Display 1 (or any display really) and you should see something similar to this screen below.
- Put the number of MB you have under Display Memory (VRAM) and Google X MB to GB (replace X with the value of MB you have). You should roughly get how much VRAM there is on your GPU.
¶ Linux
This guide is for NVIDIA GPUs only.
Install nvtop onto your system and run it (using nvtop). You should see how much VRAM you have to the top-right of the nvtop session.
¶ System Recommendations
The bigger the better(usually). Use this calculator to figure out which models you can run. It's not perfectly accurate but it gives you a good idea of what you can run.
When picking which quant to use, you should generally aim to run as big of a quant as your hardware allows, but Q5_K_M / 6.0bpw is the sweet spot where you get both speed and performance. Q4_K_M / 5.0bpw is a good alternative if Q5_K_M / 6.0bpw is too slow. Going below IQ4KS / 4.0bpw is not really recommended because the quality loss increases exponentially the smaller the quant is. Bigger models (>30B) are more forgiving when it comes to small quants, allowing you to get good outputs even at Q3_K_M / 3.75bpw.
¶ ≤6GB VRAM
Running models on 6 or less GB of VRAM is rough and you'll be limited to ≤8B models with really low quants which will heavily impact the quality. You have a few choices:
- Use KoboldCPP. This backend allows you to load models partially on your CPU.
This depends on how much RAM you have on your system. See Task Manager to see how much RAM you have.
¶ 8GB VRAM
If your system has 8GB of VRAM, it starts becoming possible to run models locally comfortably. It's recommended to:
- Use TabbyAPI. This backend specializes in running exl2 models. Highly recommended if you have enough VRAM to load the entire model.
- Use KoboldCPP to load models you can't quite fit entirely on your GPU.
¶ ≥12GB VRAM
12GB VRAM allows you to run medium models very comfortably and is usually the sweet spot when it comes to how much VRAM you want Same as before, it's recommended to:
- Use TabbyAPI. This will be your go-to.
- Use KoboldCPP if you want to get a taste of the large models (>20B).