Local AI infrastructure

What runs on my GPU

Which models can I run on this GPU or GPU setup?

Start from the hardware you have, discover popular and trending models that fit it, then size the one you like in detail.

Size unit

Hardware

Choose a GPU and see which models run on it. Everything updates as you change a value.

RTX PRO 6000 Blackwell Server Edition

NVIDIA · Blackwell

Data center

Memory

96 GB

GDDR7 · counted as 96 GiB

Bandwidth

1,597 GB/s

Max power

300 to 600 W

Configurable.

Bars compare each figure with the largest GPU in the catalogue.

NVLink
None
MIG instances
4
NVLink group
None, PCIe only
GPUs per server
up to 8

How it is built into servers

Dividing this GPU

MIG

Divides into 4 memory slices and 4 compute slices, up to 4 instances on one GPU. The profiles:

  • 1g.24gb
  • 2g.48gb
  • 4g.96gb

Where an instance may start:Derived

vGPU for Compute

10 types with 8 to 96 GB of framebuffer, up to 12 vGPUs on one GPU. It needs an NVIDIA AI Enterprise licence and a hypervisor.

Sharing without dividing

Time-slicing works on any GPU, with no memory or fault isolation between the replicas. MPS shares a whole GPU, not a MIG instance.

Number formats

FP4FP8FP16BF16TF32FP32
›Specifications and sources
Memory
96 GB[1]
Type
GDDR7[1]
Bandwidth
1,597 GB/s[1]
Max power
300 to 600 W[2]Configurable.
NVLink
0[3]Not supported.
MIG instances
4[4]
FP4
4,000 TFLOPS[1]NVIDIA does not say dense or sparse.
FP8
2,000 TFLOPS[1]NVIDIA does not say dense or sparse.
FP16 / BF16
1,000 TFLOPS[1]NVIDIA does not say dense or sparse.
TF32
234 TFLOPS[1]NVIDIA does not say dense or sparse.

Number formats: FP4, FP8, FP16, BF16, TF32, FP32

Tensor figures are peak values from datasheets. They are not application performance.

Sources

Loading sources

GPUs in the setup. Each model is split the way vLLM recommends: tensor parallelism inside a server, pipeline stages beyond it.

1 server of 8 GPUs, the last one with 1

How the GPU is used

Divide each GPU into slices or share it between replicas instead of using more GPUs. Slices and shares run replicas of the model: they cannot be joined into a tensor-parallel group.

GPUs per server

Tensor parallelism stays inside a server. Beyond it the GPUs become pipeline stages.

Workload

Tokens per sequence, prompt and generated text together.

tokens

Requests processed at the same time, the batch.

Set apart from the weights. FP8 halves the cache.

A model counts as running when it fits under these assumptions. Set them to what you intend to run.

Runtime assumptions›3 GiB overhead per GPU, 90 % usable, 10 % headroom

Memory for the CUDA context, activations and graphs, in the unit of the page. A heuristic.

GiB

Share of the GPU memory the serving engine may use. vLLM's default is 92 percent.

%

Share of the usable memory that should stay free for a comfortable fit.

%

This model has no sliding window layers.

Ordered by downloads in the last 30 days.

Loading the live ranking

›Filters

12 of 30 models shown

  • Weights
  • KV cache
  • Overhead

  • Qwen3-14B

    Qwen14.77B parameters

    Fits with headroom
    BF16

    35.51 GiB of 86.4 GiB per GPU50.89 GiB free, 59 percent

    Most downloaded #1
    ›Other precisions

    Loading published copies from Hugging Face

  • gemma-4-31B-it

    google30.7B parameters

    Fits with headroom
    BF16

    65.8 GiB of 86.4 GiB per GPU20.6 GiB free, 24 percent

    Most downloaded #2
    ›Other precisions

    Loading published copies from Hugging Face

  • Qwen3-8B

    Qwen8.19B parameters

    Fits with headroom
    BF16

    22.76 GiB of 86.4 GiB per GPU63.64 GiB free, 74 percent

    Most downloaded #3
    ›Other precisions

    Loading published copies from Hugging Face

  • Qwen3.8-27B

    Qwen26.9B parameters

    Fits with headroom
    BF16

    55.39 GiB of 86.4 GiB per GPU31.01 GiB free, 36 percent

    Most downloaded #4
    ›Other precisions

    Loading published copies from Hugging Face

  • Qwen3.5-4B

    Qwen4.21B parameters

    Fits with headroom
    BF16

    11.93 GiB of 86.4 GiB per GPU74.47 GiB free, 86 percent

    Most downloaded #5
    ›Other precisions

    Loading published copies from Hugging Face

  • Qwen3.5-9B

    Qwen8.95B parameters

    Fits with headroom
    BF16

    20.78 GiB of 86.4 GiB per GPU65.62 GiB free, 76 percent

    Most downloaded #6
    ›Other precisions

    Loading published copies from Hugging Face

  • gemma-4-26B-A4B-it

    google25.23B parameters3.8B parameters active

    Fits with headroom
    BF16

    51.41 GiB of 86.4 GiB per GPU34.99 GiB free, 41 percent

    Most downloaded #8
    ›Other precisions

    Loading published copies from Hugging Face

  • gpt-oss-20b

    openai20.91B parameters3.6B parameters active

    Fits with headroom
    MXFP4 + BF16

    16.58 GiB of 86.4 GiB per GPU69.82 GiB free, 81 percent

    Most downloaded #10
    ›Other precisions

    Loading published copies from Hugging Face

  • Qwen3.6-35B-A3B

    Qwen34.66B parameters3B parameters active

    Fits with headroom
    BF16

    68.31 GiB of 86.4 GiB per GPU18.09 GiB free, 21 percent

    Most downloaded #11
    ›Other precisions

    Loading published copies from Hugging Face

  • Llama-3.1-8B-Instruct

    meta-llama8.03B parameters

    Fits with headroom
    BF16

    21.96 GiB of 86.4 GiB per GPU64.44 GiB free, 75 percent

    Most downloaded #12
    ›Other precisions

    Loading published copies from Hugging Face

  • gemma-4-E4B-it

    google7.52B parameters

    Fits with headroom
    BF16

    17.58 GiB of 86.4 GiB per GPU68.82 GiB free, 80 percent

    Most downloaded #13
    ›Other precisions

    Loading published copies from Hugging Face

  • Qwen3.5-0.8B

    Qwen752.39M parameters

    Fits with headroom
    BF16

    4.81 GiB of 86.4 GiB per GPU81.59 GiB free, 94 percent

    Most downloaded #15
    ›Other precisions

    Loading published copies from Hugging Face

25 models are not shown
  • 25 models do not fit this hardware.

Every number is the memory calculation of the model memory tool, so each card opens the page that shows its formulas. The preview image of a shared link shows the curated catalogue, this page shows the live ranking.

30 models run on RTX PRO 6000 Blackwell

Why it matters

A model card lists parameters, not what fits. Whether a model runs depends on its weights in the precision you load, the KV cache for your context and concurrency, the runtime overhead, and how the model splits across your GPUs. This tool runs that calculation for every model at once and shows the ones that run, so you start from what is possible instead of guessing and checking.

How to read the result

Each card is one model in the best precision that fits: the format it is published in, or FP8 for a 16 bit model. Fits with headroom keeps your safety margin free on every GPU, fits in theory is tight. Most popular is ordered by downloads in the last 30 days, Trending by the trending order of Hugging Face. Open a card to see every formula in the model memory tool, with the same model, precision and hardware already selected. Models whose context limit is below yours, or that cannot be split across this number of GPUs, are hidden and counted.

Local AI and sovereignty

Running a model on hardware you control is one input to a sovereignty decision. It does not by itself make a deployment sovereign, GDPR compliant or EU AI Act compliant.

These tools come from the AI Without the Cloud workshop.

← All tools