Inference capacity

1. Step 11

   Check the driver

   Confirm that the NVIDIA driver is installed and new enough for the software below.

   Copy

   nvidia-smi

   What you should see

   • A table with the driver version and a CUDA version of at least 13.0, which the image needs.
   Help and sources

   If it does not work

   • If the CUDA version is lower than the software needs, update the driver and check again.

   • nvidia-smi(opens in a new tab)
   • CUDA minor version compatibility(opens in a new tab)

   Where the values come from

   • The GPU you chose: RTX PRO 6000 Blackwell.
2. Step 22

   Check the GPUs

   Confirm that the machine lists the GPU at index 0.

   Copy

   nvidia-smi -L

   What you should see

   • One line for each GPU, each naming RTX PRO 6000 Blackwell the way the machine spells it.
   Help and sources

   If it does not work

   • If a GPU you chose is missing, it is not in this machine or the driver cannot see it.
   • If the names do not match what you expect, the indices are not what you think. Identify the GPUs by UUID, which does not depend on the order.

   • nvidia-smi(opens in a new tab)

   Where the values come from

   • The calculator uses 1 GPU.
   • The GPU you chose: RTX PRO 6000 Blackwell.
3. Step 33

   Check that the memory is free

   The calculator expects 22.76 GiB on each GPU, of the 86.40 GiB the engine may use. Check that the memory of the GPU at index 0 is free.

   Copy

   nvidia-smi --query-gpu=index,name,memory.total,memory.used,memory.free --format=csv -i 0

   What you should see

   • Nearly all the memory of the GPU is free, at least 22.76 GiB on each.

   Keep in mind

   • Other work may already use these GPUs. If their memory is not free, stop that work or choose other GPUs before you start.
   Help and sources

   • nvidia-smi(opens in a new tab)

   Where the values come from

   • The calculator: 22.76 GiB needed of 86.40 GiB usable on each GPU.

Install the tools

1. Step 44

   Check Docker

   Confirm that Docker Engine is installed.

   Copy

   docker --version

   What you should see

   • A Docker version.
   Help and sources

   • Docker Engine: installation(opens in a new tab)
2. Step 55

   Install the NVIDIA Container Toolkit

   Needs administrator rights

   It lets containers use the GPUs. These commands are for Ubuntu and Debian, and the page linked below has the other distributions.

   Comes after step 1

   Copy

   sudo apt-get update && sudo apt-get install -y --no-install-recommends ca-certificates curl gnupg2
   curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
     && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
       sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
       sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
   sudo apt-get update
   export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.20.1-1
   sudo apt-get install -y \
     nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
     nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
     libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
     libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}

   What you should see

   • The packages install without errors.
   Help and sources

   • NVIDIA Container Toolkit: installation(opens in a new tab)
3. Step 66

   Connect Docker to the GPUs

   Needs administrator rights

   Register the NVIDIA runtime with Docker and restart it.

   Copy

   sudo nvidia-ctk runtime configure --runtime=docker
   sudo systemctl restart docker

   What you should see

   • No errors, and Docker comes back up.

   Keep in mind

   • Restarting Docker stops the containers that are running.
   Help and sources

   • NVIDIA Container Toolkit: installation(opens in a new tab)
4. Step 77

   Check that a container sees the GPUs

   Run nvidia-smi in a container that is given the GPU at index 0. Only that GPU should appear.

   Copy

   docker run --rm --runtime=nvidia --gpus '"device=0"' \
     ubuntu nvidia-smi

   What you should see

   • A table with one GPU, the ones you chose.
   Help and sources

   If it does not work

   • If Docker cannot start with the GPUs, the toolkit is not configured or the indices do not exist on this machine.
   • If the names do not match what you expect, the indices are not what you think. Identify the GPUs by UUID, which does not depend on the order.

   • NVIDIA Container Toolkit: sample workload(opens in a new tab)
   • NVIDIA Container Toolkit: Specialized configurations with Docker(opens in a new tab)

   Where the values come from

   • --gpus '"device=0"'

     Default The first GPUs, counted from 0.

Get the server

1. Step 88

   Download the image

   Download vllm/vllm-openai:v0.32.0 before the first start, so that the start only has to load the model.

   Copy

   docker pull vllm/vllm-openai:v0.32.0

   What you should see

   • The download completes and the last line names the image.

   Keep in mind

   • The image is several gigabytes.
   Help and sources

   • vllm/vllm-openai image tags(opens in a new tab)
   • vLLM: Docker deployment(opens in a new tab)

1. Step 99

   Start the server

   Start vllm-qwen3-8b with the setup you sized: the model Qwen/Qwen3-8B, the parallelism, the context and the memory it was given. The first start downloads the model and takes a few minutes.

   Comes after step 3

   Copy

   # config YSTVGT
   docker run -d --name vllm-qwen3-8b \
     --runtime nvidia --gpus '"device=0"' \
     --ipc=host \
     -v "$HOME/.cache/huggingface":/root/.cache/huggingface \
     -p 127.0.0.1:8000:8000 \
     vllm/vllm-openai:v0.32.0 \
     --model Qwen/Qwen3-8B \
     --max-model-len 8192 --max-num-seqs 4 \
     --gpu-memory-utilization 0.90

   What you should see

   • A container id.

   Keep in mind

   • The first start downloads the model and prepares the kernels, which takes minutes. The server is not ready before that.
   Help and sources

   If it does not work

   • If it stops with an out of memory message, the setup needs more than the GPU holds. Lower the context or the sequences, give the engine more of the GPU, or use more GPUs. The calculator shows by how much.
   • If vLLM says the context is longer than the KV cache can hold, lower the context or the number of sequences.
   • If vLLM says the attention heads cannot be divided by the tensor parallel size, use a size that divides them.
   • If the port is taken, choose another one in your choices.
   • If Docker cannot start with the GPUs, the toolkit is not configured or the indices do not exist on this machine.

   • vLLM: vllm serve arguments(opens in a new tab)
   • vLLM: engine arguments(opens in a new tab)

   Where the values come from

   • -d

     Default How you chose to run it.

   • --name vllm-qwen3-8b

     From the calculator The repository of the model you chose.

   • --runtime nvidia

     Default How you chose to run it.

   • --gpus '"device=0"'

     Default The first GPUs, counted from 0.

   • --ipc=host

     Default How you chose to run it.

   • -v "$HOME/.cache/huggingface":/root/.cache/huggingface

     Default Where the downloaded weights are kept.

   • -p 127.0.0.1:8000:8000

     Default The port the endpoint answers on.

   • --model Qwen/Qwen3-8B

     From the calculator The repository of the model you chose.

   • --max-model-len 8192

     From the calculator The context length you set: 8,192 tokens.

   • --max-num-seqs 4

     Worked out 4 sequences shared by 1 replica.

   • --gpu-memory-utilization 0.90

     Worked out The usable share you set, 90 percent, divided between 1 replica on the device, rounded down so the shares never add up to more than the device.

   • The repository of the model you chose.
   • The context length you set: 8,192 tokens.
   • The usable share you set, 90 percent, divided between 1 replica on the device, rounded down so the shares never add up to more than the device.

Check the endpoint

1. Step 1010

   Wait until it is ready

   The server answers /health once the model is loaded.

   Copy

   timeout 900 bash -c 'until curl -fs http://127.0.0.1:8000/health >/dev/null; do sleep 5; done' && echo ready

   What you should see

   • The word ready, after a minute or more while the model loads.
   Help and sources

   If it does not work

   • If it does not become ready, read the last log lines: the model is still loading, or the server stopped with an error.
   • If it stops with an out of memory message, the setup needs more than the GPU holds. Lower the context or the sequences, give the engine more of the GPU, or use more GPUs. The calculator shows by how much.

   • vLLM: online serving(opens in a new tab)
   • vLLM: OpenAI-compatible server(opens in a new tab)
2. Step 1111

   Check that the model is served

   Ask the server which models it serves.

   Copy

   curl -sS http://127.0.0.1:8000/v1/models

   What you should see

   • A list with Qwen/Qwen3-8B in it.
   Help and sources

   If it does not work

   • If the answer is 401, the key in this shell is not the one the server was started with.

   • vLLM: online serving(opens in a new tab)
   • vLLM: OpenAI-compatible server(opens in a new tab)
3. Step 1212

   Ask it something

   One request through the endpoint your applications will use.

   Copy

   curl -sS http://127.0.0.1:8000/v1/chat/completions \
     -H 'Content-Type: application/json' \
     -d '{"model": "Qwen/Qwen3-8B", "messages": [{"role": "user", "content": "Say hello in five words."}], "max_tokens": 32}'

   What you should see

   • A JSON answer with a message from the model.
   Help and sources

   If it does not work

   • If the answer is 401, the key in this shell is not the one the server was started with.
   • If it does not become ready, read the last log lines: the model is still loading, or the server stopped with an error.

   • vLLM: online serving(opens in a new tab)
   • vLLM: OpenAI-compatible server(opens in a new tab)

Look at the machine

1. Step 1313

   Look at the GPUs

   See the memory in use and the work on the GPU at index 0 while the server runs.

   Copy

   nvidia-smi --query-gpu=index,name,memory.used,memory.total,utilization.gpu --format=csv -i 0

   What you should see

   • Memory in use close to what the engine was given.
   Help and sources

   • nvidia-smi(opens in a new tab)
2. Step 1414

   Look at the server

   See that vllm-qwen3-8b is running, and what it last said.

   Comes after step 12

   Copy

   docker ps --filter name=vllm-qwen3-8b --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'
   docker logs --tail 20 vllm-qwen3-8b

   What you should see

   • The server is running, and its last log lines show no errors.
   Help and sources

   • vLLM: Docker deployment(opens in a new tab)

Compare with the calculator

1. Step 1515

   Compare with the calculator

   The calculator expected 22.76 GiB on each GPU, with 90 percent of the GPU given to the engine. The server's own log says what it took for the weights and what is left for the KV cache.

   Comes after step 12

   Copy

   docker logs vllm-qwen3-8b 2>&1 | grep -E 'Model loading took|Available KV cache memory|GPU KV cache size'

   What you should see

   • Lines with the memory the weights took, the memory left for the KV cache and how many tokens it holds.
   Help and sources

   • vLLM: parallelism and scaling(opens in a new tab)
   • vLLM: conserving memory(opens in a new tab)

   Where the values come from

   • The calculator: 22.76 GiB needed of 86.40 GiB usable on each GPU.

1. Step 1616

   Stop it

   Can stop or remove things

   Stop vllm-qwen3-8b and free the GPUs.

   Comes after step 9

   Copy

   docker stop vllm-qwen3-8b && docker rm vllm-qwen3-8b

   What you should see

   • No output, and the GPUs are free.

   Keep in mind

   • The endpoint stops answering.
   Help and sources

   • vLLM: Docker deployment(opens in a new tab)