AI Without the Cloud

Open models are easy to download and hard to operate

After the workshop you can put an open-weights model on a GPU, serve it through an OpenAI-compatible API, measure what it does, change one thing at a time and explain the result. You decide with numbers, and you can say what each decision costs.

The center is inference engineering: what happens between a request arriving and the last token leaving, which resource limits it, and what you trade when you change something. Every optimization trades quality, latency, throughput, concurrency, memory, cost and operational complexity against each other. No configuration is fastest for every workload, only one fits yours.

Three phases

• Understand and run. Modern models, GPU fundamentals, sizing weights and KV cache, prefill and decode, model selection. Then your first vLLM server.
• Measure and optimize. TTFT, inter-token latency and throughput from a benchmark harness. Then concurrency, continuous batching, KV cache, quantization and parallelism, one lever per experiment.
• Operate. Production architecture, containers, metrics and dashboards, scaling, security and digital sovereignty. RAG, tool calling and agents come last, as layers on top of inference, always tied back to capacity and cost.

Real GPUs, one shared server per group

You work on NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs with 96 GB of memory. The trainer provides one machine per group of three to five people, and you connect over SSH. A group shares one vLLM server, so you meet queueing, batching and contention first hand.

You keep one document through all five labs, the inference report. It starts with a workload and ends with a production architecture that you defend in front of another group.

By the end you have:

• selected models for four scenarios and defended the trade-offs
• run a vLLM server and read what it reports about memory and KV cache
• benchmarked it at several concurrency levels, with p50 and p95 latencies
• changed one setting per experiment and compared before and after
• read GPU and server metrics on a Grafana dashboard
• called a local model from code with a JSON schema
• designed a production inference architecture and named what each choice trades

Who it is for

Software, ML, platform and DevOps engineers and architects who decide how an organization runs open models, or who will operate them. You work on the command line and read Python. GPU or CUDA experience is not required, and you do not need to have trained a model.

What you need

A laptop with git, ssh, docker, python3 (3.11 or newer) and curl, about 20 GB of free disk space and an SSH key. You do not need a GPU, a cloud account or a Hugging Face token. The preparation steps wait on your workshop page once you are enrolled.

Before the first module

You arrive with a laptop that reaches your group's GPU machine.

• Preflight and access. Check your laptop, send the trainer your public SSH key and test your login to your group's GPU machine. No GPU or cloud account needed.
  Task: Run the preflight and test your login

Understand and run

You learn what a model asks of a GPU, size a deployment on paper, choose models for a workload and bring up your first vLLM server.

1. Opening: why run models yourself. Six reasons to run a model yourself and what each one costs, five sovereignty dimensions, and how training, fine-tuning and inference differ.
2. Modern language models. How a prompt becomes tokens and the next token. Layers, KV heads and parameters set memory, active parameters set compute.
3. GPU fundamentals for inference. Which limit a GPU meets first: compute, memory bandwidth, capacity or interconnect. Read datasheets without mixing dense and sparse, and use ops:byte.
4. GPU sizing: weights, KV cache, overhead. Estimate weights, KV cache and overhead for a model, then check the KV capacity against the server's start-up log with the sizing script.
   Task: Predict and check GPU memory with the sizing script
5. Inference mechanics: prefill, decode, latency. Follow one request through prefill and decode, mark TTFT, ITL and tokens per second, and see why decode waits for memory and prefill for compute.
6. Model selection. Pick the smallest model that reliably does your task: cheap checks first, licenses per repository, memory fit by arithmetic, and a small eval of your own.
7. Lab 1: select models for four scenarios. Choose model, precision, engine and context for four scenarios, check the fit with sizing.py and write down what each choice gave up.
   Task: Select and defend models for four scenarios
8. First vLLM deployment (trainer demo). See how vllm serve turns weights into an OpenAI-compatible API, call it with curl and Python, and read the start-up log.
9. Lab 2: first inference server. Follow one request from prompt to metrics on a shared vLLM server and check that your KV cache arithmetic matches the server's log within 2 %.
   Task: Follow one request through your first inference server

Measure and optimize

You predict where the bottleneck is, then measure it. Each optimization is one change, compared before and after.

10. Checkpoint: bottleneck predictions. Rebuild the chain client, API, vLLM, model, GPU and predict the bottleneck, the metric and the direction of six changes before you measure.
11. Measuring inference. Learn which metric answers which question, why percentiles need enough requests, and how closed loop, open loop and goodput differ.
12. Lab 3: benchmark the baseline. Run a concurrency sweep of 1, 4, 8 and 16 on the baseline, keep the results with provenance and find where throughput and latency turn.
    Task: Benchmark the baseline at four concurrency levels
13. Continuous batching. How the vLLM scheduler forms a batch at every step, what a cap on the running batch does to throughput and TTFT, and what chunked prefill trades.
14. KV cache, PagedAttention and prefix caching. Why the KV cache limits concurrency, what PagedAttention changes, which three signals show memory pressure, and when a shared prefix saves prefill.
15. Quantization. What fewer bits buy for weights, activations and the KV cache, what they cost, and how to read a quality probe honestly, with FP8 results on one GPU.
16. Parallelism: replicas, tensor, pipeline, expert. What replicas, tensor, pipeline and expert parallelism split, what crosses the link, and the arithmetic for when a second GPU helps. Never measured here.
    Task: Compute whether the all-reduce eats the gain from a second GPU
17. Lab 4: optimize the deployment. Test one lever with a written prediction and one changed variable, then report the trade-off first: quantization, prefix cache, KV pool or batch cap.
    Task: Test one optimization lever and report its trade-off
18. Trainer demos: speculative decoding and OOM. Speculative decoding pays on copy tasks and costs on new text under load. Two starts must fail. Three demos have no result. Skipping keeps three slides.

Operate

You turn a server into a service: architecture, containers, metrics, scaling and security. RAG and agents follow as layers on top, and you finish by defending a design.

19. Production architecture. The hops between a user's request and the GPU, what fails at each, and how state, streaming, time-outs, retries and queues behave.
20. Containerization and reproducible deployments. Why docker run does not record a deployment: host, image and mounts, what to pin, and how to build and run a client image against the group endpoint.
    Task: Build and run the client image
21. Observability: metrics, dashboards, incident investigation. Answer up, fast enough and saturated from vLLM metrics, read the Grafana dashboard row by row, and diagnose an incident from the panel that moved first.
    Task: Diagnose an incident from the dashboard
22. Scaling strategies. Bigger GPU, more replicas or a split model: what each buys, how start-up time limits autoscaling, and how to turn an SLO into a GPU count and a cost.
23. Security, compliance and digital sovereignty. Map where prompts, logs, embeddings and weights go, separate infrastructure from application security, and score a setup per workload on five dimensions.
    Task: Map where each kind of data goes
24. RAG without framework hype. RAG as seven readable steps, two of them model calls: chunking, similar versus relevant, freshness and the extra prompt tokens that retrieval adds.
    Task: Run RAG with and without retrieval
25. Tool calling, MCP and agents. Tool calling as a protocol the application owns, structured output against a JSON schema, MCP, and how one request fans out into several model calls.
    Tasks: Call the model from code with a JSON schema; Run the token ledger of a tool-calling agent
26. Lab 5: production architecture challenge. Design, size and defend an inference platform for one of four scenarios, working backwards from the SLO to GPUs, cost and risks.
    Task: Design and defend a production inference platform
27. Architecture defenses, decision framework, close. Defend your Lab 5 architecture, ask the questions that expose weak evidence, and walk the 15 decision questions for a new workload.