Selecting a Llama Model for Local Deployment

Hello again. Last lesson gave you a way to count the tokens a model must handle, including chat formatting and room for its answer. Now we can put that budget to work: which Llama model should run a particular local assistant? We’ll make a provisional choice for a private coding assistant on your RTX 3080 Ti, then see how the choice changes on your Macs.

A model name is not enough to answer this. You need four kinds of evidence: parameter count for an initial hardware estimate, context length for the intended requests, model-card evidence for capabilities and limitations, and license terms for permission to use and distribute the model as planned.

Start with the deployment, not the leaderboard

Suppose the first version of your coding assistant reads selected repository excerpts, answers questions about them, and runs locally on the desktop’s 12 GB RTX 3080 Ti. It needs text input and output, useful code responses, and enough context for a few selected files—not an entire repository in one prompt. Interactive speed matters because you will use it while developing.

Hugging Face’s short tour shows where to find model cards and filter out models that are plainly too large for local use:

For a first pass, consider Llama 3.2 1B Instruct, Llama 3.2 3B Instruct, Llama 3.1 8B Instruct, and Llama 3.3 70B Instruct. “Instruct” matters: an instruction-tuned model is intended to respond to assistant-style requests, whereas a base pretrained model is primarily a starting point for generation or adaptation. Parameter count alone cannot tell you which will write better code, but it can eliminate impractical candidates before you download them.

The Llama deployment guide is useful here for distinguishing model families—and for catching a particularly misleading size label:

For a conventional dense model, a rough weight-only estimate is parameter count multiplied by bytes per parameter. At 16-bit precision, an 8B model needs approximately $8$ billion times $2$ bytes, or 16 GB of weights. That already exceeds 12 GB of VRAM, before runtime buffers or context-related memory. At four bits per weight, the arithmetic gives roughly 4 GB, but a real quantized model uses additional storage and memory. This makes a four-bit 8B model a candidate for the desktop, not a guarantee that every configuration will fit. We’ll calculate fuller memory budgets later.

The same estimate explains why 70B is a poor first choice for this GPU: even its idealized four-bit weights alone would be about 35 GB. The desktop’s 128 GB of system RAM might permit some form of CPU or partial-GPU execution, but that is different from fitting the model in 12 GB of VRAM and may compromise interactive speed. For Llama 4’s mixture-of-experts models, sizing from “17B active” would make a similar mistake: total weights still have to be accommodated.

Read the card as evidence, not as a promise

Let’s inspect an official candidate rather than infer its properties from its name. Meta’s Llama 3.1 8B Instruct model card identifies the model’s intended use, specifications, supported languages, and governing license:

That card supports several concrete statements: this is an 8B, text-only, instruction-tuned candidate for assistant-like chat, with an advertised 128k-token context. It does not establish that 128k tokens will fit comfortably in your chosen runtime on the RTX 3080 Ti, or that the model will make effective use of every detail in a huge repository dump. An advertised maximum is a model capability; the configured context is what you actually allocate and test.

The card also lists supported languages and a knowledge cutoff. Those fields are easy to overlook. If an eventual RPG or document assistant must reliably work in a language not explicitly listed, test that requirement rather than extrapolating from English results. If it needs facts newer than the cutoff—or facts about your private repository—supply those facts in the request or through retrieval; a larger parameter count will not give the model access to files it has not seen.

Benchmarks help identify candidates, provided you read what was measured. This Ollama-hosted comparison places the two small Llama 3.2 instruction-tuned models alongside other lightweight models:

Within this table, Llama 3.2 3B scores 77.4 versus 59.5 for 1B on IFEval, an instruction-following evaluation, and 67.0 versus 25.7 on BFCL V2, a tool-use evaluation. Those differences make 3B more promising than 1B if your assistant must follow formatting instructions or eventually call tools. They do not prove that 3B beats 1B on every repository task, nor do they compare 3B directly with the 8B model we just read about. Published scores should guide a shortlist; a small set of your own representative coding questions should decide between shortlisted models.

Context and permission are separate gates

For the desktop assistant, you could begin with a configured 8,192-token context. Suppose a request includes 900 tokens of instructions and formatting, 1,000 of conversation, 4,200 of selected code, and a 700-token question. Reserving 1,000 tokens for the answer gives a planned total of 7,800 tokens, with 392 tokens of headroom. That fits the configuration. A 24,000-token document plus a 2,000-token answer does not—even though the 8B model card advertises a much larger maximum context.

This suggests a useful design rule: choose relevant excerpts first, and expand the configured context only when a real task needs it and local measurements support it. More context consumes resources and does not guarantee better attention to the right line of code. Later lessons will quantify the memory cost and build retrieval for larger document collections.

Permission requires an equally explicit check. The Llama 3.1 card identifies a custom Llama 3.1 Community License, states intended commercial and research uses, and points to an Acceptable Use Policy. “Commercial use intended” is not shorthand for “no conditions.” Before using a model in a client product, sharing weights, publishing an adapted model, or distributing a packaged assistant, read the full terms linked from the exact model version’s card and check that your distribution and intended use comply. Do not assume that a license check for Llama 3.1 automatically covers Llama 3.2 or a third-party quantized file. A local download also does not, by itself, settle the privacy behavior of the application you build around it.

A defensible first choice

For the specified private, interactive coding assistant on the RTX 3080 Ti, my provisional choice is Llama 3.1 8B Instruct in a compatible four-bit quantized format, initially configured for 8,192 tokens. The model card supports its assistant-oriented, text-and-code use; its weight estimate rules out a straightforward 16-bit GPU load but leaves a plausible quantized route; and the initial context fits the selected-excerpt workflow. The choice remains conditional on confirming the exact file’s provenance, reviewing the applicable terms, measuring memory and responsiveness, and testing answers on your repository.

I would keep Llama 3.2 3B Instruct as a smaller fallback or fast baseline, particularly on the 16 GB M2 Pro. On the 36 GB M4 Max, the 8B candidate has more memory room, though the usable allocation and speed still need measurement. I would not select 70B merely because a benchmark looks attractive, nor select a vast advertised context without a task that needs it.

The reusable decision is not a model name; it is a short record of task, hardware, estimated fit, planned context, card evidence, license review, and local test results. Next module, we’ll build the PyTorch foundations needed to understand what those billions of parameters actually do.