LLM Model LandscapeIntermediate

Quantization and Inference Flow

The Llama lineup and licence, serving with Ollama and vLLM, GPU memory sizing, and an honest look at when a hosted API is still the better choice.

By Amit UpadhyayPublished 13 Oct 202611 min

Step 1 / 4

Speed

What's happening

Load Model

Load Llama 3.1 8B with bfloat16 — runs on single consumer GPU (RTX 3090)

💡 Analogy

Like downloading and installing software — you need enough disk space (GPU memory) to run it.

Load the model from Hugging Face with 16-bit precision (bfloat16). This halves memory from ~16GB (float32) to ~8GB — fits on a single RTX 3090 or A10 GPU.

Console Output

> Loading meta-llama/Llama-3.1-8B-Instruct
> torch_dtype: bfloat16 (8GB)
> device_map: auto → cuda:0
> Model loaded in 12.3s

●llama.py

Load ModelL5–11

1from transformers import AutoTokenizer, AutoModelForCausalLM

2import torch

4# Load Llama 3.1 8B from Hugging Face Hub

5model_id = "meta-llama/Llama-3.1-8B-Instruct"

6tokenizer = AutoTokenizer.from_pretrained(model_id)

7model = AutoModelForCausalLM.from_pretrained(

8 model_id,

9 torch_dtype=torch.bfloat16, # 16-bit weights: 16GB → 8GB

10 device_map="auto", # auto-assign to GPU(s)

11)

13# Format with Llama 3 chat template

14messages = [

15 {"role": "system", "content": "You are a helpful assistant."},

16 {"role": "user", "content": "Explain gradient descent in simple terms."},

17]

18input_ids = tokenizer.apply_chat_template(

19 messages,

20 add_generation_prompt=True,

21 return_tensors="pt",

22).to(model.device)

24# Generate with sampling

25with torch.no_grad():

26 outputs = model.generate(

27 input_ids,

28 max_new_tokens=256,

29 temperature=0.6,

30 top_p=0.9,

31 do_sample=True,

32 )

34response = tokenizer.decode(

35 outputs[0][input_ids.shape[-1]:], # trim input from output

36 skip_special_tokens=True

37)

38print(response)

38 lines

Steps · 4

What's happening

Load Model

Load Llama 3.1 8B with bfloat16 — runs on single consumer GPU (RTX 3090)

💡 Analogy

Like downloading and installing software — you need enough disk space (GPU memory) to run it.

Load the model from Hugging Face with 16-bit precision (bfloat16). This halves memory from ~16GB (float32) to ~8GB — fits on a single RTX 3090 or A10 GPU.

Console Output

> Loading meta-llama/Llama-3.1-8B-Instruct
> torch_dtype: bfloat16 (8GB)
> device_map: auto → cuda:0
> Model loaded in 12.3s

Next: Test →

Llama open-weight self-hosting vLLM quantization

← PreviousEvals for Choosing a Model Next →Claude: Anthropic's Model Family in Practice

Continue reading

All LLM Model Landscape articles →

LLM Model Landscape11 min

GPT: OpenAI's Model Family in Practice

The GPT-6 tiers, what reasoning effort replaced, when to use the Responses API over Chat Completions, how the bill is calculated, and what is different on Azure.

LLM Model Landscape11 min

Claude: Anthropic's Model Family in Practice

The four Claude tiers, adaptive thinking and effort, the tool-use rules that differ from other providers, prompt caching, and where you can run it.

LLM Model Landscape10 min

Evals for Choosing a Model

Why public benchmarks do not transfer to your workload, how to build a model-selection eval, and why cost per successful task beats price per token.

Discussion

LLM Model LandscapeIntermediate

Quantization and Inference Flow

The Llama lineup and licence, serving with Ollama and vLLM, GPU memory sizing, and an honest look at when a hosted API is still the better choice.

By Amit UpadhyayPublished 13 Oct 202611 min

Step 1 / 4

Speed

What's happening

Load Model

Load Llama 3.1 8B with bfloat16 — runs on single consumer GPU (RTX 3090)

💡 Analogy

Like downloading and installing software — you need enough disk space (GPU memory) to run it.

Load the model from Hugging Face with 16-bit precision (bfloat16). This halves memory from ~16GB (float32) to ~8GB — fits on a single RTX 3090 or A10 GPU.

Console Output

> Loading meta-llama/Llama-3.1-8B-Instruct
> torch_dtype: bfloat16 (8GB)
> device_map: auto → cuda:0
> Model loaded in 12.3s

●llama.py

Load ModelL5–11

1from transformers import AutoTokenizer, AutoModelForCausalLM

2import torch

4# Load Llama 3.1 8B from Hugging Face Hub

5model_id = "meta-llama/Llama-3.1-8B-Instruct"

6tokenizer = AutoTokenizer.from_pretrained(model_id)

7model = AutoModelForCausalLM.from_pretrained(

8 model_id,

9 torch_dtype=torch.bfloat16, # 16-bit weights: 16GB → 8GB

10 device_map="auto", # auto-assign to GPU(s)

11)

13# Format with Llama 3 chat template

14messages = [

15 {"role": "system", "content": "You are a helpful assistant."},

16 {"role": "user", "content": "Explain gradient descent in simple terms."},

17]

18input_ids = tokenizer.apply_chat_template(

19 messages,

20 add_generation_prompt=True,

21 return_tensors="pt",

22).to(model.device)

24# Generate with sampling

25with torch.no_grad():

26 outputs = model.generate(

27 input_ids,

28 max_new_tokens=256,

29 temperature=0.6,

30 top_p=0.9,

31 do_sample=True,

32 )

34response = tokenizer.decode(

35 outputs[0][input_ids.shape[-1]:], # trim input from output

36 skip_special_tokens=True

37)

38print(response)

38 lines

Steps · 4

What's happening

Load Model

Load Llama 3.1 8B with bfloat16 — runs on single consumer GPU (RTX 3090)

💡 Analogy

Like downloading and installing software — you need enough disk space (GPU memory) to run it.

Load the model from Hugging Face with 16-bit precision (bfloat16). This halves memory from ~16GB (float32) to ~8GB — fits on a single RTX 3090 or A10 GPU.

Console Output

> Loading meta-llama/Llama-3.1-8B-Instruct
> torch_dtype: bfloat16 (8GB)
> device_map: auto → cuda:0
> Model loaded in 12.3s

Next: Test →

Llama open-weight self-hosting vLLM quantization

← PreviousEvals for Choosing a Model Next →Claude: Anthropic's Model Family in Practice

Continue reading

All LLM Model Landscape articles →

LLM Model Landscape11 min