Llama Animate | AI Wisdom

← Back to topic

🎞 ANIMATE · Mode 3 of 5 · +100 XP— Llama (Meta) Open Source

📋Load

⚙️Template

🔄Generate

✅Decode

🎯 Next: Test →

Step 1 / 4

Speed

What's happening

Load Model

Load Llama 3.1 8B with bfloat16 — runs on single consumer GPU (RTX 3090)

💡 Analogy

Like downloading and installing software — you need enough disk space (GPU memory) to run it.

Load the model from Hugging Face with 16-bit precision (bfloat16). This halves memory from ~16GB (float32) to ~8GB — fits on a single RTX 3090 or A10 GPU.

Console Output

> Loading meta-llama/Llama-3.1-8B-Instruct
> torch_dtype: bfloat16 (8GB)
> device_map: auto → cuda:0
> Model loaded in 12.3s

●llama.py

Load ModelL5–11

1from transformers import AutoTokenizer, AutoModelForCausalLM

2import torch

4# Load Llama 3.1 8B from Hugging Face Hub

5model_id = "meta-llama/Llama-3.1-8B-Instruct"

6tokenizer = AutoTokenizer.from_pretrained(model_id)

7model = AutoModelForCausalLM.from_pretrained(

8 model_id,

9 torch_dtype=torch.bfloat16, # 16-bit weights: 16GB → 8GB

10 device_map="auto", # auto-assign to GPU(s)

11)

13# Format with Llama 3 chat template

14messages = [

15 {"role": "system", "content": "You are a helpful assistant."},

16 {"role": "user", "content": "Explain gradient descent in simple terms."},

17]

18input_ids = tokenizer.apply_chat_template(

19 messages,

20 add_generation_prompt=True,

21 return_tensors="pt",

22).to(model.device)

24# Generate with sampling

25with torch.no_grad():

26 outputs = model.generate(

27 input_ids,

28 max_new_tokens=256,

29 temperature=0.6,

30 top_p=0.9,

31 do_sample=True,

32 )

34response = tokenizer.decode(

35 outputs[0][input_ids.shape[-1]:], # trim input from output

36 skip_special_tokens=True

37)

38print(response)

38 lines

Steps · 4

What's happening

Load Model

Load Llama 3.1 8B with bfloat16 — runs on single consumer GPU (RTX 3090)

💡 Analogy

Like downloading and installing software — you need enough disk space (GPU memory) to run it.

Load the model from Hugging Face with 16-bit precision (bfloat16). This halves memory from ~16GB (float32) to ~8GB — fits on a single RTX 3090 or A10 GPU.

Console Output

> Loading meta-llama/Llama-3.1-8B-Instruct
> torch_dtype: bfloat16 (8GB)
> device_map: auto → cuda:0
> Model loaded in 12.3s

Was this page helpful?

Discussion

← Back to topic

🎞 ANIMATE · Mode 3 of 5 · +100 XP— Llama (Meta) Open Source

📋Load

⚙️Template

🔄Generate

✅Decode

🎯 Next: Test →

Step 1 / 4

Speed

What's happening

Load Model

Load Llama 3.1 8B with bfloat16 — runs on single consumer GPU (RTX 3090)

💡 Analogy

Like downloading and installing software — you need enough disk space (GPU memory) to run it.

Load the model from Hugging Face with 16-bit precision (bfloat16). This halves memory from ~16GB (float32) to ~8GB — fits on a single RTX 3090 or A10 GPU.

Console Output

> Loading meta-llama/Llama-3.1-8B-Instruct
> torch_dtype: bfloat16 (8GB)
> device_map: auto → cuda:0
> Model loaded in 12.3s

●llama.py

Load ModelL5–11

1from transformers import AutoTokenizer, AutoModelForCausalLM

2import torch

4# Load Llama 3.1 8B from Hugging Face Hub

5model_id = "meta-llama/Llama-3.1-8B-Instruct"

6tokenizer = AutoTokenizer.from_pretrained(model_id)

7model = AutoModelForCausalLM.from_pretrained(

8 model_id,

9 torch_dtype=torch.bfloat16, # 16-bit weights: 16GB → 8GB

10 device_map="auto", # auto-assign to GPU(s)

11)

13# Format with Llama 3 chat template

14messages = [

15 {"role": "system", "content": "You are a helpful assistant."},

16 {"role": "user", "content": "Explain gradient descent in simple terms."},

17]

18input_ids = tokenizer.apply_chat_template(

19 messages,

20 add_generation_prompt=True,

21 return_tensors="pt",

22).to(model.device)

24# Generate with sampling

25with torch.no_grad():

26 outputs = model.generate(

27 input_ids,

28 max_new_tokens=256,

29 temperature=0.6,

30 top_p=0.9,

31 do_sample=True,

32 )

34response = tokenizer.decode(

35 outputs[0][input_ids.shape[-1]:], # trim input from output

36 skip_special_tokens=True

37)

38print(response)

38 lines

Steps · 4

What's happening

Load Model

Load Llama 3.1 8B with bfloat16 — runs on single consumer GPU (RTX 3090)

💡 Analogy

Like downloading and installing software — you need enough disk space (GPU memory) to run it.

Load the model from Hugging Face with 16-bit precision (bfloat16). This halves memory from ~16GB (float32) to ~8GB — fits on a single RTX 3090 or A10 GPU.

Console Output

> Loading meta-llama/Llama-3.1-8B-Instruct
> torch_dtype: bfloat16 (8GB)
> device_map: auto → cuda:0
> Model loaded in 12.3s