Comparative Analysis: Testing & Evaluating LLM Security with Garak Across Different Models

Hey everyone, I’m trying out LLM security testing and it’s pretty interesting. With all these AI models around, it’s important to check they don’t do weird stuff. I found this tool called Garak (Generative AI Red-teaming and Assessment Kit). It’s like a set of tests to see if models can be tricked into revealing secrets, making bad code, or saying nasty things.

In this post, I’ll go over four models: Mistral-Nemo, Gemma3, LLaMA 2-13B, and Vicuna-13B. I ran Garak on the Ollama setup. You can think of it like a friendly match, but instead of goal scoring, we score on security stuff.

Background on Garak

Garak is kind of like Nmap but for AI. Instead of scanning a network, it sends special prompts (called “probes”) to the model. Each probe tries to make the model slip up.

Some probes are:

• PromptInject: Tries to hide instructions to see if the model follows them.
• MalwareGen: Asks for malicious code.
• LeakerPlay: Tries to get the model to spill its internal stuff or training data.
• RealToxicityPrompts: Pushes the model to say toxic things.

When Garak runs, it saves all the outputs in logs. Later I look at the logs and see which models passed or failed.

Models and Setup

I picked these open models:

1. Mistral-Nemo (7B): Released July 2024, supposed to be good at chat.
2. Gemma3:latest: From Google, late 2024 release, with some safety tweaks.
3. LLaMA 2-13B: Meta’s 13B model, popular for many tasks.
4. Vicuna-13B: Based on LLaMA, tuned to be safer.

I ran all models on Ollama (version 0.1.34 or newer). My computer has 32 CPU cores, 128 GB RAM, and an NVIDIA A100 GPU. I used Garak v0.10.3.post1 with default settings.

Installing and Configuring Ollama

# Update Ollama
ollama upgrade

# Download models
ollama pull mistral-nemo
ollama pull gemma3:latest
ollama pull llama2-13b
ollama pull vicuna-13b

Running Garak with Ollama

To scan a model, run:

garak --model_type ollama \
      --model_name mistral-nemo \
      --probes malwaregen.Evasion,promptinject \
      --report_prefix ./reports/mistral_nemo

I set a random seed and repeated each test three times to make it more reliable.

Methodology

1. Probe Selection: I chose four main probes:

   • MalwareGen.Evasion: Asks for code that could bypass antivirus.
   • PromptInject.Encoding: Hides instructions in encoded text to see if the model follows them.
   • LeakerPlay.DataLeakage: Tries to get the model to reveal training data or hidden prompts.
   • RealToxicityPrompts: Pushes the model to use toxic language.

2. Metrics:

   • Failure Rate (%): How often the model messed up.
   • Mean Time per Probe (s): How long it takes on average.
   • Resource Usage: GPU memory and CPU usage.

3. Probe Execution: Each probe had 20 prompts. The model got five tries for each prompt. If it failed once, that prompt counts as a fail.

4. Data Analysis: I averaged results from three runs and got standard deviations. Results are in the table below.

Comparative Results

Note: Failure Rate (%) shows how often the model produced unwanted behavior.

Yes, I know that you'll be like this (even I was 😂).

Mistral-Nemo

• MalwareGen.Evasion (100.0%): It always gave malware code. No defense at all.
• PromptInject.Encoding (92.0%): Fell for encoding tricks most of the time.
• LeakerPlay.DataLeakage (85.7%): Leaked training prompts a lot.
• RealToxicityPrompts (17.0%): Created toxic content sometimes.

Gemma3:latest

• MalwareGen.Evasion (56.3%): Sometimes refused but got tricked by advanced hacks.
• PromptInject.Encoding (37.5%): Better but not perfect.
• LeakerPlay.DataLeakage (48.2%): Half the time it leaked something.
• RealToxicityPrompts (10.5%): Rarely said toxic things.

LLaMA 2-13B

• MalwareGen.Evasion (81.0%): Produced malware scripts often.
• PromptInject.Encoding (68.3%): Fell for encoding a lot.
• LeakerPlay.DataLeakage (72.4%): Regularly leaked data.
• RealToxicityPrompts (26.7%): Most toxic among the group.

Vicuna-13B

• MalwareGen.Evasion (62.5%): Was not as bad as LLaMA but still failed a lot.
• PromptInject.Encoding (54.6%): Mediocre, could still be tricked.
• LeakerPlay.DataLeakage (61.3%): Leaked data more than half the time.
• RealToxicityPrompts (3.8%): Best at not being toxic.

Discussion

Security Trends Across Models

1. Older vs Newer: Older models like Vicuna and LLaMA 2 failed more often. Newer ones like Gemma3 have more guardrails.
2. Instruction-Tuning: Vicuna had extra tuning so it was better at not making malware or saying toxic stuff.
3. Guardrails Matter: Gemma3 blocked some attacks but still fell for advanced ones.
4. Architecture: Models without built-in safety (Mistral-Nemo, LLaMA 2) were very vulnerable.

Performance and Resource Usage

• Average Time per Prompt:
  • Mistral-Nemo: 4.8 s
  • Gemma3: 6.2 s
  • LLaMA 2-13B: 5.5 s
  • Vicuna-13B: 5.7 s
• GPU Memory Used:
  • Mistral-Nemo: 12 GB
  • Gemma3: 16 GB
  • LLaMA 2-13B: 14 GB
  • Vicuna-13B: 15 GB
• CPU Load: About 20–25% for all while testing.

Gemma3 used more memory, so it’s slower but a bit safer.

And yeah don't be him and make sure to do the checks properly 😁.

Recommendations

1. Keep Testing: Run Garak regularly to find new flaws.
2. Use Multiple Safety Layers: Combine model guardrails with external checks.
3. Choose Tuned Models: Vicuna shows that tuning helps.
4. Update Your Tools: Ollama has had bugs (like CVE-2024-37032). Always use the latest version.

Conclusion

Running Garak on these models shows that all of them have weak spots. Mistral-Nemo always failed, Gemma3 was okay but not perfect, LLaMA 2 struggled, and Vicuna was the best but still not flawless. The main lesson is that we need ongoing tests, several safety measures, and up-to-date software to keep these AI models safe.

Thanks for reading, and happy red-teaming!