Security of AI

Security Controls

OWASP AI Security and Privacy Guide
OWASP Top 10 for LLM: promopt injection, insecure output handling, training data poisioning, DOS, supply chain risk, access permissions, data leakage, excessive agenctm overreliance, insecure plugins https://owasp.org/www-project-top-10-for-large-language-model-applications/assets/PDF/OWASP-Top-10-for-LLMs-2023-v05.pdf

Risk Managment Framework

Artificial Intelligence Risk Management Framework (AI RMF 1.0) https://nvlpubs.nist.gov/nistpubs/ai/nist.ai.100-1.pdf
LLM OWASP Top 10: Very useful, but some of them are a stretch. Currently at v0.5 https://owasp.org/www-project-top-10-for-large-language-model-applications/assets/PDF/OWASP-Top-10-for-LLMs-2023-v05.pdf
Prompt injection methods: GitHub - greshake/llm-security: New ways of breaking app-integrated LLMs
Skyflow data privacy for GPT: https://www.skyflow.com/post/generative-ai-data-privacy-skyflow-gpt-privacy-vault
Daniel Miessler on AI Attack Surface Map: https://danielmiessler.com/blog/the-ai-attack-surface-map-v1-0/
Generative AI: 5 Guidelines for Responsible Development - Salesforce
Nvidia’s AI red team framework: :https://developer.nvidia.com/blog/nvidia-ai-red-team-an-introduction/
IBM AI fairness 360 tools to detect bias: https://www.ibm.com/opensource/open/projects/ai-fairness-360/ -
tldrsec on a similar topic: How to securely build product features using AI APIs (tldrsec.com)
MITRE Atalas & Mitigations https://atlas.mitre.org/mitigations
Google Secure AI Framework (SAIF)

Red Teaming for AI/ML

https://www.airedteam.org/
Giskard The testing framework for ML models, LLMon monitoring solution
Microsoft https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/red-teaming
Hugging Face https://huggingface.co/blog/red-teaming
Meta’s Bot Adversarial Dialog dataset
Anthropic’s red-teaming attempts
AI2’s RealToxicityPrompts
Whitehouse https://www.whitehouse.gov/ostp/news-updates/2023/08/29/red-teaming-large-language-models-to-identify-novel-ai-risks/
Google https://blog.google/technology/safety-security/googles-ai-red-team-the-ethical-hackers-making-ai-safer/
Anthropic: https://www.anthropic.com/index/frontier-threats-red-teaming-for-ai-safety
Auto red Teaming LLM https://github.com/traghav/auto-redteam#auto-red-teaming-llms

Risk areas & tools

Rebuff: A self-hardening prompt injection detector.
LLMFuzzer: A fuzzing framework for LLMs.
LLM Guard: A security toolkit for LLM interactions ,open source LLM Firewall
LLM Gateway: Helps ensure that data being fed into and output from the LLM tool is free of proprietary company data.
Data leakage, Inadequate sandboxing
LLM vulnerability scanner - garak https://github.com/leondz/garak
Arthur Shield: The First Firewall for LLMs
Lakera is an AI security company. They have specific products to protect against Prompt injection: Lakera Guard | Unlock LLMs for Production | Lakera – Protecting AI teams that disrupt the world.
Generative Discriminator Guided Sequence Generation (GeDi) or Plug and Play Language Models (PPLM)

AI Security for Enterprise

Enterprise architectures for LLMs (a16z)
Corporate GenAI policies gartner
Gandalf chatbot security game counters privacy fireballs
Should you buy or build: When it comes to large language models, should you build or buy? | TechCrunch
Companies blocking ChatGPT and other publicly trained chatbots: Employees are banned from using ChatGPT at these companies | Fortune
Google thinks open source LLMs will be as good as OpenAI soon: Google’s Leaked Document Reveals Open Source Threat: A New Era in Language Models | BigTechWire
Triveto language model whitepaper: https://www.truveta.com/wp-content/uploads/2023/04/Truveta-Language-Model.pdf

Playgrounds

Lope Kinz: GPT4 Free, PentestGPT, . ChatGPT client
Gelei Deng: Pentest GPT
GPT 3 Vulnerability Scanner
Orange Software for Customer ML models
AutocodePro
smol.ai - Promopt engineering
GreyDGL/PentestGPT
dorkgpt
GPT Integration - Nucliei, BurpGPT, HackGPT, CloudGPT,
Aviary Explorer: A way to compare results from open source LLMs: Aviary Explorer (anyscale.com)
A playground for prompt injection. Basically tricking LLMs in revealing secrets https://gandalf.lakera.ai/
Holistic evaluation of LLMs (HELM) from Stanford: https://crfm.stanford.edu/helm/latest/

Name		Name	Last commit message	Last commit date
Latest commit History 9 Commits
LICENSE		LICENSE
README.md		README.md
main.py		main.py

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Security of AI

Security Controls

Risk Managment Framework

Red Teaming for AI/ML

Risk areas & tools

AI Security for Enterprise

Playgrounds

About

Uh oh!

Releases

Packages

Uh oh!

Contributors

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

Security of AI

Security Controls

Risk Managment Framework

Red Teaming for AI/ML

Risk areas & tools

AI Security for Enterprise

Playgrounds

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Packages