WizrAI is a modular local AI environment created by Eugeni Soto (Ximbi). It combines a local LLM inference backend (based on llama.cpp) with a semi-autonomous terminal agent that can analyze, execute, and generate code directly on your system — with no cloud dependency.

“Where code meets magic.”

WizrAI is composed of two separate but connected layers:

🧙‍♂️ wizrai.py (Local Agent)

• Runs on your host machine
• Executes commands and creates files
• Analyzes and validates generated code
• Communicates with the model via HTTP

🐳 wizrai (Docker Container)

• Runs llama-server (llama.cpp)
• Loads a local .gguf model
• Processes prompts and streams responses
• Has no access to the host system

This modular design makes the container the brain (LLM) and the local agent the mind that reasons, validates, and executes actions in your real environment.

This allows you to use cloud-hosted models when you need internet access, extended reasoning, or creative writing.

Inside wizrai.py, you’ll find two versions of the call_llm_stream() function:

• One for local inference (default) → connects to your local llama.cpp backend.

• One for OpenRouter online inference → connects to the https://openrouter.ai/api/v1/chat/completions endpoint.

To switch between them, simply comment or uncomment the desired block.

• Go to OpenRouter.ai and get your API key.

• Export it in your terminal:

  • export OPENROUTER_API_KEY="sk-xxxxxxxx"

• In wizrai.py, comment the local version and uncomment the OpenRouter version.

Run WizrAI as usual:

./wizrai.py "summarize this article and generate a research plan"

A few free and public models are already listed in the code — you can just uncomment any of them:

wizrai/

├── Dockerfile

├── docker-compose.yml

├── wizrai.py

├── models/

│ └── .gitkeep

└── cache/

The WizrAI backend runs any compatible .gguf model using llama.cpp.

You can now pick between llama.cpp and Ollama:

• ./wizrai.py "…" → usa el servidor de llama.cpp iniciado mediante Docker.
• ./wizrai.py --backend ollama "…" → envía las peticiones al daemon de Ollama que corre en http://127.0.0.1:11434 (arráncalo con ollama serve).
• Exporta WIZRAI_BACKEND=ollama si quieres que Ollama sea el default.
• Cambia el modelo utilizado por Ollama con export OLLAMA_MODEL=phi3:mini y, si es necesario, export OLLAMA_HOST=http://127.0.0.1:11434.

• Docker and Docker Compose installed

• A compatible .gguf model (e.g. Gemma, Mistral, Llama

• Clone the repository:

  • git clone https://github.com/ximbi/wizrai.git
  • cd wizrai

• Download your .gguf file in 'models' folder, you can do it via wget or webpage:

  • wget https://huggingface.co/your-model/gemma-3n-E4B-it-Q4_K_M.gguf -O models/gemma-3n-E4B-it-Q4_K_M.gguf

• Start the LLM server:

  • docker compose up

This will compile llama.cpp, load your model, and expose port 8080.

The Dockerfile is tuned for long-running, stable inference:

4K context window

10 CPU threads

batch size 256

tuned repetition penalty

persistent cache at /cache/wizrai.cache

--mlock enabled for stable performance

You can easily tweak parameters in docker-compose.yml.

wizrai.py is a smart terminal agent capable of interpreting natural objectives, creating a plan, executing commands, reviewing results, and adapting autonomously.

Python 3.11+

Packages: requests, rich

pip install requests rich

• chmod +x wizrai.py
• ./wizrai.py "create a bash script that analyzes the top CPU-consuming processes, saves the output to a file, and displays it"

1. Formulate an incremental plan

2. Generate safe POSIX commands as JSON

3. Execute steps sequentially with logs and validation

4. Detect and correct errors automatically

5. Display diffs, validations, and results in real time

1. The agent sends prompts to the local server at http://127.0.0.1:8080/completion

2. The backend streams token responses

3. The agent interprets the JSON and executes the plan

Everything runs fully offline — no data is sent to external servers.

• models Stores your .gguf model Required (mounted into the container)
• cache Stores prompt cache Optional (auto-created)
• .gitkeep Keeps empty folders under version control Safe to ignore

• The container is the LLM brain

• The agent is the active thinker and executor

• The user retains full control over the system

• No external API calls required

• Secure, auditable, and extensible

You can modify models, parameters, or behaviors easily:

• Edit OPENROUTER_MODEL inside wizrai.py

• Adjust THREADS, CONTEXT, TOKENS, etc. in docker-compose.yml

Pull requests, ideas, and improvements are welcome. If you use WizrAI in your lab or workflow, share your experience!

MIT License © 2025 — Ximbi Based on llama.cpp