Shell Patterns - Intelligent Output Compression

Every shell command run through ctx_shell passes through lean-ctx's shell compression pipeline. With 60+ pattern modules covering 60+ developer tools, the system strips boilerplate while preserving actionable output - typically saving 50-95% of tokens per command.

How Shell Compression Works

When ctx_shell executes a command, the raw output flows through a multi-stage compression pipeline before reaching the agent:

Compression Pipeline

Raw command output
  │
  ├─ 1. ANSI strip        → Remove color codes, cursor escapes
  ├─ 2. FilterEngine       → Apply user-defined TOML filters
  ├─ 3. Pattern matching   → Detect tool (git, npm, cargo, ls, find, etc.)
  ├─ 4. Pattern compress   → Tool-specific structured extraction
  ├─ 5. JSON schema        → Extract structured data from JSON output
  ├─ 6. Log dedup          → Collapse repeated log lines
  └─ 7. Test compress      → Summarize test suite results
  │
  ▼
Compressed output (~50-95% smaller)

Track Mode vs Compress Mode

ctx_shell supports two modes that control how output is processed:

# Compress mode (default) - returns compressed output
ctx_shell command="npm test"
→ ✓ 42 passed, 0 failed (3.2s)  [saved 1,847 tok]

# Track mode - returns only metadata
ctx_shell command="npm install" mode="track"
→ ✓ exit=0 (12.4s)  [output suppressed]

60+ Pattern Modules

Each pattern module knows the structure of a specific tool's output and extracts only the information an agent needs. Patterns are organized by ecosystem:

Version Control

JavaScript / Node

Python

Rust

Go

Docker & Cloud

Databases

System

Other Ecosystems

Compression Examples

# git status - 87% savings
ctx_shell command="git status"
→ branch:main ±0 ahead
  M src/auth.ts
  M src/server.ts
  ? src/new-file.ts
  [saved 523 tok, 87%]

# cargo test - 91% savings
ctx_shell command="cargo test"
→ test result: ok. 156 passed; 0 failed; 2 ignored (4.8s)
  [saved 3,412 tok, 91%]

# npm install - 71% savings
ctx_shell command="npm install"
→ added 12, removed 0, changed 3, audited 847 packages
  0 vulnerabilities
  [saved 214 tok, 71%]

Structural Output Protection

Diff-producing commands receive special treatment: git diff, git show, git blame, git log -p, git stash show, and standalone diff tools (diff, colordiff, icdiff, delta) are never aggressively compressed. Only lightweight pattern cleanup is applied (ANSI stripping, trailing whitespace removal) — all +/- lines, hunk headers, and blame annotations are preserved verbatim to ensure agents can rely on structural diff output for accurate code analysis.

Passthrough Commands

Some commands are never compressed and always return raw output. lean-ctx maintains a list of 85+ passthrough entries (added in v3.3.1) covering three categories:

1. Dev Servers & Long-Running Processes

Commands that produce continuous streaming output are passed through because compression would either buffer indefinitely or lose real-time context:

# These always return raw output:
npm run dev          # Next.js / Vite dev server
yarn dev             # Yarn dev server
docker compose up    # Docker container logs
kubectl logs -f      # Streaming pod logs
tail -f              # File watching

2. Interactive GitHub CLI Commands

gh pr commands and similar interactive flows are passed through to preserve formatting and interactivity:

gh pr create         # PR creation flow
gh pr view           # PR detail view
gh pr checks         # CI status display
gh issue create      # Issue creation

3. Smart Script-Runner Detection

lean-ctx uses a heuristic to detect script-runner commands that launch dev servers. When a package.json script name contains dev, start, or serve, the command is automatically treated as passthrough:

npm run dev          → passthrough (contains "dev")
yarn start           → passthrough (contains "start")
pnpm serve           → passthrough (contains "serve")
bun run dev:server   → passthrough (contains "dev")
npm run build        → compressed (no trigger word)

FilterEngine: Custom Filters

Beyond built-in patterns, you can define custom compression filters using TOML files. Place filter files in ~/.lean-ctx/filters/ and lean-ctx applies them before pattern matching in the pipeline.

Filter File Format

Each filter file uses [[rules]] syntax to define match-action pairs:

# ~/.lean-ctx/filters/custom.toml

[[rules]]
match = "regex"
pattern = "^\[INFO\]\s+Starting.*"
action = "drop"

[[rules]]
match = "regex"
pattern = "^WARNING: .+"
action = "keep"

[[rules]]
match = "contains"
pattern = "DEPRECATED"
action = "drop"

[[rules]]
match = "prefix"
pattern = "DEBUG:"
action = "drop"

Rule Fields

Filters are applied in order. The first matching rule wins. Lines that match no rule pass through unchanged.

Example: Suppress Verbose Logging

# ~/.lean-ctx/filters/suppress-verbose.toml

[[rules]]
match = "regex"
pattern = "^\[(TRACE|DEBUG)\]"
action = "drop"

[[rules]]
match = "contains"
pattern = "Compiling"
action = "drop"

# Keep everything else (implicit - unmatched lines pass through)

Bypassing Compression

Sometimes you need the full, uncompressed output. lean-ctx provides three escape hatches:

1. raw=true Parameter

Pass raw=true to ctx_shell to disable compression for a single command:

ctx_shell command="npm test" raw=true
→ [full uncompressed output...]

2. LEAN_CTX_RAW=1 Environment Variable

Set the environment variable to disable compression for the current command. Useful when piping or in scripts:

LEAN_CTX_RAW=1 npm test
→ [full uncompressed output, no lean-ctx processing]

3. LEAN_CTX_DISABLED=1 - Disable Entire Hook

Completely disables the lean-ctx shell hook for the current shell session. No interception, no compression, no tracking:

export LEAN_CTX_DISABLED=1
# All commands now bypass lean-ctx entirely

unset LEAN_CTX_DISABLED
# lean-ctx hook is active again

Smart Heredoc Detection

Starting with v3.3.x, lean-ctx includes intelligent heredoc handling in ctx_shell and the PreToolUse hook.

The Problem

Earlier versions would block or incorrectly process heredoc commands, because the shell hook couldn't distinguish between a heredoc (inline data) and a file redirect (writing to disk). This caused issues with commands like:

# This was incorrectly blocked:
cat <<'EOF'
some inline data
EOF

# This SHOULD be blocked (file write via heredoc):
cat <<'EOF' > /etc/config.toml
[server]
port = 8080
EOF

The Solution

lean-ctx now distinguishes between these two cases:

• Pure heredoc (no file redirect): Passed through without blocking. The command runs normally and output is compressed as usual.
• Heredoc + file redirect (>, >>): Blocked by the PreToolUse hook, as this is a file write operation that should use ctx_edit or native file editing instead.

Detection Logic

# ✓ Allowed - pure heredoc, no file redirect
cat <<EOF
inline content
EOF

# ✓ Allowed - heredoc piped to another command
cat <<EOF | grep "pattern"
search this content
EOF

# ✗ Blocked - heredoc with file redirect
cat <<EOF > output.txt
file content
EOF

# ✗ Blocked - heredoc with append redirect
cat <<EOF >> log.txt
log entry
EOF

The PreToolUse hook inspects the command string for heredoc markers (<<) combined with redirect operators (>, >>). Only the combination triggers blocking - heredocs alone pass through cleanly.