DEV Community

Cover image for 🧨 What I Broke Wednesday: Leaking API Keys
Sumit Roy
Sumit Roy

Posted on

🧨 What I Broke Wednesday: Leaking API Keys

The Setup

We'd just migrated from hardcoded config files to environment variables. Security team was happy, developers were happy, I was feeling like a DevSecOps hero:

export DATABASE_URL="postgresql://user:pass@prod-db:5432/app"
export STRIPE_SECRET_KEY="sk_live_..."
export JWT_SIGNING_KEY="super-secret-key-here"
Enter fullscreen mode Exit fullscreen mode

"Look how secure we are now!" I declared. "No more secrets in git!"

The Overconfidence

Our monitoring team wanted better visibility into our processes. I implemented a health check system that would dump process information for debugging:

// "Helpful" debugging endpoint
app.get('/debug/processes', (req, res) => {
  const processes = execSync('ps aux').toString();
  res.json({ processes });
});
Enter fullscreen mode Exit fullscreen mode

What could go wrong?

Image description

The Catastrophe

Here's what I forgot: environment variables are visible in process lists.

$ ps aux | grep node
app  1234  0.1  NODE_ENV=production DATABASE_URL=postgresql://user:supersecret@prod JWT_SIGNING_KEY=abc123...
Enter fullscreen mode Exit fullscreen mode

Our "secure" secrets were broadcast to:

  • Anyone with shell access
  • Process monitoring tools
  • Log aggregation systems
  • That debug endpoint I helpfully created

The Horror Unfolds

Security audit day arrives. Pentester runs ps aux, immediately finds our production database credentials, Stripe keys, and JWT secrets.

Pentester: "So... are these supposed to be visible?"
Me: "Visible where?"
Pentester: shows screen
Me: "Oh. OH NO."

The Real Learning Moment

Environment variables aren't actually secure for secrets. They're just... convenient. The real solutions:

Option 1: HashiCorp Vault 

// Proper secret management
const vault = require('node-vault')({
  endpoint: process.env.VAULT_ADDR,
  token: process.env.VAULT_TOKEN
});

const secrets = await vault.read('secret/data/app/prod');
const dbUrl = secrets.data.data.DATABASE_URL;
Enter fullscreen mode Exit fullscreen mode

Option 2: Kubernetes Secrets 

# Mount secrets as files, not env vars
volumeMounts:
- name: secrets
  mountPath: "/etc/secrets"
  readOnly: true
Enter fullscreen mode Exit fullscreen mode 
// Read from mounted files
const dbUrl = fs.readFileSync('/etc/secrets/database-url', 'utf8');
Enter fullscreen mode Exit fullscreen mode

The Fix (And Prevention)

  1. Never pass secrets via environment variables in production
  2. Use proper secret management (Vault, AWS Secrets Manager, etc.)
  3. Mount secrets as files when possible
  4. Audit process visibility regularly
  5. Rotate exposed secrets immediately (learned this the hard way)

Current approach:

// Startup secret fetch
async function loadSecrets() {
  if (process.env.NODE_ENV === 'production') {
    return await vault.read('secret/data/app/prod');
  }
  // Local development can still use .env files
  return process.env;
}
Enter fullscreen mode Exit fullscreen mode

The Aftermath

  • Emergency secret rotation (fun weekend that was)
  • Implemented proper Vault integration
  • Security team now reviews all env var usage
  • That debug endpoint? Yeah, that's gone.

How do you handle secrets in production? Still using env vars, or have you graduated to proper secret management? Share your war stories below - we've all leaked something at some point! 🔐

Tomorrow: Throwback Thursday - That time I thought I understood microservices (spoiler: I didn't)

Top comments (0)