How I Accidentally Locked Everyone Out of Our System (Including Admins) — And Why It Was the Best Mistake of My Career
Hmm...
My journey with Keycloak started when I joined the first tech company that I worked with as a fresh graduate and this was during my NYSC days. I was assigned a task to verify the email functionality during user registration.
That is, when a new user signs up, they should automatically receive a confirmation email.
I was told to “just check the settings” in Keycloak without any detailed instructions or documentation on how to go about this. There was no warnings about how critical certain actions on IAM platforms like Keycloak can be or the consequences that changes to keycloak settings can bring if not done properly.
As a newbie, I didn’t really know where to start from, because I didn't have any prior knowledge of keycloak or how it works. And also the tutorials I found online were either too basic or way too advanced.
I tried reaching out to our team lead for some explanations but I couldn't get across to him.
Eventually, I decided to try things myself. I went into the Keycloak Admin Console , navigated to the User Management Settings ,
And this exactly where everything went wrong. I went to:
Realm Settings → Email → Enabled ‘Verify Email’ (thinking this was just for new users).
In the Required Actions section, I Ticked ‘Verify Email’ as a default action for all users.
What I didn’t realize at the time was that Keycloak isn’t just a tool you can tweak casually. It’s a central piece of our system. And certain settings — like requiring email verification — apply globally by default. Altering a critical settings like this that controlled whether users could be verified via email affected all existing test users in the system.
So when I turned on ‘Verify Email’, I wasn’t just affecting new users — I accidentally locked out every existing user in our test database.
And this goes to show why we shouldn’t tightly couple our systems to IAM providers without planning for these kinds of disruptions.
One of the positive parts of this experience is that Mistakes like this teach us how to build better, safer, and more resilient systems.
Now this particular action forced every existing user to reconfirm their email—even admins! No one could generate tokens. Postman blew up with 401 Unauthorized errors. And our test environment was immediately locked down."
Within minutes, one of my senior colleagues noticed something was wrong. He couldn’t generate a token from Postman using any user from Keycloak — he kept on getting this 401 Unauthorized error and 403 Forbidden errors.
At first, I didn’t realize that the errors were as a result of my actions. But after some digging, I realized it was my change that caused the issue. I immediately explained to him what I did. He then escalated the problem to the Tech Lead, and I got a call from him immediately and it was not a pleasant one at all. I was strongly reprimanded over the phone.
Everyone blamed me. It was tough for me at that period. This made it felt like the end of the world.
But here’s the good part: the affected users were test users in a migration database we were using to test our new authentication system. So what we did, is to simply repopulated the database with fresh test users, and everything was okay again.
I know that you might be wondering what Keycloak is, especially if you are not a software engineer, now lemme disect keycloak for you.
Keycloak is an open-source Identity and Access Management (IAM) solution developed by Red Hat. It allows you to secure applications and services with minimal effort.
Some of its key features include:
User authentication and authorization
Single Sign-On (SSO)
Social login via Google, Facebook, etc.
Role-Based Access Control (RBAC)
And it also allowes for customizable themes and workflows
In the real world, companies use Keycloak to manage access to internal tools, APIs, web applications, mobile apps, etc —in a very secure and efficient way.
In my case, I was working with the email verification flow — which is part of Keycloak’s User Management settings. Changing that without understanding the implications caused a chain reaction across the system.
But another good part of this mistake I made was that it exposed all the existing vulnerabilities in our system and in the company.
The first vulnerability exposed was tight coupling between Keycloak and our internal application logics. our applications relied heavily on Keycloak for user state, permission, and authentication tokens.
So when I toggled “Verify Email”, Keycloak started enforcing the rule globally — and our app had no way to bypass or handle that scenario gracefully.
And as a result, all existing users were blocked from logging in because they hadn’t verified emails. The entire system broke because one centralized component which is Keycloak changed behavior unexpectedly.
Another issue exposed was the danger of lack of proper documentations and Guidance to interns. I was told to "just check the settings" — with no clear context, documentation, or mentorship. This led to my blind experimentation in a critical system. And as result of this, I toggled something without knowing its implications which is very dangerous in IAM systems like Keycloak.
Now looking back to this particular experience, there are several lessons I learnt from the whole process:
Firstly, Interns shouldn't be allowed to have access to important credentials and to even touch critical functionalities.
Secondly, If you are an intern, always ask for clear specifications before diving into a task.
Make sure to understand a system thoroughly before making changes — especially in production-like environments.
Always reach out proactively to mentors or seniors when stuck.
Make sure to use sandbox environments to test changes before applying them to real systems.
And finally document every issues you encounter and steps you took to rectify the problems for future reference.
Now after that scenario, I immediately turned that experience into a learning opportunity.
Today, I’ve grown to understand Keycloak better. In fact, before I left the company, I led the implementation of a full-scale Role-Based Access Control (RBAC) system using Keycloak + a custom Permissions Module that I engineered from scratch. I used a hybrid approach that provides us fine-grained control over user permissions across the platform.
Now, I’m considering creating a detailed, step-by-step tutorial series on how to built a robust RBAC system using Keycloak. If you would like to see that, please leave a comment below saying “Yes!”
And also to all the interns and junior devs out there — don’t be discouraged by mistakes. Just like I said in one of my previous videos, “ MISTAKE IS DATA NOT DEFEAT”. They’re not failures; they’re stepping stones.
And to the teamleads out there, mentorship and communication go a long way. A small amount of guidance can prevent big issues down the line.
Thanks for your time. If you found this story helpful or relatable or If you have ever made a ‘career-defining’ mistake, Share your story below! 👇 And if you want a deep dive on avoiding Keycloak pitfalls lemme know in the comment section.
Until next time, keep building!
🔗 Connect with me on all platforms here:
👉 https://linktr.ee/EzeDev
🎬 Subscribe to my YouTube channel for more real-world software engineering projects:
👉 youtube.com/@buildingwitheze?sub_confirmation=1
Top comments (0)