After a day or 2, I came up with an alternative solution by passing the environment values in Dockerfile while building my project.

TLDR

1. Pass your env values within dockerfile
2. Set all your env (dev, staging, prod, etc) var values in Pipeline variable.
3. Set a "settable" variable inside the Pipeline variable too, so you can set to build different environment while triggering your pipeline (eg, buildEnv)
4. Setup a bash script to perform variable text changing (eg, from firebaseApiKey to DEVfirebaseApiKey ) according to env received from buildEnv.
5. Use "replace token" task from Azure Pipeline to replace values inside Dockerfile
6. Build your docker image
7. Huaala~ now you get your environment specific build

Details

Within your Dockerfile you can place your env variable like this

RUN NEXT_PUBLIC_ENV=#{env}# \ 
  NEXT_PUBLIC_FIREBASE_API_KEY=#{firebaseApiKey}# \ 
  NEXT_PUBLIC_FIREBASE_AUTH_DOMAIN=#{firebaseAuthDomain}# \ 
  NEXT_PUBLIC_FIREBASE_PROJECT_ID=#{firebaseProjectId}# \ 
  NEXT_PUBLIC_FIREBASE_STORAGE_BUCKET=#{firebaseStorageBucket}# \ 
  NEXT_PUBLIC_FIREBASE_MESSAGING_SENDER_ID=#{firebaseMessagingSenderId}# \ 
  NEXT_PUBLIC_FIREBASE_APP_ID=#{firebaseAppId}# \ 
  NEXT_PUBLIC_FIREBASE_MEASUREMENT_ID=#{firebaseMeasurementId}# \ 
  NEXT_PUBLIC_BASE_URL=#{baseURL}# \ 
  npm run build 

These values set (eg, baseURL, firebaseMeasurementId, etc) are only a placeholder, because we need to change them later with bash script according to the buildEnv we receive. (buildEnv is settable when you trigger a build)

Bash script sample as below. What it does is that it will look within your Dockerfile for the word env and change to DEVenv / UATenv / PRODenv based on what you're passing to buildEnv

#!/bin/bash
case $(buildENV) in
dev)
sed -i -e 's/env/DEVenv/g' ./Dockerfile
;;
uat)
sed -i -e 's/env/UATenv/g' ./Dockerfile
;;
prod)
sed -i -e 's/env/PROenvD/g' ./Dockerfile
;;
*)
echo -n "unknown"
;;
esac

When this is complete, your "environment specific" docker file is sort of created. Now we'll make use of the "replace token" task from Azure Pipeline to replace the values inside Dockerfile. **Make sure you have all your values setup in Pipeline Variable!

Lastly all you may build your docker image and deploy :)