Skip to main content
Software Engineering

Return to Answer

added 3022 characters in body
Source Link
Flater
Flater
  • 59.5k
  • 8
  • 112
  • 171

Just to give you an example of how this works day to day in my team (I am the tech lead):

  • Developers work on a task locally on their machine. We have a DEV environment with cloud resources that they can hook into when running the service locally. They are also able to deploy the code to the DEV environment so it can be tested more long-term and with everyone being able to access it and work with it.
  • When they are finished with their work, they open a pull request for review.
  • When the PR is approved and merged into the main branch (if not approved, the above steps repeat until it is approved), the developers create a new pipeline run from the main branch.
  • This pipeline will (when clicking the button to do so) deploy the code, including any db migration script if necessary, to the targeted environment.
  • First, they deploy the pipeline to DEV again to ensure that the software build is in a deployable state.
  • Then, they deploy it to our TEST environment.
  • Now the pipeline goes in the freezer. Developers/QA/stakeholders use the TEST environment to vet the new version. This contains a demo to the product manager, regression tests to check for behaviors that have broken, performance tests when we expect the change to affect the application's behavior, and an overall sanity check that all requirements are met.
  • When everyone is happy, the developer opens a release request ticket. They include documentation of what the change contains, whether this deploy comes with an outage, evidence of approval from all necessary stakeholders, a step-by-step deployment plan which is almost always just "perform the PROD deploy step in [this] pipeline", and a step by step guide on what to do in case that deployment does not work or something break (usually this links to the last deployed pipeline in PROD so that we can rerun that one to bring PROD back to its previous state).
  • I review the release request, approve it, and I start the PROD deploy step of that pipeline. It is paramount that the pipeline I use to deploy the code is the exact same pipeline that was used to deploy to the TEST environment, so that I am sure that we are deploying the exact same version and not some other version that may have other changes added into it.
  • After release, the developer vets that the changes have made it into the PROD version and does a basic sanity test. If OK, the release request is marked as a success. If not OK, we perform the rollback maneuver and repeat this process depending on what it is that didn't work.

It sounds like a lot of work but it is actually significantly easier to perform these individual steps than it is to rely on developers manually doing everything on their machine and hoping that they don't make a mistake. I can rely on the pipeline deploying all necessary content and for it to behave consistently for every release, instead of opening the door to anyone being able to forget anything during any release because they have to manually keep track of everything.

I strongly suggest you move towards a pipeline-driven deployment system. This not only provides safety in not accidentally touching PROD, it also provides clear build versioning, so that you can be sure that the thing you're intending to deploy to PROD is the exact same version that you deployed to your test environment (no sneaky commits that get added inbetween your test and PROD deploy).

I strongly suggest you move towards a pipeline-driven deployment system. This not only provides safety in not accidentally touching PROD, it also provides clear build versioning, so that you can be sure that the thing you're intending to deploy to PROD is the exact same version that you deployed to your test environment (no sneaky commits that get added inbetween your test and PROD deploy).

Just to give you an example of how this works day to day in my team (I am the tech lead):

  • Developers work on a task locally on their machine. We have a DEV environment with cloud resources that they can hook into when running the service locally. They are also able to deploy the code to the DEV environment so it can be tested more long-term and with everyone being able to access it and work with it.
  • When they are finished with their work, they open a pull request for review.
  • When the PR is approved and merged into the main branch (if not approved, the above steps repeat until it is approved), the developers create a new pipeline run from the main branch.
  • This pipeline will (when clicking the button to do so) deploy the code, including any db migration script if necessary, to the targeted environment.
  • First, they deploy the pipeline to DEV again to ensure that the software build is in a deployable state.
  • Then, they deploy it to our TEST environment.
  • Now the pipeline goes in the freezer. Developers/QA/stakeholders use the TEST environment to vet the new version. This contains a demo to the product manager, regression tests to check for behaviors that have broken, performance tests when we expect the change to affect the application's behavior, and an overall sanity check that all requirements are met.
  • When everyone is happy, the developer opens a release request ticket. They include documentation of what the change contains, whether this deploy comes with an outage, evidence of approval from all necessary stakeholders, a step-by-step deployment plan which is almost always just "perform the PROD deploy step in [this] pipeline", and a step by step guide on what to do in case that deployment does not work or something break (usually this links to the last deployed pipeline in PROD so that we can rerun that one to bring PROD back to its previous state).
  • I review the release request, approve it, and I start the PROD deploy step of that pipeline. It is paramount that the pipeline I use to deploy the code is the exact same pipeline that was used to deploy to the TEST environment, so that I am sure that we are deploying the exact same version and not some other version that may have other changes added into it.
  • After release, the developer vets that the changes have made it into the PROD version and does a basic sanity test. If OK, the release request is marked as a success. If not OK, we perform the rollback maneuver and repeat this process depending on what it is that didn't work.

It sounds like a lot of work but it is actually significantly easier to perform these individual steps than it is to rely on developers manually doing everything on their machine and hoping that they don't make a mistake. I can rely on the pipeline deploying all necessary content and for it to behave consistently for every release, instead of opening the door to anyone being able to forget anything during any release because they have to manually keep track of everything.

I strongly suggest you move towards a pipeline-driven deployment system. This not only provides safety in not accidentally touching PROD, it also provides clear build versioning, so that you can be sure that the thing you're intending to deploy to PROD is the exact same version that you deployed to your test environment (no sneaky commits that get added inbetween your test and PROD deploy).

Source Link
Flater
Flater
  • 59.5k
  • 8
  • 112
  • 171

The issue here is one of not using the right tool for the job. I understand the environment you're in and that this is reality at some companies but it's difficult to give you a good answer without telling you to shift towards tooling that is built for this purpose (and brings a bunch of other benefits with it).

Just to contrast it to your scenario, in companies with a modern approach to development:

  • Developers do not have access to production databases in the first place - or at least not on their regular account, or at least without write access, only read access.
  • Deployments, including data migrations, are performed by pipelines and external build agents, not developers on their machine.
  • The idea of having a "prod solution" is non-existent. There is only one solution, and the configuration of the codebase (such as db connection strings) are stored externally to the codebase (there can be local copies in the code but they are purely for running things locally for development purposes)
  • Deployments come with sufficient red tape to ensure that no one clicks the PROD deploy button until every necessary stakeholder has verified the release and signed off on it. Generally speaking, that manifests as a much smaller group of individuals having the authority to kick off the PROD deploy step, and they rely on some kind of external mechanism to indicate to them that everyone is good to go with the PROD release - in our case we have dedicated Jira tickets for it that must pass via the appropriate release gates.

I strongly suggest you move towards a pipeline-driven deployment system. This not only provides safety in not accidentally touching PROD, it also provides clear build versioning, so that you can be sure that the thing you're intending to deploy to PROD is the exact same version that you deployed to your test environment (no sneaky commits that get added inbetween your test and PROD deploy).

How to use pipelines and build agents is a much bigger topic than this answer can reasonably contain. I strongly suggest you look into this and shift into relying on them for your release (and therefore db update) strategy.

When you dig into it, it may seem like a large piece of work, and setting it up when you're new to it will definitely require some conscious effort, but it will pay back dividends on your development process and being able to focus on development because you know your releases are gated and you don't have to run around worrying about a person mistakenly updating a PROD version (whether code or database) when they shouldn't.