Octopus Deploy is an amazing tool for managing deployments. It is easily customizable to fit any need and has an intuitive interface for getting started. Successfully deploying packages with Octopus Deploy is doesn’t take a lot. It is because of this flexibility, the ease in which projects can tailor any process, even a bad one, making our bad practices become successful. Though if left alone, these bad practices, and incorrect configurations can limit your scalability, and prevent you from getting the most out of the tool.
Here, I’m going to review a few setups I’ve seen, focusing on environments, and ways to mitigate them. The goal is to have 5 environments. Like most things, there are exceptions. If you want a personalized review of your Octopus Deploy setup, lets us know here at Clear Measure, and we’ll be happy to help you out.
The Goal
Our goal is to have 5 environments. They are dev, test, stage, UAT, and prod. Each has a purpose. How the resources in each environment are provisioned is up to the teams. This can be automatic and on demand, or static and persistent. The containing environments should be persistent and declared in your Octopus Deploy instance. First, I’ll describe the purpose of each environment, then I’ll go over some examples I’ve come across.
Dev
The dev environment is where our developers validate their changes in an environment that is more similar to production. While developing on their machines, teams may use a simulated cloud storage or database providers. The dev environment allows our teams to ensure that their changes, and integration tests pass by using the same resources as what the product will use in production. Developers should still be running unit and integration tests locally to ensure that tests will pass in the dev (and subsequent) environments.
Test
The test environment is where edge case testing and any other specialty testing should be preformed. This is where all manual testing is preformed. Any newly discovered bugs should be addressed and unit/integration tests written to ensure they don’t reappear.
Another test performed in this environment (not necessarily on the same machines) are performance/stress tests. Performance tests should use the same resource configurations as production to know how these changes will perform in production. Stress test are used to know the breaking point of your product. These tests can also be used to test your auto scaling and automated recovery, if you have those in place. If you are working specifically on performance improvements, you should still be working off a feature branch and submitting to a dev environment.
Stage
Staging is where you practice deploying to production or customers. Here you work out kinks in upgrading to your new version or new deployments of this version. This process can include taking a backup of production data and sanitizing it. Any errors found here can incorporate a hotfix pipeline to get changes to this version and be backfilled. Any applicable unit/integration tests should also be written. If any new logging or metrics need to be added, those things need to be ensured to be added here. Nothing should change after successful deployment to this environment except for blocking errors.
UAT
This environment may or may not be applicable. This is where customers may be first exposed new changes. An example would be canary deployments. Where the server that hosts the canary would be defined under the UAT environment in Octopus Deploy. It may be an entire infrastructure for UAT if you have customers requesting or paying for early access to new features. UAT is customer early access, whether by opting in, or routing small amount of traffic to test useability.
Prod
Production environment is where majority of users interact with your product. This is the targeted end goal of all software releases and metrics should be used. Everything up until this point has been practice and validation. We do our best to ensure that there are no surprises or unknown variables. This should be the smoothest, easiest deployment in the pipeline and the hardest to mess up.
The Case Studies
Next, let’s look at some setups and practices I’ve seen. We’ll look at the thought processes that led them there, what makes them less ideal, and how to correct them.
Environments Per Project
In this case, the instance had multiple project groups and projects in each group. Each group had their own set of environments. This idea can stem from a few thoughts. One of which is the idea of separation of responsibility. In the images below, take note of how much this practice inflates your infrastructure management in Octopus Deploy.
Separation of Responsibility
Teams want to keep projects separated and not have the potential of mixing one project from deploying to another. The idea is if each project have their own environment, there is no chance for the products to bleed into each environment. This desire is understandable, and sometimes dictated by organization requirements.
Multiple environments leads to overcomplicated setups. Every time you add an environment, you also increase the number of lifecycles, most of the time exponentially. For a basic 3 lifecycle setup (Full, Dev, Hotfix) each group of 5 environments will also add 3 lifecycles and renders the default lifecycle useless. Additionally, if there is any need to share infrastructure, it becomes more difficult and confusing.
Instead, teams should utilize roles to segregate the resources responsibilities. This allows more flexible deployments, less over complicated environment configuration and a more streamlined dashboard process. This also has the added benefit that all projects follow the same release lifecycles. If organization dictates that separate environments must be used, then I recommend setting up an entire new space for the project group. Spaces allow complete segregation of all things except items in the configuration and your packages found in the library. Everything else is segregated to include (but not limited to) tenants, library sets, script modules, and step templates.
Prevent Interrupting Working Processes
I’ve seen teams start creating environments for a project because they don’t want their changes to get in the way of another team’s already figured out process. This is mainly due to not knowing how Octopus Deploy works and is easily solved with some quick training.
Projects sharing environments don’t automatically share environment resources. That infrastructure sharing happens when reusing roles and tags. If you want your new project to use a new VM, tag your VM deployment target with a new role and have your project steps target this role.
Environment Per Customer
Sometimes we get all the environments correct but we get requests from our customers that they want their infrastructure environment. One response to this, I’ve seen, is to create a new environment named after the customer and add it to the production phase of the life cycle. While this works, it starts to break down when we start to get multiple requests for this. Additionally, variables like database connection strings, start to grow and get unwieldy. A better solution is to use Tenants.
Tenants
Octopus Deploy really shines because of their use of variables. One aspect where this really shines through is tenant variables. Tenants (think customer represented deployment) can have variable values specified for them. On top of that, these variables are defined on the project, but filled out on the tenant, preventing the values list for a variable from growing out of control. If you need help figuring out what kind of variable to use. you can check out an earlier post on that topic here.
Developer’s Machine
When starting a new project or proof of concept, developers may set up an environment named after them to lay claim to some infrastructure and not have someone else use their environment. These are less environments and more deployment targets. Similarly, you may have 5 dev environments, one for each team. These setup makes it complex as you have to add the environment to the dev phase of the life cycle, but it can’t be automatic.
Tenants
This is the same issue as the one environment for customer, but on the development side instead of production. The solution is the same. Yes, that’s right, you can use tenants, not only for production customers, but to segregate development deployments as well. This way you don’t have other teams stepping on each other validating changes. It doesn’t stop there, you can also have a tenant for testing specific use cases. One example is performance and stress testing. those usually have a very specific infrastructure. assigning tenants to those deployment targets is a clean way to manage these deployments. I just caution you that you have a clear branching strategy and how to go from a developer’s tenant/branch to production using an Octopus Deploy pipeline.
Geo Location Environments
Sometimes when setting up our SaaS deployments, an environment is made for each geo location. But aren’t these geo locations all still production. Again, this problem suffers from the same issues as the environment per customer setup. This is compounded by the limitation that an environment can only be assigned to one phase in a lifecycle. If you wish to deploy to UAT in these regions, you must also create a UAT environment for each location.
Using a tenant for each geo location scales better and is more flexible. Tenants can also be assigned to multiple environments in the same (and different projects). This can make managing products in different regions by just adding the tenant to the new project. If you need to add more geolocations, tenants are easier to add and manage.
Conclusion
In general, 5 environments are recommended for most pipelines. Each environment is unique from one another as it provides a role in validating deployments before your artifacts enter production. Most of these boundaries are defined by teams. If you think in terms of traditional silo’s, each team gets one environment and then production. Dev and QA are straight forward. Stage is for the operations team. UAT is for the sales team and the end customer. Here sales teams validate market changes and users provide feedback before going generally available. Outside of that, you can most likely solve your issue by incorporating tenants.
