In my daily work, I received the same request a few times: To find an automated and easy way to create a non-production environment. This could be for the developers to facilitate testing, but also as a lab environment. In most cases, the ongoing situation is that developers are sharing VMs with the needed applications, or they use their own workstation. While this works, it is far from ideal. A configuration drift will happen, people are getting in each others way and if an environment breaks, development is halted. So I found a solution, using PowerShell, ARM Templates and Azure Serverless services. In this post I want to talk about how to deploy a test environment with a calendar appointment. This blog post is part of the Applied Cloud Stories initiative.
The problem
When talking to customers, they experience two main challenges when dealing with test environments:
Static resources
Static sources are often used, either on-premises or in the cloud. These resources are used permanently and never shut down, so not very cost effective! But the bigger problem is maintenance. The resources are used and reused, deviating further and further from the production situation it was trying to simulate. Sooner or later the people using the environment will get in each others way, so maybe another one will be deployed. This will cause even more resource use and maintenance.
Manual work
Whether the environment is static or flexible, some manual work is always needed. The environment(s) need to be managed. So maybe there is a coordinator in the middle, taking care of when a developer needs an environment. This means that one person needs to keep track of all the testing/lab environments, the developers and their needs. Processes will stall if that one person isn’t available.
The alternative is to give developers permissions to control their own environments. While this could be ok, there is a big risk of either resources being forgotten and costing money, or developers using each others environments. Also, sometimes functional testers need the resources, who then have to first be educated in the use of a platform like Azure.
The solution
So I started considering the options to make a cost efficient and fully automated process. The following diagram presents the solution I came up with:
The new process goes like this:
1) A person will schedule an appointment in a calendar, that sets the date and time that the environment is needed. They will invite the people who need the resources, or create their own schedules. What is important is that a dedicated mailbox, owned by a service account, is invited
2) A bunch of magic happens. I will go into what that magic is later in this post
3) When the appointment starts, the requester will have access to the environment of their needs. This can be virtual machines or PAAS services in Azure
4) The resources are used. After the appointment finishes, the resources will be either shut down or deleted, depending on what fits best
By minimizing manual interventions and cleaning up resources as soon as possible, you get a very cost effective and low effort environment.
Technical components
So let’s talk about that magic in step 2. To clarify the process a bit more, this diagram represents the technical solution:
The script trigger: Logic Apps
The first goal is to trigger the deployment in an automated way that does not require Azure knowledge. A solution that works for almost every company is to use a calendar. To trigger action from a calendar, Logic Apps come in handy.
This requires a service account with an Office365 license, so it has access to a calendar in Exchange online (using a personal account can be a security issue). We create a connector with this account. Once every hour this logic app will check the calendar for new appointments. When a new appointment is found, it will use the data from the appointment as parameters for a runbook.
Tip: While this example was created with Office365, there are connectors for Gmail and Outlook.com which should work as well. I haven’t tried these myself though.
So let’s say someone want to use an environment tomorrow for eight hours. They create an appointment in their own calendar and invite the service account. By using the start time as well as the finish time, we can start a runbook twice. If there are multiple different environments, a certain string in the subject line can be used to pass through and define which environment to start.
And how about using the people who are invited to the schedule to set the correct permissions to the environment?
Tip: You may want to limit who in your organization can trigger this logic app. You can use a condition in the logic app to make sure not everyone can deploy the environment.
Click here for a step-by-step guide to create this Logic App
Scheduling: PowerShell runbooks
When searching for scheduling options, Azure Durable functions could work. But they are not supported for PowerShell yet (It’s coming though, so this post might need an update in the feature!). As an alternative, we will use a PowerShell runbook that can schedule other runbooks. We let the logic app trigger a runbook called “Schedule-Runbook”. That runbook sets schedules to trigger different runbooks.
Click here for the code for a runbook that can schedule other runbooks
This runbook can be used to schedule two new runbooks:
-
- One that starts the environment
This script will deploy the ARM Templates that are needed for the environment. By creating the ARM parameters within the PowerShell script and creating a loop, it is possible to create the same environment multiple times, which can come in handy when you need lab environments to teach a course.
After the environment is deployed, permissions are set so the requester can access the resources.
The ARM templates are stored in an Azure Storage Account.
The PowerShell scripts can be as complex as needed. With complexity comes a lot of flexibility, so this is where you can look at your own use case. - One that ends the environment
To save cost and prevent clutter, the end of the appointment can be used to destroy the environment that was created by removing the resources. The permissions can be removed as well. If removing the resources immediately isn’t fitting the scenario, it is also possible to schedule this script to run every week or month. Or another script might be in place that just makes sure all the VMs are shut down at the end of the appointment.
- One that starts the environment
The Test environment: DevTest Labs and ARM templates
While the structure of a test environment could change per use case, they can all be created with ARM templates. The ARM Templates are stored in a storage account so the PowerShell Runbooks can reach them.
By using DevTest Labs, developers can create their own images with the environment they need. Within the portal this can be done with a click of a button.
Because of this low threshold to create an image, it is possible for people with less coding experience to create their own environment.
By playing around with the network settings, you could create a domain controller and simulate a local environment. It is also possible to create Azure PaaS resources within the lab, whatever fits your needs.
Tip: If you create a domain controller, you can’t use Sysprep. Fortunately, you have the option to create an image without sys prep. This does mean that you need to memorize the admin account credentials or store them somewhere. How about in Keyvault?
The usage of DevTest Labs is optional, if it fits better to create resources outside of that environment, you can of course create them in ARM templates as well. I do think DevTest Labs give some extra features, like setting granular permissions and controls.
Monitoring and quality control
When something is automatically triggered, you will have to invest some time to make sure everything is working as expected. It should be clear very early if a problem has occurred. To accomplish this, you could create Azure Monitor alerts to notify you.
Click here for a guide on how to get a Teams message if a runbook fails
It is important to have correct error handling in the Runbooks in the Automation Account, otherwise they might not trigger the monitor. They should end in an error if the results are not what they should be. You can accomplish this by making the script Throw on error, for example in try-catch loops.
Conclusion
So this is how you deploy a test environment with a calendar appointment. This solution might seem complicated at first, but once you are done with the initial setup it is really not that bad. It will save the Azure administrators a lot of time and make developers happy to have quick access to their test environment.
If you would like some more in-depth information about parts of this solution, please let me know in the comments!
