Use Azure Queue storage bindings in Azure PowerShell Function Apps

I have written before about how you can start using Azure Function apps for your PowerShell scripts.
While single function apps work fine, a great advantage of the Azure Function App is that you can run more than one function and add different triggers. It is possible to create multiple runspaces in parallel, which speeds up your processes. Today I want to walk through using an Azure Queue storage message as a trigger with storage bindings in Azure PowerShell Function Apps.

But why

When you get started, your PowerShell Function app might contain only one function. With serverless though, we get some new possibilities. These might save time and provide a whole new logic to work with. I will say I found it hard to get into these options as there is not a lot of documentation if you look specifically for PowerShell. Since PowerShell in Function Apps is GA now, this will probably increase with time. It is a lot of fun to dig into though, so I hope this code inspires you to start creating your own Queue Storage bindings in Azure PowerShell Function Apps. If you have any suggestions, please let me know in the comments!

The message queue

Using the message queue is one of the many supported trigger bindings for a function app. By default it uses the storage account that the function app is using. You can add a message to the queue yourself, or allow another function/app to add the messages.
The process is as follows:
– A message enters the queue
– The function is triggered to start
– If the function fails, the message stays in the queue and function repeats itself to try to perform a correct action
– When the completes successful, the messages is cleared from the queue
– If the function fails five time, the message is places in a separate queue called the poison queue

So basically, it’s like this:

Multiple instances

The Azure Queue is able to handle more messages at once. It will get the messages in batches and start working on them. The advantage of this is that you can run multiple jobs at once, saving time in the long run.
The down side is that you can’t prioritize in one queue. If you want to prioritize, you would need to run multiple functions that respond to multiple queues.

The use case

I have had a scheduled function app running that deletes resources with a certain tag, so I could automatically delete my testresources at the end of the day. I had this in a single function for quite some time, but one of the issues I faced was time. A function app has a time out of 5 minutes by default and 10 minutes maximum for the consumption plan. I have used an App Service Plan for some time, which worked. But it felt a bit like a workaround. Something else that bugged me is that I was deleting multiple resources and a bunch of resource groups in one Function, where the best practices for Function apps call for one action per function. So I went looking for the options available.

Want to stop reading and just see the result to use for yourself? You can find it here: https://github.com/Ba4bes/AzFa-Remove-Resources

Note: this function app removes resources that can not be restored! It should be tested for your environment. To keep you safe I have commented out the remove-actions. To make use of the scripts, remove the comments from the following lines:

Remove-FirstResources – run.ps1: Line 48 and line 70
Remove-SecondResources – run.ps1: Line 43 and Line 63

The first step: Start-Function

The first thing I created was a Scheduled Function. This function searches for the resources that need to be deleted. It does so by searching for the tag “AutoDelete = True” .
The Function collects those resources and sends the resourceID to the Message queue as a string. In the run.ps1 file, this looks like this:

Push-OutputBinding -Name JobQueue -Value $ResourceId

To get this to work we need to consider the function.json file. We need to add an output binding to connect the queue. This can be done with the following code

{
  "type": "queue",
  "direction": "out",
  "name": "JobQueue",
  "queueName": "first-resourcequeue";
}

The name attribute is the same name as in the cmdlet above. The queueName is the name of the queue that is created in the storage account.
So now if we use the same queue as a triggerbinding for the next function, we’ve got some interaction here.

Remove-FirstResources

So in the function.json-file of this function, I created the following input binding

{
  "name": "ResourceId",
  "type": "queueTrigger",
  "direction": "in",
  "queueName": "first-resourcequeue";
}

In this, the Name is the variable that is used in the function. The queuename needs to be the same as the one in the first function.
This function has a goal of deleting resources. There is one problem though. There are a lot of dependencies when deleting Azure Resources. For example, a virtual network can not be removed if there is still a subnet attached.
I have not found an option yet to find out what resources are depending on each other in the script. So I went for a bit of a workaround. I select some resources that are known to have dependencies, like the virtual network we talked about. These resources are send to a second queue to be deleted. This new function has a small wait before running, so it has a better chance of the dependent resources being deleted before it runs

Error handling

I have made use of the fact that a function that is triggered by the message queue will run 5 times if it fails. Because of this, the resource will automatically retry if it fails. Those five retries are usually enough to remove all resources and the resource groups they were in if it is now empty.

Poison queue

If the 5th attempt fails, the message in the message queue is placed in the poison queue. I have made two extra function that are triggered by the poison queues. They are very simple, they create a readable output. This is meant to make monitoring of the function app easier and to clear the poison queue. A plus is that I found sometimes the Function has given up on clearing the message, but the resource has actually been removed. So I let this function dubblecheck if the resource actually hasn’t been removed.
After some testing, I found that the Poison queue functions in my environment are barely touched. I have not had any hits on the first Poison queue function yet. The second one is mostly hit by resources where the depending resource does not have the autodelete-tag.

Tip: want to see what’s going on with the storage queues? You can open them in Azure Storage Explorer to see what happens and delete stale messages.

Got confused by all that’s going on? Here you can see what’s happening

Result

This Function app will now remove the resources with the tag defined.

If you want to try it out, you can find it on Github: https://github.com/Ba4bes/AzFa-Remove-Resources. This is actually including an Azure DevOps pipeline for easy deployment. To find out how to use that, you can read my post here.