Get a Teams message when a runbook fails

When my main focus was on-Prem, I had a lot of Powershell-scripts running on scheduled tasks.
That is a few years ago and there is something that has always bugged me but I never came to fix: I didn’t have a monitor to check if the scripts were still running or if they had failed.

Fast forward, We are using runbooks now. And I have today taken the time to monitor them the way I wanted: Create a Teams message when something is off.

I’m going to show you how to set this up for yourself

The runbook

I created a runbook called “Test” which has one line.

Throw "I have broken"

Now I have a runbook that always fails, very usefull for testing.

Log analytics

The next step is to enable diagnostic settings on the Automation Account.
It’s at the bottom of the menu in the portal.
Select “Turn on diagnostics”

Provide a name and check “Send to Log Analytics”.

If you haven’t got an OMS workspace yet, you need to set it up. This is pretty straightforward. Give it a name and a resourcegroup.

You can select what information you need. I just need to see JobLogs for this, but you are free to select them all if it’s a fit for you. If you choose Jobstreams, you will get more information about the job itself, like if errors occurred

Microsoft Teams

Before moving to the alert, let’s create the webhook for Teams.

Open the Teams channel where you want the alerts to go to
Select the three dots and then connectors.

click Add for “Incoming Webhook” and then install.
You can now provide name and a custom image. They can both be changed at a later time
Select Create and it shows you an URL. Copy this and keep it somewhere ’till you set up the alert.

Azure Monitor

If you have created the log analytics-connection just now, this step will often not work yet. It apparently needs to collect data first. So wait about an hour before continuing if it was just created.

Moving on to Azure monitor. I recommend playing around with it on the logs-section with possible queries if you get a chance. For now, we’ll move to alerts.

 

In Monitor, open Alerts. Select New Alert Rule. Don’t forget to check if you have the right subscription.

The first thing you set is the target. This is the OMS workspace you have set in the earlier step. Select the subscription and “log analytics” in the resource type-dropbox. Now you can select the right workspace

Click Add criteria and select Custom Log search
Here you can fill in your query.
This will be something like below:

AzureDiagnostics | where OperationName == "Job"  | where ResultType == "Failed" or ResultType == "stopped" | where TimeGenerated > ago(60m)

 

Set the Logic to “Number of results Greater than 0” and set the timing to a period that fits the query. It should look something like this:

On to the next step: Alert Details

Click the Title and give your rule a name.
You can set the severity, which doesn’t really matter for the teams alert. Also, you can setup that the rule will suppress alerts after it has send one. This might be usefull to prevent the webhook from spamming the channel.

Next, you set up the action group. This is where you can set up the Teams-part
We are going to create a new Action group

The shortname isn’t really important for a webhook, but better make it descriptive. Now when you select a webhook in the action type, the screen will ask you for an URL. This is where you add the URL you got from Teams.

Save the settings.

Custom JSON

Now, you need to select Customize actions > Include custom Json payload for webhook.

Here, you enter JSON-code to set up the message Teams will display. You can just enter the following lines:

{"text":"#alertrulename fired with #searchresultcount records which exceeds the over threshold of #thresholdvalue"}

And you will get this:

It’s an alert, but we can do beter.

As you can see here, Teams allows you to set up some nice formatting in the message, As you can see here 

And with the Azure Alert dynamics values, the message get’s even better.

So let’s say you use this code:

{
    "@type": "MessageCard",
    "@context": "https://schema.org/extensions",
    "summary": "AzureAlert",
    "themeColor": "0078D7",
    "title": "A Runbook has failed or stopped",
    "sections": [
        {
            "facts": [
                {
                    "name": "Alertrule:",
                    "value": "#alertrulename"
                },
                {
                    "name": "Workspace Id",
                    "value": "#workspaceid"
                },
                {
                    "name": "Link To Runbook:",
                    "value": "[link](https://link.com)"
                }
            ],
            "text": "There is a problem with a runbook in the automation account.",
            "markdown": true
        }
    ]
}

And click create alert rule.

Now when the runbook fails, you get a message:

So that’s it, now Teams will alert you when your runbook is failing