Philip Van de Vyver - Azure Blog

Azure Update Manager pre and post events for turned off VMs

Wed, 10 Jan 2024 00:00:00 +0000

Introduction
Approach
Configuration
Result

Introduction

Azure Update Manager is a quite new Azure service that went Generally Available 18th of September 2023.

Azure Update Manager has been redesigned and doesn’t depend on Azure Automation or Azure Monitor Logs, as required by the Azure Automation Update Management feature.

Since the Azure Log Analytics agent, also known as the Microsoft Monitoring Agent (MMA) will be retired in August 2024, customers are recommended to move to Azure Update Manager.

In Azure Automation Update Management there was a possibility to use pre-scripts and post-scripts to handle VMs that are turned off for cost saving purposes.

In the initial release of Azure Automation Update Management this possibility wasn’t available. This recently got changed in December 2023, where Pre and Post Events got introduced for Azure Update Manager in preview.

In this post, I will explain how to implement pre and post events for VMs that are turned off for patching in Azure Update Manager.

Approach

The following approach is implemented to start/stop VMs in a Maintenance Configuration of Update Manager

1) The Maintenance Configuration about to be triggered (T -20 minutes)

2) The Maintenance Configuration triggers an event to the Event Grid System Topic with the Event Type: Microsoft.Maintenance.PreMaintenanceEvent

3) The Event Subscription for start vm gets triggered, the Event Subscription is configured to pickup events with the Event Type: Microsoft.Maintenance.PreMaintenanceEvent, the event subscription is configured to trigger a webhook with the payload from the event.

4) The Automation Webhook for starting the start vm runbook gets triggered with the payload from the event.

5) The Automation Runbook for start vm gets triggered.

6) The Automation Runbook will get all the VMs and Arc Virtual Machines in scope and starts the deallocated or stopped ones. The runbook will tag the VMs that are started with the UpdateManagerState tag and RunId value for the maintenance configuration run.

7) The actual patching will start (T -0 minutes)

8) After the configured time window for the patching, The Maintenance Configuration triggers an event to the Event Grid System Topic with the Event Type: Microsoft.Maintenance.PostMaintenanceEvent

9) The Event Subscription for stop vm gets triggered, the Event Subscription is configured to pickup events with the Event Type: Microsoft.Maintenance.PostMaintenanceEvent, the event subscription is configured to trigger a webhook with the payload from the event.

10) The Automation Webhook for starting the stop vm runbook gets triggered with the payload from the event.

11) The Automation Runbook for stop vm gets triggered.

12) The Automation Runbook will get all the VMs and Arc Virtual Machines in scope and stops the started machines with the UpdateManagerState tag and RunId value for the maintenance configuration run. The tag will be removed after stopping the VMs and Arc Virtual Machines.

Configuration

User Assigned Managed Identity

Create a User Assigned Managed Identity id-updatemanager and assign the required roles. Store the Client Id for later

Assign the following roles to the User Assigned Managed Identity at subscription scope:

Reader (To read out Azure Resource Graph for the Maintenance Runs)
Tag Contributor (To tag the VMs and identity when started)
Virtual Machine Contributor (To start/stop the VMs)

Automation Account

We will use a Automation Runbook to start all the VMs. The Runbook will get triggered from a webhook and will user the User Assigned Managed Identity to authenticate to Azure.

Automation Account Modules

The Automation Runbook requires the following PowerShell Modules:

You can import the modules using the import module guide

[Note] Make sure you select the Runtime version 7.2 for the Modules

Automation Account Variable

Create a variable with the name var-clientid-id-updatemanager that contains the ClientId from the User Assigned Managed Identity that you created earlier.

Automation Account Runbooks

There will be 2 runbooks deployed:

[Note] Make sure you select the Runtime version 7.2 for the PowerShell runbook

Here is the content of the scripts:

UpdateManager-StartVMs

param
(
    [Parameter(Mandatory=$false)]
    [object] $WebhookData
)

$miUpdateManager = Get-AutomationVariable -Name "var-clientid-id-updatemanager"

Connect-AzAccount -Identity -AccountId $miUpdateManager | Out-Null

$notificationPayload = $WebhookData.RequestBody | ConvertFrom-Json -depth 100
$maintenanceRunId = $notificationPayload[0].data.CorrelationId
$resourceSubscriptionIds = $notificationPayload[0].data.ResourceSubscriptionIds

if ($resourceSubscriptionIds.Count -eq 0) {
    Write-Output "Resource subscriptions are not present."
    break
}

Write-Output "Querying ARG to get machine details [MaintenanceRunId=$maintenanceRunId][ResourceSubscriptionIdsCount=$($resourceSubscriptionIds.Count)]"

$argQuery = @"
    maintenanceresources 
    | where type =~ 'microsoft.maintenance/applyupdates'
    | where properties.correlationId =~ '$($maintenanceRunId)'
    | where id has '/providers/microsoft.compute/virtualmachines/'
    | order by id asc
"@

Write-Output "Arg Query Used: $argQuery"

$allMachines = [System.Collections.ArrayList]@()
$skipToken = $null

do
{
    $res = Search-AzGraph -Query $argQuery -First 1000 -SkipToken $skipToken -Subscription $resourceSubscriptionIds
    $skipToken = $res.SkipToken
    $allMachines.AddRange($res.Data)
} while ($skipToken -ne $null -and $skipToken.Length -ne 0)

if ($allMachines.Count -eq 0) {
    Write-Output "No Machines were found."
    break
}

$jobIDs= New-Object System.Collections.Generic.List[System.Object]
$startableStates = "stopped" , "stopping", "deallocated", "deallocating"

$allMachines | ForEach-Object {
    $vmId = $_.properties.resourceId
    $split = $vmId -split "/";
    $subscriptionId = $split[2]; 
    $rg = $split[4];
    $name = $split[8];

    Write-Output ("Subscription Id: " + $subscriptionId)

    $mute = Set-AzContext -Subscription $subscriptionId
    $vm = Get-AzVM -ResourceGroupName $rg -Name $name -Status -DefaultProfile $mute

    $state = ($vm.PowerState -split " ")[1]
    if($state -in $startableStates) {
        Write-Output "Starting '$($name)' ..."

        $newJob = Start-ThreadJob -ScriptBlock { param($rg, $name, $sub) $context = Set-AzContext -Subscription $sub; Start-AzVM -ResourceGroupName $rg -Name $name -DefaultProfile $context} -ArgumentList $rg, $name, $subscriptionId
        $jobIDs.Add($newJob.Id)
        $tags = @{"UpdateManagerState"="$maintenanceRunId"}
        Update-AzTag -ResourceId $_.properties.resourceId -Tag $tags -operation Merge | Out-Null
    } else {
        Write-Output ($name + ": no action taken. State: " + $state) 
    }
}

$jobsList = $jobIDs.ToArray()
if ($jobsList)
{
    Write-Output "Waiting for machines to finish starting..."
    Wait-Job -Id $jobsList
}

foreach($id in $jobsList)
{
    $job = Get-Job -Id $id
    if ($job.Error)
    {
        Write-Output $job.Error
    }
}

UpdateManager-StopVMs

param
(
    [Parameter(Mandatory=$false)]
    [object] $WebhookData
)

$miUpdateManager = Get-AutomationVariable -Name "var-clientid-id-updatemanager"

Connect-AzAccount -Identity -AccountId $miUpdateManager | Out-Null

# Install the Resource Graph module from PowerShell Gallery
# Install-Module -Name Az.ResourceGraph

$notificationPayload = ConvertFrom-Json -InputObject $WebhookData.RequestBody
$maintenanceRunId = $notificationPayload[0].data.CorrelationId
$resourceSubscriptionIds = $notificationPayload[0].data.ResourceSubscriptionIds


if ($resourceSubscriptionIds.Count -eq 0) {
    Write-Output "Resource subscriptions are not present."
    break
}

Start-Sleep -Seconds 30
Write-Output "Querying ARG to get machine details [MaintenanceRunId=$maintenanceRunId][ResourceSubscriptionIdsCount=$($resourceSubscriptionIds.Count)]"

$argQuery = @"
    resources
    | where type == 'microsoft.compute/virtualmachines'
    | where tags.UpdateManagerState =~ '$($maintenanceRunId)'
"@

Write-Output "Arg Query Used: $argQuery"

$allMachines = [System.Collections.ArrayList]@()
$skipToken = $null

do
{
    $res = Search-AzGraph -Query $argQuery -First 1000 -SkipToken $skipToken -Subscription $resourceSubscriptionIds
    $skipToken = $res.SkipToken
    $allMachines.AddRange($res.Data)
} while ($skipToken -ne $null -and $skipToken.Length -ne 0)

if ($allMachines.Count -eq 0) {
    Write-Output "No Machines were found."
    break
}

$jobIDs= New-Object System.Collections.Generic.List[System.Object]
$stoppableStates = "starting", "running"

$allMachines | ForEach-Object {
    $vmId =  $_.id

    $split = $vmId -split "/";
    $subscriptionId = $split[2]; 
    $rg = $split[4];
    $name = $split[8];

    Write-Output ("Subscription Id: " + $subscriptionId)

    $mute = Set-AzContext -Subscription $subscriptionId
    $vm = Get-AzVM -ResourceGroupName $rg -Name $name -Status -DefaultProfile $mute

    $state = ($vm.PowerState -split " ")[1]
    if($state -in $stoppableStates) {
        Write-Output "Stopping '$($name)' ..."

        $newJob = Start-ThreadJob -ScriptBlock { param($resource, $vmname, $sub) $context = Set-AzContext -Subscription $sub; Stop-AzVM -ResourceGroupName $resource -Name $vmname -Force -DefaultProfile $context} -ArgumentList $rg, $name, $subscriptionId
        $jobIDs.Add($newJob.Id)
        $tags = @{"UpdateManagerState"="$maintenanceRunId"}
        Update-AzTag -ResourceId $_.properties.resourceId -Tag $tags -operation Merge | Out-Null
    } else {
        Write-Output ($name + ": no action taken. State: " + $state) 
    }
}

$jobsList = $jobIDs.ToArray()
if ($jobsList)
{
    Write-Output "Waiting for machines to finish stop operation..."
    Wait-Job -Id $jobsList
}

foreach($id in $jobsList)
{
    $job = Get-Job -Id $id
    if ($job.Error)
    {
        Write-Output $job.Error
    }
}

Automation Account Webhooks

Create a webhook for both runbooks.

Keep the URLs from the webhooks for later (it will be the only time to display)

Result:

start vm webhook

stop vm webhook

Register your subscription for public preview

Follow this guide: Register your subscription for public preview

This will enable the preview feature Pre and Post Events on your subscription.

Maintenance Configuration

Setup a Maintenance Configuration for Update Manager for In guest patching.

Event Grid Topic

Per schedule an Event Grid topic must be setup. The Event Grid Topic can be shared for multiple subscriptions.

Event Grid Subscriptions

Here’s how to create an Event Grid subscription on the Maintenance Configuration Event with a filter on Microsoft.Maintenance.PreMaintenanceEvent to start the VM.

On the selected Maintenance configuration page, under Events, select Web Hook.

In the Create Event Subscription wizard, fill in the following:

EVENT SUBSCRIPTION DETAILS
- name: [name of the Event Subscription] (Example: start-vm)
TOPIC DETAILS
- System Topic Name: [the name of the System Topic] (Example: egst-start-stop-demo)
EVENT TYPES
- Filter to Event Types: Pre Maintenance Event (This filters out the Pre Maintenance Event)
ENDPOINT DETAILS
- Endpoint for the webhook: [your endpoint] (This is the URLs from the webhook you created earlier)

Repeat the Create Event Subscription wizard for the stop-vm Event Subscription and select the Post Maintenance Event event type.

Result

As result of the configuration, the Maintenance Configuration will:

Start the Virtual Machines that are turned off using an Automation Runbook

Stop the Virtual Machines that are turned off using an Automation Runbook

Github Actions monitoring with Application Insights

Wed, 03 Jan 2024 00:00:00 +0000

Introduction
Ingest Application Insights Request Telemetry data
- Github Action
  - PowerShell script
- Result
Ingest Application Insights Release Annotation
- Github Action
  - PowerShell script
- Result

Introduction

I love working with Github Actions for Continuous Integration and Deployment.

By default you get emails regarding failed workflows for Github monitoring.

To get more insights in the performance and success of workflows, you can leverage Application Insights to log performance, failures and release annotations.

To achieve this, I use:

The Azure SDK for .NET to ingest Request Telemetry data
The Application insights Rest API together with AZ CLI to ingest release annotations.

Ingest Application Insights Request Telemetry data

In this example I’m logging a successful request (ResponseCode = 200) to Application Insights.

The PowerShell script logs the duration of the request, success and some custom properties (ActionRunURL, MyTestProperty1, MyTestProperty2)

Github Action

The Github Action consist of the following:

Workflow trigger:
- workflow dispatch trigger to trigger the workflow manually.
Environment variable:
- APPLICATIONINSIGHTS_INSTRUMENTATIONKEY: Containing the Application Insights Instrumentation keye
Jobs:
- application-insights-log-request
  - Steps:
    - set-startdatetime-var: gets the current time and sets it to an environment variable
    - login-azure: As explained in Use GitHub Actions to connect to Azure
    - application-insights-log-request: runs a PowerShell script to log the request to Application Insights

Here’s the full content of the Github Action:

name: Log - Application Insights - Request

on:
  workflow_dispatch

env:
  APPLICATIONINSIGHTS_INSTRUMENTATIONKEY: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"

jobs:
  application-insights-log-request:
    runs-on: ubuntu-latest
    steps:
      - name: set-startdatetime-var
        run: |
          START_DATETIME=$(date '+%d/%m/%Y %H:%M:%S')
          echo $START_DATETIME
          echo "START_DATETIME=$START_DATETIME" >> $GITHUB_ENV
      - name: login-azure
        uses: azure/login@v1
        with:
          creds: '{"clientId":"$","clientSecret":"$","subscriptionId": "$","tenantId":"$"}'
          enable-AzPSSession: true
      - name: application-insights-log-request
        shell: pwsh
        run: |

          # set variables
          $InstrumentationKey = $env:APPLICATIONINSIGHTS_INSTRUMENTATIONKEY
          $Name = $env:GITHUB_WORKFLOW
          $Started = $env:START_DATETIME
          $Success = $true # set to false for failed
          $ResponseCode = 200 # set to 500 for failed
          $startedDateTime = [datetime]::ParseExact($Started, "dd/MM/yyyy HH:mm:ss", $Null)

          [System.TimeSpan]$Duration = New-TimeSpan -Start $startedDateTime -End (Get-Date)
          $StartedDateTimeOffset = [System.DateTimeOffset]$startedDateTime

          # setup telemetry client
          $telemetryClient = New-Object -TypeName Microsoft.ApplicationInsights.TelemetryClient
          $telemetryClient.InstrumentationKey = $InstrumentationKey

          # initialize context information
          $TelemetryClient.Context.User.Id = "$($env:GITHUB_ACTOR)"
          $telemetryClient.Context.Operation.Name = $Name

          # set telemetry context
          $TelemetryClient.Context.Session.Id = "$($env:GITHUB_REPOSITORY)/$($env:GITHUB_WORKFLOW)/$($env:GITHUB_RUN_ID)/$($env:GITHUB_JOB)"
          $telemetryClient.Context.Operation.Id = "$($env:GITHUB_REPOSITORY)/$($env:GITHUB_WORKFLOW)/$($env:GITHUB_RUN_ID)/$($env:GITHUB_JOB)"

          Write-Host "SessionId: $($TelemetryClient.Context.Session.Id)"
          Write-Host "User Id: $($TelemetryClient.Context.User.Id)"
          Write-Host "Operation Id: $($TelemetryClient.Context.Operation.Id)"
          Write-Host "Operation Name: $($TelemetryClient.Context.Operation.Name)"

          # setup request telemetry information
          $request = New-Object -TypeName Microsoft.ApplicationInsights.DataContracts.RequestTelemetry
          $request.Name = $Name
          $request.StartTime = $StartedDateTimeOffset 
          $request.Duration = $Duration
          $request.Success = $Success
          $request.ResponseCode = $ResponseCode  

          Write-Host "StartTime: $StartedDateTimeOffset"
          Write-Host "Duration: $Duration"
          Write-Host "Success: $Success"
          Write-Host "ResponseCode: $ResponseCode"

          # set actionurl
          $ActionRunURL = "$($env:GITHUB_SERVER_URL)/$($env:GITHUB_REPOSITORY)/actions/runs/$($env:GITHUB_RUN_ID)"
          Write-Host "ActionRunURL : $ActionRunURL"

          # set properties 
          [HashTable]$Properties = @{}
          $properties["ActionRunURL"] = "$ActionRunURL"
          $properties["MyTestProperty1"] = "Test 1"
          $properties["MyTestProperty2"] = "Test 2"

          # copy properties to request
          $properties.Keys | ForEach-Object { $request.Properties[$_] = $Properties[$_] }

          # track the request
          $telemetryClient.TrackRequest($request)

          # flush
          $telemetryClient.Flush()

PowerShell script

Here’s the content of the inline PowerShell script:

# set variables
$InstrumentationKey = $env:APPLICATIONINSIGHTS_INSTRUMENTATIONKEY
$Name = $env:GITHUB_WORKFLOW
$Started = $env:START_DATETIME
$Success = $true # set to false for failed
$ResponseCode = 200 # set to 500 for failed
$startedDateTime = [datetime]::ParseExact($Started, "dd/MM/yyyy HH:mm:ss", $Null)

[System.TimeSpan]$Duration = New-TimeSpan -Start $startedDateTime -End (Get-Date)
$StartedDateTimeOffset = [System.DateTimeOffset]$startedDateTime

# setup telemetry client
$telemetryClient = New-Object -TypeName Microsoft.ApplicationInsights.TelemetryClient
$telemetryClient.InstrumentationKey = $InstrumentationKey

# initialize context information
$TelemetryClient.Context.User.Id = "$($env:GITHUB_ACTOR)"
$telemetryClient.Context.Operation.Name = $Name

# set telemetry context
$TelemetryClient.Context.Session.Id = "$($env:GITHUB_REPOSITORY)/$($env:GITHUB_WORKFLOW)/$($env:GITHUB_RUN_ID)/$($env:GITHUB_JOB)"
$telemetryClient.Context.Operation.Id = "$($env:GITHUB_REPOSITORY)/$($env:GITHUB_WORKFLOW)/$($env:GITHUB_RUN_ID)/$($env:GITHUB_JOB)"

Write-Host "SessionId: $($TelemetryClient.Context.Session.Id)"
Write-Host "User Id: $($TelemetryClient.Context.User.Id)"
Write-Host "Operation Id: $($TelemetryClient.Context.Operation.Id)"
Write-Host "Operation Name: $($TelemetryClient.Context.Operation.Name)"

# setup request telemetry information
$request = New-Object -TypeName Microsoft.ApplicationInsights.DataContracts.RequestTelemetry
$request.Name = $Name
$request.StartTime = $StartedDateTimeOffset 
$request.Duration = $Duration
$request.Success = $Success
$request.ResponseCode = $ResponseCode  

Write-Host "StartTime: $StartedDateTimeOffset"
Write-Host "Duration: $Duration"
Write-Host "Success: $Success"
Write-Host "ResponseCode: $ResponseCode"

# set actionurl
$ActionRunURL = "$($env:GITHUB_SERVER_URL)/$($env:GITHUB_REPOSITORY)/actions/runs/$($env:GITHUB_RUN_ID)"
Write-Host "ActionRunURL : $ActionRunURL"

# set properties 
[HashTable]$Properties = @{}
$properties["ActionRunURL"] = "$ActionRunURL"
$properties["MyTestProperty1"] = "Test 1"
$properties["MyTestProperty2"] = "Test 2"

# copy properties to request
$properties.Keys | ForEach-Object { $request.Properties[$_] = $Properties[$_] }

# track the request
$telemetryClient.TrackRequest($request)

# flush
$telemetryClient.Flush()

Result

When the workflow ran, the following result is shown:

In the Application Insights instance, the following appears:

When clicking on the request, the following details are shown:

Ingest Application Insights Release Annotation

In this example I’m logging a release annotation for a new release with some custom properties from the default Github environment variables.

The trigger of the request is a

Github Action

The Github Action consist of the following:

Workflow trigger:
- pull request on the main branch of the repository.
Environment variable:
- APPLICATION_INSIGHTS_RESOURCE_ID: Containing the Application Insights Resource Id
Jobs:
- create-applicationinsightsannotation
  - Steps:
    - set-startdatetime-var: gets the current time and sets it to an environment variable
    - login-azure: As explained in Use GitHub Actions to connect to Azure
    - create-applicationinsightsannotation: runs a PowerShell script to ingest an annotation in application insights

Here’s the full content of the Github Action:

name: Log - Application Insights - Annotation

on:
  pull_request:
    branches:
    - main

env:
  APPLICATION_INSIGHTS_RESOURCE_ID: "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/rg-weu-prd-githubactionsapplicationinsights/providers/microsoft.insights/components/appi-weu-prd-githubactionsapplicationinsights"

jobs:
  create-applicationinsightsannotation:
    runs-on: ubuntu-latest
    steps:
      - name: login-azure
        uses: azure/login@v1
        with:
          creds: '{"clientId":"$","clientSecret":"$","subscriptionId": "$","tenantId":"$"}'
          enable-AzPSSession: true
      - name: set-startdatetime-var
        run: |
          START_DATETIME=$(date '+%d/%m/%Y %H:%M:%S')
          echo $START_DATETIME
          echo "START_DATETIME=$START_DATETIME" >> $GITHUB_ENV
      - name: create-applicationinsightsannotation
        id: create-applicationinsightsannotation
        uses: azure/powershell@v1
        with:
          inlineScript: |

            # set annotation properties
            $annotationProperties = @{"ReleaseDescription" = "$env:GITHUB_REF"; "TriggerBy" = "$env:GITHUB_ACTOR" }
            $annotationPropertiesJson = ConvertTo-Json $annotationProperties -Compress

            # build annotation
            $annotation = @{
                Id             = [GUID]::NewGuid();
                AnnotationName = "$($env:GITHUB_REF_NAME)";
                EventTime      = (Get-Date).ToUniversalTime().GetDateTimeFormats("s")[0];
                Category       = "Deployment"; 
                Properties     = $annotationPropertiesJson
            }

            # set body 
            $body = (ConvertTo-Json $annotation -Compress) -replace '(\\+)"', '$1$1"' -replace "`"", "`"`""

            # submit annotation
            az rest --method put --uri "$($env:APPLICATION_INSIGHTS_RESOURCE_ID)/Annotations?api-version=2015-05-01" --headers "Content-Type=application/json" --body "$($body) "

          azPSVersion: "11.1.0"
          errorActionPreference: stop

PowerShell script

# set annotation properties
$annotationProperties = @{"ReleaseDescription" = "$env:GITHUB_REF"; "TriggerBy" = "$env:GITHUB_ACTOR" }
$annotationPropertiesJson = ConvertTo-Json $annotationProperties -Compress

# build annotation
$annotation = @{
    Id             = [GUID]::NewGuid();
    AnnotationName = "$($env:GITHUB_REF_NAME)";
    EventTime      = (Get-Date).ToUniversalTime().GetDateTimeFormats("s")[0];
    Category       = "Deployment"; 
    Properties     = $annotationPropertiesJson
}

# set body 
$body = (ConvertTo-Json $annotation -Compress) -replace '(\\+)"', '$1$1"' -replace "`"", "`"`""

# submit annotation
az rest --method put --uri "$($env:APPLICATION_INSIGHTS_RESOURCE_ID)/Annotations?api-version=2015-05-01" --headers "Content-Type=application/json" --body "$($body) "

Result

When the workflow ran, the following result is shown:

As result, the release annotation is shown in the graph with some custom properties:

Ingest custom data into a Log Analytics Workspace

Tue, 12 Dec 2023 00:00:00 +0000

Introduction

In my organization, we often collect custom data in Azure Monitor Logs. On those logs, we create insights in Azure Monitor Workbooks or create Azure Monitor Alerts upon the logs.

This post describes how to ingest log data in a Log Analytics workspace using the Azure Monitor Data Collector API:

Flow

I will ingest data from weather data that is collected from the Weather API API.

Post the content of a json file from all Azure Resources to the Data Collection Endpoint
Use the Data Collection Rule to transform the data and stream to a Log Analytics workspace Weather_CL custom table.

The Setup

Log Analytics workspace Weather_CL table

In order to ingest data we have to setup a Log Analytics workspace custom table with a predefined schema.

Here is the schema I used to create the table:

table.json

{
    "properties": {
        "schema": {
            "name": "Weather_CL",
            "columns": [
                {
                    "name": "TimeGenerated",
                    "type": "datetime",
                    "description": "The time at which the data was generated"
                },
                {
                    "name": "location",
                    "type": "dynamic"
                },
                {
                    "name": "current",
                    "type": "dynamic"
                }
            ]
        }
    }
}

There are a few ways to create a Log Analytics Custom Table using the Portal, API, CLI or PowerShell.

For this example I used the PowerShell way with the Invoke-RestMethod

# variables
$resourceGroup = "<your resource group>"
$logAnalyticsworkspaceId = "<your Log Analytics workspace Resource Id>"
$tableName = "Weather_CL"

function get-azCachedAccessToken() {
    $currentAzureContext = Get-AzContext
    $azProfile = [Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureRmProfileProvider]::Instance.Profile
    $profileClient = New-Object Microsoft.Azure.Commands.ResourceManager.Common.RMProfileClient($azProfile)
    $token = $profileClient.AcquireAccessToken($currentAzureContext.Tenant.TenantId)
    $token.AccessToken
}

# create authentication header
$token = get-azCachedAccessToken 
$authHeader = @{
    'Authorization' = "Bearer $token"
    'Content-Type'  = "application/json"
}

# create table
Write-Output "Creating new table '$tableName'.."
$uri = "https://management.azure.com$($logAnalyticsworkspaceId)/tables/$($tableName)?api-version=2021-12-01-preview"
$jsonBody = [String](Get-Content -Path .\table.json)
Invoke-RestMethod -Method Put -Uri $uri -Headers $authHeader -Body $jsonBody 

The result is a custom Log Analyitcs workspace Table:

Data collection endpoint

To have an endpoint to send our data to, we have to setup a Data collection endpoint.

To setup the Data collection endpoint, I use this ARM template:

dataCollectionEndpoint.json

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "dataCollectionEndpointName": {
            "type": "string",
            "metadata": {
                "description": "Specifies the name of the Data Collection Endpoint to create."
            }
        },
        "location": {
            "type": "string",
            "defaultValue": "westeurope",
            "allowedValues": [
                "westus2",
                "eastus2",
                "eastus2euap",
                "westeurope"
            ],
            "metadata": {
                "description": "Specifies the location in which to create the Data Collection Endpoint."
            }
        }
    },
    "resources": [
        {
            "type": "Microsoft.Insights/dataCollectionEndpoints",
            "name": "[parameters('dataCollectionEndpointName')]",
            "location": "[parameters('location')]",
            "apiVersion": "2021-04-01",
            "properties": {
                "networkAcls": {
                "publicNetworkAccess": "Enabled"
                }
            }
        }
    ],
    "outputs": {
        "dataCollectionEndpointId": {
            "type": "string",
            "value": "[resourceId('Microsoft.Insights/dataCollectionEndpoints', parameters('dataCollectionEndpointName'))]"
        }
    }
}

and use the following PowerShell script to deploy:

# variables
$location = "<your deployment location>"
$resourceGroup = "<your resource group>"
$dataCollectionEndpointName = "<your data collection endpoint name>"

# deploy dataCollectionEndpoint 
Write-Output "Deploying Data Collection Endpoint '$dataCollectionEndpointName'.."
New-AzResourceGroupDeployment -ResourceGroupName $resourceGroup -Name "D_dataCollectionEndpoint" -TemplateFile infra-as-code\arm\dataCollectionEndpoints\dataCollectionEndpoint.json `
    -dataCollectionEndpointName $dataCollectionEndpointName `
    -location $location | Out-Null

Data collection rule

This is where the magic happens, with the Data collection rule, we can setup:

dataCollectionEndpointId: the resourceId of the Data collection endpoint created earlier
streamDeclarations: This unique handle describes a set of data sources that will be transformed and schematized as one type. Each data source requires one or more streams, and one stream can be used by multiple data sources. All data sources in a stream share a common schema. Use multiple streams, for example, when you want to send a particular data source to multiple tables in the same Log Analytics workspace. Define a schema for the source of data. This is the schema of the Resources query that we fetched from Azure Resource Graph
destinations: This section contains a declaration of all the destinations where the data will be sent. Only Log Analytics workspace is currently supported as a destination. Each Log Analytics workspace destination requires the full workspace resourceId and a friendly name that will be used elsewhere in the DCR to refer to this workspace.
dataFlows: This section ties the other sections together. It defines the following properties for each stream declared in the streamDeclarations section:
- streams: This unique handle describes a set of data sources that will be transformed and schematized as one type. Each data source requires one or more streams, and one stream can be used by multiple data sources. All data sources in a stream share a common schema. Use multiple streams, for example, when you want to send a particular data source to multiple tables in the same Log Analytics workspace.
- destinations: is the transformation applied to the data that was sent in the input shape described in the streamDeclarations section to the shape of the target table.
- transformKql: Is the transformation applied to the data that was sent in the input shape described in the streamDeclarations section to the shape of the target table. In this usecase we will extend the table with the allowed fields in the custom Log Analytics workspace table.
- outputStream: Describes which table in the workspace specified under the destination property the data will be ingested into. The value of outputStream has the Microsoft-[tableName] shape when data is being ingested into a standard Log Analytics table, or Custom-[tableName] when ingesting data into a custom-created table. Only one destination is allowed per stream.

To deploy the Data Collection Rule, I use this ARM template:

dataCollectionRules.json

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "dataCollectionRuleName": {
            "type": "string",
            "metadata": {
                "description": "Specifies the name of the Data Collection Rule to create."
            }
        },
        "location": {
            "type": "string",
            "defaultValue": "westeurope",
            "allowedValues": [
                "westus2",
                "eastus2",
                "eastus2euap",
                "westeurope"
            ],
            "metadata": {
                "description": "Specifies the location in which to create the Data Collection Rule."
            }
        },
        "workspaceResourceId": {
            "type": "string",
            "metadata": {
                "description": "Specifies the Azure resource ID of the Log Analytics workspace to use."
            }
        },
        "endpointResourceId": {
            "type": "string",
            "metadata": {
                "description": "Specifies the Azure resource ID of the Data Collection Endpoint to use."
            }
        }
    },
    "resources": [
        {
            "type": "Microsoft.Insights/dataCollectionRules",
            "name": "[parameters('dataCollectionRuleName')]",
            "location": "[parameters('location')]",
            "apiVersion": "2022-06-01",
            "properties": {
                "dataCollectionEndpointId": "[parameters('endpointResourceId')]",
                "streamDeclarations": {
                    "Custom-WeatherRawData": {
                        "columns": [
                            {
                                "name": "location",
                                "type": "dynamic"
                            },
                            {
                                "name": "current",
                                "type": "dynamic"
                            }
                        ]
                    }
                },
                "destinations": {
                    "logAnalytics": [
                        {
                            "workspaceResourceId": "[parameters('workspaceResourceId')]",
                            "name": "myworkspace"
                        }
                    ]
                },
                "dataFlows": [
                    {
                        "streams": [
                            "Custom-WeatherRawData"
                        ],
                        "destinations": [
                            "myworkspace"
                        ],
                        "transformKql": "source | extend TimeGenerated=now()",
                        "outputStream": "Custom-Weather_CL"
                    }
                ]
            }
        }
    ],
    "outputs": {
        "dataCollectionRuleId": {
            "type": "string",
            "value": "[resourceId('Microsoft.Insights/dataCollectionRules', parameters('dataCollectionRuleName'))]"
        }
    }
}

To deploy the Data Collection rule, you can use this PowerShell script:

# variables
$location = "<your deployment location>"
$resourceGroup = "<your resource group>"
$dataCollectionRuleName = "<data collection rule name>"
$logAnalyticsWorkspaceResourceId = "<your log analytics workspace resource id>"
$dataCollectionEndpointResourceId = "<data collection endpoint resource id>"

# deploy dataCollectionRule
Write-Output "Deploying Data Collection Rule '$dataCollectionRuleName'.."
New-AzResourceGroupDeployment -ResourceGroupName $resourceGroup -Name "D_dataCollectionRule" -TemplateFile infra-as-code\arm\dataCollectionRules\dataCollectionRules.json `
    -dataCollectionRuleName $dataCollectionRuleName `
    -location $location `
    -workspaceResourceId $logAnalyticsWorkspaceId `
    -endpointResourceId "/subscriptions/$subscriptionId/resourceGroups/$resourceGroup/providers/Microsoft.Insights/dataCollectionEndpoints/$dataCollectionEndpointName"  | Out-Null

Ingest data

App Registration

Log ingestion requires an identity in the Microsoft Entra tenant. Over here you can find information on how to create a Microsoft Entra application to authenticate.

Role Assignment

To submit data to a Data Collection Endpoint you must have the Monitoring Metrics Publisher role on the Data collection rule

You can find more information on how to create that role assignment on Microsoft Learn.

NOTE: It can take some time to (up to 30 minutes) get the actual permissions to the Data collection rule!

Submitting data to the Data Collection Endpoint

PowerShell script

$DceURI = "https://dce-weu-prd-datacollection-xxxx.westeurope-1.ingest.monitor.azure.com"
$DcrImmutableId = "dcr-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
$streamName = "Custom-WeatherRawData" #name of the stream in the DCR that represents the destination table

#information needed to authenticate to AAD and obtain a bearer for app-am-alertcc-ingestion
$tenantId = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"; #Tenant ID the data collection endpoint resides in
$appId = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"; #Application ID created and granted permissions
$appSecret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"; #Secret created for the application

#Obtain a bearer token used to authenticate against the data collection endpoint
$scope= [System.Web.HttpUtility]::UrlEncode("https://monitor.azure.com//.default")   
$body = "client_id=$appId&scope=$scope&client_secret=$appSecret&grant_type=client_credentials";
$headers = @{"Content-Type"="application/x-www-form-urlencoded"};
$uri = "https://login.microsoftonline.com/$tenantId/oauth2/v2.0/token"
$token = $null
$token = (Invoke-RestMethod -Uri $uri -Method "Post" -Body $body -Headers $headers).access_token

# payload
$data = Get-Content ".\meta\weather\weatherData.json"
$data = "[" + $data + "]"

# Sending the data to Log Analytics via the DCR!
$headers = @{"Authorization" = "Bearer $token"; "Content-Type" = "application/json" };
$uri = "$DceURI/dataCollectionRules/$DcrImmutableId/streams/$streamName" + "?api-version=2021-11-01-preview";
Invoke-RestMethod -Uri $uri -Method Post -Body $data -Headers $headers;

Result

Download

You can find the templates and scripts in my GitHub Repository.

Designing your Landing Zones, the cloud adoption way

Thu, 07 Dec 2023 00:00:00 +0000

I had the opportunity to speak at Cloudbrew 2023.

My topic was “Designing your Landing Zones, the cloud adoption way”

Here’s the wrap-up of my session:

An Azure landing zone is an environment that follows key design principles across “environment” and “compliance” design areas. These design principles accommodate all application portfolios and enable application migration, modernization, and innovation at scale.

In this session I’ll discuss how to design a “Platform” and “Application” Landing Zone using these design areas.

Best practices together with experience from the field, will make you ready to successfully design your landing zones.

Here’s a link to the Presentation deck

I want to thank the organization and AZUG for the invitation and opportunity!

Deployment stacks

Fri, 25 Aug 2023 00:00:00 +0000

Table of contents
Introduction
- Deployment states
- Deployment Stacks
  - Creating / Updating Deployment Stacks
  - Deleting Deployment Stacks
Deployment Stack - Subscription Example

Introduction

Deployment states

There are multiple Infrastructure as Code languages (Terraform, Pulumi, Bicep, ARM).

Depending on the language used, there is a specific way of holding the state of resources from a deployment:

Terraform has its state file that holds the configuration
Pulumi has its state backend
Bicep & ARM have Azure as the real state of the resource

The state has always been a weakness of Bicep and ARM. The lifecycle of the resources where difficult to track, you could implement workarounds with what-if operations always an additional step to take before a deployment.

Deployment Stacks

Microsoft recently announced a public preview of ARM Deployment Stacks to tackle this. To me this was the missing element in order to use Bicep or ARM templates.

Official documentation can be found over here

Bicep or ARM template files can be submitted to a Deployment Stack, the Deployment Stacks basically define the resources that are managed by the stack.

Creating / Updating Deployment Stacks

Deployment Stacks can be created at the following scopes:

Resource Group
Subscription
Management Group

Creating/Update Deployment Stacks PowerShell & CLI commands:

Resource Group
- PowerShell: `
  - New-AzResourceGroupDeploymentStack (new)
  - Set-AzResourceGroupDeploymentStack (update)
- CLI: az stack group create (new or update)
Subscription
- PowerShell:
  - New-AzSubscriptionDeploymentStack (new)
  - Set-AzSubscriptionDeploymentStack (update)
- CLI: az stack group create (new or update)
Management Group
- PowerShell:
  - New-AzResourceGroupDeploymentStack (new)
  - Set-AzManagementGroupDeploymentStack (update)
- CLI: az stack group create (new or update)

Note: You can Block Unwanted Changes by setting a DenySettingsMode on the Deployment Stack (None, DenyDelete, DenyWriteAndDelete)

Deleting Deployment Stacks

Delete Deployment Stacks PowerShell & CLI commands:

Resource Group
- PowerShell: Remove-AzResourceGroupDeploymentStack
- CLI: az stack group delete
Subscription
- PowerShell: Remove-AzSubscriptionDeploymentStack
- CLI: az stack sub delete
Management Group
- PowerShell: Remove-AzManagementGroupDeploymentStack
- CLI: az stack mg delete

Note: A Cleanup can be done after removal of a Deployment Stack (Resources or/and Resource Groups), this is optional flag by default Resources or/and Resource Groups are detached from a Deployment Stack. Possible flags for Deployment Stack deletion: (DeleteAll, DeleteResourceGroups ,DeleteResources).

Deployment Stack - Subscription Example

In this example, the template will deploy:

a resource group
dynamic windows function app (with all contained resources)

The Deployment Stack will be scoped at subscription level.

Bicep Template & Parameter files

bicepdemo1.bicep

targetScope = 'subscription'

// **Parameters**
// Generic Parameters - Used in multiple modules 
param parLocation string
param parLocationAbbrevation string
param parEnvironment string
param parPurpose string

resource resResourceGroup 'Microsoft.Resources/resourceGroups@2022-09-01' = {
  name: 'rg-${parLocationAbbrevation}-${parEnvironment}-${parPurpose}'
  location: parLocation
}

// module dynamic windows function app
module modFunctionAppWindowsConsumption '../../modules/functionAppWindowsConsumption/functionAppWindowsConsumption.bicep' = {
  name: 'deploy-fun-${parLocationAbbrevation}-${parEnvironment}-${parPurpose}'
  scope: resourceGroup(resResourceGroup.name)
  params: {
    functionAppName: 'fun-${parLocationAbbrevation}-${parEnvironment}-${parPurpose}'
    functionWorkerRuntime: 'dotnet'
    location: parLocation
    storageAccountName: 'st${parLocationAbbrevation}${parEnvironment}${parPurpose}'
    storageAccountType: 'Standard_LRS'
    applicationInsightsName: 'appi-${parLocationAbbrevation}-${parEnvironment}-${parPurpose}'
    hostingPlanName: 'asp-${parLocationAbbrevation}-${parEnvironment}-${parPurpose}'
  }
}

tst-bicepdemo1.bicepparam

using '../bicepdemo1.bicep'

param parLocation            = 'westeurope'
param parLocationAbbrevation = 'weu'
param parEnvironment         = 'tst'
param parPurpose             = 'bicepdemo1'

Deploy a Deployment Stack

New-Deploymentstack.ps1

$inputObject = @{
    Name                        = "stack-weu-tst-bicepdemo1"
    TemplateFile                = "infra-as-code/bicep/orchestration/bicepdemo1/bicepdemo1.bicep"
    TemplateParameterFile       = "infra-as-code/bicep/orchestration/bicepdemo1/parameters/tst-bicepdemo1.bicepparam"
    DenySettingsMode            = "DenyWriteAndDelete"
    Location                    = "westeurope"
}

New-AzSubscriptionDeploymentStack  @inputObject -Force

Deployment Result

Ar subscription level, the Deployment Stack will become visible:

In the deployed Deployment Stack, the details will be displayed:

Note: The Deny status for all Resource Groups and Resources of the Deployment Stack are set to denyWriteAndDelete, the Deployment Stack will block all write and delete actions on the Resource Group or Resources controlled by the Deployment Stack.

Deny assignments:

In an attemnt of deleting a storage account controlled by the Deployment Stack, will result as a deny because of the deny assignment created by Deployment Stack:

Update a Deployment Stack

To update a Deployment Stack (update controlled Resource Group, Resources or update DenySettingsMode), you can use the following script.

Note: In this example the DenySettingMode is set to None

Set-Deploymentstack.ps1

$inputObject = @{
    Name                        = "stack-weu-tst-bicepdemo1"
    TemplateFile                = "infra-as-code/bicep/orchestration/bicepdemo1/bicepdemo1.bicep"
    TemplateParameterFile       = "infra-as-code/bicep/orchestration/bicepdemo1/parameters/tst-bicepdemo1.bicepparam"
    DenySettingsMode            = "None"
    Location                    = "westeurope"
}

Set-AzSubscriptionDeploymentStack  @inputObject -Force 

Update Result

The Deny status for the resources were updated:

Also visible in the Deny assignments blade:

Delete a Deployment Stack

To delete a Deployment Stack and delete the resource groups and resources you can use the following script.

Remove-Deploymentstack.ps1

Remove-AzSubscriptionDeploymentStack `
    -name "stack-weu-tst-bicepdemo1" `
    -DeleteAll `
    -Force

Note: The removal of Deployment Stacks support the following options:

Detach: Detach resources and resource groups (Default)

DeleteAll: Delete both the resources and the resource groups

DeleteResourceGroups: Delete the resource groups only

DeleteResources: Delete resources only

PowerShell Authentication tokens for Azure REST API

Fri, 11 Aug 2023 00:00:00 +0000

Introduction
- Azure REST API Authentication
  - JSON Web Token (JWT)
  - Authentication to the Management API
Authentication with an App Registration - Example
Authentication with current account
- Using Get-AccessToken
  - Example
- Using the token stored in the profile
  - Example

Introduction

The Azure REST API is a powerful way for interacting with Azure service resources.

In this article, I want to explain the authentication options that you have with PowerShell to interact with the Azure REST API

Azure REST API Authentication

Authentication to the Azure REST API is done by using an OAuth2 bearer token.

With this token you can authenticate your request on the Azure REST API:

JSON Web Token (JWT)

The token itself is a JSON Web Token (JWT), it consist of three parts separated by dots (.), which are:

Header
Payload
Signature

Therefore, a JWT typically looks like the following.

xxxxx.yyyyy.zzzzz

Note:
To decode a JWT token, you can use the online tool over here: https://jwt.io/

To dump the claims in your bearer token so you can validate their contents you can use the following online tool: https://jwt.ms/

Authentication to the Management API

An Access token is a Bearer token that you will have to add in all request headers to be authenticated.

For example, you might send an HTTPS GET request method for an Azure Resource Manager provider to list subscriptions by using request header fields that are similar to the following:

$path = "/subscriptions"
$apiVersion = "2022-12-01"
$token = "<your-acquired-token>"

$authHeader = @{
    'Content-Type'  = 'application/json'
    'Authorization' = "Bearer $token"
}

# invoke Azure Management REST API
$uri = "https://management.azure.com$($path)?api-version=$($apiVersion)"
Write-Host "Invoking request to: '$uri'"
$data = Invoke-RestMethod -Method Get -Uri $uri -Headers $authHeader

Authentication with an App Registration

A convenient way of authentication is using an App Registration.

The access token can be aquired using an access token request with a shared secret.

Get-AppRegistrationAuthorizationToken function:

function Get-AppRegistrationAuthorizationToken {
	[CmdletBinding()]
	param
	(
		[Parameter(Mandatory = $true)]
		[string]$TenantID,
		[Parameter(Mandatory = $true)]
		[string]$ClientID,
		[Parameter(Mandatory = $true)]
		[string]$ClientSecret
	)

	$TokenEndpoint = "https://login.windows.net/$TenantID/oauth2/token"
	$ARMResource = "https://management.core.windows.net/"

	$Body = @{
		'resource'      = $ARMResource
		'client_id'     = $ClientID
		'grant_type'    = 'client_credentials'
		'client_secret' = $ClientSecret
	}

	$params = @{
		ContentType = 'application/x-www-form-urlencoded'
		Headers     = @{'accept' = 'application/json' }
		Body        = $Body
		Method      = 'Post'
		URI         = $TokenEndpoint
	}

	$token = Invoke-RestMethod @params

	Return ($token.access_token).ToString()
}

Example

# set variables
$client_id = "<your-app-registration-client-id>"
$app_secret = "<your-app-registration-secret>"
$tenantid = "<your-tenant-id>"
$token = Get-AppRegistrationAuthorizationToken -TenantID $tenantid -ClientID $client_id -ClientSecret $app_secret
$path = "/subscriptions"
$apiVersion = "2022-12-01"
$authHeader = @{
    'Authorization' = "Bearer $($token)"
}

# invoke Azure Management REST API
$uri = "https://management.azure.com$($path)?api-version=$($apiVersion)"
$result = Invoke-RestMethod -Method Get -Uri $uri -Headers $authHeader

# return result
$result.value 

Authentication with current account

When already authenticated, you can get the access token for the identity that is already logged in into Azure.

Using Get-AccessToken

The PowerShell command Get-AccessToken (from the Az.Accounts module) can be used to get a fresh access token.

Using the ResourceTypeName

Get-AzAccessToken
   [-ResourceTypeName <String>]
   [-TenantId <String>]

Get-AzAccessToken -ResourceTypeName Arm

Using the ResourceUrl

Get-AzAccessToken
   -ResourceUrl <String>
   [-TenantId <String>]

Get-AzAccessToken -ResourceUrl "https://management.core.windows.net/"

Example

# set variables
$token = Get-AzAccessToken -ResourceUrl "https://management.core.windows.net/"
$path = "/subscriptions"
$apiVersion = "2022-12-01"
$authHeader = @{
    'Authorization' = "Bearer $($token.Token)"
}

# invoke Azure Management REST API
$uri = "https://management.azure.com$($path)?api-version=$($apiVersion)"
$result = Invoke-RestMethod -Method Get -Uri $uri -Headers $authHeader

# return result
$result.value 

Using the token stored in the profile

Using the Microsoft.Azure.Commands.ResourceManager.Common Namespace .NET Namespace, you can use the AcquireAccessToken Method to get the stored token in profile.

Get-AzureCachedAccessToken function:

function Get-AzureCachedAccessToken {
    # get current Azure Profile
	$azProfile = [Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureRmProfileProvider]::Instance.Profile

    # gets the metadata used to authenticate Azure Resource Manager requests.
	$currentAzureContext = Get-AzContext

    # initiate client session to access profile
	$profileClient = New-Object Microsoft.Azure.Commands.ResourceManager.Common.RMProfileClient($azProfile)

    # get the token from the profile
	$token = $profileClient.AcquireAccessToken($currentAzureContext.Tenant.TenantId)
	return $token.AccessToken
}

Example

$token = Get-AzureCachedAccessToken

$path = "/subscriptions"
$apiVersion = "2022-12-01"
$authHeader = @{
    'Authorization' = "Bearer $($token)"
}

# invoke Azure Management REST API
$uri = "https://management.azure.com$($path)?api-version=$($apiVersion)"
$result = Invoke-RestMethod -Method Get -Uri $uri -Headers $authHeader

$result.value 

Ingest Azure resource data into a Log Analytics Workspace Custom Table

Fri, 23 Jun 2023 00:00:00 +0000

** This post is outdated**

Log Analytics is now natively supporting Resource Graph queries

Introduction

Creating Log Analytics workspace Log queries sometimes requires data from resources that is not available in any Log Analytics workspace table. For example: tags, resource properties, …

As workaround, I developed a Logic App that queries Azure Resource Graph and injects the Logs in a Log Analytics workspace using the Azure Monitor Data Collector API:

Flow

Query the Azure Resource Graph for all Azure Resources (interval 15 minutes)
Post the retreived json data from all Azure Resources to the Data Collection Endpoint
Use the Data Collection Rule to transform the data and stream to a Log Analytics workspace Custom Table

The Setup

Log Analytics workspace Resources_CL Table

In order to ingest data we have to setup a Log Analytics workspace custom table with a predefined schema. The fields more or less match the fields from the Azure Resource Graph Resources, but some of them are not allowed to be used in a custom Log Analytics workspace Table

Here is the schema I used to create the table:

table.json

{
    "properties": {
        "schema": {
            "name": "Resources_CL",
            "columns": [
                {
                    "name": "TimeGenerated",
                    "type": "datetime",
                    "description": "The time at which the data was generated"
                },
                {
                    "name": "extendedLocation",
                    "type": "dynamic"
                },
                {
                    "name": "identity",
                    "type": "dynamic"
                },
                {
                    "name": "location",
                    "type": "dynamic"
                },
                {
                    "name": "managedBy",
                    "type": "dynamic"
                },
                {
                    "name": "name",
                    "type": "string"
                },
                {
                    "name": "plan",
                    "type": "dynamic"
                },
                {
                    "name": "properties",
                    "type": "dynamic"
                },
                {
                    "name": "resourceGroup",
                    "type": "dynamic"
                },
                {
                    "name": "resourceId",
                    "type": "string"
                },
                {
                    "name": "resourceKind",
                    "type": "dynamic"
                },
                {
                    "name": "resourceTenantId",
                    "type": "string"
                },
                {
                    "name": "resourceType",
                    "type": "string"
                },
                {
                    "name": "sku",
                    "type": "dynamic"
                },
                {
                    "name": "subscriptionId",
                    "type": "string"
                },
                {
                    "name": "tags",
                    "type": "dynamic"
                },
                {
                    "name": "zones",
                    "type": "dynamic"
                }
            ]
        }
    }
}

There are a few ways to create a Log Analytics Custom Table using the Portal, API, CLI or PowerShell.

For this example I used the PowerShell way with the Invoke-RestMethod

# variables
$resourceGroup = "<your resource group>"
$logAnalyticsworkspaceId = "<your Log Analytics workspace Resource Id>"
$tableName = "Resources_CL"

function get-azCachedAccessToken() {
    $currentAzureContext = Get-AzContext
    $azProfile = [Microsoft.Azure.Commands.Common.Authentication.Abstractions.AzureRmProfileProvider]::Instance.Profile
    $profileClient = New-Object Microsoft.Azure.Commands.ResourceManager.Common.RMProfileClient($azProfile)
    $token = $profileClient.AcquireAccessToken($currentAzureContext.Tenant.TenantId)
    $token.AccessToken
}

# create authentication header
$token = get-azCachedAccessToken 
$authHeader = @{
    'Authorization' = "Bearer $token"
    'Content-Type'  = "application/json"
}

# create table
Write-Output "Creating new table '$tableName'.."
$uri = "https://management.azure.com$($logAnalyticsworkspaceId)/tables/$($tableName)?api-version=2021-12-01-preview"
$jsonBody = [String](Get-Content -Path .\table.json)
Invoke-RestMethod -Method Put -Uri $uri -Headers $authHeader -Body $jsonBody 

The result is a custom Log Analyitcs workspace Table:

Data collection endpoint

To have an endpoint to send our data to, we have to setup a Data collection endpoint.

To setup the Data collection endpoint, I use this ARM template:

dataCollectionEndpoint.json

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "dataCollectionEndpointName": {
            "type": "string",
            "metadata": {
                "description": "Specifies the name of the Data Collection Endpoint to create."
            }
        },
        "location": {
            "type": "string",
            "defaultValue": "westeurope",
            "allowedValues": [
                "westus2",
                "eastus2",
                "eastus2euap",
                "westeurope"
            ],
            "metadata": {
                "description": "Specifies the location in which to create the Data Collection Endpoint."
            }
        }
    },
    "resources": [
        {
            "type": "Microsoft.Insights/dataCollectionEndpoints",
            "name": "[parameters('dataCollectionEndpointName')]",
            "location": "[parameters('location')]",
            "apiVersion": "2021-04-01",
            "properties": {
                "networkAcls": {
                "publicNetworkAccess": "Enabled"
                }
            }
        }
    ],
    "outputs": {
        "dataCollectionEndpointId": {
            "type": "string",
            "value": "[resourceId('Microsoft.Insights/dataCollectionEndpoints', parameters('dataCollectionEndpointName'))]"
        }
    }
}

and use the following PowerShell script to deploy:

# variables
$location = "<your deployment location>"
$resourceGroup = "<your resource group>"
$dataCollectionEndpointName = "<your data collection endpoint name>"

# deploy dataCollectionEndpoint 
Write-Output "Deploying Data Collection Endpoint '$dataCollectionEndpointName'.."
New-AzResourceGroupDeployment -ResourceGroupName $resourceGroup -Name "D_dataCollectionEndpoint" -TemplateFile .\dataCollectionEndpoint.json `
    -dataCollectionEndpointName $dataCollectionEndpointName `
    -location $location

Data collection rule

This is where the magic happens, with the Data collection rule, we can setup:

dataCollectionEndpointId: the resourceId of the Data collection endpoint created earlier
streamDeclarations: This unique handle describes a set of data sources that will be transformed and schematized as one type. Each data source requires one or more streams, and one stream can be used by multiple data sources. All data sources in a stream share a common schema. Use multiple streams, for example, when you want to send a particular data source to multiple tables in the same Log Analytics workspace. Define a schema for the source of data. This is the schema of the Resources query that we fetched from Azure Resource Graph
destinations: This section contains a declaration of all the destinations where the data will be sent. Only Log Analytics workspace is currently supported as a destination. Each Log Analytics workspace destination requires the full workspace resourceId and a friendly name that will be used elsewhere in the DCR to refer to this workspace.
dataFlows: This section ties the other sections together. It defines the following properties for each stream declared in the streamDeclarations section:
- streams: This unique handle describes a set of data sources that will be transformed and schematized as one type. Each data source requires one or more streams, and one stream can be used by multiple data sources. All data sources in a stream share a common schema. Use multiple streams, for example, when you want to send a particular data source to multiple tables in the same Log Analytics workspace.
- destinations: is the transformation applied to the data that was sent in the input shape described in the streamDeclarations section to the shape of the target table.
- transformKql: Is the transformation applied to the data that was sent in the input shape described in the streamDeclarations section to the shape of the target table. In this usecase we will extend the table with the allowed fields in the custom Log Analytics workspace table.
- outputStream: Describes which table in the workspace specified under the destination property the data will be ingested into. The value of outputStream has the Microsoft-[tableName] shape when data is being ingested into a standard Log Analytics table, or Custom-[tableName] when ingesting data into a custom-created table. Only one destination is allowed per stream.

To deploy the Data Collection Rule, I use this ARM template:

dataCollectionRules.json

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "dataCollectionRuleName": {
            "type": "string",
            "metadata": {
                "description": "Specifies the name of the Data Collection Rule to create."
            }
        },
        "location": {
            "type": "string",
            "defaultValue": "westeurope",
            "allowedValues": [
                "westus2",
                "eastus2",
                "eastus2euap",
                "westeurope"
            ],
            "metadata": {
                "description": "Specifies the location in which to create the Data Collection Rule."
            }
        },
        "workspaceResourceId": {
            "type": "string",
            "metadata": {
                "description": "Specifies the Azure resource ID of the Log Analytics workspace to use."
            }
        },
        "endpointResourceId": {
            "type": "string",
            "metadata": {
                "description": "Specifies the Azure resource ID of the Data Collection Endpoint to use."
            }
        }
    },
    "resources": [
        {
            "type": "Microsoft.Insights/dataCollectionRules",
            "name": "[parameters('dataCollectionRuleName')]",
            "location": "[parameters('location')]",
            "apiVersion": "2021-09-01-preview",
            "properties": {
                "dataCollectionEndpointId": "[parameters('endpointResourceId')]",
                "streamDeclarations": {
                    "Custom-Resources_CL": {
                        "columns": [
                            {
                                "name": "id",
                                "type": "string"
                            },
                            {
                                "name": "name",
                                "type": "string"
                            },
                            {
                                "name": "type",
                                "type": "string"
                            },
                            {
                                "name": "tenantId",
                                "type": "string"
                            },
                            {
                                "name": "kind",
                                "type": "dynamic"
                            },
                            {
                                "name": "location",
                                "type": "dynamic"
                            },
                            {
                                "name": "resourceGroup",
                                "type": "dynamic"
                            },
                            {
                                "name": "subscriptionId",
                                "type": "string"
                            },
                            {
                                "name": "managedBy",
                                "type": "dynamic"
                            },
                            {
                                "name": "sku",
                                "type": "dynamic"
                            },
                            {
                                "name": "plan",
                                "type": "dynamic"
                            },
                            {
                                "name": "properties",
                                "type": "dynamic"
                            },
                            {
                                "name": "tags",
                                "type": "dynamic"
                            },
                            {
                                "name": "identity",
                                "type": "dynamic"
                            },
                            {
                                "name": "zones",
                                "type": "dynamic"
                            },
                            {
                                "name": "extendedLocation",
                                "type": "dynamic"
                            }
                        ]
                    }
                },
                "destinations": {
                    "logAnalytics": [
                        {
                            "workspaceResourceId": "[parameters('workspaceResourceId')]",
                            "name": "myworkspace"
                        }
                    ]
                },
                "dataFlows": [
                    {
                        "streams": [
                            "Custom-Resources_CL"
                        ],
                        "destinations": [
                            "myworkspace"
                        ],
                        "transformKql": "source 
                        | extend TimeGenerated=now() 
                        | extend resourceType=[\"type\"]
                        | extend resourceId=id
                        | extend resourceTenantId=tenantId
                        | extend resourceKind=[\"kind\"]",
                        "outputStream": "Custom-Resources_CL"
                    }
                ]
            }
        }
    ],
    "outputs": {
        "dataCollectionRuleId": {
            "type": "string",
            "value": "[resourceId('Microsoft.Insights/dataCollectionRules', parameters('dataCollectionRuleName'))]"
        }
    }
}

To deploy the Data Collection rule, you can use this PowerShell script:

# variables
$location = "<your deployment location>"
$resourceGroup = "<your resource group>"
$dataCollectionRuleName = "<data collection rule name>"
$logAnalyticsWorkspaceResourceId = "<your log analytics workspace resource id>"
$dataCollectionEndpointResourceId = "<data collection endpoint resource id>"

# deploy dataCollectionRule
Write-Output "Deploying Data Collection Rule '$dataCollectionRuleName'.."
New-AzResourceGroupDeployment -ResourceGroupName $resourceGroup -Name "D_dataCollectionRule" -TemplateFile .\dataCollectionRules.json `
    -dataCollectionRuleName $dataCollectionRuleName `
    -location $location `
    -workspaceResourceId $logAnalyticsWorkspaceResourceId `
    -endpointResourceId $dataCollectionEndpointResourceId

Logic App

Now everything is prepared for ingestion, we can setup the Logic App. The Logic App will use a User Assigned Managed Identity that is linked with the Logic App. Form more details on the setup look at the article Authenticate access to Azure resources with managed identities in Azure Logic Apps

The User Assigned Managed Identity will allow us to authenticate securely to Azure Resource Graph and ingest the data into the Custom Log Analytics workspace table.

We need the following Role Based Access Control (RBAC) role assignments for the User Assigned Managed Identity:

Reader role on the subscription: To read out the resources of the subscription
Monitoring Metrics Publisher role on the Data collection rule: permissions to post data to the Data collection rule and Data collection endpoint

NOTE: It can take some time to (up to 30 minutes) get the actual permissions to the Data collection rule!

Here’s the Logic App Flow:

Logic App Design

A few variables are set
- DceURI: the Uri of the Data collection rule (can be found on the Data collection rule Essentials view in the portal)
- DcrImmutableId: The immutableId property of the DCR object (can be found on the Data collection rule Essentials JSON view in the portal)
- CustomTableName: The name of the custom table (Resource_CL)
HTTP - Get Resource Graph Resources

Using the Azure Resource Graph API query the Resources. The User Assigned Managed Identity is used for this action.
HTTP - Post - Resources to Data Collection Endpoint

Concatinate the variables to a usable endpoint for ingestion: concat(variables(‘DceURI’),’/dataCollectionRules/’,variables(‘DcrImmutableId’),’/streams/Custom-‘,variables(‘CustomTableName’),’?api-version=2021-11-01-preview’)

Ingest the data field of the previously fetched Azure Resource Graph query the Resources data.

The User Assigned Managed Identity is used for this action.

The Result

The data from Azure Resource Graph will now be ingested into the Resources_CL custom table every 15 minutes.

You can query the latest records in the Resource_CL table using the following query:

Resources_CL | summarize arg_max(TimeGenerated,*) by resourceId

The Result looks like this:

You can now start to join the table data with other data in the platform log default tables.

Download

You can find the templates, scripts and Logic App in my GitHub Repository over HERE.

Kusto Time Series Analysis for Azure Resources - Workbook

Mon, 12 Jun 2023 00:00:00 +0000

Introduction

As explained in previous blog post, you can use the Kusto Query Language (KQL) for Anomaly detection and forecasting.

This is great for ad-hoc querying the logs, and visualizing the data with the metrics explorer. However, having a tool ready to explore the metrics is much better.

Azure Monitor Workbooks are the perfect tool for that.

For your convenience, I’ve created an example workbook.

Workbook

Parameters

Timerange: The TimeRange

Subscription: The Subscription

Workspace: The Log Analytics Workspace

ResourceType: The Resource Type to filter out

Resource: The Resource Selected

Metric: The Metric

Aggregation: The type of aggregation (Avg, Min, Max, Pct5, Pct10, Pct50, Pct90, Pct95)

Forecast

This is done using the series_decompose_forecast() function.

Trendline

This is done using the series_fit_line() function.

Anomalies

This is done using the series_decompose_anomalies() function.

I also added a table that extracts the outliers for the anomalies using the mv-expand operator on the timeseries.

Download

Here’s the ARM Template the Gallery Template.

Kusto Time Series Analysis for Azure Resources (Forecast, Anomalies)

Wed, 31 May 2023 00:00:00 +0000

Introduction

Azure Monitor Logs is based on Azure Data Explorer, and log queries are written by using the same Kusto Query Language (KQL). This rich language is designed to be easy to read and author, so you should be able to start writing queries with some basic guidance.

Kusto Query Language (KQL) contains native support for creation, manipulation, and analysis of multiple time series. With KQL, you can create and analyze thousands of time series in seconds, enabling near real time monitoring solutions and workflows.

The Kusto Query language (KQL) is having a few interesting functions for Anomaly detection and forecasting.

In this article I will explain how to use these functions for Azure platform resource performance metric data stored in Logs in a Log Analytics Workspace.

Exporting platform metrics to Logs

Theory

You can export the Azure platform metrics from the Azure monitor pipeline to Azure Monitor Logs (and thus Log Analytics).

The export can be configured using the diagnostic setting for an Azure resource

The Azure platform metrics performance data is stored in the AzureMetrics table contains Metric data emitted by Azure services that measure their health and performance.

Timechart Example

A regular query to display a timechart for the Average CurrentConnections for an Application Gateway could be:

Example Query

AzureMetrics
| where ResourceId =~ "**your Application Gateway resource id**" 
| where MetricName =~ "CurrentConnections"
| summarize 
    avg = avg(Average)  
    by
    ResourceId=tolower(ResourceId),
    bin(TimeGenerated, 1h)
| render timechart 

Result:

Time Series Creation using ‘make-series’

The first step in time series analysis is to partition and transform the original telemetry table to a set of time series. The table usually contains a timestamp column, contextual dimensions, and optional metrics. The dimensions are used to partition the data. The goal is to create thousands of time series per partition at regular time intervals.

This is done by using the make-series operator.

make-series CurrentConnections=avg(avg) on TimeGenerated from ago(30d) to now(-1d) step 1h

In this example the average CurrentConnections are calculated on the Timestamp field TimeGenerated from 30 days ago until yesterday it will generate a timeseries with an interval of 1h samples.

Example Query

AzureMetrics
| where ResourceId =~ "**your Application Gateway resource id**" 
| where MetricName =~ "CurrentConnections"
| summarize 
    avg = avg(Average)  
    by
    ResourceId=tolower(ResourceId),
    bin(TimeGenerated, 1h)
| project
    avg,
    ResourceId,
    TimeGenerated
| make-series CurrentConnections=avg(avg) on TimeGenerated from ago(30d) to now(-1d) step 1h

Result

The result shows a set of time series, the average of the CurrentConnections with an interval of 1 hour from 30 days ago until yesterday.

Example Query with timechart

You can add the render operator to viualize the results in a timechart:

AzureMetrics
| where ResourceId =~ "**your Application Gateway resource id**" 
| where MetricName =~ "CurrentConnections"
| summarize 
    avg = avg(Average)  
    by
    ResourceId=tolower(ResourceId),
    bin(TimeGenerated, 1h)
| project
    avg,
    ResourceId,
    TimeGenerated
| make-series CurrentConnections=avg(avg) on TimeGenerated from ago(30d) to now(-1d) step 1h

Result

Forecasting

Now it is getting interesting. The Kusto query language has a function to decompose the timeseries using a well-known decomposition model, where each original time series is decomposed into seasonal, trend, and residual components and add predicts the values of the future timeline to it.

This is done using the series_decompose_forecast() function.

Syntax

series_decompose_forecast(Series, Points, [ Seasonality, Trend, Seasonality_threshold ])

Example Query

Note that I’ve added some variable declerations in the beginning: min_t, max_t, dt and horizon

The make-series operator is changed to extend the period to 7 days in the future (max_t+horizon).

The Forecast is added with the series_decompose_forecast() function based on the CurrentConnections in the time series.

The number of points are calculated from the number of days în the forecast devided by the interval: ‘horizon / dt’

let min_t = ago(30d);
let max_t = now(-1d);
let dt = 1h;
let horizon=7d;
AzureMetrics
| where ResourceId =~ "**your Application Gateway resource id**"
| where MetricName =~ "CurrentConnections"
| summarize 
    avg = avg(Average)  
    by
    ResourceId=tolower(ResourceId),
    bin(TimeGenerated, 1h)
| project
    avg,
    ResourceId,
    TimeGenerated
| make-series CurrentConnections=avg(avg) on TimeGenerated from min_t to max_t+horizon step dt
| extend Forecast = series_decompose_forecast(CurrentConnections, toint(horizon / dt))
| render timechart 

Query Result

The series_decompose_forecast() function adds the series for the number of points stated in the function. Resulting in a cool timechart with a future Forecast timeline:

Anomaly Detection

Similar to the series_decompose_forecast() the function series_decompose_anomalies() is based on series decomposition. The function takes an expression containing a series (dynamic numerical array) as input, and extracts anomalous points with scores.

Syntax

series_decompose_anomalies (Series, [ Threshold, Seasonality, Trend, Test_points, AD_method, Seasonality_threshold ])

Example Query

let min_t = ago(30d);
let max_t = now(-1d);
let dt = 1h;
AzureMetrics
| where ResourceId =~ "**your Application Gateway resource id**"
| where MetricName =~ "CurrentConnections"
| summarize 
    avg = avg(Average)  
    by
    ResourceId=tolower(ResourceId),
    bin(TimeGenerated, dt)
| project
    avg,
    ResourceId,
    TimeGenerated
| make-series CurrentConnections = avg(avg) on TimeGenerated from min_t to max_t step dt
| extend (Anomalies) = series_decompose_anomalies(CurrentConnections, 1.5, -1, 'linefit')
| render timechart 

Query Result

The result is nice, adding a Anomalies timeline for the anomalies. -1 showing a lower anomaly, +1 showing a higher anomaly.

The only disadvantage is that the anomalies is hard to see if the values of the timeseries are higher.

Example Query (fixing the scale of the anomanies)

You can fix the scaling of the Anomalies timeline by taking a toscalar() of the average overtime, and apply a series_multiply() to the time series:

let grain = 1h;
let metric_scale = toscalar(
    AzureMetrics
    | where ResourceId =~ "**your Application Gateway resource id**"
    | where MetricName =~ "CurrentConnections"
    | summarize 
        avg = avg(Average)  
    | summarize avg(avg));
AzureMetrics
| where ResourceId =~ "**your Application Gateway resource id**"
| where MetricName =~ "CurrentConnections"
| summarize 
    avg = avg(Average)  
    by
    ResourceId=tolower(ResourceId),
    bin(TimeGenerated, 1h)
| project
    avg,
    ResourceId,
    TimeGenerated
| make-series CurrentConnections = avg(avg) default = 0 on TimeGenerated from ago(30d) to now(-1d) step grain
| extend (anomalies) = series_decompose_anomalies(CurrentConnections, 1.5, -1, 'linefit')
| extend anomalies = 
    series_multiply(metric_scale, anomalies) // multiplied for visualization purposes
| render timechart 

Query Result

The result is showing a easier interpretable output:

Conclusion

The time series analysis functionality is realy easy to use and showing lost of potential.

In a future post, I will show you can dynamically visualize the data for Azure Platform Resources using Azure Workbooks.

Deploying Terraform using Github Actions with OpenID Connect (OIDC)

Mon, 08 May 2023 00:00:00 +0000

Introduction

Github uses a nice feature called OpenID Connect (OIDC) for Federated Authentication.

OpenID Connect (OIDC) allows your GitHub Actions workflows to access resources in Azure, without needing to store the Azure credentials as long-lived GitHub secrets.

For more information on how to setup you can read the GitHub documentation: Configuring OpenID Connect in Azure

Authentication is then done with the azure/login@v1 action.

    - name: 'Az CLI login'
      uses: azure/login@v1
      with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}

Possible Terraform issue

This is working fine for direct deployments with ARM or Bicep templates, however, when using the When tying to use the authentication with Terraform whith a Terraform plan or a terraform apply, you might bump into the following error:

Error: Error building AzureRM Client: Authenticating using the Azure CLI is only supported as a User (not a Service Principal). To authenticate to Azure using a Service Principal, you can use the separate ‘Authenticate using a Service Principal’ auth method - instructions for which can be found here: https://www.terraform.io/docs/providers/azurerm/guides/service_principal_client_secret.html

To enable Terraform to use `OpenID Connect (OIDC)`

Add the `use_oidc = true` section to the backend settings as explained in the azurerm documentation

Example azure.tf:

terraform {
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
      version = "3.30.0"
    }
  }
}

provider "azurerm" {
  use_oidc = true
  features {}
}

Declare the environment variables in the `GitHub workflow`:

env:
  ARM_SUBSCRIPTION_ID: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
  ARM_CLIENT_ID: ${{ secrets.AZURE_CLIENT_ID }}
  ARM_TENANT_ID: ${{ secrets.AZURE_TENANT_ID }}

You can find a working example on my GitHub TFdeploy repo

Philip Van de Vyver - Azure Blog

Azure Update Manager pre and post events for turned off VMs

Introduction

Approach

Configuration

User Assigned Managed Identity

Automation Account

Automation Account Modules

Automation Account Variable

Automation Account Runbooks

Automation Account Webhooks

Register your subscription for public preview

Maintenance Configuration

Event Grid Topic

Event Grid Subscriptions

Result

Github Actions monitoring with Application Insights

Introduction

Ingest Application Insights Request Telemetry data

Github Action

PowerShell script

Result

Ingest Application Insights Release Annotation

Github Action

PowerShell script

Result

Ingest custom data into a Log Analytics Workspace

Introduction

Flow

The Setup

Log Analytics workspace Weather_CL table

Data collection endpoint

Data collection rule

Ingest data

App Registration

Role Assignment

Submitting data to the Data Collection Endpoint

PowerShell script

Result

Download

Designing your Landing Zones, the cloud adoption way

Deployment stacks

Table of contents

Introduction

Deployment states

Deployment Stacks

Creating / Updating Deployment Stacks

Deleting Deployment Stacks

Deployment Stack - Subscription Example

Bicep Template & Parameter files

Deploy a Deployment Stack

Deployment Result

Update a Deployment Stack

Update Result

Delete a Deployment Stack

PowerShell Authentication tokens for Azure REST API

Table of contents

Introduction

Azure REST API Authentication

JSON Web Token (JWT)

Authentication to the Management API

Authentication with an App Registration

Example

Authentication with current account

Using Get-AccessToken

Example

Using the token stored in the profile

Example

Ingest Azure resource data into a Log Analytics Workspace Custom Table

Introduction

Flow

The Setup

Log Analytics workspace Resources_CL Table

Data collection endpoint

Data collection rule

Logic App

Logic App Design

The Result

Download

Kusto Time Series Analysis for Azure Resources - Workbook

To enable Terraform to use `OpenID Connect (OIDC)`

Add the `use_oidc = true` section to the backend settings as explained in the azurerm documentation

Declare the environment variables in the `GitHub workflow`: