Configure diagnostic logging and log retention by using Azure Monitor

Configuring diagnostic logging and log retention in Azure is an essential part of managing and maintaining the security posture of your cloud resources. Azure Monitor is the central service that provides you with the ability to collect, analyze, and act on telemetry data from your Azure and on-premises environments. It helps you maximize the performance and availability of your applications and proactively identify problems in seconds.

1. Enabling Diagnostic Logging:

Diagnostic settings in Azure enable you to specify which data platform logs and metrics should be collected and where they should be sent. This can include sending data to Azure Monitor Logs (Log Analytics workspace), Event Hubs, and Azure Storage.

To configure diagnostic settings for an Azure resource:

• Navigate to the Azure portal.
• Select the specific resource for which you want to enable diagnostic logging.
• Go to the ‘Diagnostic settings’ section and click on ‘Add diagnostic setting’.
• Give your setting a name and select the categories of logs and metrics you want to collect.
• Choose the destination for your logs and metrics (Log Analytics workspace, Event Hubs, or Storage account).
• Save the diagnostic setting.

Example:

# Use Azure CLI to configure diagnostic setting for a virtual machine
az monitor diagnostic-settings create \
–resource /subscriptions/{SubID}/resourceGroups/{RG}/providers/Microsoft.Compute/virtualMachines/{VMName} \
–name “myDiagnosticSetting” \
–logs ‘[{“category”: “AuditLogs”,”enabled”: true}]’ \
–metrics ‘[{“category”: “AllMetrics”,”enabled”: true}]’ \
–workspace /subscriptions/{SubID}/resourceGroups/{RG}/providers/Microsoft.OperationalInsights/workspaces/{WorkspaceName} \
–storage-account /subscriptions/{SubID}/resourceGroups/{RG}/providers/Microsoft.Storage/storageAccounts/{StorageAccountName} \
–event-hub {EventHubName}@{EventHubNamespace}

2. Log Retention Policies:

Azure Monitor provides the ability to configure retention policies to control how long your collected data is retained within the service. This is crucial from both a cost-management perspective and compliance with various regulatory frameworks.

By default, the data in a Log Analytics workspace is retained for 30 days at no extra cost. However, the retention period can be configured from 30 to 730 days, depending on your needs.

Example of setting a retention policy using Azure CLI:

# Set retention policy for a Log Analytics workspace
az monitor log-analytics workspace update \
–resource-group {ResourceGroupName} \
–workspace-name {WorkspaceName} \
–retention-time {NumberOfDays}

For Azure Storage, the data retention can be set indefinitely or for a specified period. This configuration can be done through the Azure portal or programmatically. For Event Hubs, the retention policy can be between 1 and 7 days.

3. Monitoring and Alerts:

Once you have set up diagnostic logging and retention policies, you can use Azure Monitor to create alerts based on specific metrics or log queries. This allows you to be notified in case of an anomaly or specific event that requires attention.

• In the Azure portal, navigate to ‘Alerts’ and then ‘New alert rule’.
• Choose the target resource and condition (metric or log).
• Define the alert logic with thresholds or query.
• Set the action group which defines what actions to take when the alert fires.

4. Review and Reporting:

Azure Monitor Logs provide powerful querying capabilities with Kusto Query Language (KQL) to extract meaningful insights from your log data.

Example of a KQL query to fetch audit log entries:

AuditLogs
| where TimeGenerated > ago(30d)
| where Category == “Write”
| order by TimeGenerated desc
| project TimeGenerated, OperationName, OperationVersion, Category, ResultType, ResultDescription, CallerIpAddress, Identity

You can also use built-in or custom workbooks in Azure Monitor for interactive, visual reporting and analysis of your log data.

In conclusion, properly configuring diagnostic logging and log retention in Azure Monitor plays a crucial role in the security management of Azure resources. Following the outlined steps ensures that you have the necessary information for troubleshooting, auditing, and compliance purposes, helping you to maintain robust security within your Azure environment.