Create custom logs in Azure Log Analytics to store custom data

Azure Log Analytics is a powerful monitoring and analytics tool within the Azure platform, offering the ability to collect, analyze, and act on telemetry data from cloud and on-premises environments. Custom logs in Azure Log Analytics provide a flexible approach to capturing custom data that is not automatically collected by Azure Monitor agents. This capability is valuable for security operations analysts who might need to log custom events, performance data, or telemetry from various applications and services for analysis and compliance purposes.

Understanding Custom Logs

Custom logs allow an organization to define a new data type in Azure Log Analytics, to which data from text files can be uploaded. For instance, you may have logs generated by a custom application or any non-standard logs you’d like to analyze alongside your standard Azure telemetry. These logs can then be used in queries, alert rules, and other Log Analytics features.

Creating Custom Logs

To create custom logs in Azure Log Analytics, follow these key steps:

1. Define a Custom Log:
   • Access your Log Analytics workspace in the Azure Portal.
   • In the Workspace Data Sources section, select ‘Custom Logs’.
   • Click on the ‘Add+’ button to start the custom log wizard, which will guide you through the process of defining your new log.
2. Upload Samples:
   • During the wizard, you will be prompted to upload a sample of the log file you wish to collect. This file is used to extract the structure of your log data.
   • Azure will parse the sample file to recommend the best timestamp format and record delimiter.
3. Define Record Structure:
   • Specify a regular expression to split the data into individual records if the auto-detection isn’t suitable.
   • Define the timestamp and its format to help Azure understand how to process the timing of the events.
4. Finalize and Create:
   • Name your custom log (postfix ‘_CL’ will be added automatically to denote a custom log).
   • Review the summary and create your custom log.

Monitoring Agent Configuration

Once the custom log is defined, it should be collected by the Azure Monitor agent:

• Determine which machines will send the custom log data.
• Install and configure the Log Analytics agent on those machines if not already present.
• Define the path where the agent can find the logs to be ingested.

Writing Queries for Custom Logs

Custom logs can be queried just like any other data type in Log Analytics. Here’s an example of how you might write a Kusto Query Language (KQL) query for your custom log data:

CustomLog_CL
| where TimeGenerated > ago(1d)
| where LogLevel == “Error”
| summarize count() by bin(TimeGenerated, 1h), Component
| render timechart

The above query counts the number of error-level logs from the CustomLog_CL custom log over the past day, grouped by hour and component, displaying the results in a timechart.

Security Considerations

As with any data collection, ensure compliance with data governance and security policies:

• Limit access to custom logs to authorized personnel.
• Consider the types of data being collected and avoid logging sensitive information without proper encryption and access controls.

Use Cases

Custom logs can be particularly beneficial in several scenarios, such as:

• Monitoring Custom Applications: Collect logs from custom-built applications that aren’t already supported by Azure Monitor.
• Compliance and Auditing: Track specific actions or events that are critical for compliance reporting.
• Advanced Security Scenarios: Capture detailed log data from security systems for incident investigation and analysis.

By leveraging custom logs in Azure Log Analytics, security operations analysts have a robust tool to extend their monitoring and analytics capabilities, ensuring they can effectively track and respond to potential security incidents in their organization’s IT environment.