Tutorial / Cram Notes
Notebooks in Threat Hunting
A Jupyter notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations, and narrative text. For threat hunting, notebooks are useful for:
- Data Collection and Management: Hunters can aggregate data from various sources to build a comprehensive view of the potential threat.
- Investigation and Analysis: Notebooks allow for the execution of complex analysis using Python and other languages.
- Visualization: Data can be visualized through various graphs and tables for better understanding.
- Automation and Reproducibility: Hunters can automate repetitive tasks and reproduce their analysis seamlessly.
Using Notebooks in Azure Sentinel
Azure Sentinel integrates with notebooks to facilitate advanced hunting. Here’s a step-by-step example of performing hunting with a notebook in Azure Sentinel:
Data Extraction
Hunters can extract data using Azure Sentinel’s Kusto Query Language (KQL). For instance, a hunter might pull logs from Azure Activity to look for anomalies:
AzureActivity
| where ActivityStatus == “Success”
| summarize Count = count() by Bin(TimeGenerated, 1h), OperationName
| order by TimeGenerated desc
Data Exploration and Cleansing
Once the data is extracted, Python can be used within the notebook to clean and prepare the data for analysis. This might include dropping null values or normalizing data types.
import pandas as pd
# Load data into a DataFrame
data = pd.DataFrame(loaded_data)
# Drop null values
data.dropna(inplace=True)
# Normalize strings to lowercase
data[‘OperationName’] = data[‘OperationName’].str.lower()
Analysis and Pattern Identification
Notebooks support various statistical and machine learning libraries to identify patterns. For example, using a clustering approach to find uncommon activity:
from sklearn.cluster import KMeans
# Feature selection
features = data[[‘OperationName’, ‘Count’]]
# KMeans clustering
kmeans = KMeans(n_clusters=2)
data[‘cluster’] = kmeans.fit_predict(features)
Visualization for Insight
Matplotlib or similar libraries enable the creation of plots and charts to visualize the findings from the analysis:
import matplotlib.pyplot as plt
# Scatter plot of the clusters
plt.scatter(data[‘OperationName’], data[‘Count’], c=data[‘cluster’])
plt.xlabel(‘Operation Name’)
plt.ylabel(‘Count’)
plt.title(‘Cluster Analysis of Azure Activity Operations’)
plt.show()
Response and Remediation Actions
Based on the insights obtained, the hunter can then take action directly from the notebook or propose actions to be taken by the security operations center (SOC).
Comparison with Traditional Hunting Tools
| Feature | Notebooks | Traditional Tools |
|---|---|---|
| Interactivity | High (live code execution, dynamic results) | Varies (may be limited to static dashboards) |
| Customization | Extensive (custom code and libraries) | Limited (depends on the tool’s capabilities) |
| Reproducibility | High (version control, shareable notebooks) | Medium (scripts might not have version control) |
| Collaboration | High (shared notebooks through Azure Sentinel) | Medium (sharing requires exporting/importing configurations) |
| Visualization | Extensive (wide range of plotting libraries) | Varies (limited to tool-specific visualization features) |
| Analysis Depth | Deep (access to state-of-the-art algorithms) | Medium (limited to tool-specific algorithms) |
In preparation for the SC-200 exam, analysts should be comfortable with creating and executing notebooks within Azure Sentinel, understanding the versatility that notebooks offer in contrast to traditional security information and event management (SIEM) tools. Notebooks go beyond rigid, predefined search templates, offering cybersecurity professionals the flexibility to innovate and improvise in their hunting techniques.
Practice Test with Explanation
True or False: Hunting notebooks in Microsoft 365 Defender can be used to proactively search for security threats.
Answer: True
Explanation: Hunting notebooks in Microsoft 365 Defender enable security analysts to proactively hunt for potential security threats using queries, code, and visualizations.
Which component do you use to perform hunting in Microsoft Sentinel?
- A) Workbooks
- B) Playbooks
- C) Notebooks
- D) Dashboards
Answer: C) Notebooks
Explanation: In Microsoft Sentinel, analysts use Notebooks, which are based on Jupyter notebooks, to run complex queries, perform data analysis, and visualize findings for threat hunting.
True or False: When using a notebook for hunting, you can only use Python for scripting.
Answer: False
Explanation: While Python is commonly used in notebooks for scripting and data analysis, other languages such as PowerShell can also be used depending on the notebook’s environment.
Notebooks can be applied to which of the following data sources?
- A) Logs from Azure Active Directory
- B) Data from Microsoft Defender for Endpoint
- C) Alerts from Microsoft Defender for Identity
- D) All of the above
Answer: D) All of the above
Explanation: Hunting notebooks can analyze data from various sources, including Azure Active Directory logs, Microsoft Defender for Endpoint, and alerts from Microsoft Defender for Identity.
What does ‘KQL’ stand for in the context of hunting with notebooks?
- A) Key Question Language
- B) Knowledge Query Language
- C) Kusto Query Language
- D) Kernel Query Language
Answer: C) Kusto Query Language
Explanation: KQL stands for Kusto Query Language, which is used extensively for querying large datasets in Microsoft’s big data analytics platforms such as Azure Log Analytics and Microsoft Sentinel.
True or False: You can share hunting notebooks with other security analysts within your organization to collaborate on security investigations.
Answer: True
Explanation: Hunting notebooks can be shared with team members, allowing for collaboration on security investigations and threat hunting within an organization.
What is the primary benefit of using hunting notebooks?
- A) Automating incident response actions
- B) Simplifying data export procedures
- C) Performing interactive data analysis and visualization
- D) Generating compliance reports
Answer: C) Performing interactive data analysis and visualization
Explanation: The primary benefit of using hunting notebooks is to perform interactive data analysis and visualization, enabling security analysts to investigate threats more thoroughly.
Which one of the following is NOT a typical feature of a hunting notebook?
- A) Integrating with data sources
- B) Visualization libraries
- C) Pre-built query templates
- D) Phishing email blocking
Answer: D) Phishing email blocking
Explanation: Phishing email blocking is not a typical feature of a hunting notebook. Notebooks focus on data analysis and visualization, rather than direct preventive actions like blocking emails.
True or False: Hunting notebooks in Microsoft security solutions are only useful for large enterprises and have limited value for small or medium-sized organizations.
Answer: False
Explanation: Hunting notebooks are beneficial for organizations of all sizes as they provide a platform for proactive threat hunting and can be scaled according to the specific needs of any organization.
True or False: You must be proficient in advanced data science to use hunting notebooks effectively.
Answer: False
Explanation: While having data science skills is beneficial for making the most of hunting notebooks, Microsoft provides templates and guidance to assist security analysts who may not have advanced data science knowledge.
What should you use to automate the running of queries in hunting notebooks at regular intervals?
- A) Conditional Access Policies
- B) Azure Logic Apps
- C) Scheduled tasks in the operating system
- D) Azure Automation runbooks
Answer: B) Azure Logic Apps
Explanation: Azure Logic Apps can be used to automate the running of queries in hunting notebooks on a schedule, allowing for regular hunting activities without manual initiation.
True or False: Custom threat intelligence sources cannot be integrated into hunting notebooks.
Answer: False
Explanation: Hunting notebooks can integrate various threat intelligence sources, including custom sources, to enrich the data and improve the outcomes of threat-hunting activities.
Interview Questions
What is Microsoft Sentinel notebook?
Microsoft Sentinel notebook is a web-based tool that allows you to create and share interactive documents that contain code, visualizations, and explanatory text.
What is the purpose of using notebooks in Microsoft Sentinel?
Notebooks provide a flexible and powerful way to perform hunting, investigation, and visualization of data in Microsoft Sentinel.
What are the key features of Microsoft Sentinel notebooks?
Some key features of Microsoft Sentinel notebooks include support for Kusto Query Language (KQL), integration with the Microsoft Graph Security API, and the ability to create and share custom visualizations.
How do you create a new notebook in Microsoft Sentinel?
To create a new notebook in Microsoft Sentinel, you can click the “New” button on the Notebooks page and then choose a template or start with a blank notebook.
What is a KQL query, and how do you use it in a notebook?
KQL is a query language used in Microsoft Sentinel to search and analyze data. You can use KQL queries in a notebook by adding a new cell and entering your query in the appropriate format.
What is the Microsoft Graph Security API, and how does it work with notebooks?
The Microsoft Graph Security API is a RESTful API that provides access to security-related data across Microsoft products and services. Notebooks can use the Microsoft Graph Security API to retrieve and analyze this data.
How can you use custom visualizations in a notebook?
You can use custom visualizations in a notebook by creating a Python package that contains your visualization code, and then importing that package into your notebook.
How can you share a notebook with others in Microsoft Sentinel?
You can share a notebook with others in Microsoft Sentinel by clicking the “Share” button and selecting the appropriate options for sharing the notebook.
What are some common use cases for using notebooks in Microsoft Sentinel?
Some common use cases for using notebooks in Microsoft Sentinel include performing ad-hoc queries, investigating security incidents, and creating custom reports and visualizations.
What are some best practices for using notebooks in Microsoft Sentinel?
Some best practices for using notebooks in Microsoft Sentinel include using version control, keeping notebooks organized and easy to understand, and validating and testing your code before running it on live data.
Great blog post! Notebooks are incredibly powerful for threat hunting in the SC-200 exam.
Can anyone explain how Jupyter Notebooks are better compared to traditional SIEM solutions for threat hunting?
Can Jupyter Notebooks be used to automate repetitive tasks in threat hunting?
Thanks for sharing this! Really appreciate the insights on Jupyter Notebooks.
I found it difficult to set up a Jupyter Notebook environment. Any tips?
The practical examples in hunting provided here were very useful for my SC-200 preparation.
For those who have completed the SC-200 exam, how relevant were Jupyter Notebooks in your practical test?
Awesome content! Helped clarify many doubts about using Notebooks in security operations.