Tutorial / Cram Notes
Service and application logging are critical components of monitoring, security, and compliance in the AWS ecosystem. AWS CloudTrail and CloudWatch Logs are two powerful services that help in tracking user activity and API usage, as well as in collecting, monitoring, and storing log files, respectively. Configuring these services effectively is crucial for an AWS Certified DevOps Engineer – Professional.
AWS CloudTrail
AWS CloudTrail is a service that enables governance, compliance, operational auditing, and risk auditing of your AWS account. By default, CloudTrail logs all account activity across your AWS infrastructure, allowing for detailed auditing and analysis.
Setting Up AWS CloudTrail
- Log in to the AWS Management Console and navigate to the CloudTrail service.
- Create a new trail and configure the trail settings:
- Trail Name: Assign a name to your trail.
- Apply Trail to All Regions: Decide whether the trail should apply to all regions or just the region you’re currently in.
- Read/Write Events: Choose to log read-only events, write-only events, or all.
- Choose a target S3 bucket for log file delivery. Ensure that the bucket has the proper policies for CloudTrail to write logs.
- Optionally, configure SNS notifications to notify you whenever log files are delivered to the S3 bucket.
- If you need to capture more detailed information about the Lambda functions, S3 buckets, etc., enable data event logging.
- Enable log file encryption using KMS keys to secure access to the log information.
Best Practices with CloudTrail
- Enable CloudTrail across all AWS regions to ensure activities from any region are logged.
- Use CloudTrail log file integrity validation to ensure your logs have not been tampered with.
- Regularly review and rotate your KMS keys for encryption purposes, adhering to strict IAM policies and roles.
AWS CloudWatch Logs
AWS CloudWatch Logs allows you to monitor, store, and access your log files from Amazon EC2 instances, AWS CloudTrail, and other sources. With CloudWatch Logs, you can create alarms and visualizations, export data, and perform real-time analysis of log data.
Setting Up AWS CloudWatch Logs
- Open the AWS Management Console and access the CloudWatch service.
- Navigate to Logs and then select ‘Create log group’:
- Log Group Name: Provide a name that is indicative of the type of logs being stored.
- Assign retention policies based on how long you need the logs to be preserved (e.g., 30 days, 90 days, indefinitely).
- To publish logs to this group, use the AWS CLI or CloudWatch Logs SDK to send log data. This requires installing and configuring the CloudWatch Logs agent or using the unified CloudWatch agent.
Example of a log stream creation using AWS CLI:
aws logs create-log-stream –log-group-name “MyLogGroup” –log-stream-name “MyLogStream”
Monitoring Application Logs
- To monitor application logs, you can either manually send log data from your EC2 instances or leverage existing automation through IAM roles and CloudWatch agent.
- For Lambda functions, logs are automatically sent to CloudWatch Logs under the `/aws/lambda/function-name` log group.
Best Practices with CloudWatch Logs
- Create Metric Filters to turn log data into numerical CloudWatch metrics that you can graph and set alarms on.
- Use CloudWatch Logs Insights for interactive log analytics and querying capabilities.
- Structure your logs with JSON formatting to enhance search and filtering capabilities in CloudWatch Logs Insights.
Comparison of CloudTrail and CloudWatch Logs
| Feature | CloudTrail | CloudWatch Logs |
|---|---|---|
| Primary Use Case | Auditing AWS account activity | Monitoring & storing log data |
| Default Logging | Enabled by default; tracks API calls | Must be configured to receive logs |
| Data Sources | AWS Management Console, SDKs, CLI, etc. | EC2 Instances, CloudTrail, Lambda, etc. |
| Log Storage & Analysis | Stores logs in S3, analysis outside | Provides log storage & analysis tools |
| Real-time Monitoring | No real-time; near-real-time logs in S3 | Supports real-time log stream monitoring |
| Encryption | Supports encryption with KMS | Supports encryption with KMS |
| Custom Log Data | Not designed for custom log data | Can accept custom log data |
| Integrated Services | Integrated with other AWS services for tracking | Integrated with metrics, alarms, and dashboard |
In summary, while CloudTrail is tailored specifically for auditing and tracking API and console actions across AWS services, CloudWatch Logs focuses on generic log storage and analysis, offering powerful features for real-time monitoring and insights into your infrastructure and applications.
AWS Certified DevOps Engineers should be skilled in setting up and managing both CloudTrail and CloudWatch Logs to maintain robust monitoring and ensure operational excellence. Hands-on experience and a solid understanding of these services’ capabilities are crucial for the DOP-C02 exam.
Practice Test with Explanation
True or False: AWS CloudTrail is used to monitor and record user activity and API usage across your AWS infrastructure.
- True
AWS CloudTrail is a service designed to enable governance, compliance, operational auditing, and risk auditing of your AWS account through logging and monitoring of account activity.
True or False: CloudWatch Logs can trigger a Lambda function when a specific message is logged.
- True
CloudWatch Logs can be configured to invoke a Lambda function using CloudWatch Logs triggers when a specific pattern is matched in the logs.
Which of the following AWS services can be used to collect and process logs from multiple sources like EC2 instances and AWS Lambda functions?
- A) AWS Config
- B) Amazon Kinesis
- C) AWS CloudTrail
- D) Amazon CloudWatch Logs
Answer: D) Amazon CloudWatch Logs
Amazon CloudWatch Logs enables you to centralize the logs from all your systems, applications, and AWS services in a single, highly scalable service.
Which AWS service provides a history of AWS API calls for your account?
- A) Amazon CloudWatch
- B) AWS X-Ray
- C) AWS CloudTrail
- D) AWS Config
Answer: C) AWS CloudTrail
AWS CloudTrail is a service that provides a record of actions taken by a user, role, or an AWS service in the account.
True or False: When using AWS CloudWatch Logs, log data can be exported to Amazon S3 for long-term archival.
- True
AWS CloudWatch Logs supports exporting log data to Amazon S3 buckets for archival purposes.
True or False: AWS Config rules can be used to continuously evaluate AWS resource configurations and automatically take action when non-compliant changes occur.
- True
AWS Config rules provide automatic checks on the configuration of AWS resources within your account to ensure compliance with desired configurations.
To make AWS CloudTrail log file delivery faster, you can:
- A) Enable log file integrity validation
- B) Create a new trail for each region
- C) Increase the frequency of log file delivery settings
- D) None of the above
Answer: D) None of the above
Delivery speed for AWS CloudTrail log files is not configurable; AWS manages the delivery frequency. Log integrity validation is for ensuring log files are not tampered with, and creating a new trail increases coverage, not delivery speed.
Which AWS service helps you gain system-wide visibility into application performance by providing a dashboard for monitoring Amazon CloudWatch metrics and logs?
- A) AWS CloudTrail
- B) Amazon CloudWatch
- C) AWS X-Ray
- D) AWS CodePipeline
Answer: B) Amazon CloudWatch
Amazon CloudWatch provides a platform for monitoring and operational control, with a dashboard for visualization of metrics and logs.
True or False: AWS CloudTrail cannot log data events for S3 object-level activities such as GetObject and DeleteObject.
- False
AWS CloudTrail does support logging of data events, which can be configured to include S3 object-level activities like GetObject and DeleteObject.
What is the purpose of AWS CloudTrail log file validation feature?
- A) To ensure the integrity of the log files.
- B) To verify that the log files have been delivered to the correct S3 bucket.
- C) To encrypt log files.
- D) To make the log file delivery faster.
Answer: A) To ensure the integrity of the log files.
AWS CloudTrail log file validation is a feature that creates a hash for every log file delivered to your S3 bucket and a signed digest file that can be used to verify the integrity of the log files.
Which of the following IAM policies allows a user to fully manage CloudWatch Logs?
- A) CloudWatchLogsReadOnlyAccess
- B) CloudWatchLogsFullAccess
- C) CloudTrailFullAccess
- D) CloudWatchReadOnlyAccess
Answer: B) CloudWatchLogsFullAccess
CloudWatchLogsFullAccess policy grants permissions to access all CloudWatch Logs capabilities to create, configure, and manage logs.
You can use AWS CloudTrail to receive a notification when a specific API activity occurs?
- A) True, by integrating with Amazon SNS.
- B) False, CloudTrail only stores log files in S3 and does not provide notifications.
Answer: A) True, by integrating with Amazon SNS.
AWS CloudTrail can be integrated with Amazon Simple Notification Service (SNS) to provide notifications when new log files are delivered, and further custom alerts can be configured using CloudWatch or AWS Lambda.
Interview Questions
What are the main differences between AWS CloudTrail and AWS CloudWatch Logs?
AWS CloudTrail is a service that provides a record of actions taken by a user, role, or AWS service in AWS. It’s primarily used for auditing and governance. AWS CloudWatch Logs, on the other hand, is a monitoring service for logging and real-time monitoring of events and logs from AWS resources, applications, and services. While CloudTrail focuses on who did what action within AWS, CloudWatch Logs focuses on logging application and system log data.
How can you secure your CloudTrail log files?
You can secure AWS CloudTrail log files by enabling log file encryption using AWS Key Management Service (KMS) keys. Additionally, you should store the logs in a dedicated S3 bucket with the proper access policies, versioning, and MFA Delete enabled to prevent unauthorized access or accidental deletion.
How would you set up CloudWatch Logs to monitor the logs of a specific EC2 instance or a group of instances?
To set up CloudWatch Logs for monitoring EC2 instances, you would install and configure the CloudWatch Logs agent on the desired EC2 instances to send logs to CloudWatch. Then, create Log Groups and define Log Streams for the organized collection and analysis of log data.
Can you set up alarms in CloudWatch to notify you of specific events in your application logs? If so, how?
Yes, you can create metric filters in CloudWatch Logs to search for and match terms, phrases, or values in your logs. After creating a metric filter, you can set up a CloudWatch Alarm based on the metric filter. When the defined metric threshold is breached, the alarm can send notifications via Amazon SNS or take other automated actions.
How do you grant an IAM user access to CloudWatch Logs?
To grant an IAM user access to CloudWatch Logs, you need to create and attach an IAM policy that defines the appropriate permissions for CloudWatch Logs actions, such as logs:PutLogEvents, logs:CreateLogStream, and logs:DescribeLogStreams, and associate it with the IAM user or group.
How can you forward logs from CloudWatch to an external system, like a SIEM or log analysis tool?
You can forward logs from CloudWatch to an external system by exporting the log data to an Amazon S3 bucket and then using an AWS Lambda function or another processing system to ingest the logs into the SIEM or log analysis tool. Alternatively, you can set up a subscription filter in CloudWatch Logs to stream the data directly to Amazon Kinesis Data Firehose, which can then forward the logs to the desired tool.
How does AWS CloudTrail integrate with AWS CloudWatch?
AWS CloudTrail integrates with AWS CloudWatch by delivering log files to a CloudWatch Logs log group, which allows you to set real-time monitoring, alarms, and access to historical audit data within CloudWatch.
What is the retention period for logs in CloudWatch logs and how can it be configured?
Log data in CloudWatch Logs can be retained indefinitely or on a custom schedule, which is user-configurable. You can set the retention period for each log group to a specific duration such as 1 day, 1 week, 1 month, etc., or to “never expire,” using either the AWS Management Console, AWS CLI, or CloudWatch API.
Explain how you would monitor AWS Lambda function execution logs using CloudWatch.
To monitor AWS Lambda function execution logs, you can enable logging within the Lambda function’s configuration. The logs generated during execution will automatically be sent to CloudWatch Logs, where you can view, search, and apply metric filters for monitoring and alerting purposes.
How can you ensure that log data is not lost if a logging failure occurs in an EC2 instance?
To ensure log data is not lost, it is important to store log files on persistent storage like EBS or stream them directly to CloudWatch Logs. Additionally, implementing a robust log rotation and backup strategy is crucial. The CloudWatch Logs agent can also be configured to watch and send log files to CloudWatch Logs in near-real-time to mitigate the risk of data loss.
Describe a scenario where CloudTrail Multi-Region logging would be beneficial.
CloudTrail Multi-Region logging would be beneficial for organizations that operate in multiple AWS regions and require centralized logging of API activity across those regions for simplified compliance auditing, security analysis, and global resource tracking.
Is it possible to change the log level of a running application without redeploying it? How would you achieve that using AWS CloudWatch?
Yes, it’s possible to change the log level of a running application without redeployment by using environment variables or configuration files that the application reads at runtime, which can be updated dynamically. With AWS Systems Manager Parameter Store or AppConfig, you can manage these configurations centrally and dynamically adjust log levels that your application periodically checks. CloudWatch Logs will then collect logs according to the updated settings.
Great blog post! Helped me understand the basics of CloudTrail and CloudWatch Logs.
Can someone explain the difference between CloudTrail and CloudWatch Logs?
Appreciate the step-by-step guide, thanks!
Having issues with setting up CloudWatch Log streams. Any advice?
I used this post to set up my application logging, and it’s working perfectly!
Are there any cost implications of extensive logging?
Could someone help me understand VPC flow logs in CloudWatch?
How often should one review CloudTrail logs?