Skip to main content

Hands-On: Logs, Retention, and a CPU Alarm

What We Are Building​

Three things every production account needs on day one: fast access to Lambda and EC2 logs from the terminal, retention policies so logs stop accumulating forever, and an alarm that emails me when an instance's CPU pins. All CLI, all verifiable.

I assume a Lambda function named order-processor and an EC2 instance i-0abc123def456789a already exist — swap in your own names.

Step 1: Find the Log Groups​

aws logs describe-log-groups \
  --query "logGroups[].[logGroupName,storedBytes]" --output table

Lambda groups follow the /aws/lambda/function-name convention. EC2 groups only exist if the CloudWatch agent is shipping them; mine ships nginx logs to /ec2/nginx/access.

Step 2: Tail Lambda Logs Live​

The single most useful command in this whole page:

aws logs tail /aws/lambda/order-processor --follow

Invoke the function in another terminal and watch output stream in:

aws lambda invoke \
  --function-name order-processor \
  --payload '{"orderId": 42}' \
  --cli-binary-format raw-in-base64-out /dev/null
2026-07-06T10:14:02 START RequestId: 8f2e... Version: $LATEST
2026-07-06T10:14:02 INFO processing order 42
2026-07-06T10:14:03 REPORT RequestId: 8f2e...  Duration: 812 ms  Max Memory Used: 91 MB

For a historical slice instead of a live tail:

aws logs tail /aws/lambda/order-processor --since 2h --filter-pattern "ERROR"

--filter-pattern runs server-side, so it stays fast even on large groups. The same command works on any group, including the EC2 ones:

aws logs tail /ec2/nginx/access --since 30m --filter-pattern '" 500 "'

Step 3: Set Retention on Every Log Group​

New log groups default to Never expire, which quietly becomes a real line item at $0.03 per GB-month. Set an explicit policy:

aws logs put-retention-policy \
  --log-group-name /aws/lambda/order-processor \
  --retention-in-days 30

aws logs put-retention-policy \
  --log-group-name /ec2/nginx/access \
  --retention-in-days 14

Valid values are fixed steps: 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365 and up. My defaults: 14 days for chatty access logs, 30 for application logs, 90 or more only where compliance demands it.

Audit for groups that slipped through:

aws logs describe-log-groups \
  --query "logGroups[?retentionInDays==null].logGroupName" --output text

Empty output means every group has a policy.

Step 4: Create the SNS Topic for Notifications​

Alarms do not send email themselves — they publish to an SNS topic, and the topic fans out to subscribers.

aws sns create-topic --name ops-alerts

aws sns subscribe \
  --topic-arn arn:aws:sns:us-east-1:123456789012:ops-alerts \
  --protocol email \
  --notification-endpoint rizwanashi215@gmail.com

SNS sends a confirmation email with a link. The subscription delivers nothing until that link is clicked — forgetting this is the classic "my alarm never fired" bug. Verify:

aws sns list-subscriptions-by-topic \
  --topic-arn arn:aws:sns:us-east-1:123456789012:ops-alerts \
  --query "Subscriptions[].SubscriptionArn"

A real ARN means confirmed; PendingConfirmation means go check the inbox.

Step 5: Create the CPU Alarm​

Alert when the instance averages above 80 percent CPU for two consecutive 5-minute periods — sustained load, not a brief spike:

aws cloudwatch put-metric-alarm \
  --alarm-name ec2-high-cpu-api-server \
  --alarm-description "API server CPU above 80% for 10 minutes" \
  --namespace AWS/EC2 \
  --metric-name CPUUtilization \
  --dimensions Name=InstanceId,Value=i-0abc123def456789a \
  --statistic Average \
  --period 300 \
  --evaluation-periods 2 \
  --threshold 80 \
  --comparison-operator GreaterThanThreshold \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:ops-alerts \
  --ok-actions arn:aws:sns:us-east-1:123456789012:ops-alerts \
  --treat-missing-data breaching

Choices worth explaining:

  • --evaluation-periods 2 with --period 300 means 10 minutes of sustained breach before paging — deploys and cron spikes stay quiet.
  • --ok-actions sends a recovery email too, so I know when it resolved without logging in.
  • --treat-missing-data breaching makes a stopped or unreachable instance count as an alert. For CPU that is what I want; for spiky custom metrics choose notBreaching.

The alarm starts in INSUFFICIENT_DATA and settles to OK after a couple of periods:

aws cloudwatch describe-alarms --alarm-names ec2-high-cpu-api-server \
  --query "MetricAlarms[0].StateValue"

Step 6: Test the Alarm Without Burning CPU​

Force the state transition instead of running a stress tool:

aws cloudwatch set-alarm-state \
  --alarm-name ec2-high-cpu-api-server \
  --state-value ALARM \
  --state-reason "manual test of the notification path"

An email lands within a minute: subject ALARM: "ec2-high-cpu-api-server" in US East (N. Virginia). CloudWatch re-evaluates against real metrics on the next period and flips back to OK on its own — which conveniently also tests the recovery email.

If nothing arrives, the checklist is short: subscription still pending confirmation, wrong topic ARN in --alarm-actions, or the message went to spam.

Recap​

  • aws logs tail --follow replaces clicking through log streams in the console
  • Every log group gets an explicit retention policy the day it is created
  • Alarm plus SNS topic plus a confirmed email subscription is the minimum viable alerting stack
  • set-alarm-state lets you test the whole notification path in seconds
Last updated on