in aws, DevOps

Centralized Log Management using AWS CloudWatch Logs Agent

True to AWS form, getting started is incredibly simple. Using the defaults for all of the installer questions, I was able to dump all of my log data from “/var/log/messages” to CloudWatch:

1. Create an EC2 role with the following perms:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:*",
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:logs:us-east-1:*:*",
        "arn:aws:s3:::*"
      ]
    }
  ]
}

2. Launch a Linux instance with the above role assigned.
3. Log on to the new instance and then download and install the client:

wget https://s3.amazonaws.com/aws-cloudwatch/downloads/awslogs-agent-setup-v1.0.py
sudo python ./awslogs-agent-setup-v1.0.py --region us-east-1

4. Select all the defaults.
5. Go to Cloudwatch in the AWS Console and select “Logs” in the left pane.
6. You should now see “/var/log/messages” in the “Log Groups” list.
7. Select this new log group and click “Create Metric Filter”
8. For our example, just put “Warning:” in the Filter Pattern box (With quotes).
9. Click “Assign Metric”
10. Give your metric a name (I chose “Warning”).
11. Click “Create Filter”

That’s it! You can now create alarms based on filters like this one. In its most basic form, this is very simple and I’m very excited about this new feature.