A hands-on messaging lab in eighteen slides, in the console and the CLI: make the project bucket again; create a dead-letter queue, a work queue and an audit queue; create an SNS topic, subscribe both queues and see one message reach both; let S3 publish each upload under uploads/ to the topic; run workers in an Auto Scaling group that scales on the backlog per instance, upload 200 files and watch it grow; catch a poison message in the dead-letter queue; and clean up.
The slides
The tool is a hands-on lab in eighteen slides, one step each, with the steps to follow beside a terminal and a picture
of the bucket, the topic, the three queues with their messages, and the worker group:
- Introduction: what the lab makes.
- 1. Before we start
- Make the project bucket again: the S3 lab deleted it.
- 2. The queues
- Create the dead-letter queue:
project-thumbs-dlq, 14 days of retention. - Create the work queue project-thumbs: a 60-second visibility timeout, long polling, the DLQ after 3 receives.
- Create the queue project-audit from the CLI.
- 3. The topic
- Create the SNS topic project-uploads.
- Subscribe both queues to the topic, and read the queue policy it writes.
- Publish once, receive twice: SNS's envelope in the audit queue.
- 4. Uploads to the topic
- Let S3 publish to the topic: a statement in the topic's access policy.
- Add the event notification for uploads/.
- Upload a file and read its event: S3's event inside SNS's envelope.
- 5. The workers
- Create the workers' role and security group: receive and delete on one queue.
- Create the launch template with the worker: the worker loop in user data.
- Create the Auto Scaling group project-worker-asg: one worker, up to four, group metrics on.
- Add the scaling policy on the backlog: target tracking on metric math, 10 messages per worker.
- Upload 200 files and watch the fleet grow: from one worker to four.
- 6. A poison message
- Catch a poison message in the dead-letter queue: three receives, then the DLQ.
- 7. Clean up: the group, the template, the role, the topic, the queues and the bucket.
The buttons on the left and right edges move between slides. So do the arrow keys and a swipe.
The lab starts where
Getting Started with AWS ends: the CLI works as jane.
The slides
Messaging and Events explain what it uses.
The scaling policy
The workers scale by target tracking on the backlog per instance: CloudWatch metric math divides the queue's
ApproximateNumberOfMessagesVisible by the group's GroupInServiceInstances, and the policy keeps it near 10. It is the
EC2 Auto Scaling guide's recommended way, and the simpler correct one: metric math needs no program of ours to publish
a custom metric, while a step policy on the queue's length alone keeps adding workers as the queue drains, since that
number does not fall when workers are added. The console's target tracking offers only predefined metrics, so the lab
creates the policy from the CLI.
The placeholders
The account ID is AWS's documentation placeholder, 111122223333; the VPC, security group and message IDs and the
alarm names are AWS's placeholders too. The bucket is firstname-lastname-project: use your own name. The lab has not
yet been run in a real account: the console steps follow the AWS guides, and the outputs the CLI's service model.
Sources