A hands-on analytics lab in fourteen slides, in the console and the CLI: rebuild the project bucket and a small ALB with fixed responses; create a Firehose stream and deliver the ALB's access logs to it as JSON; send 20,000 requests and find the log files in S3; catalog them with a Glue crawler; count the requests by status with Athena and read the data scanned; copy the table to Parquet with CTAS and compare; clean up.
The slides
The tool is a hands-on lab in fourteen slides, one step each, with the steps to follow beside a terminal and a picture of
the pipeline, from the ALB to Athena:
- Introduction: what the lab does, part by part.
- 1. Rebuild
- Rebuild the bucket and a small ALB: fixed responses,
200 and 403.
- 2. Firehose
- Create the Firehose stream:
project-alb-logs, to logs/ in the project bucket. - Send the ALB's access logs to Firehose: a CloudWatch Logs delivery, as JSON.
- Send 20,000 requests.
- Find the log files in S3: one JSON line per request.
- 3. Glue
- Catalog the files with a Glue crawler: the table
project_logs.logs. - Read the table the crawler made: its columns and partitions.
- 4. Athena
- Set Athena's query result location.
- Count the requests with Athena: and the data scanned.
- 5. Parquet
- Copy the table to Parquet with CTAS.
- Count the requests on the Parquet table.
- Compare the data scanned.
- 6. Clean up
The buttons on the left and right edges move between slides. So do the arrow keys and a swipe.
The slides
Data Ingestion, Analytics, Migration and ML Services explain what it uses.
How the logs reach Firehose
An ALB's classic access logs, set on its Attributes tab, go to S3 only, as a compressed file every 5 minutes. Since
the CloudWatch Logs integration, an ALB can also deliver its access logs as vended logs to CloudWatch Logs, S3 or
Firehose, from its Integrations tab or with the CLI's put-delivery-source, put-delivery-destination and
create-delivery. The lab uses that path, with the log type ALB_ACCESS_LOGS and the JSON format, so the logs flow
through Firehose as the syllabus asks.
The placeholders
The account ID is 111122223333, 192.0.2.10 is our computer, and the ALB's DNS name and IDs are examples. The JSON
record's field names are the ones CloudWatch Logs lists for ALB_ACCESS_LOGS; its values, the file sizes and the data
scanned are examples of the expected order, not measurements.
Sources