I recently reviewed my classics about AWS Kinesis services and I wanted to share this digest with you. So, no more boring introduction stuff, let's start!

Kinesis Data Stream

🛠 Workflow:

• Producer (Kinesis Producer Library, API via SDK)
• Kinesis Data Stream
• Consumer (EC2, Lambda, EMR, Kinesis Data Analytics)
• Storage/Analysation (S3, DynamoDB, Redshift, BI Tools…)

💡 Benefits

• Data retention
• Real-time

🧪 Use cases

• Analyze logs in real-time
• Transform real-time streaming data and fed it to a custom ML application

Kinesis Data Firehose

🛠 Workflow

• Producer
• Kinesis Data Firehose
• Processing tool (optional: Lambda)
• Storage (S3, Redshift…)
• Additional steps (send on event from S3 to DynamoDB table)

💡 Benefits

• Collect streaming data and send to data store

🧪 Use cases

• Delivery service for streaming data
• Apache logs from EC2 instance to S3 or Redshift
• Streaming data from IoT devices to data lake

Kinesis Video Stream

🛠 Workflow

• Producer
• Kinesis Video Stream
• Consumer (EC2 continuous/Batch consumer program)
• Storage (S3…) or other service

💡 Benefits

• Real-time streaming of video data (images, audio, radar…)
• Batch process and store streaming video data
• Feed video data to other AWS services

🧪 Use cases

• Stream event video coverage to customers
• Ingest data for ML applications

Kinesis Data Analytics

🛠 Workflow

• Input (Kinesis Data Stream, Kinesis Data Firehose)
• Kinesis Data Analytics
• Storage (S3, Redshift…) / Visual tools (Quicksight, ELK…)

💡 Benefits

• Run SQL queries on streaming data and output to S3
• Create dashboards
• Metrics
• Alarms

🧪 Use cases

• Query real-time data
• Enrich data for ETL jobs
• Responsive real-time analytics
• Metric graphs

Conclusion

AWS Kinesis services can be confusing at the beginning but I hope this cheat sheet is going to help you understand and memorize to make the best out of these services.

Don’t forget to react and to share this article if you liked it! Feel free to reach me on Twitter (@FlolightC) to tell me about your Kinesis use case or to ask me questions ! I’m always happy to discuss with you !