You need serverless Spark ETL with no cluster to manage. Which service fits?
Why A is the answer
AWS Glue
Glue runs Spark jobs serverlessly.
In detail
- Glue gives serverless Spark.
- You pay while jobs run.
Use case
An illustrative scenario, not a real incident.
A small team runs Spark ETL on Glue, with no clusters to size or patch.
See it in different ways
flowchart TD
A{"Transform job"} -->|"Small, event-driven"| B["AWS Lambda"]
A -->|"Serverless Spark ETL"| C["AWS Glue"]
A -->|"Big clusters you tune"| D["Amazon EMR"]mindmap
root((Transform))
Glue
Serverless Spark
Job bookmarks
EMR
Custom tuning
Lambda
Small files
DataBrew
No code
Redshift
Stored proceduresflowchart TD
Q{"What do you need?"}
Q -->|"No cluster to manage"| A0["AWS Glue"]
Q -->|"Custom tuning"| A1["Amazon EMR"]Key point: Glue is serverless Spark ETL.
Why the other options are wrong
- B. Amazon EMR on EC2, with node groups that you size and patch yourself
- EMR on EC2 has clusters to manage.
- C. A self-managed Hadoop cluster running on EC2 instances
- Self-managed Hadoop is the opposite.
- D. An AWS Lambda function with a very long timeout per file
- Lambda is not Spark.