An API source returns data page by page. What must your ingestion handle?
Why D is the answer
Pagination and rate limits
Ingesting from APIs means managing pages and limits.
In detail
- An API source needs pagination and rate limits handled.
- Save progress, so a failed run can resume.
Use case
An illustrative scenario, not a real incident.
A partner API returns 100 rows at a time, with a rate limit. The job follows pages and waits between calls.
See it in different ways
mindmap
root((Ingestion))
Streams
Kinesis Data Streams
Amazon MSK
Delivery
Data Firehose
Databases
AWS DMS
Files
S3 eventssequenceDiagram participant P0 as Ingestion job participant P1 as Partner API P0->>P1: Page 1 P1-->>P0: Rows and next token P0->>P1: Page 2 P1-->>P0: Rate limit, wait
Key point: API ingestion handles pages and limits.
Why the other options are wrong
- A. Compaction of each response
- Compaction concerns stored files.
- B. Encryption with a new key per page
- Per-page keys add nothing.
- C. A separate Region for each page
- Regions are unrelated to paging.