Design App that Downloads User Data

You're asked to design a "download your information" feature like Facebook's data export. A user requests everything the platform holds about them and later receives a link to a downloadable archive. The system must gather data from many services, support accounts ranging from a few megabytes to hundreds of gigabytes, survive partial failures, and show useful progress.

Accepting the request takes milliseconds, but building the archive can take hours. We'll return a job ID immediately and move extraction, assembly, and notification work to the background.

Clarify the requirements

• What's included? Posts, photos, messages, comments, likes, ad interactions, login history. Each lives in a different service with a different API and a different volume profile.
• What's the deadline? Regulations typically allow days rather than minutes, which makes a fully asynchronous design appropriate.
• What format? A zip containing JSON or HTML per category is typical. Format matters less than the fact that it's one archive assembled from many sources.
• How large can an export get? Ask explicitly. A heavy user with fifteen years of photos may need hundreds of gigabytes, which affects chunking and storage.
• How sensitive is the archive? It contains every piece of data about one person in a single file, so access control and link expiration are functional requirements.

Back-of-envelope numbers

• Request rate: $3\text{B users} \times 0.01\% \approx 300\text{k exports/month} \approx 0.1/\text{sec}$, which is trivial
• Archive volume: $300\text{k exports} \times 1\text{ GB avg} \approx 300\text{ TB/month}$, with heavy accounts well above the average
• At a 30-day retention window, steady-state storage is roughly 300 TB

The request rate is negligible. Optimize the system for reliable, resumable work rather than low-latency completion.

High-level architecture

Deep dive 1: parent and child jobs

Do not treat the entire export as one job. If a monolithic job fails at 90%, it must start over, and a large account may never finish before another failure occurs.

Have the parent fan out into one child job per data category and derive the parent's state from its children. A failure in photo extraction then retries only the photos. Twelve completed children out of fifteen gives the user measurable progress. Categories also run in parallel, so total time depends on the slowest category rather than the sum of all extraction times.

For very large categories, children subdivide further. Fifteen years of photos becomes one child job per time range, each writing its own chunk to object storage.

Deep dive 2: surviving failures partway through

Each child job must be independently retryable and idempotent. A repeated child should overwrite its output chunk rather than append to it, so a retry produces the same archive as the first attempt. Key chunks by (job_id, category, range) to make overwriting the default behavior.

Use the rest of the job machinery as well:

• Leases with heartbeats, so a worker that dies mid-extraction releases its child job for another worker instead of stranding it.
• Classified retries. A rate-limit response from a source service is retryable with backoff; a permanently deleted account is not.
• A checkpoint per child, so a category that is 80% extracted resumes near that point instead of starting over.
• Per-source rate limiting. The photo service was not built to serve one customer's entire history as fast as possible. Throttle extraction so an export never degrades the live product.

Deep dive 3: assembly and delivery

After every child succeeds, the finalizer packages the chunks. Stream them from object storage into a zip file written back to object storage rather than loading the archive into memory. A 200 GB export then uses the same amount of memory as a 200 MB export.

Secure the delivery path with the following controls:

• Time-limited signed URLs. The link expires, typically in a few days, and is single-use or tied to an authenticated session.
• Re-authentication before download. The archive is every piece of data about a person, so requiring a fresh login before handing it over is proportionate.
• Automatic deletion. Archives are deleted after the retention window, which caps both storage cost and the blast radius if a link leaks.
• Encryption at rest, and optionally a password-protected archive.

Deep dive 4: reporting progress to the user

The parent-child structure makes progress reporting straightforward. Calculate completion from the fraction of child jobs that are done, optionally weighted by expected size so photos count more than login history. Store the result in the job record and let clients poll for updates.

For a very large account, final packaging may become the bottleneck, which is why the finalizer streams chunks. A source service may also impose enough rate limiting to cause hours of backoff, so throttle each source deliberately. Schedule archive deletion so abandoned exports do not accumulate indefinitely.

Rate-limit export requests per account. Repeated requests consume substantial resources. Return a recently generated archive when possible instead of rebuilding the same export.

Common pitfalls

• Treating the export as a single job. A failure anywhere means starting from zero, and progress becomes unreportable.
• Holding the archive in memory. Works for a test account, fails for a real one.
• No idempotency on child jobs. A retry produces a corrupted archive with duplicated content.
• Ignoring source rate limits. One user's export degrades the live product for everyone else.
• Permanent download links. An unexpiring URL to a complete personal data archive is a serious exposure.

Leveling signals

Related lessons