Let users export their data
Build an account's export as a ZIP in a background job, keep it in R2, and email the person a link once it is ready.
Last updated
After reading this guide, you will know
- What guides build a data export, and in what order to follow them
- How one job assembles the archive, stores it and emails its link
- Where a file too large for a Worker goes in the export
At the end of this recipe a person asks for their data, carries on using your app, and gets an email with a download link once a ZIP of their account, their contacts as a CSV and their stored files is ready.
npm add @sdxc/jobs @sdxc/resultAssemble the archive
The archive is what the job builds, so write it first. Follow
Stream an export for the paged
contactRows generator, then Add the entries
and Open stored files when the archive reaches them
for accountArchive.
Keep the file in R2
The finished archive outlives the job that wrote it, so it goes in a bucket. Open an
accountExports() storage over an EXPORTS bucket the way
Crop it square and store a clean copy
opens avatars(), and serve exports/<account id>.zip to the signed-in account as
Serve either one does.
Build it in a job
Follow Send email through
Publish a mailer to requests and jobs
for an exports.ready email whose one prop is the download url, and
Background jobs and cron for an exports.build
job with an accountId input, which the export action enqueues before it redirects. The job
already runs off the request, so it sends the email itself, and a failed send retries the run:
import { createJobHandler } from "@sdxc/jobs";
import { isFailure } from "@sdxc/result";
import emails from "~/app/emails";
import jobs from "~/app/jobs";
import { accountExports } from "~/app/models/account-exports";
import { Accounts } from "~/app/models/accounts";
import { accountArchive } from "~/app/services/exports/account-archive";
import { SITE_URL } from "~/config/constants";
import routes from "~/routes/web";
export default createJobHandler(jobs.exports.build, async (ctx) => {
let account = await Accounts.find(ctx.db, ctx.input.accountId);
if (account === null) return ctx.exit("The account was removed");
let archive = await accountArchive(ctx.db, account);
if (isFailure(archive)) return ctx.exit(archive.error.message);
let bytes = await archive.data.bytes();
if (isFailure(bytes))
return ctx.retry({ delay: "5 minutes", cause: bytes.error });
let file = new File([bytes.data.slice()], `${account.slug}-export.zip`, {
type: "application/zip",
});
await accountExports().set(`exports/${account.id}.zip`, file);
let url = new URL(routes.account.download.href(), SITE_URL);
let sent = await ctx.email.send(emails.exports.ready, {
to: account.email,
locale: account.locale,
props: { url: url.href },
});
if (isFailure(sent)) return ctx.retry({ cause: sent.error });
});Test it
Test each stage as its guide does: Test the archive checks the archive's bytes, and Run jobs in a test drains the job with the memory transport from Test with the memory transport.
Where to go next
Stream several files as a ZIP — what happens when a stored file fails while the archive is written.
Stream JSON Lines — an export format for records with nested data.
Logs, traces and timings — reading the record each export run emits.