--- title: "A public/private data tier from a single pipeline" description: "The problem wasn't producing a database export: a classic ETL pipeline already knows how to do that. The problem was that this same export had to exist in two…" date: "2026-06-25T09:00:00.000Z" updated: "2026-08-24T15:05:22.393Z" locale: "en" canonical: "https://ninhache.fr/en/blog/tiers-donnees-public-prive" author: "Néo Almeida" tags: "architecture, data, etl" categories: "dev" --- # A public/private data tier from a single pipeline The problem wasn't producing a database export: a classic ETL pipeline already knows how to do that. The problem was that this same export had to exist in two distinct forms, one that anyone can clone and run locally, another restricted to a handful of authorized people, without duplicating the extraction logic or maintaining two pipelines that drift apart over time. The concrete context: an ETL that periodically dumps the state of a live game into a Postgres database (effects, spells, items, and other data tables derived from a binary source format). A web-app then consumes this dump to populate its own development database. A seemingly simple question: how do you distribute this dump to external developers who contribute to the project, without exposing certain categories of data not yet ready to be public? ## Two flavors, one source dump The solution that emerged isn't two separate pipelines but a single complete extraction, followed by a derivation. **Full dump** The pipeline always dumps the entirety of the data into a first file: this is the private flavor, complete, unfiltered. **Filtering** From this same file, a filter applies an allowlist and produces a second file: the public flavor. The important point is the direction of the filtering. Concretely, that means when a new category of items appears in the dump (a new type of cosmetic, a new content set), it doesn't automatically land in the public flavor. **Blacklist** The list says "here's what needs to be removed". An oversight leaks by default: new, unlisted content ships automatically. **Allowlist (chosen)** The list says "here's what's allowed out". Everything else is excluded by default: new content stays absent until someone has explicitly added it. That's the direction of error you want when the stakes are not publishing unfinished content. The filter itself remains an implementation detail: a script that reads the dump line by line and only lets through lines whose identifier belongs to the list, for the tables concerned. The rest of the dump (schema, sequences, non-sensitive tables) passes through unmodified. What matters in the pattern isn't this particular script but the idea that filtering happens downstream of a single source dump, never by duplicating the extraction query. ## Releases as the poor developer's CDN Once the two files are produced, they need to be distributed. No need for a dedicated CDN or an object storage bucket: GitHub releases do the job perfectly well for this volume and publishing frequency. Each release is tagged with the data source version, and carries a single asset (the compressed dump). The distinction between the two flavors plays out on two axes: the target repository and a tag suffix. **Public flavor** Tag `data-v`, main repository `/`, open access to everyone. **Private flavor** Tag `data-v-private`, restricted repository `/`, marked as prerelease. The prerelease flag isn't cosmetic: it signals that this release isn't meant for general consumption, even for someone who might have access to the repository by mistake or curiosity. It's a second barrier, independent of the repository's own access control. ## The contract on the consumer side On the web-app side, the command that fetches these dumps to populate a local database doesn't know the notion of "confidential" as such. It knows a flavor and a repository, both supplied via environment variables, with default values that point to the public case: ```bash title=".env (web-app)" SEED_FLAVOR=public # or "private" SEED_REPO=/ # or the restricted repo ``` Resolving the right release then amounts to querying the target repository's releases API and filtering by a tag pattern matching the requested flavor (ending in the suffix for private, not ending in it for public). An external developer who provides no variable gets, by construction, the public flavor of the public repository. A developer authorized on the private tier must explicitly point to the other repository and the other flavor, generally via an authentication token that only exists for the people who need it. What makes this contract clean is that it doesn't rely on any hidden client-side logic: the entire authorization decision lives in "who has access to which repository and which token", not in the seed command's code. The resolution code is identical for both tiers, only the configuration changes. ## Why separate rather than open everything The natural temptation would be to publish everything and let each person filter what interests them client-side. That doesn't work here because the reason for exclusion isn't a volume or relevance problem, it's a content maturity problem. A database derived from raw game files inevitably contains artifacts that were never meant to ship: content still being finalized, test entries left by internal development tools, elements removed but not yet cleaned up from the source. None of that has any value for an external contributor. **Warning** Prematurely publishing one of these elements can have consequences (an unwanted announcement, incomplete content visible ahead of time) that no client-side filtering can fix after the fact. ## The general lesson **Lesson** This pattern goes well beyond the case of a game dump. Whenever a pipeline produces derived data of which part must remain restricted, the same structure applies: a single source extraction, filtering by an allowlist that's closed by default rather than open by default, versioned distribution that serves as both channel and history, and a prerelease flag or equivalent that marks the internal tier independently of the repository's access control. The contract between producer and consumer then reduces to a handful of environment variables (which flavor, which repository, which tag suffix), which makes it testable and reusable without ever coupling the consumer's code to the notion of confidentiality itself.