One flowgroup, every environment¶

A Lakehouse Plumber flowgroup is written once and stays identical across dev, staging, and prod. The logic of a pipeline — read this source, reshape these rows, write that table — does not change when you promote it. What changes is the surroundings: which catalog it writes to, which host it reads from, which credential it uses. Keeping those two things apart is the whole point of substitutions. Everything environment-specific is a token, and every token resolves from a per-environment file when you generate. The flowgroup you commit carries the shape of the pipeline; the substitution file carries the environment.

This is the same idea that runs through the rest of Lakehouse Plumber — declare your ETL, don’t hand-write it — applied to promotion: declare the values that vary, don’t hand-copy the pipeline for each place it runs.

From definition to runtime¶

During flowgroup resolution, LHP resolves local variables, expands template actions and applies presets, then substitutes environment values. Secret references become lookup calls in the generated Python. The calls fetch values when that code runs on Databricks.

At build time, local variables resolve before template and preset expansion and environment substitution. The same catalog token produces dev_catalog.bronze.orders for dev and prod_catalog.bronze.orders for prod. A secret reference becomes a dbutils.secrets.get call in generated Python. Only at runtime does Databricks execute the call and retrieve the value from a secret scope. — Environment values change the generated references; secret values remain runtime lookups. Logical secret-scope names can map to different scopes per environment. Select the diagram to view it full size.¶

The alternative is config drift¶

The obvious way to run one pipeline in two places is to keep two copies — orders_dev.yaml and orders_prod.yaml — and hand-edit the catalog, the host, and the password in each. It works on the first day. Then every change has to be made twice, and the two files drift apart: a column added in dev, a flag flipped in prod, a fix applied to one copy and forgotten in the other. The pipeline that “worked in dev” fails in prod, and the failure hides in a diff between two files that were supposed to be identical.

Environment drift is the most common shape of production bug, and a copy per environment manufactures it. When the flowgroup is one file, that whole class of bug closes: a difference between dev and prod can only come from the substitution file, which is a small, reviewable YAML document. Generate for two environments, diff the output, and every difference is one you declared and can point at.

Four tiers, widening outward¶

“Environment-specific” is not one thing, so Lakehouse Plumber does not resolve it with one mechanism. It resolves four kinds of placeholder, in a fixed order, each with a wider scope than the last:

%{local_var} — local variables, scoped to one flowgroup. Resolved first in flowgroup resolution, before template expansion. For a value that repeats inside a single flowgroup and is the same in every environment.
{{ template_param }} — template parameters, scoped to one instance of a template. Expanded when a flowgroup stamps out a template. For the pieces that vary between uses of one shared pattern.
${env_token} — environment tokens, scoped to one environment. Substituted from substitutions/<env>.yaml at generation time. For the values that actually change between dev, staging, and prod.
${secret:scope/key} — secret references, scoped to the secret store. Turned into a dbutils.secrets.get(...) call in the generated Python, so the value is fetched when the pipeline runs — never written into a file.

Read that list top to bottom and the scope widens at each step: one flowgroup, one template instance, one environment, the workspace’s secret store. The order is not cosmetic. Each layer resolves before the next one sees the YAML, so the output of one can feed the next: a local variable can sit inside a template parameter, a template parameter can expand into text that contains an environment token, and an environment token can resolve to a string that holds a secret reference. Resolving them in any other order would break that chaining.

The order also tracks when each value becomes known. A local variable expands when the flowgroup is resolved. A template parameter is fixed when the template is stamped. An environment token is fixed when you choose --env at generate time. A secret is not fixed at generate time at all — the generated code carries a lookup, and the value arrives at run time.

Choosing the tier is choosing the scope¶

Because the tiers differ by scope, picking the right one is a question about where a value belongs, not about syntax:

The value is the same everywhere but repeats inside one flowgroup — a table name reused across three actions. That is a local variable. It expands before environment substitution, so its text can contain an environment token. A hard-coded dev value in variables: still remains a dev value.
The value differs between dev and prod — a catalog, a schema, a storage path, an alert address. That is an environment token, and it lives in the substitution file.
The value is a credential. That is a secret reference. A literal password in a substitution file is a leak the moment the file is committed, because substitution files ship to version control with the rest of the project. The reference keeps the value in the Databricks secret scope and puts only the indirection under version control.
The value is one knob in a pattern reused across many flowgroups. That is a template parameter, filled in each time the template is used.

The mistake the model guards against is putting an environment-varying value in the wrong tier — most often a local variable, which is resolved before Lakehouse Plumber even knows which environment is in play. The rule that falls out of the scope ladder: if a value differs between environments, it is an environment token or a secret; if it is constant but repeated, it is a local variable or a template parameter.

Note

Always write environment tokens as ${token}. The bare single-brace form {token} is deprecated: it collides with Python f-string syntax inside the .py and .sql files a flowgroup can reference, where {name} is an ordinary runtime variable. The ${...} form is unambiguous because ${} is not valid f-string syntax.

The model to keep¶

Hold on to the four-tier ladder, narrowest scope to widest:

%{local_var} — one flowgroup.
{{ template_param }} — one template instance.
${env_token} — one environment.
${secret:scope/key} — the secret store, resolved at run time.

A flowgroup written against that ladder promotes across every environment untouched, and the only thing that changes from dev to prod is a small file you can read in one sitting. For the mechanics — writing a substitution file, generating for an environment, and reading the diff across environments — see the Substitutions and Secrets guide. For the exhaustive precedence rules, recursive token expansion, and the default-scope secret form, see the substitutions reference.