On this page

• pipeline_config.yaml
  • Top-level keys
    • Pipeline selection
    • Event-log overrides

pipeline_config.yaml

File: path passed with -pc / --pipeline-config. For setup, see Configure and deploy bundles.

A multi-document YAML file. The first document holds project_defaults; each subsequent document targets one or more pipelines via a pipeline key (a single name or a list). Pass its path with -pc / --pipeline-config.

project_defaults:
  <shared keys>
---
pipeline: <name>          # or [name1, name2]
<per-pipeline keys>

Top-level keys

Valid under both project_defaults and a per-pipeline document.

Key	Type	Required	Default	Description
catalog	string	Yes	—	Unity Catalog name. Must be set with schema (both or neither) and non-empty after substitution. Supports ${token}.
schema	string	Yes	—	Schema name. Must be set with catalog. Supports ${token}.
serverless	bool	No	true	Pipeline compute mode.
edition	string	No	ADVANCED	One of CORE, PRO, ADVANCED. Ignored when serverless: true.
channel	string	No	CURRENT	One of CURRENT, PREVIEW.
continuous	bool	No	false	Streaming/continuous pipeline mode.
packaging	string	No	source	One of source, wheel. Selects how generated code ships (see Bundle configuration). Consumed by LHP; never written to the resource YAML.

Any other top-level key (clusters, configuration, notifications, tags, event_log, environment, permissions, photon, and any Databricks Pipelines API field such as run_as) is passed through verbatim into the generated pipeline resource, except the three keys LHP writes itself:

• name and root_path are rejected with LHP-CFG-072. LHP names every pipeline resource <pipeline>_pipeline. It sets root_path to the pipeline’s generated code folder; Databricks adds that folder to sys.path, and the generated code’s imports of its helper modules rely on it.

• libraries entries are added after the glob library LHP generates for the pipeline’s code. Every LHP pipeline already has that glob library, and Databricks does not allow glob libraries together with notebook or file libraries, so each entry must be exactly glob: {include: <path>}: glob is the entry’s only key, and it holds only a non-empty include path. An include can point to one file or notebook, or to a folder ending /**. The value must be a list, not a bundle variable reference such as ${var.libs}; libraries: [] adds nothing. Any other value fails with LHP-CFG-072.

LHP checks these keys when it loads the file, so lhp validate (with bundle output enabled) and lhp generate both report the error. For example, this setting:

project_defaults:
  libraries:
    - glob:
        include: ${workspace.file_path}/pipeline_extras/**

gives every pipeline resource both libraries:

libraries:
  - glob:
      include: ${workspace.file_path}/generated/${bundle.target}/<pipeline>/**
  - glob:
      include: ${workspace.file_path}/pipeline_extras/**

As with every list, a per-pipeline libraries list replaces the project_defaults list, and libraries: [] in a pipeline document drops it for that pipeline. project_defaults also applies to the generated monitoring pipeline; use pipeline: __eventlog_monitoring with libraries: [] to leave it out.

Merge precedence (lowest to highest): built-in defaults → project_defaults → per-pipeline document. Nested mappings are deep-merged; lists are replaced wholesale. Token substitution from substitutions/<env>.yaml applies to every field. A plain ${name} still present after substitution fails lhp validate and lhp generate with LHP-CFG-010 when bundle output is enabled. Databricks bundle references such as ${var.x} or ${bundle.target} are dotted and are written to the resource as they are.

project_defaults:
  catalog: "${catalog}"
  schema: "${bronze_schema}"
  serverless: true

---
pipeline: bronze_load
packaging: wheel

External pipeline settings follow the Databricks pipeline resource reference.

Pipeline selection

The pipeline selector is a string or non-empty list of strings. A pipeline may occur only once across override documents. Defaults and overrides apply to selected pipelines by their YAML pipeline name, independently of file paths.

Use pipeline: __eventlog_monitoring to target the generated monitoring pipeline without repeating its resolved name. Defining both that alias and the actual monitoring pipeline name is an error. When monitoring is unconfigured or disabled, the alias entry is ignored with a warning.

Event-log overrides

A per-pipeline event_log mapping replaces the project’s injected event-log configuration for that pipeline. event_log: false opts that pipeline out. The generated monitoring pipeline itself does not receive event-log injection. See Monitoring configuration for the project settings and collection pipeline.