On this page

Apply shared defaults with a preset

Use a preset when several actions share reader options or table properties. Start with a working flowgroup from Build your own pipeline.

The worked reuse diagram compares shared settings, template actions and blueprint expansion.

Create the shared settings

presets/bronze_defaults.yaml
# Shared bronze-ingestion defaults. Presets apply by implicit type matching:
# load_actions.cloudfiles merges into any cloudfiles load, and
# write_actions.streaming_table merges into any streaming-table write.
# The preset is the single place this shared bronze standard is defined.
name: bronze_defaults
version: "1.0"
description: "Bronze ingestion standard: CSV Auto Loader options + streaming-table properties."
defaults:
  load_actions:
    cloudfiles:
      options:
        cloudFiles.rescuedDataColumn: "_rescued_data"
        cloudFiles.maxFilesPerTrigger: 200
  write_actions:
    streaming_table:
      table_properties:
        delta.enableChangeDataFeed: "true"
        quality: "bronze"

load_actions.cloudfiles targets the load action’s source mapping; write_actions.streaming_table targets its write_target mapping. The preset name inside the file is the name used by a flowgroup.

Apply it to a flowgroup

pipelines/orders_ingest.yaml
# Bronze ingest for orders. The shared cloudFiles + streaming-table config
# lives in the bronze_defaults preset; this flowgroup declares only what is
# specific to orders: the source path, the view name, and the table name.
pipeline: bronze_ingest
flowgroup: orders_ingest
presets:
  - bronze_defaults

actions:
  - name: load_orders_csv
    type: load
    source:
      type: cloudfiles
      path: "${landing_path}/orders/*.csv"
      format: csv
    target: v_orders_raw

  - name: write_orders_bronze
    type: write
    source: v_orders_raw
    write_target:
      type: streaming_table
      catalog: "${catalog}"
      schema: "${bronze_schema}"
      table: orders

Validate and generate, then inspect the reader options and table properties in generated/dev/bronze_ingest/orders_ingest.py:

lhp validate --env dev
lhp generate --env dev

In a bundle project, add -pc config/pipeline_config.yaml to both commands.

In the current resolver, preset values win conflicts in load-source and write-target mappings. A flowgroup’s list of preset names is applied in order; later presets win. Read Presets before combining layers, especially for transform or flowgroup defaults, whose behaviour differs.

When the whole action sequence repeats, continue to Parameterize a template for many tables.