On this page

• Flowgroup YAML
  • Minimal example
  • Fields
  • Action syntax

Flowgroup YAML

Files: YAML files discovered under pipelines/. Each document describes one flowgroup. Use --- to put multiple flowgroups in one file. For composition examples, see Compose flowgroups.

Minimal example

pipeline: bronze
flowgroup: orders
actions:
  - name: read_orders
    type: load
    source:
      type: sql
      sql: SELECT 1 AS order_id
    target: v_orders
  - name: write_orders
    type: write
    source: v_orders
    write_target:
      type: streaming_table
      catalog: main
      schema: bronze
      table: orders

This loads a temporary view and writes an output table. A normal flowgroup needs at least one load and one write action. A materialized-view write with sql/sql_path or a snapshot-CDC write with source_function can supply its own data without a load. Test-only flowgroups are also supported. See Write actions for targets.

Fields

Field	Type	Required / default	Meaning
pipeline	string	Required	Generated pipeline name. All flowgroups with this name contribute to the same pipeline.
flowgroup	string	Required	Flowgroup identifier; also used for its generated Python filename.
job_name	string or null	unset	Orchestration job grouping, consumed by lhp dag. See Orchestration job configuration.
variables	mapping[string, string]	unset	Local %{name} substitutions. See Substitutions for resolution order.
presets	list[string]	[]	Presets applied in order. See Presets for merge behaviour.
use_template	string or null	unset	Template name to expand into actions. See Templates.
template_parameters	mapping	unset	Values for the selected template’s declared parameters.
actions	list[mapping]	[] in the model	Explicit actions or template output. Use the references below for the required fields of each action.
operational_metadata	boolean or list[string]	unset	Column selection. Use explicit lists; see Operational metadata for selection and disabling rules.

Action syntax

• Load actions: source.type selects the reader.

• Transform actions: transform_type selects the operation.

• Write actions: write_target.type selects the output.

• Test actions: test_type selects the assertion.

File paths do not determine pipeline membership; pipeline does. Reference temporary views only within their pipeline. Cross-pipeline dependencies use persisted tables. See Dependency analysis reference for depends_on.