On this page

Build your own pipeline

Use this after the sample course to build with your own data. This example reads CSV files and writes a bronze streaming table. You need LHP installed; generation works locally without a running Spark cluster.

Create an empty project

mkdir my_project
cd my_project
lhp init my_project --no-bundle

This starts with local code generation. Add the bundle configuration in Configure and deploy bundles when you are ready to deploy.

Set the environment

Edit substitutions/dev.yaml with your catalog, bronze schema and landing volume. These must be accessible to the pipeline’s run-as identity when deployed.

substitutions/dev.yaml
# Development environment tokens.
# Referenced from flowgroups with ${token} syntax.
dev:
  catalog: dev_catalog
  bronze_schema: bronze
  landing_path: /Volumes/dev_catalog/landing/files

${catalog} in an action uses the catalog value inside the selected dev mapping. See Configure your project for the other configuration scopes.

Declare one flowgroup

Create pipelines/bronze_ingest.yaml:

pipelines/bronze_ingest.yaml
# Your first pipeline: ingest CSV files from a volume into a bronze table.
pipeline: bronze_ingest
flowgroup: orders_ingest

actions:
  # 1. Load: stream CSV files from the landing volume with Auto Loader.
  - name: load_orders_csv
    type: load
    source:
      type: cloudfiles
      path: "${landing_path}/orders/*.csv"
      format: csv
    target: v_orders_raw

  # 2. Write: append the streamed rows into a bronze streaming table.
  - name: write_orders_bronze
    type: write
    source: v_orders_raw
    write_target:
      type: streaming_table
      catalog: "${catalog}"
      schema: "${bronze_schema}"
      table: orders

pipeline names the generated pipeline. flowgroup names this group of actions. The load creates v_orders_raw and the write consumes it. A pipeline can contain several flowgroups and a file can contain several YAML documents; one file per flowgroup is a useful starting convention.

For another input, use the source chooser. For exact fields, use Auto Loader (cloudfiles) and Write actions.

Validate and inspect

Run from the project root:

lhp validate --env dev
lhp generate --env dev

Open generated/dev/bronze_ingest/orders_ingest.py. Check the resolved input path, catalog and schema before deploying. Generation produces Python and does not read your source data or create tables in Databricks.

Continue with Configure and deploy bundles. As the project grows, add transforms, quality checks and Compose flowgroups. When configuration repeats, introduce Apply shared defaults with a preset.