LHP LHP Documentation
/
Website ↗ GitHub ↗
Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode Skip to content
Lakehouse Plumber
Website ↗ · GitHub ↗
Overview

Documentation

  • Get Started
    • Install and scaffold
    • Configure the project
    • Explore the pipelines
    • Generate and inspect
    • Deploy and run
    • Explore LHP Web
    • Where to next
  • Build pipelines
    • Build your own pipeline
    • Configure your project
      • Environments, substitutions and secrets
      • How environment substitution works
    • Understand LHP
      • How LHP works
      • The action model
    • Read data
      • Auto Loader (files)
      • Delta table
      • SQL query
      • JDBC database
      • Kafka topic
      • Python function
      • Custom data source
    • Transform data
      • SQL transform
      • Python transform
      • Schema enforcement
      • Temp tables
    • Write tables and outputs
      • Streaming table
      • CDC (change feed)
      • CDC (snapshots)
      • Replace (replace using)
      • Replace where (recent rows)
      • Materialized view
      • External sinks
    • Check data quality
      • Row expectations
      • Quarantine
      • Data tests
      • Test reporting
    • Compose flowgroups
      • Build a pipeline from several flowgroups
      • Flowgroups and dependencies
    • Reuse patterns
      • Choosing a reuse tool
      • Presets
      • Templates
      • Blueprints and instances
    • Add table metadata
  • Develop and deploy
    • Local development tools
      • Web IDE
      • AI assistant
      • Sandbox
      • Editor setup
      • Coding agents
      • How coding agents use LHP
    • Configure and deploy bundles
    • Generate and schedule orchestration jobs
    • Validate and deploy through CI
    • Package as wheels
    • Migrate an existing DLT pipeline
  • Monitor and troubleshoot
    • Monitoring
      • Enable pipeline monitoring
      • Adjust monitoring settings
    • Troubleshoot a pipeline
    • Inspect pipeline dependencies
  • Reference
    • Load actions
      • Auto Loader (cloudfiles)
      • Delta table (delta)
      • SQL query (sql)
      • Python function (python)
      • JDBC database (jdbc)
      • Kafka stream (kafka)
      • Custom DataSource (custom_datasource)
    • Transform actions
      • SQL transform (sql)
      • Python transform (python)
      • Row expectations (data_quality)
      • Temporary table (temp_table)
      • Schema transform (schema)
    • Write actions
      • Append to a streaming table
      • CDC streaming table
      • Snapshot CDC streaming table
      • Replace rows in a streaming table
      • Replace rows matching a predicate
      • Materialized view
      • Sink
    • Test actions
      • Row counts (row_count)
      • Unique keys (uniqueness)
      • Referential integrity (referential_integrity)
      • Required values (completeness)
      • Value range (range)
      • Schema comparison (schema_match)
      • Lookup matches (all_lookups_found)
      • SQL checks (custom_sql)
      • Custom expectations (custom_expectations)
    • Feature configuration
      • Operational metadata
      • Unity Catalog tags
      • Row quarantine (DLQ)
      • Test reporting
      • Monitoring and event logs
        • Monitoring job settings
      • Sandbox
    • Configuration files
      • lhp.yaml
      • Flowgroup YAML
        • Dependency declarations
      • pipeline_config.yaml
      • Orchestration job_config.yaml
      • Substitutions
      • Presets
      • Templates
      • Blueprints and instances
      • Schema and tag files
      • Expectation formats
      • Databricks bundle integration
    • CLI
    • Python API
    • LHP error code catalog
    • Glossary
    • Telemetry
    • Changelog
Back to top
View this page
Docs /Develop and deploy

Develop and deploy¶

Develop a pipeline locally, generate its Databricks resources, then validate and deploy the bundle. Use the optional authoring tools when they help your work.

  • Local development tools
  • Configure and deploy bundles
  • Generate and schedule orchestration jobs
  • Validate and deploy through CI
  • Package as wheels
  • Migrate an existing DLT pipeline

Already know the workflow? Look up pipeline_config.yaml, Orchestration job configuration or CLI directly.

Next
Local development tools
Previous
Add table metadata
Copyright © 2026, Lakehouse Plumber
Made with Sphinx and @pradyunsg's Furo

YOUR NEXT STEP

Start with your agent

Set up a project or learn LHP. Choose your agent and copy the prompt.

What would you like to do?
Choose your agent
1Choose your agent

Open Genie Code in your Databricks workspace.

2

Send the instruction

Your agent will check its environment, set up LHP and continue your project request.