============================= Dependency analysis reference ============================= .. meta:: :description: Reference for how Lakehouse Plumber builds dependency graphs, including the action-level depends_on field, table and view matching, warning suppression, and trust-depends-on mode. ``lhp dag`` builds its graph from the data each action reads and produces. Most edges come from an action's configured ``source`` or from table reads extracted from its SQL and Python bodies. Use the action-level ``depends_on`` field when an upstream cannot be discovered statically. This field declares **data dependencies**. Its values are table or view references, not action names, flowgroup names, or pipeline names. .. important:: Do not confuse the action-level ``depends_on`` field documented here with the ``depends_on`` keys in dependency-analysis JSON or generated Databricks job YAML. Those contain pipeline or task names derived from the completed graph. ``depends_on`` field ==================== ``depends_on`` is optional and is accepted on every action type. .. list-table:: :header-rows: 1 :widths: 24 76 * - Property - Contract * - Location - On an action, alongside ``name``, ``type``, ``source``, and ``target``. * - Type - List of non-empty strings. * - Default - Unset. An empty list has no effect. * - Accepted shapes - ``catalog.schema.table``, ``schema.table``, or a single-part name. See `How references match`_ for the different matching rules. * - Graph behavior - Additive by default: declared references are added to sources discovered from configuration, SQL, and Python. * - Runtime behavior - None. The field is used by dependency analysis only and is never emitted into generated pipeline code. * - Validation - Invalid values raise ``LHP-VAL-063``. Example ------- This Python transform reads exchange rates through runtime-only logic that static analysis cannot resolve: .. code-block:: yaml :caption: pipelines/silver_orders.yaml actions: - name: enrich_orders type: transform transform_type: python source: v_orders module_path: transforms/enrich_orders.py function_name: run depends_on: - ${catalog}.${reference_schema}.exchange_rates target: v_enriched_orders If another action in the analyzed project produces the identically authored ``${catalog}.${reference_schema}.exchange_rates`` table, LHP creates an edge from that producer to ``enrich_orders``. If no producer matches, the reference is reported as an external source. How the DAG is built ==================== LHP constructs the graph in four steps: #. **Collect producers.** A load or transform ``target`` registers a temporary view. A streaming-table or materialized-view write registers its table. A Delta sink with ``options.tableName`` also registers a table; other sinks do not. #. **Collect sources.** For each action, LHP combines configured source references, table reads extracted from SQL and Python, and the action's ``depends_on`` entries. #. **Match sources to producers.** Each match creates an action edge pointing from producer to consumer. An unmatched source remains external. #. **Roll up the graph.** Cross-action edges become flowgroup edges when the actions belong to different flowgroups, then pipeline edges when those flowgroups belong to different pipelines. The rolled-up pipeline graph determines execution stages, cycle detection, JSON and DOT output, and the task dependencies in orchestration jobs generated by ``lhp dag --format job``. Default mode: additive declarations =================================== In the default mode, ``depends_on`` adds declarations without replacing what LHP can discover: .. code-block:: text action sources = configured sources + SQL/Python-extracted sources + depends_on declarations This makes the field suitable as an escape hatch for one dynamic or otherwise opaque read while retaining every statically discoverable edge on the action. Repeated references collapse to the same graph edge. Warning suppression ------------------- A non-empty ``depends_on`` suppresses that action's ``LHP-DEP-002`` and ``LHP-DEP-003`` extraction advisories. Suppression applies to the whole action, not to an individual read: LHP cannot determine which declared reference was intended to cover which unresolved expression. .. important:: If one action has several unresolved reads, declaring only one upstream still suppresses the advisories for all of them. Declare every upstream the action requires. Default mode still reads and parses the action's SQL and Python bodies. A non-empty ``depends_on`` suppresses extraction advisories, but it does not bypass missing-file errors or other failures encountered while loading those bodies. Authoritative mode ================== Use ``--trust-depends-on`` when actions fully declare their upstreams and body extraction is unnecessarily expensive: .. code-block:: console $ lhp dag --trust-depends-on For each action with a non-empty ``depends_on`` list, LHP skips SQL and Python body extraction entirely. Its source set becomes: .. code-block:: text action sources = configured sources + depends_on declarations Actions without ``depends_on`` continue to use normal extraction. .. list-table:: :header-rows: 1 :widths: 34 33 33 * - Behavior - Default - ``--trust-depends-on`` * - Parse bodies for an action with declarations - Yes - No * - Keep statically extracted reads - Yes - No * - Keep configured ``source`` references - Yes - Yes * - Add declared references - Yes - Yes * - Surface extraction advisories for that action - No - No extraction runs * - Encounter missing SQL/Python body files for that action - Possible - No .. warning:: Trust mode treats declarations as authoritative. An incomplete list silently omits the undeclared body-derived edges, which can produce an incomplete DAG and incorrect orchestration order. How references match ==================== ``depends_on`` entries use the same matcher as automatically discovered reads. Before comparison, LHP lowercases each dotted part, removes backticks, and trims surrounding whitespace. .. list-table:: :header-rows: 1 :widths: 32 68 * - Declared reference - Matching behavior * - ``catalog.schema.table`` - Matches the same canonical three-part table. It can also match a two-part Delta-sink producer with an omitted catalog; it never suffix-matches a producer in a different explicit catalog. * - ``schema.table`` - Matches a three-part producer only when exactly one registered catalog produces that ``schema.table``. Multiple catalogs make the reference ambiguous, so it remains external. * - ``view_name`` - Matches a load or transform ``target`` only within the same pipeline. A single-part name does not match a global table producer. * - No matching producer - Remains an external source; LHP does not create a phantom edge. Views are pipeline-scoped because Lakeflow temporary views exist only within their pipeline. Tables are global and can form edges across pipelines. Substitution tokens ------------------- Dependency analysis does not resolve ``${...}`` substitutions. Tokens remain literal parts of the reference and must be authored consistently on both sides of the edge. For example, a declaration containing ``${catalog}.silver.orders`` does not match a producer named ``dev_catalog.silver.orders``. Spell ``depends_on`` in the same literal form as the producer when the producer and consumer otherwise use different substitution forms. Validation and edge cases ========================= Each entry must be a non-empty string with no more than three dot-separated parts and no blank part. These are invalid: .. code-block:: yaml depends_on: - "" - catalog..table - workspace.catalog.schema.table They raise ``LHP-VAL-063``. The following values are structurally valid but can still remain external: - a well-formed table that no analyzed action produces; - an ambiguous two-part table produced in multiple catalogs; - a single-part name with no matching view in the same pipeline; or - a tokenized reference whose literal spelling differs from the producer. Manually declared edges participate in cycle detection exactly like discovered edges. A declaration that points back to the consumer, directly or through other actions, can produce ``LHP-DEP-001``. What the field does not do ========================== Action-level ``depends_on`` does not: - add a Spark read to generated code; - change generation-time action ordering inside a flowgroup; - rename references for sandbox table isolation; - require an upstream to exist in the project; or - directly name a Databricks job task. It changes the dependency-analysis graph. Pipeline and job ordering are then derived from that graph. Related pages ============= - :doc:`/guides/ops/dependency-analysis` — run ``lhp dag`` and read each output format. - :doc:`/concepts/flowgroups-and-dependencies` — understand producers, consumers, pipeline-scoped views, and execution stages. - :doc:`/reference/cli` — look up every ``lhp dag`` option. - :doc:`/reference/errors` — look up ``LHP-VAL-063``, ``LHP-DEP-001``, ``LHP-DEP-002``, and ``LHP-DEP-003``.