Documentation

Website ↗ GitHub ↗

On this page

Python transform (python)

Python transform (python)¶

Calls a function in a project Python module and emits a @dp.temporary_view. The whole module file is copied to generated/<pipeline>/custom_python_functions/ and imported as from custom_python_functions.<module> import <function>.

Minimal example¶

Add this entry to the actions list in your flowgroup.

- name: transform_enrich
  type: transform
  transform_type: python
  source: v_orders
  target: v_orders_enriched
  module_path: "transforms/enrich.py"
  function_name: enrich
  parameters:
    lookup: regions

Options¶

Field	Type	Required	Default	Description
`source`	string / list	Yes	—	Input view name(s); `None` is rejected. A list passes a `List[DataFrame]` to the function.
`module_path`	string	Yes	—	Path to a `.py` file, relative to project root.
`function_name`	string	Yes	—	Name of the function to call.
`parameters`	dict	No	`{}`	Passed to the function as its `parameters` argument.
`readMode`	string	No	`batch`	`batch` or `stream`; controls `spark.read` vs `spark.readStream` on each source.

Function signature: single source def fn(df, spark, parameters); list source def fn(dataframes, spark, parameters). A source is required. To create a DataFrame without an input view, use a Python load.

Common fields¶

Fields accepted by every transform_type. Type-specific fields are listed in the options above.

Field	Type	Required	Default	Description
`name`	string	Yes	—	Action name, unique within the flowgroup.
`type`	string	Yes	—	Always `transform`.
`transform_type`	string	Yes	—	One of `sql`, `python`, `data_quality`, `temp_table`, `schema`.
`target`	string	Yes	—	Name of the produced view (or temporary table). Required for all transforms.
`source`	string / list / mapping	Yes	—	Upstream view name(s). Accepted shape and requirement vary per type — see this type’s options.
`description`	string	No	auto	Comment on the generated view/table. Defaults to an auto-generated string.
`operational_metadata`	bool / list	No	—	Add operational metadata columns; a list selects column names. Honored by `sql`, `python`, `data_quality`, `temp_table`; ignored by `schema`.
`depends_on`	list	No	—	Extra upstream table or view references added to the dependency graph. See Dependency analysis reference.