On this page

• Custom DataSource (custom_datasource)
  • Minimal example
  • Source fields
  • Action-level fields

Custom DataSource (custom_datasource)

YAML location: an entry in a flowgroup’s actions list. Source selector: source.type: custom_datasource.

Reads through a user-supplied PySpark DataSource class. Fields live under source:.

Minimal example

Add this entry to the actions list in your flowgroup.

- name: load_custom
  type: load
  source:
    type: custom_datasource
    module_path: "sources/my_source.py"
    custom_datasource_class: MyDataSource
    options:
      endpoint: "${api_endpoint}"
  target: v_custom

Source fields

Field	Type	Required	Default	Description
type	string	Yes	—	Must be custom_datasource.
module_path	string	Yes	—	Path to the .py DataSource module.
custom_datasource_class	string	Yes	—	Name of the DataSource class to register and use.
options	mapping	No	{}	Passed to the reader as .option(...) calls.

readMode defaults to stream; batch is also supported. The class’s name() classmethod return value is used as the .format(...) name.

Action-level fields

These fields apply to every load sub-type. name, type, source and target are required.

Field	Type	Required	Default	Description
name	string	Yes	—	Unique action name within the flowgroup.
type	string	Yes	—	Always load.
source	mapping	Yes	—	Source configuration. source.type selects the sub-type. Use a mapping for every load type, including SQL.
target	string	Yes	—	Name of the temporary view this action creates.
readMode	string	No	sub-type	batch or stream; selects spark.read vs spark.readStream. Default and constraints vary by sub-type (below). Ignored by sql, python, and jdbc. The view this action creates is a stream or a batch view, and later actions in the flowgroup must read it the same way (LHP-VAL-070); see Read mode.
description	string	No	auto	Docstring for the generated view function.
operational_metadata	bool or list[string]	No	—	Explicit column names to add; action-level false disables inherited selection. true alone adds no columns. See Operational metadata.
depends_on	list[string]	No	—	Extra upstream table or view references added to the dependency graph. See Dependency analysis reference.

See the worked guide or choose another load source.