On this page

• Auto Loader (cloudfiles)
  • Minimal example
  • Source fields
  • Read mode placement
  • Action-level fields
  • Option precedence

Auto Loader (cloudfiles)

YAML location: an entry in a flowgroup’s actions list. Source selector: source.type: cloudfiles.

Streams files into a temporary view via Databricks Auto Loader. Fields live under source:.

Minimal example

Add this entry to the actions list in your flowgroup.

- name: load_orders
  type: load
  source:
    type: cloudfiles
    path: "${landing_volume}/orders/*.json"
    format: json
  target: v_orders_raw

Source fields

Field	Type	Required	Default	Description
type	string	Yes	—	Must be cloudfiles.
path	string	Yes	—	File path or glob to ingest.
format	string	Yes	—	File format; feeds cloudFiles.format (e.g. json, csv, parquet).
options	mapping	No	—	cloudFiles.* and reader options passed to Auto Loader.
reader_options	mapping	No	—	Options merged verbatim into the reader.
format_options	mapping	No	—	Options prefixed with <format>..
schema	string or mapping	No	—	Schema-file path, or {file: <path>}. Mutually exclusive with schema_file and options.cloudFiles.schemaHints.
schema_file	string	No	—	Back-compat schema-file path.

readMode must be stream (batch is rejected). Legacy scalar options (schema_location, schema_infer_column_types, max_files_per_trigger, schema_evolution_mode, rescue_data_column) are accepted for back-compat but are superseded by options.

Read mode placement

Prefer action-level readMode alongside source and target. The generator also accepts source.readMode as a fallback. An action-level value takes precedence when both are supplied.

Action-level fields

These fields apply to every load sub-type. name, type, source and target are required.

Field	Type	Required	Default	Description
name	string	Yes	—	Unique action name within the flowgroup.
type	string	Yes	—	Always load.
source	mapping	Yes	—	Source configuration. source.type selects the sub-type. Use a mapping for every load type, including SQL.
target	string	Yes	—	Name of the temporary view this action creates.
readMode	string	No	sub-type	batch or stream; selects spark.read vs spark.readStream. Default and constraints vary by sub-type (below). Ignored by sql, python, and jdbc. The view this action creates is a stream or a batch view, and later actions in the flowgroup must read it the same way (LHP-VAL-070); see Read mode.
description	string	No	auto	Docstring for the generated view function.
operational_metadata	bool or list[string]	No	—	Explicit column names to add; action-level false disables inherited selection. true alone adds no columns. See Operational metadata.
depends_on	list[string]	No	—	Extra upstream table or view references added to the dependency graph. See Dependency analysis reference.

See the worked guide or choose another load source.

Reader options are open-ended mappings. For the full Databricks option set, see Spark API options.

Option precedence

LHP processes options first, then merges reader_options, then format_options (prefixing keys with the file format when needed). Later mappings overwrite the same emitted option. Legacy scalar option fields fill only options that have not already been supplied. cloudFiles.format defaults to source.format when it is absent from the options.