On this page

• Kafka stream (kafka)
  • Minimal example
  • Source fields
  • Read mode placement
  • Action-level fields

Kafka stream (kafka)

YAML location: an entry in a flowgroup’s actions list. Source selector: source.type: kafka.

Streams from Apache Kafka (or Kafka-protocol-compatible endpoints) into a temporary view. Fields live under source:.

Minimal example

Add this entry to the actions list in your flowgroup.

- name: load_events
  type: load
  source:
    type: kafka
    bootstrap_servers: "broker1:9092,broker2:9092"
    subscribe: orders
  target: v_events

Source fields

Field	Type	Required	Default	Description
type	string	Yes	—	Must be kafka.
bootstrap_servers	string	Yes	—	Maps to kafka.bootstrap.servers.
subscribe	string	One of	—	Topic(s). Provide exactly one of subscribe / subscribePattern / assign.
subscribePattern	string	One of	—	Topic regex. One of the three subscription methods.
assign	string (JSON)	One of	—	Partition assignment. One of the three subscription methods.
options	mapping	No	—	Extra kafka.* options.

readMode must be stream (batch is rejected). Kafka returns binary key/value columns — deserialize them in a downstream transform.

Read mode placement

Prefer action-level readMode alongside source and target. The generator also accepts source.readMode as a fallback. An action-level value takes precedence when both are supplied.

Action-level fields

These fields apply to every load sub-type. name, type, source and target are required.

Field	Type	Required	Default	Description
name	string	Yes	—	Unique action name within the flowgroup.
type	string	Yes	—	Always load.
source	mapping	Yes	—	Source configuration. source.type selects the sub-type. Use a mapping for every load type, including SQL.
target	string	Yes	—	Name of the temporary view this action creates.
readMode	string	No	sub-type	batch or stream; selects spark.read vs spark.readStream. Default and constraints vary by sub-type (below). Ignored by sql, python, and jdbc. The view this action creates is a stream or a batch view, and later actions in the flowgroup must read it the same way (LHP-VAL-070); see Read mode.
description	string	No	auto	Docstring for the generated view function.
operational_metadata	bool or list[string]	No	—	Explicit column names to add; action-level false disables inherited selection. true alone adds no columns. See Operational metadata.
depends_on	list[string]	No	—	Extra upstream table or view references added to the dependency graph. See Dependency analysis reference.

See the worked guide or choose another load source.

For the upstream option definitions, see the Spark Kafka integration reference.