On this page

Schema transform (schema)

Renames, casts, and (in strict mode) filters columns of the source view via a schema mapping, emitting a @dp.temporary_view. Requires exactly one of schema_inline or schema_file.

Minimal example

Add this entry to the actions list in your flowgroup.

- name: enforce_schema
  type: transform
  transform_type: schema
  source: v_orders
  target: v_orders_typed
  schema_inline: |
    order_id: BIGINT
    amount: DECIMAL(18,2)
  enforcement: strict

Options

Field

Type

Required

Default

Description

source

string

Yes

—

Input view name. Must be a plain string (the nested source.view form is not supported).

schema_inline

string

Cond.

—

Inline schema definition (arrow or structured YAML). Exactly one of schema_inline / schema_file.

schema_file

string

Cond.

—

Path to an external schema file. Exactly one of schema_inline / schema_file.

enforcement

string

No

permissive

strict keeps only defined columns; permissive transforms defined columns and passes the rest through unchanged.

readMode

string

No

stream

batch or stream.

Schema definitions use arrow syntax, one entry per column:

  • old_col -> new_col: BIGINT — rename and cast.

  • old_col -> new_col — rename only.

  • col: DECIMAL(18,2) — cast in place.

  • col — pass through / explicit keep (strict mode).

A source-column name (left of ->, a cast-only col, or a pass-through col) may contain $. Write any other source name — one with spaces, ., -, : or other characters — in backticks, as in Spark SQL. A backtick inside such a name is written as two backticks. In a columns: list, quote the entry, because YAML cannot start an unquoted value with a backtick:

columns:
  - "`customer name` -> customer_name"
  - "`order.id` -> order_id: BIGINT"
  - "`unit-price`: DECIMAL(18,2)"

A plain schema_inline line is not a YAML string, so it is written as-is, without quotes:

schema_inline: |
  `customer name` -> customer_name
  `unit-price`: DECIMAL(18,2)

A rename target (right of ->) must be a clean identifier of letters, digits, and underscores; anything else raises LHP-VAL-011.

Common fields

Fields accepted by every transform_type. Type-specific fields are listed in the options above.

Field

Type

Required

Default

Description

name

string

Yes

—

Action name, unique within the flowgroup.

type

string

Yes

—

Always transform.

transform_type

string

Yes

—

One of sql, python, data_quality, temp_table, schema.

target

string

Yes

—

Name of the produced view (or temporary table). Required for all transforms.

source

string / list / mapping

Yes

—

Upstream view name(s). Accepted shape and requirement vary per type — see this type’s options.

description

string

No

auto

Comment on the generated view/table. Defaults to an auto-generated string.

operational_metadata

bool / list

No

—

Add operational metadata columns; a list selects column names. Honored by sql, python, data_quality, temp_table; ignored by schema.

depends_on

list

No

—

Extra upstream table or view references added to the dependency graph. See Dependency analysis reference.