Monitoring job configuration

File: the path selected by monitoring.job_config_path in lhp.yaml. For orchestration jobs generated by lhp dag, use Orchestration job configuration.

monitoring.job_config_path points to a flat single-document YAML mapping. LHP deep-merges it over its defaults (max_concurrent_runs: 1, queue.enabled: true, performance_target: STANDARD) and then token-substitutes it per environment. The job name is fixed at <pipeline_name>_job and the pipeline_task is generated automatically. All fields are optional.

Field	Description
max_concurrent_runs	Maximum concurrent job runs. Default 1.
performance_target	STANDARD (default) or PERFORMANCE_OPTIMIZED.
queue	queue.enabled toggles job-run queueing.
notebook_cluster.new_cluster	New-cluster spec for the notebook task. Mutually exclusive with existing_cluster_id; serverless when neither is set.
notebook_cluster.existing_cluster_id	Attach the notebook task to an existing cluster.
timeout_seconds	Job-level timeout in seconds.
tags	Free-form {key: value} mapping.
email_notifications	on_start / on_success / on_failure recipient lists.
webhook_notifications	on_start / on_success / on_failure webhook-id lists.
permissions	User/group permission entries (level plus user_name or group_name).
schedule	Quartz cron schedule: quartz_cron_expression, timezone_id, pause_status.
tasks	Extra tasks, added after the generated union_event_logs notebook task and, when the monitoring pipeline is generated, the <pipeline_name>_pipeline task. Must be a list; each task needs a non-empty string task_key that no other task uses. List a generated task key under depends_on to run after it.

Any key not listed above is passed through verbatim into the job resource, so new Databricks Jobs fields work without an LHP change. The exception is name: LHP sets it and rejects a value here. A name or an invalid tasks value fails lhp generate with LHP-CFG-072. Use the flat shape shown below. LHP does not interpret a project_defaults wrapper or job_name selector here; unsupported resource keys can pass through and fail Databricks validation. LHP generates the job name and its own tasks.

# config/monitoring_job_config.yaml
max_concurrent_runs: 1
performance_target: STANDARD
queue:
  enabled: true
tags:
  environment: "${bundle.target}"

Use the Databricks job resource reference for external fields. See Adjust monitoring settings for change instructions.