On this page

Adjust monitoring settings

Use this after enabling monitoring. Edit the source configuration, regenerate and deploy the resulting changes.

Choose the setting

Change

Setting

File

Concurrent event-log collection streams

monitoring.max_concurrent_streams

lhp.yaml

Summary queries and materialized views

monitoring.materialized_views

lhp.yaml

Where monitoring data is stored

monitoring.catalog, monitoring.schema, monitoring.streaming_table

lhp.yaml

Checkpoint storage

monitoring.checkpoint_path

lhp.yaml

Schedule, notifications or notebook compute

Monitoring job settings

File named by monitoring.job_config_path

For example, add max_concurrent_streams: 5 to your existing monitoring mapping. Preserve the required checkpoint_path and job_config_path fields. The complete defaults, ranges and inheritance rules are in Monitoring configuration.

Review and apply the change

lhp validate --env dev -pc config/pipeline_config.yaml
lhp generate --env dev -pc config/pipeline_config.yaml
databricks bundle validate -t dev

Review the generated monitoring notebook, pipeline and job resource. Deploy the bundle and run its monitoring job, then check the job’s tasks and resulting monitoring tables. Generation alone does not apply changes to a running job.

Treat checkpoint and destination changes as state changes: verify the intended history and replay behaviour before changing them. See checkpoint behaviour. To diagnose missing data or a failed job, use Troubleshoot a pipeline.