On this page
Telemetry¶
Lakehouse Plumber (LHP) reports anonymous usage telemetry so the project can see which commands and features are actually used. It is on by default and opt-out. LHP records one event per CLI command; the web integrated development environment (IDE) adds one per browser tab and per run. Every field is a bounded enum, a boolean, a counter, a version string or an opaque identifier — never a name, a path, file content or an error message.
LHP appends the event to a local spool file and posts it in the background, so telemetry never blocks a command, never changes an exit code and never raises. When the endpoint is unreachable the upload fails silently and the events stay on disk.
Run lhp telemetry show to print the shape of the event this machine would
send — the preview’s duration_ms and exit_code are zero and flags
is empty — and lhp telemetry off to turn telemetry off.
Added in version 0.9.2.
What is sent¶
Envelope¶
Every event carries the same envelope.
Field |
Type |
Value |
|---|---|---|
|
integer |
|
|
string |
A fresh UUID v4 per event. The receiver can store an event more than once, and |
|
string |
|
|
string |
UTC timestamp with millisecond precision, |
|
string or null |
The installation’s UUID v4 (see Identifiers). Always null under continuous integration (CI). |
|
string or null |
32 lowercase hex characters, a salted hash (see Identifiers). Null outside a project. |
|
string |
|
|
string |
The installed LHP version. |
|
string |
Major and minor only, for example |
|
string |
|
|
string |
|
|
string |
|
|
string |
|
|
string |
|
|
boolean |
Whether |
|
boolean |
Whether stderr is a terminal. |
|
object |
The per-event fields below. |
How install_kind is decided¶
LHP reads install_kind from the distribution’s PEP 610 direct_url.json
metadata, and only its dir_info.editable field. The record’s url field
is a local filesystem path and is never read.
Value |
Meaning |
|---|---|
|
Installed from a built wheel: either no |
|
An editable (PEP 610 |
|
The metadata is missing, unreadable, or describes a direct URL with no |
cli.command¶
One event per command run, including failed runs.
Field |
Type |
Value |
|---|---|---|
|
string |
The command that ran, with spaces replaced by dots: |
|
list of strings |
The sorted names of the parameters you passed on the command line. Values are never read, so |
|
string or null |
|
|
integer |
Wall-clock time inside the command’s error boundary. |
|
integer |
|
|
string or null |
The |
|
string or null |
The exception’s class name only. No message, no stack trace. |
|
object |
Error codes counted, for example |
|
object |
The same shape for failures, with the same |
|
integer or null |
How many files the run wrote. |
|
boolean or null |
Whether bundle support was active for the run. |
|
boolean or null |
Whether the run used the discovery cache. |
|
object or null |
The project shape below. Present for |
Project shape¶
project is a fixed allowlist of 51 keys — counters and booleans that
describe a project’s size and which features it configures. A value LHP does
not recognize folds into its family’s *_other key, so the number of keys
that reach the wire is a property of LHP, not of your project.
The counts come from project discovery, which expands blueprint instances but
not templates: actions that a template supplies are missing from actions
and from every per-type counter. The pipeline LHP generates when
monitoring is enabled is counted, with
its flowgroup and actions.
Keys |
What they count |
|---|---|
|
Project totals. |
|
Distinct write targets declared in the discovered flowgroups, counted once each however many actions write to them. Discovery expands neither templates nor presets, so a target that only a template supplies is not counted. Sinks are not counted. |
|
Load actions by source type. |
|
Transform actions by transform type. |
|
Write actions by target type. |
|
Table write actions by mode. Sinks carry no mode. |
|
Test actions by test type. |
|
How many of each the project declares, and how many flowgroups use a template. |
|
Booleans: whether the project configures the feature. The configured values are never collected. |
web.session¶
LHP emits one event per browser tab of lhp web, when the tab’s event
stream disconnects, when the tab goes idle, or when the server shuts down. It
drops a session shorter than five seconds that moved no counter rather than
sending it.
Field |
Type |
Value |
|---|---|---|
|
string |
A UUID v4 the tab mints for itself and keeps in |
|
integer |
Seconds from the tab’s first recorded activity to its last, or to the close of its event stream when that is later. Activity is any application programming interface (API) request the tab makes other than the health check, including opening the event stream, plus the end of each run the tab starts. A tab still connected when the server shuts down counts up to the shutdown. LHP does not count the waits that end a session: 30 seconds after the event stream closes, or 30 minutes without activity. |
|
string |
|
|
boolean |
Whether the tab ever opened the event stream. |
|
object |
API calls counted by route family, for example |
|
object |
File mutations counted by kind: |
|
object |
Counts for |
|
integer |
Derived from the |
|
boolean |
Whether the AI assistant was invoked. |
|
string or null |
|
|
string or null |
|
|
object |
Counts keyed |
web.run¶
One event per validate or generate run started from the web IDE.
Field |
Type |
Value |
|---|---|---|
|
string |
The tab that started the run. |
|
string |
|
|
string |
|
|
string |
As for |
|
boolean |
Whether the run used sandbox mode. |
|
boolean |
Whether a pipeline filter was applied. The pipelines it selected are never collected. |
|
boolean or null |
Whether bundle support was active. |
|
integer |
Wall-clock time of the run. |
|
boolean |
Whether the run reported success. |
|
boolean |
True when the stream ended with no terminal result, for example because the browser disconnected. |
|
string or null |
An LHP error code such as |
|
integer |
For a run that reports totals, such as a completed |
|
integer or null |
How many files a generate run wrote. |
Install events¶
LHP records install.first_seen when it mints the install id; the event
carries no fields. It records install.upgraded, which carries
previous_version, on the first run after the installed version changes. It
records neither in CI, where it never mints an install id.
Never collected¶
LHP never collects any of the following.
Identity — usernames, hostnames, email addresses, machine identifiers, IP addresses. The receiver never reads or stores the IP address a request comes from. It runs on Cloudflare, whose network sees that address to deliver the request and to apply a rate limit (see Endpoint, transport and retention).
Project content — the names of projects, pipelines, flowgroups, actions, tables, catalogs, schemas and environments; paths; YAML, SQL or Python content; generated code.
Diagnostics — error and warning messages, environment-variable values, secrets, git remotes.
Assistant — prompts, responses, tool arguments, token counts.
LHP reports a failure as the LHP-<CATEGORY>-<NUMBER> code plus the
exception’s class name, and nothing else.
LHP never sends the project name itself either. When a project declares
neither project_id nor bundle.uuid, LHP sends a salted hash of the name
instead, and that identifier is pseudonymous rather than anonymous: the salt
is a public constant, so anyone who guesses the name can reproduce the hash.
Identifiers explains how to replace it with an
opaque value.
Identifiers¶
Two identifiers travel with an event, and neither identifies a person.
Install id¶
A UUID v4 stored in the state file, created on the first run that records an event. It distinguishes one installation from another so that twenty commands from one machine are not read as twenty users. LHP never creates it and never sends it in CI, where a fresh runner would otherwise look like a new developer on every build.
Project id¶
The first 32 characters of the lowercase hex digest of
sha256("lhp-project:" + <raw identifier, trimmed and lowercased>).
project_id_source says which raw identifier was hashed.
Order |
Raw identifier |
Read from |
|
|---|---|---|---|
1 |
|
|
|
2 |
|
|
|
3 |
|
|
|
4 |
none — |
— |
|
The same hash is applied whichever source is found, so the raw identifier
never leaves the machine — a bundle.uuid is sent hashed, not as the value
your bundle deployment uses. Sources 1 and 2 normally hash the UUID that
lhp init minted, which is opaque, and the result is anonymous. Source 3
hashes a guessable string, which makes it pseudonymous — add a project_id
to lhp.yaml to move to source 1.
Project identity — project_id in lhp.yaml¶
Field |
Type |
Default |
Description |
|---|---|---|---|
|
string |
— (absent) |
Opaque per-project identifier, a UUID v4 by convention (any non-empty string is accepted). Used only in hashed form, and only by telemetry: nothing in code generation reads it. Safe to commit — that is the point, because it makes every developer’s and every CI run’s events roll up to one project. |
lhp init writes a fresh project_id into the scaffolded lhp.yaml,
and writes the same value as bundle.uuid in databricks.yml when the
project is scaffolded with bundle support.
name: my_project
project_id: 3f1c9e4a-6b2d-4e8f-9a70-5c1d2e3f4a5b
version: "1.0"
LHP never edits an existing lhp.yaml to add the key. A project created
before 0.9.2 keeps working and resolves to source 2 or source 3 above; adding
the key yourself changes the project’s identifier once, after which it is
stable.
Endpoint, transport and retention¶
LHP posts events as one POST request with a JSON body to
https://telemetry.lakehouse-plumber.dev/v1/events, a receiver that the LHP
project operates on Cloudflare. The endpoint is fixed at release time, and
lhp telemetry status prints the one this build uses. A build whose endpoint
is unreachable keeps events in the local spool until the caps drop them. The
request carries Content-Type: application/json and a
User-Agent: lhp/<version> header, and nothing else — no authentication, no
cookies. LHP uses the default urllib opener, so HTTPS_PROXY and
NO_PROXY are honored like every other Python HTTP client.
Property |
Value |
|---|---|
Attempts |
At most one per command, and at most one per event the web IDE emits; sessions that end together share one attempt. LHP starts no attempt while another from the same process is still in flight. There is no retry loop inside a run: a batch kept after a failed upload is offered again by a later upload (see the response rows below). |
Timeout |
3 seconds. |
Added exit latency |
At most 1 second in the worst case. The upload runs on a daemon thread and the command waits up to one second for it before exiting. |
Batch caps |
At most 500 events and 512 KB per request. |
Spool caps |
At most 500 events and 512 KB (524,288 bytes) on disk. When either cap is exceeded, the oldest events are dropped. |
Per-event cap |
8 KB; an event larger than that is dropped rather than spooled. |
2xx response |
The receiver accepted the batch, so LHP removes it from the spool. An empty body is a plain success; a non-empty body that is not JSON came from a proxy or a captive portal rather than the receiver, so LHP keeps the batch. |
400, 413 and any other 4xx except 429 |
LHP discards the batch. The receiver does not accept it however often LHP offers it. |
3xx redirect |
LHP discards the batch. The endpoint must not redirect: the HTTP client re-sends a redirected upload as a request without its body, so the reply says nothing about the batch. |
429, 5xx, or a connection that failed before the request was fully sent |
The receiver has not accepted the batch, so LHP keeps it and offers it again on the next upload, with no limit other than the spool caps. |
Sent but unanswered, or cut off at exit |
The receiver may already hold the batch, so LHP limits the resends: it offers the batch again and discards an event whose send goes unanswered a second time. A request that times out waiting for the reply, or whose connection drops, is unanswered. So is a batch still in flight when the command exits: LHP leaves it on disk after the one-second exit wait, and the first upload it attempts more than a minute after claiming the batch puts it back in the spool. A resent event keeps its |
Remote pause |
A 2xx response may carry |
Receiver |
|
Delivery |
A 200 reply means the receiver has queued the batch’s events durably. It then writes them to storage and retries on failure, so a storage outage never reaches LHP. It rejects a malformed event on its own and accepts the rest of the batch. Delivery is at least once, so an event can be stored more than once; |
Rate limit |
10 requests per 10 seconds from one IP address, applied by Cloudflare. A request over the limit gets 429 and its batch waits for the next upload. CI runners behind one network address translation (NAT) gateway share its IP address and can reach the limit together, which delays their events; a runner discarded at the end of its job discards whatever is still in its spool. |
Storage |
A Databricks workspace on Azure, in the West US 2 region (United States). |
IP addresses |
The receiver never reads the IP address a request comes from, never stores it, and derives no location from it. Cloudflare’s network sees the address to deliver the request and to apply the rate limit. |
Retention |
Stored events are kept: no automatic deletion is in place yet. Two things are planned but not in place: a job that deletes raw events after 12 months, and published monthly aggregates that carry no identifiers. |
Note
The telemetry upload is the only outbound network request the lhp
command line makes. (The lhp web AI assistant talks to whichever
provider you configure for it, which is a separate feature with its own
settings.)
Off switches¶
Switch |
Turns telemetry off when |
Scope |
|---|---|---|
|
set to |
The process. |
|
set to anything except an empty string, |
The process. |
|
the same values as |
The process. A permanent alias, kept for existing CI configurations. |
|
set — pytest sets it for every test phase |
Any pytest run, so a test suite that imports LHP never emits. |
|
run once |
This user on this machine, until |
Any off wins. The switches are checked in the order above and the first one
that says off decides; lhp telemetry status names it. The stored
preference is read only after every environment switch has passed, so an
opted-out environment never touches the config directory when LHP records an
event. The lhp telemetry commands are the exception: status reads the
state file and the spool, show reads the spool, and on and off
write the state file, whatever the switches say.
Blocking telemetry.lakehouse-plumber.dev at the network level also stops
anything leaving the machine, but it is not an off switch: events are still
recorded and still accumulate in the spool, up to the caps above.
Environment variables¶
Variable |
Effect |
|---|---|
|
|
|
Turns telemetry off. |
|
Turns telemetry off. A permanent alias of |
|
Overrides the config directory holding the state file and the spool. |
|
Overrides the endpoint. Accepted only for an |
|
|
The lhp telemetry command¶
lhp telemetry status— print whether telemetry is on, which layer decided that, the mode, the config directory, the install id, the endpoint, how many events are waiting to be delivered, and a link to this page. The count includes any batch an upload has claimed and not yet settled.lhp telemetry show— print thecli.commandevent this invocation would send, then the newest events waiting to be delivered, in-flight batches included, one compact JSON object per line. Standard output carries only those JSON lines, so it pipes intojqunchanged; the closing count, and the notice that nothing would be sent when telemetry is off, go to standard error.lhp telemetry on— turn telemetry on for this user on this machine.lhp telemetry off— turn telemetry off for this user on this machine. The install id and any spooled events are kept, and nothing is recorded or sent while it is off.
Every option is listed in the CLI reference.
lhp telemetry on and lhp telemetry off are the only telemetry
operations that can fail a command: when the state file cannot be written they
raise LHP-IO-028, because a preference you asked to change has to be
reported when it does not take. Every other telemetry failure is logged at
debug level and ignored.
Update hint¶
The receiver’s reply to a telemetry upload may name the newest final release of LHP on PyPI. LHP stores it and, on a later run, prints one line:
lhp 0.9.3 is available (installed 0.9.2): pip install -U lakehouse-plumber [LHP_UPDATE_CHECK=off to silence]
LHP prints the line only when the command succeeded, the run is interactive
and outside CI, LHP_UPDATE_CHECK does not opt out, the stored version is
newer than the installed one, and it has shown no hint in the past 24 hours.
Pre-releases are ignored unless the installed version is itself a
pre-release. The check never makes a request of its own, so it works only
while telemetry is on.
Local files¶
LHP resolves the config directory in this order: LHP_CONFIG_DIR verbatim;
then, on Windows, %APPDATA%\lhp (or ~\AppData\Roaming\lhp when that
variable is unset); elsewhere $XDG_CONFIG_HOME/lhp when it holds an
absolute path, otherwise ~/.config/lhp.
Path |
Contents |
|---|---|
|
Its own |
|
Events waiting to be sent, one JSON object per line. An event whose send went unanswered ends with an extra |
|
A batch an upload has claimed from the spool, named after the claiming process’s id and the claim time in milliseconds. LHP deletes it once the upload is settled. One left behind by a process that exited mid-upload goes back into the spool at the first upload attempted more than a minute after the claim. |
LHP creates these files, and the directories holding them, with owner-only
permissions. Apart from lhp telemetry on and lhp telemetry off, which
write the state file whatever the mode, the switches or CI and never mint an
install id, LHP creates none of them in LHP_TELEMETRY=log mode or on a run
where telemetry is off, and never creates the state file in CI.
install_id is always null in CI, because LHP neither mints nor sends it
there. The spool is still used in CI,
because that is how a CI run’s events reach the endpoint.
Deleting any of them is safe: the install id is minted again on the next recorded event, and a deleted spool or in-flight file discards the events it was holding.