CHouseUI
Using CHouse UIData

How data is watched

The collector, learned baselines, trust states and criticality behind every Data tab — and the grants it needs.

In the app
Data/data

The Data page answers one question for every table on the active connection: is the data right, right now? This page explains where its answers come from. Each tab has its own page: Overview, Incidents, Lineage, Pipelines, Datasets & promises, Coverage, Context and Scheduled queries.

Data replaced the DataOps page in 3.14; old /dataops/* links redirect.

The collector

A collector runs inside every CHouse UI server pod. On a schedule it reads ClickHouse's own metadata for each connection and stores what it learns — the evidence — in the RBAC database:

CollectorReadsFeeds
Catalog & tablessystem.tables, system.columns, system.parts, system.part_logFreshness, volume, schema history
Usage & queriessystem.query_logCriticality, read patterns, lineage from INSERT … SELECT, performance baselines
PipelinesEngine-specific system tables (Kafka, S3Queue, views, …)Pipeline status
Changes & capacitySettings, versions, DDL, system.disksChange timeline, capacity forecasts
ProfilesA 0.1% sample of critical tablesColumn distribution drift
Fleetsystem.metrics, system.processes, …Fleet and alerts

It never runs SELECT * over your data; only the profiler reads table rows, and only a small sample of the tables that matter most. A lease per connection and collector means each piece of work runs on one pod at a time, however many replicas you run.

Evidence queries respect data access rules: a table you can't read never appears, not even as a lineage node or inside someone else's query text.

What every table gets, from day one

Without any setup, each table learns:

  • Freshness — its normal gap between writes (p50 and p99), so stale means stale for this table.
  • Volume — an hourly band of rows written; an hour well outside it marks the table degraded.
  • Criticality — critical, important or normal, from how much it is actually read. Pin it by hand on the table's detail view (needs observe:edit).

Each table has a trust state:

StateMeaning
learningNot enough history yet to judge
trustedWrites and volume are within the learned baseline
degradedVolume is outside the learned band
staleNo write for longer than the table normally goes without one, outside its usual quiet hours

A table written by a scheduled query or a refreshable view follows that job instead. It is stale only when the job is failing or overdue. A run that succeeded with nothing new to write leaves the table trusted, and so does a paused job. Several parts written by one insert count as one write, and a quiet night or weekend that the table has already gone through this week is not stale.

Promises add explicit expectations on top — "loaded by 07:00", "no duplicate order ids" — when the learned baseline isn't enough.

Grant the collector what it needs

The connection's ClickHouse user must be able to read the system tables above. A connection is only saved once its user can, and Admin › Connections › Edit › Test connection lists anything missing with the exact GRANT statements to run — see ClickHouse privileges. If a grant is revoked later, it shows up as unsupported or missing evidence for the features that need it.

Collector health is shown as a chip on the Data page header. Settings that tune it (OBSERVE_*) are on Environment variables.

How often it runs

Each collector has a built-in interval:

CollectorBuilt-in intervalFastest allowed
Fleet health30 s (OBSERVE_FLEET_INTERVAL)10 s
Pipelines1 min30 s
Catalog, Table baselines, Observed lineage, Changes5 min1 min
Usage, Query performance, Capacity, Column profiles15 min5 min

Every collector can run as slowly as once a day. Change the intervals in Admin › Connections › Collection schedule, for every connection or for one — see Collection schedule. The schedule shows what each collector read on its last run and what that adds up to per day, so you can see where the load comes from before you change anything. A slower interval means slower answers: a stale table is noticed later, a new reader appears later.

The collectors that read system.query_log and system.part_log read only the rows written since their last run. Usage keeps 30 days of query shapes as daily rollups in the RBAC database instead of scanning 30 days of query_log on every run; after an install or upgrade it reads the newest 3 days first and fills in the older days 3 days per run. Two numbers it keeps are approximate: a query shape's distinct users (the largest daily count) and its p95 duration (the run-weighted average of daily p95s).

Permissions

TabNeeds any of
Overview, Lineage, Pipelines, Coverage, Contextobserve:view
Incidents, Datasetsobserve:view, data_health:view
Scheduled queriesscheduled_queries:view

Changing things needs more: observe:edit (criticality, dismissing suggestions, pipeline incidents), context:edit (table context and metrics), data_health:edit (promises), and the remediation:* permissions for fixes.

Edit this page on GitHub ↗