Histograph

Lineage-aware observability &root-cause tracing for incidentresponse in production ML

Run a demo
Model health degrades after a source change. Histograph traces the cause backward through DataHub lineage, then verifies recovery separately through fresh runtime evidence.
DataHub
GitHub
Model runtime

Production ML incidents, fully connected.

Model health, lineage evidence, deployment state, and recovery in one workspace.

mobile-money-fraud

production · incident HG-184

Incident HG-184

Production recall crossed its allowed decrease

Threshold crossed

Reference

14.3%

stable · v1

Observed

0.0%

feature release · v2

Decrease

−14.3 pp

14.3 percentage points

Sample

240

10 min evaluation

Recall · labeled outcomes

12:00—14:30 UTC
Candidate model recall degradationRecall remains near 14.3 percent, then falls to zero after the feature release.FEATURE V2 OBSERVED

Next evidence

Trace the changed input

DataHub links the affected model to the amount feature released before the breach.

Lineage matches1 upstream change

Compare releases on production evidence.

Stable and candidate versions share the same labeled window and an explicit performance boundary.

Production recall

Same-window comparison

Boundary crossed
Reference
14.3%
stable · v1
Released
0.0%
feature release · v2
Allowed decrease
5.0 pp
healthy at 9.3%

Shared labeled outcomes

Both releases evaluated on the same production window

240 shared

Stable version 1 and feature release version 2 are compared on the same 240 labeled production outcomes. Recall is 14.3 percent for stable version 1 and 0 percent for released version 2, crossing the 9.3 percent healthy boundary implied by the allowed 5 percentage point decrease.

See which change reached production.

Histograph correlates monitor evidence with DataHub lineage and records the exact path on the incident.

DataHub investigation

Persisted lineage evidence

Path isolated

Dataset

momtsim.transactions

source

Changed feature

amount

v2

Affected model

mobile-money-fraud-detection

model

Deployment

production

reached

Unaffected branch · account_velocity_24h

Root-cause evidence

Changed feature
amount · v2
Reached
mobile-money-fraud-detection
Change time
14:06 UTC
Monitor window
14:12–14:18 UTC
Corroborated
DataHub highlights the implicated path from the MoMTSim transaction dataset through the changed amount feature and mobile-money fraud model to production. The unaffected account velocity feature remains muted, and the incident retains the corroborating change and monitor window.

Approval stays in your workflow.

Merge a rollback PR or authorize your remediation adapter. Histograph records who approved what.

Rollback mobile-money-amount-feature

histograph/rollback-hg-184

Approved
demo/deployment-feature-release.yaml
spec:
features:
- name: mobile-money-amount-feature
version: v2
version: v1
scaleMultiplier: 100
scaleMultiplier: 1

Action

Rollback feature

Target

v1 · ×1

Approval record

Proposed by
Histograph
Approved by
risk-lead
Proof
Signed merge
Merge commit
87ac2d1
Runtime stateReconciling

Desired state changed. Observed state is verified separately.

Histograph proposes a rollback by restoring the amount feature from version 2 with a scale multiplier of 100 to version 1 with a multiplier of 1. A signed merge records the approving engineer, while runtime reconciliation remains a separate state.

Recovery requires fresh production evidence.

Execution, observed runtime state, and a new healthy monitor window must agree before the incident resolves.

Recovery verification

Post-remediation evidence

Resolved
Feature scaleProduction recallHealthy boundary
Rollback observed14:18
Feature scale×100 → ×1Production recall0.0% → 14.3%Healthy boundary9.3%

240 fresh labeled outcomes

Action result

Succeeded

Observed state

feature v1 · ×1

Fresh window

240 labeled outcomes

Incident

Recovery verified

After the feature rollback is observed, the scale multiplier returns to one and a new labeled monitor window shows production recall above the healthy boundary. The incident resolves only after the action result, runtime state, and fresh evidence agree.

Connect without replacing your stack.

DataHub for lineage. GitHub for desired state and approvals. Your runtime for telemetry and recovery evidence.

Connector register

Existing systems stay authoritative

3 connected

DataHub

Configured

Lineage context

entity + upstream/downstream lineage

Destination

Incident record

GitHub

Configured

Desired state + approval

manifest + signed merge + deploy status

Destination

Incident record

Runtime

Configured

Telemetry + runtime state

predictions + outcomes + runtime state

Destination

Incident record

Histograph connects DataHub lineage context, GitHub deployment state and approvals, and the organization's model runtime telemetry and observed state into one incident record without replacing any of those systems.

Explore the full incident workflow.

Run a demo