Databricks

Databricks Cost Intelligence · Architecture

OBSERVE → MODEL → CONSTRAIN → SIMULATE → EXPLAIN → APPROVE

READ-ONLY FIRST
CONCEPT v0.1
Architecture view active Databricks Cost Intelligence architecture A read-only architecture that observes a running Databricks estate, builds workload digital twins, simulates constrained alternatives, uses governed AI for explanation, and requires human approval before validation or action. OBSERVED DATABRICKS ESTATE OPTIMIZATION INTELLIGENCE PLATFORM DECISION & ACTION EVIDENCE-BASED · CONSTRAINED · EXPLAINABLE READ-ONLY COLLECTION · NO WORKLOAD REDESIGN Current Compute Fleet All-purpose · Jobs · Pipelines CONFIG 01 Runtime Telemetry CPU · memory · shuffle · spill OBSERVED 02 Cost & Job History DBU · active hours · SLA history COST 03 Live Capability Catalog Node types · DBR · policies CURRENT 04 Cloud & Contract Rates Infrastructure · discounts · region RATES 05 Telemetry & Catalog Sync Read-only · scheduled · versioned COLLECT System Table Aggregator Selective 30–90 day evidence windows Spark Event Parser Stages · tasks · spill · skew · backlog Capability Snapshot Nodes · runtimes · policies · lifecycle Pricing Adapters DBU + cloud + customer effective rate Release & Docs Sync DBR notes · EOL · compatibility RAG Schema Normalization Cloud-neutral node & config model NO RAW CUSTOMER DATA SENT TO AI Workload Digital Twin Observed behavior · not a questionnaire PROFILE Optimization Lakehouse Telemetry snapshots · config history Catalog versions · outcomes · audit Workload Signature Interactive · batch · streaming CPU · memory · I/O · shuffle · driver Operational Envelope Concurrency · failure · retries · SLA Idle windows · peak vs. average Current Cost Baseline DBU + infrastructure + active hours Per cluster · job · owner · workspace VERSIONED EVIDENCE WINDOW Candidate Engine Hard rules before AI Workspace / region availability DBR · Photon · ARM compatibility Policies · access mode · libraries SLA · spot · GPU · local disk Full config candidates, not SKUs VALID CANDIDATES ONLY Cost & Risk Simulator Cost · performance · risk $10K ILLUSTRATIVE MONTHLY SAVE Runtime & worker profile Failure / OOM / SLA probability Savings range + confidence Configurable AI Reasoning Structured evidence in · explanation and migration plan out AI GATEWAY FAST Extract · classify Release-note sync HAIKU CLASS STANDARD Analyze · explain Config diff · Q&A SONNET CLASS DEEP Complex migration High-value · low confidence OPUS CLASS Fresh DBR RAG · champion/challenger · token budgets · prompt versioning LLM never invents candidates or calculates the authoritative price Recommendation Evidence-backed savings decision CURRENT All-Purpose DBR 14.3 · fixed 8 PROPOSED Jobs Compute LTS · autoscale 2–6 $9–11K ILLUSTRATIVE SAVINGS RANGE Performance impact Confidence & evidence strength Risk & migration warnings Databricks App Review · approve · validate · export HUMAN Approval required before change Canary / shadow validation Export JSON · policy · Terraform OPTIONAL APPLY · REVALIDATE · ROLLBACK Safety Envelope MVP DEFAULT Read-only workspace access No automatic cluster mutation Savings shown as a range High-risk changes require canary START: ALL-PURPOSE + JOBS · ONE CLOUD Trust, Governance & Continuous Learning Controls every recommendation from evidence window to realized outcome CONTROL PLANE OAuth · RBAC · Least Privilege Catalog & Price Snapshot Lineage Model · Prompt · Rule Versions AI Usage · Tokens · Budgets Approval & Immutable Audit Trail Evaluation Dataset & Regression Champion / Challenger Rollout Validation & Rollback Evidence Realized Savings & Feedback LEGEND & OPERATING PRINCIPLE Evidence / decision flow Control / catalog flow Outcome / learning feedback THE LLM EXPLAINS — IT DOES NOT AUTHORIZE YASAR KOCYIGIT / DAQ ARCHITECTURE SIGNATURE · CONCEPT v0.1 VALIDATION OUTCOME · REALIZED SAVINGS · CALIBRATION