Skip to main content

Databricks integration

Connect Databricks so Context's Data tabs show Genie, cost, jobs and SQL queries, person by person.

Connect Databricks to Leanmote to see in Context's Data tabs what your organization does in Databricks: who uses Genie and what it costs against each person's cap, cost at list price, SQL queries and warehouses, and job and pipeline runs. Leanmote only reads Databricks' system tables, once a day: it never changes anything in your Databricks and never reads the text of your queries.


​

What Leanmote reads from Databricks


​

  • Cost — system.billing.usage, system.billing.list_prices. What Databricks bills, by person and workspace.

  • Genie — system.access.assistant_events. Messages to Genie Code, by person.

  • Jobs and pipelines — system.lakeflow.jobs, system.lakeflow.job_run_timeline, system.lakeflow.pipelines, system.lakeflow.pipeline_update_timeline. How each run ended and how long it took.

  • SQL queries — system.query.history. Who sends them, how long they take and what they cost. Never their text.

  • Workspaces — system.access.workspaces_latest. Their names.


​

Connect Databricks


​

An Owner or Manager connects Databricks once for the whole organization, in Integrations › Databricks. There are three ways:


​

  • Automatic (recommended) — any cloud. You need to be a workspace admin. In Databricks, create a token that lasts 1 day (your name › Settings › Developer › Access tokens › Generate new token). In Leanmote, paste it with your workspace address, pick the SQL warehouse and click Connect. Leanmote creates its own service principal, «Leanmote», gives it «Can use» on that warehouse and read access to the system tables, and checks that it works. It never stores your token: you can delete it in Databricks right after.

  • Step by step — any cloud. You create the service principal and its secret, give it «Can use» on a SQL warehouse, run the lines the window shows to give it read access to the system tables, and click Save and verify.

  • With your Microsoft account — Azure Databricks only. You sign in with your work account and Leanmote reads with your permissions. If you leave the company, the connection stops.


​

Read access to the system tables is given by a metastore admin: being a workspace admin is not enough. The role is assigned in the account console: Catalog › your metastore › Configuration › Metastore admin. If your user cannot give that access, Leanmote connects anyway and the window shows the lines to send to a metastore admin: they paste them in the SQL Editor and click «Run». If your workspace only accepts some IP addresses, add 100.27.182.179 first.


​

What it costs you


​

Leanmote's reads are about 10 short queries a day on the SQL warehouse you pick. Databricks bills the warehouse while it is on, so pick the smallest «Cluster Size» with a short «Auto Stop», or one your team already keeps on.


​

Questions to Genie by space (optional)


​

Which Genie space each question went to is only in Databricks' audit log, and Leanmote does not read the whole log. To see it, create a view with only the messages to Genie and share that view alone: the window has the lines to create it. Type its name (catalog.schema.view) in Genie events view (optional).


​

Check what Leanmote can read


​

In Integrations › Databricks › Manage each system table shows Readable, Missing grant or Not read yet. A table Leanmote cannot read leaves out only the figures that come from it; to fix it, open Change connection and run the lines it shows.


​

Genie caps


​

Databricks does not let Leanmote read your Genie spending caps: they are only in your account console. Type them in Settings → Organization Settings → Tabs → Data. See Data tabs in Context.


​

If the connection stops working


​

If Databricks stops accepting the connection — the secret expired or was deleted, the warehouse is gone, or it lost «Can use» — Leanmote marks it to be reconnected and lets your admins know, as with any other integration. Fix it in Databricks and save again in Integrations › Databricks.


​

Related articles


​


​

Did this answer your question?