# Unity Catalog Governance

A governance layer over a lakehouse providing centralised access control, lineage and discovery across data and models.

**Rests on:** Regulatory Compliance Controls

**Leads to:** Semantic Layer

*Data Engineering and Analytics · Data Governance*


## Why learn this

You need this because without it, terms like 'governed model registry' or 'lineage' stay as vague labels that offer no control over your data. With Unity Catalog governance understood, you can reason about exactly who is allowed to see which table, which model, or which prompt, and know the machinery that enforces those rules. This clarity lets you answer concrete questions like 'can you prove this model was approved by legal?' with a query into the system rather than a scramble through email archives.


## The intuition

Imagine an office building where every file cabinet, every whiteboard, every model prototype, and every data report is in rooms scattered across floors. Anyone with a badge can wander in and read whatever they want. There is no list of who looked at what, no record of where a number came from, and no way to know if the report on your desk was based on last month's data or a guess.

Now imagine a security office on the ground floor. Every file, every whiteboard, every report is registered there with an owner. To open any drawer, you need a badge that is checked against that registration. Every time you pull out a document, the security office writes down a log line: who, what, when. If you are curious where a number in a report came from, you can walk back through the log to see the source documents, and those sources might themselves be logs, and so on until you reach raw data.

Unity Catalog is that security office, but for your data storage system. It centralises three things: access control (who can see what), lineage (what a number depended on), and discovery (what exists at all). Instead of each team bolting their own rules onto their own tools, one system governs everything: tables, files, models, functions, even prompts.

The key insight is that a model registry is not just a folder of saved weights; with Unity Catalog it becomes a governed asset with permissions and an audit trail, just like a table.


## An analogy

Think of a large hospital. Every patient record, every lab result, every X-ray image, every prescription is a piece of information. Before a unified system, each department kept its own logs. A nurse in cardiology might write results in one book, the pharmacy in another, and the radiology images in a third. To treat a patient, you would have to walk from department to department, hope the right person is there, and trust that the notebook is up to date. Worse, nobody could easily answer: 'Why is this patient on this dose of this drug?' because the chain from the original lab value to the prescription was scattered.

Now imagine the hospital adopts a single electronic health record system. Every test, every image, every prescription is entered into one central system, tagged with the patient's ID, the person who entered it, and the time. The system records who opened which record and when. A doctor can see the full history of a single lab value: where it came from (the machine, the technician, the calibration), and which prescriptions later used it. The chain is visible. The same idea applies to data: Unity Catalog is the electronic health record for a company's data assets. It tracks who can access what, and it records how each result was derived. But the analogy breaks in one important way. In a hospital, the goal is patient safety and legal compliance, and the system is used by a limited set of trained staff. In a data lakehouse, the users include data scientists, analysts, and automated pipelines, and the volume of accesses is millions per day. So Unity Catalog must be automated and designed for scale, not a manual logging system. Also, in a hospital, everyone is on the same network; a lakehouse often spans clouds and regions, so the catalog must work across those boundaries. The lessons from the hospital—centralised, auditable, with clear ownership—carry over; the technical machinery is different.


## Definition

A governance layer over your data storage system providing centralised access control, lineage, and discovery across data and models, so that one permission model spans tables, files, models, and functions, and every number can be traced back to its sources.


## How it connects

You have notes on Data Governance as a parent concept: 'policies, ownership, catalogues and controls that determine who may access which data for what purpose, and how that is documented and audited'. Unity Catalog is a concrete implementation of those ideas on Databricks. Your notes also mention Regulatory Compliance Controls under the same concept; Unity Catalog is the technical substrate that such controls rely on. In the book you studied ('Practical MLflow for Generative AI on Databricks'), you read about MLflow Tracing, Prompt Registry and Versioning, and Model Serving APIs. Unity Catalog governs all of those: the prompt registry is not a loose folder but a governed catalog entry, and the model serving endpoints check permissions against the catalog. Your notes on Data Modeling are the prerequisite for understanding what a 'table' is in this context, and later, AI Governance and Regulation will build on the audit and lineage capabilities you learn here.


## From scratch, in Python

```python
This concept is about permissions and tracking, not arithmetic, but you can still see the logic in plain Python. The following is a tiny simulation of what Unity Catalog does underneath: it keeps a dictionary of assets, each with an owner and allowed readers, and it has a log list. You are not required to run this, but it shows the core ideas: a permission check, an audit log, and a lineage map.

# A dictionary maps an asset name to its metadata.
# A dictionary in Python looks things up by name, not by position,
# like a phone book.
assets = {
    'customer_table': {'owner': 'alice', 'readers': ['bob', 'carol']},
    'sales_model': {'owner': 'dave', 'readers': ['bob']},
    'sales_data_file': {'owner': 'alice', 'readers': []},
}

# A list is an ordered collection. We will append log entries to it.
audit_log = []

# A function is a reusable block of code.
# It takes a username and an asset name, and returns True or False.
def can_read(username, asset_name):
    if asset_name not in assets:
        return False
    # A dictionary's .get() method returns None if the key is missing.
    metadata = assets[asset_name]
    # 'owner' is a key in the inner dictionary.
    if username == metadata['owner']:
        return True
    # 'readers' is a list; we check membership with 'in'.
    if username in metadata['readers']:
        return True
    return False

# Simulate a request from bob to read the customer_table.
request_username = 'bob'
request_asset = 'customer_table'
allowed = can_read(request_username, request_asset)

# Append a tuple to the log: (who, what, allowed). A tuple is
# like a list but cannot be changed after creation.
audit_log.append((request_username, request_asset, allowed))

print('Allowed?', allowed)  # True
print('Log so far:', audit_log)

# Lineage: a dictionary mapping an output to the inputs it used.
# In unity catalog this is a graph, but we show a simple version.
lineage = {
    'sales_report': ['customer_table', 'sales_model'],
    'customer_table': ['raw_import_file'],
}

# To see what a report depended on, look it up in the dictionary.
def get_sources(asset_name):
    if asset_name in lineage:
        return lineage[asset_name]
    return []

print('Sources of sales_report:', get_sources('sales_report'))
```


## The same thing, with a framework

```python
In real Unity Catalog, you do not write your own permission logic. You use SQL or the Databricks API. For example, to grant read access to a table:

GRANT SELECT ON TABLE main.default.customer_table TO `bob@example.com`;

And to see the audit log, you query the system table:

SELECT * FROM system.access.audit WHERE action_name = 'read' AND asset = 'customer_table';

Underneath, that SQL executes the same logic as our Python: it checks the permissions dictionary, logs the request, and returns the result.
```


## What the code does

The code defines a dictionary called `assets` with three entries. Each entry is itself a dictionary with keys 'owner' and 'readers'. Then it defines an empty list `audit_log` and a function `can_read`. The function first checks if the asset exists; if not, it returns False. Then it looks up the metadata dictionary and checks if the requester is the owner. If not, it checks the 'readers' list for membership. The request is simulated by setting `request_username = 'bob'` and `request_asset = 'customer_table'`, then calling `can_read`. The result is appended to `audit_log` as a tuple. The `lineage` dictionary maps an output to a list of inputs. The function `get_sources` simply returns that list. This is a much-simplified version of what Unity Catalog does at scale: it stores permissions, logs access, and maintains a graph of what depends on what.


## Common misconceptions

**Unity Catalog is just a folder organizer** — It's like a better file browser; it just makes it easier to find tables.

If it were a folder organizer, it would not enforce permissions. A folder browser shows you what files exist; it does not check your badge before showing them. Unity Catalog's core is the permission check and the audit log, not the navigation. In fact: Unity Catalog is an access control and audit system. It has a nice interface for discovery, but every read goes through a permission check and is logged.

**If I can see the table, I can access it** — If a table shows up in my search, I should be able to query it.

Discoverability and access are separate. In Unity Catalog, you might see that a table exists (discovery) but not have SELECT permission. The catalog will return a 'permission denied' error when you try to read it. In fact: Discovery lists what exists; access control decides who can read. You may need to request a grant even if you can see the table's name.

**Lineage shows two numbers are equal** — Lineage tells me that the number in this report is exactly the same as the number in another report.

Lineage suggests dependency, not equality. A report might use the same sales table but apply a different filter or a different aggregation (e.g., only last month vs. all years). The numbers will differ. In fact: Lineage tells you which upstream data a value was derived from, not that it is numerically identical. You still need to understand the transformations.

**Governance makes data unusable** — If I add governance, analysts will never get access to anything; it's just roadblocks.

If governance blocks all access, people find ways around it (copying to unsanctioned locations), which is worse. Good governance is designed for safe access, not no access. Unity Catalog supports fine-grained permissions and self-service requests. In fact: Governance should permit use without over-exposure. It is a risk trade-off, not a prohibition. If it is too strict, it loses adoption and becomes ineffective—your own notes say exactly that.

**The model registry is just a folder of weights** — The model registry is simply where I save my model files, like a Dropbox for .pt files.

A model registry under Unity Catalog is more: it has metadata (owner, version, stage), permission controls (who can read or approve), and lineage (which dataset trained it). It is a governed asset. In fact: A governed model registry is an asset with controlled access and a full history. 'The model registry is a governed asset rather than a folder' is a key point in your notes.


## In short

Unity Catalog is a single governance layer for a lakehouse. It centralises access control (who can read what), lineage (what a output depended on), and discovery (what assets exist). It spans tables, files, models, and functions, so the same permission logic applies everywhere. It logs every access, enabling audit. It turns the model registry into a governed asset, not a folder. Good governance is granular enough to allow work, but it never becomes an obstacle so permissive that people bypass it.


## Where this is used

Unity Catalog is used in most AI and data work on Databricks, and the same ideas appear in other governance systems like AWS Lake Formation or Azure Purview. The mechanism is always the same: a central service that stores permissions, logs access, and records lineage.

**In large language models (LLMs)** — When you train or fine-tune an LLM on proprietary data, the training datasets are tables in the lakehouse. A data scientist needs SELECT permission on those tables, but the security team wants to know who touched them. Unity Catalog grants the SELECT role to the training job's service principal (a technical account), and every read is logged. The model itself, after training, is registered in the model registry that is governed by Unity Catalog. The model's version, its owner, and its permissions are all catalog entries. To serve the model, the serving endpoint checks the caller's permissions against the catalog. If a new analyst wants to query the model, an admin runs `GRANT USAGE ON MODEL main.default.sales_model TO analyst`. Without governance, you would have exported the data to a shared folder and lost all tracking.

**In small language models on devices (TinyML)** — On a phone or laptop, there is no central catalog; the model is just a file. However, the pipeline that produces that file lives in the cloud. The training data, the quantised conversion, and the validation results are all catalogued. The model's lineage tells you which version of the training data produced this exact mobile binary. If a data breach happens in the source database, you can query lineage to find every model that used that data and revoke their serving permissions in one `REVOKE` statement. Without governance, you would have to manually search every device firmware release.

**In retrieval-augmented generation (RAG)** — A RAG pipeline retrieves documents from a vector database. Those documents often come from source tables (PDF, Confluence, etc.). Unity Catalog governs the source tables, and when you build an index, the index is stored as a table too. The retrieval step at runtime reads from that index table, and that read is logged. If you have a policy that only certain documents may be used for answering, you can enforce it by granting SELECT on only the allowed source tables. The lineage from an answer back to its source documents is visible in the catalog. When a user asks for a source, the system can show which catalog entries were involved. If a document must be expired due to a retention policy, you delete its row from the index table, and Unity Catalog ensures the underlying source is also deleted, and the audit log records it.

**In agentic systems (agents that plan and call tools)** — Agents call functions. In Unity Catalog, you can register Python functions as assets. Each function has an owner and set of users who can invoke it. An agent that needs to query a database must be granted permission to that function. When the agent calls the function, that call is logged. The audit trail shows exactly which tool the agent used and when. If the agent is allowed to call a function that returns customer names, but not a function that returns credit scores, Unity Catalog enforces that. The agent code itself is also versioned and governed. This prevents an agent from making an unapproved SQL query, because the SQL is wrapped in a registered function with a permission check.

**In everyday applied ML (recommendation, forecasting, fraud detection)** — A recommendation model reads user interaction tables; a forecasting model reads sales tables; a fraud detection model reads transaction tables. Each of those tables is governed. The data scientist who builds the model needs SELECT; the model gets deployed to a serving endpoint that reads a feature table. All reads are logged. If a fraud model is suspected of using a deprecated field, lineage will show which tables it read, and you can find all models that used that table. The model registry itself is a governed catalog, so you can control who can change model versions and who can approve a production deployment. The same for feature store tables: granting access to a feature table is a catalog operation.

**What it breaks when done badly** — If you did not have a governance layer, you would have each team writing its own scripts to copy data to local folders, Zip files, and shared drives. People would download the data to their laptop, build models, and lose track of the source. When an auditor asks 'who accessed this personally identifiable information', you would have no answer. When a data source changes or is deleted, models would silently break because there is no lineage to tell you what depends on it. Governance that is too strict, with no self-service, gets bypassed; that is why Unity Catalog allows fine-grained grants and sharing, so people can get access through a request workflow rather than copying data wholesale. The key is that access control must be granular enough to permit use without over-exposure, as your notes say.


---

Read this on the web: https://basavapurushottam.com/learn-ai/unity-catalog-governance
