---
name: data-engineering-and-analytics
description: "Data Engineering and Analytics, in 27 concepts."
version: 1.0.0
author: Learn AI
license: CC-BY-4.0
metadata:
  hermes:
    tags: [Data, Engineering, And, Analytics]
    related_skills: [llms-and-generative-ai, machine-learning]
    source: https://basavapurushottam.com/learn-ai
---
# Data Engineering and Analytics

27 concepts, written for an adult with no computer science and no mathematics past school arithmetic. Each concept is a file under `references/`; load one when it is the thing being asked about rather than reading the whole domain.


## When to Use

- The reader asks about data engineering and analytics and wants it explained from the ground up rather than summarised.

- A term below comes up and you want the plain-language account of it, with a worked example.

- You need to know what a concept rests on before teaching it — every reference lists its prerequisites.


## When NOT to Use

- The reader wants current events, papers or library documentation. This is a curriculum, not a news source, and it does not update itself.

- The reader wants code they can ship. Examples here are written to be understood, not deployed.


## How to Run

Open the file listed beside a concept in Quick Reference below. The files live in the `references` directory, are plain Markdown with no front matter and no dependencies — read the one you want and ignore the rest.


## Quick Reference


### Data Engineering

- `references/continuous-learning-loops.md` — **Continuous Learning Loops**: Pipelines feeding production outcomes back into retraining, so models improve from thei…
- `references/data-contracts.md` — **Data Contracts**: Explicit agreements between data producers and consumers covering schema, semantics, qu…
- `references/data-fabric-and-mesh.md` — **Data Fabric and Mesh**: Two architectural responses to scale: fabric unifies access through metadata and automa…
- `references/entity-resolution.md` — **Entity Resolution**: Determining when different records refer to the same real-world entity, across systems…
- `references/parquet-and-flat-file-hygiene.md` — **Parquet and Flat-File Hygiene**: Practices for durable file-based data — columnar formats, schema evolution, partitionin…
- `references/point-in-time-data.md` — **Point-in-Time Data**: Reconstructing exactly what was known at a given moment, so models are never trained on…
- `references/semantic-layer.md` — **Semantic Layer**: A shared definition layer giving business meaning to raw data, so metrics mean the same…
- `references/streaming-ingestion.md` — **Streaming Ingestion**: Continuously accepting events as they arrive, with buffering, ordering and backpressure…
- `references/text-mining-at-scale.md` — **Text Mining at Scale**: Extracting structure from large document collections, where scalability rather than acc…

### Data Governance

- `references/regulatory-compliance-controls.md` — **Regulatory Compliance Controls**: Technical and procedural controls implementing regulatory obligations — retention, acce…
- `references/unity-catalog-governance.md` — **Unity Catalog Governance**: A governance layer over a lakehouse providing centralised access control, lineage and d…

### Data Quality

- `references/constraint-generation.md` — **Constraint Generation**: Automatically discovering constraints from existing data, then enforcing them on future…
- `references/data-cleaning.md` — **Data Cleaning**: Correcting or removing inaccurate, malformed and inconsistent records so downstream ana…
- `references/data-dictionaries.md` — **Data Dictionaries**: Documentation defining each field's meaning, type, units, valid values and provenance.
- `references/data-literacy.md` — **Data Literacy**: The ability to read, interpret, question and communicate with data accurately and respo…
- `references/data-transformation.md` — **Data Transformation**: Reshaping data into the form analysis requires — type conversion, normalisation, encodi…
- `references/data-validation-constraints.md` — **Data Validation Constraints**: Explicit rules a dataset must satisfy — types, ranges, uniqueness, referential integrit…
- `references/found-data.md` — **Found Data**: Data generated for another purpose and repurposed for analysis, as opposed to data coll…
- `references/handling-missing-data.md` — **Handling Missing Data**: Deciding what to do about absent values — drop, impute, or model the missingness itself.
- `references/null-and-duplicate-handling.md` — **Null and Duplicate Handling**: Managing absent values and repeated records, the two most common structural defects in…
- `references/outlier-handling.md` — **Outlier Handling**: Identifying extreme values and deciding whether they are errors to remove or genuine ob…
- `references/regular-expression-constraints.md` — **Regular Expression Constraints**: Validating that string fields conform to expected patterns — identifiers, codes, dates,…
- `references/roll-up-constraints.md` — **Roll-Up Constraints**: Rules asserting that aggregates reconcile — that details sum to totals across hierarchies.
- `references/unicode-and-encoding-pitfalls.md` — **Unicode and Encoding Pitfalls**: Text failures caused by encoding mismatches, normalisation differences and the gap betw…
- `references/unstructured-data.md` — **Unstructured Data**: Text, image, audio and video content lacking a predefined schema, which now dominates d…

### Data Visualization

- `references/power-bi-visualization.md` — **Power BI Visualization**: Building interactive dashboards and reports over connected data models with a drag-and-…
- `references/r-in-power-bi.md` — **R in Power BI**: Embedding R scripts inside dashboards to reach statistical visualisations the native ch…


## Pitfalls

- The concepts are ordered by what rests on what, not by difficulty. A later concept is not always harder; it is later because something else has to come first.

- Explanations avoid notation the reader has not met. If you reintroduce standard symbols, say what they mean, or you undo the thing this curriculum is for.


## Verification

`ls references/ | wc -l` should print 27.


---

From [Learn AI](https://basavapurushottam.com/learn-ai). Prose is licensed CC BY 4.0.
