Data Engineering

A record store that only ever adds rows: corrections are new rows that point at the ones they replace, so history, including the mistake, stays readable.

A delivery guarantee under which every message or job arrives one or more times, never zero — so duplicates are possible and must be tolerated.

Loading or recomputing data for past periods after the fact — and recording when each value was really loaded, so history is not made to look earlier-known than it was.

Data that records two times for every fact: when it was true in the world, and when the system learned it.

A guard that stops calling a failing service after repeated errors, waits, then tests carefully before resuming.

Storing a table column by column instead of row by row, so analytical queries that need a few columns read far less data.

Storing each object under a name derived from a hash of its bytes, so identical content has one address and any change creates a new one.

A function that turns any input into a short fixed-size fingerprint: the same bytes always give the same fingerprint, and any change gives an unrelated one.

A written, versioned agreement between the producer of a dataset and the people who use it, covering structure, meaning, timing and quality.

The recorded path of a piece of data: where it came from, how it was changed, and what depends on it.

Named aspects — accuracy, completeness, consistency, timeliness, uniqueness — that turn the vague idea of clean data into checks you can measure.

A complete record of a list or table as it stood at one moment, kept as a unit with its time, scope and completeness, so later versions can be compared against it.

Deciding which records, in one source or many, refer to the same real-world thing, and linking them under one identifier.

Two orders for moving data into an analysis store: transform it first and then load it (ETL), or load it raw and transform it afterwards (ELT).

Storing every change as an immutable event and working out the current state by replaying the events.

Waiting longer after each failed attempt, usually doubling the delay and adding randomness, so retries relieve a struggling service instead of piling on.

Turning raw data into inputs a model or rule can use, such as ratios, rates of change and counts over windows. Each must be computable from what was known at the time.

Floating-point numbers are fast but cannot hold many decimal fractions exactly; money and other exact quantities are better stored as whole numbers of a small unit.

An operation is idempotent when doing it several times has the same effect as doing it once — which is what makes retries safe.

A table mapping each word to the documents that contain it, so a search engine can find matches without scanning every document.

Records that reach a pipeline after the period they describe has already been processed or reported. They are normal, so a pipeline needs a rule for them.

Values that should exist but were not recorded. Why they are missing decides whether ignoring them is harmless or biases the result.

Replacing each point of a series with the average of the points around it, to smooth short-term noise. A longer window smooths more but reacts later.

An observation far from the rest. It may be an error, a rare but real event, or a sign of two kinds of things mixed together; look before deleting.

Splitting a long list into pages fetched one at a time. How you page decides whether items can be skipped or repeated when the list changes.

Reading data as it was known at a past moment — not as it looks today after later corrections and additions.

A code that names a product so records can be matched: a seller's SKU, a global GTIN (the number under a barcode), or a marketplace's own catalogue number.

The hierarchy of categories a platform uses to organise products. It differs between platforms and changes over time, so a category means something only with its version.

One version of a product that differs in a few attributes such as size or colour, usually grouped under a parent listing. Counts may refer to the parent or to a variant.

Changing the structure of data over time while old data stays readable and existing consumers keep working.

A descriptive attribute, such as a product's category, that changes rarely. Type 2 handling keeps every version with validity dates.

Logic with a third value, unknown, besides true and false. A rule that depends on a missing value is unknown, not false.

Observations of one quantity recorded in time order. Neighbouring values are related and order matters, so time series need methods that respect time.

UTC is the world's reference time scale and a time zone is a local offset from it. Store instants in UTC; convert for display.

A marker of how far a pipeline has progressed — usually the latest time or sequence number fully processed — so the next run only asks for what comes after.