Data Collection

Bias that arises when units drop out of a study for reasons related to what is measured, so the units that remain no longer represent the original group.

A guard that stops calling a failing service after repeated errors, waits, then tests carefully before resuming.

The terms that say what you may do with a dataset or feed: keep a copy, compute from it, publish results, pass it on. Access alone grants none of these.

A complete record of a list or table as it stood at one moment, kept as a unit with its time, scope and completeness, so later versions can be compared against it.

Waiting longer after each failed attempt, usually doubling the delay and adding randomness, so retries relieve a struggling service instead of piling on.

A real web browser run without a visible window and controlled by code; it runs the page's scripts, so it can read pages assembled in the browser.

Splitting a long list into pages fetched one at a time. How you page decides whether items can be skipped or repeated when the list changes.

Observations of the same units, such as products, firms or people, repeated over time, so change can be studied within each unit.

Capping how many requests a client may make in a period. Servers apply it to protect themselves; responsible clients also impose it on themselves.

A plain-text file at a site's root in which the owner states which paths automated clients are asked not to fetch. A convention, not a lock.

The list or source a sample is actually drawn from. Whatever the frame leaves out cannot appear in the sample, however large the sample is.

A standard file format that stores the raw requests and responses of web captures together with when and how each was made.

Collecting data by having a program request web pages meant for people and pull out the fields it needs. A fallback with real fragility.