Collecting a Time Series You Can Compare: Fixed Conditions, Snapshots and Honest Gaps
A series of daily observations is only comparable if each was taken the same way. How to fix the conditions, record completeness, keep failures visible and avoid comparing across a change.
Synthetic dataExamples on this page use synthetic data created for illustration.
On this page
Some data cannot be collected later. A price, a ranking, the contents of a page: the world overwrites them, and the only way to know what they were last Tuesday is to have written them down last Tuesday. Starting a daily collection early is valuable, and doing it carelessly wastes the value. A series is useful only if each observation can be compared with the others. This article describes the habits that make a collected series comparable and honest.
Comparable means “taken the same way”
Suppose you record the price of one product every day. On day 10 the figure drops 8%. Did the price fall, or did something about the measurement change? Many things can alter a number without any change in the thing measured:
- the time of day (prices and rankings move through the day),
- the region or postcode (price, stock and delivery date depend on location),
- the language and currency of the page,
- the type of page (a list page and a detail page may show different prices),
- the account state (logged in or not, new or returning visitor),
- the version of the parser that reads the page.
Decide on one setting for each, write it down, and keep it. That setting is the series’ scope. If any part of it must change, start a new scope version and do not compare across the boundary, or compare only after checking an overlap period in which both versions were run side by side (Dynamic Pricing).
Keep the raw evidence, then parse it
Store what you received, as you received it (the page or the response, with its address, the time, and the request conditions) before extracting any fields. File formats such as WARC exist for this (WARC). Parsing can then be redone when you find a mistake or want a new field; collecting the page again would give you today’s page, not last Tuesday’s. Record the parser version with every extracted value, so a change in results can be traced to the parser or to the world.
Record completeness, every time
A collection run can succeed, partly succeed, or fail. Record which, as part of the snapshot (Data Snapshot):
- Complete: every part read (every page of the list).
- Partial: some part missing (the second page failed).
- Failed: nothing usable.
The reason is what such records prevent: a half-read list looks like a list that shrank, and you will “discover” that products left it. With a completeness flag, absence from a partial snapshot is not evidence of anything.
Keep gaps as gaps
When a day is missing, resist the urge to tidy it. Do not fill it with zero (that is a measurement of “nothing”); do not copy yesterday’s value forward (that invents a measurement); do not drop the day silently (the series then looks continuous). Record it as missing, with the reason (Missing Data), and let the analysis decide how to cope: leave it out, or use a method that handles gaps explicitly. If you later add data from another route to cover a gap, mark them as backfilled with the date they were obtained, so the series does not pretend they were collected on time (Backfill).
Two kinds of time
Every observation has two times: when it was true in the world (the moment you looked) and when it became available to your analysis (after parsing and loading). Keep both. Usually they differ by minutes, but when you reprocess old evidence with a new parser, the values for last month are available today, not last month (Bitemporal Data, Point-in-Time). Any study that replays history needs to know the difference, or it will use numbers that did not exist yet.
Check for silent changes
Build a few tests into the collection itself:
| Check | What it catches |
|---|---|
| Number of items in a list, compared with the recent median | A page redesign that breaks the parser, or a half-read list |
| Share of items with each field filled in | A field that has silently gone empty |
| Range of values (a price of 0 or 1,000,000) | A wrong field, a unit change |
| Same item seen twice in one snapshot | A paging problem (Pagination) |
| Collection time within the planned window | A scheduler that drifted |
Alarm on a sudden change, and look before you trust the next day’s data.
Make it repeatable
Each run should be safe to repeat: running a day twice must not produce two copies of its observations (Idempotency). Name each snapshot by scope, date and run; keep the evidence by content (Content-addressable Storage); and write down what the retention and licence terms allow, so the evidence is not kept longer than permitted (Data License).
A short checklist
- Scope written down: time, region, language, currency, page type, account state.
- Raw evidence stored before parsing, with time and conditions.
- Parser version recorded with every value.
- Every run marked complete, partial or failed.
- Gaps kept as gaps, with reasons; backfills labelled.
- Both times kept: when true, and when available.
- Checks that alarm on sudden change.
- A new scope version whenever the conditions change, and no comparison across it.
References
- Ralph Kimball, Margy Ross. The Data Warehouse Toolkit . Wiley, 2013. ISBN 978-1118530801.
- The WARC Format 1.1 . International Internet Preservation Consortium.
- Martin Fowler. Bitemporal History . martinfowler.com, 2021.