Overview / Resources
Resources for working with data
A glossary, a checklist for preparing an export safely and a guide to reading a change log. Free to use and share inside your organisation.
Checklist
Before you export anything
- Write the question the data must answer in one sentence.
- List the columns that question needs, and remove the rest.
- Remove names, contact details and free-text notes unless they are essential for matching.
- Note the date range and any filters applied in the source system.
- Record how many rows the source system says it exported.
- Save in CSV with a header row; avoid merged cells and colour coding as meaning.
- Write down any local rules people follow, such as "we leave the field blank for trade customers".
- Agree a secure transfer route before sending; never use an open email attachment for personal data.
Glossary
Terms in plain English
- Profile
- A description of a dataset's columns: types, blanks, distinct values and inconsistencies.
- Canonical value
- The single agreed spelling or code that variants are mapped to.
- Exception list
- Records that failed a rule and were set aside for a human decision.
- Change log
- The record of every rule applied and how many rows it affected.
- Reconciliation
- Checking that counts or totals match between the source and the output.
- Grain
- What one row represents: an order, an order line, a customer per month. Mixing grains causes double counting.
- ISO 8601
- The international date format YYYY-MM-DD, which sorts correctly and cannot be misread as month-first.
- Data minimisation
- Using only the personal data that a task genuinely needs.
Guide
Reading a change log
Each line of a change log names a rule, the column it touched and the number of rows affected. Read it top to bottom: rules are listed in the order applied, because order matters. A large count on a rule you expected to be rare is worth a question. A rule that affected zero rows can usually be retired. If totals in and out do not reconcile, the log should explain the difference line by line.
Need a practical aid? The column profiler gives you a quick first profile of a small sample.