Our angle, and it is a modest one: we are Geonode and we sell proxies to people who collect data from the web, so we see a lot of datasets at the moment they are created. The observation worth passing on is that the expensive mistakes happen at collection time and are discovered months later. Not writing down when the data was gathered, not recording which fields could be missing, not keeping the raw responses — none of these hurt on day one, and all of them make a dataset unusable when someone asks a question you did not anticipate. Nothing in this article requires buying anything; the good habits are free and most of them take minutes.
The Basic Structure
Most datasets share the same shape, whatever the format.
Records — the individual items. Rows in a table, objects in a JSON array, files in a directory. One record is one thing you are describing: a person, a transaction, a product, a photograph.
Fields — the attributes of each record. Columns in a table, keys in an object. Name, price, date, category.
Values — what a given field holds for a given record.
Schema — the description of which fields exist, what types they hold, and which are required. Sometimes formal and enforced, sometimes an informal understanding, and occasionally nothing at all — which is the case that causes trouble later.
Metadata — data about the dataset itself. When it was collected, by whom, from where, under what licence, with what known limitations.
The last one is the one beginners skip and experienced practitioners insist on. A dataset without metadata is a set of numbers with no way to judge whether they answer your question.
Structured, Semi-Structured and Unstructured
A useful three-way distinction, because it determines what you can do without preprocessing.
Structured data has a fixed schema and consistent types. A database table, a CSV with a stable header, a spreadsheet. You can query it, filter it and aggregate it directly.
Semi-structured data has organisation but not a rigid schema. JSON where records may have different fields, XML, log lines. There is structure to parse, and it varies between records.
Unstructured data has no inherent record structure. Text documents, images, audio, video. It is not that there is no information; it is that extracting fields from it requires a model or a human.
The boundaries blur in practice. A directory of images with a CSV listing filenames, dimensions and labels is unstructured content with a structured index — which is the standard arrangement for machine learning datasets and a good pattern generally.
Most real work involves converting between these. Scraping turns unstructured web pages into structured records; the conversion is where most of the errors enter, and it is why keeping the raw source matters.
Formats and What They Cost You
The choice matters more than it appears.
| Format | Structure | Types preserved | Streams | Good for |
|---|---|---|---|---|
| CSV | Flat table | No — everything is text | Yes | Spreadsheets, database loading |
| JSON | Nested | Yes | No, needs whole document | APIs, config, nested records |
| JSON Lines | Nested per line | Yes | Yes | Collection, logs, event streams |
| Parquet | Columnar, typed | Yes, exactly | Partly | Analytics, archival, large data |
| SQLite | Relational | Yes | Query-based | Portable relational data |
Three points that decide most cases.
CSV has no formal specification. RFC 4180 says so about itself, describing what "seems to be followed by most implementations". That is the source of the delimiter, encoding and quoting problems everyone has met. It is also compact and universally readable, which is why it persists.
JSON Lines is under-used and usually right for collection. One JSON document per line: it streams in constant memory, appends safely, and a truncated file still yields every complete record. A JSON array does none of those, and a crashed collection job leaves an unparseable file.
Parquet is the right destination for anything large and analytical. Columnar storage means a query touching three of forty columns reads only those three, compression is far better than text because similar values sit together, and types are preserved exactly. It is not human-readable, which is the trade.
A sensible pipeline collects to JSON Lines because it tolerates interruption, then converts to Parquet in batches for analysis. We compared the text formats in detail in JSON vs CSV.
What Makes a Dataset Usable: FAIR
The scientific community formalised this, and the framework applies well beyond research.
The FAIR principles, published in 2016, set out four properties.
Findable. F1 requires that "(Meta)data are assigned a globally unique and persistent identifier"; F2 that "Data are described with rich metadata"; F3 that metadata "clearly and explicitly include the identifier of the data they describe"; and F4 that they are "registered or indexed in a searchable resource".
Accessible. A1 requires retrieval "by their identifier using a standardised communications protocol". A2 is the one people find surprising and is arguably the most valuable: "Metadata are accessible, even when the data are no longer available." The description of a dataset should outlive the dataset, so that someone reading a paper years later can tell what was used.
Interoperable. I1 asks for "a formal, accessible, shared, and broadly applicable language for knowledge representation"; I2 for vocabularies that themselves follow FAIR; I3 for "qualified references to other (meta)data".
Reusable. R1 requires that data are "richly described with a plurality of accurate and relevant attributes".
Translated into practice for an ordinary project: give your dataset a stable identifier, write down what it contains and where it came from, use standard field names and units where they exist, state the licence, and keep the documentation even if you delete the data.
Documenting a Dataset
The FAIR principles say documentation matters. "Datasheets for Datasets" says what to write.
That 2018 proposal borrows from the electronics industry, where every component ships with a datasheet. It argues that each dataset should be accompanied by documentation covering "its motivation, composition, collection process, recommended uses, and so on", in order to "facilitate better communication between dataset creators and dataset consumers, and encourage the machine learning community to prioritize transparency and accountability".
The practical checklist that falls out of it:
Why does this exist? What question was it collected to answer. This determines whether it answers yours.
What is in it? Records, fields, types, units, and what counts as missing.
How was it collected? Method, dates, sources, sampling. A dataset scraped from one site in one week is a different object from one aggregated over a year.
What is it not? Known gaps, biases, populations excluded, periods missing. The single most valuable section and the one most often absent.
How should it be used, and how not? Intended applications and known unsuitable ones.
What is the licence? And who to contact.
How is it maintained? Whether it will be updated, and how versions are identified.
Writing this takes an hour when the dataset is fresh and is close to impossible eighteen months later when the person who collected it has left.
Judging Quality
Six dimensions, and each has a check you can actually run.
Completeness. How much is missing, and is the missingness random? Count nulls per field. A field that is 40% empty is telling you something — either a collection failure or a genuine optionality you should document.
Accuracy. Do the values reflect reality? Hard to verify in general, tractable in specifics: check a random sample against the source by hand. Twenty records takes fifteen minutes and finds most systematic errors.
Consistency. Do the same things appear the same way? Dates in mixed formats, countries as both UK and United Kingdom, prices with and without currency symbols. Count distinct values per categorical field — a field with three hundred distinct "countries" has a normalisation problem.
Timeliness. When was it collected, and does that matter for your question? Prices from last quarter are history rather than data.
Representativeness. Does the sample match the population you care about? A dataset of reviews is a dataset of people who write reviews.
Provenance. Can you trace each record to its source? This is what lets you re-derive, audit and correct — and it is why keeping raw responses alongside parsed records is worth the storage.
Run these before analysis, not after. Discovering a normalisation problem in a chart is considerably more expensive than discovering it in a value count.
Splitting Data for Machine Learning
If the dataset is for training a model, the split matters as much as the data.
Training set — what the model learns from. Usually the majority. Validation set — used to tune and choose between models. Test set — held out entirely, used once, to estimate real-world performance.
Three ways this goes wrong, all common.
Leakage. Information from the test set influences training. Scaling or imputing using statistics computed over the whole dataset before splitting is the classic version, and it inflates your results silently.
Duplicate records across splits. Near-identical items in both training and test mean the model has seen the answer. Web-collected data is particularly prone to this, since the same content appears at multiple URLs.
Temporal leakage. For time-ordered data, a random split lets the model learn from the future. Split by date instead.
The general principle: the test set should resemble the situation you will actually face. If you will predict tomorrow from today, split by time. If you will see new users, split by user.
Licensing: The Part That Decides What You Can Do
A dataset you cannot legally use is not a dataset you have. Licensing is checked far less often than it should be, usually because it is boring right up to the point where it is the only thing that matters.
Open data licences. Creative Commons is the common family. CC0 places work in the public domain as far as the law allows and imposes no conditions. CC BY requires attribution. CC BY-SA adds a share-alike condition, meaning derivatives must carry the same licence — which can be incompatible with a commercial product. CC BY-NC prohibits commercial use, and "non-commercial" is defined loosely enough to be a genuine risk if your use is ambiguous. Government portals often use bespoke open licences that are permissive in practice; read them once rather than assuming.
Database rights are separate from copyright. In the EU and UK, a sui generis right protects substantial investment in obtaining, verifying or presenting a database's contents — independently of whether any individual record is copyrightable. A dataset of bare facts can still be protected as a database, which is precisely the situation aggregation projects find themselves in.
Terms of service are contractual. A dataset collected from a site whose terms prohibit automated access carries that problem regardless of what the individual records are. This is a different question from copyright and it applies even where the facts themselves are not protected.
Personal data brings its own regime. If records identify people — directly or in combination — data protection law applies to your holding and processing of them, not merely to their collection. That means a lawful basis, retention limits, and rights that data subjects can exercise against you. "It was publicly visible" is not a lawful basis on its own.
And derived datasets inherit constraints. Training a model on a share-alike dataset, or aggregating several sources with different licences, produces obligations that are the union of the inputs rather than the most permissive one.
The practical habit is to record the licence in the dataset's documentation at the moment of collection, alongside a link to the terms as they read that day. Licences change, terms pages get rewritten, and being able to show what you agreed to when you collected is worth more than remembering it.
Where Datasets Come From
Five sources, in descending order of how much work each requires.
Published open datasets. Government portals, research repositories, institutional archives. Free, documented, and frequently good enough. Check first — the amount of re-collection of data that already exists publicly is considerable.
APIs. Structured, sanctioned, stable. If a source publishes one, it is almost always the right route.
Commercial data providers. Licensed, supported, and priced accordingly. Frequently cheaper than building the same thing once engineering time is counted.
Your own systems. Logs, transactions, telemetry. Usually the most valuable data available to an organisation and the most neglected.
Web collection. What we sell into. Appropriate when the data is publicly visible and no sanctioned route exists — and it carries obligations: robots.txt, terms of service, copyright, database rights and, where personal data is involved, data protection law. It is also the source that most requires documenting, because a scraped dataset without a record of when and from where is very hard to defend or reproduce.
People Also Ask
What is a dataset in simple terms?
A collection of related data organised so it can be worked with as a unit — typically records (the items) with fields (their attributes), plus a description of what the fields mean. A spreadsheet, a database table and a folder of labelled images are all datasets.
What is the difference between data and a dataset?
Data is the raw material; a dataset is a bounded, organised collection of it assembled for a purpose. The organisation and the boundary are what make it usable — you can count a dataset's records, describe its fields and state where it came from.
What are structured and unstructured data?
Structured data has a fixed schema and consistent types, like a database table. Unstructured data has no inherent record structure — text, images, audio. Semi-structured sits between, with organisation but variable shape, like JSON or log files.
What format should I use for a dataset?
CSV for flat tables going into spreadsheets or a database loader. JSON Lines for anything collected incrementally, because it streams and survives truncation. Parquet for large analytical data, because columnar storage and typing make queries far cheaper.
What makes a dataset good quality?
Completeness, accuracy, internal consistency, timeliness, representativeness of the population you care about, and traceable provenance. Each has a concrete check — null counts, a hand-verified sample, distinct value counts per categorical field.
How should I document a dataset?
Record why it exists, what it contains, how and when it was collected, what its known gaps and biases are, how it should and should not be used, its licence, and how it is maintained. The "Datasheets for Datasets" proposal is the standard reference for this.
What are the FAIR principles?
Findable, Accessible, Interoperable, Reusable — a framework for data management published in 2016. The most under-appreciated requirement is that metadata should remain accessible "even when the data are no longer available", so a description outlives the dataset itself.
How do I split a dataset for machine learning?
Into training, validation and held-out test sets, with the test set used once. Compute any transformations from the training set only, remove near-duplicates across splits, and split by time rather than randomly for time-ordered data — all three are common sources of silently inflated results.
Wrapping Up
A dataset is records, fields and values, plus the description that makes them mean something. The description is the part that determines whether anyone can use it in six months, including you.
The formats matter less than the habits. CSV where it fits, JSON Lines for anything you are collecting incrementally because it survives an interrupted job, Parquet when the data is large and the queries are analytical. Any of these works; the failure mode is choosing a JSON array for a long-running collection and finding an unparseable file after a crash.
What repays the most effort is documentation, written while the dataset is fresh. Why it exists, how and when it was collected, what is missing and what it should not be used for. The FAIR principles put it formally, the "Datasheets for Datasets" proposal gives you a checklist, and both point the same way: metadata should outlive the data.
And check quality before you analyse. Null counts per field, distinct values per categorical field, and twenty records verified by hand against the source will find most systematic problems in under an hour — which is a great deal cheaper than finding them in a conclusion.