pandas.read_html(): Full Guide with Examples

Our position: we are Geonode and we sell proxies, so tables on the public web are adjacent to our business. The relevant caution is that read_html fetching a URL directly gives you no control over the request — no custom headers, no session, no retry logic, and no way to route it through anything. For a one-off table on a cooperative site that is fine and it is the fastest thing available. For anything recurring, fetch the HTML yourself with a proper HTTP client and pass the string to read_html. That separation costs one extra line and gives you everything the built-in fetch withholds.

The Basics

import pandas as pd

tables = pd.read_html("https://example.com/data")
print(len(tables))
df = tables[0]

The critical detail, from the pandas documentation: it returns "a list of DataFrames". Not one DataFrame. A page with six tables gives you six, in document order, and tables[0] may well be a navigation layout rather than the data you wanted.

The documentation is also clear that it "will always return a list of DataFrames or fail entirely" — it will not return an empty list except in unusual cases such as "single row with <td> containing only whitespaces". So an empty result is a signal that something odd happened rather than a normal outcome.

You can also pass HTML directly, which is the form to prefer for anything beyond a quick look:

import requests
html = requests.get(url, headers={"User-Agent": "MyBot/1.0 (+https://example.com/bot)"}).text
tables = pd.read_html(html)

What It Can and Cannot See

Understanding the scope prevents most disappointment.

It reads <table> elements only. The documentation states that it "searches for <table> elements and only processes <tr>, <th> rows and <td> elements within tables". A layout built from <div> elements styled as a grid — which is most modern web design — contains no table for it to find, regardless of how table-like it looks on screen.

It does not execute JavaScript. If the table is rendered client-side, the HTML read_html sees does not contain it. This is the most common reason for "it found no tables" on a page that visibly has one.

It handles spanning correctly. colspan and rowspan attributes "are handled properly", which is more than many hand-rolled parsers manage.

It prefers <thead> for headers, and falls back to finding them in the body if there is none.

It respects display: none by default. The displayed_only=True default "excludes elements with display: none", which is usually what you want and occasionally hides data a site has deliberately made available only to certain viewports.

Two dependency notes. The parsing engines are lxml first, falling back to bs4 plus html5lib — the documentation notes that "'bs4' and 'html5lib' are synonymous" as flavour names. And there is a URL quirk worth knowing: "lxml only accepts the http, ftp and file url protocols. If you have a URL that starts with 'https' you might try removing the 's'." In practice, fetching the page yourself avoids this entirely.

The Parameters That Matter

The full signature has eighteen parameters. Six do most of the work.

match (default '.+') filters to tables whose text matches a regular expression. This is the single most useful parameter and it is under-used:

tables = pd.read_html(html, match="Population")

Instead of guessing an index into a list, you name something the table contains. Far more robust when a page's layout changes, since the content usually survives a redesign that moves the table's position.

attrs filters by HTML attributes, which is the other way to identify a specific table:

tables = pd.read_html(html, attrs={"id": "results", "class": "data"})

header names the header row. The documentation flags an ordering detail that catches people: "the header argument is applied after skiprows is applied". So if you skip two rows and then ask for header=0, you get the first row after the skip.

skiprows removes rows before parsing — useful for tables with title rows above the real header.

index_col sets a column as the index.

thousands (default ',') and decimal (default '.') handle numeric formatting. These matter more than they look: a European table using 1.234,56 needs thousands='.' and decimal=',', and without them every number silently becomes a string or a wrong value.

Two more worth knowing:

converters applies a function per column at parse time, which is cleaner than fixing types afterwards.

extract_links captures the href of links inside cells rather than only their text — genuinely useful when the table's rows link to detail pages you also want.

The Cleanup Nobody Avoids

The documentation sets expectations honestly, and it is worth reading the disclaimer rather than discovering it:

Expect to do some cleanup after you call this function. For example, you might need to manually assign column names if the column names are converted to NaN when you pass the header=0 argument.

And:

We try to assume as little as possible about the structure of the table and push the idiosyncrasies of the HTML contained in the table to the user.

That second sentence is the design philosophy stated plainly. read_html gives you what the HTML contains; making it tidy is your job.

The cleanup that comes up every time:

MultiIndex columns from spanning headers. A table with a two-row header produces a MultiIndex, which is correct and awkward. Flatten it:

df.columns = [" ".join(str(c) for c in col).strip() for col in df.columns]

Whitespace and non-breaking spaces. HTML is full of  , which arrives as \xa0 and defeats a plain .strip():

df = df.replace("\xa0", " ", regex=True)
df.columns = df.columns.str.replace("\xa0", " ", regex=False).str.strip()

Numbers that are strings. Currency symbols, percent signs and footnote markers:

df["Price"] = (df["Price"].astype(str)
               .str.replace(r"[^\d.,-]", "", regex=True)
               .str.replace(",", "")
               .pipe(pd.to_numeric, errors="coerce"))

errors="coerce" turns unparseable values into NaN rather than raising, which lets you count how many failed rather than losing the whole operation to one bad cell.

Footnote rows and totals. Many tables end with a summary row that is not data. Filter it explicitly rather than assuming the last row is safe.

Selecting the Right Table

Four approaches, in increasing order of robustness.

By index — tables[0]. Fine for exploration, fragile in a script. A new table added above yours breaks it silently, because index 0 still exists and now contains something else.

By match — the best default. Name a string that appears in the table you want and nothing else.

By attrs — best when the table has an id or a distinctive class, since those are chosen deliberately by a developer.

By shape, after loading — when nothing else identifies it:

candidates = [t for t in pd.read_html(html)
              if {"Name", "Price"}.issubset(t.columns)]
if len(candidates) != 1:
    raise ValueError(f"expected 1 matching table, found {len(candidates)}")
df = candidates[0]

That assertion is the important part. A script that silently takes the first of three matches will produce wrong data for months. Failing when the count is unexpected turns a data problem into an error message.

Reading Multiple Pages Into One DataFrame

The natural next step once a single table works, and the place where a few habits save real trouble.

Concatenate rather than appending in a loop. Building a DataFrame incrementally is slow and produces a fragmented index. Collect the frames and combine once:

frames = []
for page in range(1, 11):
    df = fetch_table(f"https://example.com/data?page={page}",
                     match="Population",
                     expected_columns=["Country", "Population"])
    df["source_page"] = page
    frames.append(df)

combined = pd.concat(frames, ignore_index=True)

Record where each row came from. The source_page column above costs nothing and answers the question you will eventually be asked — which page produced this odd value. For anything collected over time, add a timestamp too. A dataset without provenance is very hard to debug and impossible to audit.

Pace the loop. Ten pages fetched as fast as the connection allows is a burst that looks like an attack from the server's side. A one-second pause between requests is both polite and, on most sites, the difference between finishing and being rate-limited:

import time, random
time.sleep(1 + random.random())

Handle failures per page rather than abandoning the run. One page that changed layout should not lose you the other nine:

frames, failures = [], []
for page in range(1, 11):
    try:
        frames.append(fetch_table(url_for(page), match="Population", expected_columns=COLS))
    except Exception as exc:
        failures.append((page, str(exc)))

if failures:
    print(f"{len(failures)} pages failed:", failures)

Check that the shapes agree before concatenating. If page seven has a column the others do not, pd.concat will happily produce a frame full of NaN for the mismatched rows — valid, well-formed and wrong. Comparing column sets before combining turns that into an error you can see.

And deduplicate afterwards. Paginated tables frequently repeat rows across page boundaries, particularly when the underlying data changes mid-crawl. combined.drop_duplicates() on a meaningful subset of columns is a cheap guard against counting the same record twice.

When to Use Something Else

read_html is a convenience wrapper. Four situations call for a different tool.

When the data is not in a <table>. Card layouts, definition lists, <div> grids. Use a real parser — lxml or BeautifulSoup with CSS selectors or XPath — and build the DataFrame yourself:

from lxml import html as lh
tree = lh.fromstring(page)
rows = [{"name": c.cssselect("h3")[0].text_content().strip(),
         "price": c.cssselect(".price")[0].text_content().strip()}
        for c in tree.cssselect("div.product-card")]
df = pd.DataFrame(rows)

When the page needs JavaScript. Render it first with Playwright or a similar tool, then pass the rendered HTML to read_html. That combination works well and is frequently the shortest route:

html = page.content()          # rendered DOM, not source
tables = pd.read_html(html)

When you need the request controlled. Headers, cookies, sessions, retries, timeouts, proxies — none of which read_html exposes when it fetches for you. Fetch separately and pass the string.

When the site offers structured data. A CSV download, an API, or JSON-LD embedded in the page. Any of these beats parsing rendered HTML for stability, and it is worth thirty seconds checking before writing anything.

Doing It Properly in a Script

The pattern for anything that runs more than once.

import pandas as pd
import requests

HEADERS = {"User-Agent": "AcmeDataBot/1.0 (+https://acme.example.com/bot)"}

def fetch_table(url, match, expected_columns):
    resp = requests.get(url, headers=HEADERS, timeout=30)
    resp.raise_for_status()

    tables = pd.read_html(resp.text, match=match)
    if len(tables) != 1:
        raise ValueError(f"{url}: expected 1 table matching {match!r}, got {len(tables)}")

    df = tables[0]
    df.columns = [str(c).replace("\xa0", " ").strip() for c in df.columns]

    missing = set(expected_columns) - set(df.columns)
    if missing:
        raise ValueError(f"{url}: missing columns {missing}; got {list(df.columns)}")

    if df.empty:
        raise ValueError(f"{url}: table matched but contains no rows")

    return df

Five things in there are the difference between a script that fails loudly and one that fails quietly.

raise_for_status() catches HTTP errors, because read_html on an error page will either find no tables or find the wrong ones.

match rather than an index, so a layout change produces an error rather than the wrong table.

Asserting exactly one match, so an ambiguous result stops rather than silently taking the first.

Checking the expected columns, which catches a redesign that renames or reorders them — the failure that otherwise produces well-formed wrong data indefinitely.

Checking for emptiness, since a matched table with no rows is almost always a symptom rather than a result.

An honest user agent with a contact URL costs nothing and makes you a client an operator can choose to allow rather than one they have to block.

People Also Ask

What does pandas.read_html do?

It parses HTML and returns every <table> element it finds as a list of DataFrames. It handles colspan and rowspan, uses <thead> for headers where present, and by default excludes elements hidden with display: none.

Why does read_html return a list?

Because a page can contain any number of tables, and pandas returns all of them in document order. It always returns a list or fails — it will not return an empty one except in unusual cases — so an empty result is itself a signal that something went wrong.

Why does read_html say "No tables found"?

Either the page has no <table> elements — modern layouts frequently use styled <div> elements instead — or the table is rendered by JavaScript that pandas does not execute. Check the raw HTML source rather than the browser's rendered view to tell which.

How do I select a specific table?

Use match with a regular expression matching text inside the table you want, or attrs to filter on an id or class. Both are far more robust than indexing into the list, which breaks silently when a table is added above yours.

Does read_html work with JavaScript-rendered pages?

No. It parses HTML and does not execute scripts. Render the page first with a browser automation tool, then pass the resulting HTML string to read_html — that combination works well and is usually the shortest route.

How do I clean up the DataFrame afterwards?

Expect to flatten MultiIndex columns from spanning headers, strip non-breaking spaces that arrive as \xa0, convert currency and percentage strings to numbers with pd.to_numeric(errors="coerce"), and remove footnote or total rows. The pandas documentation says explicitly to expect cleanup.

Can I use a proxy or custom headers with read_html?

Not when it fetches the URL itself — it exposes no request options. Fetch the page with requests or another client, where you control headers, timeouts, sessions and proxies, then pass the HTML string to read_html.

What do the thousands and decimal parameters do?

They tell pandas how numbers are formatted, defaulting to ',' and '.' respectively. For European formatting such as 1.234,56 you need thousands='.' and decimal=',' — without them, values are silently mis-parsed or left as strings.

Wrapping Up

read_html is a genuinely good convenience function with a narrow scope: it finds <table> elements in HTML you give it and turns them into DataFrames. Within that scope it handles the awkward parts — spanning cells, header detection, hidden elements — better than most hand-written parsers.

The two things to internalise are that it returns a list rather than a DataFrame, and that the documentation itself tells you to expect cleanup. Selecting by match or attrs rather than by index, and asserting that exactly one table matched, converts the most common silent failure into an error message.

For anything running more than once, fetch the HTML yourself. One extra line buys you headers, timeouts, retries, sessions and everything else the built-in fetch withholds — and it sidesteps the lxml URL-protocol quirk at the same time.

And when the page has no tables, stop reaching for parameters. A <div> grid or a client-rendered page is a different problem, and the answer is a real parser, a rendering step, or — best of all — the structured data the site is probably already publishing somewhere you have not looked.