Structured Data Extraction: Turning Unstructured Web Pages into Usable Datasets

What Structured Data Extraction Means

Structured data extraction is the process of turning unstructured or semi-structured content (web pages, PDFs, emails, documents) into clean, organised datasets that can be analysed, stored in a database or fed into another system.

A web page is semi-structured. It has HTML markup that gives it some structure, but the data you want (a price, a name, an address, an email, a product specification) is mixed in with navigation, advertising, styling and layout elements. Structured data extraction separates the signal from the noise.

Examples of structured data extraction:

For email address extraction specifically, upload your files to Email Extractor, which pulls email addresses from 19 file formats including HTML, XML, JSON, CSV, PDF, DOCX, XLSX and more.

How Web Pages Store Data

HTML structure

HTML is a tree of nested elements. Data lives inside elements, as text content or as attribute values.

<div class="product-card">
  <h2 class="product-name">Widget Pro 3000</h2>
  <span class="price">$49.99</span>
  <p class="description">The best widget for professional use.</p>
  <a href="/products/widget-pro-3000" class="product-link">View details</a>
  <span class="availability in-stock">In Stock</span>
</div>

In this example:

• The product name is the text inside the <h2> with class product-name.
• The price is the text inside the <span> with class price.
• The product URL is the href attribute of the <a> tag.
• Availability is both the text content and the class name of the <span>.

Structured data markup

Many websites embed structured data in machine-readable formats alongside the human-readable HTML. This makes extraction much easier when it is present.

JSON-LD (JavaScript Object Notation for Linked Data):

The most common format. Embedded in a <script> tag:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "Widget Pro 3000",
  "description": "The best widget for professional use.",
  "offers": {
    "@type": "Offer",
    "price": "49.99",
    "priceCurrency": "USD",
    "availability": "https://schema.org/InStock"
  }
}
</script>

Microdata:

Embedded as attributes on HTML elements:

<div itemscope itemtype="https://schema.org/Product">
  <h2 itemprop="name">Widget Pro 3000</h2>
  <span itemprop="price" content="49.99">$49.99</span>
</div>

RDFa:

Similar to microdata but uses different attributes:

<div vocab="https://schema.org/" typeof="Product">
  <h2 property="name">Widget Pro 3000</h2>
  <span property="price" content="49.99">$49.99</span>
</div>

APIs

Some websites provide APIs that return structured data directly, making scraping unnecessary:

• Public APIs. Many sites offer free or paid APIs for their data (GitHub, Twitter/X, Reddit, many e-commerce platforms).
• Internal APIs. JavaScript-heavy websites often fetch data from internal API endpoints. Browser developer tools (Network tab) reveal these endpoints, and they often return clean JSON.
• RSS feeds. Blogs and news sites offer RSS feeds with structured article data.

Always check for an API before building a scraper. API data is cleaner, more stable and explicitly permitted.

Extraction Methods

CSS selectors

CSS selectors identify HTML elements by tag name, class, ID, attributes and position in the document tree.

Python example with Beautiful Soup:

from bs4 import BeautifulSoup

html = '<div class="product-card"><h2 class="product-name">Widget</h2><span class="price">$49.99</span></div>'
soup = BeautifulSoup(html, 'html.parser')

name = soup.select_one('.product-name').text
price = soup.select_one('.price').text

XPath

XPath is more powerful than CSS selectors. It can navigate up the tree (to parents), use conditions, and match text content.

Python example with lxml:

from lxml import html

tree = html.fromstring(page_content)
names = tree.xpath('//h2[@class="product-name"]/text()')
prices = tree.xpath('//span[@class="price"]/text()')

Regular expressions

Regular expressions match text patterns. They are useful for extracting data from unstructured text (not HTML).

Warning: Do not parse HTML with regular expressions. HTML is a nested, recursive structure that regular expressions cannot handle correctly. Use a proper HTML parser (Beautiful Soup, lxml) for HTML, and regular expressions only for flat text patterns within extracted text.

JSON-LD extraction

When a page contains JSON-LD, extraction is trivial:

import json
from bs4 import BeautifulSoup

soup = BeautifulSoup(page_content, 'html.parser')
scripts = soup.find_all('script', type='application/ld+json')

for script in scripts:
    data = json.loads(script.string)
    if data.get('@type') == 'Product':
        print(data['name'], data['offers']['price'])

JSON-LD is pre-structured. If a page has it, always extract from JSON-LD rather than parsing the HTML.

Building an Extraction Pipeline

Step 1: Analyse the source

Before writing code, understand the source:

1. Open the page in a browser. Look at the data you want to extract.
2. View the page source. Check for JSON-LD or structured data markup.
3. Inspect elements. Use browser developer tools (right-click, Inspect) to examine the HTML structure around the data.
4. Check the Network tab. Look for API calls that return the data as JSON.
5. Check robots.txt. Review the site's robots.txt for scraping policies.
6. Look for an API. Search for "[site name] API" before building a scraper.

Step 2: Build the extractor

For a single page type (e.g., all product pages on one site):

import requests
from bs4 import BeautifulSoup
import csv
import time

def extract_product(url):
    response = requests.get(url, headers={
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)'
    })
    soup = BeautifulSoup(response.text, 'html.parser')

    # Try JSON-LD first
    ld_scripts = soup.find_all('script', type='application/ld+json')
    for script in ld_scripts:
        try:
            data = json.loads(script.string)
            if data.get('@type') == 'Product':
                return {
                    'name': data.get('name'),
                    'price': data.get('offers', {}).get('price'),
                    'description': data.get('description'),
                    'url': url
                }
        except json.JSONDecodeError:
            continue

    # Fall back to HTML parsing
    return {
        'name': soup.select_one('.product-name').text.strip() if soup.select_one('.product-name') else None,
        'price': soup.select_one('.price').text.strip() if soup.select_one('.price') else None,
        'description': soup.select_one('.description').text.strip() if soup.select_one('.description') else None,
        'url': url
    }

Step 3: Handle pagination and listing pages

Most scraping targets have listing pages (search results, category pages) that link to detail pages.

def get_listing_urls(listing_url):
    """Extract product URLs from a listing page."""
    response = requests.get(listing_url)
    soup = BeautifulSoup(response.text, 'html.parser')

    product_urls = []
    for link in soup.select('.product-card a.product-link'):
        href = link.get('href')
        if href:
            full_url = urljoin(listing_url, href)
            product_urls.append(full_url)

    # Handle pagination
    next_page = soup.select_one('a.next-page')
    if next_page:
        next_url = urljoin(listing_url, next_page['href'])
        product_urls.extend(get_listing_urls(next_url))

    return product_urls

Step 4: Clean the data

Raw extracted data is messy. Clean it before storage:

Step 5: Store the data

Tools

Libraries

Managed services

No-code tools

Legal and Ethical Considerations

• Robots.txt. Check and respect the site's robots.txt file.
• Terms of service. Many sites prohibit scraping in their ToS.
• Rate limiting. Do not overload servers. Add delays between requests.
• Copyright. Extracted data may be copyrighted. Extraction for analysis and personal use is generally more defensible than republication.
• Personal data. Extracting and storing personal data must comply with GDPR, CCPA and other privacy regulations.

See Is Web Scraping Legal and Scraping Ethics Best Practices.

Related Guides

• Web Scraping for Market Research
• Building a Scraping Pipeline
• Scraping Directories for Leads
• Best Web Scraping Tools
• Regex for Email Extraction