Scraping Government and Public Records for Business Leads
On this page
Why Government Data Is Valuable for Lead Generation
Government databases contain business information that is publicly available, generally accurate and regularly updated. Unlike third-party data providers, government sources are authoritative: a business registration is a legal record, not a web crawl.
Types of business data available through government sources:
| Data type | What it contains | Why it matters for prospecting |
|---|---|---|
| Business registrations | Company name, registered agent, address, formation date, status | Identifies new businesses and active companies |
| Professional licences | Licence holder name, licence type, address, status | Identifies professionals in regulated industries |
| Building permits | Property owner, contractor, project type, value | Identifies construction activity and property investment |
| Government contracts | Contractor name, contract value, agency, scope | Identifies companies winning government work |
| Court filings | Parties, case type, attorney names | Identifies legal activity (use with caution) |
| Property records | Owner name, address, property value, transaction history | Identifies property owners and recent buyers |
| SEC filings (US) | Company financials, officer names, addresses | Identifies public company contacts |
| Charity/nonprofit filings | Organisation name, officers, financials | Identifies nonprofit decision makers |
| Import/export records | Company name, products, trade volume | Identifies companies in international trade |
Government Data Sources by Country
United States
| Source | URL pattern | Data available |
|---|---|---|
| Secretary of State (each state) | Varies by state (e.g., sos.ca.gov, dos.ny.gov) | Business registrations, corporate filings, registered agents |
| SAM.gov (System for Award Management) | sam.gov | Federal contract awards, entity registrations |
| USASpending.gov | usaspending.gov | Federal spending data, contractor details |
| SEC EDGAR | sec.gov/edgar | Public company filings, officer names |
| USPTO (Patent and Trademark Office) | uspto.gov | Patent and trademark filings, inventor/applicant names |
| FDA databases | fda.gov | Facility registrations, warning letters, drug approvals |
| OSHA inspection data | osha.gov | Workplace inspection records |
| State professional licence databases | Varies by state | Licensed professionals (real estate, medical, legal, construction) |
| County property records | Varies by county | Property ownership, transactions |
| State permit databases | Varies by state | Building permits, environmental permits |
United Kingdom
| Source | Data available |
|---|---|
| Companies House | Company registrations, director names, financial filings |
| Contracts Finder | Government contract opportunities and awards |
| Land Registry | Property ownership and transaction data |
| FCA Register | Regulated financial services firms and individuals |
| Charity Commission | Charity registrations, trustee names, financials |
European Union
| Source | Data available |
|---|---|
| TED (Tenders Electronic Daily) | EU public procurement tenders and contract awards |
| National company registries | Business registrations (format varies by country) |
| EU Transparency Register | Lobbying organisations and their representatives |
| EUIPO (Intellectual Property Office) | Trademark and design applications |
Australia
| Source | Data available |
|---|---|
| ASIC (Australian Securities and Investments Commission) | Company registrations, director names |
| AusTender | Australian government contract awards |
| ABN Lookup | Business registration details |
| State business licence databases | Professional and business licences |
Canada
| Source | Data available |
|---|---|
| Corporations Canada | Federal business registrations |
| Provincial registries | Provincial business registrations |
| BuyAndSell.gc.ca | Federal government procurement |
| Canada Revenue Agency (charity listings) | Registered charities |
Extracting Leads from Government Data
Step 1: Identify the right database
Match your target market to the appropriate government database:
| Target market | Best government source |
|---|---|
| New businesses | Secretary of State / Companies House (filter by formation date) |
| Construction companies | Building permit databases |
| Government contractors | SAM.gov, USASpending, Contracts Finder, AusTender |
| Licensed professionals (lawyers, doctors, real estate agents) | State/provincial professional licence databases |
| Property investors | County property records, Land Registry |
| Importers/exporters | Import/export records, customs databases |
| Nonprofits | Charity Commission, IRS Exempt Organizations (Form 990) |
| Regulated financial firms | FCA Register, SEC, state financial regulators |
Step 2: Access the data
Government data is available through several mechanisms:
| Access method | Pros | Cons |
|---|---|---|
| Web search interface | No technical skill required; immediate results | Slow for bulk data; often limited to one record at a time |
| Bulk data downloads (CSV, XML) | Complete dataset; process locally | Large files; may require cleaning |
| API access | Structured, programmable access | Not always available; may require registration |
| FOIA/FOI requests | Can access data not published online | Slow (weeks to months); may be redacted |
| Web scraping | Can extract data from search-only interfaces | Technical skill required; check terms of use |
Step 3: Extract contact information
Government records often contain:
- Business addresses (registered office, principal place of business).
- Registered agent names (the person authorised to receive legal documents).
- Officer and director names (from corporate filings).
- Contact email addresses (sometimes, particularly in permit applications and licence records).
- Phone numbers (in some databases).
For records available as downloadable files (CSV, PDF, XML), upload them to Email Extractor to extract any email addresses they contain. The tool processes CSV, PDF, XML and other common government data formats.
Step 4: Enrich with email addresses
Government records often provide names and companies but not email addresses. To find email addresses:
- Extract what is available. Some government databases include email addresses in contact fields or attached documents.
- Use the company domain. Look up the company's website from their registered address or name.
- Guess and verify. If you know the person's name and company domain, use common email patterns and verify with an email verification service.
- Check the company website. Many companies list contact information on their website that corresponds to the officers named in government filings.
Working with Specific Data Types
New business registrations
New business registrations are high-value leads for companies selling to new businesses (banking, insurance, accounting, legal services, office supplies, website design, marketing services).
How to use:
- Access your state's Secretary of State database (or equivalent).
- Filter by formation date (past 30, 60 or 90 days).
- Filter by entity type (LLC, Corporation, etc.) and/or industry classification if available.
- Export or scrape the results.
- Enrich with email addresses from the registered agent or company website.
Timing matters. New businesses make purchasing decisions quickly in their first 90 days. The sooner you reach them after formation, the more likely they are to need your service.
Government contract awards
Contract award data identifies companies that are growing and have budget:
How to use:
- Search SAM.gov, USASpending.gov or equivalent for contracts in your target industry.
- Identify the prime contractors and subcontractors.
- Note the contract value, scope and timeline.
- Research the company to find the appropriate contact.
- Reference the specific contract in your outreach.
Why this works: A company that just won a large government contract often needs to hire staff, buy equipment, engage subcontractors or purchase supporting services. Your outreach is timed to a known need.
Professional licence databases
Professional licence databases are valuable for targeting specific professions:
| Profession | Licence database | What you can sell |
|---|---|---|
| Real estate agents | State real estate commission | CRM, marketing, lead generation |
| Lawyers | State bar association | Practice management, marketing, technology |
| Doctors and dentists | State medical/dental board | Practice management, equipment, marketing |
| Contractors | State contractor licensing board | Insurance, equipment, materials, software |
| Financial advisors | SEC, FINRA, state regulators | Compliance, technology, marketing |
| Accountants | State board of accountancy | Software, marketing, continuing education |
Building permits
Building permit data identifies active construction projects:
How to use:
- Access your city or county's building permit database.
- Filter by permit type (commercial, residential, renovation), date and value.
- Identify the property owner and contractor.
- Extract contact information from the permit application.
- Reach out with services relevant to the project (materials, equipment, subcontracting, insurance, financing).
Scraping Government Websites
Legal considerations
Government data is public information, but scraping government websites has considerations:
| Consideration | Guidance |
|---|---|
| Terms of use | Some government websites prohibit automated access; check before scraping |
| Rate limiting | Government servers are often under-resourced; be polite (long delays between requests) |
| Bulk download alternatives | Many agencies offer bulk data downloads; use these instead of scraping when available |
| API availability | Check for an official API before scraping (SAM.gov, EDGAR and others offer APIs) |
| FOIA/FOI alternative | If data is not available online, a Freedom of Information request may be more appropriate than scraping |
Technical approaches
For search-only interfaces (no bulk download):
- Identify the search parameters (date range, entity type, location).
- Break the search into small batches to avoid overwhelming the server.
- Add long delays between requests (5-10 seconds minimum for government servers).
- Save raw responses before processing to avoid re-scraping.
- Parse the results to extract structured data.
For downloadable datasets:
- Download the full dataset (often available as CSV, XML or JSON).
- Load into a spreadsheet or database.
- Filter and clean the data locally.
- Extract email addresses from any text fields using Email Extractor.
Data Quality Considerations
Government data has specific quality characteristics:
| Characteristic | Impact |
|---|---|
| Generally accurate at time of filing | Business names, addresses and officer names are legal records |
| May be outdated | Businesses do not always update records when they move or change officers |
| Inconsistent formatting | Different agencies use different formats; normalisation is needed |
| May include inactive records | Filter for active/current status when available |
| Address is often a registered agent, not the business location | Registered agent addresses may be a law firm or service company |
| Not all fields are populated | Some databases have optional fields that are frequently left blank |
Compliance and Ethics
| Principle | Application |
|---|---|
| The data is public | Government records are, by definition, public information |
| Use responsibly | Just because data is public does not mean all uses are appropriate |
| Respect opt-out requests | If someone you found through public records asks not to be contacted, honour that immediately |
| Follow email regulations | CAN-SPAM, GDPR, CASL and other regulations apply regardless of data source |
| Do not misrepresent | Do not imply a government endorsement or partnership in your outreach |
| Be transparent | If asked how you found someone, be honest about the data source |
