Data Reconciliation for Email Lists: Merging and Resolving Conflicts

What Data Reconciliation Solves

When email lists come from multiple sources (CRM exports, web scraping, event lists, purchased data, manual collection), the same person often appears in multiple lists with inconsistent information. Data reconciliation is the process of identifying these overlaps, resolving conflicts and producing a single, accurate record for each contact.

Without reconciliation, you get:

• Duplicate outreach. The same person receives your email from two different campaigns, sometimes on the same day.
• Conflicting data. Your CRM says a contact works at Company A; your event list says Company B (they may have changed jobs).
• Inflated metrics. Your "10,000-contact list" may contain 7,000 unique email strings after address deduplication; the number of people requires separate identity reconciliation.
• Compliance risk. A contact who unsubscribed from one source still receives email from another.

The Reconciliation Process

Step 1: Normalise all sources

Before comparing records across sources, standardise the format:

def normalise_email(email):
    """Normalise an email address for matching."""
    if not email:
        return None
    email = email.strip().lower()
    # Handle common typos
    email = email.replace(" ", "")
    email = email.replace("..", ".")
    return email

def normalise_company(company):
    """Normalise company name for fuzzy matching."""
    if not company:
        return None
    company = company.strip()
    # Remove common suffixes
    for suffix in [
        " Inc.", " Inc", " LLC", " Ltd.", " Ltd",
        " Corp.", " Corp", " Co.", " Co",
        " GmbH", " S.A.", " B.V.", " Pty Ltd",
    ]:
        if company.endswith(suffix):
            company = company[: -len(suffix)]
    return company.strip()

Step 2: Identify matches

Matching records across sources requires comparing on multiple fields, because no single field is reliable alone.

Match keys (in order of reliability):

Match strategy:

def find_matches(record, existing_records):
    """Find potential matches for a record in existing data."""
    matches = []
    email = normalise_email(record.get("email"))

    # Tier 1: Exact email match
    for existing in existing_records:
        if normalise_email(existing.get("email")) == email:
            matches.append(("exact_email", existing))
            return matches  # Exact email is definitive

    # Tier 2: Name + company match
    name = f"{record.get('first_name', '')} {record.get('last_name', '')}".strip().lower()
    company = normalise_company(record.get("company"))

    for existing in existing_records:
        existing_name = f"{existing.get('first_name', '')} {existing.get('last_name', '')}".strip().lower()
        existing_company = normalise_company(existing.get("company"))

        if name and existing_name and name == existing_name:
            if company and existing_company and company.lower() == existing_company.lower():
                matches.append(("name_company", existing))

    return matches

Step 3: Resolve conflicts

When two records match but contain different information, you need conflict resolution rules:

Resolution strategies:

Source priority example:

Conflict resolution logic:

def resolve_conflict(field_name, values_by_source):
    """Resolve conflicting values for a field using source priority."""
    SOURCE_PRIORITY = {
        "crm": 1,
        "linkedin": 2,
        "verification": 3,
        "conference": 4,
        "scraping": 5,
        "purchased": 6,
    }

    # Sort by source priority (lowest number = highest priority)
    sorted_values = sorted(
        values_by_source.items(),
        key=lambda x: SOURCE_PRIORITY.get(x[0], 99),
    )

    # Return the highest-priority non-empty value
    for source, value in sorted_values:
        if value and str(value).strip():
            return value, source

    return None, None

Step 4: Merge records

After resolving conflicts, merge matched records into a single canonical record:

def merge_records(records_to_merge, source_priority):
    """Merge multiple records into one canonical record."""
    merged = {}
    provenance = {}

    all_fields = set()
    for record in records_to_merge:
        all_fields.update(record.keys())

    for field in all_fields:
        values_by_source = {}
        for record in records_to_merge:
            source = record.get("_source", "unknown")
            if field in record and record[field]:
                values_by_source[source] = record[field]

        if values_by_source:
            value, source = resolve_conflict(field, values_by_source)
            merged[field] = value
            provenance[field] = source

    merged["_sources"] = list(
        {r.get("_source") for r in records_to_merge}
    )
    merged["_provenance"] = provenance
    return merged

Step 5: Track provenance

After merging, the canonical record should show where each field came from. This is essential for:

• Debugging. When a field is wrong, you need to know which source provided it.
• Re-reconciliation. When you get new data from a source, you can selectively update fields.
• Compliance. Under GDPR and similar regulations, you may need to demonstrate where personal data originated.

Common Reconciliation Scenarios

Scenario: Merging CRM with a conference attendee list

The conference list has a newer job title, so the reconciled record uses the conference data for that field. The CRM has a phone number the conference list lacks, so the CRM value is kept.

Scenario: Merging scraped data with purchased data

The purchased list has a more complete name and a title that was not found during scraping. The scraped data provides a verifiable source URL.

Scenario: Same person, different email addresses

This scenario requires human judgment or additional signals (like checking LinkedIn) to confirm the match. Automated matching on name alone risks false positives.

Handling Suppression Lists During Reconciliation

Suppression lists (unsubscribes, bounces, complaints) must be respected across all sources:

Tools for Reconciliation

For small lists (under 10,000 records)

For larger lists (10,000-100,000 records)

For enterprise scale (100,000+ records)

Quality Checks After Reconciliation

After reconciling, verify the output:

Related Guides

• Removing Duplicates at Scale
• Data Normalization Workflows
• Data Quality Scoring for Email Lists
• CRM Data Hygiene Guide
• How Email Deduplication Works