Operating a global genealogy platform requires managing two massive, distinct engineering challenges: querying billions of unstructured historical records in real time and comparing billions of genomic data points to map human relationships. MyHeritage addresses these challenges through a cloud-first infrastructure (AWS and Microsoft Azure), high-scale Apache Solr search clusters, and a cutting-edge shift toward low-pass Whole Genome Sequencing (WGS).
This deep dive details the explicit algorithms, data storage patterns, and genetic processing methods that power the platform.
1. The Genomics Pipeline: Low-Pass WGS, Phasing, and Imputation
Historically, consumer genetic testing relied on genotyping arrays, which read roughly 700,000 specific Single Nucleotide Polymorphisms (SNPs) across the human genome. MyHeritage revolutionized this approach by partnering with Ultima Genomics to scale low-pass Whole Genome Sequencing (WGS) at 2x coverage.
The Sequencing Physics (Ultima Genomics Wafers)
Instead of traditional flow cells, the sequencing platform utilizes large, spinning circular silicon wafers adapted from the semiconductor industry.
During processing:
- Spin-Coating: Sequencing reagents are dispensed at the center of the spinning wafer and spread uniformly by centrifugal force.
- Sequencing-by-Synthesis: The system uses flow-based chemistry to build the DNA sequence.
- Optical Readout: High-speed cameras capture optical signals from active DNA clusters continuously as the wafer spins, drastically reducing the cost per gigabase of sequenced data.
Resolving the “Low-Pass” 2x Coverage
Clinical WGS requires 30x coverage (reading each base pair 30 times) to guarantee medical accuracy. For genealogical relationship matching, 30x is cost-prohibitive. Instead, MyHeritage uses 2x coverage combined with two statistical algorithms to reconstruct the missing information:
- Phasing: This step takes the unordered raw genetic reads and assigns them to their respective chromosomes of origin (maternal vs. paternal). MyHeritage implements a phasing heuristic that analyzes local haplotype structures, grouping co-inherited SNPs into physical strands.
- Imputation: Because 2x coverage leaves physical gaps in the read files, statistical imputation software compares the phased 2x scaffold against a vast reference panel of fully sequenced genomes. By identifying known haplotype blocks, the platform predicts the missing unobserved alleles with over 99% accuracy.
2. DNA Matching: Segment Detection and Chromosome Browsing
Once the genomes are phased and imputed, the matching engine looks for Identical-by-Descent (IBD) segments—stretches of DNA that are identical because they were inherited from a common ancestor.
User Genome: [---A-G-C-T-T-A-G-C-C-A---] <-- Shared Segment (IBD)
Match Genome: [---A-G-C-T-T-A-G-C-C-A---]
The Segment-Matching Algorithm
The algorithm scans genomes in parallel to identify matching blocks measured in centimorgans (cM), a unit of genetic linkage.
- The Matching Threshold: The engine searches for continuous blocks of shared DNA. Currently, the matching threshold is set to a minimum of 8 cM to balance the detection of true relatives with the elimination of false-positive matches (which typically occur on very short segments).
- Stitching: DNA extraction and phasing are prone to minor read errors. The engine utilizes a stitching algorithm that bridges minor, artificial gaps within a single long segment, ensuring that a single maternal/paternal inheritance line isn’t falsely split into separate, smaller matches.
3. High-Scale Search Architecture: Billions of Historical Records
To query more than 20 billion historical records (census documents, birth certificates, and directories), MyHeritage uses a highly optimized search tier.
The Core Tech Stack
- Web & Application Tier: Powered by an enterprise PHP application layer that has been progressively upgraded using automated AI-refactoring workflows, alongside Node.js microservices.
- Database Layer: Large relational clusters run on MySQL. To ensure 24/7 availability, upgrades and migrations on these massive production databases are executed using automated replication-lag monitors to achieve zero-downtime failovers.
- Distributed Search: Apache Solr forms the search backend. The engineering team conducts continuous, live A/B infrastructure testing on Solr clusters to balance indexing throughput against search query latency.
Exactly-Once Processing in Pipelines
Data ingested from global archives must flow into user databases without duplication or loss. MyHeritage achieves this using Apache Spark Structured Streaming. The pipelines use an Exactly-Once processing guarantee by utilizing transactional state stores and idempotent sinks. This ensures that even if a pipeline node crashes mid-ingestion, no historical records are duplicated or omitted in the target database.
[Historical Data Source]
│
▼
[Spark Structured Streaming (Idempotent Sinks)] ──► Ensures Exactly-Once Ingestion
│
├────────────────────────┐
▼ ▼
[MySQL Clusters] [Apache Solr Indexes]
(Transactional Storage) (Fuzzy & Phenotypic Search)
Smart Matching & Phenotypic Record Matching
Historical records are notoriously messy due to OCR (Optical Character Recognition) errors, spelling variations, and translation issues. MyHeritage bridges this gap using proprietary matching algorithms:
- Global Name Translation: When a user searches for a name, the engine translates the query across languages (e.g., matching a Hebrew document for “יצחק” with a search for “Isaac”).
- Record Detective: This semantic link-analysis engine matches one historical record to another. If a user views a 1910 census record, the engine analyzes the household’s structural features (relationships, ages, geographic transitions) and dynamically queries Solr to find related documents (such as a 1920 census or a death certificate) without requiring manual search queries.
4. Document Structuring via Scribe AI
Historically, extracting genealogical information from unstructured scanned records was a manual indexing bottleneck. In 2026, MyHeritage introduced Scribe AI to automate this translation pipeline at scale.
Scribe AI leverages Large Language Models (LLMs) to read unstructured text—such as historical wills, deeds, or newspaper obituaries—and map the raw text to highly structured, queryable schemas.
JSON
{
"document_type": "Will and Testament",
"testator": {
"name": "William Harrison",
"death_year": 1845
},
"relationships_extracted": [
{
"name": "Mary Harrison",
"relationship": "spouse",
"inheritance": "family estate"
},
{
"name": "Thomas Harrison",
"relationship": "son",
"inheritance": "agricultural equipment"
}
]
}
By standardizing raw, free-form text into strict JSON payloads, the data immediately becomes indexable within the Apache Solr cluster, making previously hidden family connections instantly discoverable to the platform’s millions of active users.
Also Read: Dashlane Password Manager For Complete Digital Security
