This event has ended. View the official site or create your own event → Check it out
This event has ended. Create your own
View analytic
Thursday, October 13 • 2:20pm - 3:10pm
Solr Cross-Datacenter Replication and Consistency at Scale
Replicating a SolrCloud index to multiple availability zones serves two primary purposes: redundancy for rapid disaster recovery and data locality for minimizing request latencies across different regions. These features are particularly interesting for services where SolrCloud is used as the primary data store or when availability of the search index is directly linked to uptime. In our view, a reliable cross-availability zone replication system possesses two qualities. First, it should be fault-tolerant and provide certain guarantees about data loss and availability, even in the event of a complete datacenter outage. Second, it should be capable of detecting and resolving any eventual inconsistencies. To the first of these ends, we developed a Solr plugin which uses a distributed queue to achieve non-blocking, failure-tolerant writes without compromising local indexing performance. To the second, we are currently working on a time-based Merkle tree comparison technique to detect and resolve inconsistencies during online indexing. In this talk, we will present the design of these components as well as the overall system architecture, and discuss their guarantees and limitations in the context of similar efforts in the community.


Oliver Bates

Software Engineer, Cloud Infrastructure, Apple Inc.
Oliver started his career in biomedical engineering, working on numerical modeling of complex biological systems. After several years in academia, collecting and crunching huge swaths of data, he turned his attention to distributed systems.

Thursday October 13, 2016 2:20pm - 3:10pm
Independence Sheraton Boston