Webinar: Solr & Fusion for Big Data

Editor's Notes

1. I don’t have to tell you that big data is a popular topic of discussion in IT circles these days. What I want to talk about today is how Solr and Lucidworks Fusion fit into the big data landscape. Don’t worry, I’ll try to keep the hype and grandiose statements to a minimum. I will get technical in a few places because it’s important to understand the details. Search is a critical component of any big data strategy Fusion & Solr are first-class citizens in the Hadoop ecosystem Big data doesn’t have to be hard – Fusion makes it easy
2. Search engines contain mission-critical data and are typically on the front-line, directly serving users
3. Before I was a Solr committer, I was a Solr user, one of the first adopters of SolrCloud actually. I worked on a team that built and supported a big data framework built on Hadoop, Storm, Cassandra, Solr, and Postgres. Effectively, we computed performance metrics for brands by analyzing social media data IT organizations are consolidating data infrastructure for improved ROI, efficiency, security, and governance. Solr is included as part of Cloudera, Hortonworks, and MapR Hadoop distributions Users get search because they see it everyday; BI / dashboards / SQL are powerful, but not necessarily intuitive Vast amount of exhaust created by users interacting with searchable content Often times, it’s a small department in a larger organization that uses search to expose medium data to deliver business insights and then the “search engine” evolves into an insights engine on larger and larger data sets. If you could plan for all the possible queries you need to serve, then traditional BI / data warehousing techniques will still serve you well. Search fills the void where users need fast, ad hoc query capabilities to do exploratory analysis.
4. Let’s imagine we have time-ordered data, such as logs of user activity. You can insert any scale that fits your needs here. We work with customers that have a billions log events per day up to 10’s of billions. Let’s work through a quick example to illustrate some of the questions that come up and how we tackle them at Lucidworks Bunch of log data HDFS, want to index it for ad hoc queries and basic visualizations, i.e. the kinds that you can power with simple analytical functions like faceting First thing we have to identify is what data are we indexing? where is it coming from? how much data is there? how quickly do we need it to be indexed? But wait … step back a sec … how are people going to search this data? There are three important decisions emerge when designing your search solution: data partitioning scheme (time-based: hourly, daily, 15-minutes, etc) doc values: fields you need to sort and facet on should have doc values range queries need trie-fields to be indexed what fields must be stored / indexed What type of visualizations make sense for this data? What type of aggregations do you want to perform and at what time granularity? So here we’re starting to see some of the same considerations when designing data warehouse, i.e. there’s no free lunch, esp. at scale
5. The key-takeaway here is that you use your investment in Hadoop to scale complex document processing using Fusion pipelines by running a pipeline in each map or reduce task