Case study of Rujhaan.com (A social news app )
•Download as PPTX, PDF•
4 likes•2,441 views
Rujhaan.com is a news aggregation app that collects trending news and social media discussions from topics of interest to users. It uses various technologies including a crawler to collect data from social media, Apache Solr for search, MongoDB for storage, Redis for caching, and machine learning techniques like classification and clustering. The presenter discussed the technical architecture and challenges of building Rujhaan.com to provide fast, personalized news content to over 16,000 monthly users while scaling to growing traffic levels.
Report
Share
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Case study of Rujhaan.com (A social news app )
Similar to Case study of Rujhaan.com (A social news app ) (20)
More from Rahul Jain
Recently uploaded
Recently uploaded (20)
Case study of Rujhaan.com (A social news app )
- 1. Case Study of Rujhaan.com November 2014 Meetup Rahul Jain @rahuldausa
- 2. About Me… • Big-data/Search Consultant based out of Hyderabad, India • Provide Consulting services and solutions for Solr, Elasticsearch and other Big data solutions (Apache Hadoop and Spark) • Organizer of two Meetup groups in Hyderabad • Hyderabad Apache Solr/Lucene • Big Data Hyderabad
- 3. What it does? Rujhaan which means "#interest" is a news app that aggregates the Trending #News, #trends with #buzz around them from social media. It also works as a content discovery where user can see information based on his interest (under development).
- 4. What I am going to talk • Introduction • Software Stack • Crawler • Apache Solr • MongoDB • Redis • Machine Learning stack • Classification • Clustering • NER • POS Tagging
- 5. How it look ? http://www.rujhaan.com
- 7. Trends : Arpita Khan http://www.rujhaan.com/topic/Arpita-Khan.html
- 8. Trends : Phil Hughes http://www.rujhaan.com/topic/Phillip-Hughes.html
- 10. Major challenge: Response time of 500ms is Critical
- 11. High level Flow: Processing Fetch Managed Cache Internet 2 1 3 4 Topics Extraction 1 8 5 Language Detectio 6 Classification/ Clustering 7 Parse MongoDB HTML Cleaner Junk/Sp am Cleaner (Text) n Scoring Summary (Most Meaningful text of Story) Social Media Apache Solr 9 0 1 1
- 12. High level Flow: View HAProxy Redis Managed Cache Internet 2 1 3 Nginx MongoDB Tomcat (App) Apache Solr 4 5
- 13. Current Traffic Stats Traffic: • 16k users/month • ~38k pageviews/month • 200k requests/day by 24+ bots • Traffic growing by 60-70%/month • Alexa rank : ~211000
- 14. Application Stack • Crawler • Apache Solr • MongoDB • Redis
- 15. Crawler • A web crawler (also known as a web spider or ant) is a program, which browses the World Wide Web in a methodical, automated manner. • Web crawlers are mainly used to create a copy of all the visited pages for later processing by a search engine, that will index the downloaded pages to provide fast searches. http://www.codeproject.com/Articles/13486/A-Simple-Crawler-Using-C-Sockets
- 16. How it work? http://wiki.eclipse.org/SMILA/Documentation/Importing/Crawler/Web
- 17. Search@ApacheSolr • Enterprise Search platform for Apache Lucene • Open source • Highly reliable, scalable, fault tolerant • Support distributed Indexing (SolrCloud), Replication, and load balanced querying • http://lucene.apache.org/solr 17
- 18. High level overview Source: http://www.slideshare.net/erikhatcher/solr-search-at-the-speed-of-light
- 19. Apache Solr - Features • full-text search • faceted search (similar to GroupBy clause in RDBMS) • scalability – caching – replication – distributed search • near real-time indexing • geospatial search • and many more : highlighting, database integration, rich document (e.g., Word, PDF) handling 19
- 20. Database: #MongoDB • Document Oriented NoSQL database • Dynamic Schema • JSON based • Fast read and write • Quite suitable for Non Relational data Stats: • 2 million tweets • 70k news articles • ~25GB rawhtml unstructured data • ~16GB structured data
- 21. Why NoSQL • Large Volume of Data • Dynamic Schemas • Auto-sharding • Replication • Horizontally Scalable * Some of these above Operations can be achieved by Enterprise class RDBMS software but with very High cost
- 22. Major NoSQL Categories • Document databases • pair each key with a complex data structure known as a document. • MongoDB • Graph databases • store information about networks, such as social connections • Neo4j Contd.
- 23. Major NoSQL Categories • Key-Value stores • Every single item in the database is stored as an attribute name (or "key"), • Riak , Voldemort, Redis • Wide-column stores • store data in columns together, instead of row • Google’s Bigtable, Cassandra and HBase
- 24. Sample Record (JSON) { "_id" : ObjectId("53f087c69144ca452acadfb0"), "id" : "7a622c50e95d4debb1376d4f6e2d0a47", "title" : "Yelp Swings To Profitability In Strong Q2 With $88.8M In Revenue, EPS Of $0.04", "summary_gs" : "Today after the bell Yelp reported its second-quarter financial performance, including revenue of $88.79 million, and a profit of $0.04 per share. The company had net income of $2.7 million in the period, up from a $878,000 loss in the year-ago quarter. Investors had expected Yelp to lose 3 cents per share on revenue of $86.32 million. The company’s revenue tally for its most recent quarter is up 61 percent on a year-over-year basis. The company also reported strong guidance for its third quarter, with revenues forecasted to land in the $98 to $99 million range. ", "link" : "http://techcrunch.com/2014/07/30/yelp-swings-to-profitability-in-strong-q2-with-88-8m-in-revenue- eps-of-0-04/", "category_label" : "business", “image_url”:” http://tctechcrunch2011.files.wordpress.com/2014/04/yelp-earnings.jpg”, “score”: 38.0, “boost”:1.0, “keywords”:[“news”, “yelp”, “revenue”] }
- 25. Cache: #Redis • Advanced In-Memory key-value store • Insane fast • Response time in order of 5-10ms • Provides Cache behavior (set, get) with advance data structures like hashes, lists, sets, sorted sets, bitmaps etc. • http://redis.io/
- 26. Machine Learning • Classification • Clustering • NER (Named Entity Recognition) • Summarization (Relevant text) • Topics Extraction
- 27. ML Workflow
- 28. Classification • classify a document into a predefined category. – For e.g news can be classified into business, politics, finance etc. • documents can be text, images • Popular one is Naive Bayes Classifier. • Steps: – Step1 : Train the program (Building a Model) using a training set with a category for e.g. sports, cricket, news, – Classifier will compute probability for each word, the probability that it makes a document belong to each of considered categories – Step2 : Test with a test data set against this Model • http://en.wikipedia.org/wiki/Naive_Bayes_classifier
- 29. Clustering • clustering is the task of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar to each other • objects are not predefined • For e.g. these keywords – “man’s shoe” – “women’s shoe” – “women’s t-shirt” – “man’s t-shirt” – can be cluster into 2 categories “shoe” and “t-shirt” or “man” and “women” • Popular ones are K-means clustering and Hierarchical clustering
- 30. K-means Clustering • partition n observations into k clusters in which each observation belongs to the cluster with the nearest mean, serving as a prototype of the cluster. • http://en.wikipedia.org/wiki/K-means_clustering http://pypr.sourceforge.net/kmeans.html
- 31. Summarization • Finding the most relevant text related to story/article • There can be multiple approaches related to accuracy. • Below is our approach: Cleaned Text 1 Find low 3 2 value cluster 4 5 Cluster based on stop words Score each cluster Take Highest score cluster Sentence Extractor Some more Scoring… Summary text 6 7 *Summary can be a content curated by computer system. i.e. translating the story into its own sentences (out of scope)
- 32. POS (Part of Speech) Tagging • process of marking up a word in a text (corpus) as corresponding to a particular part of speech, its definition, as well as its context • relationship with adjacent and related words in a phrase, sentence, or paragraph. • 9 parts of speech in English: noun, verb, article, adjective, preposition, pronoun, adverb, conjunction, and interjection. • “This is a sample sentence” will be output as • This/DT is/VBZ a/DT sample/NN sentence/NN • We use Stanford MaxentTagger • http://nlp.stanford.edu/software/tagger.shtml Number Tag Description 1. CC Coordinating conjunction 2. CD Cardinal number 3. DT Determiner 4. JJ Adjective 8. JJR Adjective, comparative 9. JJS Adjective, superlative 10. LS List item marker 11. MD Modal 12. NN Noun, singular or mass 13. NNS Noun, plural 14. NNP Proper noun, singular 15. NNPS Proper noun, plural 16. PDT Predeterminer 17. POS Possessive ending 18. PRP Personal pronoun 19. PRP$ Possessive pronoun 20. RB Adverb 21. RBR Adverb, comparative 22. RBS Adverb, superlative 23. RP Particle 24. SYM Symbol 25. TO to 26. UH Interjection 27. VBD Verb, past tense 32. VBZ Verb, 3rd person singular present
- 33. NER • Identifying the Named Entities like Person name, location, organization from a text • Need a pre built trained model.
- 34. Machine Learning Stack • Stanford NER & Tagger • LingPipe • OpenNLP • Carrot2
- 35. We are Hiring! rockstar@rujhaan.com 35 Want to make an impact on millions of lives ? Join Us
- 36. Thanks! @rahuldausa on twitter and slideshare http://www.linkedin.com/in/rahuldausa 36 Join us @ For Solr, Lucene, Elasticsearch, Machine Learning, IR http://www.meetup.com/Hyderabad-Apache-Solr-Lucene-Group/ Join us @ For Hadoop, Spark, Cascading, Scala, NoSQL, Crawlers and all cutting edge technologies. http://www.meetup.com/Big-Data-Hyderabad/