Ashutosh Kulkarni’s Post

Rockset is shutting down on the 30th of September. While it is amazing news for the industry that proves the point that companies that can effectively store, process, and retrieve data will be the ultimate winners in the AI race led by OpenAI. It is also tough to be in Rockset customer's shoes. If you are looking for a real-time search and analytics database, we got you covered. CrateDB is a distributed SQL database purposely built for real-time analytics across large datasets of structured and semi-structured data. Similarly to Rockset, it indexes all data in real-time (text, vector, geospatial, time-series, and JSON) for the most efficient search and fast ad hoc query execution at any scale. It is built on top of Apache Lucene and unlike Rockset is fully open-source (https://lnkd.in/dJW9VG_Z).

🚀 Search with Typesense! Elasticsearch's Alternattive! 🚀 Hey LinkedIn community! 👋 I recently had an exhilarating experience while working on a project involving the development of a sophisticated search feature. 🛠️ Our task was to create a search that not only matched keywords with item names and their synonyms but also prioritized results based on specific fields, giving preference to item names over it's synonyms. To add to the complexity, we had to accommodate up to 2-character spelling mistakes. 🕵️♂️ Enter Elasticsearch, a powerful search engine. 🚀 We initially opted for it but soon encountered hurdles for it's massive container (docker instance) size (1.3+ GB) and complex configurations. Even our dataset was not that much in size. 'Typesense' turned out to be a game-changer! Not only did it meet our requirements seamlessly, but it also boasted a container size less than half of Elasticsearch (630+ MB). 😲 The documentation was user-friendly, and the reduction in code size brought a sigh of relief. Data syncing became faster, and the search experience was spot-on. Here's what blew us away: 📦 **Container Size:** Only 630+ MB - less than half of Elasticsearch! 🤯 📚 **Documentation:** Super cool and user-friendly. 💻 **Code Size:** Reduced to half of the original, making it cleaner and more manageable. ⚡ **Data Syncing:** Lightning-fast performance. 🔍 **Searching:** Natively supports the complex searches we needed, and I can dynamically choose fields at runtime. 🧠 **Machine Learning:** Built-in model for semantic search - next-level search intelligence. What truly impressed me about Typesense was its native support for fuzzy searching, runtime field querying, and a built-in machine learning model for semantic search. 🚀   What won me over completely? Typesense's motto: "A software alternate to Elasticsearch, developer-friendly, needs less configuration tuning, and easy to integrate." 🌟 I'm genuinely thrilled to explore more features that Typesense has to offer. If you're into search technology, I highly recommend checking it out! Let's simplify search together! 💻🔍 #TechInnovation #SearchEngine #Typesense #ElasticsearchAlternative

How Elasticsearch Search Fast? Ever wonder how this open source Elasticsearch achieves near real time search experience. Short answer is by inverted index. Let's break it down with a hands-on example! Consider three simple documents indexed in Elasticsearch: { "_id": 1  "question": "What is LLM ?" } { "_id": 2  "question": "How to Fine tune LLM Models ?" } { "_id": 3  "question": "Top Ten LLM Models" } When these documents get indexed inside Elasticsearch, the data will be stored in inverted index format. Let us have a look on inverted indexed structure after indexing. Term      | Document Id    --------------|------------------ Fine      2         How      2         LLM      1, 2, 3      Models  2, 3        Ten      3         Top      3         What      1         is       1         to       2         tune      2        Now when a user searches for the term 'LLM' Elasticsearch, utilizing the inverted index fetches Document 1,2, 3 as results. Without this inverted index search, we need to go through each document and filter all the documents that match with the search results. This will hugely impact performance when we work with millions of documents. #Elasticsearch #ELK #SearchEngine #Tech

“Compute embedding with Cohere and store them for efficient vector or hybrid search in Elasticsearch. This guide uses a dataset of Wikipedia articles to set up a pipeline for semantic search. It will cover: Creating an Elastic inference processor using Cohere embeddings Creating an Elasticsearch index with embeddings Performing hybrid search on the Elasticsearch index and reranking results Performing basic RAG To see the full code sample, refer to this notebook.”

#Datainfrastructure - is the cornerstone behind the scenes that enables most of #AI use cases.  Companies that can effectively store, process and analyse the data for intelligent decision making will be the ultimate winners in this #AI race. This is why OpenAI just bought Rockset and we believe CrateDB is a great fully #opensource alternative. CrateDB is the intelligent database purpose built for the #AI #generativeapplications in #realtime

Search is hard, but keeping a Biomedical Search Engine engine up-to-date is even harder. Why is this harder? Because...of the data: 1) The data comes from multiple sources (Pubmed, Pmc, ClinicalTrial, bioarxiv) thus we need different crawlers for different websites. 2) The data comes in multiple formats (XML, JSON) with different fields (e.g some say "doi" some others say "digital_object_identifier"), so we need different parsers for each website to normalize them and have uniform data. 3) The relationship of the data is complicated. Some data are new (new ingestions) and some data give additional information for the data that already exists in our engine (updates), while some other data signifies we need to delete. Thus, we need to give specific instructions to each data point on how will it be ingested in our engine. 4) There is a lot of data. We had to build a parallelized architecture for processing and indexing so it wouldn't take 100 years to finish. How do we manage all that 🛠️? With 20 servers running in Oracle Cloud, our own Kubernetes with our custom elasticsearch, and combining python and java code for the ingestion instructions. #searchengines #ai #softwaredevelopment #softwarearchitecture #bigdata #kubernetes #elasticsearch

Follow along to see how we can leverage HarperDB’s application layer to periodically fetch data and index it in Elastic. In this tutorial, we’ll go over a proof of concept to index new data from HarperDB into Elasticsearch. #HarperDB #Elasticsearch #Connector #Coding https://hubs.li/Q02dGg0X0

Your .𝐍𝐄𝐓 𝐩𝐫𝐨𝐣𝐞𝐜𝐭 needs lightning-fast search? Look no further than 𝐄𝐥𝐚𝐬𝐭𝐢𝐜𝐬𝐞𝐚𝐫𝐜𝐡, 🔎 the king of search engines! But how do you get your data in there? Enter indexing, the secret sauce to unleashing search power. Think of it like this: Indexing is like building a detailed library catalog. You tell Elasticsearch what's in your documents, where it is, and how to find it. This lets users zero in on what they need with laser-like accuracy. 𝐖𝐡𝐲 𝐢𝐧𝐝𝐞𝐱 𝐢𝐧 .𝐍𝐄𝐓 𝐰𝐢𝐭𝐡 𝐄𝐥𝐚𝐬𝐭𝐢𝐜𝐬𝐞𝐚𝐫𝐜𝐡? ◾ 𝐁𝐥𝐚𝐳𝐢𝐧𝐠-𝐟𝐚𝐬𝐭 𝐬𝐞𝐚𝐫𝐜𝐡𝐞𝐬: Find what you need in milliseconds, not minutes. ⚡ ◾ 𝐑𝐞𝐥𝐞𝐯𝐚𝐧𝐜𝐞 𝐫𝐞𝐝𝐞𝐟𝐢𝐧𝐞𝐝: Fine-tune your search with advanced query options and aggregations. ◾ 𝐒𝐜𝐚𝐥𝐚𝐛𝐢𝐥𝐢𝐭𝐲 𝐭𝐡𝐚𝐭 𝐬𝐜𝐚𝐥𝐞𝐬: Handle mountains of data with ease, thanks to Elasticsearch's distributed nature. ◾ 𝐅𝐥𝐞𝐱𝐢𝐛𝐢𝐥𝐢𝐭𝐲, 𝐨𝐡 𝐬𝐨 𝐟𝐥𝐞𝐱𝐢𝐛𝐥𝐞: Index any type of data you can imagine, from text to images to geospatial coordinates. ️ 𝐑𝐞𝐚𝐝𝐲 𝐭𝐨 𝐝𝐢𝐯𝐞 𝐢𝐧𝐭𝐨 𝐢𝐧𝐝𝐞𝐱𝐢𝐧𝐠 𝐦𝐚𝐠𝐢𝐜? 𝑪𝒉𝒆𝒄𝒌 𝒐𝒖𝒕 𝒕𝒉𝒆 𝒐𝒇𝒇𝒊𝒄𝒊𝒂𝒍 𝑬𝒍𝒂𝒔𝒕𝒊𝒄𝒔𝒆𝒂𝒓𝒄𝒉 .𝑵𝑬𝑻 𝒄𝒍𝒊𝒆𝒏𝒕 𝒇𝒐𝒓 𝒔𝒎𝒐𝒐𝒕𝒉 𝒊𝒏𝒕𝒆𝒈𝒓𝒂𝒕𝒊𝒐𝒏. 𝑬𝒙𝒑𝒍𝒐𝒓𝒆 𝒉𝒐𝒘 𝑵𝒆𝒔𝒕, 𝒂 𝒑𝒐𝒑𝒖𝒍𝒂𝒓 .𝑵𝑬𝑻 𝒍𝒊𝒃𝒓𝒂𝒓𝒚, 𝒔𝒊𝒎𝒑𝒍𝒊𝒇𝒊𝒆𝒔 𝒊𝒏𝒅𝒆𝒙𝒊𝒏𝒈 𝒂𝒏𝒅 𝒒𝒖𝒆𝒓𝒚𝒊𝒏𝒈. 𝑱𝒐𝒊𝒏 𝒕𝒉𝒆 𝑬𝒍𝒂𝒔𝒕𝒊𝒄𝒔𝒆𝒂𝒓𝒄𝒉 𝒄𝒐𝒎𝒎𝒖𝒏𝒊𝒕𝒚 𝒇𝒐𝒓 𝒆𝒙𝒑𝒆𝒓𝒕 𝒕𝒊𝒑𝒔 𝒂𝒏𝒅 𝒕𝒓𝒊𝒄𝒌𝒔. Don't settle for mediocre search. Indexing your .NET project with Elasticsearch is the game changer you've been waiting for! #elasticsearch #dotnet #indexing #search #performance #scalability #developer #software

Tutorial on RAG implementation using Postgres extension as Vector DB #postgres #pgvector #rag #llm #vectordb

It's the first day of Supabase Launch Week. For a long time, we've be trying to figure out how to make Postgres Row Level Security easier for developers. Needless to say, AI provided the answer. We started stripping out all the custom interfaces in our Dashboard, moving instead towards a "text-to-sql" approach. Now, developers can just describe what they need in natural language. AI is extremely powerful when combined with a schema-based database, like Postgres, because it can infer so much context from the schema and the database provides stricter guarantees with generated code. This is just one of four features we're releasing today. Check out the rest here: https://lnkd.in/g9-r97cN