CEO of Writer.com | Enterprise generative AI | Hiring in ML, eng, design, mktg, sales + CS
Monday's post on RAG (posted in comments) hit a nerve, and it's worth a follow up post. I get why: there are dozens of vectorDB companies and every db company is adding vector storage capabilities; surely that's the way to go?? The fact that there ARE so many vector DB companies should be a red flag: there aren't hundreds of graph DB / relational DB companies. Once an approach scales, a few winners emerge, and it's clear that that has yet to happen in vector DB land for generative AI, and you should be asking yourself why. First, context: The primary advantage of embeddings + vector search is that you can search for semantic similarity, not just literal keyword matches. The reason you can search by similarity and not linguistic matches is bc you use embeddings models to turn the query into a numerical representation, and do the same thing to your data: so you're comparing numbers to numbers, and thus better able to approximate conceptual meaning. And bc the embeddings representation of your data sits in multi-dimensional space, you need a vector DB, which was invented for genetic research, to store those representations. Embeddings capture semantic similarity between your data and a **query**, but *do not* also store or connect the relationships *between* data in said multi-dimensional space. OK, back to my Monday post. "May, don't be silly, no one does fixed chunking, we solve the chunking problems you describe with hierarchical and context-aware chunking." Rebuttal: It's chunks vs nodes and edges!! A node in your graph could be a single portion of a page or table, for ex, and the *relationships* between the nodes in your graph are called edges: and those nodes and edges are specific pieces of information that can provide more context to support a query than hierarchical ranking of chunks. There's just no comparison in knowledge management use cases between graph-based approaches and vector DB-based approaches to RAG bc of the level of DETAIL that can be drawn upon to answer multi-hop questions. Most organizations don't have the NLP engineering / linguistics expertise in house to be building the primitives for the constant sentence boundary recognition, PoS tagging, parsing, table logic, chart logic, etc that's required across hundreds of different file types to do hierarchical chunking well — plus, you're rebuilding a TON of logic / doing a ton of small LLM fine-tuning to recreate the temporal / topical / semantic / entity context you get with graph-based approaches, that all has to be scaled AND maintained. Add multi-modalities to that, and your team is fried before they've scaled a single AI assistant. Writer uses specially-trained LLMs to build your knowledge graphs, THEN does graph-based RAG. imo the combination of AI knowledge graphs and LLMs **is** basically gonna be AGI in the enterprise. Especially in enterprise contexts, nodes and edges are your alpha.
David Fackler
5mo
Regardless of how the solution finally gets addressed, people experimenting with RAG have to acknowledge that a knowledge retrieval bot that can only confidently answer shallow questions that require little to no context is a fundamental limit to making a GREAT product. As you said in an earlier post, users aren't going to be trained to know that they can only ask a question whose question could be gleaned from just one or two paragraphs in a vector database. They have been promised a knowledge bot and they are going to need a knowledge bot that can combine concepts and synthesize whole concepts. That limit has been there in every approach I have seen barring some massive loop where you call repeated LLM calls on every chunk. At this point, the overhead makes it functionally useless in production. Excited to see how your knowledge graph approach works! I was reminded of a webinar I saw a couple of months ago with someone warning that they have seen the rise and fall of dozens of new specialized database types. And to be wary of committing to building a whole system on a specific vector database that can't handle other modalities.
Abhishek Ratna
5mo
Using LLMs to build knowledge graphs 🤯
Andre Zayarni
5mo
«The fact that there ARE so many vector DB companies should be a red flag: there aren't hundreds of graph DB / relational DB companies.» 🤔 Are you sure about that fact? I know only a handful of vector DB’s. Let’s check a source of truth. Relational DB’s: > 160 https://db-engines.com/en/ranking/relational+dbms Graph DBs: 38 https://db-engines.com/en/ranking/graph+dbm Key value stores: 66 https://db-engines.com/en/ranking/key-value+store Vector DBs: 13 https://db-engines.com/en/ranking/vector+dbms And by far not all of them are really databases.
Rene Santana
5mo
Getting specific highly contextual answers from an AI powered app is the next level in building out knowledge bases. I see RAG brought up a lot in these discussions, but as you describe the needs to be able to do so, it seems incredibly demanding. As mentioned, using the LLMs (specially trained, logic built out, and scalable) to build the knowledge graphs (hyper-focused content generated for answering questions) seems to be the sweet spot for pushing AGI forward, but also providing an ultra useful case for Enterprise level AI.
Manj Chenna
5mo
Excellent and well-argued post, May Habib. It’s clear that while graph-based search is key for applications needing to understand data relationships, we shouldn’t overlook the importance of vector databases in semantic search, especially in fields like image recognition and voice search. And regarding AGI, it’s a moving target – what we consider the pinnacle of AI today will evolve and be replaced by newer advancements as we move forward 5, 10 and 50 years from now.
THIS POST May Habib is just outstanding! You've just schooled a vast number of people in how to think about this practically - and threw in a strong point-of-view for free! Awesome. My CIO/CTO, Nils Senvalds, has been shouting at me about something similar for weeks, but it took YOUR post to make my brain go, 'Oh, now I get it!' For my part, there are a few enterprise use-cases with Knowledge Graphs that are incredibly powerful, and that I've not yet seen very (if any) considering... It has been something of a 'forgotten art' given all the recency around LLMs, vector dBs, etc. Check out Paco Nathan - he's been in AI/ML for 40 years! One of his most powerful insights to add to your reflection might be the idea of making sure that you first have 'structure, and meaning, and semantic attributes' BEFORE you start 'creating content' with GenAI. Very much the age-old, 'if we only knew what we knew' phrase. And, 'What's old is new again,' as another phrase goes. [n.b.. I did NOT just call you 'old', Paco! 🤐 ]
Jacob Warren 🤖
5mo
This is the way. There's so much more that goes into making RAG valuable than just similarity comparisons. My (compared to Writer's) LLM is fine-tuned on 57 unique dimensions. And I'm just generating LinkedIn posts. For the enterprise, the data preprocessing to fine-tune the LLM for use with RAG is realistically more important than the model itself.
John S.
5mo
Preach!! 🙌 Excellent breakdown, couldn't agree more about the ⛳️. It's another example of the perils of getting caught in a big wave, vs knowing _why_ a technology is a good fit. Graphs + LLMs will be true game changers for orgs, _especially_ those in highly complicated and regulated industries that demand a lot of tribal knowledge. Stoked to see this take off at scale. 🚀
Ash Garner
5mo
Awesome post!! Using LLMs to build graph is what we're all about. Rishabh Sagar you'll love this. If anyone is nerdy enough to read this - here's an overview on how you can do it... https://tomoro.ai/insights/graph-databases-as-rag-backends
CEO of Writer.com | Enterprise generative AI | Hiring in ML, eng, design, mktg, sales + CS
5mohttps://www.linkedin.com/feed/update/urn:li:activity:7157743924189491200/