How AI Is Built 🛠’s Post

When should you use Spark to process your data for your AI Systems? In today’s episode of How AI Is Built 🛠, Abhishek Choudhary, principal data engineer at Bayer, breaks down data processing with Spark: - When to use Spark vs. alternatives for data processing - Key components of Spark: RDDs, DataFrames, and SQL - Integrating AI into data pipelines - Challenges with LLM latency and consistency - Data storage strategies for AI workloads - Orchestration tools for data pipelines - Tips for making LLMs more reliable in production Key takeaways: 1. Data Volume: Spark shines when dealing with terabytes+ of data. For datasets under 100GB, simpler tools like Pandas or DuckDB often suffice. 2. Uncertainty in Data Growth: If you expect rapid, unpredictable growth in your data volume, Spark's scalability becomes a major advantage. 3. Complex Data Pipelines: Spark excels when you need to perform multiple operations (e.g., loading, transforming, aggregating) on large datasets. 4. Existing Infrastructure: If you already have a Spark cluster set up (e.g., in Databricks), leveraging it for AI data processing can be efficient. 5. Team Expertise: Consider your team's familiarity with Spark. If you're starting from scratch, simpler Python-based tools might be easier to adopt. 6. Performance Requirements: For compute-intensive operations on large datasets, Spark's distributed computing model can significantly boost performance. Link to the episode is in the comments below! Let me know what tools you use for data processing and why. #dataengineering #llms #ai

How can you standardize data from various formats into a single, usable structure? 1️⃣ The "Two-Sided Funnel" Model: Input: Data comes from many different sources. Middle: All data is changed into one standard format (canonical format). Output: The standardized data is then shared or used. 2️⃣ Unified Data Streams: Turn different types of data into one consistent stream. Create parsers for each data source that convert the source's native format into your ingestible object format. 3️⃣ Parallel Processing Power: Use the "competing consumer model" to handle data faster and more efficiently. These workers (processes or threads) take tasks from a shared list (message queue), allowing faster and more efficient processing. 4️⃣ Actor Model Magic: Treat each part of your system (like data collectors or processors) as separate units. These units manage themselves and talk to each other through messages. This makes the system more stable and easier to grow. 5️⃣ Smart Polling: Instead of constantly checking all data, only ask for what's new or changed. This saves time and resources by focusing on updates rather than repeating checks on unchanged data. Want to go deeper? My latest podcast on How AI Is Built 🛠 with Kirk Marple, CEO of Graphlit, by Unstruk Data, unpacks these concepts and more. Listen here: https://lnkd.in/dkVaJtmE Which one does fit the best into agentic workflows?

How can you bring the data to your LLM? Chunks, context, metadata. All need to be stored, prepared, and delivered. While most people tweak prompts, tweaking this pipeline often brings the biggest marginal results. Some key points: 1. Capturing and representing the rich metadata is as important as the content itself when processing conversational data. 2. Implementing a tagging system for data stored in AI systems can greatly enhance retrieval efficiency and accuracy. 3. The best results are not always in the most complex tools. AWS Textract can provide better OCR results than LLM-based solutions. If you want to learn more, read below or listen to my episode with Derek Tu (Carbon) on How AI Is Built 🛠.

Bringing AI into data orchestration or orchestrating data workflows for AI. Today, you can learn about both. In today’s episode of How AI Is Built, I learned from Hugo Lu how to build robust, cost-efficient, and scalable data pipelines that are easy to monitor. Hugo is the founder of Orchestra, a serverless data orchestration tool that aims to provide a unified control plane for managing data pipelines, infrastructure, and analytics across an organization's modern data stack. If you only take away three things, here they are: Find the right level of abstraction when building data orchestration tasks/workflows. "I think the right level of abstraction is always good. I think like Prefect do this really well, right? Their big sell was, just put a decorator on a function and it becomes a task. That is a great idea. You know, just make tasks modular and have them do all the boilerplate stuff like error logging, monitoring of data, all of that stuff.” Modularize data pipeline components: "It's just around understanding what that dev workflow should look like. I think it should be a bit more modular." Having a modular architecture where different components like data ingestion, transformation, model training are decoupled allows better flexibility and scalability." Adopt a streaming/event-driven architecture for low-latency AI use cases: "If you've got an event-driven architecture, then, you know, that's not what you use an orchestration tool for...if you're having a conversation with a chatbot, like, you know, you're sending messages, you're sending events, you're getting a response back. That I would argue should be dealt with by microservices." Listen now: https://lnkd.in/dPgZQ6Am Question to you: How are AI workloads changing the way you approach data orchestration? Are you using specialized tools or adapting existing ones? Stay tuned for next week, when I discuss how to build data pipelines specifically for generative AI with Derek Tu from Carbon. #genai #llms #data #dataengineering #dataorchestration