Girish A.

Real world experience surpasses text book examples, where you can get stuck into one method thinking. Years ago I served as a Data Architect for a large bank in Ireland 🍀, which is when I really learned data modeling. This article is a good modern take on different techniques popular today. Worth a read. https://lnkd.in/ec5awd99 #DataAI #DataModeling #DataEngineering

1

🚀 Master Nested JSON Integration: Read from Azure Storage in Databricks Like a Pro! Unlock the full potential of your data with our latest blog post on reading nested array JSON from Azure Storage using Databricks. Learn step-by-step methods, best practices, and insider tips to streamline your data workflows and enhance productivity. This guide is perfect for data engineers, analysts, and enthusiasts looking to efficiently process and analyze complex JSON structures. Read the full blog here: https://lnkd.in/dZvrTpWb Let's discuss and share ideas on how we can make data processing smoother and more effective. Feel free to leave your thoughts, questions, and feedback in the comments #DataEngineering #BigData #Azure #Databricks #JSON #DataIntegration #DataAnalytics #TechBlog #DataScience #DataProcessing #TechInsights

5

1 Comment

Optimise your data across the entire value chain with Databricks and DVT DVT, a certified partner in Databricks services, offers data engineering expertise crucial for the effective implementation and ROI maximisation of your Databricks projects. For more information - https://lnkd.in/dttES4bW. Karen Heydenrych Dantes de Beer #retail #dataengineering #data #ai #innovation

4

Here's my latest article on Medium about a very small but key Design aspect of any Data Practitioner's journey, data checksum. #snowflake #snowflakeadvocate #checksum

3

In our latest case study, we explore how Intellinexus assisted a manufacturing company to migrate from 'On Premise IBM Netezza' to 'Snowflake Data Cloud'. The company undertook the migration in three phases. The first was focusing on data and code, followed by ETL to ELT optimisation along with data sharing and data monetisation considerations, and then concluded with integrating advanced analytics capabilities Snowflake provides for predictive workloads, forecasting and data apps. Follow this link to read the full case study: https://buff.ly/3JWNRbC #DataInsights #DataDriven #Technology #Manufacturing

14

1 Comment

Hi Team, This is another video on snowflake Schema Detection and Schema Evolution. The video covers the json and csv use cases. The steps are so simple: -Setup Snowflake environment. -Upload the files to the stage using snowflake UI or Snowsql Put Command or other 3rd party tools. -Create the table from a template using INFER_SCHEMA option. -Load the 1st file. -Enable the SCHEMO_EVOLUTION option for the table. -Load the 2nd file. -Check the table description and the data content. Thanks. https://lnkd.in/eEuuNVDx

1

As we navigate the era of big data, the challenge isn’t just in handling the sheer volume—it’s in processing it smartly and swiftly.  Efficiency & Scalability are the cornerstones of modern data pipelines. Embracing strategies like partitioning, sharding, and distributed computing can transform your data architecture. This leads to a more compressed structure, Faster querying and Efficient execution. Want to stay ahead of the curve? Dive into our latest article for insights on scaling your data pipelines for peak performance. Let’s build systems that not only manage the data deluge but also deliver insights at the speed of thought!   Read the full article here: https://lnkd.in/gAkygWji #DataEngineering #DataPipelines #Efficieny #CostManagement #BigData

17

Production kdb+ stack has just fallen over 🔘 All hands call, get SAs & dev team on the phone, send out client impact notification, Sev N 🔘Support have manually moved DR to primary, Sev N-1, notify client impact/resolution 🔘 I see all traffic is diverted to COB/DR, can we get someone to look at recovering that at next opportunity 🔘 The remaining X stacks appear to be managing the load well, can we get an instance spun back up Which one does your kdb+ application fall into? In our latest ARK blog, from Ross Edens, the focus is on Disaster Recovery and ensuring Continuity of Business. How many hours are lost each year due to firefighting systems that a bit of preparation and resourcing could prevent/reduce. We all want to trend toward 5 9s of uptime. If you'd like us to take a look and do a full or partial architecture review, we'd be more than happy to. Let that spark be an idea rather than money going up in smoke. https://lnkd.in/ew7xt6w4 #kdb #dataintellect #challengeaccepted

21

Interesting read on Databricks Fundamentals. Delta Lake and Photon are two pivotal technologies within the Databricks ecosystem, revolutionizing the way organizations manage and analyze their data. 1. Delta Lake - Open-source storage layer that brings ACID Atomicity, Consistency, Isolation, Durability) transactions to Apache Spark and big data workloads. It provides reliability, performance, and data management capabilities for big data analytics. Other Key Features in Delta Lake - Schema enforcement prevents users from accidentally polluting their tables with mistakes or garbage data. - Schema Evolution - Delta Lake supports schema evolution, allowing schemas to evolve over time while maintaining backward compatibility. This enables flexibility in data ingestion and schema management, reducing development overhead and improving productivity. 2. Photon: Photon is a distributed query engine optimized for Databricks Delta Lake. It accelerates query performance and improves resource utilization for analytics workloads by leveraging advanced query optimization techniques and in-memory processing. https://lnkd.in/gpxTGEDm

6

📌Databricks vs Snowflake Although they cater to different needs, it was always a dicussion point while migrating legacy data warehouses to cloud becuase of some similatities they posses. Good to know thier capabilities at high level to start with. Blogs by others I read this week: https://lnkd.in/dpCfG_ZP

8

Most enterprises store data across multiple tables, with each table storing data for an entity and its attributes. Each single table captures just one part of a complex pattern. For example, as Neha Patki and Frances Hartwell explain, a customer's age may correlate with the amount of money they spent on a purchase. Even though the "age" and "amount per purchase" columns are present in different tables, the connection between the tables means that intertable trends exist. To fully understand these trends, and emulate how an entity and its data connect and evolve, a generative model must work across tables and learn these patterns. We at the The Synthetic Data Vault developed our first multi-table approach — popularly known as a Hierarchical Modeling Algorithm (HMA) — in 2016. Over the years, we have improved it and created many different generative modeling techniques for multi-table data. Each of our approaches strikes a delicate balance between modeling speed vs. how exhaustively the model learns the patterns, affecting the usability of the tool and the quality of the #syntheticdata generated, and letting users find an approach that matches their needs. Read our primer on multi-table synthesizers and the SDV on our blog: https://lnkd.in/etjqnJ9f. As always, we learn and evolve based on your feedback. Join our Slack: https://lnkd.in/ePnJgHhc Try out SDV Community: https://docs.sdv.dev/sdv Send us a request for SDV Enterprise: https://lnkd.in/e97Ce89i

23

Extract, Transform, Load (ETL) processes form the backbone of data engineering, crucial for every data, analytics, and AI workload. As data warehouses, data lakes, and the newer data lakehouses evolve, data engineers need to develop a deeper understanding of ETL. Building ETL pipelines in this modern landscape comes with challenges such as managing real-time data ingestion, ensuring data quality, and orchestrating and monitoring pipelines effectively. Explore this guide to learn about: 1. The fundamentals of ETL 2. Key components of ETL, including data ingestion, transformation, and orchestration 3. Best practices to tackle pipeline issues, enhance efficiency, and ensure scalability 16 Best+FREE Data Engineering Courses- https://lnkd.in/eEcU7Jww Stay ahead in the world of data engineering by mastering these essential ETL concepts. PDF Credit- Matt Palmer #ETL #DataEngineering #DataPipelines #DataArchitecture #DataWarehousing #DataLakes #AI #DataScience #BigData #Analytics

43

🚀 At the Data Engineering Summit 2024 in Bangalore, Vishal Singh, Senior Director of Engineering at Slice, highlighted the crucial role of data observability. Vishal shared how Slice monitors business-critical metrics across various products to ensure reliable data-driven decisions. He also stressed aligning observability with business objectives, cost management, and compliance, especially in fintech. In summary, Vishal's session underscored data observability's importance in maintaining trust and driving value in modern data ecosystems. 🚀 https://lnkd.in/gAFPt6NJ #DES2024 #DataEngineering #DataObservability #Fintech #RealTimeInsights

5

👨‍Join John Joyce at Snowflake, for "𝗗𝗲𝗺𝗼𝗰𝗿𝗮𝘁𝗶𝘇𝗶𝗻𝗴 𝗗𝗮𝘁𝗮 𝗤𝘂𝗮𝗹𝗶𝘁𝘆: 𝗜𝗻𝗰𝗿𝗲𝗮𝘀𝗶𝗻𝗴 𝗔𝗰𝗰𝗲𝘀𝘀𝗶𝗯𝗶𝗹𝗶𝘁𝘆 𝗙𝗼𝗿 𝗦𝘂𝘀𝘁𝗮𝗶𝗻𝗮𝗯𝗹𝗲 𝗦𝘂𝗰𝗰𝗲𝘀𝘀"! 📈 In today's data-driven world, organizations face challenges in maintaining high-quality data and manageable processes. Acryl Data introduces a unified method that integrates essential elements from separate tools. Join 𝗝𝗼𝗵𝗻 𝗝𝗼𝘆𝗰𝗲, our Co-Founder and Tech Lead, at Snowflake’s #DataCloudSummit to learn how this approach provides a real-time view of data quality and compliance across your organization. The session will also discuss how a 𝘂𝗻𝗶𝗳𝗶𝗲𝗱 𝗮𝗻𝗱 𝗮𝗰𝗰𝗲𝘀𝘀𝗶𝗯𝗹𝗲 𝗺𝗲𝘁𝗵𝗼𝗱 can foster 𝗰𝗼𝗹𝗹𝗮𝗯𝗼𝗿𝗮𝘁𝗶𝘃𝗲, 𝗲𝗳𝗳𝗲𝗰𝘁𝗶𝘃𝗲, 𝗮𝗻𝗱 𝘀𝘂𝘀𝘁𝗮𝗶𝗻𝗮𝗯𝗹𝗲 𝗱𝗮𝘁𝗮 𝗾𝘂𝗮𝗹𝗶𝘁𝘆 𝗽𝗿𝗼𝗰𝗲𝘀𝘀𝗲𝘀, laying the foundation for long-term competitive advantages. 🗓️ 𝗪𝗵𝗲𝗻: Monday, June 3 at 4:00 PM PDT 📍𝗪𝗵𝗲𝗿𝗲: Basecamp S. Theater 2 (Rm MD225) 🚀𝗗𝗼𝗻'𝘁 𝗺𝗶𝘀𝘀 𝘁𝗵𝗶𝘀 𝗼𝗽𝗽𝗼𝗿𝘁𝘂𝗻𝗶𝘁𝘆 𝘁𝗼 𝗲𝗹𝗲𝘃𝗮𝘁𝗲 𝘆𝗼𝘂𝗿 𝗱𝗮𝘁𝗮 𝗾𝘂𝗮𝗹𝗶𝘁𝘆 𝘀𝘁𝗿𝗮𝘁𝗲𝗴𝗶𝗲𝘀. 🎫 Click for more info👇 . . . . #DataQuality #SustainableSuccess #AcrylData #DataManagement #Snowflake

12

4 Comments

I've read some debate around whether or not to skip this badge, but I'd definitely recommend to do the Databricks Certified Data Engineer Associate exam, it reinforces and demonstrates core product knowledge. Looking forward to self-prepping the Professional badge over the coming weeks! #databricks #dataengineer #dataarchitect #databrickscertified

22

5 Comments

Friends in Data... Much has changed in the last 2 decades in how data is stored, moved and utilisied within an enterprise. From initial Single Customer View, through Data Warehouses, ETL Data Integration Tooling, DataMarts, Lakes, Big Data through to Predictive and Prescriptive Data Analytic applications its fair to say the way we consume data has shifted. What has remained a constant are the traits of the best people to help you make sense of it... Data experts who can understand the business needs, understand the complexity of the data and design and deliver an elegant solution. Right now I am working with a bunch of very talented people in this space with expertise in Data Governance, Engineering, Data Solution Sales, Program Management and Data Analysis. If your job needs you to connect with excellent data people we should be talking. #Recruiters, #talentaquisition, #practiceleads, #Dataleads #DataGovernance #DataEngineer #DataAnalyst #DataProgram #Sydney #Brisbane #Melbourne.

11

Look who else is coming to PLAY at Snowflake Data Cloud Summit... Alex Read! ❄ Alex Read, Director of Data Platforms at EDF (UK), will share how EDF UK streamlined data platforms over the course of 12 months, helping Britain achieve net zero. #SnowflakeDataCloudSummit #SnowflakeSummit2024 #MatillionAtSnowflake #DataEngineering #AI

56