Josh Klahr’s Post

Did you know that 40% of Snowflake ❄ accounts also use Databricks 🧱 - and 46% of Databricks 🧱 accounts also use Snowflake ❄? Organizations are increasingly deploying multiple data silos just to keep up with the needs of their data teams. 🤷 If you’re struggling to: - Ingest data from batch and streaming sources with up-to-the-minute freshness - Manage your data centrally in a cost-effective way - Support your downstream BI, analytics, and data science teams with their processing engines of choice, such as Spark and SQL 👉 Then the Universal Data Lake may be the right option for you. Check out this short video that introduces the concept in simple terms. https://lnkd.in/gs8fwyup #onehouse #opensource #nolockin #datalakehouse #lakehouse  #dataengineering #universaldatalakehouse

Did you know that 40% of Snowflake ❄ accounts also use Databricks 🧱 and 46% of Databricks 🧱 accounts also use Snowflake ❄? Organizations are increasingly deploying multiple data silos just to keep up with analytics requirements. ❓If you’re struggling to... - Ingest data from batch and streaming sources with up-to-the-minute freshness - Manage your data centrally in a cost-effective way - Support your downstream BI, analytics, and data science teams with their processing and analytics engines of choice such as SQL and Spark 👉 ...then the Universal Data Lake may be the right option for you. Check out this short video that introduces the concept in simple terms. https://lnkd.in/gs8fwyup #onehouse #opensource #nolockin #datalakehouse #lakehouse  #dataengineering #universaldatalakehouse

Did you know that Spark supports for Parquet also Aggregate Pushdowns? Max, min and count can be consumed from the Parquet Statistics. This makes those calculations so much faster. The feature seems a bit secretive even though released with Spark 3.3. Quite exciting for me to discover and play around with it. Find more in this video. 😊 https://lnkd.in/dudwuT9R In this session we will: - Learn about the Data Source API v2 - Recap Predicate Pushdown on column and row level - Discover Aggregate Pushdown on ParquetYou want to master Data Engineering with PySpark? Subscribe here: https://lnkd.in/duVbCwRz Feel free to comment or challenge my explanations as always. Happy to learn also myself more by the community.    Video link here: https://lnkd.in/dudwuT9R   #spark #pyspark #dataengineering #dataengineeringessentials

🎯 Advantages of using Parquet file in your data project 1️⃣ Easy data retrieval: Ideal for OLAP systems due to its columnar storage. 2️⃣ Reduced storage needs: Efficient compression algorithms minimize storage requirements. 3️⃣ Predicate pushdown: Queries optimized to scan only relevant data, enhancing query performance. 4️⃣ Schema evolution: Supports adding new columns without rewriting the entire dataset. 5️⃣ Splitting and partitioning: Enables parallel processing and efficient data retrieval. 6️⃣ Wide compatibility: Supported by major big data processing frameworks. Follow me Sampath Vijayan for more insights and tips. #pyspark #spark #parquet #dataengineer #dataanalysis #ctgdatabit #trendytech

New Data Video: Browsing Parquet Files from VSCode ---------------------------------- ** Watch Here: https://lnkd.in/eUcj6_-d ** Intro to Data Playlist: https://lnkd.in/eP9RxwcF ----------------------------------- #DataEngineering #DataAnalytics #DataScience #DataLake #DataLakehouse #BigData

Spatial SQL is coming to Databricks! 🌐📌 This major announcement underscores the importance of geospatial capabilities for the #analytics community. Get ready to start turning location data into powerful insights! 🚀 Learn more & get started 👉 https://hubs.ly/Q02CBmk80

𝗧𝗲𝘀𝘁 𝗬𝗼𝘂𝗿 𝗦𝗽𝗮𝗿𝗸 𝗞𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲. I was teaching repartition and coalesce today and Just thought to have a quiz around this. Let's see how many of you can answer this correctly. 𝗪𝗵𝗶𝗰𝗵 𝗼𝗻𝗲 𝗶𝘀 𝘁𝗿𝘂𝗲 𝗮𝗯𝗼𝘂𝘁 𝗖𝗼𝗮𝗹𝗲𝘀𝗰𝗲 𝗮𝗻𝗱 𝗿𝗲𝗽𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻? 1. Coalesce can only decrease the partition, repartition can increase/decrease the partition. 2. coalesce does not shuffle the data, repartition shuffle the data 3. coalesce tries to avoid shuffling the data, repartition always shuffles the data. 4. Both coalesce and repartition can increase/decrease the partition 5. Both coalesce and repartition can cause data shuffling. There can be Multiple Answers too. Let me know in comment Loved it? Hit 👍 show ❤️ Follow our page for more updates OnlineLearningCenter #dataengineering #spark #datascience

Learn DataScience from DataCamp Pay 75$ for an annual subscription Enroll from here: https://lnkd.in/dNhxgSpp Guided - Unguided Projects Review https://cutt.ly/GEwAgev #ml #research #data #DataScience #MachineLearning #DeepLearning #Rstats #100DaysOfCode

Learn DataScience from DataCamp Pay 75$ for an annual subscription  Enroll from here: https://lnkd.in/dNhxgSpp Skill Tracks Review https://cutt.ly/TEwAoRs #ml #research #data #DataScience #MachineLearning #DeepLearning #Rstats #100DaysOfCode

I just completed the course "Categorical Data in the Tidyverse" via DataCamp