Heribert Bastos

🚀 Released Data Contract CLI v0.10.10 with two new imports! ✅ dbt import Create a data contract from your dbt models, thanks to Rodrigo Carneiro $ datacontract import --format dbt --source my_dbt_models.yaml --dbt_models="orders,line_items" ✅ spark import Create a data contract for your spark StructTypes, thanks to Pierre Monnet $ datacontract import --format spark # imports directly from spark context And a lot of minor improvements. Thanks to all contributors for making this open source tool so great! Curious? Try out the tool and its new features at cli.datacontract.com #datacontracts

99

1 comentário

In this course I learned about data acquisition, how to obtain basic insights from a data set, data organization, exploratory data analysis and development model evaluation. #Python #IBM #Coursera.org #SaishruthiSwaminathan #Pooja #Matplotlib #Pandas #Waffles #WordCloud #Seaborn

2

Achieving a production-grade data lakehouse requires careful attention to maintenance and optimization. During his talk, "A case for Data Lakehousekeeping", Flavio Junqueira will focus on techniques that have been recently used across existing systems such as Z-ordering and, in particular, one that Qbeast uses based on a tree algorithm for indexing. He will discuss the reasons for choosing this approach and argue that it provides multiple benefits from improving overall performance to enabling features such as processing with samples. 👉 Know more about Flavio, CTO at Qbeast, and his talk at #JOTB24 here: https://lc.cx/rBfNji #data #datalakehouse #datalake #computerscience #distributedsystems

30

Understanding Data Debt: The Next Technical Debt As digital transformations accelerate, data becomes central to our business strategies. However, the exponential growth in data creation and storage is leading us to confront a new challenge: data debt 📈. Diogo Silva Santos explores this concept thoroughly in his insightful article, and I want to share some key takeaways and expand upon them with my own observations. What is Data Debt? Data debt occurs when organizations delay or avoid the necessary investment in managing their vast data assets, similar to technical debt but with data. This neglect can lead to decreased efficiency, higher costs, and significant risks 🚧. Santos describes data debt as a byproduct of rapid data accumulation without adequate governance, leading to issues like data silos and inconsistent data definitions. Key Causes of Data Debt: Data Silos and Duplication: As companies grow, data management often fails to keep pace, leading to fragmented and duplicated data across the organization 🔄. Poor Data Governance: Lack of strict governance policies results in inconsistencies, redundancies, and inaccuracies 🔍.Low Data Literacy: Organizations often underestimate the need to cultivate a data-driven culture, which hampers effective data management 📊Lack of Collaboration: The absence of proper data contracts between software and data engineers can cause disruptions when production systems are updated without consulting the data teams 🤝.Real-World Consequences: Unused Dashboards and Complicated SQL Data Pipelines:hese consume resources for maintenance but fail to provide value, adding to the confusion and operational inefficiency 📉Breaks in Critical Business Models: Minor changes in data management can have major repercussions, disrupting key business operations and decision-making processes ⚠️Strategies to Manage Data Debt: The article emphasizes proactive measures: Prioritizing Data Quality: It's crucial for organizations to enforce rigorous data management standards and practices ✔️. Fostering Collaboration: Enhancing communication and understanding between different teams can mitigate the risks associated with data change 🤝Empowering Data Ownership: Assigning clear ownership of data processes and assets ensures accountability and facilitates quicker resolutions to issues 🔑Moving Forward: Santos suggests that like managing financial debt, addressing data debt requires strategic decision-making to balance immediate costs against long-term benefits. This approach will help organizations not only reduce data debt but also leverage their data assets more effectively for competitive advantage 🌟. Let's discuss how we can turn data management from a liability into an asset! 🚀 #DataDebt #TechnicalDebt #DataGovernance #DataManagement #DigitalTransformation #DataLiteracy #DataQuality #BigData #DataScience #Analytics

41

5 comentários

PySpark is the Python API for Apache Spark. It enables you to perform real-time, large-scale data processing in a distributed environment using Python. It also provides a PySpark shell for interactively analyzing your data. PySpark combines Python’s learnability and ease of use with the power of Apache Spark to enable processing and analysis of data at any size for everyone familiar with Python. PySpark supports all of Spark’s features such as Spark SQL, DataFrames, Structured Streaming, Machine Learning (MLlib) and Spark Core. Overcoming Challenges: Connecting PySpark to a SQL Database Engine for Massive Data Processing In the world of big data, integrating various technologies seamlessly is often a daunting task. One of the significant challenges I faced was connecting from a PySpark session to a SQL database engine and efficiently distributing the memory and CPU cache for massive data processing. Here’s how I tackled this complex problem. 1.Establishing a reliable connection between PySpark and a SQL database engine. 2.Ensuring the efficient distribution of memory and CPU resources. 3.Handling massive datasets without compromising performance. Pyspark Scala

5

EN Hey guys, post I wrote about how to use PySpark with Airflow, I hope you like it! PT Hey guys, post que escrevi sobre como usar PySpark com Airflow, espero que gostem! https://lnkd.in/du9T7kGQ #dataengineering #datascience

5

🚀 Still think centralized data teams are the future? Think again! It’s time to embrace the power of Data Mesh! 🌐 Our "Data Mesh Simplified PDF Guide" covers: The shift from monolithic to microservices-based data management 🧩 Domain-oriented ownership for better scalability and autonomy 🌍 Principles of Data Mesh: Data as a product, federated governance, and self-serve data platforms 🔄 🚀 Unveiling the world of data science through Shivai, my own startup venture. I'm Tripathi Aditya Prakash, and I'm here to make data science, AI, and machine learning approachable and exciting for everyone. 🌟 🎧 Podcasts Alert! Dive into the world of data science with my podcasts: Shivai Data Revelations: Join me as we explore the latest trends and breakthroughs in data science. Whether you're just starting out or are a seasoned expert, this podcast offers a wealth of knowledge and insights. Listen on Spotify: https://lnkd.in/dA2iwwFD That Data Guy: A special podcast for those navigating the corporate data world. Packed with practical tips and real-world experiences, it's a must-listen for data professionals at the start of their journey. Tune in for advice that hits the mark. 📊💼 . Listen on Spotify: https://lnkd.in/ectC_Es2 📚 Expand your data knowledge on my Medium page: https://lnkd.in/dirGC7ca 🎥 Explore the visual side of analytics and machine learning on my YouTube channel: https://lnkd.in/dvvWVxgF , https://lnkd.in/eFdu7TeX 📸 Discover the art of data on my Instagram: https://lnkd.in/dHwMhezE , https://lnkd.in/esjCmu6V 🐦 Get quick bites of data science on my Twitter: https://lnkd.in/gc7xvMya , https://lnkd.in/gthe8zcY Substack: Data Dilemmas – Join the community for unique insights at the nexus of data and personal development : https://lnkd.in/gBRDwjqV With Shivai, let's navigate the vast ocean of data science and decode the future, one byte at a time. ✨ 🤔 How could Data Mesh transform your organization’s data strategy? Share your thoughts below! Unlock the future of data with our guide 📘 #DataMesh #DataScience #TechInnovation #ShivaiTech 🔄 Appreciate my content? Stay connected and engage with my posts. Follow me, Tripathi Aditya Prakash, and Shivai for a continuous stream of enlightening data science content!

1

Data analysts, data scientists and data engineers often struggle to find and share relevant data and insights, especially when data is sensitive and cannot be readily shared. The issue is further exacerbated when an organization’s data assets are spread across multiple catalogs and engines. Learn how Snowflake Horizon enables data teams to quickly discover and share data, apps and models, both within and outside their organizations.

64

Data quality in ETL with Pandera and Python. - Repeated bugs break trust. - Data contracts to ensure a clear Pipeline. Thanks to Luciano Vasconcelos Filho and Renan Heckert.

7

1 comentário

Efficient Architectures in data engineering Efficient architectures in data engineering are essential for maximizing the performance and scalability of data pipelines and systems. These architectures encompass various components, including data storage, processing frameworks, and computational resources, all designed to optimize the flow of data from ingestion to analysis. Key considerations in designing efficient architectures include choosing the right data storage technologies, such as relational databases, NoSQL databases, or data lakes, based on the specific requirements of the application. Additionally, employing distributed computing frameworks like Apache Hadoop or Apache Spark enables parallel processing of large datasets, improving overall performance. For more info read : https://lnkd.in/gnhZSZG7 #nixontechnologiesllc #nixon #nixontechnologysystems #ITservice #nixontech #enterprisesintegration #dataanalytics #cloudcomputing #ITconsultantinindia #USAITconsultant #bestitconsultant #usitconsultant #CRM #dataengineering #softwarejobsinusa #itjobsinusa #usaconsultant #itjobsconsultant #ITInfrastructure2024 #SecOps #SecurityManagement #dataSecurity #CyberSecurity #datascience #cloud #logistics #devops #automachine

1

A Case for Data Lakehousekeeping 📝 Meet Flavio Junqueira's new blog post! It covers the evolution of data lakehouse architecture and how it's changing the game for data management. It also covers how open-table formats like Apache Iceberg and Delta Lake are simplifying processes and boosting performance. It shows the benefits of multi-dimensional indexing with Qbeast-Spark. Flavio uses experiments with the NYC Yellow Taxi dataset to demonstrate how optimizing your data layout can dramatically improve query performance and cut down processing time. Explore all the details: https://lnkd.in/d9BSeFR7 This post is about a presentation at J On The Beach, a conference on multiple topics including data engineering and distributed systems, in May 2024. #DataLakehouse #DataEngineering #QbeastSpark #JOTB24

9

❤️ My 3 Rules for Mentoring Data Engineers Article by Tim Webster ➡ https://lnkd.in/egMSVNU7 - - - - - - - - - - - - - - - - - - - - - - - - - - - 💡 Follow for more great content! ✅ ✍️ Want to contribute? Connect with us. 🔗 Let's create together! ✅ - - - - - - - - - - - - - - - - - - - - - - - - - - - #dataengineering  #datascience #dataanalysis #careerdevelopment

4

One of my targets this year is to master Data Analysis skills, so I'm coming back to review my knowledge in SQL starting this week! #data #sql #dataanalysis #dataanalytics #product #productmanager #datadriven #strategy #showmedata #innovation #decisionmaking

8

New badge that I have achieved helps me in the fundamental understanding of models, sources, tests, documents, and deployment in dbt.

9

Data mesh and data fabric offer different approaches to data management. While data fabric overlays existing data pipelines to provide a unified view, data mesh is more about decentralization and domain-specific ownership. The former maintains legacy infrastructure, while the latter redistributes data control to individual units. Each framework poses unique benefits and challenges, shaping the data landscape in different ways. Which approach resonates more with your organization's data strategy? #DataMesh #DataFabric #DataAnalysis Video by GrowthMatch

10