Astrafy

How close is AI to replacing product managers? 🤖 We read this interesting article from Lenny’s Newsletter [https://lnkd.in/dNTWf3an] last week, and the image below illustrates one of the main takeaways. Explicit 🆚 Tacit information In any organization, a significant portion of valuable information is tacit (hidden knowledge, emotions, etc). This information often has more value than explicit information and frequently drives final decisions. And without much surprise, AI cannot make sense of tacit information because it’s simply not accessible anywhere to consume. As long as tacit information remains significant in each organization, humans will still have a central place. The more you transition from a tacit-oriented organization to an explicit-oriented organization, the more you can benefit from and leverage the power of AI. How is your organization balancing tacit and explicit information? 🤔 #AI #information #data

Everyone talks about Data Governance, but no one fully masters it 🤔. There are multiple reasons, but the main one is that shipping data operations always prioritize “nice to have” tasks such as data governance. Then, automation and tooling around data governance can be complex and involve some manual steps. This latter barrier is being tackled seriously by BigQuery by integrating data governance directly within BigQuery UI. • Automatic Data Quality: Built-in rule recommendations, serverless execution, and alerting ensure high-quality data at scale. You can view quality metrics directly in BigQuery Studio. • Unified Data Catalog: Search and discover insights from all your data, including Vertex AI models, datasets, and features, directly within BigQuery. • Comprehensive Data Lineage: Track the end-to-end movement of data, including column-level lineage for BigQuery and Vertex AI pipelines (preview). • Automated Governance Rules: Define and enforce data access policies at scale with fine-grained controls, streamlining governance workflows. 💡 Data Governance is now at your fingertips and can greatly democratize access to your data and improve data quality — no more excuse to consider it as a second-class citizen. #BigQuery #DataGovernance #DataQuality #DataCatalog #DataLineage

Have you ever struggled as a data/analytics engineer to get data from source applications or get clear guidelines from the business for the data they need 🤔? If yes, you are probably experiencing the effects of having silos departments around your data instead of transversal data product teams. Traditionally, you would have: 👉 Software engineers responsible for dumping the data from internal databases or 3rd party applications 👉 Data/Analytics engineers responsible for moving the data from ingestion to curated datamarts to be used by end users  👉 End users (data analysts, BI engineers, ML engineers, etc.) and Business stakeholders consuming this data This approach has the downside that each team has different goals instead of having a common goal to bring value to the business. Having transversal data product teams where software engineers, data engineers, analytics engineers, and end users all collaborate leads to streamlined operations with increased data quality and alignment on the business value. While transversal data product teams are the optimal design, it’s important to have a centralized data platform and governance teams that provide the infrastructure and governance guidelines to each data product team. Infra and governance need to be standardized and monitored by central teams so that security is consistent and that the infra wheel is not reinvented with each new data product. #datamesh #datateams

Looking for a stack to build your next-gen data department? Ultimately, it’s less about the tools (as there are hundreds of good ones) and more about how you use those tools, integrate those tools with the rest of your stack, and how you treat data as a product. That being said, here is our preferred data stack that can lay the foundations for a great data ecosystem to thrive 🚀 ➡️ Ingestion: Airbyte - OSS tool of choice with hundreds of built-in connectors to ingest data from databases and 3rd party applications. ➡️ Raw storage: Google Cloud Storage (GCS) - stores any kind of data formats and acts as the data lake for all your raw data from your internal database and 3rd party applications. ➡️ Analytical database: BigQuery - Best of breed with advanced SQL functions, possibility to run spark code, to trigger external Cloud Functions, and so on. There is really not much you cannot do on BigQuery. ➡️ Transformation framework: dbt - Reference SQL framework for the last five years, it brings software engineering best practices to the data world. ➡️ Orchestrator: Airflow - It has been leading the game of data orchestration for the last ten years and it’s doing a great job at moving your data from the source up to your downstream applications (datamarts, ML models, etc.) Of course, there is much more to a data stack but to get buy-in, it’s important to go by iterations, and by setting up properly the five components above, you will be ready to start your modern data journey. Ready to get your next data stack? Get in touch! #DataLake #GoogleCloudStorage #BigQuery #CloudComposer

We are excited to have our team speaking at the Airflow Summit for the second time! Join us to explore dynamic workflow orchestration with Airflow Datasets, live data pipelines, and dbt in Google Cloud. See you there! https://airflowsummit.org/

Regulations can be daunting and lead to severe consequences if not followed thoroughly. The data world is no exception and regulations are evolving quicker than ever. If you are looking for the best Cloud to meet regulations and keep your data safe, look no further. Google Cloud offers a robust suite of services to empower you to secure your sensitive data while adhering to industry standards. 🔒 Multi-Layered Security on GCP: ➜ Google Cloud Identity and Access Management (IAM): Fine-grained access control system enforces the principle of least privilege. ➜ Cloud Key Management Service (KMS): Centrally manage and protect encryption keys used to safeguard your data at rest and in transit. ➜ Cloud Data Loss Prevention (DLP): Identify and prevent sensitive data exfiltration attempts. ➜ Cloud Security Command Center (SCC): Gain centralized visibility into your security posture on Google Cloud. SCC also offers a tool to diagnose your percentage of compliance with different regulations and policies. And much more—those are just some of the key products. Why is it important? ✅ Meet stringent data privacy regulations like GDPR, CCPA, and HIPAA. 🌐 ✅ Safeguard sensitive data throughout its lifecycle with encryption, access controls, and data loss prevention mechanisms. 🛡️ ✅ Demonstrate your commitment to data privacy and build trust with customers and partners. 💼 Would you like to leverage Google Cloud services to fortify your data security posture and comply with the regulations in place within your sector/country? Let’s connect! 💬 #DataPrivacy #Compliance #GoogleCloud #IAM #CloudIdentity #KMS #DLP #SecurityCommandCenter #BigQuery #VertexAI

Ever been taken by surprise by a huge Cloud bill? 😲💸  Effective financial operations (FinOps) are crucial for managing and optimizing cloud costs in the cloud. By implementing FinOps practices, you can get the most value out of your cloud investment. If you are on Google Cloud, here are a few tools to get you started on your FinOps journey: ➜ Google Cloud Billing: This core service provides a comprehensive view of your cloud spending across projects, accounts, and regions. You can set up budgets with notifications and also set up a sink of your billing data to BigQuery. ➜ BigQuery: Once your billing data is In BigQuery, you can leverage SQL to get insights about all your billing data. You can identify trends, uncover potential cost savings opportunities, and create custom reports to gain deeper insights into your cloud spend. ➜ FinOps Hub: This AI-powered service analyzes your Google Cloud usage and recommends cost-saving optimizations. These recommendations can include switching to committed use discounts, using managed services that can be more cost-effective than self-managed deployments, or utilizing regional pricing options. ➜ Cloud Identity and Access Management (IAM): Having “least privileged access” is an important step to limit employees inadvertently creating resources they are not allowed to create. FinOps Benefits: ✅ Improved Cost Visibility: Gain a clear understanding of your cloud spending patterns and identify areas for optimization. ✅ Optimized Resource Usage: Right-size your cloud resources to ensure you're paying only for what you need. ✅ Enhanced Financial Accountability: Empower teams to make informed decisions about cloud resource utilization and spending. ✅ Increased Agility and Innovation: By controlling cloud costs, you can free up resources for innovation and new business initiatives. Astrafy helps clients implement FinOps practices to achieve significant cost savings. We have built over the years a FinOps accelerator that can be deployed in a matter of hours with the power to get double-digit savings. 💰 #FinOps #CostManagement #GoogleCloud #BigQuery #CloudFunctions

🤔 Are you struggling to automate your data flow from ingestion to distribution while ensuring data quality? DataOps addresses the same challenges in data engineering that DevOps handles in software engineering. Make sure that your data operations run smoothly by removing the toil and automating as much as possible. 🔧 Our Toolkit: ➜ Application deployment with ArgoCD: Automate deployment of data applications on Kubernetes. Your favorite tools need a strong and scalable engine. ➜ IaC with Terraform: Reference tool for infrastructure as code. Your data workloads cannot run  ➜ CI/CD with Gitlab CI / GitHub Actions: Each push of new transformation code must trigger CI pipelines to check for data quality (through unit tests and data-diff), data linting with SQLfluff, and then packaging your code into a versioned docker image to be used by your orchestrator. ➜ Orchestration with Airflow: The glue that puts it all together. For your data to move seamlessly from ingestion to distribution with no manual steps, you need an orchestrator to achieve this. Why DataOps? ✅ Enhance data quality and consistency. 🛡️ Reduce manual errors. ⚡ Speed up deployment cycles. From our experience, DataOps is where data teams lack the most skills and experience and this often leads to inefficiencies for deploying data products in production. Production requires automation and consistency in deployment, testing, rolling back quickly to previous versions, etc. #DataOps #CICD #GoogleCloud #ArgoCD #Terraform #CloudBuild

🚀 Traditional maintenance schedules can be costly and inefficient. But with AI and machine learning, we can predict equipment failures before they happen, ensuring smooth operations and cost savings. Tools of the Trade: ➜ Vertex AI: Develop and deploy machine learning models for predictive maintenance. ➜ BigQuery ML: Run machine learning directly on your data warehouse for real-time insights. ➜ Cloud Functions: Automate responses to maintenance alerts. Benefits: ✅ Reduce operational costs. ✅ Increase equipment lifespan. ✅ Enhance safety and reliability. At Astrafy, we’ve helped clients integrate these tools to predict failures and optimize maintenance schedules. Ready to revolutionize your maintenance strategy? Let's chat! 💬 #AIPredictiveMaintenance #GoogleCloud #VertexAI #BigQueryML #CloudFunctions

🔥Yesterday's #dbt Madrid #4 was an amazing experience, thanks to everyone who joined us! A special thanks to, Rubén Ibáñez Pinillo and Miguel Martín Tapia from Mercadona Tech 🤝 for their insightful talk on the benefits and challenges of using dbt for efficient data transformations and quality management. The event was a wonderful opportunity for learning and networking, bringing together professionals passionate about data. Your interest and insights are what make these meetups valuable. 👉 Thank you for making #dbt Madrid #4 memorable. Join our meetup group for details on our next event 🔗 https://lnkd.in/dANrdtiT #dbtmeetup #dbtsemanticlayer #techtalks #dataengineering