Yang Xiang’s Post

Prometheus + OpenTelemetry = 🌟 The Prometheus community is deeply invested in advancing the capabilities of OpenTelemetry, an observability framework designed to manage telemetry data such as traces, metrics, and logs. Collaborating closely with the OpenTelemetry community, the Prometheus team has worked to ensure seamless integration between the two systems. Notable achievements include the development of an official specification for converting data between Prometheus and OpenTelemetry, as well as implementations allowing the ingestion of Prometheus metrics into OpenTelemetry Collector and vice versa. Looking ahead to 2024, the Prometheus community is committed to further enhancing OpenTelemetry support, with plans to release Prometheus 3.0 featuring OTel support as a key feature. This initiative aims to establish Prometheus as the default store for OpenTelemetry metrics, with improvements including OTLP ingestion GA, UTF-8 support for metric and label names, native support for resource attributes, and expanded OTLP export capabilities across the Prometheus ecosystem. Read more here: https://hubs.li/Q02tLLP50 Explore the unparalleled capabilities of @Codegiant, your gateway to powerful and cost-effective observability solutions. With seamless integration with both Prometheus and OpenTelemetry, Codegiant Observability empowers you to harness the full potential of these leading frameworks. Gain valuable insights into your infrastructure and applications with our in-built Grafana dashboard, whether monitoring performance metrics, analyzing logs, or tracing application behavior. Elevate your observability strategy with Codegiant today: https://hubs.li/Q02tLCdz0 #monitoring #observability #sre #devops #codegiant

There is a common painful workflow with many #observability solutions. Each data type is separated into its own user interface, creating a disjointed workflow that increases cognitive load and slows down Mean Time to Diagnose (MTTD). At Coralogix, we aim to give our customers the maximum possible insights for the minimum possible effort. We’ve expanded our APM features (see documentation) to provide deep, contextual insights into applications – but we’ve done something different. See what it is 👇👇

Instrumentation for Event Driven eventdriven Woovi invests a lot in observability as discussed in this article https://lnkd.in/e_uJw3YA. In this article, we explain how we instrument our events in our event-driven architecture. Observability Concepts Before dive in, let's define a few basic observability concepts. A Transaction represents one full request, one job, one unit of work. All POST /charge will have the same transaction type and name. A Transaction has a start and an end. A Span represents a unit of work inside a transaction. A Span has a start and an end. Examples of spans: database query, http request, redis request, etc. A Label is a metadata attached to a transaction or span. Event-Driven Instrumentation We use Elastic APM as our Application Performance Monitoring. elastic-apm-node auto instrument many packages like koa, mongodb and redis, but we also need to manually instrument to address Woovi needs. We use bull-js for our distributed queue and event-driven library. To register a function as a handler for an event, we do this in bull-js https://lnkd.in/egUzweY7

We just dropped a detailed comparison between Weights and Biases and Helicone. Both Helicone and WandB are #opensource and can handle massive scale. Recently, we integrated Upstash Kafka into our core data pipeline to ensure 100% log coverage and we use Cloudflare Workers to ensure sub-millisecond latency impact. Helicone is ideal if: ✅ you want a simple way to track and manage production metrics for your LLM. ✅ you are looking for a cost-effective option for high-volume usage. ✅ you want your entire team (technical and non-technical users) to easily derive value from the dashboard. Ask us a question below! Or check out the detailed comparison → https://lnkd.in/ggX_ZA7Y

I'm proud to announce support for the ingestion of #OTLP logs with our latest Datadog Agent release (v7.48)! With this new feature, our Datadog Agent now supports the ingestion of all three pillars of observability (traces, metrics and logs) in OTLP format from #OpenTelemetry (OTel)-instrumented applications. This allows you to easily access OOTB Datadog capabilities with OpenTelemetry data to achieve your monitoring goals. Learn the benefits of Agent OTel Logs ingestion and how to enable this feature today in the blog post! Kudos to our intern Ibraheem Aboulnaga and mentorship from our eng Yang Song for making this happen! #datadog #observability #SRE #devops

👀 Wow… Your Data! Your Ops! The models you choose! Super simple to register your own Llama2 model to MLflow allowing you to host/serve it on your own for chat, apps, embedding, etc.

Elastic just released our new piped query language, ES|QL (Elasticsearch Query Language), and transforms, enriches, and simplifies data investigations. If you are here at #kubecon2023 come to our booth to learn how it helps investigate with #Kubernetes and #OpenTelemetry. Also check out my latest blog https://lnkd.in/gXycY4CY

Llama2 in your laptop, with MLflow Model Management! Folks - I developed a MLflow custom pyfunc model for Llama2+llama.cpp. The two tasks supported are Chat and Embedding Generation. Using this Apache Licensed open source code, you can register your Llama2 model as a MLflow model. You can realize the value of MLflow model management such as model lifecycle management, model authorization, etc. Here's a detailed write up: #mlflow #llama2 #llamacpp #databricks #infinstor #sagemaker #azureml #llm #generativeai https://lnkd.in/gZ5qQW9S

One of the things I love about #hamiltonOS is that it can really help simplify the modern data stack by integrating exciting new techologies! In this post, Thierry Jean shows how to use #Ibis., dltHub, and #hamiltonOS to transform slack data. This also has a great overview of how #dataengineering works (ETL versus ELT) and uses #LLMs to summarize the results. I'm really exciteda bout the approach represented here -- it is simple, powerful, portable across orchestrators, and highly extensible. https://lnkd.in/gX-BC8Ym

Airbyte is now able to write both structured and unstructured data to Vector Databases! There are so many new use cases this can unlock. Here’s a tutorial by Joe Reuter showing how to build a “connector development support” chat bot that knows Airbyte APIs, open feature requests and previous Slack conversations. 🤖 By the end of this tutorial, you will know how to: - Extract unstructured data from a variety of sources using Airbyte - Use Airbyte to efficiently load data into a vector database, preparing the data for LLM usage along the way - Integrate a vector database into your LLM to ask questions about your proprietary data 🔗 Dive into the tutorial: https://lnkd.in/dTtvdSsg #dataengineering #llm #openai