The document provides an introduction and overview of Apache Kafka presented by Jeff Holoman. It begins with an agenda and background on the presenter. It then covers basic Kafka concepts like topics, partitions, producers, consumers and consumer groups. It discusses efficiency and delivery guarantees. Finally, it presents some use cases for Kafka and positioning around when it may or may not be a good fit compared to other technologies.
AWS EMR을 사용하면서 비용을 최적화하기 위해 필요한 다양한 관점의 방안을 검토하여 정리한 자료.
비용 최적화 대상은 zeppelin/jupyter notebook과 apache spark를 활용하는 서비스를 대상으로 하였으며, 해당 작업이 aws emr에서 어��게 동작하는지 내부 구조을 파악하여 확인함.
- AWS EMR이란?
- AWS EMR의 과금 방식은?
- 어떻게 비용을 최적화 할 것인가?
- 최적의 EMR 클러스터 구성 방안
- 가성비 높은 Instance 선정 방안
- Apache Spark 성능 개선 방안
가장 중요한 것은 실행할 job의 자원사용량/성능을 모니터링하고, 이에 맞게 자원을 최적화하는 것이 필요함.
fluent-plugin-beats at Elasticsearch meetup #14N Masahiro
This document summarizes a presentation about integrating Fluentd with Elastic Beats data collection agents. It introduces Beats, their supported outputs including Elasticsearch, and various third party Beats. It then describes the fluent-plugin-beats plugin which allows Fluentd to receive events from Beats using the Lumberjack protocol. An example configuration is shown. Performance tests show Fluentd can handle over 100,000 events/sec while Filebeat is slower at 18,000 events/sec. The conclusion is that Beats are useful for collection but Fluentd may be better than Filebeat for high volume environments.
A Thorough Comparison of Delta Lake, Iceberg and HudiDatabricks
Recently, a set of modern table formats such as Delta Lake, Hudi, Iceberg spring out. Along with Hive Metastore these table formats are trying to solve problems that stand in traditional data lake for a long time with their declared features like ACID, schema evolution, upsert, time travel, incremental consumption etc.
Fluentd is an open source data collector that allows flexible data collection, processing, and storage. It collects log data from various sources using input plugins and sends the data to various outputs like files, databases or forward to other Fluentd servers. It uses a pluggable architecture so new input/output plugins can be added through Ruby gems. It provides features like buffering, retries and reliability.
Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안SANG WON PARK
Apache Kafak의 빅데이터 아키텍처에서 역할이 점차 커지고, 중요한 비중을 차지하게 되면서, 성능에 대한 고민도 늘어나고 있다.
다양한 프로젝트를 진행하면서 Apache Kafka를 모니터링 하기 위해 필요한 Metrics들을 이해하고, 이를 최적화 하기 위한 Configruation 설정을 정리해 보았다.
[Apache kafka 모니터링을 위한 Metrics 이해 및 최적화 방안]
Apache Kafka 성능 모니터링에 필요한 metrics에 대해 이해하고, 4가지 관점(처리량, 지연, Durability, 가용성)에서 성능을 최적화 하는 방안을 정리함. Kafka를 구성하는 3개 모듈(Producer, Broker, Consumer)별로 성능 최적화를 위한 …
[Apache Kafka 모니터링을 위한 Metrics 이해]
Apache Kafka의 상태를 모니터링 하기 위해서는 4개(System(OS), Producer, Broker, Consumer)에서 발생하는 metrics들을 살펴봐야 한다.
이번 글에서는 JVM에서 제공하는 JMX metrics를 중심으로 producer/broker/consumer의 지표를 정리하였다.
모든 지표를 정리하진 않았고, 내 관점에서 유의미한 지표들을 중심으로 이해한 내용임
[Apache Kafka 성능 Configuration 최적화]
성능목표를 4개로 구분(Throughtput, Latency, Durability, Avalibility)하고, 각 목표에 따라 어떤 Kafka configuration의 조정을 어떻게 해야하는지 정리하였다.
튜닝한 파라미터를 적용한 후, 성능테스트를 수행하면서 추출된 Metrics를 모니터링하여 현재 업무에 최적화 되도록 최적화를 수행하는 것이 필요하다.
This document summarizes the key features and changes between versions of Fluentd, an open source data collector.
The main points are:
1) Fluentd v1.0 will provide stable APIs and features while remaining compatible with v0.12 and v0.14. It will have no breaking API changes.
2) New features in v0.14 and v1.0 include nanosecond time resolution, multi-core processing, Windows support, improved buffering and plugins, and more.
3) The goals for v1.0 include migrating more plugins to the new APIs, addressing issues, and improving documentation. A release is planned for Q2 2017.
Apache Kafka becoming the message bus to transfer huge volumes of data from various sources into Hadoop.
It's also enabling many real-time system frameworks and use cases.
Managing and building clients around Apache Kafka can be challenging. In this talk, we will go through the best practices in deploying Apache Kafka
in production. How to Secure a Kafka Cluster, How to pick topic-partitions and upgrading to newer versions. Migrating to new Kafka Producer and Consumer API.
Also talk about the best practices involved in running a producer/consumer.
In Kafka 0.9 release, we’ve added SSL wire encryption, SASL/Kerberos for user authentication, and pluggable authorization. Now Kafka allows authentication of users, access control on who can read and write to a Kafka topic. Apache Ranger also uses pluggable authorization mechanism to centralize security for Kafka and other Hadoop ecosystem projects.
We will showcase open sourced Kafka REST API and an Admin UI that will help users in creating topics, re-assign partitions, Issuing
Kafka ACLs and monitoring Consumer offsets.
Log Management
Log Monitoring
Log Analysis
Need for Log Analysis
Problem with Log Analysis
Some of Log Management Tool
What is ELK Stack
ELK Stack Working
Beats
Different Types of Server Logs
Example of Winlog beat, Packetbeat, Apache2 and Nginx Server log analysis
Mimikatz
Malicious File Detection using ELK
Practical Setup
Conclusion
HTTP Analytics for 6M requests per second using ClickHouse, by Alexander Boc...Altinity Ltd
This document summarizes Cloudflare's use of ClickHouse to analyze over 6 million HTTP requests per second. Some key points:
- Cloudflare previously used PostgreSQL, Citus, and Flink but these did not scale sufficiently.
- ClickHouse was chosen as it is fast, scalable, fault tolerant, and Cloudflare had existing expertise in it.
- Cloudflare designed ClickHouse schemas to aggregate HTTP data into totals, breakdowns by category, and unique counts into two tables using different engines.
- Tuning ClickHouse index granularity improved query latency by 50% and throughput by 3x.
- The new ClickHouse pipeline is more scalable, fault tolerant
Apache Spark on K8S Best Practice and Performance in the CloudDatabricks
Kubernetes As of Spark 2.3, Spark can run on clusters managed by Kubernetes. we will describes the best practices about running Spark SQL on Kubernetes upon Tencent cloud includes how to deploy Kubernetes against public cloud platform to maximum resource utilization and how to tune configurations of Spark to take advantage of Kubernetes resource manager to achieve best performance. To evaluate performance, the TPC-DS benchmarking tool will be used to analysis performance impact of queries between configurations set.
Speakers: Junjie Chen, Junping Du
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013mumrah
Apache Kafka is a distributed publish-subscribe messaging system that allows both publishing and subscribing to streams of records. It uses a distributed commit log that provides low latency and high throughput for handling real-time data feeds. Key features include persistence, replication, partitioning, and clustering.
From cache to in-memory data grid. Introduction to Hazelcast.Taras Matyashovsky
This presentation:
* covers basics of caching and popular cache types
* explains evolution from simple cache to distributed, and from distributed to IMDG
* not describes usage of NoSQL solutions for caching
* is not intended for products comparison or for promotion of Hazelcast as the best solution
Producer Performance Tuning for Apache KafkaJiangjie Qin
Kafka is well known for high throughput ingestion. However, to get the best latency characteristics without compromising on throughput and durability, we need to tune Kafka. In this talk, we share our experiences to achieve the optimal combination of latency, throughput and durability for different scenarios.
In the session, we discussed the End-to-end working of Apache Airflow that mainly focused on "Why What and How" factors. It includes the DAG creation/implementation, Architecture, pros & cons. It also includes how the DAG is created for scheduling the Job and what all steps are required to create the DAG using python script & finally with the working demo.
Arrow Flight is a proposed RPC layer for Apache Arrow that allows for efficient transfer of Arrow record batches between systems. It uses GRPC as the foundation to define streams of Arrow data that can be consumed in parallel across locations. Arrow Flight supports custom actions that can be used to build services on top of the generic API. By extending GRPC, Arrow Flight aims to simplify the creation of data applications while enabling high performance data transfer and locality awareness.
This document describes a presentation about introducing black magic programming patterns in Ruby and their pragmatic uses. It provides an overview of Fluentd, including what it is, its versions, and the changes between versions 0.12 and 0.14. Specifically, it discusses how the plugin API was updated in version 0.14 to address problems with the version 0.12 API. It also explains how a compatibility layer was implemented to allow most existing 0.12 plugins to work without modification in 0.14.
This document discusses middleware in Ruby and provides examples of considerations when writing middleware:
- Middleware should be a long-running daemon process that is compatible across platforms and environments and handles various data formats and traffic volumes.
- Tests must be run on all supported platforms to ensure compatibility as thread and process scheduling differs between operating systems.
- Memory usage and object leaks must be carefully managed in long-running processes to avoid consuming resources over time.
- Performance of JSON parsing/generation should be benchmarked and the most optimized library used to avoid unnecessary CPU usage.
This document discusses making the Norikra stream processing software more perfect. It outlines how Norikra currently works well for small to medium sites but has limitations for large deployments. The concept of a "Perfect Norikra" is introduced that would add distributed execution, high availability, and dynamic scaling capabilities. A rough design is sketched that involves a new query executor, dataflow manager, and strategies for dynamic scaling through intermediate results and merging across nodes. Challenges mentioned include resource monitoring, multi-tenancy, and supporting queries without aggregations.
Fighting API Compatibility On Fluentd Using "Black Magic"SATOSHI TAGOMORI
The document discusses Fluentd's changes to its plugin API between versions 0.12 and 0.14. In 0.14, the API was overhauled to separate entry points from implementations and introduce a plugin base class to control data and control flow. A compatibility layer was added to allow most 0.12 plugins to work unmodified in 0.14 by handling calls to overridden methods. However, plugins that override certain methods like #emit may cause errors due to changes in how buffering works.
Open Source Software, Distributed Systems, Database as a Cloud ServiceSATOSHI TAGOMORI
- Treasure Data is a database as a cloud service company that collects and stores customer data beyond the cloud [1].
- It uses open source software like Fluentd and MessagePack to easily integrate and collect data from customers [2]. It also uses open source distributed systems software like Hadoop and Presto to store, process and query large amounts of customer data [3].
- As a database service, it needs to share computer resources securely for many customers. It contributes to open source to build and maintain the distributed systems software that powers its cloud database service [4].
This document discusses the pros and cons of building an in-house data analytics platform versus using cloud-based services. It notes that in startups it is generally better not to build your own platform and instead use cloud services from AWS, Google, or Treasure Data. However, the options have expanded in recent years to include on-premise or cloud-based platforms from vendors like Cloudera, Hortonworks, or cloud services from various providers. The document does not make a definitive conclusion, but discusses factors to consider around distributed processing, data management, process management, platform management, visualization, and connecting different data sources.
This document discusses features that could make Norikra, an open source stream processing software, even more "perfect". It describes how Norikra currently works and highlights areas for improvement, such as enabling queries to resume processing from historical batch query results, sharing operators between queries to reduce memory usage, and developing a true lambda architecture with a single query language for both streaming and batch processing. The document envisions a "perfect stream processing engine" with these enhanced capabilities.
This document discusses using Ruby for distributed storage systems. It describes components like Bigdam, which is Treasure Data's new data ingestion pipeline. Bigdam uses microservices and a distributed key-value store called Bigdam-pool to buffer data. The document discusses designing and testing Bigdam using mocking, interfaces, and integration tests in Ruby. It also explores porting Bigdam-pool from Java to Ruby and investigating Ruby's suitability for tasks like asynchronous I/O, threading, and serialization/deserialization.
Distributed Logging Architecture in Container EraSATOSHI TAGOMORI
Distributed Logging Architecture in Container Era
The document discusses distributed logging architecture in the container era. It covers: 1) The difficulties of logging with microservices and containers due to their ephemeral and distributed nature, 2) The need to redesign logging to push logs from containers to destinations quickly without fixed addresses or mappings; 3) Common patterns for distributed logging architectures including source aggregation, destination aggregation, and scaling; and 4) A case study using Docker and Fluentd to implement source aggregation and scaling for logging. Open source solutions are important to keep the logging layer transparent, interoperable, and able to scale independently of applications and infrastructure.
Distributed Logging Architecture in the Container EraGlenn Davis
Presentation given at LinuxCon Japan 2016 by Satoshi "Moris" Tagomori (@tagomoris), Treasure Data. Describes various strategies for aggregating log data in a microservices architecture using containers, e.g. Docker.
Fluentd and Distributed Logging at KubeconN Masahiro
This document discusses distributed logging with containers using Fluentd. It notes the challenges of logging in container environments where logs need to be collected from ephemeral containers and transferred to storage. It introduces Fluentd as a flexible data collection tool that can collect logs from containers using various plugins and methods like log drivers, shared volumes, and application libraries. The document discusses deployment patterns for Fluentd including using it for source-side aggregation to buffer and transfer logs more efficiently and for destination-side aggregation to scale log storage.
This document discusses real-time analytics on streaming data. It describes why real-time data streaming and analytics are important due to the perishable nature of data value over time. It then covers key components of real-time analytics systems including data sources, stream storage, stream ingestion, stream processing, and stream delivery. Finally, it discusses streaming data processing techniques like filtering, enriching, and converting streaming data.
The document discusses a tool called the BizTalk Migrator that helps migrate BizTalk applications to Azure Integration Services. It summarizes the tool's capabilities like discovering and parsing BizTalk artifacts. It also discusses what is and isn't supported when migrating things like adapters, pipelines, and orchestrations. The document provides installation instructions and highlights differences between the original and migrated applications that users may encounter.
MongoDB .local Houston 2019: Building an IoT Streaming Analytics Platform to ...MongoDB
Corva's analytics platform enables real-time engineering and machine learning predictions and powers faster and safer drilling. The platform utilizes AWS serverless Lambda & extensible, data-driven API with MongoDB to handle 100,000+ requests per minute of streaming sensor data.
Ai big dataconference_ml_fastdata_vitalii bondarenkoOlga Zinkevych
This document discusses machine learning on fast data. It presents an agenda covering ML on production systems, TensorFlow, Kafka, Docker and Kubernetes. It then describes the machine learning process and shows how an enterprise analytics platform can integrate data sources, a machine learning cluster using Kafka, and data destinations. It provides details on TensorFlow and how it can be used for linear regression and neural networks. It also explains Apache Kafka as a streaming data service bus and how Confluent Platform extends it. Finally, it briefly introduces Docker and Kubernetes.
Vitalii Bondarenko "Machine Learning on Fast Data"DataConf
This document discusses machine learning on fast data. It presents an agenda covering ML on production systems, TensorFlow, Kafka, Docker and Kubernetes. It then describes the machine learning process and shows how an enterprise analytics platform can integrate data sources, a machine learning cluster using Kafka, and data destinations. Details are provided on using TensorFlow for linear regression and neural networks. Apache Kafka is explained as a distributed streaming platform using topics, brokers, and consumer groups. The Confluent platform, KStream and KTable APIs are also summarized. Docker and Kubernetes are mentioned for containerization.
Serverless SQL provides a serverless analytics platform that allows users to analyze data stored in object storage without having to manage infrastructure. Key features include seamless elasticity, pay-per-query consumption, and the ability to analyze data directly in object storage without having to move it. The platform includes serverless storage, data ingest, data transformation, analytics, and automation capabilities. It aims to create a sharing economy for analytics by allowing various users like developers, data engineers, and analysts flexible access to data and analytics.
MongoDB 3.6 helps you *move at the speed of your data* - turning developers, operations teams, and analysts into a growth engine for the business. It enables new apps to be delivered to market faster, running reliably and securely at scale, and unlocking insights and intelligence in real time. Learn more: https://www.mongodb.com/mongodb-3.6
Following simple patterns of good application design can allow you to scale your application for your customers easily. This presentation dives into the 12 factor application design and demo how this applies to containers and deployments on Amazon ECS and Fargate. We'll take a look at tooling that can be used to simplify your workflow and help you adopt the principles of the 12 factor application.
This presentation will describe how to go beyond a "Hello world" stream application and build a real-time data-driven product. We will present architectural patterns, go through tradeoffs and considerations when deciding on technology and implementation strategy, and describe how to put the pieces together. We will also cover necessary practical pieces for building real products: testing streaming applications, and how to evolve products over time.
Presented at highloadstrategy.com 2016 by Øyvind Løkling (Schibsted Products & Technology), joint work with Lars Albertsson (independent, www.mapflat.com).
Data Analytics Service Company and Its Ruby UsageSATOSHI TAGOMORI
This document summarizes Satoshi Tagomori's presentation on Treasure Data, a data analytics service company. It discusses Treasure Data's use of Ruby for various components of its platform including its logging (Fluentd), ETL (Embulk), scheduling (PerfectSched), and storage (PlazmaDB) technologies. The document also provides an overview of Treasure Data's architecture including how it collects, stores, processes, and visualizes customer data using open source tools integrated with services like Hadoop and Presto.
Reactive Development: Commands, Actors and Events. Oh My!!David Hoerster
Distributed applications are becoming more popular with the increasing popularity of microservices (however you want to define that term). But the principles of distributed application development are key if you want to build a system that is resilient, responsive, elastic and maintainable. In this workshop, we’ll review the principles of CQRS and the Reactive Manifesto, and how they complement each other. We’ll build an application that can handle a large stream of data, and allow users to still have a responsive experience while interacting with real-time and near-real-time data.
We’ll look at Akka.NET as the workhorse inside your services, and how the principles of CQRS can help with your service-to-service communications.
We’ll also look at how Event Sourcing can aid in managing your domain state, and how an event stream can be used to project data for your system for a number of different uses. We’ll build our own simple event store, but also look at commercially available stores, too.
This session will focus on using Akka.NET along with a few other tools and technologies, such as EventStore and MongoDB. The concepts learned in this session will be applicable to a number of different tools, technologies and languages.
BDA402 Deep Dive: Log Analytics with Amazon Elasticsearch ServiceAmazon Web Services
Everything generates logs. Applications, infrastructure, security ... everything. Keeping track of the flood of log data is a big challenge, yet critical to your ability to understand your systems and troubleshoot (or prevent) issues. In this session, we will use both Amazon CloudWatch and application logs to show you how to build an end-to-end log analytics solution. First, we cover how to configure an Amazon Elaticsearch Service domain and ingest data into it using Amazon Kinesis Firehose, demonstrating how easy it is to transform data with Firehose. We look at best practices for choosing instance types, storage options, shard counts, and index rotations based on the throughput of incoming data and configure a secure analytics environment. We demonstrate how to set up a Kibana dashboard and build custom dashboard widgets. Finally, we dive deep into the Elasticsearch query DSL and review approaches for generating custom, ad-hoc reports.
How Netflix Monitors Applications in Near Real-time w Amazon Kinesis - ABD401...Amazon Web Services
Thousands of services work in concert to deliver millions of hours of video streams to Netflix customers every day. These applications vary in size, function, and technology, but they all make use of the Netflix network to communicate. Understanding the interactions between these services is a daunting challenge both because of the sheer volume of traffic and the dynamic nature of deployments. In this session, we first discuss why Netflix chose Kinesis Streams to address these challenges at scale. We then dive deep into how Netflix uses Kinesis Streams to enrich network traffic logs and identify usage patterns in real time. Lastly, we cover how Netflix uses this system to build comprehensive dependency maps, increase network efficiency, and improve failure resiliency. From this session, you'll learn how to build a real-time application monitoring system using network traffic logs and get real-time, actionable insights.
An Azure of Things, a developer’s perspectiveBizTalk360
The world of integration is changing very quickly and we have the opportunity to use a lot of different technologies. There are many ways to solve the same problem and new technologies being introduced all of the time. Azure is now full of very interesting features and the real challenge is understanding how to use and combine all of these together in an effective way to create a good solution. In this session Nino will talk about his experiences and thoughts from the last year around areas such as BizTalk, Hybrid Integration, Microservices, Event Hubs, Stream Analytics and more.
You’ve decided to develop in Azure and need to make a decision on the messaging technology. Storage Queues, Service Bus, Event Grid, Event Hubs, etc. Which technology should you use? How do you pick the right one if they all deal with messages? This session will help you answer these questions.
Highlights of AWS ReInvent 2023 (Announcements and Best Practices)Emprovise
Highlights of AWS ReInvent 2023 in Las Vegas. Contains new announcements, deep dive into existing services and best practices, recommended design patterns.
Talk we gave during IT Press Tour #17, describing the vision, company profile and technology of OpenIO.
See: http://www.theregister.co.uk/2015/12/02/openio_object_storage_upstart/
Similar to The Patterns of Distributed Logging and Containers (20)
Ractor is a new experimental feature in Ruby 3.0 that allows Ruby code to run in parallel on CPUs. It manages objects per Ractor and can move objects between Ractors, making moved objects invisible to the original Ractor. It can share certain "shareable" objects like modules, classes, and frozen objects between Ractors. For web applications to fully utilize Ractors, an experimental application server called Right Speed was created that uses Rack and runs processing workers on Ractors. However, there are still problems to address like exceptions when closing connections and accessing non-shareable constants and instance variables across Ractors before Ractors can be ready for production use in web applications.
Good Things and Hard Things of SaaS Development/OperationsSATOSHI TAGOMORI
This document discusses the good and hard things about developing and operating a SaaS platform. It describes how the backend team at Treasure Data owns and manages various components of their distributed platform. It also discusses how they have modernized their deployment process from a periodic Chef-based approach to using CodeDeploy for more frequent deployments. This allows them to move faster by doing many small releases that minimize the number of affected components and customers.
The document is an invitation to a presentation about Maccro, a Ruby macro system that allows rewriting Ruby methods by registering rules. It summarizes Maccro's capabilities such as registering before/after matchers, applying rules to methods, matching method ASTs to rules, and rewriting method code by replacing placeholders. It also lists some limitations including only supporting Ruby 2.6+, not handling method call order dependencies, and not updating method visibility. The presentation aims to recruit more developers to Maccro to handle its remaining challenges.
The document discusses constants in Ruby programming. It notes that constant names start with capital letters and can be overwritten with warnings. It provides some constant name samples and encourages trying out different Ruby versions. It also discusses making Ruby scripts confusing by using Unicode characters and constant names that are difficult to understand for future maintainers.
This document summarizes a presentation given by @joker1007 and @tagomoris on hijacking Ruby syntax. It discusses various Ruby features like bindings, tracepoints, and refinements that can be used to modify Ruby's behavior. It then demonstrates several "hacks" that leverage these features, including finalist, overrider, and abstriker which add method modifiers, and binding_ninja which allows implicitly passing a binding. Other hacks discussed are with_resources and deferral which add "with" and "defer" statements to Ruby. The presentation emphasizes how these modifications are implemented using hooks, bindings, tracepoints and other Ruby internals.
Lock, Concurrency and Throughput of Exclusive OperationsSATOSHI TAGOMORI
1. The document discusses different patterns for implementing locks in a distributed key-value store to maximize concurrency and throughput of operations.
2. It describes a naive giant lock approach that locks the entire storage for any operation, resulting in poor concurrency and throughput.
3. Better approaches use metadata locks plus simple resource locks, and reference counting locks, to allow concurrent updates to different resources while minimizing critical sections.
This document discusses Ruby's role in data processing. It outlines the typical steps of data processing - collect, summarize, analyze, visualize. It then provides examples of open-source Ruby tools that can be used for each step, such as Fluentd for collection, and libraries for numerical analysis, bioinformatics, and machine learning. Services that use Ruby for collection and processing are also mentioned, like Log Analytics and Stackdriver Logging. The document encourages continuing to improve Ruby tools to make data processing better and celebrates Ruby's 25th anniversary.
Bigdam is a planet-scale data ingestion pipeline designed for large-scale data ingestion. It addresses issues with the traditional pipeline such as imperfectqueue throughput limitations, latency in queries from event collectors, difficulty maintaining event collector code, many small temporary and imported files. The redesigned pipeline includes Bigdam-Gateway for HTTP endpoints, Bigdam-Pool for distributed buffer storage, Bigdam-Scheduler to schedule import tasks, Bigdam-Queue as a high throughput queue, and Bigdam-Import for data conversion and import. Consistency is ensured through at-least-once design and deduplication is performed at the end of the pipeline for simplicity and reliability. Components are designed to scale out horizontally.
Technologies, Data Analytics Service and Enterprise BusinessSATOSHI TAGOMORI
This document discusses technologies for data analytics services for enterprise businesses. It begins by defining enterprise businesses as those "not about IT" and data analytics services as providing insights into business metrics like customer reach, ad views, purchases, and more using data. It then outlines some key technologies needed for such services, including data management systems, distributed processing systems, queues and schedulers, tools for connecting systems, and methods for controlling jobs and workflows with retries to handle failures. Specific challenges around deadlines, idempotent operations, and replay-able workflows are also addressed.
Fluentd is an open source data collector that provides a unified logging layer between data sources and backend systems. It decouples these systems by collecting and processing logs and events in a flexible and scalable way. Fluentd uses plugins and buffers to make data collection reliable even in the case of errors or failures. It can forward data between Fluentd nodes for high availability and load balancing.
Overview of data analytics service: Treasure Data ServiceSATOSHI TAGOMORI
Treasure Data provides a data analytics service with the following key components:
- Data is collected from various sources using Fluentd and loaded into PlazmaDB.
- PlazmaDB is the distributed time-series database that stores metadata and data.
- Jobs like queries, imports, and optimizations are executed on Hadoop and Presto clusters using queues, workers, and a scheduler.
- The console and APIs allow users to access the service and submit jobs for processing and analyzing their data.
The document discusses various techniques used to optimize Hive query execution and deployment in Treasure Data, including:
1) Running Hive queries through a custom QueryRunner that handles query planning, execution, and statistics reporting.
2) Using an in-memory metastore and schema-on-read from Treasure Data's columnar storage to manage schemas and tables.
3) Configuring jobs through HiveConf properties to control mappings, partitions, and storage handlers for efficient execution on Hadoop clusters.
Data Analytics Service Company and Its Ruby UsageSATOSHI TAGOMORI
Treasure Data is a data analytics service company that makes heavy use of Ruby in its platform and services. It uses Ruby for components like Fluentd (log collection), Embulk (data loading), scheduling, and its Rails-based API and console. Java and JRuby are also used for components involving Hadoop and Presto processing. The company's architecture includes collectors that ingest data, a PlazmaDB for storage, workers that process jobs on Hadoop and Presto clusters, and schedulers that queue and schedule those jobs using technologies like PerfectSched and PerfectQueue which are written in Ruby. Hive jobs are built programmatically using Ruby to generate configurations and submit the jobs to underlying Hadoop clusters.
The document discusses ISUCON5 and its benchmark tools. It describes how the benchmark tools were developed with two main requirements - high performance and integrity checks. It then provides details on the key features and implementation of the benchmark tools using Java and the Jetty client library to send requests and check responses in parallel. Distributed benchmarking using multiple nodes is also covered. Performance results from ISUCON5 are shared showing it could handle around 194,000 requests per minute using up to 30 nodes.
Data-Driven Development Era and Its TechnologiesSATOSHI TAGOMORI
This document discusses data-driven development and the technologies used in the data analytics process. It covers topics like data collection, storage, processing, and visualization. The document advocates using managed cloud services for data and analytics to focus on data instead of managing infrastructure. Choosing technologies should be based on the type of data and problems to solve, not the other way around. Services like Google BigQuery, Amazon Redshift, and Treasure Data are recommended for their ease of use.
1. The document discusses the role of an engineer at a tech company, focusing on how engineers can lead both technical and business aspects.
2. It argues that tech-led businesses involve engineers finding customer needs, creating valuable products/services, providing them to customers, collecting feedback, and continuously improving.
3. The most important thing is for engineers to enjoy their work and find meaning in creating things that provide value both for customers and themselves.
In today's dynamic business landscape, ERP software systems are essential tools for businesses worldwide, including those in the UAE. These systems cater to the unique needs of the UAE's rapidly changing economy and expanding industries.
This blog examines the top 10 ERP companies in the UAE, highlighting their innovative products, exceptional customer support, and significant impact on the regional business community. These companies excel in providing ERP solutions that enhance efficiency and growth for businesses throughout the UAE.
1. **Odoo**
- Odoo ERP is a comprehensive business management solution with features like accounting, HR, sales, inventory control, and CRM. Its user-friendly interface simplifies processes and boosts productivity. Banibro IT Solutions leverages Odoo to transform business operations.
- **Details:**
- Suitable for: Small, Medium, Large Businesses
- Open Source: Yes
- Cloud-based: Yes (Cloud and On-premises)
- Support: Phone, Chat, Email
- Payment: Yearly, Monthly
- Multi-Language: Yes
- OS Support: Windows, Mac, iOS, Android
- API: Available
2. **Microsoft Dynamics 365**
- Dynamics 365 offers a centralized platform for small and medium-sized businesses, integrating with Microsoft apps and cloud services for scalability. It simplifies data processing with user-friendly interfaces and customizable reporting.
- **Details:**
- Suitable for: Small, Medium, Large Businesses
- Support: Phone, Chat, Email, Knowledge Base
- Payment: One-Time, Yearly, Monthly
- Multi-Language: No
- OS Support: Web App, Windows, iOS, Android
- API: Not specified
3. **FirstBIT ERP**
- Known for serving small and medium-sized businesses, FirstBIT ERP offers comprehensive solutions and exceptional customer service, enhancing productivity and efficiency.
- **Details:**
- Suitable for: Medium, Large Businesses
- Open Source: Yes/No
- Cloud-based: Yes (Cloud and On-premises)
- Support: Phone, Email, Video Tutorials
- Payment: Yearly, Monthly
- Multi-Language: Yes
- OS Support: Web App, Windows, Mac, iOS, Android
- API: Available
4. **Ezware Technologies**
- Ezware Technologies provides top-notch ERP solutions for various industries with user-friendly modules that streamline complex business processes.
- **Details:**
- Suitable for: Small, Medium, Large Businesses
- Support: Phone, Chat, Email, Knowledge Base
- Payment: One-Time, Yearly, Monthly
- Multi-Language: No
- OS Support: Web App, Windows, Mac, iOS, Android
- API: Not specified
5. **RealSoft**
- RealSoft by Coral is popular in Dubai, offering modules for contracting, real estate, job costing, manufacturing, trading, and finance. It's VAT-enabled and affordable for medium-sized businesses.
- **Details:**
- Suitable for: Small, Medium, Large Businesses
- Open Source: No
- Cloud-based: On-premises
-
Unlocking the Future of Artificial IntelligencedorinIonescu
Unlock the Future: Dive into AI Today! Videnda AI specializes in developing advanced artificial intelligence solutions, including visual dictionaries and language learning tools that leverage immersive virtual travel experiences. Stay Ahead of the Curve: Master AI Now! Our AI technology integrates machine learning and neural networks to enhance education and business applications. AI: The Next Frontier. Are You Ready to Explore? With a focus on real-time AI solutions and deep learning models, Videnda AI provides innovative tools for multilingual communication and immersive learning.
In this course, you'll find a series of engaging videos packed with vibrant animations that break down complex AI concepts into digestible pieces. Our curriculum covers AI models such as Convolutional Neural Networks (CNN), Multi-Layer Perceptrons (MLP), Generative Adversarial Networks (GAN), and Transformers, providing a solid understanding of these models and their real-world applications. We also offer hands-on experience with Generative AI tools like ChatGPT and Midjourney, and Python programming tutorials to help you implement AI algorithms and build your own AI applications.
We are proud participants in the Nvidia Inception Program, driving AI innovation across various industries. By the end of our course, you'll have a strong understanding of AI principles, enhanced Python programming skills, and practical experience with state-of-the-art Generative AI tools. Whether you're looking to kickstart a career in AI or simply curious about this revolutionary technology, Videnda AI is your partner in mastering the future of artificial intelligence.
Waze vs. Google Maps vs. Apple Maps, Who Else.pdfBen Ramedani
Let’s face it, getting lost isn’t really part of the adventure anymore (unless you’re into that sort of thing!). Nowadays, a good navigation app is like your trusty compass, guiding you through busy city streets and winding country roads. But with so many options out there—from big names like Waze, Google Maps, and Apple Maps to some lesser-known contenders—choosing the right one can feel a bit overwhelming.
Think about it: you're about to head out on a road trip, and the last thing you want is to end up in the middle of nowhere because you took a wrong turn. Or maybe you're just trying to navigate your daily commute without hitting every single red light. That's where a solid navigation app comes in handy.
Google Maps is like the old reliable friend who knows every shortcut and scenic route. It's packed with features, from real-time traffic updates to detailed directions, making it a top choice for many. But then there's Waze, the social butterfly of navigation apps. It's all about community, with drivers sharing real-time updates on traffic, accidents, and even speed traps. It’s perfect if you want to feel like you’re part of a huge driving club, all working together to get everyone to their destination faster.
And let’s not forget Apple Maps, which has come a long way since its rocky start. If you're deep into the Apple ecosystem, it's a seamless choice, integrating smoothly with all your devices and offering some pretty neat features like Flyover for 3D city views.
But wait, there are also some underdog apps worth considering! Have you heard of MapQuest? It's still around and offers some great features, especially for planning long trips with multiple stops. Then there's HERE WeGo, which is fantastic for offline navigation—a real lifesaver if you're heading somewhere with spotty cell service.
So, whether you're planning a cross-country adventure or just trying to find the quickest route to work, we’ll help you sift through these options. We’ll dive into what makes each app unique, their pros and cons, and ultimately, guide you to the perfect navigation app for your needs. Buckle up and get ready for a smooth ride!
Tube Magic Software | Youtube Software | Best AI Tool For Growing Youtube Cha...David D. Scott
Tube Magic Software is your ultimate tool for creating stunning video content with ease. Designed with both beginners and professionals in mind, it offers a user-friendly interface packed with powerful features. From seamless editing to eye-catching effects, Tube Magic helps you bring your creative vision to life. Elevate your videos and captivate your audience effortlessly. Join our community of content creators and experience the magic today!
Fix Production Bugs Quickly - The Power of Structured Logging in Ruby on Rail...John Gallagher
Rails apps can be a black box. Have you ever tried to fix a bug where you just can’t understand what’s going on? This talk will give you practical steps to improve the observability of your Rails app, taking the time to understand and fix defects from hours or days to minutes. Rails 8 will bring an exciting new feature: built-in structured logging. This talk will delve into the transformative impact of structured logging on fixing bugs and saving engineers time. Structured logging, as a cornerstone of observability, offers a powerful way to handle logs compared to traditional text-based logs. This session will guide you through the nuances of structured logging in Rails, demonstrating how it can be used to gain better insights into your application’s behavior. This talk will be a practical, technical deep dive into how to make structured logging work with an existing Rails app.
I talk about the Steps to Observable Software - a practical five step process for improving the observability of your Rails app.
The SQDC (Safety, Quality, Delivery, Cost) process enhances manufacturing performance through daily safety meetings, defect tracking, and waste reduction. Orcalean’s FactoryKPI digital dashboard streamlines this process, providing real-time data and AI-powered analytics for continuous improvement.
Unlocking value with event-driven architecture by Confluentconfluent
Sfrutta il potere dello streaming di dati in tempo reale e dei microservizi basati su eventi per il futuro di Sky con Confluent e Kafka®.
In questo tech talk esploreremo le potenzialità di Confluent e Apache Kafka® per rivoluzionare l'architettura aziendale e sbloccare nuove opportunità di business. Ne approfondiremo i concetti chiave, guidandoti nella creazione di applicazioni scalabili, resilienti e fruibili in tempo reale per lo streaming di dati.
Scoprirai come costruire microservizi basati su eventi con Confluent, sfruttando i vantaggi di un'architettura moderna e reattiva.
Il talk presenterà inoltre casi d'uso reali di Confluent e Kafka®, dimostrando come queste tecnologie possano ottimizzare i processi aziendali e generare valore concreto.
Crowd Strike\Windows Update Issue: Overview and Current Statusramaganesan0504
Crowd Strike\Windows Update Issue: Overview and Current Status
Discover the latest on the CrowdStrike Windows update issue, including an overview, current status, and support steps for affected customers. Learn about the identified defect, its impact on Windows hosts, and CrowdStrike's committed actions to ensure ongoing security and stability.
What is CrowdStrike?
CrowdStrike is a prominent cybersecurity technology company that specializes in providing advanced threat intelligence and endpoint protection solutions. Founded in 2011 by George Kurtz, Dmitri Alperovitch, and Gregg Marston, CrowdStrike has quickly established itself as a leader in the cybersecurity industry. Here are some key aspects of
Three available editions of Windows Servers crucial to your organization’s op...Q-Advise
Three available editions of Windows Servers crucial to your organization’s operations
Windows Server, Microsoft’s robust operating system, is the cornerstone of enterprise IT infrastructure, tailored for mission-critical operations. It helps in managing enterprise-level tasks, including data storage, applications, and communication.
Proper licensing of Windows Server is essential for both legal compliance and optimal functionality within business environments.
Windows Server comes in various edition and before any edition is used in your organization, it is required you license them appropriately. The licensing can be complex and capital demanding when you don’t know what you want or understand the licensing requirements.
Even if successfully licensed, there are various activities you can practice as an organization to make sure your Server is operating optimally and there is real value for money. This requires a deeper understanding of best practices and our team of cloud and licensing experts can be of support.
Send the team an email, info@q-advise.com let’s have a look at your needs, together with you decide which licensing model will best work in your case, assist you with savings options and share with you how pre-owned licensing can help you get licensed adequately also.
Get to know Autonomous 2.0, the latest innovation from Applitools, in this sneak peek session showcasing how our AI-powered testing solutions revolutionize how you create, debug, and manage test scripts. See more and sign up for a free trial at https://applitools.info/ml6
7. Logging in Industries
• Service Logs
• Web access logs
• Ad logs
• Commercial transaction logs for analytics (EC, Game, ...)
• System Logs
• Syslog and other OS logs
• Audit logs
• Performance metrics
Logs for
Business Growth
Logs for
Service Stability
9. Microservices and Containers
• Microservices
• Isolated dependencies
• Agile deployment
• Containers
• Isolated environments & resources
• Simple pull&restart deployment
• Less overhead, high density
10. Logging Challenges with Microservices/Containers
• Containerization changes everything:
• No permanent storages
• No fixed physical/network addresses
• No fixed mapping between servers and roles
11. Logging Challenges with Microservices/Containers
• Containerization changes everything:
• No permanent storages
• No fixed physical/network addresses
• No fixed mapping between servers and roles
Transfer Logs to Anywhere ASAP
12. Logging Challenges with Microservices/Containers
• Containerization changes everything:
• No permanent storages
• No fixed physical/network addresses
• No fixed mapping between servers and roles
Push Logs From Containers
13. Logging Challenges with Microservices/Containers
• Containerization changes everything:
• No permanent storages
• No fixed physical/network addresses
• No fixed mapping between servers and roles
Label Logs With Service Names/Tags
14. Logging Challenges with Microservices/Containers
• Containerization changes everything:
• No permanent storages
• No fixed physical/network addresses
• No fixed mapping between servers and roles
Label Logs With Service Names/Tags
Parse Logs & Label Values At Source
Structured Logs
15. Structured Logs: tag, time, key-value pairs
Original log:
the customer put an item to cart: item_id=101, items=10, client=web
Structured log:
ec_service.shopping_cart
2017-03-30 16:35:37 +0100
{
"container_id": "bfdd5b9....",
"container_name": "/infallible_mayer",
"source": "stdout",
"event": "put an item to cart",
"item_id": 101,
"items": 10,
"client": "web"
}
tag
timestamp
record
16. How to Ship Logs from Docker Containers
nginx, mysql, ....
log files
agents
read files,
parse plain texts
apps, middleware
json log files
agents
read files,
parse json lines
applications
agents
just receive
transferred logs
apps, middleware
agents
just receive
transferred logs
Using
mounted volume
Using
container json logs
Sending logs
to agents directly
Using
logging drivers
+ disk I/O penalty
+ mount points
+ disk I/O penalty
+ logger code
+ agent config 😃
19. Distributed Logging Workflow
• Retrieve raw logs: file system / network
• Parse log content
Collector
Aggregator
Destination
• Get data from multiple sources
• Split/merge incoming data into streams
• Retrieve structured logs from Aggregator
• Store formatted logs
20. Scaling Logging
• Network Traffic
• Split heavy log traffic into traffics to nodes
• CPU Load
• Distribute processing to nodes about parsing/formatting logs
• High Availability
• Switch traffic from a node to another for failures
• Agility
• Reconfigure whole logging layer to modify destinations
25. Aggregation Pattern without Source-side Aggregation
• Pros:
• Simple configuration
• Cons:
• Fixed aggregator (destination endpoint) address
configured in containers
• Many network connections
• High load in aggregator / destination
26. Aggregation Pattern with Source-side Aggregation
• Pros:
• Less connections
• Lower load in aggregator / destination
• Less configurations in containers
• More agility
(aggregate containers can be reconfigured)
• Cons:
• Need more resources (+1 container per host)
Aggregate
Container
31. HOW TO SCALE HERE
Source
Transferring
Aggregation
Destination
Now I'm Talking About:
32. Scaling Destination Patterns
Scaling Up
Aggregator/Destination Endpoints
Scaling Out
Aggregator/Destination Endpoints
Destination-side
Aggregator
or
Destination
Load balancer
or
Huge queue
Backend
nodes
Collector
nodes
Using HTTP Load Balancer
or Huge Queues
Using Round Robin Clients
33. • Pros:
• Simple configuration:
specifying load balancer only
in collector nodes
• Cons:
• Upper limits about scaling up
on Load balancer (or queue)
Scaling Up Destination
Backend
nodes
Load balancer
or
Huge queue
34. Scaling Out Destination
• Pros:
• Unlimited scaling by adding nodes
• Cons:
• Complex configuration in collector nodes
• Client feature required for round-robin
• Unavailable for traffic over Internet
35. Destination-side Aggregation and Destination Scaling
Destination Side Aggregation
Scaling Up
Destination
Endpoints
YESNO
Scaling Out
Destination
Endpoints
Early Stage Systems
Collect Logs over
Internet
or
Using Queues
Collect Logs
in Data Center
All Collector Nodes Must Know
All Destination Nodes
↓
Uncontrollable
37. Practices: Docker + Fluentd
• Docker Fluentd Logging Driver
• Docker containers can send these logs to Fluentd directly,
with less overhead
• Fluentd's Pluggable Architecture
• Various destination systems (storage/database/service) are available
by changing configuration
• Small Memory Footprint
• Source aggregation requires +1 container per hosts:
less additional resource usage is fine!
39. Practice 2: Source-side Aggregation + Scaling Up
• Containerized Applications
• w/ Google Stackdriver for Monitoring
• w/ Treasure Data for Analytics
Google Stackdriver Logging
apps, middleware
40. Practice 3: Source/Destination-side Aggregation + Scaling Out
• Containerized Application
• w/ Log processing on Hadoop
• writing files on HDFS via WebHDFS
• Hadoop HDFS prefers large files on HDFS:
• Destination-side aggregation works well
apps, middleware
41. Practice 4: Source/Destination-side Aggregation + Scaling Out
• Containerized Application
• w/ Log processing on Google BigQuery
• putting logs via HTTPS
• BigQuery has quota about write requests:
• Destination-side aggregation works well
apps, middleware
42. Best practices?
• Source aggregation: do it
• it makes app containers free from logging problems (buffering, HA, ...)
• Destination aggregation: it depends
• no need for cloud logging services/storages
• may need for self-hosted distributed filesystems/databases
• may need for cloud services which charges per requests
• Destination scaling: it depends on destinations