Disaster Recovery with MySQL InnoDB ClusterSet - What is it and how do I use it?Miguel Araújo
MySQL InnoDB ClusterSet brings multi-datacenter capabilities to our solutions and make it very easy to setup a disaster recovery architecture. Think multiple MySQL InnoDB Clusters into one single database architecture, fully managed from MySQL Shell and with full MySQL Router integration to make it easy to access the entire architecture.
This presentation covers:
- The various features of InnoDB Clusterset
- How to setup MySQL InnoDB ClusterSet
- Ways to migrate from an existing MySQL InnoDB Cluster into MySQL InnoDB ClusterSet
- How to deal with various failures
- The various features of router integration which makes connection to the database architecture easy.
Presentation at FOSS ASIA 2015 in Singapore, on March 15th 2015. This presentation explains benefit of "SQL and NoSQL" hybrid data management solution MySQL Cluster. MySQL Cluster is designed as high available and high scalable database cluster. In the same time MySQL Cluster can work as ACID compliant transactional Key-Value Datastore aka KVS #FOSSASIA
MySQL Shell - The DevOps Tool for MySQLMiguel Araújo
The document discusses MySQL Shell and MySQL InnoDB Cluster. MySQL Shell is a development and administration tool that supports multiple languages and provides unified interfaces for developers and DBAs. It allows for both relational and document-based operations. MySQL InnoDB Cluster provides a native high availability solution with Group Replication and automatic failover capabilities. It offers an integrated full stack HA solution that is easy to use. Live demos are presented showing MySQL Shell for development and administration of MySQL InnoDB Cluster.
20140722 Taiwan MySQL User Group Meeting Tech UpdatesRyusuke Kajiyama
This document discusses MySQL and its technology updates. It highlights that MySQL 5.7 provides improved performance and scalability over previous versions, with up to a 2x performance gain shown on benchmarks. It also outlines new features for MySQL products like Workbench 6.1 which adds performance dashboards. MySQL Cluster is mentioned as providing shared-nothing active-active clusters with both SQL and NoSQL access for applications.
The document discusses Oracle Cloud and its services, including Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). It describes Oracle's cloud database, Java, developer, and application services which provide fully managed services to developers. It emphasizes that Oracle Cloud can support applications both in cloud or on-premise environments.
WebLogic 12.1.3 was released late last year. It brings a large set of changes including support for some key new Java EE 7 APIs such as WebSocket, JAX-RS 2, JSON-P and JPA 2.1, support for Java SE 8, WebSocket fallback support, support for Server-Sent Events (SSE), improved Maven support, enhanced REST administration support, Oracle Database 12c driver support and much, much more. In this session we will take a detailed tour of these features. In addition we will also cover updated WebLogic support in the Oracle Cloud, the new Oracle public Maven repository, using WebLogic with Arquillian for testing and well as official Docker support for WebLogic.
Towards the end of the session we will discuss what's coming in WebLogic 12.2.1 this year including full support for Java EE 7, multi-tenancy and more.
The document discusses JavaServer Faces (JSF) 2.3 and opportunities for developers to get involved. It covers UI trends like mobile-first design and JavaScript frameworks. For JSF, this means using mobile-optimized component libraries and designing for statelessness. JSF 2.3 in Java EE 8 will focus on specification clarifications, CDI alignment, and small new features. Developers can engage with JSF development by filing issues, joining the expert group, or adopting a JSR specification.
WebSocket in Enterprise Applications 2015Pavel Bucek
Presentation from JavaOne 2015.
This session, which covers use cases of JSR 356 (Java API for WebSocket) and some features of Oracle’s implementation related to enterprise applications, contains description of standard use cases and recommends optimizations and best practices for using the JSR 356 API. After that, it presents more-complex schemes involving authentication support, fallback support, and clustering.
This document discusses Connector/J, the JDBC driver for MySQL. It provides an overview of Connector/J's features and support for various MySQL versions. It also covers how to install Connector/J and configure connections for load balancing, failover, and secure connections using SSL. Tips are provided for performance including enabling streaming result sets and caching server configurations. Resources for learning more about using MySQL with Java are listed at the end.
MySQL Manchester TT - Replication FeaturesMark Swarbrick
The document is a presentation about replication features in MySQL 5.7 given at Percona Live in Amsterdam. It provides background on replication components like binary logs and replicas. It then discusses new usability and online features for replication in MySQL 5.7 like online reconfiguration of global transaction identifiers to allow configuration changes without downtime. The presentation outlines these new features and previews upcoming lab developments and the future roadmap.
The document provides an overview of Java EE 7 APIs supported in WebLogic Server 12c, including JAX-RS 2.0, JSON Processing API 1.0, Java Persistence API 2.1, and Java API for WebSocket 1.0. It discusses how these APIs are enabled and configured in WebLogic 12c. It also describes key features of the APIs such as asynchronous processing in JAX-RS 2.0, schema generation in JPA 2.1, and developing WebSocket applications using the Java API for WebSocket 1.0.
MySQL Group Replication - HandsOn TutorialKenny Gryp
Group Replication is a plugin for MySQL that provides multi-master replication. It works by having each node send write transactions to other nodes through a group communication system. The writes are certified locally in an asynchronous manner to ensure total order of transactions across all nodes. Group Replication uses optimistic locking where local locks are released right after commit, and conflict detection happens during certification rather than at the start of transactions.
JavaOne 2014 BOF4241 What's Next for JSF?Edward Burns
The document discusses recent developments and future plans for JavaServer Faces (JSF). Key points include:
- JSF 2.2 introduced HTML5 friendly markup, flows for modularizing behavior, and resource library contracts for modularizing appearance.
- Emerging trends like mobile-first design, JavaScript frameworks, and statelessness need to be considered for JSF.
- JSF 2.3 planned updates include further CDI alignment, small new features like JSON ajax rendering, and ensuring compatibility with the new MVC specification.
Tweet4Beer - Beertap powered by Java goes IoT and JavaFXBruno Borges
This document discusses a project that uses Java technologies to power a beer tap via Twitter. It involves a JavaFX application that tweets to turn on a relay connected to a beer tap. A REST service in Oracle Java Cloud Service is used to control the relay. The project demonstrates using Java ME, SE Embedded, and Java EE to connect devices to the cloud as part of an Internet of Things application.
MySQL London Tech Tour March 2015 - MySQL FabricMark Swarbrick
This document discusses MySQL Fabric, which is a framework for managing high availability and sharding of MySQL servers. It provides simple and transparent high availability through automatic failover and replication management. It also allows scaling out through sharding where application data is partitioned across multiple MySQL servers. MySQL Fabric handles routing of queries to appropriate servers and coordination between servers to enable features like global updates, shard splitting and moving.
This document provides summaries of updates and new releases for MySQL products between 2010 and 2015. It highlights improvements made under Oracle stewardship, including doubling the engineering staff. New generally available releases include MySQL 5.7, MySQL Cluster 7.4, MySQL Workbench 6.2, and MySQL Enterprise Encryption. Performance gains of up to 47% were achieved in MySQL Cluster 7.4 compared to previous versions.
Group Replication: A Journey to the Group Communication CoreAlfranio Júnior
This document discusses the journey of MySQL Group Replication from using a third-party group communication system (Corosync) to its own built-in group communication engine (XCOM) based on Paxos algorithms. It describes the key components of MySQL Group Replication including the group communication interface, XCOM engine, and optimizations in XCOM like pipelining and batching. It also provides performance results of MySQL Group Replication with multiple writers.
JSONB introduction and comparison with other frameworksDmitry Kornilov
This document discusses JSON Binding (JSON-B), including what it is, the JSR specification, default mappings, and comparisons to other frameworks. JSON-B is a standard for converting Java objects to and from JSON documents. It has a default mapping that handles basic types, dates, classes and collections, but also allows for customized mappings. The specification is hosted on Java.net and has a reference implementation.
MySQL Troubleshooting with the Performance SchemaSveta Smirnova
This document discusses using the Performance Schema in MySQL to troubleshoot performance issues. It provides an overview of the Performance Schema and what information it collects. It then discusses how to use specific Performance Schema tables like events_statements_history_long, events_stages_history_long, and others to identify statements that examine too many rows, issues with index usage, and which internal operations are taking a long time. The document provides examples of queries to run and what to look for in the Performance Schema output to help troubleshoot and optimize SQL statements.
Performance Schema in MySQL (Danil Zburivsky)Ontico
The document discusses the Performance Schema feature in MySQL 5.5, which instruments and collects data about internal operations to help identify performance bottlenecks. It is implemented as a storage engine that collects data about events like query execution steps, locks, I/O, and threads into tables that provide visibility into where the server spends its time. This helps address the lack of good instrumentation previously available in MySQL for performance tuning.
MySQL Performance - SydPHP October 2011Graham Weldon
A talk on optimisations around MySQL on the server side, and through the use of PHP extensions to reduce disk writes to provide for more IO access for MySQL. This was presented at SydPHP in October 2011
MySQL Tech Tour 2015 - Progettare, installare e configurare MySQL ClusterPar-Tec S.p.A.
Il TechAdvisor Mirko Conte spiega come progettare, installare e configurare MySQL Cluster, la versione di punta del database open source più utilizzato al mondo.
Durante la presentazione, Mirko ha condiviso numerose informazioni teoriche e pratiche per comprendere dove, come e quando utilizzare al meglio MySQL Cluster. In questa sessione ha trattato i seguenti punti:
- Valutare MySQL Cluster nel proprio progetto
- Esempi di architettura
- Requisiti hardware/network
- Sessione hands-on
Per saperne di più, scaricate le slide e guardate il video della presentazione del nostro TechAdvisor su http://www.par-tec.it/progettare-installare-gestire-e-ottimizzare-mysql-cluster#progettare
The document provides an overview of new replication features in MySQL 5.7, including:
1. Online reconfiguration of global transaction identifiers and replication filters which allow changing replication settings without restarting servers or interrupting reads/writes.
2. Online reconfiguration of replication receivers and appliers which enables changing the replication topology during failover without stopping applier threads.
3. Improved replication monitoring through new performance schema tables that provide more accurate and extensive monitoring of replication components.
This document provides 10 tips for optimizing MySQL database performance at the operating system level. The tips include using SSDs instead of HDDs for faster I/O, allocating large amounts of memory, avoiding swap space, keeping the MySQL version up to date, using file systems without barriers, configuring RAID cards for write-back caching, and leveraging huge pages. Overall, the tips aim to improve I/O speeds and memory usage to enhance MySQL query processing performance.
Conference slides: MySQL Cluster Performance TuningSeveralnines
This presentation goes through performance tuning basics in MySQL Cluster.
It also covers the new parameters and status variables of MySQL Cluster 7.2 to determine issues with e.g disk data performance and query (join) performance.
Performance Schema for MySQL TroubleshootingSveta Smirnova
The Performance Schema in MySQL provides tables and instruments for troubleshooting issues like locks, I/O bottlenecks, slow queries, memory usage, and replication failures. It contains over 500 instruments in MySQL 5.6 and over 800 in 5.7. The tables provide visibility into the internal workings of MySQL to analyze and optimize performance.
This document summarizes MySQL's monitoring mechanisms and how they have evolved over time. It discusses tools like SHOW statements, INFORMATION_SCHEMA, slow/general query logs, and EXPLAIN that provided limited visibility in past versions. MySQL 5.5 introduced the Performance Schema framework for detailed instrumentation. Subsequent versions have expanded instrumentation to provide more developer-focused statistics on statements, stages, I/O, locks and more. New INFORMATION_SCHEMA tables in 5.6 provide additional InnoDB statistics on data dictionary, buffer pool, transactions and compression. The optimizer trace exposes query transformations. Enhanced EXPLAIN now supports more statement types and future improvements will provide a structured EXPLAIN output.
Priyanka, a MySQL cluster developer, presented MySQL cluster in the MySQL User camp. The slide deck contains an introduction to the cluster module- the architecture,
auto-sharding, failover etc in the cluster module.
Performance Schema for MySQL TroubleshootingSveta Smirnova
Percona Live (https://www.percona.com/live/data-performance-conference-2016/sessions/performance-schema-mysql-troubleshooting)
The performance schema in MySQL version 5.6, released in February, 2013, is a very powerful tool that can help DBAs discover why even the trickiest performance issues occur. Version 5.7 introduces even more instruments and tables. And while all these give you great power, you can get stuck choosing which instrument to use.
In this session, I will start with a description of a typical problem, then guide you how to use the performance schema to find out what causes the issue, the reason for unwanted behavior and how the received information can help you solve a particular problem.
Traditionally, performance schema sessions teach what is in contained in tables. I will, in contrast, start from a performance issue, then demonstrate which instruments and tables can help solve it. We will discuss how to setup the performance schema so that it has minimal impact on your server.
This document discusses the Performance Schema in MySQL, which records instrumentation data to help profile and monitor database activity. It provides an overview of the Performance Schema's components and tables, how it has evolved between MySQL versions to include more metrics and functionality, and examples of how to query the tables to analyze wait events, statements, stages and other performance data.
This document provides an overview of MySQL high availability solutions including InnoDB Cluster and NDB Cluster. InnoDB Cluster allows setting up a highly available MySQL cluster with auto-sharding using Group Replication and MySQL Router for transparent application routing. NDB Cluster is a memory-optimized database for low-latency applications requiring high scalability and availability. MySQL Shell provides a unified interface for deploying, managing and monitoring these MySQL HA solutions.
MySQL 5.7 New Features for Developers session for DOAG (Oracle user group conference) in 2016. A similar version was also presented in Israel MySQL User Group on November 2016.
This presentation review new features in MySQL 5.7: Optimizer, InnoDB engine, JSON native data type, performance and sys schemas
Lecture 7 regionalism in india and search for indiannessNipesh P Narayanan
The document discusses the history of regionalism and search for Indian identity in architecture. It covers key topics such as the emergence of regionalism in the mid-20th century focused on vernacular architecture. Important figures who promoted regionalism including Bernard Rudofsky, Paul Oliver and Hassan Fathy are highlighted. The document also examines how regionalism emerged in India through figures like Mahatma Gandhi and the need to define an architecture for post-independence India drawing from local culture and traditions. Key Indian architects discussed include Laurie Baker, Raj Rewal, Charles Correa and their approach towards regional modernism.
This document provides an overview of MySQL query optimization. It discusses MySQL features like storage engines, InnoDB, and indexing. It explains that query optimization is important for performance as data grows. Techniques like explaining query plans, indexing, and rewriting queries to make better use of indexes can improve query performance by 10-100 times. The document includes examples of indexing, query rewriting, and using EXPLAIN plans.
FOSDEM 2015 - NoSQL and SQL the best of both worldsAndrew Morgan
This document discusses the benefits and limitations of both SQL and NoSQL databases. It argues that while NoSQL databases provide benefits like simple data formats and scalability, relying solely on them can result in data duplication and inconsistent data when relationships are not properly modeled. The document suggests that MySQL Cluster provides a hybrid approach, allowing both SQL queries and NoSQL interfaces while ensuring ACID compliance and referential integrity through its transactional capabilities and handling of foreign keys.
MySQL 5.7 proposes several changes to improve performance and consistency including:
1. Making replication durable by default by setting sync_binlog and repository options.
2. Deprecating features like INNODB monitor tables and ALTER IGNORE TABLE in favor of newer standards.
3. Simplifying and restricting SQL modes to encourage stricter querying and remove ambiguous options. Explanations for errors and modes will also be improved.
Venkatesh Duggirala from the MySQL Replication Team gave a presentation on Multi Source Replication. The presentation covered the background on why replication is used, an introduction to multi-source replication including how a slave can have multiple masters, use cases like data aggregation, and technical details on how channels and slave appliers work in multi-source replication. Monitoring of multi-source replication was also discussed.
MySQL Group Replication @osi days 2014Manish Kumar
MySQL Group Replication allows multiple MySQL servers to act as a single logical master by replicating transactions between them in parallel. It provides multi-master replication with automatic conflict detection and resolution. When a new server joins the replication group, it synchronizes by retrieving missing transactions from another member before participating.
MySQL High Availability with Replication New FeaturesShivji Kumar Jha
The session was presented at open source India 2014 (http://osidays.com/osidays/) by Shivji (me) and Manish Kumar. It talks of the new features in MySQL-5.7 Replication. It covered work on
1) performance enhancements in MySQL Replication
2) Usability improvements
3) More flexibility to provide more options to our users so
they can chose what is best for their application.
4) Semisynchronous and MySQL Group Replication
At then end, there are a lot of links to the blogs written on these features by the MySQL Replication engineers.
Introduction to MySQL Enterprise MonitorMark Leith
The document is a presentation on MySQL Enterprise Monitor (MEM) by Mark Leith of Oracle. It introduces MEM as a distributed monitoring system for MySQL with a central Service Manager and agents installed on monitored hosts. The presentation includes sections on MEM architecture showing its core components, and a demo of features in the MEM UI like viewing instances, advisors, events, graphs, and query analysis.
NoSQL & SQL - Best of both worlds - BarCamp Berkshire 2013Andrew Morgan
The document discusses blending NoSQL and SQL databases by leveraging the strengths of both. It describes how MySQL Cluster provides massively scalable performance through its NoSQL-style data storage and replication abilities, while also supporting SQL queries, joins, and ACID transactions like a traditional relational database. This allows applications to use NoSQL for simple operations and scalability while still using SQL for complex queries and transactions as needed.
This document discusses MySQL Fabric, which is a framework for managing a farm of MySQL servers to provide high availability and sharding capabilities. It describes how MySQL Fabric allows for easy management of MySQL servers, including load balancing, read/write splitting, distributed transactions, global updates, and sharding of tables. It also covers how application connectors can be made aware of MySQL Fabric to properly route queries and transactions to the backend MySQL servers.
MySQL High Availability: Managing Farms of Distributed Servers (MySQL Fabric)Alfranio Júnior
This document provides an overview and introduction to MySQL Fabric, a new high availability and distributed database solution from Oracle. The summary includes:
- MySQL Fabric is a distributed framework that allows farms of MySQL servers to be managed as highly available groups. It uses extensions and connectors to provide fault tolerance.
- Failure detection and failover works by having MySQL Fabric monitor the servers in an availability group. If the master fails, it will trigger a failover to promote a slave to become the new master.
- MySQL Fabric-aware connectors are available for Python, Java, and PHP that can route transactions, cache information, and handle failures by retrying operations on a different server if needed.
MySQL 5.7 New Features to Exploit -- PHPTek/Chicago MySQL User Group May 2014Dave Stokes
MySQL 5.7 is on the way and this presentation outlines the changes and how to best take advantage of them. Presentations May 2014 to PHPTek and Chicago MySQL User Group.
Programming-best practices( beginner) ADF_fusionappsBerry Clemens
This document provides an overview and best practices for using Oracle Application Development Framework (ADF). The presentation covers simplifying development choices, ADF Business Components, ADF Task Flow, ADF Binding, ADF Faces, and JavaScript. It discusses architectural best practices such as using ADF Business Component preferences and exception handling. Implementation best practices covered include using managed beans, data control scope, and a custom ADF model error handler.
The document provides an introduction to AngularJS, including:
- What AngularJS is and the challenges it addresses like rich user interfaces, forms, and testability
- Key terminology like modules, directives, views, controllers, and scopes
- Examples of basic AngularJS apps demonstrating data binding, routing, forms, REST communication, and directives
- How to test AngularJS apps using unit tests and end-to-end tests
- Services provided by AngularJS for tasks like accessing the browser and mocking HTTP requests
Scaling MySQl 1 to N Servers -- Los Angelese MySQL User Group Feb 2014Dave Stokes
The document discusses various options for scaling MySQL databases to handle increasing load. It begins with simple options like upgrading MySQL versions, adding caching layers, and read/write splitting. More complex and reliable options include using MySQL replication, cloud hosting, MySQL Cluster, and columnar storage engines. Scaling to very large "big data" workloads may involve using NoSQL technologies, Hadoop, and data partitioning/sharding. The key challenges discussed are defining business and technical requirements, planning for high availability, and managing increased complexity.
MySQL InnoDB Cluster and Group Replication - OSI 2017 BangaloreSujatha Sivakumar
The document discusses MySQL InnoDB Cluster and Group Replication. It provides an introduction and overview of InnoDB Cluster, outlining the key features and how to get an InnoDB Cluster up and running in 3 steps: deploying instances, creating a cluster, and adding more instances. It also covers setting up and starting a router. For Group Replication, it discusses the concept of replicating writes across multiple servers for high availability and read scaling. It shows how Group Replication achieves consensus on membership, message delivery and state updates across the group.
MySQL Fabric is an extensible framework for managing high availability and sharding across a farm of MySQL servers. It allows creating high availability groups, adding MySQL servers to those groups to manage redundancy and load balancing. The framework includes connectors, a central node to manage the farm, and extensions for high availability and sharding functionality.
Sharding and Scale-out using MySQL FabricMats Kindahl
MySQL Fabric is an open-source solution released by the MySQL Engineering team at Oracle. It make management of farms of MySQL servers easy and available for both applications with small and large number of servers.
This is the presentation from Percona Live Santa Clara.
1) The document discusses an Oracle MySQL OEM Tech Tour that took place in Lima, Peru in November 2013. It includes contact information for various MySQL team members from Oracle.
2) It then lists the top 10 reasons for using MySQL and discusses MySQL editions, trends, large data volumes, and editions of MySQL between Enterprise and Open Source.
3) Finally, it discusses how to become an ISV partner and thanks attendees. The document appears to be from a presentation promoting Oracle's MySQL database products and services.
Solution Use Case Demo: The Power of Relationships in Your Big DataInfiniteGraph
In this security solution demo, we have integrated Oracle NoSQL DB with InfiniteGraph to demonstrate the power of using the right tools for the solution. By integrating the key value technology of Oracle with the InfiniteGraph distributed graph database, we are able to create new views of existing Call Detail Record (CDR) details to enable discovery of connections, paths and behaviors that may otherwise be missed.
Discover how to add value to your existing Big Data to increase revenues and performance!
Keynote (Nandini Ramani) - The Role of Java in Heterogeneous Computing & How ...AMD Developer Central
Keynote presentation, The Role of Java in Heterogeneous Computing, and How You Can Help, by Nandini Ramani, VP, Java Platform, Oracle Corporation, at the AMD Developer Summit (APU13), Nov. 11-13, 2013.
206590 mobilizing your primavera workforcep6academy
The document discusses Oracle's ADF Mobile framework. It provides an overview of what ADF Mobile is, including that it is a hybrid mobile application framework that allows developing once and deploying apps across multiple platforms. It also reviews what capabilities ADF Mobile provides, such as building user interfaces with drag and drop, accessing native device features, and testing interfaces on different devices. The document concludes by pointing to additional resources for using ADF Mobile.
The document discusses Oracle Database Cloud services and whether they are suitable for different users' needs. It describes the differences between public and private database clouds offered by Oracle. The public database cloud provides instant provisioning but has limitations around storage capacity and customization options. The private database cloud allows more control and customization through options like RAC and pluggable databases but requires managing infrastructure.
Similar to Open source India - MySQL Labs: Multi-Source Replication (20)
Batch to near-realtime: inspired by a real production incidentShivji Kumar Jha
This slide deck was used for the platformatory streams meetup in Bengaluru on July 7, 2024.
This is a real world account from an Apache Druid cluster in production. A story of 48 hours of debugging, learning and understanding batch vs stream better, filing a couple of issues in Druid open source projects and finally a stable production pipeline again thanks to the Druid community. We will discuss what parts of your design could be impacted, how you should change the related systems so the cascading failures don’t bring down your complete production availability. As an example, we will discuss the bottlenecks we had in overlord, slot issues for Peons in middle managers, coordinator bottlenecks, how to mitigated task and segment flooding, what configs we changed sprinkled with real world numbers and snapshots from our Grafana dashboards.
Finally we will list all the leanings and how we made sure we never repeat the same mistakes in production systems.
Navigating Transactions: ACID Complexity in Modern DatabasesShivji Kumar Jha
Transactions are anything but straightforward, with each database vendor offering its unique interpretation of the term. By scrutinising the internal architectures of these databases, engineers can gain valuable insights, enabling them to write more stable applications.This talk explores the intricacies of transactions, focusing on modern databases. Delving into distributed transactions, we discuss network challenges and cloud deployments in the contemporary era. The session provides a concise examination of the internal architectures of cloud-scale, multi-tenant databases such as Spanner, DynamoDB, and Amazon Aurora.
Druid Summit 2023 : Changing Druid Ingestion from 3 hours to 5 minutesShivji Kumar Jha
This is a real world account from a Druid cluster in production. A story of 48 hours of debugging, learning and understanding Druid better, filing a couple of issues in Druid github and finally a stable production pipeline again thanks to the Druid community.
We will discuss the bottlenecks we had in overlord, slot issues for Peons in middle managers, coordinator bottlenecks, how we mitigated task and segment flooding, what configs we changed sprinkled with real world numbers and snapshots from our grafana dashboards.
In this slide deck, we go exploring the database landscape today and the common lego blocks that are used to build these different falvours of databses. We will dive through internals of a database, explore some choices and towards the end also explore some real world database architectures in view of the concepts (legos) we explored earlier.
This document provides an overview of Apache Pulsar:
- It introduces Apache Pulsar and shares some stats on its adoption and contributors.
- It describes Pulsar's architecture including brokers, Zookeeper, BookKeeper, topics, and subscribers.
- It explains how Pulsar stores data across tenants, namespaces, bundles, ledgers and topics to enable features like multi-tenancy, load balancing, and geo-replication.
Pulsar Summit Asia 2022 - Streaming wars and How Apache Pulsar is acing the b...Shivji Kumar Jha
This presentation will cover why we prefer Apache Pulsar over other streaming solutions. Given the streaming requirements of near-realtime action, scalability, high availability, disaster recovery, load balancing, low cost of operations, multi-tenancy and flexibility to fit a variety of use cases, we have run kafka, kinesis and NATS Jetstream across different use cases. And we chose Apache Pulsar as our platform of choice for cloud-native messaging.
This talk presents the operational challenges we have faced running Pulsar for over 4 years and how Pulsar fit into different use cases given its multi-tenancy and configurability. We will also talk about how we have aced these challenges to stick to pulsar and even moved application from other messaging solutions to Pulsar. We will end with the challenges and learnings on moving to Pulsar from Kafka and Kinesis.
After this session, you will learn more on common messaging requirements, why you should also choose Apache Pulsar as your platform of choice and how you can safely transition to Pulsar if you have been running other messaging solutions.
Pulsar Summit Asia 2022 - Keeping on top of hybrid cloud usage with PulsarShivji Kumar Jha
This presentation will cover how we force controls on an application over a hybrid cloud infrastructure built from a combination of different clouds that could include private and public clouds. For instance, you could deploy your microservice in AWS but use BigTable as your data store.
Every cloud or on-premise infrastructure provider provides monitoring, alerting, metering, audit trail etc. In a hybrid cloud use case, the IT team needs a single view of the usage across the cloud providers. Such a platform needs to combine the data sourcing of these utilities from different infrastructure providers, parse them into a common format and build an integrated data sink. Adding to it the challenge of each data source evolving its data formats, volume, velocity, throughput, latency etc. You have a challenging task to understand data from varied sources and present it in one view.
We will present an architecture that has been battle-tested in production for over five years. The components include Pulsar, Flink, PostgreSQL, Redis, Neo4J DB, rule/ML engine etc., to name a few technologies.
After this presentation, you will learn more about
1. Combining infrastructure from multiple clouds and on-premise providers to build your application.
2. Appreciate the need for lambda architecture.
3. How to stream ever-evolving multi-schema data using pulsar
4. How to write custom rules over a stream analytics framework to make your application.
Pulsar summit asia 2021: Designing Pulsar for IsolationShivji Kumar Jha
This document discusses isolation in Apache Pulsar. It introduces the presenters as experts in distributed systems and the Pulsar open source project. It then outlines ways to isolate resources in Pulsar like brokers, bookies, and clusters to separate namespaces and tenants. The key methods covered are namespace isolation policies, failure domains, anti-affinity groups, and bookie affinity groups. It provides examples of how these are configured and allows scaling resources up and down independently per namespace. Finally, it invites questions and provides contact details.
Event sourcing Live 2021: Streaming App Changes to Event StoreShivji Kumar Jha
This document discusses streaming app changes to event stores. It covers change data capture (CDC) which involves identifying, capturing, and delivering changes made to data. CDC can be done by capturing events from app code or by tailing database transaction logs. Capturing from app code provides flexibility but requires extra code, while capturing from databases is easier to control but depends on database log formats. The document also discusses using event stores for data warehouses, data lakes, CQRS patterns, and hybrid transactional/analytical processing (HTAP) databases.
Type safety is extremely important in any application built around a stream / queue. Type definition and evolution can either be built in the application or relied upon the data layer to support it out of the box allowing the application to only concentrate on business logic, not how of data store and evolution. It is this property of the good old relational databases (among others) that make them a favourite among all the modern NoSQL databases. Modern software architectures requires asynchronous communication (via stream / queue). While the data store and query design changes with asynchronous communication, type safety is still equally important.
In this slide deck, used for Apache Con 2021 talk, we will go over ways in which one can force structure (schema) over the streaming data. As an example, we will talk about Apache Pulsar. Apache pulsar offers server as well as client side support for the structured streaming. We have been using pulsar for asynchronous communication among microservices in our nutanix beam and flow security central apps for over 1.5 years in production. This deck presents the technical details on what is schema, how to represent schema, what is available in the apache pulsar server and client side, how we have used pulsar’s schema support to build our use cases and our learnings from them.
Apache Con 2021 : Apache Bookkeeper Key Value Store and use casesShivji Kumar Jha
In order to leverage the best performance characters of your data or stream backend, it is important to understand the nitty gritty details of how your backend store and compute works, how data is stored, how is it indexed and how the read path is. Understanding this empowers you to design your use case solutioning so as to make the best use of resources at hand as well as get the optimum amount of consistency, availability, latency and throughput for a given amount of resources at hand.
With this underlying philosophy, in this slide deck, we will get to the bottom of storage tier of pulsar (apache bookkeeper), the barebones of the bookkeeper storage semantics, how it is used in different use cases ( even other than pulsar), understand the object models of storage in pulsar, different kinds of data structures and algorithms pulsar uses therein and how that maps to the semantics of the storage class shipped with pulsar by default. Oh yes, you can change the storage backend too with some additional code!
The focus will be more on storage backend so as to not keep this tailored to pulsar specifically but to be able to apply it different data stores or streams.
How pulsar stores data at Pulsar-na-summit-2021.pptx (1)Shivji Kumar Jha
In order to leverage the best performance characters of your stream backend, it is important to understand the nitty gritty details of how pulsar stores your data. Understanding this empowers you to design your use case solutioning so as to make the best use of resources at hand as well as get the optimum amount of consistency, availability, latency and throughput for a given amount of resources at hand.
With this underlying philosophy, in this talk, we will get to the bottom of storage tier of pulsar (apache bookkeeper), the barebones of the bookkeeper storage semantics, how it is used in different use cases ( even other than pulsar), understand the object models of storage in pulsar, different kinds of data structures and algorithms pulsar uses therein and how that maps to the semantics of the storage class shipped with pulsar by default. Oh yes, you can change the storage backend too with some additional code!
This session will empower you with the right background to map your data right with pulsar.
Pulsar Summit Asia - Structured Data Stream with Apache PulsarShivji Kumar Jha
This document discusses Apache Pulsar schemas. It begins with background on Pulsar, serialization, and schema evolution. It then discusses the benefits of using schemas with Pulsar, including different schema types like primitive, JSON, and Avro schemas. It describes how Pulsar uses a schema registry to store schemas on the server side rather than client side. Key learnings are to use structured schemas like Avro to model domain objects, consider compatibility and ordering when designing topics, and manage schemas through a code review process. The document provides references for further reading on Pulsar schemas and schema evolution.
Pulsar Summit Asia - Running a secure pulsar clusterShivji Kumar Jha
This document provides an overview of securing Apache Pulsar. It discusses securing the different cluster components like Zookeeper, Bookkeeper and brokers. It describes how to enable TLS for securing communication between these components. It also covers setting up TLS, keystores and truststores for brokers and clients. The document references Pulsar and Zookeeper documentation for more details on configuring security.
Having used apache pulsar in production for an year for our pub sub use cases such as stream analytics, event sourcing etc, this slide deck presents the lesson learned per se understanding the architecture, tuning the cluster, managing to keep it highly available and fault tolerant and much more.
While the slides are presented in terms of apache pulsar, a lot of the concepts can be easily extended to a lot of distributed systems.
The views here are my own and do not represent the view of nutanix corporation.
Retrieval Augmented Generation Evaluation with RagasZilliz
Retrieval Augmented Generation (RAG) enhances chatbots by incorporating custom data in the prompt. Using large language models (LLMs) as judge has gained prominence in modern RAG systems. This talk will demo Ragas, an open-source automation tool for RAG evaluations. Christy will talk about and demo evaluating a RAG pipeline using Milvus and RAG metrics like context F1-score and answer correctness.
It's your unstructured data: How to get your GenAI app to production (and spe...Zilliz
So you've successfully built a GenAI app POC for your company -- now comes the hard part: bringing it to production. Aparavi addresses the challenges of AI projects while addressing data privacy and PII. Our Service for RAG helps AI developers and data scientists to scale their app to 1000s to millions of users using corporate unstructured data. Aparavi’s AI Data Loader cleans, prepares and then loads only the relevant unstructured data for each AI project/app, enabling you to operationalize the creation of GenAI apps easily and accurately while giving you the time to focus on what you really want to do - building a great AI application with useful and relevant context. All within your environment and never having to share private corporate data with anyone - not even Aparavi.
Demystifying Neural Networks And Building Cybersecurity ApplicationsPriyanka Aash
In today's rapidly evolving technological landscape, Artificial Neural Networks (ANNs) have emerged as a cornerstone of artificial intelligence, revolutionizing various fields including cybersecurity. Inspired by the intricacies of the human brain, ANNs have a rich history and a complex structure that enables them to learn and make decisions. This blog aims to unravel the mysteries of neural networks, explore their mathematical foundations, and demonstrate their practical applications, particularly in building robust malware detection systems using Convolutional Neural Networks (CNNs).
TrustArc Webinar - Innovating with TRUSTe Responsible AI CertificationTrustArc
In a landmark year marked by significant AI advancements, it’s vital to prioritize transparency, accountability, and respect for privacy rights with your AI innovation.
Learn how to navigate the shifting AI landscape with our innovative solution TRUSTe Responsible AI Certification, the first AI certification designed for data protection and privacy. Crafted by a team with 10,000+ privacy certifications issued, this framework integrated industry standards and laws for responsible AI governance.
This webinar will review:
- How compliance can play a role in the development and deployment of AI systems
- How to model trust and transparency across products and services
- How to save time and work smarter in understanding regulatory obligations, including AI
- How to operationalize and deploy AI governance best practices in your organization
Choosing the Best Outlook OST to PST Converter: Key Features and Considerationswebbyacad software
When looking for a good software utility to convert Outlook OST files to PST format, it is important to find one that is easy to use and has useful features. WebbyAcad OST to PST Converter Tool is a great choice because it is simple to use for anyone, whether you are tech-savvy or not. It can smoothly change your files to PST while keeping all your data safe and secure. Plus, it can handle large amounts of data and convert multiple files at once, which can save you a lot of time. It even comes with 24*7 technical support assistance and a free trial, so you can try it out before making a decision. Whether you need to recover, move, or back up your data, Webbyacad OST to PST Converter is a reliable option that gives you all the support you need to manage your Outlook data effectively.
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...Zilliz
Enterprises have traditionally prioritized data quantity, assuming more is better for AI performance. However, a new reality is setting in: high-quality data, not just volume, is the key. This shift exposes a critical gap – many organizations struggle to understand their existing data and lack effective curation strategies and tools. This talk dives into these data challenges and explores the methods of automating data curation.
UiPath Community Day Amsterdam: Code, Collaborate, ConnectUiPathCommunity
Welcome to our third live UiPath Community Day Amsterdam! Come join us for a half-day of networking and UiPath Platform deep-dives, for devs and non-devs alike, in the middle of summer ☀.
📕 Agenda:
12:30 Welcome Coffee/Light Lunch ☕
13:00 Event opening speech
Ebert Knol, Managing Partner, Tacstone Technology
Jonathan Smith, UiPath MVP, RPA Lead, Ciphix
Cristina Vidu, Senior Marketing Manager, UiPath Community EMEA
Dion Mes, Principal Sales Engineer, UiPath
13:15 ASML: RPA as Tactical Automation
Tactical robotic process automation for solving short-term challenges, while establishing standard and re-usable interfaces that fit IT's long-term goals and objectives.
Yannic Suurmeijer, System Architect, ASML
13:30 PostNL: an insight into RPA at PostNL
Showcasing the solutions our automations have provided, the challenges we’ve faced, and the best practices we’ve developed to support our logistics operations.
Leonard Renne, RPA Developer, PostNL
13:45 Break (30')
14:15 Breakout Sessions: Round 1
Modern Document Understanding in the cloud platform: AI-driven UiPath Document Understanding
Mike Bos, Senior Automation Developer, Tacstone Technology
Process Orchestration: scale up and have your Robots work in harmony
Jon Smith, UiPath MVP, RPA Lead, Ciphix
UiPath Integration Service: connect applications, leverage prebuilt connectors, and set up customer connectors
Johans Brink, CTO, MvR digital workforce
15:00 Breakout Sessions: Round 2
Automation, and GenAI: practical use cases for value generation
Thomas Janssen, UiPath MVP, Senior Automation Developer, Automation Heroes
Human in the Loop/Action Center
Dion Mes, Principal Sales Engineer @UiPath
Improving development with coded workflows
Idris Janszen, Technical Consultant, Ilionx
15:45 End remarks
16:00 Community fun games, sharing knowledge, drinks, and bites 🍻
Discovery Series - Zero to Hero - Task Mining Session 1DianaGray10
This session is focused on providing you with an introduction to task mining. We will go over different types of task mining and provide you with a real-world demo on each type of task mining in detail.
How UiPath Discovery Suite supports identification of Agentic Process Automat...DianaGray10
📚 Understand the basics of the newly persona-based LLM-powered Agentic Process Automation and discover how existing UiPath Discovery Suite products like Communication Mining, Process Mining, and Task Mining can be leveraged to identify APA candidates.
Topics Covered:
💡 Idea Behind APA: Explore the innovative concept of Agentic Process Automation and its significance in modern workflows.
🔄 How APA is Different from RPA: Learn the key differences between Agentic Process Automation and Robotic Process Automation.
🚀 Discover the Advantages of APA: Uncover the unique benefits of implementing APA in your organization.
🔍 Identifying APA Candidates with UiPath Discovery Products: See how UiPath's Communication Mining, Process Mining, and Task Mining tools can help pinpoint potential APA candidates.
🔮 Discussion on Expected Future Impacts: Engage in a discussion on the potential future impacts of APA on various industries and business processes.
Enhance your knowledge on the forefront of automation technology and stay ahead with Agentic Process Automation. 🧠💼✨
Speakers:
Arun Kumar Asokan, Delivery Director (US) @ qBotica and UiPath MVP
Naveen Chatlapalli, Solution Architect @ Ashling Partners and UiPath MVP
Finetuning GenAI For Hacking and DefendingPriyanka Aash
Generative AI, particularly through the lens of large language models (LLMs), represents a transformative leap in artificial intelligence. With advancements that have fundamentally altered our approach to AI, understanding and leveraging these technologies is crucial for innovators and practitioners alike. This comprehensive exploration delves into the intricacies of GenAI, from its foundational principles and historical evolution to its practical applications in security and beyond.
"Building Future-Ready Apps with .NET 8 and Azure Serverless Ecosystem", Stan...Fwdays
.NET 8 brought a lot of improvements for developers and maturity to the Azure serverless container ecosystem. So, this talk will cover these changes and explain how you can apply them to your projects. Another reason for this talk is the re-invention of Serverless from a DevOps perspective as a Platform Engineering trend with Backstage and the recent Radius project from Microsoft. So now is the perfect time to look at developer productivity tooling and serverless apps from Microsoft's perspective.
Generative AI technology is a fascinating field that focuses on creating comp...Nohoax Kanont
Generative AI technology is a fascinating field that focuses on creating computer models capable of generating new, original content. It leverages the power of large language models, neural networks, and machine learning to produce content that can mimic human creativity. This technology has seen a surge in innovation and adoption since the introduction of ChatGPT in 2022, leading to significant productivity benefits across various industries. With its ability to generate text, images, video, and audio, generative AI is transforming how we interact with technology and the types of tasks that can be automated.