Apache Ignite is a high-performance, integrated and distributed in-memory platform for computing and transacting on large-scale data sets in real-time, orders of magnitude faster than possible with traditional disk-based or flash technologies.
Despite the increase of deep learning practitioners and researchers, many of them do not use GPUs, this may lead to long training/evaluation cycles and non-practical research.
In his talk, Lior shares how to get started with GPUs and some of the best practices that helped him during research and work. The talk is for everyone who works with machine learning (deep learning experience is NOT mandatory!), It covers the very basics of how GPU works, CUDA drivers, IDE configuration, training, inference, and multi-GPU training.
How to win a machine learning competition pavel pleskovDataFest Tbilisi
This document provides tips for winning machine learning competitions on Kaggle from a Kaggle Grandmaster. It discusses choosing the right competition based on factors like dataset size and number of participants. It also offers strategies like using specialized machine learning software and hardware, collaborating on teams, leveraging data leakages, and ensemble methods like stacking. The document emphasizes the benefits of competitions for rapidly advancing skills and building experience and portfolios, as well as some of the cons like the significant time commitment required.
How we broke Apache Ignite by adding persistence, by Stephen Darlington (Grid...Altinity Ltd
Apache Ignite is an in-memory computing platform that combines fast in-memory performance with disk durability. The developers wanted to add persistence to Ignite to store more data than memory and enable crash recovery. They started with an ARIES architecture using page-based write-ahead logging to store everything off-heap. This worked initially but performance degraded with disk I/O. To maintain predictable speeds, they throttled load based on dirty page production and disk write rates. They also avoided doubling memory usage with the OS page cache by using direct I/O.
Continuously improving factory operations is of critical importance to manufacturers. Consider the facts: the total cost of poor quality amounts to a staggering 20% of sales (American Society of Quality) and unplanned downtime costs plants approximately $50 billion per year (Deloitte).
The most pressing questions are: which process variables effect quality and yield and which process variables predict equipment failure? Getting to those answers is providing forward thinking manufacturers a leg up over competitors.
The speakers address the data management challenges facing today's manufacturers, including proprietary systems and silo'ed data sources, as well as an inability to make sensor-based data usable.
Integrating enterprise data from ERP, MES, maintenance systems and other sources with real time operations data from sensors, PLCs, SCADA systems and historians represents a major first step. But how to get started? What is the value of a data lake? How are AI/ML being applied to enable real time action?
Join us for this educational session, which includes a rare view from one of our SWAT team experts into our roadmap for an open source industrial IoT data management platform.
Key Takeaways:
• How to choose an initial project from which to quickly demonstrate high value returns
• Understand the value of multivariate data sources, as opposed to a single sensor on a piece of equipment
• Understand advances in big data management and streaming analytics that are paving the way to next-generation factory performance
MICHAEL GER, General Manager, Manufacturing and Automotive, Hortonworks and RYAN TEMPLETON, Senior Solutions Engineer, Hortonworks
Case Study: Stream Processing on AWS using Kappa ArchitectureJoey Bolduc-Gilbert
In the summer of 2016, XpertSea decided to migrate its operations to AWS and to build a data processing system that is able to scale to the extent of our ambitions. Come see how we built our platform inspired by Kappa Architecture, able to support connected devices located all-around the globe and state-of-the-art machine learning algorithms.
Modernizing the Analytics and Data Science Lifecycle for the Scalable Enterpr...Data Con LA
Data Con LA 2020
Description
It’s no secret that the roots of Data Science date back to the 1960’s and were first mainstreamed in the 1990’s with the emergence of Data Mining. This occurred when commercially affordable computers started offering the horsepower and storage necessary to perform advanced statistics to scale.
However, the words “to scale” have evolved over time. The leap to “Big Data” is only one serial aspect of growth. Beyond the typical 1-off studies that catalyzed the field of Data Mining, Data Science now fulfills enterprise and multi-enterprise use cases spanning much broader and deeper data sets and integrations. For example, AI and Machine Learning frameworks can interoperate with a variety of other systems to drive alerting, feedback loops, predictive frameworks, prescriptive engines, continual learning, and more. The deployment of AI/ML processes themselves often involves integration with contemporary DevOps tools.
Now segue to SEAL – the Scalable Enterprise Analytic Lifecycle. In this presentation, you’ll learn how to cover the major bases of a modern Data Science projects – and Citizen Data Science as well – from conception, learning, and evaluation through integration, implementation, monitoring, and continual improvement. And as the name implies, your deployments will be performant and scale as expected in today’s environments.
Speaker
Jeff Bertman, CTO, Dfuse Technologies
Real time analytics at uber @ strata data 2019Zhenxiao Luo
This document summarizes Uber's use of Presto, an open source distributed SQL query engine, for real-time analytics and business intelligence. Presto allows Uber to query petabytes of data across different data sources like HDFS, Elasticsearch, Pinot and databases in seconds. Uber has optimized Presto for its scale with contributions like geospatial support, security features and connectors. Presto is critical for Uber's data scientists, analysts and operations to power applications, machine learning and business decisions.
Organizations need to perform increasingly complex analysis on data — streaming analytics, ad-hoc querying, and predictive analytics — in order to get better customer insights and actionable business intelligence. Apache Spark has recently emerged as the framework of choice to address many of these challenges. In this session, we show you how to use Apache Spark on AWS to implement and scale common big data use cases such as real-time data processing, interactive data science, predictive analytics, and more. We will talk about common architectures, best practices to quickly create Spark clusters using Amazon EMR, and ways to integrate Spark with other big data services in AWS.
Learning Objectives:
• Learn why Spark is great for ad-hoc interactive analysis and real-time stream processing.
• How to deploy and tune scalable clusters running Spark on Amazon EMR.
• How to use EMR File System (EMRFS) with Spark to query data directly in Amazon S3.
• Common architectures to leverage Spark with Amazon DynamoDB, Amazon Redshift, Amazon Kinesis, and more.
Transitioning Compute Models: Hadoop MapReduce to SparkSlim Baltagi
This presentation is an analysis of the observed trends in the transition from the Hadoop ecosystem to the Spark ecosystem. The related talk took place at the Chicago Hadoop User Group (CHUG) meetup held on February 12, 2015.
This document discusses Route Origin Authorization (ROA) using the Resource Public Key Infrastructure (RPKI). It provides an overview of RPKI and how it uses digital certificates to validate the association between network resources and their holders. It describes APNIC's involvement in promoting RPKI adoption in the Asia Pacific region. It outlines the benefits of ROA, such as preventing route hijacking, and minimizing routing errors. It provides step-by-step instructions on creating ROAs using the MyAPNIC portal. Finally, it shares statistics on ROA adoption rates in South Asia and details APNIC's outreach efforts to encourage more networks to validate and filter routes using RPKI.
re:Invent 2022 DAT326 Deep dive into Amazon Aurora and its innovationsGrant McAlister
With an innovative architecture that decouples compute from storage as well as advanced features like Global Database and low-latency read replicas, Amazon Aurora reimagines what it means to be a relational database. The result is a modern database service that offers performance and high availability at scale, fully open-source MySQL- and PostgreSQL-compatible editions, and a range of developer tools for building serverless and machine learning-driven applications. In this session, dive deep into some of the most exciting features Aurora offers, including Aurora Serverless v2 and Global Database. Also learn about recent innovations that enhance performance, scalability, and security while reducing operational challenges.
Segment Routing Technology Deep Dive and Advanced Use CasesCisco Canada
The document provides an overview of Segment Routing technologies including SRv6. It begins with a recap of Segment Routing concepts and how it simplifies network operations. It then covers SRv6 which extends Segment Routing to IPv6 networks to take advantage of growing IPv6 adoption. The document discusses how SRv6 can further simplify networks and support new services and traffic patterns from 5G, IoT, and container-based microservices.
Realtime Indexing for Fast Queries on Massive Semi-Structured DataScyllaDB
Rockset is a realtime indexing database that powers fast SQL over semi-structured data such as JSON, Parquet, or XML without requiring any schematization. All data loaded into Rockset are automatically indexed and a fully featured SQL engine powers fast queries over semi-structured data without requiring any database tuning. Rockset exploits the hardware fluidity available in the cloud and automatically grows and shrinks the cluster footprint based on demand. Available as a serverless cloud service, Rockset is used by developers to build data-driven applications and microservices.
In this talk, we discuss some of the key design aspects of Rockset, such as Smart Schema and Converged Index. We describe Rockset's Aggregator Leaf Tailer (ALT) architecture that provides low latency queries on large datasets.Then we describe how you can combine lightweight transactions in ScyllaDB with realtime analytics on Rockset to power an user-facing application.
Tutorial: Using GoBGP as an IXP connecting routerShu Sugimoto
- Show you how GoBGP can be used as a software router in conjunction with quagga
- (Tutorial) Walk through the setup of IXP connecting router using GoBGP
Apache Arrow and Pandas UDF on Apache SparkTakuya UESHIN
The document discusses Apache Arrow and Pandas UDF on Apache Spark. It provides an overview of PySpark and Pandas, describes Python UDF and the new Pandas UDF feature, and explains how Pandas UDFs use Apache Arrow for efficient serialization and communication between the JVM and Python workers. The document outlines the physical operators implemented for different Pandas UDF types and the ongoing work to improve performance and functionality.
The document lists the names Akshay Rai and Jithesh T Rajan. It contains several bullet point sections that seem to list various skills, experiences, and qualifications for the two individuals. However, the document does not provide enough contextual information to determine its overall purpose or meaning.
Practical experiences and best practices for SSD and IBM iCOMMON Europe
This document discusses using solid state drives (SSDs) to optimize performance on IBM Power Systems. It provides examples showing that SSDs can significantly reduce batch window times, lower response times for transactions, and accelerate analytics. SSDs offer much faster read/write speeds than hard disk drives (HDDs) and can cut batch windows by 40-50%. For queries on data warehouses, SSDs deliver reductions of 60-87% in processing time. SSDs also allow for 85% faster data building and compression for deep analytics. Mixed SSD/HDD configurations with SSDs holding hot data can deliver the best price/performance.
Docker Inside/Out: the ‘real’ real-world of stacking containers in production...Codemotion
So you’ve already containerized the shit out of your code, broken down monoliths, microserviced the hell out of your app and have run some awesome workloads in your local, dev and test environments. It’s all looking good, but now what? Running Docker commands is one thing, but maintaining containers in production is a whole other ballgame. So, during this talk, I’ll show you the REAL wild world of Docker in production. With the added benefit of talking to and observing how over 900 of our customers have been using Docker in production.
S3, Cassandra or Outer Space? Dumping Time Series Data using Spark - Demi Be...Codemotion
The document discusses time series data processing using Apache Spark. It begins with an introduction of the speaker and an overview of Spark. It then describes the structure and flow of maritime location data. The challenges of processing this streaming time series data are discussed, including delays in data arrival that can cause holes in computations. The document outlines the evolution of their solution from using MongoDB to using Apache Spark with Cassandra and S3 for improved performance and scalability. It concludes with lessons about understanding data characteristics and choosing the right persistence layers.
Component-Based UI Architectures for the Web - Andrew Rota - Codemotion Rome...Codemotion
Today UI frameworks for the web are embracing the concept of “components”. But what does a component-focused architecture really mean? In this talk we’ll dive into the theory behind component-based UIs and what it means for the future of user interfaces on the web. At the conclusion of this talk, attendees will have an understanding of what makes component-based architectures distinct, and why such an approach might be the ideal solution for building web-based UIs.
Pronti per la legge sulla data protection GDPR? No Panic! - Domenico Maracci,...Codemotion
This document summarizes an upcoming presentation on preparing for the General Data Protection Regulation (GDPR). The presentation will cover what GDPR is, its key impacts and highlights, how to approach GDPR from an IT developer perspective securely, and conclude with a question and answer section. It will discuss GDPR requirements around personal data definitions, data subject rights, penalties for non-compliance, and how to design security and privacy into systems by default. It will also provide recommendations for organizations to discover, centralize and protect personal data, improve test data management practices, and securely expose data to users.
Monitoring Big Data Systems Done "The Simple Way" - Demi Ben-Ari - Codemotion...Codemotion
Once you start working with Big Data systems, you discover a whole bunch of problems you won’t find in monolithic systems. Monitoring all of the components becomes a big data problem itself. In the talk, we’ll mention all of the aspects that you should take into consideration when monitoring a distributed system using tools like Web Services, Spark, Cassandra, MongoDB, AWS. Not only the tools, what should you monitor about the actual data that flows in the system? We’ll cover the simplest solution with your day to day open source tools, the surprising thing, that it comes not from an Ops Guy.
Full-Text Search Explained - Philipp Krenn - Codemotion Rome 2017Codemotion
Today’s applications are expected to provide powerful full-text search. But how does that work in general and how do I implement it on my site or in my application? Actually, this is not as hard as it sounds at first. This talk covers: * How full-text search works in general and what the differences to databases are. * How the score or quality of a search result is calculated. * How to implement this with Elasticsearch. Attendees will learn how to add common search patterns to their applications without breaking a sweat.
From a Developer's POV: is Machine Learning Reshaping the World? - Simone Sca...Codemotion
There is no denying that machine learning is rapidly reshaping the technological horizon, fueled by increasing availability of data, computing power, and software (e.g., TensorFlow). Classical ML techniques are becoming a common tool for the everyday programmer, at the same time that sophisticated deep learning models are fueling driverless cars, advanced AI players, and more. This talk will survey the ways in which ML is impacting the programming world, as we try to answer the following questions: are we truly witnessing a new AI resurgence? If yes, what should any developer be aware of?
Microservices in GO - Massimiliano Dessì - Codemotion Rome 2017Codemotion
In this talk we'll see how to write a cloud native microservice with Go language, the microservices will be: Cloud native A twelve factor app Scalable with the GO built in concurrency Monitored with a distributed tracing system to check the latency Testable with a load test during the development Communications with different protocols.
Comics and immersive storytelling in Virtual Reality - Fabio Corrirossi - Cod...Codemotion
Virtual Reality is an undoubtedly ideal storytelling platform, whichever the story. After starting with the very first VR comic in the world, "Magnetique", a GearVR exclusive, we'll focus on telling virtual reality stories without resorting to 360° videos. Drawing techniques, stereoscopic coding, sequential art tips and tricks. And more. Our times allow for a unique opportunity to tell old stories, anew.
Anche per te "Open Source" = "qualcuno ha già fatto il lavoro al posto mio, e per di più gratis"? Ottimo, allora sei nel posto giusto e con l'approccio giusto! In questo talk, attraverso tanti episodi di vita vissuta come utente, contributor e maintainer, discuteremo di come trarre una serie di altri vantaggi da questo magico mondo, di come approcciarsi alle community e, perché no, anche delle gioie e dei dolori che ti aspettano se decidi di saltare la staccionata e di rendere (veramente) open il tuo codice.
Il game audio come processo ingegneristico - Davide Pensato - Codemotion Rome...Codemotion
Mostrare come il game audio è a tutti gli effetti una professionalità che unisce ad aspetti artistico creativi, forti competenze tecnico informatiche. L'audio designer può a tutti gli effetti considerarsi un ingegnere del suono, che applica modelli, regole e metodi rigorosi per ottenere il risultato. Tutto questo all'interno del ciclo di produzione, integrandosi con grafici, designer e programmatori
Cyber Security in Multi Cloud Architecture - Luca Di Bari - Codemotion Rome 2017Codemotion
Nuovi modelli di sicurezza in ambienti multi-cloud. Ridefinizione del concetto di Front-End. Nuovi approcci alle tematiche di sicurezza in scenari magmatici.
The busy developer guide to Docker - Maurice de Beijer - Codemotion Rome 2017Codemotion
Docker is all the rage these days and you are told all the time you need to use Docker to host your applications. But what is Docker and why has it become such a hot topic? Why is Microsoft updating Windows 2016 so be a Docker container host? What does using Docker mean for your application architecture or can you just take any application and host it using Docker? In this session Maurice de Beijer will explain the history of Docker as well as explain how you could use it with your applications. He will also explain what else, besides Docker, you will need to add to your architecture.
Xamarin.Forms is a framework for building cross-platform applications that share most of the UI codebase among the UWP, iOS and Android platforms. Due to the higher level of abstraction compared to Xamarin.Native, Xamarin.Forms applications may suffer from memory leaks and slow rendering times at the expense of the final user experience. In the session, we will explore the mechanisms used by Xamarin.Forms to translate abstract UI components into native ones, highlight with demos what are the main bottlenecks met by developer, how to solve them and get close to native performances.
Barbarians at the Gate(way) - Dave Lewis - Codemotion Rome 2017Codemotion
This talk will examine the tools, methods and data behind the DDoS attacks that are prevalent in the news headlines. Using information collected, I will demonstrate what the attackers are using to cause their mischief and mayhem and examine the timeline and progression of attackers as they move from the historical page defacers to the motivated DDoS attacker. I will look at the motivations and rationale that they have and try to share some sort of understanding as to what patterns to be aware of for their own protection.
Commodore 64 Mon Amour(2): sprite multiplexing. Il caso Catalypse e altre sto...Codemotion
Continuiamo il viaggio iniziato lo scorso anno nel magico mondo della moderna programmazione del Commodore 64. La scena italiana e romana è molto attiva. Dopo una brevissima introduzione sugli sprite in generale, il mitico Andrea Pompili, autore dello sparattutto Catalypse pubblicato da Genias nel 1992, ci spiegherà la tecnica dello sprite multiplexing, utilizzata per superare il noto limite degli 8 sprite contemporanei a schermo, applicata al suo gioco.
Container orchestration: the cold war - Giulio De Donato - Codemotion Rome 2017Codemotion
L’ecosistema degli orchestratori di container è in rapido movimento, una galassia di piattaforme e framework. Come si fa a scegliere quello giusto per le vostre esigenze? Vediamo tutti gli orchestratori in commercio, con i loro pro e contro: DC/OS, Kubernetes, Docker e anche quelli meno famosi ma saranno promesse, e anche le dinamiche e le scelte fatte.
Web Based Virtual Reality - Tanay Pant - Codemotion Rome 2017Codemotion
There has been a surge in the development of virtual reality applications with the production of easily accessible and sophisticated VR devices such as Oculus Rift, HTC Vive and Samsung Gear. Frameworks like A-Frame developed by the MozVR team combined with cheap alternatives such as Google Cardboard allows the developers to leverage the power of the web. The attendees of this talk would learn about the WebVR API, using A-Frame to build virtual worlds, creating virtual worlds for modern content display (such as reddit posts, news feeds, Instagram photos) as well as game development.
Handle insane devices traffic using Google Cloud Platform - Andrea Ulisse - C...Codemotion
In a world of connected devices it is really important to be prepared receiving and managing a huge amount of messages. In this context what is making the real difference is the backend that has to be able to handle safely every request in real time. In this talk we will show how the broad spectrum of highly scalable services makes Google Cloud Platform the perfect habitat for such as workloads.
Apache Ignite is an integrated and distributed In-Memory Data Fabric for computing and transacting on large-scale data sets in real-time, orders of magnitude faster than possible with traditional disk-based or flash technologies. It is designed to easily power both existing and new applications in a distributed, massively parallel architecture on affordable, industry-standard hardware. Apache Ignite addresses today's Fast Data and Big Data needs by providing a comprehensive in-memory data fabric, which includes a data grid with SQL and transactional capabilities, in-memory streaming, an in-memory file system, and more.
Spark Summit EU talk by Christos ErotocritouSpark Summit
This document discusses Apache Ignite and how it can be used with Apache Spark for fast data applications. It provides an overview of Ignite's in-memory data fabric capabilities, how it compares to Spark, and how Ignite can be integrated with Spark to provide shared resilient storage and distributed computing. Examples are given of reading and writing data between Ignite and Spark and using Ignite's in-memory file system and SQL support from Spark.
Getting Started with Apache Ignite as a Distributed DatabaseRoman Shtykh
This document summarizes a presentation about using Apache Ignite as a distributed database. It discusses the limitations of standard RDBMS and NoSQL databases, and how Ignite addresses these by providing a consistent and scalable solution. The presentation outlines Ignite's features and architecture, and shares reference cases from companies like ING Group and Ping An Insurance that have benefited from using Ignite. It concludes by providing additional Ignite learning resources and information on downloading a free trial.
From Data to Services at the Speed of BusinessAli Hodroj
From Data to Services at the Speed of Business: Applying cloud-native paradigm to combine fast data analytics with microservices architecture for hybrid workloads.
PayPal datalake journey | teradata - edge of next | san diego | 2017 october ...Deepak Chandramouli
PayPal Data Lake Journey | 2017-Oct | San Diego | Teradata Edge of Next
Gimel [http://www.gimel.io] is a Big Data Processing Library, open sourced by PayPal.
https://www.youtube.com/watch?v=52PdNno_9cU&t=3s
Gimel empowers analysts, scientists, data engineers alike to access a variety of Big Data / Traditional Data Stores - with just SQL or a single line of code (Unified Data API).
This is possible via the Catalog of Technical properties abstracted from users, along with a rich collection of Data Store Connectors available in Gimel Library.
A Catalog provider can be Hive or User Supplied (runtime) or UDC.
In addition, PayPal recently open sourced UDC [Unified Data Catalog], which can host and serve the Technical Metatada of the Data Stores & Objects. Visit http://www.unifieddatacatalog.io to experience first hand.
GridGain 6.0: Open Source In-Memory Computing Platform - Nikita IvanovJAXLondon2014
This document discusses Apache Ignite, an open source in-memory data fabric for real-time processing of big data. It provides an overview of Ignite's capabilities including in-memory caching and computing, distributed processing, streaming and complex event processing. Examples are given of how to use Ignite for distributed caching, transactions, MapReduce jobs and accelerating Hadoop workloads.
In memory computing principles by Mac Moore of GridGainData Con LA
This document provides an overview of in-memory computing principles and GridGain's in-memory data fabric technology. It discusses why in-memory computing is needed to handle today's data volumes and velocities, how architectures have evolved from traditional databases to in-memory data grids, key considerations for in-memory data grids, use cases for GridGain's technology, and highlights of GridGain's Release 6.5 including cross-language interoperability and dynamic schema changes.
Apache Ignite: In-Memory Hammer for Your Data Science ToolkitDenis Magda
Machine learning is a method of data analysis that automates the building of analytical models. By using algorithms that iteratively learn from data, computers are able to find hidden insights without the help of explicit programming. These insights bring tremendous benefits into many different domains. For business users, in particular, these insights help organizations improve customer experience, become more competitive, and respond much faster to opportunities or threats.
The availability of very powerful in-memory computing platforms, such as Apache Ignite, means that more organizations can benefit from machine learning today. In this presentation, we will discuss how the Compute Grid, Data Grid, and Machine Learning Grid components of Apache Ignite work together to enable your business to start reaping the benefits of machine learning. Through examples, attendees will learn how Apache Ignite can be used for data analysis and be the in-memory hammer in your machine learning toolkit.
In this talk I’ll present the SharedRDD – a high-performance in-memory caching layer for Spark jobs. We’ll work through 1) design & architecture of this component, 2) configuration and 3) actual Java and Scala usage examples.
A Big Data Lake Based on Spark for BBVA Bank-(Oscar Mendez, STRATIO)Spark Summit
This document describes BBVA's implementation of a Big Data Lake using Apache Spark for log collection, storage, and analytics. It discusses:
1) Using Syslog-ng for log collection from over 2,000 applications and devices, distributing logs to Kafka.
2) Storing normalized logs in HDFS and performing analytics using Spark, with outputs to analytics, compliance, and indexing systems.
3) Choosing Spark because it allows interactive, batch, and stream processing with one system using RDDs, SQL, streaming, and machine learning.
Simplifying Real-Time Architectures for IoT with Apache KuduCloudera, Inc.
3 Things to Learn About:
*Building scalable real time architectures for managing data from IoT
*Processing data in real time with components such as Kudu & Spark
*Customer case studies highlighting real-time IoT use cases
(ARC346) Scaling To 25 Billion Daily Requests Within 3 Months On AWSAmazon Web Services
What if you were told that within three months, you had to scale your existing platform from 1,000 req/sec (requests per second) to handle 300,000 req/sec with an average latency of 25 milliseconds? And that you had to accomplish this with a tight budget, expand globally, and keep the project confidential until officially announced by well-known global mobile device manufacturers? That’s what exactly happened to us. This session explains how The Weather Company partnered with AWS to scale our data distribution platform to prepare for unpredictable global demand. We cover the many challenges that we faced as we worked on architecture design, technology and tools selection, load testing, deployment and monitoring, and how we solved these challenges using AWS.
Apache Spark and Apache Ignite: Where Fast Data Meets the IoTDenis Magda
It is not enough to build a mesh of sensors or embedded devices to obtain more insights about the surrounding environment and optimize your production systems. Usually, your IoT solution needs to be capable of transferring enormous amounts of data to storage or the cloud where the data have to be processed further. Quite often, the processing of the endless streams of data has to be done in real-time so that you can react on the IoT subsystem's state accordingly.
This session will show attendees how to build a Fast Data solution that will receive endless streams from the IoT side and will be capable of processing the streams in real-time using Apache Ignite's cluster resources.
Solving enterprise challenges through scale out storage & big compute finalAvere Systems
Google Cloud Platform, Avere Systems, and Cycle Computing experts will share best practices for advancing solutions to big challenges faced by enterprises with growing compute and storage needs. In this “best practices” webinar, you’ll hear how these companies are working to improve results that drive businesses forward through scalability, performance, and ease of management.
The slides were from a webinar presented January 24, 2017. The audience learned:
- How enterprises are using Google Cloud Platform to gain compute and storage capacity on-demand
- Best practices for efficient use of cloud compute and storage resources
- Overcoming the need for file systems within a hybrid cloud environment
- Understand how to eliminate latency between cloud and data center architectures
- Learn how to best manage simulation, analytics, and big data workloads in dynamic environments
- Look at market dynamics drawing companies to new storage models over the next several years
Presenters communicated a foundation to build infrastructure to support ongoing demand growth.
This document discusses high performance spatial-temporal trajectory analysis using Spark. It covers the background of analyzing mobile signaling data to enable smarter urban planning. The solution architecture includes data sources, distributed file system, computation engine, and visualization. Technical designs address the big data platform, data governance, algorithm models, and Spark spatial computing. Example scenarios are presented for population heatmaps, commute routes, and office-residence imbalance analysis.
From Mainframe to Microservices: Vanguard’s Move to the Cloud - ENT331 - re:I...Amazon Web Services
The document discusses Vanguard's move from a mainframe-based architecture to microservices in the cloud. It describes Vanguard's initial complex IT environment with monolithic applications and a mainframe. Vanguard's approach was to replicate data from the mainframe to the cloud, refactor applications to make API calls to microservices, and migrate batch processes. This "strangulation strategy" allowed the monolith to be gradually replaced. The document outlines Vanguard's cloud data architecture and how it leveraged AWS services like RDS, DynamoDB, Lambda and Kinesis while addressing compliance and operational requirements. Lessons learned included preparing for regulatory needs and pushback to cloud migration.
If you're like most of the world, you're on an aggressive race to implement machine learning applications and on a path to get to deep learning. If you can give better service at a lower cost, you will be the winners in 2030. But infrastructure is a key challenge to getting there. What does the technology infrastructure look like over the next decade as you move from Petabytes to Exabytes? How are you budgeting for more colossal data growth over the next decade? How do your data scientists share data today and will it scale for 5-10 years? Do you have the appropriate security, governance, back-up and archiving processes in place? This session will address these issues and discuss strategies for customers as they ramp up their AI journey with a long term view.
SnappyData is a new open source project started by Pivotal GemFire founders to build a unified cluster capable of OLTP, OLAP, and streaming analytics using Spark. SnappyData fuses an elastic, highly available in-memory store for OLTP with Spark's memory manager and query engine to provide a single system for mixed workloads with fast ingestion, high concurrency and the ability to work with live, mutable data.
Similar to An Introduction to Apache Ignite - Mandhir Gidda - Codemotion Rome 2017 (20)
Fuzz-testing: A hacker's approach to making your code more secure | Pascal Ze...Codemotion
Increased complexity makes it very hard and time-consuming to keep your software bug-free and secure. We introduce fuzz-testing as a method for automatically and continuously discovering vulnerabilities hidden in your code. The talk will explain how fuzzing works and how to integrate fuzz-testing into your Software Development Life Cycle to increase your code’s security.
Pompili - From hero to_zero: The FatalNoise neverending storyCodemotion
It was 1993 when we decided to venture in a beat'em up game for Amiga. The Catalypse's success story pushed me and my comrade to create something astonishing for this incredible game machine... but things went harder, assumptions were slightly different, and italian competitors appeared out of nowhere... the project died in 1996. Story ended? Probably not...
Il Commodore 65 è un prototipo di personal computer che Commodore avrebbe dovuto mettere in commercio quale successore del Commodore 64. Purtroppo la sua realizzazione si fermò appunto allo stadio prototipale. Racconterò l'affascinante storia del suo sviluppo ed il perchè della soppressione del progetto ormai ad un passo dalla immissione in commercio.
Rivivere l'ebbrezza di progettare un vecchio computer o una consolle da bar è oggi possibile sfruttando le FPGA, ovvero logiche programmabili che consentono a chiunque di progettare il proprio hardware o di ricrearne uno del passato. In questa sessione si racconta come dal reverse engineering dell'hardware di vecchie glorie come il Commodore 64 e lo ZX Spectrum sia stato possibile farle rivivere attraverso tecnologie oggi alla portata di tutti.
Michel Schudel - Let's build a blockchain... in 40 minutes! - Codemotion Amst...Codemotion
There's a lot of talk about blockchain, but how does the technology behind it actually work? For developers, getting some hands-on experience is the fastest way to get familiair with new technologies. So let's build a blockchain, then! In this session, we're going to build one in plain old Java, and have it working in 40 minutes. We'll cover key concepts of a blockchain: transactions, blocks, mining, proof-of-work, and reaching consensus in the blockchain network. After this session, you'll have a better understanding of core aspects of blockchain technology.
Richard Süselbeck - Building your own ride share app - Codemotion Amsterdam 2019Codemotion
When was the last time you were truly lost? Thanks to the maps and location technology in our phones, a whole generation has now grown up in a world where getting lost is truly a thing of the past. Location technology goes far beyond maps in the palm of our hand, however. In this talk, we will explore how a ridesharing app works. How do we discover our destination?How do we find the closest driver? How do we display this information on a map? How do we find the best route?To answer these questions,we will be learning about a variety of location APIs, including Maps, Positioning, Geocoding etc.
Eward Driehuis - What we learned from 20.000 attacks - Codemotion Amsterdam 2019Codemotion
Eward Driehuis, SecureLink's research chief, will guide you through the bumpy ride we call the cyber threat landscape. As the industry has over a decade of experience of dealing with increasingly sophisticated attacks, you might be surprised to hear more attacks slip through the cracks than ever. From analyzing 20.000 of them in 2018, backed by a quarter of a million security events and over ten trillion data points, Eward will outline why this happens, how attacks are changing, and why it doesn't matter how neatly or securely you code.
Francesco Baldassarri - Deliver Data at Scale - Codemotion Amsterdam 2019 - Codemotion
IoT revolution is ended. Thanks to hardware improvement, building an intelligent ecosystem is easier than never before for both startups and large-scale enterprises. The real challenge is now to connect, process, store and analyze data: in the cloud, but also, at the edge. We’ll give a quick look on frameworks that aggregate dispersed devices data into a single global optimized system allowing to improve operational efficiency, to predict maintenance, to track asset in real-time, to secure cloud-connected devices and much more.
Martin Förtsch, Thomas Endres - Stereoscopic Style Transfer AI - Codemotion A...Codemotion
What if Virtual Reality glasses could transform your environment into a three-dimensional work of art in realtime in the style of a painting from Van Gogh? One of the many interesting developments in the field of Deep Learning is the so called "Style Transfer". It describes a possibility to create a patchwork (or pastiche) from two images. While one of these images defines the the artistic style of the result picture, the other one is used for extracting the image content. A team from TNG Technology Consulting managed to build an AI showcase using OpenCV and Tensorflow to realize such goggles.
Melanie Rieback, Klaus Kursawe - Blockchain Security: Melting the "Silver Bul...Codemotion
The document summarizes some of the security issues with blockchain technology. It discusses how blockchain is not a "silver bullet" and does not inherently solve problems like privacy and security of smart devices. It outlines various application security issues with complex code, protocols, and difficulty of updates on blockchains. Concerns over data immutability and security of smart contracts are also covered. The document questions whether blockchain truly provides the level of decentralization and anonymity claimed, and outlines some impossibility results and limitations of existing approaches to achieving security and privacy in blockchain systems.
Angelo van der Sijpt - How well do you know your network stack? - Codemotion ...Codemotion
The document provides an overview of the HTTP network protocol in its early stages of development. It summarizes the initial IMP (Interface Message Processor) software used to establish connections and transmit messages over the ARPANET. It outlines some early requirements for host-to-host software to enable simple and advanced use between computer systems. The document also describes the initial host software specifications, including establishing connections, transmitting data efficiently, and implementing error checking between connected systems. This was one of the first documents to define core aspects of the early HTTP network protocol to enable information exchange over the fledgling internet.
Lars Wolff - Performance Testing for DevOps in the Cloud - Codemotion Amsterd...Codemotion
Performance tests are not only an important instrument for understanding a system and its runtime environment. It is also essential in order to check stability and scalability – non-functional requirements that might be decisive for success. But won't my cloud hosting service scale for me as long as I can afford it? Yes, but… It only operates and scales resources. It won't automatically make your system fast, stable and scalable. This talk shows how such and comparable questions can be clarified with performance tests and how DevOps teams benefit from regular test practise.
Sascha Wolter - Conversational AI Demystified - Codemotion Amsterdam 2019Codemotion
Sascha will demonstrate the opportunities and challenges of Conversational AI learned from the practice. Both Technology and User Experience will be covered introducing a process finding micro-moments, writing happy paths, gathering intents, designing the conversational flow, and finally publishing on almost all channels including Voice Services and Chatbots. Valuable for enterprises, developers, and designers. All live on stage in just minutes and with almost no code.
Michele Tonutti - Scaling is caring - Codemotion Amsterdam 2019Codemotion
A key challenge we face at Pacmed is quickly calibrating and deploying our tools for clinical decision support in different hospitals, where data formats may vary greatly. Using Intensive Care Units as a case study, I’ll delve into our scalable Python pipeline, which leverages Pandas’ split-apply-combine approach to perform complex feature engineering and automatic quality checks on large time-varying data, e.g. vital signs. I’ll show how we use the resulting flexible and interpretable dataframes to quickly (re)train our models to predict mortality, discharge, and medical complications.
Pat Hermens - From 100 to 1,000+ deployments a day - Codemotion Amsterdam 2019Codemotion
Coolblue is a proud Dutch company, with a large internal development department; one that truly takes CI/CD to heart. Empowerment through automation is at the heart of these development teams, and with more than 1000 deployments a day, we think it's working out quite well. In this session, Pat Hermens (a Development Managers) will step you through what enables us to move so quickly, which tools we use, and most importantly, the mindset that is required to enable development teams to deliver at such a rapid pace.
James Birnie - Using Many Worlds of Compute Power with Quantum - Codemotion A...Codemotion
Quantum computers can use all of the possible pathways generated by quantum decisions to solve problems that will forever remain intractable to classical compute power. As the mega players vie for quantum supremacy and Rigetti announces its $1M "quantum advantage" prize, we live in exciting times. IBM-Q and Microsoft Q# are two ways you can learn to program quantum computers so that you're ready when the quantum revolution comes. I'll demonstrate some quantum solutions to problems that will forever be out of reach of classical, including organic chemistry and large number factorisation.
Don Goodman-Wilson - Chinese food, motor scooters, and open source developmen...Codemotion
Chinese food exploded across America in the early 20th century, rapidly adapting to local tastes while also spreading like wildfire. How was it able to spread so fast? The GY6 is a family of scooter engines that has achieved near total ubiquity in Europe. It is reliable and cheap to manufacture, and it's made in factories across China. How are these factories able to remain afloat? Chinese-American food and the GY6 are both riveting studies in product-market fit, and both are the product of a distributed open source-like development model. What lessons can we learn for open source software?
Pieter Omvlee - The story behind Sketch - Codemotion Amsterdam 2019Codemotion
The design space has exploded in size within the last few years and Sketch is one of the most important milestones to represent the phenomenon. But behind the scenes of this growing reality there is a remote team that revolutionizes the design space all without leaving the home office. This talk will present how Sketch has grown to become a modern, product designer's tool.
Dave Farley - Taking Back “Software Engineering” - Codemotion Amsterdam 2019Codemotion
Would you fly in a plane designed by a craftsman or would you prefer your aircraft to be designed by engineers? We are learning that science and empiricism works in software development, maybe now is the time to redefine what “Software Engineering” really means. Software isn't bridge-building, it is not car or aircraft development either, but then neither is Chemical Engineering. Engineering is different in different disciplines. Maybe it is time for us to begin thinking about retrieving the term "Software Engineering" maybe it is time to define what our "Engineering" discipline should be.
Joshua Hoffman - Should the CTO be Coding? - Codemotion Amsterdam 2019Codemotion
What is the job of a CTO and how does it change as a startup grows in size and scale? As a CTO, where should you spend your focus? As an engineer aspiring to be a CTO, what skills should you pursue? In this inspiring and personal talk, I describe my journey from early Red Hat engineer to CTO at Bloomon. I will share my view on what it means to be a CTO, and ultimately answer the question: Should the CTO be coding?
Retrieval Augmented Generation Evaluation with RagasZilliz
Retrieval Augmented Generation (RAG) enhances chatbots by incorporating custom data in the prompt. Using large language models (LLMs) as judge has gained prominence in modern RAG systems. This talk will demo Ragas, an open-source automation tool for RAG evaluations. Christy will talk about and demo evaluating a RAG pipeline using Milvus and RAG metrics like context F1-score and answer correctness.
Self-Healing Test Automation Framework - HealeniumKnoldus Inc.
Revolutionize your test automation with Healenium's self-healing framework. Automate test maintenance, reduce flakes, and increase efficiency. Learn how to build a robust test automation foundation. Discover the power of self-healing tests. Transform your testing experience.
Discovery Series - Zero to Hero - Task Mining Session 1DianaGray10
This session is focused on providing you with an introduction to task mining. We will go over different types of task mining and provide you with a real-world demo on each type of task mining in detail.
UiPath Community Day Amsterdam: Code, Collaborate, ConnectUiPathCommunity
Welcome to our third live UiPath Community Day Amsterdam! Come join us for a half-day of networking and UiPath Platform deep-dives, for devs and non-devs alike, in the middle of summer ☀.
📕 Agenda:
12:30 Welcome Coffee/Light Lunch ☕
13:00 Event opening speech
Ebert Knol, Managing Partner, Tacstone Technology
Jonathan Smith, UiPath MVP, RPA Lead, Ciphix
Cristina Vidu, Senior Marketing Manager, UiPath Community EMEA
Dion Mes, Principal Sales Engineer, UiPath
13:15 ASML: RPA as Tactical Automation
Tactical robotic process automation for solving short-term challenges, while establishing standard and re-usable interfaces that fit IT's long-term goals and objectives.
Yannic Suurmeijer, System Architect, ASML
13:30 PostNL: an insight into RPA at PostNL
Showcasing the solutions our automations have provided, the challenges we’ve faced, and the best practices we’ve developed to support our logistics operations.
Leonard Renne, RPA Developer, PostNL
13:45 Break (30')
14:15 Breakout Sessions: Round 1
Modern Document Understanding in the cloud platform: AI-driven UiPath Document Understanding
Mike Bos, Senior Automation Developer, Tacstone Technology
Process Orchestration: scale up and have your Robots work in harmony
Jon Smith, UiPath MVP, RPA Lead, Ciphix
UiPath Integration Service: connect applications, leverage prebuilt connectors, and set up customer connectors
Johans Brink, CTO, MvR digital workforce
15:00 Breakout Sessions: Round 2
Automation, and GenAI: practical use cases for value generation
Thomas Janssen, UiPath MVP, Senior Automation Developer, Automation Heroes
Human in the Loop/Action Center
Dion Mes, Principal Sales Engineer @UiPath
Improving development with coded workflows
Idris Janszen, Technical Consultant, Ilionx
15:45 End remarks
16:00 Community fun games, sharing knowledge, drinks, and bites 🍻
How UiPath Discovery Suite supports identification of Agentic Process Automat...DianaGray10
📚 Understand the basics of the newly persona-based LLM-powered Agentic Process Automation and discover how existing UiPath Discovery Suite products like Communication Mining, Process Mining, and Task Mining can be leveraged to identify APA candidates.
Topics Covered:
💡 Idea Behind APA: Explore the innovative concept of Agentic Process Automation and its significance in modern workflows.
🔄 How APA is Different from RPA: Learn the key differences between Agentic Process Automation and Robotic Process Automation.
🚀 Discover the Advantages of APA: Uncover the unique benefits of implementing APA in your organization.
🔍 Identifying APA Candidates with UiPath Discovery Products: See how UiPath's Communication Mining, Process Mining, and Task Mining tools can help pinpoint potential APA candidates.
🔮 Discussion on Expected Future Impacts: Engage in a discussion on the potential future impacts of APA on various industries and business processes.
Enhance your knowledge on the forefront of automation technology and stay ahead with Agentic Process Automation. 🧠💼✨
Speakers:
Arun Kumar Asokan, Delivery Director (US) @ qBotica and UiPath MVP
Naveen Chatlapalli, Solution Architect @ Ashling Partners and UiPath MVP
Keynote : AI & Future Of Offensive SecurityPriyanka Aash
In the presentation, the focus is on the transformative impact of artificial intelligence (AI) in cybersecurity, particularly in the context of malware generation and adversarial attacks. AI promises to revolutionize the field by enabling scalable solutions to historically challenging problems such as continuous threat simulation, autonomous attack path generation, and the creation of sophisticated attack payloads. The discussions underscore how AI-powered tools like AI-based penetration testing can outpace traditional methods, enhancing security posture by efficiently identifying and mitigating vulnerabilities across complex attack surfaces. The use of AI in red teaming further amplifies these capabilities, allowing organizations to validate security controls effectively against diverse adversarial scenarios. These advancements not only streamline testing processes but also bolster defense strategies, ensuring readiness against evolving cyber threats.
The Challenge of Interpretability in Generative AI Models.pdfSara Kroft
Navigating the intricacies of generative AI models reveals a pressing challenge: interpretability. Our blog delves into the complexities of understanding how these advanced models make decisions, shedding light on the mechanisms behind their outputs. Explore the latest research, practical implications, and ethical considerations, as we unravel the opaque processes that drive generative AI. Join us in this insightful journey to demystify the black box of artificial intelligence.
Dive into the complexities of generative AI with our blog on interpretability. Find out why making AI models understandable is key to trust and ethical use and discover current efforts to tackle this big challenge.
Welcome to Cyberbiosecurity. Because regular cybersecurity wasn't complicated...Snarky Security
How wonderful it is that in our modern age, every bit of our biological data can be digitized, stored, and potentially pilfered by cyber thieves! Isn't it just splendid to think that while scientists are busy pushing the boundaries of biotechnology, hackers could be plotting the next big bio-data heist? This delightful scenario is brought to you by the ever-expanding digital landscape of biology and biotechnology, where the integration of computer science, engineering, and data science transforms our understanding and manipulation of biological systems.
While the fusion of technology and biology offers immense benefits, it also necessitates a careful consideration of the ethical, security, and associated social implications. But let's be honest, in the grand scheme of things, what's a little risk compared to potential scientific achievements? After all, progress in biotechnology waits for no one, and we're just along for the ride in this thrilling, slightly terrifying, adventure.
So, as we continue to navigate this complex landscape, let's not forget the importance of robust data protection measures and collaborative international efforts to safeguard sensitive biological information. After all, what could possibly go wrong?
-------------------------
This document provides a comprehensive analysis of the security implications biological data use. The analysis explores various aspects of biological data security, including the vulnerabilities associated with data access, the potential for misuse by state and non-state actors, and the implications for national and transnational security. Key aspects considered include the impact of technological advancements on data security, the role of international policies in data governance, and the strategies for mitigating risks associated with unauthorized data access.
This view offers valuable insights for security professionals, policymakers, and industry leaders across various sectors, highlighting the importance of robust data protection measures and collaborative international efforts to safeguard sensitive biological information. The analysis serves as a crucial resource for understanding the complex dynamics at the intersection of biotechnology and security, providing actionable recommendations to enhance biosecurity in an digital and interconnected world.
The evolving landscape of biology and biotechnology, significantly influenced by advancements in computer science, engineering, and data science, is reshaping our understanding and manipulation of biological systems. The integration of these disciplines has led to the development of fields such as computational biology and synthetic biology, which utilize computational power and engineering principles to solve complex biological problems and innovate new biotechnological applications. This interdisciplinary approach has not only accelerated research and development but also introduced new capabilities such as gene editing and biomanufact
This PDF delves into the aspects of information security from a forensic perspective, focusing on privacy leaks. It provides insights into the methods and tools used in forensic investigations to uncover and mitigate privacy breaches in mobile and cloud environments.
Choosing the Best Outlook OST to PST Converter: Key Features and Considerationswebbyacad software
When looking for a good software utility to convert Outlook OST files to PST format, it is important to find one that is easy to use and has useful features. WebbyAcad OST to PST Converter Tool is a great choice because it is simple to use for anyone, whether you are tech-savvy or not. It can smoothly change your files to PST while keeping all your data safe and secure. Plus, it can handle large amounts of data and convert multiple files at once, which can save you a lot of time. It even comes with 24*7 technical support assistance and a free trial, so you can try it out before making a decision. Whether you need to recover, move, or back up your data, Webbyacad OST to PST Converter is a reliable option that gives you all the support you need to manage your Outlook data effectively.
"Hands-on development experience using wasm Blazor", Furdak Vladyslav.pptxFwdays
I will share my personal experience of full-time development on wasm Blazor
What difficulties our team faced: life hacks with Blazor app routing, whether it is necessary to write JavaScript, which technology stack and architectural patterns we chose
What conclusions we made and what mistakes we committed
Collection of logical in-memory components that solve high performance and scalability challenges
Collection of logical in-memory components that solve high performance and scalability challenges
Collection of logical in-memory components that solve high performance and scalability challenges
application level soft-locking using versioning
deadlock protection
When it comes to querying and acting on data — including in Big Data/Fast Data environments — SQL still dominates. And no other database-agnostic in-memory solution handles SQL functionality like the Apache Ignite In-Memory Data Fabric.
application level soft-locking using versioning
deadlock protection (Serializable)
Apache Ignite allows for most of the data structures from the java.util.concurrent framework to be used in a distributed fashion
Collection of logical in-memory components that solve high performance and scalability challenges
ComputeTask with service injection
Direct API for Fork/Join
Collection of logical in-memory components that solve high performance and scalability challenges
Direct API for Fork/Join
Affinity run of spark jobs
- supports multiple infrastructure including Google Cloud, AWS & Docker