Jonathan S. Katz gave a talk on safely protecting passwords in PostgreSQL. He discussed:
- The evolution of password management in PostgreSQL, from storing passwords in plain text to using md5 hashes to modern SCRAM authentication.
- How plain text and md5 password storage are insecure as passwords can be intercepted or cracked.
- The SCRAM authentication standard which allows two parties to verify they know a secret without exchanging the secret directly.
- How PostgreSQL implements SCRAM-SHA-256 to generate a secure verifier from the password and authenticate users with random salts and iterations to secure against brute force attacks.
PostgreSQL 10 had added built-in logical replication which tackles some of the limitations of physical replication and opens up the possibility of promising new areas of replication. In this webinar, we will introduce the concept of logical replication and demonstrate how one can configure a logical replication in no time.
Highlights include:
- Basic architecture, including the publisher and subscriber model
- Configuration, administration, and monitoring
- Limitations and future plans
O documento apresenta um mapa mental sobre o Capítulo 1 do Syllabus Foundation Level - CTFL, criado por Ramon de Pieri Saraiva para facilitar o estudo e memorização do conteúdo. O autor do mapa enfatiza que ele deve ser usado em conjunto com a leitura do próprio Syllabus.
PostgreSQL Replication High Availability MethodsMydbops
This slides illustrates the need for replication in PostgreSQL, why do you need a replication DB topology, terminologies, replication nodes and many more.
How to Lock Down Apache Kafka and Keep Your Streams Safeconfluent
The document discusses how to secure Apache Kafka clusters through authentication. It describes several authentication mechanisms including TLS, SASL/GSSAPI using Kerberos, and SASL/PLAIN and SASL/SCRAM for username and password authentication. TLS provides server and client authentication but has performance overhead while SASL mechanisms like GSSAPI and SCRAM integrate with existing authentication systems with lower performance impact. The document provides configuration details and security considerations for each mechanism.
This talk will provide a brief overview about some of the latest developments in the Linux networking world: Things like VLAN-aware-bridges, VXLAN, VRF-Lites, as well as MPLS support will be shown with practical examples.
Everyone still using »ifconfig«, »route«, »arp« etc. might want to attend to get an idea how to use the Linux swiss army knife for networkers (»ip«) which already has replaced or will replace all the old tools on current distributions.
For Debian based systems ifupdown2 provides a convenient replacement for the old ifupdown toolchain including configuration for VLAN interfaces and LAGs which previously required auxiliary tools.
At the end you will get a glimpse into building your own SDN with Debian Linux, ifupdown2, Salt Stack and Python.
back to top
Managing python at scale without breaking the bankPyData
By Misha Tselman
PyData New York City 2017
Athena is the largest Python-based ecosystem in J.P. Morgan Chase and among the largest in the world. Maintaining consistency and stability while enabling creativity, research, and speed of development are critical for supporting our clients and staying ahead. Can we fit everything into the same platform?
PGConf.ASIA 2019 Bali - Setup a High-Availability and Load Balancing PostgreS...Equnix Business Solutions
PGConf.ASIA 2019 Bali - 10 September 2019
Speaker: Bo Peng
Room: SQL
Title: Setup a High-Availability and Load Balancing PostgreSQL Cluster - New Features of Pgpool-II 4.1
Best practices for MySQL/MariaDB Server/Percona Server High AvailabilityColin Charles
Best practices for MySQL/MariaDB Server/Percona Server High Availability - presented at Percona Live Amsterdam 2016. The focus is on picking the right High Availability solution, discussing replication, handling failure (yes, you can achieve a quick automatic failover), proxies (there are plenty), HA in the cloud/geographical redundancy, sharding solutions, how newer versions of MySQL help you, and what to watch for next.
PostgreSQL is a very popular and feature-rich DBMS. At the same time, PostgreSQL has a set of annoying wicked problems, which haven't been resolved in decades. Miraculously, with just a small patch to PostgreSQL core extending this API, it appears possible to solve wicked PostgreSQL problems in a new engine made within an extension.
The document summarizes a presentation about Apache Ratis, a Raft consensus library. It introduces Raft consensus and describes Ratis' features like leader election, log replication, pluggable components, and use cases in Hadoop projects like Ozone. It also outlines Ratis' development status and future work areas like performance, metrics, security, and documentation.
This document discusses PostgreSQL replication. It provides an overview of replication, including its history and features. Replication allows data to be copied from a primary database to one or more standby databases. This allows for high availability, load balancing, and read scaling. The document describes asynchronous and synchronous replication modes.
This document discusses exactly once semantics in Apache Kafka 0.11. It provides an overview of how Kafka achieved exactly once delivery between producers and consumers. Key points include:
- Kafka 0.11 introduced exactly once semantics with changes to support transactions and deduplication.
- Producers can write in a transactional fashion and receive acknowledgments of committed writes from brokers.
- Brokers store commit markers to track the progress of transactions and ensure no data loss during failures.
- Consumers can read from brokers in a transactional mode and receive data only from committed transactions, guaranteeing no duplication of records.
- This allows reliable message delivery semantics between producers and consumers with Kafka acting as
Delivered as plenary at USENIX LISA 2013. video here: https://www.youtube.com/watch?v=nZfNehCzGdw and https://www.usenix.org/conference/lisa13/technical-sessions/plenary/gregg . "How did we ever analyze performance before Flame Graphs?" This new visualization invented by Brendan can help you quickly understand application and kernel performance, especially CPU usage, where stacks (call graphs) can be sampled and then visualized as an interactive flame graph. Flame Graphs are now used for a growing variety of targets: for applications and kernels on Linux, SmartOS, Mac OS X, and Windows; for languages including C, C++, node.js, ruby, and Lua; and in WebKit Web Inspector. This talk will explain them and provide use cases and new visualizations for other event types, including I/O, memory usage, and latency.
[Open Infrastructure & Cloud Native Days Korea 2019]
커뮤니티 버전의 OpenStack 과 Ceph를 활용하여 대고객서비스를 구축한 사례를 공유합니다. 유연성을 확보한 기업용 클라우드 서비스 구축 사례와 높은 수준의 보안을 요구하는 거래소 서비스를 구축, 운영한 사례를 소개합니다. 또한 이 프로젝트에 사용된 기술 스택 및 장애 해결사례와 최적화 방안을 소개합니다. 오픈스택은 역시 오픈소스컨설팅입니다.
#openstack #ceph #openinfraday #cloudnative #opensourceconsulting
This document discusses logical replication with pglogical. It begins by explaining that pglogical performs row-oriented replication and outputs replication data that can be used in various ways. It then covers the architectures of standalone PostgreSQL, physical replication, and logical replication. The rest of the document discusses key aspects of pglogical such as its output plugin, selective replication capabilities, performance and future plans, and examples of using the output with other applications.
From Zero to Hero with Kafka Connect (Robin Moffat, Confluent) Kafka Summit L...confluent
Integrating Apache Kafka with other systems in a reliable and scalable way is often a key part of a streaming platform. Fortunately, Apache Kafka includes the Connect API that enables streaming integration both in and out of Kafka. Like any technology, understanding its architecture and deployment patterns is key to successful use, as is knowing where to go looking when things aren’t working. This talk will discuss the key design concepts within Kafka Connect and the pros and cons of standalone vs distributed deployment modes. We’ll do a live demo of building pipelines with Kafka Connect for streaming data in from databases, and out to targets including Elasticsearch. With some gremlins along the way, we’ll go hands-on in methodically diagnosing and resolving common issues encountered with Kafka Connect. The talk will finish off by discussing more advanced topics including Single Message Transforms, and deployment of Kafka Connect in containers.
Get Your Insecure PostgreSQL Passwords to SCRAMJonathan Katz
Passwords: they just seem to work. You connect to your PostgreSQL database and you are prompted for your password. You type in the correct character combination, and presto! you're in, safe and sound.
But what if I told you that all was not as it seemed. What if I told you there was a better, safer way to use passwords with PostgreSQL? What if I told you it was imperative that you upgraded, too?
PostgreSQL 10 introduced SCRAM (Salted Challenge Response Authentication Mechanism), introduced in RFC 5802, as a way to securely authenticate passwords. The SCRAM algorithm lets a client and server validate a password without ever sending the password, whether plaintext or a hashed form of it, to each other, using a series of cryptographic methods.
In this talk, we will look at:
* A history of the evolution of password storage and authentication in PostgreSQL
* How SCRAM works with a step-by-step deep dive into the algorithm (and convince you why you need to upgrade!)
* SCRAM channel binding, which helps prevent MITM attacks during authentication
* How to safely set and modify your passwords, as well as how to upgrade to SCRAM-SHA-256 (which we will do live!)
all of which will be explained by some adorable elephants and hippos!
At the end of this talk, you will understand how SCRAM works, how to ensure your PostgreSQL drivers supports it, how to upgrade your passwords to using SCRAM-SHA-256, and why you want to tell other PostgreSQL password mechanisms to SCRAM!
Rspamd is a spam filtering system that is:
- Written in C for performance and uses an event-driven model to process messages asynchronously for scalability.
- Capable of detecting spam through a variety of filtering methods like policies, DNS lists, headers, text patterns, and machine learning.
- Integrates with mail transfer agents using plugins to modify or reject messages based on spam detection.
This document discusses tools for static analysis of files, including ClamAV and YARA. ClamAV is an open-source antivirus engine that uses signatures to detect malware. Signatures can include strings, hashes, and byte patterns. YARA allows for more flexible identification of malware through rules that can detect strings, regular expressions, and byte patterns. Examples of ClamAV and YARA signatures are provided.
The document discusses various methods for encrypting and authenticating data in PHP, including:
1. Encrypting data with md5() hash functions, the MCrypt package, and file-based authentication.
2. MCrypt supports two-way encryption algorithms like DES and allows encrypting and decrypting data.
3. File-based authentication parses a text file into an array to authenticate users by comparing hashed passwords.
[CB20] Vulnerabilities of Machine Learning Infrastructure by Sergey GordeychikCODE BLUE
The boom of AI brought to the market a set of impressive solutions both on the hardware and software side. On the other hand, massive implementation of AI in various areas brings about problems, and security is one of the greatest concerns.
In this talk we will present results of hands-on vulnerability research of different components of AI infrastructure including NVIDIA DGX GPU servers, ML frameworks such as Pytorch, Keras and Tensorflow, data processing pipelines and specific applications, including Medical Imaging and face recognition powered CCTV. Updated Internet Census toolkit based on the Grinder framework will be introduced.
This document discusses test-driven development (TDD) with Puppet. It explains why testing is important, such as having confidence to change code without breaking things. It covers what to test, like parameters, resources, and conditional logic. Tools for testing like rspec-puppet are also presented. The document demonstrates writing tests first, then code to pass the tests, for a sample MOTD module. It encourages testing all parameters and provides examples for validating values. Finally, it discusses integrating tests with Travis CI for continuous integration.
The document discusses OpenSSL commands for encryption and decryption. It covers symmetric encryption techniques like AES-256 and asymmetric encryption with RSA key pairs. It provides examples of encrypting and decrypting a file using OpenSSL commands like openssl enc and openssl dec. It also discusses hashing algorithms like MD5 and SHA1 for generating hashes of files.
The document discusses encryption techniques and best practices. It recommends using high-level encryption APIs that handle keys and algorithms securely rather than implementing encryption directly. Symmetric encryption with AES is recommended for encrypting data, and hashing with SHA-256 for passwords. HMAC is recommended to protect data integrity. Home-grown ciphers and algorithms are to be avoided.
This document discusses password security and best practices. It notes that 66% of corporate network breaches are due to weak passwords. Passwords should not be stored in plaintext or with reversible encryption, but rather hashed with salts. Hashing strengthens security but using salts is important to avoid rainbow table attacks. The document recommends choosing long, unique passphrases rather than short, dictionary words for passwords. It also advocates the use of password managers and two-factor authentication for strongest security.
The document discusses various methods of cracking salted password hashes, including determining the hashing algorithm used based on hash length, brute forcing hashes when the salt is known, and exploiting situations where the salt is constant rather than random to facilitate cracking multiple hashes. It provides examples of insecure password hashing implementations and advises using unique random salts with each hashed password for proper security.
This document discusses monitoring MySQL performance using StatsD and Graphite. It provides an overview of the tools and how they are used. StatsD collects metrics from applications and services and sends them to Graphite for storage and visualization. The document describes how a custom MySQL StatsD daemon was created to gather MySQL metrics and send them to StatsD in real-time for high granularity monitoring and graphing in Graphite.
This document discusses security vulnerabilities in SAP systems. It notes that many SAP systems have non-web services exposed that could allow remote access. It also details how passwords are sometimes stored insecurely in SAP shortcuts, log files, and database tables, allowing attackers to gain access to systems and steal sensitive data. The document recommends steps companies can take to prevent such vulnerabilities, like patching systems, not storing passwords in shortcuts, and using more secure authentication methods.
Efficient Spark Analytics on Encrypted Data with Gidon GershinskyDatabricks
This document discusses efficient Spark analytics on encrypted data using Parquet modular encryption. It provides an overview of the problem of protecting sensitive data at rest while preserving analytics performance. It then describes Parquet modular encryption which enables columnar projection, predicate pushdown and fine-grained access control on encrypted Parquet data. Finally, it demonstrates a connected car use case and shows the performance implications of encryption on Spark analytics are minimal.
Rspamd is an open-source spam filtering system written in C that uses an event-driven architecture. It processes emails using multiple rules to evaluate scores, supports plugins in Lua, and has a self-contained web interface. Performance is a key design goal through techniques like optimized regular expressions, event-driven network processing, and optimized memory usage. It uses techniques like Bayesian filtering with n-grams, fuzzy hashes, and DNS-based filtering while prioritizing security through input validation, encryption for network traffic, and limiting DNS queries.
2014 05-07-fr - add dev series - session 6 - deploying your application-2MongoDB
The document discusses MongoDB replication and sharding. Replication uses replica sets for high availability and disaster recovery. Sharding partitions data across multiple servers (shards) to improve scalability. The key points covered include:
- Replication maintains copies of data on multiple servers for redundancy and high availability. It uses replica sets and elections for failover.
- Sharding partitions data by a shard key across multiple mongod instances (shards) to scale reads and writes. It requires config servers to store metadata and mongos instances as query routers.
- Write concerns allow controlling acknowledgments and replication of write operations. Tag-aware sharding allows controlling data distribution across shards.
Next Generation DevOps in Drupal: DrupalCamp London 2014Barney Hanlon
In this talk, Barney will be discussing and demonstrating how to:
- Use nginx, Varnish and Apache together in a "SPDY sandwich" to support HTTP 2.0
- Setting up SSL properly to mitigate against attack vectors
- Performance improvements with mod_pagespeed and nginx
- Deploying Drupal sites with Docker containers
Barney is a Technical Team Leader at Inviqa, a Drupal Association member and writes for Techportal on using technologies to improve website performance. He first started using PHP professionally in 2003, and has over seventeen years experience in software development. He is an advocate of Scrum methodology and has an interest in performance optimization, researching and speaking on various techniques to improve user experience through faster load times.
Spy hard, challenges of 100G deep packet inspection on x86 platformRedge Technologies
This document discusses challenges and approaches for performing deep packet inspection (DPI) at speeds of 100 gigabits per second and beyond on x86 platforms. It begins by explaining why DPI is needed at such high speeds, for tasks like large-scale intrusion detection. It then examines the performance requirements for scanning payloads at 100Gbps rates. The document reviews different software approaches for payload matching, such as regular expressions, and hardware that can assist, such as Intel's Hyperscan technology. It also provides examples of how Hyperscan can be integrated into real-world intrusion detection and prevention systems.
Security Best Practice: Oracle passwords, but secure!Stefan Oehrli
Authentication is an integral part of database security. If authentication or passwords are insufficient or inadequate, all further security measures are generally useless. But how do you ensure that passwords are complex and authentication is secure? In this presentation, the password hashes will be explained and it will be shown how to make sure passwords and authentication are state of the art. Focusing on the current versions of the Oracle database, the following topics will be discussed:
- Oracle database authentication
- Password verification and hashes
- Where can I find password hashes?
- Check and password hashes.
- Discussion of various risks related to authentication.
- Discussion of password policies and strong passwords.
- Customer Use Case in the DB Vault environment "ups we have forgotten the passwords".
The presentation will be supplemented by corresponding examples and live demos.
PLNOG 18 - Paweł Małachowski - Spy hard czyli regexpem po pakietachPROIDEA
This document discusses challenges related to implementing 100G deep packet inspection on x86 platforms. It begins by explaining why DPI is needed at speeds of 100G+ and examples of large DDoS attacks where DPI could help. It then sizes the requirements for scanning packets at 100Gbit/s rates. Next, it covers approaches to software payload lookup like regular expressions and finite state machines. The rest of the document discusses specific regex and DPI techniques like Hyperscan, benchmarking regex performance, and examples of hardware that could accelerate such inspection.
DEFCON 23 - Eijah - crypto for hackersFelipe Prado
This document provides an introduction to cryptography concepts and algorithms. It defines common crypto terminology like encryption, hashing, and key derivation. It then demonstrates examples of symmetric ciphers like AES, cryptographic hashes like MD5 and SHA256, HMAC to provide integrity to hashes, and key agreement schemes like Diffie-Hellman. The examples are shown using C++ code snippets from crypto libraries like Crypto++ to encrypt, hash, derive keys from passwords, and perform key exchange between two parties. The document aims to educate readers on fundamental crypto concepts and their usage through code examples.
Similar to Safely Protect PostgreSQL Passwords - Tell Others to SCRAM (20)
Vectors are the new JSON in PostgreSQL (SCaLE 21x)Jonathan Katz
Vectors are a centuries old, well-studied mathematical concept, yet they pose many challenges around efficient storage and retrieval in database systems. The heightened ease-of-use of AI/ML has lead to a surge of interested of storing vector data alongside application data, leading to some unique challenges. PostgreSQL has seen this story before with JSON, when JSON became the lingua franca of the web. So how can you use PostgreSQL to manage your vector data, and what challenges should you be aware of?
In this session, we'll review what vectors are, how they are used in applications, and what users are looking for in vector storage and search systems. We'll then see how you can search for vector data in PostgreSQL, including looking at best practices for using pgvector, an extension that adds additional vector search capabilities to PostgreSQL. Finally, we'll review ongoing development in both PostgreSQL and pgvector that will make it easier and more performant to search vector data in PostgreSQL.
There are parallels between storing JSON data in PostgreSQL and storing vectors that are produced from AI/ML systems. This lightning talk briefly covers the similarities in use-cases in storing JSON and vectors in PostgreSQL, shows some of the use-cases developers have for querying vectors in Postgres, and some roadmap items for improving PostgreSQL as a vector database.
Build a Complex, Realtime Data Management App with Postgres 14!Jonathan Katz
Congratulations: you've been selected to build an application that will manage reservations for rooms!
On the surface, this sounds simple, but you are building a system for managing a high traffic reservation web page, so we know that a lot of people will be accessing the system. Therefore, we need to ensure that the system can handle all of the eager users that will be flooding the website checking to see what availability each room has.
Fortunately, PostgreSQL is prepared for this! And even better, we will be using Postgres 14 to make the problem even easier!
We will explore the following PostgreSQL features:
* Data types and their functionality, such as:
* Data/Time types
* Ranges / Multirnages
Indexes such as:
* GiST
* Common Table Expressions and Recursion (though multiranges will make things easier!)
* Set generating functions and LATERAL queries
* Functions and the PL/PGSQL
* Triggers
* Logical decoding and streaming
We will be writing our application primary with SQL, though we will sneak in a little bit of Python and using Kafka to demonstrate the power of logical decoding.
At the end of the presentation, we will have a working application, and you will be happy knowing that you provided a wonderful user experience for all users made possible by the innovation of PostgreSQL!
Operating PostgreSQL at Scale with KubernetesJonathan Katz
The maturation of containerization platforms has changed how people think about creating development environments and has eliminated many inefficiencies for deploying applications. These concept and technologies have made its way into the PostgreSQL ecosystem as well, and tools such as Docker and Kubernetes have enabled teams to run their own “database-as-a-service” on the infrastructure of their choosing.
All this sounds great, but if you are new to the world of containers, it can be very overwhelming to find a place to start. In this talk, which centers around demos, we will see how you can get PostgreSQL up and running in a containerized environment with some advanced sidecars in only a few steps! We will also see how it extends to a larger production environment with Kubernetes, and what the future holds for PostgreSQL in a containerized world.
We will cover the following:
* Why containers are important and what they mean for PostgreSQL
* Create a development environment with PostgreSQL, pgadmin4, monitoring, and more
* How to use Kubernetes to create your own "database-as-a-service"-like PostgreSQL environment
* Trends in the container world and how it will affect PostgreSQL
At the conclusion of the talk, you will understand the fundamentals of how to use container technologies with PostgreSQL and be on your way to running a containerized PostgreSQL environment at scale!
Building a Complex, Real-Time Data Management ApplicationJonathan Katz
Congratulations: you've been selected to build an application that will manage whether or not the rooms for PGConf.EU are being occupied by a session!
On the surface, this sounds simple, but we will be managing the rooms of PGConf.EU, so we know that a lot of people will be accessing the system. Therefore, we need to ensure that the system can handle all of the eager users that will be flooding the PGConf.EU website checking to see what availability each of the PGConf.EU rooms has.
To do this, we will explore the following PGConf.EU features:
* Data types and their functionality, such as:
* Data/Time types
* Ranges
Indexes such as:
* GiST
* SP-Gist
* Common Table Expressions and Recursion
* Set generating functions and LATERAL queries
* Functions and the PL/PGSQL
* Triggers
* Logical decoding and streaming
We will be writing our application primary with SQL, though we will sneak in a little bit of Python and using Kafka to demonstrate the power of logical decoding.
At the end of the presentation, we will have a working application, and you will be happy knowing that you provided a wonderful user experience for all PGConf.EU attendees made possible by the innovation of PGConf.EU!
Using PostgreSQL With Docker & Kubernetes - July 2018Jonathan Katz
The maturation of containerization platforms has changed how people think about creating development environments and has eliminated many inefficiencies for deploying applications. These concept and technologies have made its way into the PostgreSQL ecosystem as well, and tools such as Docker and Kubernetes have enabled teams to run their own “database-as-a-service” on the infrastructure of their choosing.
In this talk, we will cover the following:
- Why containers are important and what they mean for PostgreSQL
- Setting up and managing a PostgreSQL along with pgadmin4 and monitoring
- Running PostgreSQL on Kubernetes with a Demo
- Trends in the container world and how it will affect PostgreSQL
An Introduction to Using PostgreSQL with Docker & KubernetesJonathan Katz
The maturation of containerization platforms has changed how people think about creating development environments and has eliminated many inefficiencies for deploying applications. These concept and technologies have made its way into the PostgreSQL ecosystem as well, and tools such as Docker and Kubernetes have enabled teams to run their own “database-as-a-service” on the infrastructure of their choosing.
In this talk, we will cover the following:
- Why containers are important and what they mean for PostgreSQL
- Setting up and managing a PostgreSQL container
- Extending your setup with a pgadmin4 container
- Container orchestration: What this means, and how to use Kubernetes to leverage database-as-a-service with PostgreSQL
- Trends in the container world and how it will affect PostgreSQL
Developing and Deploying Apps with the Postgres FDWJonathan Katz
This document summarizes Jonathan Katz's experience building a foreign data wrapper (FDW) between two PostgreSQL databases to enable an API for his company VenueBook. He created separate "app" and "api" databases, with the api database using FDWs to access tables in the app database. This allowed inserting and querying data across databases. However, he encountered permission errors and had to grant various privileges on the remote database to make it work properly, demonstrating the importance of permissions management with FDWs.
What's the great thing about a database? Why, it stores data of course! However, one feature that makes a database useful is the different data types that can be stored in it, and the breadth and sophistication of the data types in PostgreSQL is second-to-none, including some novel data types that do not exist in any other database software!
This talk will take an in-depth look at the special data types built right into PostgreSQL version 9.4, including:
* INET types
* UUIDs
* Geometries
* Arrays
* Ranges
* Document-based Data Types:
* Key-value store (hstore)
* JSON (text [JSON] & binary [JSONB])
We will also have some cleverly concocted examples to show how all of these data types can work together harmoniously.
Accelerating Local Search with PostgreSQL (KNN-Search)Jonathan Katz
KNN-GiST indexes were added in PostgreSQL 9.1 and greatly accelerate some common queries in the geospatial and textual search realms. This presentation will demonstrate the power of KNN-GiST indexes on geospatial and text searching queries, but also their present limitations through some of my experimentations. I will also discuss some of the theory behind KNN (k-nearest neighbor) as well as some of the applications this feature can be applied too.
To see a version of the talk given at PostgresOpen 2011, please visit http://www.youtube.com/watch?v=N-MD08QqGEM
Webscale PostgreSQL - JSONB and Horizontal Scaling StrategiesJonathan Katz
All data is relational and can be represented through relational algebra, right? Perhaps, but there are other ways to represent data, and the PostgreSQL team continues to work on making it easier and more efficient to do so!
With the upcoming 9.4 release, PostgreSQL is introducing the "JSONB" data type which allows for fast, compressed, storage of JSON formatted data, and for quick retrieval. And JSONB comes with all the benefits of PostgreSQL, like its data durability, MVCC, and of course, access to all the other data types and features in PostgreSQL.
How fast is JSONB? How do we access data stored with this type? What can it do with the rest of PostgreSQL? What can't it do? How can we leverage this new data type and make PostgreSQL scale horizontally? Follow along with our presentation as we try to answer these questions.
PostgreSQL comes built-in with a variety of indexes, some of which are further extensible to build powerful new indexing schemes. But what are all these index types? What are some of the special features of these indexes? What are the size & performance tradeoffs? How do I know which ones are appropriate for my application?
Fortunately, this talk aims to answer all of these questions as we explore the whole family of PostgreSQL indexes: B-tree, expression, GiST (of all flavors), GIN and how they are used in theory and practice.
UiPath Community Day Amsterdam: Code, Collaborate, ConnectUiPathCommunity
Welcome to our third live UiPath Community Day Amsterdam! Come join us for a half-day of networking and UiPath Platform deep-dives, for devs and non-devs alike, in the middle of summer ☀.
📕 Agenda:
12:30 Welcome Coffee/Light Lunch ☕
13:00 Event opening speech
Ebert Knol, Managing Partner, Tacstone Technology
Jonathan Smith, UiPath MVP, RPA Lead, Ciphix
Cristina Vidu, Senior Marketing Manager, UiPath Community EMEA
Dion Mes, Principal Sales Engineer, UiPath
13:15 ASML: RPA as Tactical Automation
Tactical robotic process automation for solving short-term challenges, while establishing standard and re-usable interfaces that fit IT's long-term goals and objectives.
Yannic Suurmeijer, System Architect, ASML
13:30 PostNL: an insight into RPA at PostNL
Showcasing the solutions our automations have provided, the challenges we’ve faced, and the best practices we’ve developed to support our logistics operations.
Leonard Renne, RPA Developer, PostNL
13:45 Break (30')
14:15 Breakout Sessions: Round 1
Modern Document Understanding in the cloud platform: AI-driven UiPath Document Understanding
Mike Bos, Senior Automation Developer, Tacstone Technology
Process Orchestration: scale up and have your Robots work in harmony
Jon Smith, UiPath MVP, RPA Lead, Ciphix
UiPath Integration Service: connect applications, leverage prebuilt connectors, and set up customer connectors
Johans Brink, CTO, MvR digital workforce
15:00 Breakout Sessions: Round 2
Automation, and GenAI: practical use cases for value generation
Thomas Janssen, UiPath MVP, Senior Automation Developer, Automation Heroes
Human in the Loop/Action Center
Dion Mes, Principal Sales Engineer @UiPath
Improving development with coded workflows
Idris Janszen, Technical Consultant, Ilionx
15:45 End remarks
16:00 Community fun games, sharing knowledge, drinks, and bites 🍻
Increase Quality with User Access Policies - July 2024Peter Caitens
⭐️ Increase Quality with User Access Policies ⭐️, presented by Peter Caitens and Adam Best of Salesforce. View the slides from this session to hear all about “User Access Policies” and how they can help you onboard users faster with greater quality.
Cracking AI Black Box - Strategies for Customer-centric Enterprise ExcellenceQuentin Reul
The democratization of Generative AI is ushering in a new era of innovation for enterprises. Discover how you can harness this powerful technology to deliver unparalleled customer value and securing a formidable competitive advantage in today's competitive market. In this session, you will learn how to:
- Identify high-impact customer needs with precision
- Harness the power of large language models to address specific customer needs effectively
- Implement AI responsibly to build trust and foster strong customer relationships
Whether you're at the early stages of your AI journey or looking to optimize existing initiatives, this session will provide you with actionable insights and strategies needed to leverage AI as a powerful catalyst for customer-driven enterprise success.
"Making .NET Application Even Faster", Sergey Teplyakov.pptxFwdays
In this talk we're going to explore performance improvement lifecycle, starting with setting the performance goals, using profilers to figure out the bottle necks, making a fix and validating that the fix works by benchmarking it. The talk will be useful for novice and seasoned .NET developers and architects interested in making their application fast and understanding how things work under the hood.
How UiPath Discovery Suite supports identification of Agentic Process Automat...DianaGray10
📚 Understand the basics of the newly persona-based LLM-powered Agentic Process Automation and discover how existing UiPath Discovery Suite products like Communication Mining, Process Mining, and Task Mining can be leveraged to identify APA candidates.
Topics Covered:
💡 Idea Behind APA: Explore the innovative concept of Agentic Process Automation and its significance in modern workflows.
🔄 How APA is Different from RPA: Learn the key differences between Agentic Process Automation and Robotic Process Automation.
🚀 Discover the Advantages of APA: Uncover the unique benefits of implementing APA in your organization.
🔍 Identifying APA Candidates with UiPath Discovery Products: See how UiPath's Communication Mining, Process Mining, and Task Mining tools can help pinpoint potential APA candidates.
🔮 Discussion on Expected Future Impacts: Engage in a discussion on the potential future impacts of APA on various industries and business processes.
Enhance your knowledge on the forefront of automation technology and stay ahead with Agentic Process Automation. 🧠💼✨
Speakers:
Arun Kumar Asokan, Delivery Director (US) @ qBotica and UiPath MVP
Naveen Chatlapalli, Solution Architect @ Ashling Partners and UiPath MVP
Retrieval Augmented Generation Evaluation with RagasZilliz
Retrieval Augmented Generation (RAG) enhances chatbots by incorporating custom data in the prompt. Using large language models (LLMs) as judge has gained prominence in modern RAG systems. This talk will demo Ragas, an open-source automation tool for RAG evaluations. Christy will talk about and demo evaluating a RAG pipeline using Milvus and RAG metrics like context F1-score and answer correctness.
TrustArc Webinar - Innovating with TRUSTe Responsible AI CertificationTrustArc
In a landmark year marked by significant AI advancements, it’s vital to prioritize transparency, accountability, and respect for privacy rights with your AI innovation.
Learn how to navigate the shifting AI landscape with our innovative solution TRUSTe Responsible AI Certification, the first AI certification designed for data protection and privacy. Crafted by a team with 10,000+ privacy certifications issued, this framework integrated industry standards and laws for responsible AI governance.
This webinar will review:
- How compliance can play a role in the development and deployment of AI systems
- How to model trust and transparency across products and services
- How to save time and work smarter in understanding regulatory obligations, including AI
- How to operationalize and deploy AI governance best practices in your organization
Choosing the Best Outlook OST to PST Converter: Key Features and Considerationswebbyacad software
When looking for a good software utility to convert Outlook OST files to PST format, it is important to find one that is easy to use and has useful features. WebbyAcad OST to PST Converter Tool is a great choice because it is simple to use for anyone, whether you are tech-savvy or not. It can smoothly change your files to PST while keeping all your data safe and secure. Plus, it can handle large amounts of data and convert multiple files at once, which can save you a lot of time. It even comes with 24*7 technical support assistance and a free trial, so you can try it out before making a decision. Whether you need to recover, move, or back up your data, Webbyacad OST to PST Converter is a reliable option that gives you all the support you need to manage your Outlook data effectively.
"Building Future-Ready Apps with .NET 8 and Azure Serverless Ecosystem", Stan...Fwdays
.NET 8 brought a lot of improvements for developers and maturity to the Azure serverless container ecosystem. So, this talk will cover these changes and explain how you can apply them to your projects. Another reason for this talk is the re-invention of Serverless from a DevOps perspective as a Platform Engineering trend with Backstage and the recent Radius project from Microsoft. So now is the perfect time to look at developer productivity tooling and serverless apps from Microsoft's perspective.
Generative AI technology is a fascinating field that focuses on creating comp...Nohoax Kanont
Generative AI technology is a fascinating field that focuses on creating computer models capable of generating new, original content. It leverages the power of large language models, neural networks, and machine learning to produce content that can mimic human creativity. This technology has seen a surge in innovation and adoption since the introduction of ChatGPT in 2022, leading to significant productivity benefits across various industries. With its ability to generate text, images, video, and audio, generative AI is transforming how we interact with technology and the types of tasks that can be automated.
Self-Healing Test Automation Framework - HealeniumKnoldus Inc.
Revolutionize your test automation with Healenium's self-healing framework. Automate test maintenance, reduce flakes, and increase efficiency. Learn how to build a robust test automation foundation. Discover the power of self-healing tests. Transform your testing experience.
The History of Embeddings & Multimodal EmbeddingsZilliz
Frank Liu will walk through the history of embeddings and how we got to the cool embedding models used today. He'll end with a demo on how multimodal RAG is used.
5. • Director of Cloud Engineering & Growth,
Crunchy Data
• Previously: Engineering leadership in startups
• Longtime PostgreSQL community contributor
• Advocacy & various committees for PGDG
• @postgresql + .org content
• Director, PgUS
• Conference organization + speaking
• @jkatz05
About Me
5
6. Market Leading
Data Security
Crunchy Data is the leader in
PostgreSQL security. Common
Criteria certification and essential
security enhancements make Crunchy
Certified PostgreSQL the trusted open
source PostgreSQL distribution for
the enterprise.
Cloud Ready
Data Management
Whether deploying to public or
private clouds, Crunchy Data
provides market leading, open
source, Kubernetes-based
technology solutions, giving your
team the choice and flexibility for how
you deploy your data.
Leader in Open Source
Enterprise PostgreSQL
Crunchy Data gives organizations the
technology, support, and confidence to
enjoy the power and efficiency of open
source PostgreSQL.
7. • You’ll understand the evolution of how password management in
PostgreSQL
• If you use PostgreSQL’s password mechanisms in production, you should
want to start using SCRAM-SHA-256
• You will know how to upgrade your systems to use SCRAM-SHA-256
By The End of This Talk
7
13. • It’s very easy to take password management for granted when it just
works
• Before version 10, offered two methods of storing passwords and
validating passwords
• plain
• crypt
• md5
How Do Passwords Work in PostgreSQL?
13
15. Meet grayhippo
15
This is "grayhippo"
grayhippo loves PostgreSQL,
and loves storing large
amounts of data.
grayhippo's password is
"datalake"
16. Meet redhippo
16
This is "redhippo"
redhippo also loves
PostgreSQL. redhippo likes
to play pretend, and in
particular, likes to pretend to
be grayhippo.
17. • Stored in plain text
• Communicated in plain text
• Method was kept available while drivers
were updating to “md5” method.
• Dropped in PostgreSQL 10*
• “Plaintext” validation is still requested for some
methods, such as LDAP
PostgreSQL Plaintext Passwords
17
password
31. • Supersedes Securely Socket Layers (SSL)
• PostgreSQL refers to “SSL” everywhere in the configuration
• Performs a “secure handshake” between two parties and encrypts all
traffic between them
• Use the “hostssl” authentication type to require TLS connections
• Can be used as a method of authentication in PostgreSQL…but that’s a
different talk
• (PostgreSQL 12: clientert=verify-full – allows for two-factor authentication using
certificates + another authentication method!)
Transport Layer Security
31
37. 37
--
-- Roles
--
CREATE ROLE grayhippo;ALTER ROLE grayhippo WITH NOSUPERUSER
INHERIT NOCREATEROLE NOCREATEDB LOGIN NOREPLICATION
NOBYPASSRLS PASSWORD 'datalake';
…and if we inspect the file
40. • Derived by computing md5 hash of the
combination of the plaintext password
and the username
• Stored in hexadecimal format with "md5"
prepended to it
• Communicated using the PostgreSQL
"md5 protocol"
• PostgreSQL server sends a 4 byte random
salt
• Connecting client computes md5 hash of
password + username, then appends the 4
byte salt and computes another md5 hash
PostgreSQL MD5 Password Authentication
40
"md5" +
md5(password + username)
56. 56
--
-- Roles
--
CREATE ROLE grayhippo;ALTER ROLE grayhippo WITH NOSUPERUSER
INHERIT NOCREATEROLE NOCREATEDB LOGIN NOREPLICATION
NOBYPASSRLS PASSWORD 'md55133cd3d6af5b408fa0bd7fbbde1dedd';
…and if we inspect the file
63. • CREATE ROLE grayhippo PASSWORD 'datalake';
• If redhippo can read the server logs, redhippos now has the plain text password
• Alternatives
• Via psql: Use password
• Via other: pre-hash the password with the formula: "md5" + md5(password + username)
• …but both of these methods still log the md5 hashed password to the server logs
• SELECT rolname, rolpassword FROM pg_authid;
• Requires a privileged user, but if redhippo has said privileges…
Other Ways redhippo Can Get the Password
63
65. • Defined in RFC 5802
• Part of the Simple Authentication and Security Layer (SASL) family (RFC 4422)
• PostgreSQL uses RFC7767 which specifies the use of the SHA-256 hashing algorithm
• Designed to be easy to implement and use all standard, accepted methods
• SHA-256 (RFC 6234)
• HMAC (RFC 2014)
• SASLprep (RFC 4013), based on stringprep (RFC 3434)
• …and we'll talk about how PostgreSQL does this
• Allows for two parties to verify they both know a secret without
exchanging the secret
Salted Challenge Response Authentication Mechanism
65
66. Two Parties Can Verify They
Both Know a Secret Without
Exchanging the Secret
68. • In PostgreSQL need to build a "SCRAM verifier"
• (Might have just been changed to "SCRAM secret" in PostgreSQL 13?)
• A client can build the SCRAM verifier and transmit it to the server.
• Even in this form, an eavesdropper will be unable to access the secret
Creating a Password For SCRAM
68
<DIGEST>$<ITERATIONS>:<SALT>$<STORED_KEY>:<SERVER_KEY>
69. • PostgreSQL 10 introduced support for "SCRAM-SHA-256"
• PostgreSQL 11 introduced support for "SCRAM-SHA-256-PLUS", which is
used for channel binding (more on that later)
Building a SCRAM Secret - DIGEST
69
"SCRAM-SHA-256"
70. • Iterations is the number of times the HMAC signature is computed with
hashing function (SCRAM-SHA-256)
• We'll start talking about the full computation of this in two slides
• By default, PostgreSQL uses 4096 iterations. If you built your own secret,
you can customize the number of iterations.
Building a SCRAM Secret - ITERATIONS
70
"4096"
71. • A cryptographically secure randomly generated value
• PostgreSQL defaults to 16 bytes
• Ultimately stored in base64 representation
Building a SCRAM Secret - SALT
71
"s+1VLTv5oCfNEymVKi01Fw=="
72. • Before creating the hashed password, it needs to be normalized using SASLPrep*
• *SASLprep is for UTF-8 encoded strings, but PostgreSQL accepts multiple encodings…so we
perform a modified SASLprep
• If password is ASCII or is not UTF-8 encoded, this step is skipped
• 4 Steps:
• Step 1: Remove non-ASCII space characters. If password ends up being empty, then return original
password
• Step 2: Normalize the password using NFKC form. If empty, then return original password
• Step 3: Check for prohibited characters. If any are detected, then return original password
• Step 4: Bi-directional characters. If any prohibited Bi-directional characters detected, return original
password. If there is a "RandALCat" character and either a) at least one "LCat" character OR b) does
not end with a "RandALCat" character, return original password
• If we get through that, return normalized password!
Building a SCRAM Secret – SASLPrep the Password
72
It's still
"datalake"
73. • The first iteration for generating the salted password uses the following
formula:
• HMAC using SHA-256 with "password" as the key signing a message with "salt"
concatenated with a 32-bit value with final bit set to 1
• Store this value as an aggregator
• For the remaining iterations:
• Calculate the HMAC using SHA-256 with "password" as the key signing a message
that is composed of HMAC calculated from the previous iteration
• XOR this HMAC with the aggregator (aggregator = aggregator XOR HMAC)
Building a SCRAM Secret – Generate the Salted Password
73
In base64 for this example, the value is:
"oJZnL+tf7yE1QWp5fykec7xEgw1cwfyr5Jh3SudbWio="
74. • The SHA-256 hash of the "Client Key", stored in a base64 representation
• …the "Client Key" is a HMAC using the salted password as the key and
"Client Key" as the message
Building a SCRAM Secret – STORED KEY
74
"z+tqsnBaCmgSJQBLf1cPOmq7n80PW3aRjzeuAGkkuq4="
76. • A HMAC using the salted password as the key and "Server Key" as the
message
Building a SCRAM Secret – SERVER KEY
76
"9ZWHsv+XZAmrpkiEnDPm05wg55jLRrITWspTqsz1zD4="
78. • You can just use "password" from psql
• If your PostgreSQL driver supports it, its password creation facilities
• It may delegate it to libpq, which has said facilities
Building a SCRAM Secret – "Easy Button"
78
That sounds
much simpler…
79. Send the SCRAM Secret to PostgreSQL
79
ALTER ROLE grayhippo
PASSWORD 'SCRAM-SHA-
256$4096:s+1VLTv5oCfNEymVKi01Fw==$z+tqsnBaCmgSJQBLf1cPOmq7n
80PW3aRjzeuAGkkuq4=:9ZWHsv+XZAmrpkiEnDPm05wg55jLRrITWspTqsz
1zD4=';
(We'll let redhippo listen)
80. • Identifying information sent:
• Stored Key: A hash of a HMAC using the computed salted password and "Client Key"
• Server Key: A HMAC using the computed salted password and "Server Key"
• In other words, we did not send the original secret in any recognizable form
• redhippo can still try to brute force the password or try other offline
attacks, but with a high number of iterations (and good password
selection), this is hard.
We Don't Care That redhippo Listens?
80
Just wait until my
quantum computer
boots up…
82. • The goal of SCRAM authentication is for two parties to verify that each
ones knows the shared secret
• To prevent against replay attacks, one-time nonces are used for the
session being authenticated
SCRAM Authentication
82
Just to save space,
pretend I'm listening
on the next few
slides
86. SCRAM Authentication Flow
86
Cool. Here is my nonce added to your nonce:
"FVazuC8Hjl46XPPCs2L9RFhqMxMjEzh5txq6gZv6iuW813Aa"
Here is the number of iterations to use to computed a salted password: "4096"
Here is the salt for you to use: "s+1VLTv5oCfNEymVKi01Fw=="
Send me proof that you know the password.
88. SCRAM Authentication Flow: Generating Proof
88
My password is "datalake" so from that I am going to computed
the salted password using the "SHA-256" hashing method, the
"s+1VLTv5oCfNEymVKi01Fw==" salt, and applying the method
for "4096" iterations.
Basically, like generating the password.
Which was
"oJZnL+tf7yE1QWp5fykec7xEgw1cwfyr5Jh3SudbWio="
in base64.
91. SCRAM Authentication Flow: Generating Proof
91
From that I can derive the client key (see password generation)
which is
"eq9WtJSH7PLTxsdv8SJQqJA4+6SU3f03nWE1TW4Ppqg="
And the stored key which is
"z+tqsnBaCmgSJQBLf1cPOmq7n80PW3aRjzeuAGkkuq4="
I can create the client signature by taking a HMAC using the SHA-
256 hashing method with the stored key and the message
containing information from the authorization sessions headers
93. SCRAM Authentication Flow: Generating Proof
93
And I can generate the client proof
by calculating
client key XOR client signature
94. SCRAM Authentication Flow: Sending Proof
94
I'm sending you the client proof that I built up. It's:
"MhvH1xAaVVhUrp4d5TCyjIiMsmlNQQAFTgI57AksP7g="
Also, here is our combined nonce.
95. SCRAM Authentication Flow: Server Verification
95
Okay, I can compute the client signature
because I know the stored key and I have
the authentication session info, too.
I can get what you think the client key by
calculating
client proof XOR client signature
If the SHA-256 digest of your client key
matches the stored key, then I know you
know the password!
96. SCRAM Authentication Flow: Server Verification
96
But…I still need to prove to you that I know
the password.
I'm going to send you my server signature
which is an HMAC using the SHA-256
method using my stored server key and
the authorization session info as the
message.
100. SCRAM Authentication Flow: Client Verification
100
First let me generate the server key. This is just like
generating the SCRAM secret, and I have the salted
password available.
I can then generate an HMAC using the SHA-256
method and the server key and the authorization
session info as the message.
If this value matches your server signature, then I
know you know my password!
102. • Identifying information sent:
• Stored Key: A hash of the HMAC using the computed salted password and "Client
Key"
• Server Key: A HMAC using the computed salted password and "Server Key"
• A combined client/server nonce that can only be used once for this session
• Information about the session from the authorization headers
Recall What We Transmitted
102
Grr…
106. • Recall: As the final step of SCRAM, the server sends a server signature to
the client.
• The client is authenticated, but what if the server is not who you think it is?
Case #1: Server "Claims" To Know Secret
106
Here's my server
signature, you're
authenticated now!
107. Case #1: Server "Claims" To Know Secret
107
Awesome, here is all of
my secret information
108. • This is why the client must also verify the server, to ensure the server
actually knows the client's password.
Case #1: Server "Claims" To Know Secret
108
Thanks! I'm going to go
sell this for a lot of
money and buy up so
many fruits and
vegatables!
109. • Recall: We can use TLS to secure the connection endpoints between the
client and server
• But…how do we know that the server sending the signature is the same
instance we originally connected to?
Case #2: Elephant-in-the-Middle Attack
109
TLS
110. Case #2: Elephant-in-the-Middle Attack
110
TLS
OK. I choose "SCRAM-
SHA-256".
Here is a nonce I generated:
Hmm…I want to get
in on this…
111. Case #2: Elephant-in-the-Middle Attack
111
TLS
OK, here's a bunch of stuff that
looks normal because your
connection is already of a secure
socket so this must appear to be
normal…nothing weird going on
here…
113. • Introduced in PostgreSQL 11, channel binding ensures that the TLS
handshake is still the same when the client and server are identifying each
other
• Specified for SCRAM in RFC 5802, TLS channel bindings in RFC 5929
• PostgreSQL uses "tls-server-end-point", which uses ahash of the server certificate to bind
the channels
• A server that supports channel binding advertises it as "SCRAM-SHA-256-
PLUS"
• A client then requests to use channel binding (and in PostgreSQL 13
require to use channel binding!)
Channel Binding
113
114. Channel Binding
114
TLS
OK. I choose "SCRAM-SHA-256-PLUS".
Here is a channel binding header with
your certificate. And the nonce
Oh no…I won't be
able to prove that I
can validate that
certificate…
116. • As of this presentation, all client drivers and libraries support SCRAM
except for the Swift driver and pgpool
• https://wiki.postgresql.org/wiki/List_of_drivers#Drivers
• In postgresql.conf set password_encryption to scram-sha-256
• Keep md5 as your authentication method in pg_hba.conf until all your
users have re-hashed their passwords
• ...have your users re-hash their passwords. Easiest way is password
• Once all of your users have re-hashed their password, go back to
pg_hba.conf and switch your authentication method to scram-sha-256
Upgrading to SCRAM
116
117. • As of this presentation, all client drivers and libraries support SCRAM
except for the Swift driver
https://wiki.postgresql.org/wiki/List_of_drivers#Drivers
Driver Support for SCRAM
117
119. • It's good to question "it just works" as this can lead to better solutions
• Beyond PostgreSQL, SCRAM is a general purpose solution. Consider
implementing it in your other applications where you need to verify secrets
• Example PostgreSQL Password Creator:
https://gist.github.com/jkatz/e0a1f52f66fa03b732945f6eb94d9c21
• Example (poorly written) SCRAM implementation with PostgreSQL:
https://gist.github.com/jkatz/7444eda78a6fff18ab5d74c024e3761d
• …oh, upgrade your passwords to use SCRAM. Now.
• Also look into using "clientcert=verify-full" if you're running PostgreSQL 12 – two
factor authentication!
Not All Password Methods Are the Same
119