The document discusses how the PostgreSQL query planner works. It explains that a query goes through several stages including parsing, rewriting, planning/optimizing, and execution. The optimizer or planner has to estimate things like the number of rows and cost to determine the most efficient query plan. Statistics collected by ANALYZE are used for these estimates but can sometimes be inaccurate, especially for n_distinct values. Increasing the default_statistics_target or overriding statistics on columns can help address underestimation issues. The document also discusses different plan types like joins, scans, and aggregates that the planner may choose between.
The document provides an overview of PostgreSQL performance tuning. It discusses caching, query processing internals, and optimization of storage and memory usage. Specific topics covered include the PostgreSQL configuration parameters for tuning shared buffers, work memory, and free space map settings.
How to use Parquet as a basis for ETL and analyticsJulien Le Dem
Parquet is a columnar format designed to be extremely efficient and interoperable across the hadoop ecosystem. Its integration in most of the Hadoop processing frameworks (Impala, Hive, Pig, Cascading, Crunch, Scalding, Spark, …) and serialization models (Thrift, Avro, Protocol Buffers, …) makes it easy to use in existing ETL and processing pipelines, while giving flexibility of choice on the query engine (whether in Java or C++). In this talk, we will describe how one can us Parquet with a wide variety of data analysis tools like Spark, Impala, Pig, Hive, and Cascading to create powerful, efficient data analysis pipelines. Data management is simplified as the format is self describing and handles schema evolution. Support for nested structures enables more natural modeling of data for Hadoop compared to flat representations that create the need for often costly joins.
How a Developer can Troubleshoot a SQL performing poorly on a Production DBCarlos Sierra
This session is about some free small scripts you can execute from SQL*Plus, which provide you with some basics about a SQL statement, like the Execution Plan from multiple child cursors, their SQL Monitor report if your site has the Tuning Pack, and some useful Active Session History (ASH) summaries for your SQL if your site has the Diagnostics Pack. And if you have neither the Tuning nor the Diagnostics Pack then you may want to learn about some alternatives to collect important performance metrics.
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013Jaime Crespo
Tutorial delivered at Percona MySQL Conference Live London 2013.
It doesn't matter what new SSD technologies appear, or what are the latest breakthroughs in flushing algorithms: the number one cause for MySQL applications being slow is poor execution plan of SQL queries. While the latest GA version provided a huge amount of transparent optimizations -specially for JOINS and subqueries- it is still the developer's responsibility to take advantage of all new MySQL 5.6 features.
In this tutorial we will propose the attendants a sample PHP application with bad response time. Through practical examples, we will suggest step-by-step strategies to improve its performance, including:
* Checking MySQL & InnoDB configuration
* Internal (performance_schema) and external tools for profiling (pt-query-digest)
* New EXPLAIN tools
* Simple and multiple column indexing
* Covering index technique
* Index condition pushdown
* Batch key access
* Subquery optimization
The columnar roadmap: Apache Parquet and Apache ArrowDataWorks Summit
The Hadoop ecosystem has standardized on columnar formats—Apache Parquet for on-disk storage and Apache Arrow for in-memory. With this trend, deep integration with columnar formats is a key differentiator for big data technologies. Vertical integration from storage to execution greatly improves the latency of accessing data by pushing projections and filters to the storage layer, reducing time spent in IO reading from disk, as well as CPU time spent decompressing and decoding. Standards like Arrow and Parquet make this integration even more valuable as data can now cross system boundaries without incurring costly translation. Cross-system programming using languages such as Spark, Python, or SQL can becomes as fast as native internal performance.
In this talk we’ll explain how Parquet is improving at the storage level, with metadata and statistics that will facilitate more optimizations in query engines in the future. We’ll detail how the new vectorized reader from Parquet to Arrow enables much faster reads by removing abstractions as well as several future improvements. We will also discuss how standard Arrow-based APIs pave the way to breaking the silos of big data. One example is Arrow-based universal function libraries that can be written in any language (Java, Scala, C++, Python, R, ...) and will be usable in any big data system (Spark, Impala, Presto, Drill). Another is a standard data access API with projection and predicate push downs, which will greatly simplify data access optimizations across the board.
Speaker
Julien Le Dem, Principal Engineer, WeWork
The landscape for storing your big data is quite complex, with several competing formats and different implementations of each format. Understanding your use of the data is critical for picking the format. Depending on your use case, the different formats perform very differently. Although you can use a hammer to drive a screw, it isn’t fast or easy to do so.
The use cases that we’ve examined are:
* reading all of the columns
* reading a few of the columns
* filtering using a filter predicate
* writing the data
Furthermore, different kinds of data have distinct properties. We've used three real schemas:
* the NYC taxi data http://tinyurl.com/nyc-taxi-analysis
* the Github access logs http://githubarchive.org
* a typical sales fact table with generated data
Finally, the value of having open source benchmarks that are available to all interested parties is hugely important and all of the code is available from Apache.
This document discusses PostgreSQL statistics and how to use them effectively. It provides an overview of various PostgreSQL statistics sources like views, functions and third-party tools. It then demonstrates how to analyze specific statistics like those for databases, tables, indexes, replication and query activity to identify anomalies, optimize performance and troubleshoot issues.
- The document discusses advanced techniques for optimizing MySQL queries, including topics like temporary tables, file sorting, order optimizations, and calculated fields.
- It provides examples of using indexes and index optimizations, explaining concepts like index types, index usage, key lengths, and covering indexes.
- One example shows how to optimize a query involving a calculated year() expression by rewriting the query to use a range on the date field instead.
Apache phoenix: Past, Present and Future of SQL over HBAseenissoz
HBase as the NoSQL database of choice in the Hadoop ecosystem has already been proven itself in scale and in many mission critical workloads in hundreds of companies. Phoenix as the SQL layer on top of HBase, has been increasingly becoming the tool of choice as the perfect complementary for HBase. Phoenix is now being used more and more for super low latency querying and fast analytics across a large number of users in production deployments. In this talk, we will cover what makes Phoenix attractive among current and prospective HBase users, like SQL support, JDBC, data modeling, secondary indexing, UDFs, and also go over recent improvements like Query Server, ODBC drivers, ACID transactions, Spark integration, etc. We will conclude by looking into items in the pipeline and how Phoenix and HBase interacts with other engines like Hive and Spark.
First Normal Form (1NF) Second Normal Form (2NF) Third Normal Form (3NF) Boyce-Codd Normal Form (BCNF) Boyce-Codd Normal Form (BCNF) Fourth Normal Form (4NF) Fifth Normal Form (5NF)
The document discusses compaction in RocksDB, an embedded key-value storage engine. It describes the two compaction styles in RocksDB: level style compaction and universal style compaction. Level style compaction stores data in multiple levels and performs compactions by merging files from lower to higher levels. Universal style compaction keeps all files in level 0 and performs compactions by merging adjacent files in time order. The document provides details on the compaction process and configuration options for both styles.
Database tuning is the process of optimizing a database to maximize performance. It involves activities like configuring disks, tuning SQL statements, and sizing memory properly. Database performance issues commonly stem from slow physical I/O, excessive CPU usage, or latch contention. Tuning opportunities exist at the level of database design, application code, memory settings, disk I/O, and eliminating contention. Performance monitoring tools like the Automatic Workload Repository and wait events help identify problem areas.
Apache Arrow Flight: A New Gold Standard for Data TransportWes McKinney
This document discusses how structured data is often moved inefficiently between systems, causing waste. It introduces Apache Arrow, an open standard for in-memory data, and how Arrow can help make data movement more efficient. Systems like Snowflake and BigQuery are now using Arrow to help speed up query result fetching by enabling zero-copy data transfers and sharing file formats between query processing and storage.
How to use histograms to get better performanceMariaDB plc
Sergei Petrunia and Varun Gupta, software engineers MariaDB, show how histograms can be used to improve query performance. They begin by introducing histrograms and explaining why they’re needed by the query optimizer. Next, they discuss how to determine whether or not histrograms are needed, and if so, how to determine which tables and columns they should be applied. Finally, they cover best practices and recent improvements to histograms.
The document discusses various Oracle performance monitoring tools including Oracle Enterprise Manager (OEM), Automatic Workload Repository (AWR), Automatic Database Diagnostic Monitor (ADDM), Active Session History (ASH), and eDB360. It provides overviews of each tool and examples of using AWR, ADDM, ASH and eDB360 for performance analysis through demos. The conclusions recommend OEM as the primary tool and how the other tools like AWR, ADDM and ASH complement it for deeper performance insights.
Parquet is an open-source columnar storage format that provides an efficient data layout for analytical queries. Twitter uses Parquet to store logs and analytics data across multiple large Hadoop clusters, saving petabytes of storage and reducing query times by up to 66% by reading only needed columns. Parquet defines a language-independent file format that stores data by column rather than row to optimize analytical access patterns.
This document provides a summary of improvements made to Hive's performance through the use of Apache Tez and other optimizations. Some key points include:
- Hive was improved to use Apache Tez as its execution engine instead of MapReduce, reducing latency for interactive queries and improving throughput for batch queries.
- Statistics collection was optimized to gather column-level statistics from ORC file footers, speeding up statistics gathering.
- The cost-based optimizer Optiq was added to Hive, allowing it to choose better execution plans.
- Vectorized query processing, broadcast joins, dynamic partitioning, and other optimizations improved individual query performance by over 100x in some cases.
A Practical Introduction to Handling Log Data in ClickHouse, by Robert Hodges...Altinity Ltd
This document discusses using ClickHouse to manage log data. It begins with an introduction to ClickHouse and its features. It then covers different ways to model log data in ClickHouse, including storing logs as JSON blobs or converting them to a tabular format. The document demonstrates using materialized views to ingest logs into ClickHouse tables in an efficient manner, extracting values from JSON and converting to columns. It shows how this approach allows flexible querying of log data while scaling to large volumes.
Structuring Spark: DataFrames, Datasets, and Streaming by Michael ArmbrustSpark Summit
This document summarizes Spark's structured APIs including SQL, DataFrames, and Datasets. It discusses how structuring computation in Spark enables optimizations by limiting what can be expressed. The structured APIs provide type safety, avoid errors, and share an optimization and execution pipeline. Functions allow expressing complex logic on columns. Encoders map between objects and Spark's internal data format. Structured streaming provides a high-level API to continuously query streaming data similar to batch queries.
InfluxDB IOx Tech Talks: Query Engine Design and the Rust-Based DataFusion in...InfluxData
The document discusses updates to InfluxDB IOx, a new columnar time series database. It covers changes and improvements to the API, CLI, query capabilities, and path to open sourcing builds. Key points include moving to gRPC for management, adding PostgreSQL string functions to queries, optimizing functions for scalar values and columns, and monitoring internal systems as the first step to releasing open source builds.
This document discusses PostgreSQL query optimization techniques. It covers identifying slow queries, understanding query plans, and provides examples of optimizations like adding indexes and changing query structures. The key steps are finding queries to optimize using tools like EXPLAIN and pg_stat_statements, analyzing queries and plans to understand performance bottlenecks, and then making changes like creating indexes, restructuring queries, and adjusting configuration settings to improve performance.
The document discusses various techniques for optimizing query performance in MySQL, including using indexes appropriately, avoiding full table scans, and tools like EXPLAIN, Performance Schema, and pt-query-digest for analyzing queries and identifying optimization opportunities. It provides recommendations for index usage, covering indexes, sorting and joins, and analyzing slow queries.
Query Optimization with MySQL 5.6: Old and New TricksMYXPLAIN
The document discusses query optimization techniques for MySQL 5.6, including both established techniques and new features in 5.6. It provides an overview of tools for profiling queries such as EXPLAIN, the slow query log, and the performance schema. It also covers indexing strategies like compound indexes and index condition pushdown.
This document discusses Parallel Query, a feature of POLARDB for MySQL that allows queries to run in parallel across multiple CPU cores for improved performance. It begins with an introduction to Parallel Query and how it works, then discusses how to use Parallel Query, how it is implemented internally, examples of performance improvements seen, and some current limitations and plans for future work.
Introduction to MySQL Query Tuning for Dev[Op]sSveta Smirnova
To get data, we query the database. MySQL does its best to return requested bytes as fast as possible. However, it needs human help to identify what is important and should be accessed in the first place.
Queries, written smartly, can significantly outperform automatically generated ones. Indexes and Optimizer statistics, not limited to the Histograms only, help to increase the speed of the query a lot.
In this session, I will demonstrate by examples of how MySQL query performance can be improved. I will focus on techniques, accessible by Developers and DevOps rather on those which are usually used by Database Administrators. In the end, I will present troubleshooting tools which will help you to identify why your queries do not perform. Then you could use the knowledge from the beginning of the session to improve them.
In Memory Database In Action by Tanel Poder and Kerry OsborneEnkitec
The document discusses Oracle Database In-Memory option and how it improves performance of data retrieval and processing queries. It provides examples of running a simple aggregation query with and without various performance features like In-Memory, vector processing and bloom filters enabled. Enabling these features reduces query elapsed time from 17 seconds to just 3 seconds by minimizing disk I/O and leveraging CPU optimizations like SIMD vector processing.
Oracle Database In-Memory Option in ActionTanel Poder
The document discusses Oracle Database In-Memory option and how it improves performance of data retrieval and processing queries. It provides examples of running a simple aggregation query with and without various performance features like In-Memory, vector processing and bloom filters enabled. Enabling these features reduces query elapsed time from 17 seconds to just 3 seconds by minimizing disk I/O and leveraging CPU optimizations like SIMD vector processing.
This document discusses third party patches for MySQL that provide quick wins and new features. It summarizes five such patches: 1) Slow query filtering which helps identify expensive queries, 2) Index statistics which helps determine unused indexes, 3) An InnoDB dictionary limit which constrains memory usage, 4) A global long query time setting, and 5) A "fix" for InnoDB group commit performance regressions in MySQL 5.0. The document encourages using third party patches to gain features and improvements not yet available in the MySQL core.
Managing Statistics for Optimal Query PerformanceKaren Morton
Half the battle of writing good SQL is in understanding how the Oracle query optimizer analyzes your code and applies statistics in order to derive the “best” execution plan. The other half of the battle is successfully applying that knowledge to the databases that you manage. The optimizer uses statistics as input to develop query execution plans, and so these statistics are the foundation of good plans. If the statistics supplied aren’t representative of your actual data, you can expect bad plans. However, if the statistics are representative of your data, then the optimizer will probably choose an optimal plan.
The document discusses strategies for optimizing queries by shaping the optimizer's search space. It recommends:
1. Maximizing data locality by using basic B-tree indexes rather than more complex options like partitions or clusters.
2. Writing queries to explicitly exploit indexes by using range conditions, ordering results to match the index order, and terminating scans after a specified number of rows.
3. Ordering columns in multi-column indexes to match the predicates in common queries, with equality conditions before range conditions.
How the query planner in PostgreSQL works? Index access methods, join execution types, aggregation & pipelining. Optimizing queries with WHERE conditions, ORDER BY and GROUP BY. Composite indexes, partial and expression indexes. Exploiting assumptions about data and denormalization.
Matt Smiley
This is a basic primer aimed primarily at developers or DBAs new to Postgres. The format is a Q/A style tour with examples, based on common questions and pitfalls. Begin with a quick tour of relevant parts of the postgres catalog, with an aim to answer simple but important questions like:
How many rows does the optimizer think my table has?
When was it last analyzed?
Which other tables also have a column named "foo"?
How often is this index used?
Basic Query Tuning Primer - Pg West 2009mattsmiley
Intro to query tuning in Postgres, for beginners or intermediate software developers. Lists your basic toolkit, common problems, a series of examples. Assumes the audience knows basic SQL but has little or no experience with reading or adjusting execution plans. Accompanies 45-90 minute talk; meant to encourage Q/A.
John Melesky - Federating Queries Using Postgres FDW @ Postgres OpenPostgresOpen
This document discusses federating queries across PostgreSQL databases using foreign data wrappers (FDWs). It begins by introducing the author and their background. It then covers using FDWs to partition tables across multiple nodes for queries, the benefits over traditional views, and demonstrates counting rows across nodes. It notes limitations like network overhead, lack of keys/constraints, and single-threaded execution. Finally, it discusses strategies like using many small nodes, node-level partitioning, distributed processing, and multi-headed setups to optimize federated querying.
Percona Live 2016 (https://www.percona.com/live/data-performance-conference-2016/sessions/why-use-explain-formatjson). Although EXPLAIN FORMAT=JSON was first presented a long time ago, there still aren't many resources that explain how and why to use it. The most advertised feature is visual EXPLAIN in MySQL Workbench, but this format can do more than create nice pictures. It prints additional information that can't be found in good old tabular EXPLAIN, and can help to solve many tricky performance issues. In this session, I will not only describe which additional information we can get with the new syntax, but also provide examples showing how to use it to diagnose production issues.
The document discusses Oracle database performance tuning. It covers identifying and resolving performance issues through tools like AWR and ASH reports. Common causes of performance problems include wait events, old statistics, incorrect execution plans, and I/O issues. The document recommends collecting specific data when analyzing problems and provides references and scripts for further tuning tasks.
Talk at "Istanbul Tech Talks" in Istanbul, April, 17, 2018. http://www.istanbultechtalks.com/
In this talk I will show how to get started with MySQL Query Tuning. I will make short introduction into physical table structure and demonstrate how it may influence query execution time. Then we will discuss basic query tuning instruments and techniques, mainly EXPLAIN command with its latest variations. You will learn how to understand its output and how to rewrite query or change table structure to achieve better performance.
Performance Schema for MySQL TroubleshootingSveta Smirnova
The Performance Schema provides detailed information for troubleshooting and optimizing MySQL. It collects instrumentation data on server operations, statements, memory usage, locks and connections. The data can be used to identify slow queries, statements not using indexes, memory consumption trends over time, and more. Configuration and enabling specific instruments allows controlling the level of detail collected.
Similar to PostgreSQL query planner's internals (20)
Welcome to Cyberbiosecurity. Because regular cybersecurity wasn't complicated...Snarky Security
How wonderful it is that in our modern age, every bit of our biological data can be digitized, stored, and potentially pilfered by cyber thieves! Isn't it just splendid to think that while scientists are busy pushing the boundaries of biotechnology, hackers could be plotting the next big bio-data heist? This delightful scenario is brought to you by the ever-expanding digital landscape of biology and biotechnology, where the integration of computer science, engineering, and data science transforms our understanding and manipulation of biological systems.
While the fusion of technology and biology offers immense benefits, it also necessitates a careful consideration of the ethical, security, and associated social implications. But let's be honest, in the grand scheme of things, what's a little risk compared to potential scientific achievements? After all, progress in biotechnology waits for no one, and we're just along for the ride in this thrilling, slightly terrifying, adventure.
So, as we continue to navigate this complex landscape, let's not forget the importance of robust data protection measures and collaborative international efforts to safeguard sensitive biological information. After all, what could possibly go wrong?
-------------------------
This document provides a comprehensive analysis of the security implications biological data use. The analysis explores various aspects of biological data security, including the vulnerabilities associated with data access, the potential for misuse by state and non-state actors, and the implications for national and transnational security. Key aspects considered include the impact of technological advancements on data security, the role of international policies in data governance, and the strategies for mitigating risks associated with unauthorized data access.
This view offers valuable insights for security professionals, policymakers, and industry leaders across various sectors, highlighting the importance of robust data protection measures and collaborative international efforts to safeguard sensitive biological information. The analysis serves as a crucial resource for understanding the complex dynamics at the intersection of biotechnology and security, providing actionable recommendations to enhance biosecurity in an digital and interconnected world.
The evolving landscape of biology and biotechnology, significantly influenced by advancements in computer science, engineering, and data science, is reshaping our understanding and manipulation of biological systems. The integration of these disciplines has led to the development of fields such as computational biology and synthetic biology, which utilize computational power and engineering principles to solve complex biological problems and innovate new biotechnological applications. This interdisciplinary approach has not only accelerated research and development but also introduced new capabilities such as gene editing and biomanufact
Redefining Cybersecurity with AI CapabilitiesPriyanka Aash
In this comprehensive overview of Cisco's latest innovations in cybersecurity, the focus is squarely on resilience and adaptation in the face of evolving threats. The discussion covers the imperative of tackling Mal information, the increasing sophistication of insider attacks, and the expanding attack surfaces in a hybrid work environment. Emphasizing a shift towards integrated platforms over fragmented tools, Cisco introduces its Security Cloud, designed to provide end-to-end visibility and robust protection across user interactions, cloud environments, and breaches. AI emerges as a pivotal tool, from enhancing user experiences to predicting and defending against cyber threats. The blog underscores Cisco's commitment to simplifying security stacks while ensuring efficacy and economic feasibility, making a compelling case for their platform approach in safeguarding digital landscapes.
TrustArc Webinar - Innovating with TRUSTe Responsible AI CertificationTrustArc
In a landmark year marked by significant AI advancements, it’s vital to prioritize transparency, accountability, and respect for privacy rights with your AI innovation.
Learn how to navigate the shifting AI landscape with our innovative solution TRUSTe Responsible AI Certification, the first AI certification designed for data protection and privacy. Crafted by a team with 10,000+ privacy certifications issued, this framework integrated industry standards and laws for responsible AI governance.
This webinar will review:
- How compliance can play a role in the development and deployment of AI systems
- How to model trust and transparency across products and services
- How to save time and work smarter in understanding regulatory obligations, including AI
- How to operationalize and deploy AI governance best practices in your organization
Keynote : Presentation on SASE TechnologyPriyanka Aash
Secure Access Service Edge (SASE) solutions are revolutionizing enterprise networks by integrating SD-WAN with comprehensive security services. Traditionally, enterprises managed multiple point solutions for network and security needs, leading to complexity and resource-intensive operations. SASE, as defined by Gartner, consolidates these functions into a unified cloud-based service, offering SD-WAN capabilities alongside advanced security features like secure web gateways, CASB, and remote browser isolation. This convergence not only simplifies management but also enhances security posture and application performance across global networks and cloud environments. Discover how adopting SASE can streamline operations and fortify your enterprise's digital transformation strategy.
Garbage In, Garbage Out: Why poor data curation is killing your AI models (an...Zilliz
Enterprises have traditionally prioritized data quantity, assuming more is better for AI performance. However, a new reality is setting in: high-quality data, not just volume, is the key. This shift exposes a critical gap – many organizations struggle to understand their existing data and lack effective curation strategies and tools. This talk dives into these data challenges and explores the methods of automating data curation.
Increase Quality with User Access Policies - July 2024Peter Caitens
���️ Increase Quality with User Access Policies ⭐️, presented by Peter Caitens and Adam Best of Salesforce. View the slides from this session to hear all about “User Access Policies” and how they can help you onboard users faster with greater quality.
Discovery Series - Zero to Hero - Task Mining Session 1DianaGray10
This session is focused on providing you with an introduction to task mining. We will go over different types of task mining and provide you with a real-world demo on each type of task mining in detail.
The History of Embeddings & Multimodal EmbeddingsZilliz
Frank Liu will walk through the history of embeddings and how we got to the cool embedding models used today. He'll end with a demo on how multimodal RAG is used.
Choosing the Best Outlook OST to PST Converter: Key Features and Considerationswebbyacad software
When looking for a good software utility to convert Outlook OST files to PST format, it is important to find one that is easy to use and has useful features. WebbyAcad OST to PST Converter Tool is a great choice because it is simple to use for anyone, whether you are tech-savvy or not. It can smoothly change your files to PST while keeping all your data safe and secure. Plus, it can handle large amounts of data and convert multiple files at once, which can save you a lot of time. It even comes with 24*7 technical support assistance and a free trial, so you can try it out before making a decision. Whether you need to recover, move, or back up your data, Webbyacad OST to PST Converter is a reliable option that gives you all the support you need to manage your Outlook data effectively.
2. 2 Why this talk?
• Why this query is so slow?
• Why planner is not using my index?
• What to do?
dataegret.com
3. 3 Where are we going?
• How planner works
• How we can affect it’s work
• When it can go wrong
• Known limitations
dataegret.com
4. 4 The Path of a Query
Connection
↓
Parser
↓
Rewrite system
↓
Planner/Optimizer
↓
Executor ↔ [Workers]
↓
Send results
all in single process (backend) beside background workers (parallel seq scan,
9.6+)
dataegret.com
5. 5 EXPLAIN command
explain (ANALYZE,VERBOSE,COSTS,BUFFERS,TIMING1) select * from t1;
QUERY PLAN
-------------------------------------------------------------------
Seq Scan on public.t1 (cost=0.00..104424.80 rows=10000000 width=8)
(actual time=0.218..2316.688 rows=10000000 loops=1)
Output: f1, f2
Buffers: shared read=44248
I/O Timings: read=322.7142
Planning time: 0.024 ms
Execution time: 3852.588 ms
1
COSTS and TIMING options are on by default
2
I/O Timings shown when track_io_timing is enabled
dataegret.com
6. 6 Planner have to guess
Seq Scan on public.t1 (cost=0.00..104424.80 rows=10000000 width=8)
• startup cost
• total cost
• rows
• average row width
dataegret.com
7. 7 Cost stability principles
Quote from “Common issues with planner statistics by Tomas Vondra”: 3
• correlation to query duration: The estimated cost is correlated with
duration of the query, i.e. higher cost means longer execution.
• estimation stability: A small difference in estimation causes only small
difference in costs, i.e. small error in estimation causes only small cost
differences.
• cost stability: Small cost difference means small difference in duration.
• cost comparability: For a given query, two plans with (almost) the same
costs should result in (almost) the same duration.
3
https://blog.pgaddict.com/posts/common-issues-with-planner-statistics
dataegret.com
8. 8 Data retrieval methods
• seq scan – sequential scan of whole table
• index scan – random io (read index + read table)
• index only scan – read only index (9.2+)4
• bitmap index scan – something in between seq scan/index scan, possible
to use several indexes at same time in OR/AND conditions
4
https://wiki.postgresql.org/wiki/Index-only_scans
dataegret.com
9. 9 Join methods
• nested loop – optimal for small relations
• hash join – optimal for big relations
• merge join – optimal for big relations if they’re sorted
dataegret.com
11. 11 Planner Cost Constants
#seq_page_cost = 1.0 # cost of a sequentially-fetched disk page
#random_page_cost = 4.0 # cost of a non-sequentially-fetched disk page
#cpu_tuple_cost = 0.01 # cost of processing each row during a query
#cpu_index_tuple_cost = 0.005 # cost of processing each index entry
#cpu_operator_cost = 0.0025 # cost of processing each operator or function
so basically cost is just
i
cini . How hard could it be?
dataegret.com
12. 12 Well, kind of hard
• How many rows we’ll get when we’ll filter table by this condition?
• How many pages is that? Will we read them sequentially or not?
• How many rows we’ll get when we join 2 relations?
dataegret.com
13. 13 We have stats!
• pg_statistic – only readable by a superuser
• pg_stats view – the same but human-readable and available to all users
(permissions apply)
dataegret.com
14. 14 pg_stats
pgday=# d pg_stats
Column | Type |
------------------------+----------+----------------------------------------------------------------
tablename | name | name of the table or functional index
attname | name | name of the column or index column
null_frac | real | fraction of column entries that are null
avg_width | integer | average width in bytes of column’s entries
n_distinct | real | number (or fraction of number of rows) of distinct values
most_common_vals | anyarray | list of the most common values in the column
most_common_freqs | real[] | list of the frequencies of the most common values
histogram_bounds | anyarray | list of intervals with approximately equal population
correlation | real | correlation between physical row ordering and logical ordering
most_common_elems | anyarray |
most_common_elem_freqs | real[] |
elem_count_histogram | real[] |
dataegret.com
15. 15 Analyze
table pages (8Kb)
pick 300*stats_target random pages pick 300*stats_target random rows
rows
5
5
Algorithm Z from Vitter, Jeffrey S. (1 March 1985). “Random sampling with a reservoir”
dataegret.com
18. 18 n_distinct underestimation example
select setseed(0.5);
create table test_ndistinct as
select
(case when random() < 0.1 then f1 end)::int f1
from normal_rand(10000000, 50000, 50000/3) as nr(f1);
10M rows, 90% nulls, ≈ 99.7% of values in between 0..100000
dataegret.com
19. 19 n_distinct underestimation example
# analyze verbose test_ndistinct;
INFO: analyzing "public.test_ndistinct"
INFO: "test_ndistinct": scanned 30000 of 35314 pages, containing 8495268 live rows and 0 dead rows;
30000 rows in sample, 10000067 estimated total rows
select * from pg_stats where tablename = ’test_ndistinct’ and attname = ’f1’;
...
null_frac | 0.904067
avg_width | 4
n_distinct | 3080
most_common_vals |
most_common_freqs |
histogram_bounds | {-8505,10072,15513,18933,21260,22574,24082,25695,26953,27898,28645...
correlation | -0.00286606
dataegret.com
20. 20 n_distinct underestimation example
# explain analyze select distinct f1 from test_ndistinct ;
QUERY PLAN
---------------------------------------------------------------------------------------
HashAggregate (cost=160314.84..160345.64 rows=3080 width=4)
(actual time=2558.751..2581.286 rows=90020 loops=1)
Group Key: f1
-> Seq Scan on test_ndistinct (cost=0.00..135314.67 rows=10000067 width=4)
(actual time=0.045..931.687 rows=10000000 loops=1)
Planning time: 0.048 ms
Execution time: 2586.550 ms
dataegret.com
21. 21 n_distinct underestimation example
# set default_statistics_target = 50;
# analyze verbose test_ndistinct;
INFO: analyzing "public.test_ndistinct"
INFO: "test_ndistinct": scanned 15000 of 35314 pages, containing 4247361 live rows and 0 dead rows;
15000 rows in sample, 9999792 estimated total rows
# explain analyze select distinct f1 from test_ndistinct ;
QUERY PLAN
---------------------------------------------------------------------------------------
HashAggregate (cost=160311.40..160328.51 rows=1711 width=4)
(actual time=2436.392..2455.851 rows=90020 loops=1)
Group Key: f1
-> Seq Scan on test_ndistinct (cost=0.00..135311.92 rows=9999792 width=4)
(actual time=0.029..892.596 rows=10000000 loops=1)
Planning time: 0.096 ms
Execution time: 2461.160 ms
dataegret.com
22. 22 n_distinct underestimation example
# explain analyze select * from test_ndistinct where f1 < 5000;
QUERY PLAN
---------------------------------------------------------------------------------------
Seq Scan on test_ndistinct (cost=0.00..160316.36 rows=99 width=4)
(actual time=2.325..1436.792 rows=3480 loops=1)
Filter: (f1 < 5000)
Rows Removed by Filter: 9996520
Planning time: 0.058 ms
Execution time: 1437.424 ms
dataegret.com
23. 23 n_distinct underestimation example
alter table test_ndistinct alter column f1 set (n_distinct = 100000);
analyze verbose test_ndistinct;
INFO: analyzing "public.test_ndistinct"
INFO: "test_ndistinct": scanned 15000 of 35314 pages, containing 4247670 live rows and 0 dead rows;
15000 rows in sample, 10000012 estimated total rows
ANALYZE
dataegret.com
24. 24 n_distinct underestimation example
# explain analyze select distinct f1 from test_ndistinct ;
QUERY PLAN
-------------------------------------------------------------------------------------------
Unique (cost=1571431.43..1621431.49 rows=100000 width=4)
(actual time=4791.872..7551.150 rows=90020 loops=1)
-> Sort (cost=1571431.43..1596431.46 rows=10000012 width=4)
(actual time=4791.870..6893.413 rows=10000000 loops=1)
Sort Key: f1
Sort Method: external merge Disk: 101648kB
-> Seq Scan on test_ndistinct (cost=0.00..135314.12 rows=10000012 width=4)
(actual time=0.041..938.093 rows=10000000 loops=1)
Planning time: 0.099 ms
Execution time: 7714.701 ms
dataegret.com
25. 25 n_distinct underestimation example
set work_mem = ’8MB’;
SET
# explain analyze select distinct f1 from test_ndistinct ;
QUERY PLAN
-------------------------------------------------------------------------------------------
HashAggregate (cost=160314.15..161314.15 rows=100000 width=4)
(actual time=2371.902..2391.415 rows=90020 loops=1)
Group Key: f1
-> Seq Scan on test_ndistinct (cost=0.00..135314.12 rows=10000012 width=4)
(actual time=0.093..871.619 rows=10000000 loops=1)
Planning time: 0.048 ms
Execution time: 2396.186 ms
dataegret.com
26. 26 n_distinct underestimation example
# explain analyze select * from test_ndistinct where f1 < 5000;
QUERY PLAN
-------------------------------------------------------------------------------------------
Seq Scan on test_ndistinct (cost=0.00..160316.61 rows=7550 width=4)
(actual time=0.723..839.347 rows=3480 loops=1)
Filter: (f1 < 5000)
Rows Removed by Filter: 9996520
Planning time: 0.262 ms
Execution time: 839.774 ms
dataegret.com
27. 27 n_distinct
• n_distinct plays important role in rows estimation when values are not in
MCV list
• In very big tables it’s possible to underestimate it in some cases
• It’s possible to override n_distinct estimation via
alter table xx alter column yy set (n_distinct = zz);
dataegret.com
28. 28 default_statistics_target
• Increasing default_statistics_target setting in config could help in this case
but not recommended
• Default value 100 usually is good enough
• When it’s not enough better increase it on selected columns only via
alter table xx alter column yy set statistics zz;
• Otherwise it could lead to much longer planning time (autoanalyze will
work longer too)
dataegret.com
29. 29 high default_statistics_target real example
# show default_statistics_target ;
default_statistics_target
---------------------------
6000
explain analyze SELECT "seven_charlie"."id" FROM "xray" JOIN "seven_charlie" ON
( "xray"."lima_seven" = "seven_charlie"."lima_seven" ) WHERE
( "xray"."alpha" = ’139505’ ) AND ( "seven_charlie"."seven_five" IS TRUE );
Nested Loop (cost=0.850..798.110 rows=58 width=4) (actual time=0.081..3.314 rows=169 loops=1)
-> Index Scan using romeo on xray (cost=0.420..205.390 rows=242 width=4) (actual time=0.023..0.79
Index Cond: (alpha = 139505)
-> Index Scan using lima_two on seven_charlie (cost=0.430..2.440 rows=1 width=8) (actual time=0.0
Index Cond: ((lima_seven = papa3six.lima_seven) AND (seven_five = true))
Planning time: 433.630 ms
Execution time: 3.397 ms
dataegret.com
30. 30 high default_statistics_target real example
set default_statistics_target = 1000;
SET
# analyze verbose xray;
INFO: analyzing "public.xray"
INFO: "xray": scanned 6760 of 6760 pages, containing 851656 live rows and 2004 dead rows; 300000 row
ANALYZE
Nested Loop (cost=0.850..782.110 rows=57 width=4) (actual time=0.066..2.992 rows=169 loops=1)
-> Index Scan using romeo on xray (cost=0.420..199.220 rows=238 width=4) (actual time=0.021..0.70
Index Cond: (alpha = 139505)
-> Index Scan using lima_two on seven_charlie (cost=0.430..2.440 rows=1 width=8) (actual time=0.0
Index Cond: ((lima_seven = papa3six.lima_seven) AND (seven_five = true))
Planning time: 75.196 ms
Execution time: 3.071 ms
dataegret.com
31. 31 high default_statistics_target real example
# analyze verbose seven_charlie;
INFO: "seven_charlie": scanned 300000 of 1548230 pages, containing 2184079 live rows and 8293 dead r
Nested Loop (cost=0.850..782.110 rows=65 width=4) (actual time=0.197..26.517 rows=169 loops=1)
-> Index Scan using romeo on xray (cost=0.420..199.220 rows=238 width=4) (actual time=0.064..3.15
Index Cond: (alpha = 139505)
-> Index Scan using lima_two on seven_charlie (cost=0.430..2.440 rows=1 width=8) (actual time=0.0
Index Cond: ((lima_seven = papa3six.lima_seven) AND (seven_five = true))
Planning time: 14.256 ms
Execution time: 26.617 ms
dataegret.com
32. 32 Hacks
select name from pg_settings where name ~ ‘enable_’;
enable_bitmapscan
enable_indexscan
enable_indexonlyscan
enable_seqscan
enable_tidscan
enable_nestloop
enable_hashjoin
enable_mergejoin
enable_sort
enable_hashagg
enable_material
startup_cost += disable_cost
disable_cost = 1010
dataegret.com
33. 33 Hacks
• Very good for testing
• Affects whole query
• Possible to use in functions in some bad cases via
alter function xxx() set enable_? = false
• pg_hint_plan6 extension (not in contrib) which provide hints
6
https://osdn.net/projects/pghintplan/
dataegret.com
34. 34 Join ordering problem
• There are O(n!) ways to join n relations which grows very fast (10! ≈ 3.6M)
• ORMs like to join everything
• It’s possible to break this number down by using CTEs but be careful
• join_collapse_limit, from_collapse_limit (default 8 relations)
dataegret.com
35. 35 Genetic Query Optimizer (geqo)
• geqo_threshold (default 12 relations)
• Chose suboptimal plan in reasonable time
• “Mutation” and selection phases
dataegret.com
37. 37 What planner can’t do properly
• Estimate number of rows correctly for conditions like “a=x and b=y” where
a and b statistically dependent
• Use indexes for conditions like created_at + interval ‘1 day’ >= NOW()
• Use index to count distinct values8
• Cope with lots of partitions
• Estimate correctly how many rows need to be read when using index scan
on a for “where condition order by a limit n”
8
https://wiki.postgresql.org/wiki/Loose_indexscan
dataegret.com
38. 38 Query rewriting tricks
• Disable index usage in where clause: a = x => a+0 = x
• Disable index usage in order by clause: order by a => order by a+0
• Restrict push up/pull downs from subquery with offset 0
• Replace left join with exists/not exists to force nested loop
• Move non-limiting join after limit
dataegret.com
39. 39 What have we learned?
Planner
query
pg_statistic
Analyze
Plan
config
default_statistics_target
autovacuum_analyze_scale_factor
pg_classpg_attribute
custom per-column
stats_target
n_distinct
custom per-table
autovacuum_analyze_scale_factor
pg_tablespace
costs
custom per-tablespace
seq_page_cost
random_page_cost
memory
work_mem
effective_cache_size
hacks
enable_*
pg_index
seq_page_cost
random_page_cost
cpu_tuple_cost
cpu_index_tuple_cost
cpu_operator_cost
parallel
parallel_setup_cost
parallel_tuple_cost
max_parallel_workers_per_gather
query rewriting tricks
other
from_collapse_limit
join_collapse_limit
geqo*
or session/per database/per user settings
dataegret.com
40. 40 Troubleshooting
• Don’t panic!
• Check if planner’s estimates are wrong (off by orders of magnitude)
• Check for missing indexes when a lot of filtering is done
• For complex plans https://explain.depesz.com/ could help
• Extract problem part
• Check for outdated/incomplete stats
• Play with hacks and query rewriting tricks
dataegret.com
41. 41 Would you like to know more?
• Robert Haas – The PostgreSQL Query Planner, PostgreSQL East 2010
• Tom Lane – Hacking the Query Planner, PGCon 2011
• Bruce Momjian – Explaining the Postgres Query Optimizer
• PostgreSQL Manual 67.1. Row Estimation Examples
• PostgreSQL Manual 14.1. Using EXPLAIN
• depesz: Implement multivariate n-distinct coefficients
• depesz: Explaining the unexplainable
• www.slideshare.net/alexius2/
dataegret.com