Apache Calcite: One planner fits all

If SQL is the universal language of data, why do we author our most important data applications (metrics, analytics, business intelligence) in languages other than SQL? Multidimensional databases and languages such as MDX, DAX and Tableau LOD solve these problems but introduce others: they require specialized knowledge, complicate the data pipeline and don’t integrate well. Is it possible to define and query business intelligence models in SQL? Apache Calcite has extended SQL to support measures, evaluation context, and multidimensional expressions. With these concepts you can define data models that contain measures, use them in queries, and define new measures in queries. A talk given by Julian Hyde to Apache Calcite's virtual meetup, 2023-03-15.

SQL for NoSQL and how Apache Calcite can help

Christian Tzolov

https://fosdem.org/2017/schedule/event/hpc_bigdata_calcite/ When working with BigData & IoT systems we often feel the need for a Common Query Language. The platform specific languages are often harder to integrate with and require longer adoption time. To fill this gap many NoSql (Not-only-Sql) vendors are building SQL layers for their platforms. It is worth exploring the driving forces behind this trend, how it fits in your BigData stacks and how we can adopt it in our favorite tools. However building SQL engine from scratch is a daunting job and frameworks like Apache Calcite can help you with the heavy lifting. Calcite allow you to integrate SQL parser, cost-based optimizer, and JDBC with your big data system. Calcite has been used to empower many Big-Data platforms such as Hive, Spark, Drill Phoenix to name some. I will walk you through the process of building a SQL access layer for Apache Geode (In-Memory Data Grid). I will share my experience, pitfalls and technical consideration like balancing between the SQL/RDBMS semantics and the design choices and limitations of the data system. Hopefully this will enable you to add SQL capabilities to your prefered NoSQL data system.

Cost-based Query Optimization in Apache Phoenix using Apache Calcite

This document summarizes a presentation on using Apache Calcite for cost-based query optimization in Apache Phoenix. Key points include: - Phoenix is adding Calcite's query planning capabilities to improve performance and SQL compliance over its existing query optimizer. - Calcite models queries as relational algebra expressions and uses rules, statistics, and a cost model to choose the most efficient execution plan. - Examples show how Calcite rules like filter pushdown and exploiting sortedness can generate better plans than Phoenix's existing optimizer. - Materialized views and interoperability with other Calcite data sources like Apache Drill are areas for future improvement beyond the initial Phoenix+Calcite integration.

Data all over the place! How SQL and Apache Calcite bring sanity to streaming...

The revolution has happened. We are living the age of the deconstructed database. The modern enterprises are powered by data, and that data lives in many formats and locations, in-flight and at rest, but somewhat surprisingly, the lingua franca for remains SQL. In this talk, Julian describes Apache Calcite, a toolkit for relational algebra that powers many systems including Apache Beam, Flink and Hive. He discusses some areas of development in Calcite: streaming SQL, materialized views, enabling spatial query on vanilla databases, and what a mash-up of all three might look like. He also describes how SQL is being extended to handle streaming, and the challenges that will need to be solved if it is to become standard. A talk given by Julian Hyde at Lyft, San Francisco, on 2018/06/27.

Open Source SQL - beyond parsers: ZetaSQL and Apache Calcite

Smarter Together - Bringing Relational Algebra, Powered by Apache Calcite, in...

What if Looker saw the queries you just executed and could predict your next query? Could it make those queries faster, by smarter caching, or aggregate navigation? Could it read your past SQL queries and help you write your LookML model? Those are some of the reasons to add relational algebra into Looker’s query engine, and why Looker hired Julian Hyde, author of Apache Calcite, to lead the effort. In this talk about the internals of Looker’s query engine, Julian Hyde will describe how the engine works, how Looker queries are described in Calcite’s relational algebra, and some features that it makes possible. A talk by Julian Hyde at JOIN 2019 in San Francisco.

The Apache Spark File Format Ecosystem

A Deep Dive into Query Execution Engine of Spark SQL

Spark SQL enables Spark to perform efficient and fault-tolerant relational query processing with analytics database technologies. The relational queries are compiled to the executable physical plans consisting of transformations and actions on RDDs with the generated Java code. The code is compiled to Java bytecode, executed at runtime by JVM and optimized by JIT to native machine code at runtime. This talk will take a deep dive into Spark SQL execution engine. The talk includes pipelined execution, whole-stage code generation, UDF execution, memory management, vectorized readers, lineage based RDD transformation and action.

Apache Calcite: One Frontend to Rule Them All

Michael Mior

Apache Calcite is an open source framework that allows for a unified query interface over heterogeneous data sources. It provides an ANSI-compliant SQL parser, a logical query optimizer, and acts as a middleware layer that can integrate data from multiple sources. Calcite uses a relational algebra approach and has pluggable adapters that allow it to connect to different backends like MySQL, MongoDB, and streaming data sources. It supports features like SQL queries, views, optimization rules, and works across both batch and streaming data. The project aims to continue adding new capabilities like geospatial queries and improved cost modeling.

Optimizing Apache Spark SQL Joins

Join operations in Apache Spark is often the biggest source of performance problems and even full-blown exceptions in Spark. After this talk, you will understand the two most basic methods Spark employs for joining DataFrames – to the level of detail of how Spark distributes the data within the cluster. You’ll also find out how to work out common errors and even handle the trickiest corner cases we’ve encountered! After this talk, you should be able to write performance joins in Spark SQL that scale and are zippy fast! This session will cover different ways of joining tables in Apache Spark. Speaker: Vida Ha This talk was originally presented at Spark Summit East 2017.

Building a SIMD Supported Vectorized Native Engine for Spark SQL

Spark SQL works very well with structured row-based data. Vectorized reader and writer for parquet/orc can make I/O much faster. It also used WholeStageCodeGen to improve the performance by Java JIT code. However Java JIT is usually not working very well on utilizing latest SIMD instructions under complicated queries. Apache Arrow provides columnar in-memory layout and SIMD optimized kernels as well as a LLVM based SQL engine Gandiva. These native based libraries can accelerate Spark SQL by reduce the CPU usage for both I/O and execution.

Apache Spark Data Source V2 with Wenchen Fan and Gengliang Wang

As a general computing engine, Spark can process data from various data management/storage systems, including HDFS, Hive, Cassandra and Kafka. For flexibility and high throughput, Spark defines the Data Source API, which is an abstraction of the storage layer. The Data Source API has two requirements. 1) Generality: support reading/writing most data management/storage systems. 2) Flexibility: customize and optimize the read and write paths for different systems based on their capabilities. Data Source API V2 is one of the most important features coming with Spark 2.3. This talk will dive into the design and implementation of Data Source API V2, with comparison to the Data Source API V1. We also demonstrate how to implement a file-based data source using the Data Source API V2 for showing its generality and flexibility.

Apache Spark Architecture

Alexey Grishchenko

Understanding Query Plans and Spark UIs

"The common use cases of Spark SQL include ad hoc analysis, logical warehouse, query federation, and ETL processing. Spark SQL also powers the other Spark libraries, including structured streaming for stream processing, MLlib for machine learning, and GraphFrame for graph-parallel computation. For boosting the speed of your Spark applications, you can perform the optimization efforts on the queries prior employing to the production systems. Spark query plans and Spark UIs provide you insight on the performance of your queries. This talk discloses how to read and tune the query plans for enhanced performance. It will also cover the major related features in the recent and upcoming releases of Apache Spark. "

Introducing DataFrames in Spark for Large Scale Data Science

Hortonworks Technical Workshop: Interactive Query with Apache Hive

Hortonworks

Apache Hive is the defacto standard for SQL queries over petabytes of data in Hadoop. It is a comprehensive and compliant engine that offers the broadest range of SQL semantics for Hadoop, providing a powerful set of tools for analysts and developers to access Hadoop data. The session will cover the latest advancements in Hive and provide practical tips for maximizing Hive Performance. Audience: Developers, Architects and System Engineers from the Hortonworks Technology Partner community. Recording: https://hortonworks.webex.com/hortonworks/lsr.php?RCID=7c8f800cbbef256680db14c78b871f97

Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...

Streaming SQL with Apache Calcite

With the rise of the Internet of Things (IoT) and low-latency analytics, streaming data becomes ever more important. Surprisingly, one of the most promising approaches for processing streaming data is SQL. In this presentation, Julian Hyde shows how to build streaming SQL analytics that deliver results with low latency, adapt to network changes, and play nicely with BI tools and stored data. He also describes how Apache Calcite optimizes streaming queries, and the ongoing collaborations between Calcite and the Storm, Flink and Samza projects. This talk was given Julian Hyde at Apache Big Data conference, Vancouver, on 2016/05/09.

Apache Spark Overview

Vadim Y. Bichutskiy

This document provides an overview of a talk on Apache Spark. It introduces the speaker and their background. It acknowledges inspiration from a previous Spark training. It then outlines the structure of the talk, which will include: a brief history of big data; a tour of Spark including its advantages over MapReduce; and explanations of Spark concepts like RDDs, transformations, and actions. The document serves to introduce the topics that will be covered in the talk.

Morel, a Functional Query Language

Is it easier to add functional programming features to a query language, or to add query capabilities to a functional language? In Morel, we have done the latter. Functional and query languages have much in common, and yet much to learn from each other. Functional languages have a rich type system that includes polymorphism and functions-as-values and Turing-complete expressiveness; query languages have optimization techniques that can make programs several orders of magnitude faster, and runtimes that can use thousands of nodes to execute queries over terabytes of data. Morel is an implementation of Standard ML on the JVM, with language extensions to allow relational expressions. Its compiler can translate programs to relational algebra and, via Apache Calcite’s query optimizer, run those programs on relational backends. In this talk, we describe the principles that drove Morel’s design, the problems that we had to solve in order to implement a hybrid functional/relational language, and how Morel can be applied to implement data-intensive systems. (A talk given by Julian Hyde at Strange Loop 2021, St. Louis, MO, on October 1st, 2021.)

What's hot (20)

Adding measures to Calcite SQL

SQL for NoSQL and how Apache Calcite can help

Cost-based Query Optimization in Apache Phoenix using Apache Calcite

Data all over the place! How SQL and Apache Calcite bring sanity to streaming...

Open Source SQL - beyond parsers: ZetaSQL and Apache Calcite

Smarter Together - Bringing Relational Algebra, Powered by Apache Calcite, in...

The Apache Spark File Format Ecosystem

A Deep Dive into Query Execution Engine of Spark SQL

Apache Calcite: One Frontend to Rule Them All

Optimizing Apache Spark SQL Joins

Building a SIMD Supported Vectorized Native Engine for Spark SQL

Apache Spark Data Source V2 with Wenchen Fan and Gengliang Wang

Apache Spark Architecture

Understanding Query Plans and Spark UIs

Introducing DataFrames in Spark for Large Scale Data Science

Hortonworks Technical Workshop: Interactive Query with Apache Hive

Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...

Streaming SQL with Apache Calcite

Apache Spark Overview

Morel, a Functional Query Language

Viewers also liked

Options for Data Prep - A Survey of the Current Market

Data comes in many shapes and sizes, and every company struggles to find ways to transform, validate, and enrich data for multiple purposes. The problem has been around as long as data, and the market has an overwhelming number of options. In this presentation we look at the problem and key options from vendors in the market today. Dremio is a new approach that eliminates the need for stand alone data prep tools.

Bi on Big Data - Strata 2016 in London

Building a Virtual Data Lake with Apache Arrow

Apache Arrow - An Overview

The twins that everyone loved too much

The Future of Column-Oriented Data Processing With Apache Arrow and Apache Pa...

Essentially every successful analytical DBMS in the market today makes use of column-oriented data structures. In the Hadoop ecosystem, Apache Parquet (and Apache ORC) provide similar advantages in terms of processing and storage efficiency. Apache Arrow is the in-memory counterpart to these formats and has been been embraced by over a dozen open source projects as the de facto standard for in-memory processing. In this session the PMC Chair for Apache Arrow and the PMC Chair for Apache Parquet discuss the future of column-oriented processing.

Data Science Languages and Industry Analytics

Wes McKinney

SQL on everything, in memory

Enterprise data is moving into Hadoop, but some data has to stay in operational systems. Apache Calcite (the technology behind Hive’s new cost-based optimizer, formerly known as Optiq) is a query-optimization and data federation technology that allows you to combine data in Hadoop with data in NoSQL systems such as MongoDB and Splunk, and access it all via SQL. Hyde shows how to quickly build a SQL interface to a NoSQL system using Calcite. He shows how to add rules and operators to Calcite to push down processing to the source system, and how to automatically build materialized data sets in memory for blazing-fast interactive analysis.

Don’t optimize my queries, optimize my data!

The document discusses strategies for optimizing data through materialized views and how data systems can learn to optimize themselves. It proposes an algorithm that uses sketches and information theory to profile data cardinalities and recommend materialized views. The algorithm aims to defeat the combinatorial search space by only considering combinations with "surprising" cardinalities. This profiling provides the cost and benefit information needed to optimize data structures. The document also discusses using query logs and statistics to infer relationships between tables and design summary tables through lattices.

Apache Arrow: In Theory, In Practice

This document discusses Apache Arrow, an open source cross-language development platform for in-memory analytics. It provides an overview of Arrow's goals of being cross-language compatible, optimized for modern CPUs, and enabling interoperability between systems. Key components include core C++/Java libraries, integrations with projects like Pandas and Spark, and common message patterns for sharing data. The document also describes how Arrow is implemented in practice in systems like Dremio's Sabot query engine.

Using Apache Arrow, Calcite, and Parquet to Build a Relational Cache

From DataEngConf 2017 - Everybody wants to get to data faster. As we move from more general solution to specific optimization techniques, the level of performance impact grows. This talk will discuss how layering in-memory caching, columnar storage and relational caching can combine to provide a substantial improvement in overall data science and analytical workloads. It will include a detailed overview of how you can use Apache Arrow, Calcite and Parquet to achieve multiple magnitudes improvement in performance over what is currently possible.

Viewers also liked (11)

Options for Data Prep - A Survey of the Current Market

Bi on Big Data - Strata 2016 in London

Building a Virtual Data Lake with Apache Arrow

Apache Arrow - An Overview

The twins that everyone loved too much

The Future of Column-Oriented Data Processing With Apache Arrow and Apache Pa...

Data Science Languages and Industry Analytics

SQL on everything, in memory

Don’t optimize my queries, optimize my data!

Apache Arrow: In Theory, In Practice

Using Apache Arrow, Calcite, and Parquet to Build a Relational Cache

Similar to Apache Calcite: One planner fits all

HQL over Tiered Data Warehouse

DataWorks Summit

This document discusses Inmobi's analytics platform and the challenges it aimed to address with Grill. It provides an overview of Hive and how Grill leverages Hive to provide a unified query layer across different execution engines and data stores. Key points include how Grill allows OLAP queries on cubes, rewrites queries for different storages, and selects the most efficient execution engine to run the query. The demo then shows Grill in action.

Fifth elephant-grill

amarsri

Grill is a unified analytics platform developed by Inmobi to address problems with disparate data storage and querying across their Hadoop and SQL data warehouses. Grill provides a single interface and catalog to allow both ad-hoc and canned queries to run interactively or in batch mode over billions of records. It uses a pluggable execution engine architecture to optimize query costs and allows queries to span multiple storage systems like Hadoop, HBase and SQL data stores.

Apache Kylin Extreme OLAP Engine for Big Data

Luke Han

This document provides an overview of Apache Kylin, an open source distributed analytics engine that provides SQL interface and multi-dimensional analysis (OLAP) on Hadoop supporting extremely large datasets. It discusses Kylin's features such as fast query performance, SQL interface, seamless integration with BI tools, and job management capabilities. It also describes Kylin's technical architecture including its use of MapReduce for cube building, storage of cubes in HBase, and routing of SQL queries to the query engine. The document outlines Kylin's roadmap including plans to improve cube building algorithms and support real-time analysis using streaming and Spark.

Grill at HadoopSummit

amarsri

- The document discusses Inmobi's analytics data warehouse which contains 170 TB of data and the challenges of querying across different data stores and execution engines. - It introduces Apache Hive and the OLAP cube model for representing multi-dimensional data, and provides examples of queries on cubes. - Grill is presented as Inmobi's solution to unify querying across Hive, Impala, and other engines through a single interface and metadata catalog. A demo of Grill's capabilities is included in the agenda.

Testing Big Data: Automated Testing of Hadoop with QuerySurge

RTTS

Are You Ready? Stepping Up To The Big Data Challenge In 2016 - Learn why Testing is pivotal to the success of your Big Data Strategy. According to a new report by analyst firm IDG, 70% of enterprises have either deployed or are planning to deploy big data projects and programs this year due to the increase in the amount of data they need to manage. The growing variety of new data sources is pushing organizations to look for streamlined ways to manage complexities and get the most out of their data-related investments. The companies that do this correctly are realizing the power of big data for business expansion and growth. Learn why testing your enterprise's data is pivotal for success with big data and Hadoop. Learn how to increase your testing speed, boost your testing coverage (up to 100%), and improve the level of quality within your data - all with one data testing tool.

Building Modern Data Platform with Microsoft Azure

Dmitry Anoshin

This document provides an overview of building a modern cloud analytics solution using Microsoft Azure. It discusses the role of analytics, a history of cloud computing, and a data warehouse modernization project. Key challenges covered include lack of notifications, logging, self-service BI, and integrating streaming data. The document proposes solutions to these challenges using Azure services like Data Factory, Kafka, Databricks, and SQL Data Warehouse. It also discusses alternative implementations using tools like Matillion ETL and Snowflake.

From Business Intelligence to Big Data - hack/reduce Dec 2014

Adam Ferrari

Apache Kylin: OLAP Engine on Hadoop - Tech Deep Dive

Xu Jiang

Kylin is an open source Distributed Analytics Engine from eBay Inc. that provides SQL interface and multi-dimensional analysis (OLAP) on Hadoop supporting extremely large datasets. If you want to do multi-dimension analysis on large data sets (billion+ rows) with low query latency (sub-seconds), Kylin is a good option. Kylin also provides seamless integration with existing BI tools (e.g Tableau).

Develop a Custom Data Solution Architecture with NorthBay

Amazon Web Services

Organizations that have vast amounts of data in legacy applications often experience difficulties delivering that data to business unit end-users. Register to learn how Eliza Corporation and Scholastic overcame this challenge by leveraging a Data Lake solution from NorthBay on AWS to optimize data analytics and provide greater visibility. AWS and NorthBay will give you an in-depth overview of how you can use a Data Lake in conjunction with your existing on-premises or cloud-based Data Warehouse. NorthBay helps organizations scale their ETL and data warehousing workloads using Amazon EMR and Amazon Redshift. Join us to learn: • Best practices for using a Data Lake in conjunction with your existing data warehouse • The key aspects of introducing agile and scrum methodologies into an enterprise • The most impactful cost-savings levers that are addressed via a cloud data warehouse migration Who should attend: Heads of Analytics, Heads of BI, Analytics Managers, BI Teams, Senior Analysts

SQL Analytics for Search Engineers - Timothy Potter, Lucidworksngineers

Lucidworks

This document discusses how SQL can be used in Lucidworks Fusion for various purposes like aggregating signals to compute relevance scores, ingesting and transforming data from various sources using Spark SQL, enabling self-service analytics through tools like Tableau and PowerBI, and running experiments to compare variants. It provides examples of using SQL for tasks like sessionization with window functions, joining multiple data sources, hiding complex logic in user-defined functions, and powering recommendations. The document recommends SQL in Fusion for tasks like analytics, data ingestion, machine learning, and experimentation.

ONE FOR ALL! Using Apache Calcite to make SQL smart

Evans Ye

In the past when Hadoop was born, the big data world were focusing on how to build systems that scales. Now the world has evolved. HBase hits 2.0, Cassandra hits 3.0, Hive hits 3.0, etc. When scalability is conquered, what's next? That’s right, usability comes into play. If we look back into the history, NoSQL is really just using divide and concur mechanism to tackle big data problems by trading off SQL capabilities. But once big data problem solved, we see more and more NoSQL and data processing engines start to build up SQL or SQL-like interfaces. Therefore, a generic SQL engine that provides core SQL capabilities such as query parsing, relational algebra, and query optimization starts to shine. In this talk, I'll walk you through the architecture, functionality, and design concept of Apache Calcite. Notice that Calcite itself is not a database, but many well known systems already incorporate Calcite as a library. For instance, Hive, Drill, Druid, Phoenix, Apex, Flink, Storm, Samza, and more. To better illustrate how Calcite works, I'll choose some of the systems and describe how they adopt Calcite and which part is enhanced by Calcite. Furthermore, I'll talk about several features that Calcite provides such as query optimization, heterogeneous data source, materialized view, and Stream SQL. From user's perspective, knowing better how these systems work behind the scene equips you with more knowledge to chose a system that ultimately suits your needs.

Apache® Spark™ MLlib: From Quick Start to Scikit-Learn

These are the slides to support the Apache® Spark™ MLlib: From Quick Start to Scikit-Learn webinar. In this webcast, Joseph Bradley from Databricks will be speaking about Apache Spark’s distributed Machine Learning Library - MLlib. We will start off with a quick primer on machine learning, Spark MLlib, and a quick overview of some Spark machine learning use cases. We will continue with multiple Spark MLlib quick start demos. Afterwards, the talk will transition toward the integration of common data science tools like Python pandas, scikit-learn, and R with MLlib

Sf big analytics_2018_04_18: Evolution of the GoPro's data platform

Chester Chen

Talk 1 : Evolution of the GoPro's data platform In this talk, we will share GoPro’s experiences in building Data Analytics Cluster in Cloud. We will discuss: evolution of data platform from fixed-size Hadoop clusters to Cloud-based Spark Cluster with Centralized Hive Metastore +S3: Cost Benefits and DevOp Impact; Configurable, spark-based batch Ingestion/ETL framework; Migration Streaming framework to Cloud + S3; Analytics metrics delivery with Slack integration; BedRock: Data Platform Management, Visualization & Self-Service Portal Visualizing Machine learning Features via Google Facets + Spark Speakers: Chester Chen Chester Chen is the Head of Data Science & Engineering, GoPro. Previously, he was the Director of Engineering at Alpine Data Lab. David Winters David is an Architect in the Data Science and Engineering team at GoPro and the creator of their Spark-Kafka data ingestion pipeline. Previously He worked at Apple & Splice Machines. Hao Zou Hao is a Senior big data engineer at Data Science and Engineering team. Previously He worked as Alpine Data Labs and Pivotal

The modern analytics architecture

Joseph D'Antoni

This document summarizes the history and evolution of data warehousing and analytics architectures. It discusses how data warehouses emerged in the 1970s and were further developed in the late 1980s and 1990s. It then covers how big data and Hadoop have changed architectures, providing more scalability and lower costs. Finally, it outlines components of modern analytics architectures, including Hadoop, data warehouses, analytics engines, and visualization tools that integrate these technologies.

Introduction to Dremio

An introduction to self-service data with Dremio. Dremio reimagines analytics for modern data. Created by veterans of open source and big data technologies, Dremio is a fundamentally new approach that dramatically simplifies and accelerates time to insight. Dremio empowers business users to curate precisely the data they need, from any data source, then accelerate analytical processing for BI tools, machine learning, data science, and SQL clients. Dremio starts to deliver value in minutes, and learns from your data and queries, making your data engineers, analysts, and data scientists more productive.

Apache Kylin: Hadoop OLAP Engine, 2014 Dec

Yang Li

Microsoft Azure BI Solutions in the Cloud

Mark Kromer

This document provides an overview of several Microsoft Azure cloud data and analytics services: - Azure Data Factory is a data integration service that can move and transform data between cloud and on-premises data stores as part of scheduled or event-driven workflows. - Azure SQL Data Warehouse is a cloud data warehouse that provides elastic scaling for large BI and analytics workloads. It can scale compute resources on demand. - Azure Machine Learning enables building, training, and deploying machine learning models and creating APIs for predictive analytics. - Power BI provides interactive reports, visualizations, and dashboards that can combine multiple datasets and be embedded in applications.

Power BI / AAS Data Model Optimization 101 v2

Dan English

Dan English gave a presentation on optimizing Power BI and Analysis Services Tabular data models for performance. He discussed required tools like DAX Studio, Vertipaq Analyzer and Tabular Editor. Key steps included reviewing table and column sizes, DAX expressions, and best practices like star schema design and simplifying calculations. Resources on optimization best practices and troubleshooting performance were provided.

SolrCloud on Hadoop

Alex Moundalexis

This document provides an overview of SolrCloud on Hadoop. It discusses how SolrCloud allows for distributed, highly scalable search capabilities on Hadoop clusters. Key components that work with SolrCloud are also summarized, including HDFS for storage, MapReduce for processing, and ZooKeeper for coordination services. The document demonstrates how SolrCloud can index and query large datasets stored in Hadoop.

Apache Kylin @ Big Data Europe 2015

Seshu Adunuthula

The document discusses Apache Kylin, an open source distributed analytics engine that provides SQL interface and multi-dimensional analysis (OLAP) on Hadoop for extremely large datasets. It provides an overview of Kylin's features such as sub-second query latency, ANSI SQL support, and seamless integration with BI tools. The document also covers Kylin's architecture, cube storage in HBase, query processing using Calcite, and optimization techniques for cube building.

Similar to Apache Calcite: One planner fits all (20)

HQL over Tiered Data Warehouse

Fifth elephant-grill

Apache Kylin Extreme OLAP Engine for Big Data

Grill at HadoopSummit

Testing Big Data: Automated Testing of Hadoop with QuerySurge

Building Modern Data Platform with Microsoft Azure

From Business Intelligence to Big Data - hack/reduce Dec 2014

Apache Kylin: OLAP Engine on Hadoop - Tech Deep Dive

Develop a Custom Data Solution Architecture with NorthBay

SQL Analytics for Search Engineers - Timothy Potter, Lucidworksngineers

ONE FOR ALL! Using Apache Calcite to make SQL smart

Apache® Spark™ MLlib: From Quick Start to Scikit-Learn

Sf big analytics_2018_04_18: Evolution of the GoPro's data platform

The modern analytics architecture

Introduction to Dremio

Apache Kylin: Hadoop OLAP Engine, 2014 Dec

Microsoft Azure BI Solutions in the Cloud

Power BI / AAS Data Model Optimization 101 v2

SolrCloud on Hadoop

Apache Kylin @ Big Data Europe 2015

More from Julian Hyde

Measures in SQL (SIGMOD 2024, Santiago, Chile)

SQL has attained widespread adoption, but Business Intelligence tools still use their own higher level languages based upon a multidimensional paradigm. Composable calculations are what is missing from SQL, and we propose a new kind of column, called a measure, that attaches a calculation to a table. Like regular tables, tables with measures are composable and closed when used in queries. SQL-with-measures has the power, conciseness and reusability of multidimensional languages but retains SQL semantics. Measure invocations can be expanded in place to simple, clear SQL. To define the evaluation semantics for measures, we introduce context-sensitive expressions (a way to evaluate multidimensional expressions that is consistent with existing SQL semantics), a concept called evaluation context, and several operations for setting and modifying the evaluation context. A talk at SIGMOD, June 9–15, 2024, Santiago, Chile Authors: Julian Hyde (Google) and John Fremlin (Google) https://doi.org/10.1145/3626246.3653374

Measures in SQL (a talk at SF Distributed Systems meetup, 2024-05-22)

Building a semantic/metrics layer using Calcite

A semantic layer, also known as a metrics layer, lies between business users and the database, and lets those users compose queries in the concepts that they understand. It also governs access to the data, manages data transformations, and can tune the database by defining materializations. Like many new ideas, the semantic layer is a distillation and evolution of many old ideas, such as query languages, multidimensional OLAP, and query federation. In this talk, we describe the features we are adding to Calcite to define business views, query measures, and optimize performance. A talk given at Community over Code, the annual conference of the Apache Software Foundation, in Halifax, NS, on 9th October, 2023.

Cubing and Metrics in SQL, oh my!

This document discusses processing business intelligence (BI) queries in SQL rather than BI-specific languages. It proposes extensions to Apache Calcite's SQL dialect to support measures, context-sensitive expressions, and analytic views that contain metrics and calculations. The talk will describe the SQL syntax for measures, how to define and use them for cross-dimensional calculations, and approaches for optimizing such queries. It provides examples of multidimensional queries that can be expressed in both MDX and the proposed SQL extensions.

Morel, a data-parallel programming language

This document discusses Morel, a data-parallel programming language that is an extension of Standard ML with relational operators. Morel aims to provide the expressiveness of a functional programming language, the power and conciseness of SQL, and efficient execution on different hardware. It is implemented on top of Apache Calcite's relational algebra framework. The talk describes Morel's evolution and how it is pushing Calcite's capabilities with graph and recursive queries. Standard ML concepts like functions, recursion, and higher-order functions are extended in Morel with relational operators like "from" to enable data-parallel programming over immutable datasets. Functions can also be treated as values in Morel.

Is there a perfect data-parallel programming language? (Experiments with More...

The perfect data parallel language has not yet been invented. SQL queries can achieve great performance and scale, but there are many general purpose algorithms that it cannot express. In Morel, we build on the functional and relational roots of MapReduce in an elegant and strongly-typed general-purpose programming language. But Morel is, in a real sense, a query language; programs are executed on relational frameworks such as Google BigQuery and Spark. In this talk, we describe the principles that drove Morel’s design, the problems that we had to solve in order to implement a hybrid functional/relational language, and how Morel can be applied to implement data-intensive systems. We also introduce Apache Calcite, the popular open source framework for query planning, and describe how Morel's compiler uses Calcite's relational algebra and rewrite rules to generate efficient plans.

What to expect when you're Incubating

Efficient spatial queries on vanilla databases

A talk given by Julian Hyde at the Apache Calcite online meetup, 2021/01/20. Spatial and GIS applications have traditionally required specialized databases, or at least specialized data structures like r-trees. Unfortunately this means that hybrid applications such as spatial analytics are not well served, and many people are unaware of the power of spatial queries because their favorite database does not support them. In this talk, we describe how Apache Calcite enables efficient spatial queries using generic data structures such as HBase’s key-sorted tables, using techniques like Hilbert space-filling curves and materialized views. Calcite implements much of the OpenGIS function set and recognizes query patterns that can be rewritten to use particular spatial indexes. Calcite is bringing spatial query to the masses!

Tactical data engineering

A talk given by Julian Hyde at DataCouncil SF on April 18, 2019 How do you organize your data so that your users get the right answers at the right time? That question is a pretty good definition of data engineering — but it is also describes the purpose of every DBMS (database management system). And it’s not a coincidence that these are so similar. This talk looks at the patterns that reoccur throughout data management — such as caching, partitioning, sorting, and derived data sets. As the speaker is the author of Apache Calcite, we first look at these patterns through the lens of Relational Algebra and DBMS architecture. But then we apply these patterns to the modern data pipeline, ETL and analytics. As a case study, we look at how Looker’s “derived tables” blur the line between ETL and caching, and leverage the power of cloud databases.

Don't optimize my queries, organize my data!

Your queries won't run fast if your data is not organized right. Apache Calcite optimizes queries, but can we make it optimize data? We had to solve several challenges. Users are too busy to tell us the structure of their database, and the query load changes daily, so Calcite has to learn and adapt. We talk about new algorithms we developed for gathering statistics on massive database, and how we infer and evolve the data model based on the queries.

Spatial query on vanilla databases

A talk given by Julian Hyde at ApacheCon NA 2018 in Montreal on September 26th, 2018. Spatial and GIS applications have traditionally required specialized databases, or at least specialized data structures like r-trees. Unfortunately this means that hybrid applications such as spatial analytics are not well served, and many people are unaware of the power of spatial queries because their favorite database does not support them. In this talk, we describe how Apache Calcite enables efficient spatial queries using generic data structures such as HBase’s key-sorted tables, using techniques like Hilbert space-filling curves and materialized views. Calcite implements much of the OpenGIS function set and recognizes query patterns that can be rewritten to use particular spatial indexes. Calcite is bringing spatial query to the masses!

Lazy beats Smart and Fast

A talk given by Julian Hyde at DataEngConf SF on April 17th 2018. Did you know that databases often “cheat”? Even with a scalable query engine and smart optimizer, many real-world queries would be too slow if the engine read all the data, so the engine re-writes your query to use a pre-materialized result. B-tree indexes made the first relational databases possible, and there are now many flavors of materialization, from explicit materialized views to OLAP-style caching and spatial indexes. Materialization is more relevant than ever in today’s heterogenous, distributed systems. If you are evaluating data engines, we describe what materialization features to look for in your next engine. If you are implementing an engine, we describe the features provided by Apache Calcite to design, maintain and use materializations.

Data profiling with Apache Calcite

Query optimizers and people have one thing in common: the better they understand their data, the better they can do their jobs. Optimizing queries is hard if you don't have good estimates for the sizes of the intermediate join and aggregate results. Data profiling is a technique that scans data, looking for patterns within the data such as keys, functional dependencies, and correlated columns. These richer statistics can be used in Apache Calcite's query optimizer, and the projects that use it, such as Apache Hive, Phoenix and Drill. We describe how we built a data profiler as a table function in Apache Calcite, review the recent research and algorithms that made it possible, and show how you can use the profiler to improve the quality of your data. A talk given by Julian Hyde at DataWorks Summit, San Jose, on June 14th 2017.

A smarter Pig: Building a SQL interface to Apache Pig using Apache Calcite

What if Apache Pig had a SQL front-end and query optimizer? What if Apache Calcite was able to use Pig and MapReduce to run queries? In this project, we aimed to answer both questions by adding a Pig adapter for Calcite. In this talk, we describe Calcite's adapter framework, how we used it to write a Pig adapter, and how you can use this SQL interface to Pig for interactive and long-running queries. A talk given by Eli Levine and Julian Hyde at Apache: Big Data, Miami, on May 17th, 2017.

Data Profiling in Apache Calcite

Streaming SQL

Streaming is necessary to handle data rates and latency, but SQL is unquestionably the lingua franca of data. Is it possible to combine SQL with streaming, and if so, what does the resulting language look like? Apache Calcite is extending SQL to include streaming, and Apache Apex is using Calcite to support streaming SQL. In this talk, Julian Hyde describes streaming SQL in detail and shows how you can use streaming SQL in your application. He also describes how Calcite’s planner optimizes queries for throughput and latency. Julian Hyde gave this talk at Apex Big Data World, Mountain View, on April 4, 2017.

Streaming SQL (at FlinkForward, Berlin, 2016/09/12)

A talk given by Julian Hyde at FlinkForward, Berlin, on 2016/09/12. Streaming is necessary to handle data rates and latency, but SQL is unquestionably the lingua franca of data. Is it possible to combine SQL with streaming, and if so, what does the resulting language look like? Apache Calcite is extending SQL to include streaming, and Apache Flink is using Calcite to support both regular and streaming SQL. In this talk, Julian Hyde describes streaming SQL in detail and shows how you can use streaming SQL in your application. He also describes how Calcite’s planner optimizes queries for throughput and latency.

Streaming SQL

Streaming is necessary to handle IoT data rates and latency but SQL is unquestionably the lingua franca of data. Apache Samza and Apache Storm have new high-level query interfaces based on standard SQL with streaming extensions, both powered by Apache Calcite. Calcite's relational algebra allows query optimization and federation with data-at-rest in databases, memory, or HDFS. A talk given by Julian Hyde at Hadoop Summit, San Jose, on 2016/06/29.

Streaming SQL