ORC File Introduction

This document discusses improvements to ORC support in Apache Spark 2.3. It describes previous issues with ORC performance and compatibility in Spark. The current approach in Spark 2.3 introduces a new native ORC file format that provides significantly better performance compared to the previous Hive ORC implementation. It allows configuring the ORC implementation and reader type. The document also demonstrates ORC usage in Spark and PySpark. Benchmark results show the native ORC reader provides up to 15x faster performance for scans and predicate pushdown. Future work items are discussed to further improve ORC support in Spark.

SAP ASE 16 SP02 Performance Features

SAP Technology

Local Secondary Indexes in Apache Phoenix

Rajeshbabu Chintaguntla

The document discusses local secondary indexes in Apache Phoenix. Local indexes are stored in the same region as the base table data, providing faster index building and reads compared to global indexes. The write process involves preparing index updates along with data updates and writing them atomically to memstores and the write ahead log. Reads scan the local index and retrieve any missing columns from the base table. Local indexes improve write performance over global indexes due to reduced network utilization. The document provides performance results and tips on using local indexes.

7. Key-Value Databases: In Depth

Fabio Fumarola

DynamoDB is a key-value database that achieves high availability and scalability through several techniques: 1. It uses consistent hashing to partition and replicate data across multiple storage nodes, allowing incremental scalability. 2. It employs vector clocks to maintain consistency among replicas during writes, decoupling version size from update rates. 3. For handling temporary failures, it uses sloppy quorum and hinted handoff to provide high availability and durability guarantees when some replicas are unavailable.

Parquet overview

Parquet is a column-oriented storage format for Hadoop that supports efficient compression and encoding techniques. It uses a row group structure to store data in columns in a compressed and encoded column chunk format. The schema and metadata are stored in the file footer to allow for efficient reads and scans of selected columns. The format is designed to be extensible through pluggable components for schema conversion, record materialization, and encodings.

Choosing an HDFS data storage format- Avro vs. Parquet and more - StampedeCon...

StampedeCon

At the StampedeCon 2015 Big Data Conference: Picking your distribution and platform is just the first decision of many you need to make in order to create a successful data ecosystem. In addition to things like replication factor and node configuration, the choice of file format can have a profound impact on cluster performance. Each of the data formats have different strengths and weaknesses, depending on how you want to store and retrieve your data. For instance, we have observed performance differences on the order of 25x between Parquet and Plain Text files for certain workloads. However, it isn’t the case that one is always better than the others.

Polyglot persistence @ netflix (CDE Meetup)

Roopa Tangirala

Netflix’s architecture involves thousands of microservices built to serve unique business needs. As this architecture grew, it became clear that the data storage and query needs were unique to each area; there is no one silver bullet which fits the data needs for all microservices. CDE (Cloud Database Engineering team) offers polyglot persistence, which promises to offer ideal matches between problem spaces and persistence solutions. In this meetup you will get a deep dive into the Self service platform, our solution to repairing Cassandra data reliably across different datacenters, Memcached Flash and cross region replication and Graph database evolution at Netflix.

NoSql

Girish Khanzode

Optimizing Hive Queries

DataWorks Summit/Hadoop Summit

Apache Hive is a data warehousing system for large volumes of data stored in Hadoop. However, the data is useless unless you can use it to add value to your company. Hive provides a SQL-based query language that dramatically simplifies the process of querying your large data sets. That is especially important while your data scientists are developing and refining their queries to improve their understanding of the data. In many companies, such as Facebook, Hive accounts for a large percentage of the total MapReduce queries that are run on the system. Although Hive makes writing large data queries easier for the user, there are many performance traps for the unwary. Many of them are artifacts of the way Hive has evolved over the years and the requirement that the default behavior must be safe for all users. This talk will present examples of how Hive users have made mistakes that made their queries run much much longer than necessary. It will also present guidelines for how to get better performance for your queries and how to look at the query plan to understand what Hive is doing.

The Data Lake Engine Data Microservices in Spark using Apache Arrow Flight

Databricks

Apache Atlas: Tracking dataset lineage across Hadoop components

Apache Atlas provides centralized metadata services and cross-component dataset lineage tracking for Hadoop components. It aims to enable transparent, reproducible, auditable and consistent data governance across structured, unstructured, and traditional database systems. The near term roadmap includes dynamic access policy driven by metadata and enhanced Hive integration. Apache Atlas also pursues metadata exchange with non-Hadoop systems and third party vendors through REST APIs and custom reporters.

How Kafka Powers the World's Most Popular Vector Database System with Charles...

HostedbyConfluent

We use Kafka as the data backbone to build Milvus, an open-source vector database system that has been adopted by thousands of organizations worldwide for vector similarity search. In this presentation, we will share how Milvus uses Kafka to enable both real-time processing and batch processing on vector data at scale. We will walk through the challenges of unified streaming and batching in vector data processing, as well as the design choices and the Kafka-based data architecture.

An introduction to Apache Thrift

Mike Frampton

RocksDB compaction

MIJIN AN

The document discusses compaction in RocksDB, an embedded key-value storage engine. It describes the two compaction styles in RocksDB: level style compaction and universal style compaction. Level style compaction stores data in multiple levels and performs compactions by merging files from lower to higher levels. Universal style compaction keeps all files in level 0 and performs compactions by merging adjacent files in time order. The document provides details on the compaction process and configuration options for both styles.

LineairDBの紹介

Sho Nakazono

Quand utiliser MongoDB … Et quand vous en passer…

MongoDB

MyRocks Deep Dive

Yoshinori Matsunobu

Dynamic Column Masking and Row-Level Filtering in HDP

DataWorks Summit/Hadoop Summit

As enterprises around the world bring more of their sensitive data into Hadoop data lakes, balancing the need for democratization of access to data without sacrificing strong security principles becomes paramount. In this webinar, Srikanth Venkat, director of product management for security & governance will demonstrate two new data protection capabilities in Apache Ranger – dynamic column masking and row level filtering of data stored in Apache Hive. These features have been introduced as part of HDP 2.5 platform release.

Neo4j graph database

Prashant Bhargava

Apache Phoenix + Apache HBase

The document summarizes Apache Phoenix and HBase as an enterprise data warehouse solution. It discusses how Phoenix provides OLTP and analytics capabilities over HBase. It then covers various use cases where companies are using Phoenix and HBase, including for web analytics and time series data. Finally, it discusses optimizations that can be made to the schema design, queries, and writes in Phoenix to improve performance.

What's hot (20)

ORC improvement in Apache Spark 2.3

SAP ASE 16 SP02 Performance Features

Local Secondary Indexes in Apache Phoenix

7. Key-Value Databases: In Depth

Parquet overview

Choosing an HDFS data storage format- Avro vs. Parquet and more - StampedeCon...

Polyglot persistence @ netflix (CDE Meetup)

NoSql

Optimizing Hive Queries

The Data Lake Engine Data Microservices in Spark using Apache Arrow Flight

Apache Atlas: Tracking dataset lineage across Hadoop components

How Kafka Powers the World's Most Popular Vector Database System with Charles...

An introduction to Apache Thrift

RocksDB compaction

LineairDBの紹介

Quand utiliser MongoDB … Et quand vous en passer…

MyRocks Deep Dive

Dynamic Column Masking and Row-Level Filtering in HDP

Neo4j graph database

Apache Phoenix + Apache HBase

Viewers also liked

File Format Benchmark - Avro, JSON, ORC & Parquet

DataWorks Summit/Hadoop Summit

This document summarizes a benchmark study of file formats for Hadoop, including Avro, JSON, ORC, and Parquet. It found that ORC with zlib compression generally performed best for full table scans. However, Avro with Snappy compression worked better for datasets with many shared strings. The document recommends experimenting with the benchmarks, as performance can vary based on data characteristics and use cases like column projections.

ORC File and Vectorization - Hadoop Summit 2013

Eric Hanson and I gave this presentation at Hadoop Summit 2013: Hive’s RCFile has been the standard format for storing Hive data for the last 3 years. However, RCFile has limitations because it treats each column as a binary blob without semantics. Hive 0.11 added a new file format named Optimized Row Columnar (ORC) file that uses and retains the type information from the table definition. ORC uses type specific readers and writers that provide light weight compression techniques such as dictionary encoding, bit packing, delta encoding, and run length encoding — resulting in dramatically smaller files. Additionally, ORC can apply generic compression using zlib, LZO, or Snappy on top of the lightweight compression for even smaller files. However, storage savings are only part of the gain. ORC supports projection, which selects subsets of the columns for reading, so that queries reading only one column read only the required bytes. Furthermore, ORC files include light weight indexes that include the minimum and maximum values for each column in each set of 10,000 rows and the entire file. Using pushdown filters from Hive, the file reader can skip entire sets of rows that aren’t important for this query. Columnar storage formats like ORC reduce I/O and storage use, but it’s just as important to reduce CPU usage. A technical breakthrough called vectorized query execution works nicely with column store formats to do this. Vectorized query execution has proven to give dramatic performance speedups, on the order of 10X to 100X, for structured data processing. We describe how we’re adding vectorized query execution to Hive, coupling it with ORC with a vectorized iterator.

Structor - Automated Building of Virtual Hadoop Clusters

This document describes Structor, a tool that automates the creation of virtual Hadoop clusters using Vagrant and Puppet. It allows users to quickly set up development, testing, and demo environments for Hadoop without manual configuration. Structor addresses the difficulties of manually setting up Hadoop clusters, particularly around configuration, security testing, and experimentation. It provides pre-defined profiles that stand up clusters of different sizes on various operating systems with or without security enabled. Puppet modules configure and provision the Hadoop services while Vagrant manages the underlying virtual machines.

Adding ACID Updates to Hive

Protecting Enterprise Data in Apache Hadoop

From Hadoop Summit 2015, San Jose From Apache BigData 2016, Vancouver Hadoop has long had strong authentication via integration with Kerberos, authorization via User/Group/Other HDFS permissions, and auditing via the audit log. Recent developments in Hadoop have added HDFS file access control lists, pluggable encryption key provider APIs, HDFS snapshots, and HDFS encryption zones. These features combine to give important new data protection features that every company should be using to protect their data. This talk will cover what the new features are and when and how to use them in enterprise production environments. Upcoming features including columnar encryption in the ORC columnar format will also be covered.

Data protection2015

Hadoop has long had strong authentication via integration with Kerberos, authorization via user/group/other HDFS permissions and auditing via the audit log. Recent developments in Hadoop have added HDFS file access control lists, pluggable encryption key provider APIs, HDFS snapshots, and HDFS encryption zones. These features combine to given important new data protection features that every company should be using to protect their data. This talk will cover what the new features are and when and how to use them in enterprise production environments. Upcoming features including columnar encryption in the ORC file format will also be covered.

Plugging the Holes: Security and Compatability in Hadoop

Next Generation MapReduce

The next generation of Hadoop MapReduce Arun C. Murthy presented the plans for the next generation of Apache Hadoop MapReduce. The MapReduce framework has hit a scalability limit around 4,000 machines. We are developing the next generation of MapReduce that factors the framework into a generic resource scheduler and a per-job, user-defined component that manages the application execution. Since downtime is more expensive at scale high-availability is built-in from the beginning; as are security and multi-tenancy to support many users on the larger clusters. The new architecture will also increase innovation, agility and hardware utilization. More information and video available at: http://developer.yahoo.com/blogs/hadoop/posts/2011/02/hug-feb-2011-recap/

Bay Area HUG Feb 2011 Intro

Next Generation Hadoop Operations

Optimizing Hive Queries

Hadoop Security Architecture

Strata London 2016: The future of column oriented data processing with Arrow ...

Sql on everything with drill

Mapreduce total order sorting technique

Uday Vakalapudi

1) Total order sorting is another kind of sorting technique, where map output keys are sorted across all the reducers. 2) This technique uses, where you want to extract the most popular URLs from a web graph. 1) By default Mapreduce uses HashPartitioner as its Partitioner class, which partitions using a hash of the map output keys. 2) Also HashPartitioner ensures that all records with the same map output key goes to the same reducer, but it doesn’t perform total sorting of the map output keys across all the reducers. 3) For this reason only TotalOrderPartitioner class is introduced, which is by default packed with the Hadoop distribution. 1) If you want to work with Total order sorting, we need to create Partition file, and then we have to run Mapreduce job using TotalOrderPartitioner class. 2) We will create partition file, by using InputSampler class, which is used to do sampling of the whole dataset. 3) There are basically two kinds of samplers that we mostly use. 4) First one is RandomSampler, which is mainly used to pick random samples from the original dataset. And the second one is, IntervalSampler, which is mainly used to pick the sample for every R number of records. In the practical demonstration I have used RandomSampler class to pick the samples from Original dataset. 5) Once all the meaningful samples are extracted from the dataset, it will sort those keys, and pick N-1 keys from those sorted keys where N is number of reducers and it places in a Partition file which is used for Total order sorting. 1) This is an overview of Total Order Sorting, here it show how it generates the Partition file and also it shows how the Mapreduce job uses this Partition file during Total Order Sorting. 1) This is a code Sample for Total Order Sorting, in this we have specified the sampler object as RandomSample class. And we also set the Number of reducers using setNumReduceTasks(). And also we specified the Partionfile location unsing setPartionfile() of TotalOrderPartitioner class. And at last we have used writePartitionFile() of InputSampler class for creating Partition file.

Hive integration: HBase and Rcfile__HadoopSummit2010

Yahoo Developer Network

Strata NY 2016: The future of column-oriented data processing with Arrow and ...

ORC File & Vectorization - Improving Hive Data Storage and Query Performance

Hive’s RCFile has been the standard format for storing Hive data for the last 3 years. However, RCFile has limitations because it treats each column as a binary blob without semantics. The upcoming Hive 0.11 will add a new file format named Optimized Row Columnar (ORC) file that uses and retains the type information from the table definition. ORC uses type specific readers and writers that provide light weight compression techniques such as dictionary encoding, bit packing, delta encoding, and run length encoding — resulting in dramatically smaller files. Additionally, ORC can apply generic compression using zlib, LZO, or Snappy on top of the lightweight compression for even smaller files. However, storage savings are only part of the gain. ORC supports projection, which selects subsets of the columns for reading, so that queries reading only one column read only the required bytes. Furthermore, ORC files include light weight indexes that include the minimum and maximum values for each column in each set of 10,000 rows and the entire file. Using pushdown filters from Hive, the file reader can skip entire sets of rows that aren’t important for this query. Columnar storage formats like ORC reduce I/O and storage use, but it’s just as important to reduce CPU usage. A technical breakthrough called vectorized query execution works nicely with column store formats to do this. Vectorized query execution has proven to give dramatic performance speedups, on the order of 10X to 100X, for structured data processing. We describe how we’re adding vectorized query execution to Hive, coupling it with ORC with a vectorized iterator.

Data Eng Conf NY Nov 2016 Parquet Arrow

- Arrow and Parquet are open source projects focused on column-oriented data formats for efficient in-memory (Arrow) and on-disk (Parquet) analytics. - They allow for interoperability across systems by eliminating the overhead of data serialization and enabling common data representations. - Column-oriented formats improve performance by reducing storage needs, enabling projection of only needed columns, and better utilization of CPU/memory through cache locality and vectorized processing.

Hive and Apache Tez: Benchmarked at Yahoo! Scale

This document discusses benchmarking Hive at Yahoo scale. Some key points: - Hive is the fastest growing product on Yahoo's Hadoop clusters which process 750k jobs per day across 32500 nodes. - Benchmarking was done using TPC-H queries on 100GB, 1TB, and 10TB datasets stored in ORC format. - Significant performance improvements were seen over earlier Hive versions, with 18x speedup over Hive 0.10 on text files for the 100GB dataset. - Average query time was reduced from 530 seconds to 28 seconds for the 100GB dataset, and from 729 seconds to 172 seconds for the 1TB dataset.

Viewers also liked (20)

File Format Benchmark - Avro, JSON, ORC & Parquet

ORC File and Vectorization - Hadoop Summit 2013

Structor - Automated Building of Virtual Hadoop Clusters

Adding ACID Updates to Hive

Protecting Enterprise Data in Apache Hadoop

Data protection2015

Plugging the Holes: Security and Compatability in Hadoop

Next Generation MapReduce

Bay Area HUG Feb 2011 Intro

Next Generation Hadoop Operations

Optimizing Hive Queries

Hadoop Security Architecture

Strata London 2016: The future of column oriented data processing with Arrow ...

Sql on everything with drill

Mapreduce total order sorting technique

Hive integration: HBase and Rcfile__HadoopSummit2010

Strata NY 2016: The future of column-oriented data processing with Arrow and ...

ORC File & Vectorization - Improving Hive Data Storage and Query Performance

Data Eng Conf NY Nov 2016 Parquet Arrow

Hive and Apache Tez: Benchmarked at Yahoo! Scale

Similar to ORC File Introduction

Inside hadoop-dev

Availability and Integrity in hadoop (Strata EU Edition)

The document discusses data availability and integrity in Apache Hadoop. It explains that Hadoop uses replication and checksums to ensure data safety. While the NameNode is a single point of failure, Hadoop 1 provides cold failover capability using Linux HA clustering. Hadoop 2 will introduce live NameNode failover using ZooKeeper. The goal is full stack high availability to make the entire Hadoop system resilient to planned and unplanned outages.

HA Hadoop -ApacheCon talk

Hadoop provides high availability through replication of data across multiple nodes. Replication handles data integrity through checksums and automatic re-replication of corrupt blocks. Rack failures are reduced by dual networking and more replication bandwidth. NameNode failures are rare but cause downtime, so Hadoop 1 adds cold failover of Namenodes using VMware HA or RedHat HA. Hadoop 2 introduces live failover of Namenodes using a quorum journal manager to eliminate single points of failure. Full stack high availability adds monitoring and restart of all services.

Mobile Development Meets Semantic Technology

Blue Slate Solutions

Orange County HUG - Agile Data on HDP

The document discusses strategies for developing agile analytics applications on Hadoop, emphasizing an iterative approach where the data model and insights evolve through exploration of data in an interactive web application, rather than trying to design insights up front, in order to discover insights rather than define them. It also highlights using techniques like storing data in documents rather than relational structures and using Pig for its ability to handle diverse data formats.

An Introduction to Spring Data

Oliver Gierke

LA HUG - Agile Analytics Applications on HDP

The document discusses publishing structured event data to applications for analytics. It describes serializing event data from sources like emails and logs into Avro documents and loading them into Pig for processing. The data is then published from Pig to databases and analytics stacks using various options like ElasticSearch, MongoDB, HBase, and Hive/HCatalog for exploration and building analytics applications. Code examples demonstrate loading Avro data into Pig, illustrating the data schema, and publishing the data from Pig to MongoDB. The overall approach emphasizes agility, iteration, and flexibility in building analytics applications on Hadoop.

Hadoop: today and tomorrow

Cloud Consolidation with Oracle (RAC) - How much is too much?

Markus Michalewicz

This document discusses database consolidation on Oracle RAC systems in cloud environments. It covers general considerations for database consolidation including memory management, CPU management, and database resource management. The document also discusses Oracle RAC-specific considerations for consolidation limits per server and cluster limits. It provides an overview of common database cloud architectures and guidelines for evaluating workload characteristics during capacity planning for database consolidation.

Sentri SharePoint Performance webinar

Sentri

$DB2 z/OS & Java - What\'s New?$ $DB2 z/OS & Java - What\'s New?$

DB2 z/OS & Java - What\'s New?

Laura Hood

Hortonworks: Agile Analytics Applications

russell_jurney

The document discusses building agile analytics applications using Hadoop. It recommends setting up an environment where insights can be repeatedly produced through iterative and interactive exploration of data. The document emphasizes making an application for exploring data rather than trying to design insights directly. Insights are discovered through many iterations of refining the data and interacting with it.

Agile analytics applications on hadoop

The document discusses strategies for developing agile analytics applications using Hadoop, emphasizing an iterative approach where data is explored interactively to discover insights which then form the basis for shipped applications, rather than trying to design insights up front. It recommends setting up an environment where insights are repeatedly produced and shared with the team using an interactive application from the start to facilitate collaboration between data scientists and developers.

Compaction and Splitting in Apache Accumulo

The document discusses compaction and splitting in Apache Accumulo distributed key-value stores. It explains that Accumulo tables are divided into non-overlapping ranges called tablets, and that compaction merges sorted files within a tablet into a single file to improve read performance. Splitting divides large tablets into two in order to balance workload. The document provides details on Accumulo's and HBase's compaction algorithms and how they determine when to compact and split tablets.

Introduction to DDD

Radosław Mejer

Domain-driven design (DDD) is a software development approach that centers around modeling a domain into objects based on how an expert in that domain would naturally describe it. The presentation introduces key DDD concepts like ubiquitous language, context, domain models using entities, value objects, repositories, services, and commands. It notes pros of DDD like separated business rules, readable code, and ease of maintenance, as well as cons like code overhead and a steep learning curve. The presentation concludes by thanking the audience.

Ozone and HDFS’s evolution

The document discusses scaling challenges with HDFS and proposed solutions from Hortonworks called HDDS and Ozone. HDFS scales well for data and IO but has limitations scaling the namespace beyond 500 million files. HDDS aims to scale the block layer using block containers which can reduce block reports. Ozone uses a flat key-value namespace that is easier to shard and scale beyond billions of objects compared to HDFS hierarchical namespace. It also provides an HDFS compatible filesystem called OzoneFS. Together HDDS and Ozone aim to retain HDFS features while scaling to exabytes of data and trillions of files.

ORC 2015

t3rmin4t0r

The document discusses the Optimized Row Columnar (ORC) file format. ORC provides columnar storage, row groups, optimized compression, vectorized processing and other features to improve performance of Hive queries up to 100x. It describes how ORC has been implemented at companies like Facebook and Spotify to significantly reduce storage needs, improve query performance and reduce CPU usage. The document outlines various optimizations in ORC including column projection, stripe splitting, statistics gathering, vectorized processing using SIMD and improvements in compression and encoding of integer, double and string data types.

Storage Characteristics Of Call Data Records In Column Store Databases

David Walker

This document summarizes the storage characteristics of call data records (CDRs) in column store databases. It discusses what CDRs are, what a column store database is, and how efficient column stores are for storing CDR and similar machine-generated data. It provides details on the structure and content of sample CDR data, how the data was loaded into a Sybase IQ column store database for testing purposes, and the results in terms of storage characteristics and what would be needed for a production environment.

Java and Mongo

Marcio Mangar

This document provides an introduction to using Java with MongoDB. It begins with an agenda that covers an introduction, motivations for using MongoDB, and coding with Java and MongoDB. The introduction section defines what MongoDB is and is not, including that it is a NoSQL document database that uses BSON and sharding to improve performance, availability, and scalability. The motivations section discusses when MongoDB would be a good fit, such as for fast querying, temporary data storage, and horizontally scaling applications. The coding section demonstrates how to connect and install drivers in Java, use annotations to map classes to collections, and perform common operations like creating, deleting, querying, and updating documents.

VA Code Completion

ESUG

This document summarizes the code completion features of VA Smalltalk. It describes the introduction of code completion to address limited recall, provides the release history, reviews the new features in upcoming releases like support for inspectors and debuggers, describes the different contexts in which code completion is available, lists the types of suggestions provided, and outlines the 150ms workflow to provide suggestions.

Similar to ORC File Introduction (20)

Inside hadoop-dev

Availability and Integrity in hadoop (Strata EU Edition)

HA Hadoop -ApacheCon talk

Mobile Development Meets Semantic Technology

Orange County HUG - Agile Data on HDP

An Introduction to Spring Data

LA HUG - Agile Analytics Applications on HDP

Hadoop: today and tomorrow

Cloud Consolidation with Oracle (RAC) - How much is too much?

Sentri SharePoint Performance webinar

$DB2 z/OS & Java - What\'s New?$ $DB2 z/OS & Java - What\'s New?$

DB2 z/OS & Java - What\'s New?

Hortonworks: Agile Analytics Applications

Agile analytics applications on hadoop

Compaction and Splitting in Apache Accumulo

Introduction to DDD

Ozone and HDFS’s evolution

ORC 2015

Storage Characteristics Of Call Data Records In Column Store Databases

Java and Mongo

VA Code Completion

More from Owen O'Malley

Running An Apache Project: 10 Traps and How to Avoid Them

This document discusses 10 common mistakes made when running an Apache project and how to avoid them. These include starting development on GitHub without an IP agreement, keeping the project secret instead of promoting it, not fostering diversity among contributors, holding exclusive in-person meetings, including binary artifacts, having a high barrier for committer status, ignoring trademark issues, not rewarding open source work, engaging in stealth development, and licensing problems. The key recommendations are to build an open community, do work transparently, train employees on open source best practices, and ensure proper licensing.

Big Data's Journey to ACID

This document summarizes several big data systems and their approaches to providing ACID transactions. It finds that supporting object stores like S3 is becoming critical as streaming data ingest grows. Hive ACID is adding support for Presto and Impala queries. Iceberg is improving by adding delta files and Hive support. Overall, this is an active area of development that will continue changing in the next six months to better handle tasks like GDPR data removal and updating large datasets.

ORC Deep Dive 2020

This document provides an overview of the ORC file format. It describes the key requirements and design decisions, including file structure, stripe structure, encoding columns, run length encoding, compression, indexing, and versioning. It also discusses optimizations, debugging, and using ORC from SQL, Java, C++, and the command line. The document is intended to help users and developers better understand how ORC works.

Protect your private data with ORC column encryption

Fine-grained data protection at a column level in data lake environments has become a mandatory requirement to demonstrate compliance with multiple local and international regulations across many industries today. ORC is a self-describing type-aware columnar file format designed for Hadoop workloads that provides optimized streaming reads but with integrated support for finding required rows quickly. Owen O’Malley dives into the progress the Apache community made for adding fine-grained column-level encryption natively into ORC format, which also provides capabilities to mask or redact data on write while protecting sensitive column metadata such as statistics to avoid information leakage. The column encryption capabilities will be fully compatible with Hadoop Key Management Server (KMS) and use the KMS to manage master keys, providing the additional flexibility to use and manage keys per column centrally.

Fine Grain Access Control for Big Data: ORC Column Encryption

Fine-grained data protection at a column level in data lake environments has become a mandatory requirement to demonstrate compliance with multiple local and international regulations across many industries today. ORC is a self-describing type-aware columnar file format designed for Hadoop workloads that provides optimized streaming reads, but with integrated support for finding required rows quickly. In this talk, we will outline the progress made in Apache community for adding fine-grained column level encryption natively into ORC format that will also provide capabilities to mask or redact data on write while protecting sensitive column metadata such as statistics to avoid information leakage. The column encryption capabilities will be fully compatible with Hadoop Key Management Server (KMS) and use the KMS to manage master keys providing the additional flexibility to use and manage keys per column centrally.

Fast Access to Your Data - Avro, JSON, ORC, and Parquet

The landscape for storing your big data is quite complex, with several competing formats and different implementations of each format. Understanding your use of the data is critical for picking the format. Depending on your use case, the different formats perform very differently. Although you can use a hammer to drive a screw, it isn’t fast or easy to do so. The use cases that we’ve examined are: * reading all of the columns * reading a few of the columns * filtering using a filter predicate While previous work has compared the size and speed from Hive, this presentation will present benchmarks from Spark including the new work that radically improves the performance of Spark on ORC. This presentation will also include tips and suggestions to optimize the performance of your application while reading and writing the data. Finally, the value of having open source benchmarks that are available to all interested parties is hugely important and all of the code is available from Apache.

Strata NYC 2018 Iceberg

Hive tables are an integral part of the big data ecosystem, but the simple directory-based design that made them ubiquitous is increasingly problematic. Netflix uses tables backed by S3 that, like other object stores, don’t fit this directory-based model: listings are much slower, renames are not atomic, and results are eventually consistent. Even tables in HDFS are problematic at scale, and reliable query behavior requires readers to acquire locks and wait. Owen O’Malley and Ryan Blue offer an overview of Iceberg, a new open source project that defines a new table layout addresses the challenges of current Hive tables, with properties specifically designed for cloud object stores, such as S3. Iceberg is an Apache-licensed open source project. It specifies the portable table format and standardizes many important features, including: * All reads use snapshot isolation without locking. * No directory listings are required for query planning. * Files can be added, removed, or replaced atomically. * Full schema evolution supports changes in the table over time. * Partitioning evolution enables changes to the physical layout without breaking existing queries. * Data files are stored as Avro, ORC, or Parquet. * Support for Spark, Hive, and Presto.

Fast Spark Access To Your Complex Data - Avro, JSON, ORC, and Parquet

The landscape for storing your big data is quite complex, with several competing formats and different implementations of each format. Understanding your use of the data is critical for picking the format. Depending on your use case, the different formats perform very differently. Although you can use a hammer to drive a screw, it isn’t fast or easy to do so. The use cases that we’ve examined are: reading all of the columns reading a few of the columns filtering using a filter predicate While previous work has compared the size and speed from Hive, this presentation will present benchmarks from Spark including the new work that radically improves the performance of Spark on ORC. This presentation will also include tips and suggestions to optimize the performance of your application while reading and writing the data.

ORC Column Encryption