Feeding Data Curiosity: The Singular Reason I Joined Incorta

I have been very fortunate to have played diverse roles alongside some of the top leaders in some of the best-run enterprise software companies in the world. In my opinion, what has set these leaders and organizations apart is their consistent data curiosity coupled with the ability to dispassionately look at information and drive business decisions. I would attribute these two traits as the main reasons that their organizations consistently delighted their customers while outsmarting the competition.

What’s interesting about data curiosity, though, is that it has a time window beyond which it begins to wane and eventually just expires. The waning is only natural as leaders are barraged with critical decisions every moment of the day, and if their attention moves on to more pressing topics so does their data curiosity. And if the curiosity is not nurtured within a reasonable amount of time, they are compelled to make decisions that are either based on partial information or are instinct-driven, neither of which is a desirable outcome.

I’ll tell you up front that Incorta is the answer to feeding the data curiosity of every organization. But before we jump into the Incorta magic, let’s look at what is the current state of the data analytics market and the opportunities it presents for Incorta.

State of the Data Analytics Market 

Data analytics has remained top of mind for enterprises for the past two decades; so much so that Gartner tracks the space as three distinct multibillion dollar markets:

• Database Management Solutions for Analytics (includes data warehousing solutions): $37.1B TAM with 8% CAGR

• Data Integration Tools (such as ETL): $2.7B TAM with 6% CAGR

• Analytics and BI Platforms (front-end analytics): $3.5B TAM with 16% CAGR

Enterprises routinely use a combination of these three solutions to feed data curiosity, the reason for which lies in a bit of a history lesson in database management systems. 

From their inception, query engines have had one fundamental and severe limitation: They were designed for online transaction processing (OLTP) systems that consisted of lots of transaction writes, reads, and updates; they were not designed for the online analytical processing (OLAP) style queries that typically require scaling beyond three or four joins. Today, massive and complex data sets are par for the analytics course for any reasonable-sized organization. These data sets are pushing the limits of the query engine limitations, resulting in exponentially growing response times that have become huge execution bottlenecks for today’s data curious leadership. 

I. The Data Warehousing Market: Star Schemas to the Rescue?

This query engine limitation means that data warehouses—whether running in the cloud or on-premises—simply cannot handle queries against large (more than a couple hundred million rows) and complex (normalized data sets/3NF that need more than four joins) data sets. In order to overcome this limitation, additional concepts and technologies were introduced into the data warehousing value chain. 

One such key concept is the star schema. The star schema was specifically developed to denormalize 3NF data so that the query engine is able to handle these complex query use cases within acceptable response times. Unfortunately, rather than fixing performance issues completely, these schemas only temporarily delay the inevitable exponential growth in query response times. 

II. The Data Integration Tools Market: Adding More Moving Parts

One undesirable side effect of these schemas is that practitioners are forced to restrict the amount and complexity of data and need to be selective about which data subsets from different source applications are reshaped into the schema. To enable these selective reshaping and data transfers, they are forced to introduce another step to the value chain: the extract-transform-load (ETL) process. ETL was specifically invented to simplify the movement and reshaping of data to make it fit into the star schemas. 

This additional ETL steps adds to the moving parts. They are costly and brittle, and break often when the source application schemas change and are one of the main reasons for the significant delays incurred in IT delivering on new report requests. To be clear, not all ETL is bad ETL; the transformation part of ETL is often necessary where data formats are standardized across multiple sources or where data is enriched to make it more valuable for analysis.

III. The Analytics and BI Platforms Market: Limited to the Lower End of Scale and Complexity

Vendors in this market have primarily focused on integrating and streamlining the data warehousing value chain. However, given that they haven’t fixed the root of the issues—poor performing query engines—the integrated stack can scale only up to a few hundred million records and three to four joins. Or else they force IT to pre-aggregate their datasets before making them available for analysis, oftentimes losing data fidelity along with the ability to drill down from top-line analysis to a single line of record. 

Incremental Innovation Down the Wrong Path

Unfortunately, the latest innovation in the analytics market has been incremental and limited to that of moving the warehouses, data marts, and ETL pipelines to the cloud to leverage the value proposition of cloud infrastructures—on-demand scale and a pay-as-you-go charge model. Some vendors have gone a step further and offer data warehousing automation (DWA) tools to automate the same cumbersome data modeling and ETL tasks and try to lessen the pain. Which is like making a horse carriage faster rather than reimagining the transportation problem.

Enter Incorta: Calling Scale vs. Complexity a Faux Choice

Incorta has reimagined the query engine and solved the join challenge with its innovative Direct Data Mapping™ technology. Direct Data Mapping delivers queries in subseconds against the most complex and large data sets. As a result of this capability, it dramatically pares down the data warehousing value chain by eliminating data modeling and the unnecessary parts of ETL. It is the one-stop-shop for your business’s analytical needs without compromising on the scale or complexity of your data.

IT can now ingest all the data from all the application sources while inheriting their entity relationships. And since Incorta can ingest entire schemas, data is available for analysis at 100% fidelity. There is no information loss or the data inaccuracies typically introduced by data transformations or pre-aggregations.

The Incorta Promise

With Incorta, it takes only a couple of hours (if not minutes) for IT to pull in all the data needed to answer the first question asked by business. With Incorta’s subsecond query response times, any subsequent question can be answered in seconds as well, meaning IT and business users can peel the proverbial onion to get to the heart of any business challenge. IT certainly breathes easier as their backlog goes down dramatically with the quick turnaround times on new and change analytics requests, and business is excited to feed their data curiosity, all in real time!

I feel honored to join such an amazing team of innovators and am excited to be part of this journey of scaling Incorta to greater heights. Our collective promise remains to feed data curiosity in every organization of this world. Come learn more about us at incorta.com.