Walking the balance beam – why it’s smart to combine specific business goal with a common big data engine

Original article published on CiOStory.com

Data is the new currency. Social media is now the hottest way to raise consumer support ticket. Sentiment analysis are being more and more accurate in understanding customer desire. There is enough evidence these days that enterprises are no longer debating about the value of big data. The challenge, however, is how to implement its big data strategy.

There are two distinctly different approaches so far, centralized and departmental. Each of them comes with its advantages as well as challenges.

The centralized approach is mostly originated from the centralized Information Technology organization of the enterprise. They have experienced the complexity of data management first-hand, and they are in the deep realization of all the data compliance needs. Therefore it’s natural for the IT organization to recommend a centralized approach that usually starts with three steps: first, dig a data lake; Second, secure the data lake; Third, establish a process of who and how the data can be accessed.

The approach has many advantages such as data security. It’s also very effective in meeting compliance requirements such as customer privacy and confidentiality audits. On the other hand, it’s at a disadvantage because of the conflict between control and speed. The more centralized and secure the big data repository is, the less flexible it seems when there is a need to harness the data, store the data, or retrieve the data. The processes will get into the way so much that business units start to create local copies of spreadsheet “just for the sake of speed,” hence resulting in multiple copies of data, therefore, losing the value of having a “single version of the truth.” The data lake invested by the centralized IT organization would start to become a secure but less popular lake, often in need of frequent and active business users. When I speak to some of my CIO customers for large enterprises, they would tell me that they have been building their data lake for the last few years yet they are still “not done” and are under the pressure of seeking more concrete use cases and justifying much quicker ROIs (Return on Investment).

Figure 1 Art work by Kyoko Uchida, Sr. Designer, CA Technologies

The departmental approach is almost the opposite of the centralized approach. It usually got started with a well-defined objective (“We need to know what product features our customers are using so that we can prioritize our investment”), some rudimentary data collection (“I talked to a few customers and here is a spreadsheet of their feature usage mapping”), and the need of turning data into actions quickly (“Let’s make a change now by prioritizing this set of features and get them out to customers in 2 weeks”). While this approach can lead to quick time to value because of its laser focus, it creates a lot of repetitive work when each business unit started from zero for their big data needs. With the staff implementing these mostly being business experts rather than technology experts, the architectural design may lack the foresight for future growth. Lastly, there can be multiple copies of customer Twitter data stored at multiple locations by different business units, which is not only a waste of resources but also a hindering factor to prevented an enterprise to leverage the data in a more collaborative fashion.

Figure 2

Who wouldn’t want to have the best of both world? Same is true in our enterprise big data challenge. While this is not easy, we do see some success of a “hybrid approach” with a “common big data engine strategy” which focuses on the initial and early win (i.e. show me the value and Return on Investment) but also allowing thoughtful scalability and data sharing down the road. Rather than hiding a private copy secretly in the centralized approach or rebuilding everything from the ground up for each use case in the departmental approach, the hybrid approach offers speed to results while allowing enough room for potential growth. An illustration of this approach would look like this:

Figure – 3

To balance time to value and room for growth, there are three key considerations as we design and initiate our big data hybrid approach with common engine strategy:

1. Initiate with a joint mindset, shared by stakeholders who bring expertise both from business units as well as from centralized IT best practices. Note that the initial task force doesn’t have to have representatives from ALL business units, for the sake of speed. This is the most crucial part of your overall investment because it gains buy-in and builds the team. It’s important to glue the players with a clear objective after putting out all the risk and concerns from different groups on the table.
2. Identify a project with well-defined scope, objective and success criteria. Is our first win about increasing the demand-gen funnel, or about reducing the escalations and complaints from customers seeking support? The outcome of this exercise allows focus and ability to demonstrate the Time to Value and ROI. After identifying such project, some adjustment of the project team members may be needed to align further to ensure there is “some skin in the game” for each group participating. This is a key step to recruit additional stakeholders from within, which will further increase the ROI of the big data investment as it scales.
3. Incorporate architectural foresight into design since day one. This is probably the most debated concept. Some believe that you only worry about scalability when the demand is in front of you. A well-known example would be early-days Facebook where they were so small, and all they can afford to focus on is the immediate deliverables. Others believe that just like when building a house you leave electric outlets and plumbing hooks for future expansion, the same should apply so that additional use cases can be added and economy of scale can be achieved without too much duplicate work. In working with CIOs and business leaders for their strategic initiatives, I learned that the foresight is important for enterprise cost benefit.

To sum it up, we are still in an early stage of exploring the full benefit of big data implementation. Technology leaders should be mindful of the advantages and disadvantages of both centralized and departmental approach, thus orchestrate a hybrid approach with a common big data engine carefully so that enterprises can benefit from early wins followed by the economy of scale.