Hadoop: Structuring the future

The Innovation Index analyses the innovation impact of a given trend or technology. Infrastructure and support is the last article of a series of three-dimensional analyses to provide a final Innovation Index for Hadoop as a supporting technology to Big Data. Two other articles estimate the market attractiveness and business model maturity of Hadoop. The infrastructure and support part reviews trends for technical readiness to market, structural readiness to market, together with their wider legal, social and economics impact.

Technical readiness to market: Hybrid models, risks and opportunities

Technical Readiness Analysis
Technical Readiness Analysis

Technology: Big Solutions for Big Problems

1. “Intel has customized some processors from the previous Xeon E7 v2 generation for Oracle database solutions, in order to meet Google, Amazon and Facebook requirements.” NoSQLCenter

2. “The objective of Google Cloud BigTable is to offer cloud space for companies and organizations willing to store great amounts of data, which could also be combined to analysis tools.” NoSQLCenter

3. “Big data technologies are capable of solving both questions when they are implemented under governance that supports new solutions.” Svetlana Sicular, Gartner

4. “But along with these cost advantages come challenges including security, data privacy, regulatory compliance, access management, and auditing that prevent companies from moving everything to the cloud. A hybrid cloud solution can address these challenges while delivering the cost savings that cloud provides. ” Data Informed

Expertise: Converging knowledges

5. “In 2011, McKinsey Global Institute warned of an upcoming shortage of “140,000 to 190,000 people with deep analytic skills” and “1.5 million managers and analysts”. Gartner estimated in 2012 that only one-third of the 4.4m IT jobs needed globally to support big data would be filled by 2015. This year, the EU said it expected upwards of a 825,000 labour shortfall by 2020.” GE Look Ahead

6. “Alas, while many developers are beginning to recognize the synergies and overlaps between BI and big data, quite a few still consider and run both in individual silos.” Boris Evelson, Forrester

7. “Gartner has recently predicted that by 2017, most business users in addition to analysts in organisations will have access to self-service business intelligence (BI) tools to prepare data for analysis.” Chloe Green, Information Age

Research and Development: Connecting to the Future

8. “With industry analysts predicting that the majority of new application deployments will be SaaS-based by 2017, the requirement of having connected data should not be negotiable. It is a must have.” Julie Lockner, Informatica

9. “Hölzle is the man who oversees Google’s worldwide network of data centers, and he believes that efficiencies brought by devices such as Internet-connected thermostats, lighting systems, and self-driving cars will balance out the extra power needed to drive our computing centers.” Ariel Zambelich, WIRE

10. “Much of what is being done with wearable devices is happening simply because it can be done. However, several users still are not sure about wearables and whether they want to walk around with devices strapped to them all day. Is this the paradox of wearables?” Rohit Yadav, Data Science Central

WAI Comment: Analysts seem to agree on the requirements of Big Data and therefore hadoop based solutions needed, as much as they agree on the potential market opportunities and threats. They question the ability of organizations to support the roll-out of such solutions given the lack of specialized resources able to build Hadoop based solutions and the difficulty they have to adapt to new governance models. Overall, the initial technical readiness to market for Hadoop reaches an average 2.5 out of 5.

Structural readiness to market: It’s all about integration

Structural Readiness Analysis

Governance models: Identifying the right targets

1. “Regardless of who creates or manages the applications of the enterprise, it is important to remember that integration is to a large degree the nervous system that connects applications into business capabilities. As such, the CIO’s team has a critical and continuing role in managing this linkage.” Myles Suer, Informatica

2. “When we see industry shifts with a new technology, the second wave of the shift comes with partnerships. Embracing new technology among partners is a nod to the validity of the solution, and definitely increases the velocity of adoption.” Eric Wright, VMTurbo

3. “Being data-driven gives you a clear competitive advantage over your competitors. You take the right decision when they don’t. You know what’s working when they don’t. You get it.” Pierre Lechelle

Mid-management : redefining responsibilities

4. “To survive and thrive in this new world, marketers need to start using data integration to devise a host of new quantifiable marketing metrics that can analyze a number of factors such as number of wins, cost per win, and the ROI earned from specific marketing campaigns.” Mark Johnson, Loyalty360

5. “Deepak Taneja, chief technologist at RSA, the security division of Hopkinton-based IT storage firm EMC (NYSE: EMC), said it’s important for startups to think about integrating cybersecurity technology into a connected device starting from the first prototype.” Sara Castellanos, Boston Business Journal

6. “Just to be clear, no enterprise, no matter how large, has all the capabilities in house to bring a complex IoT product or service to market without some outside help along the way.” Aria Systems

Operational level: Coping with change

7. “In this report, “machine generated” data projects were 41.2% of all big data projects, rising from 23.7% in 2012. This jump was at the expense of “human sourced” data dropping to 20.9%. In addition, over 40% valued big data projects as a “vital part” or “important part” of their business. This demonstrates both a jump in operational big data use-case deployments and a large share of these deployments making a significant impact on the business”. Steve Smythe, SM360

8. Refining data as a must have for data analytics and business decision quality.

9. “With more than 90% of companies reporting some level of cloud adoption, stringent security requirements are necessary to keep information private. As adoption and use increases, security will remain a top priority. Yet, keeping private information private is easier said than done. In 2014 alone, security breaches increased 27.5%.” Aria Systems

WAI Comment: The radical changes implied by Big Data and Hadoop to enterprises are such that a complete unlearning and relearning process needs to take place throughout organisations. On the governance side, companies need to redefine their visions and objectives. On the management side, they need to open the strategic thinking models. On the operational side, they need to apply change in the midst of an immature market structure and business models. The level of challenge brings the intermediate grade to 2 out of 5.

Legal, Social and Economic impact

Hadoop Legal Social and Economics Impact
Hadoop Legal Social and Economics Impact

Legal: shaping a concrete framework for Big Data

1. “Concerns over automated decision-making are sometimes raised and, certainly, automated decision processing is particularly problematic under European Union data protection law. (…)  Accordingly, employers that use big data analytics in recruitment need to ensure that there is an element of human judgment involved in decision-making.” Susan McLean, Socially Aware Blog

2. “To secure today’s critical business asset – data — companies need to implement security frameworks that account for today’s dynamic and global commercial environment. There is simply no technology fix for the “where” of data security.” Diane D Reynolds, Privacy and Data Security Insight

3. “On Wednesday, April 8, the Federal Communications Commission (FCC) entered a consent decree and levied a $25 million civil penalty against AT&T to settle a data breach that exposed the information of nearly 280,000 customers. “Alexander Stout, JDSupra

Social: unanswered uncertainties

4. “And because each solution must have its own ecosystem, there are no synergies that can be leveraged to bring down costs. When you combine fractured experiences with higher costs, it’s not long before people give up and go back to the old way of doing things or try yet another solution only to become even more frustrated.” Dave Evans, TechCrunch

5. “This is exactly where doors will open to capture and use all this necessary information to later be able to implement a smart city.” No SQL Center

6. “Big Data has a tremendously positive potential, but just because of the catastrophes which could result from a Big Data breach – whether accidental or not –  it can create create much serious issues.” Jose Manuel Abad Linan, El Pais

7. “Encourage the habit of refining questions and relooking at questionable decisions. Never settle for easy answers and do not let your analytics group become an answering machine.” Kshira Saagar, Information Management 

Economics: a potential to be refined

8. “Around 30% of companies within the financial sector in Spain are using the opportunities offered by (Big Data and analytics) technologies to grow innovation.” No SQL Center

9. “I do agree that, leveraged properly, there’s much more value to be extracted from the cloud by government. Further, I agree that cloud technologies have sufficiently matured to the point that it is feasible to consider migrating mission critical applications. Yet, is it possible that the government’s “fear of commitment” is, in some ways, justified ?” Bobby Caudill, Informatica

10. “But enterprises should beware as the definition of “policy” varies wildly. Some say it’s the ability to control the resources apps’ use or where the apps run. Others view policy as governing the permissions and delivering security. Policy is all of that and more.” Pat Romansky, CloudExpo Blog

WAI Comment: External players supporting Big Data and Hadoop markets are taking a stance on opportunities. Although still under definition for the greatest part, the legal framework is starting to take action and protect end-users. Analysts ring alarms against the lack of clarity and common definitions of Big Data consequences (whether positive or negative). Finally, innovation specialists highlight the application potential that lays beyond the many doubts we will surely find innovative answers to. Still, the overall Infrastructure and Support impact for Hadoop hits a final grade of 2 out of 5.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s