The Innovation Index analyses the market attractiveness, business model maturity and infrastructure and support impact of a given sector or technology. This is the first part of the new Innovation Index for Hadoop. It aims at analyzing Hadoop market attractiveness through facts, figures and key words analysis, as collected through WAI networks. The second part will evaluate infrastructure and support impact, and the last part will analyze business model maturity to come to a final index value. To know more, your can watch “The Internet of Things, of What, Exactly?“, an end-to-end Innovation Index from earlier this year.

Hard Facts Analysis: Hadoop between fiction and reality

Below is a pull-push fact analysis based on articles collected through WAI networks from September 2014 to October 2014. It is based on a relative weight analysis in context, and relates to facts as listed below the graph, with no specific order.

1. Nick Bostrom, Live Science: Super Intelligence is a scary scenario for many users
2. Derick Harris, GigaOm: NortonWorks develops purpose-built object store for Hadoop, developing its connectivity.
3. Joe Panettieri, Information Management: Tata Communications, Horizons Ventures and Access Industries invest in Watson competitor
4. YEVGENIY SVERDLIK, Data Centre Knowledge: Germany and Japan eyed by Data Centre real-estate giant Digital Realty
5. Richi Jennings, Forbes: Cloud is primarily business driven value, with a technology flavor
6. Todd Goldman, Informatica: Hadoop is reaching market critical mass
7. Steven Saslow, Information Technology Group: Business people still have no idea what cloud means
8. Lee Gomes, IEEE Spectrum: “The overeager adoption of big data is likely to result in catastrophes of analysis comparable to a national epidemic of collapsing bridges.”
9. Mike Combs, Veristorm: “Customer-centric marketing practices will drive everything from how promotions are constructed to how stores are stocked, decorated, and sited.”

Conclusion: The initial grade reflects the overall uncertainty related to such a wide topic and market opportunity for Hadoop, as a core environment for Big Data. The fact that we hear “fearful” voices enlightens the earliness of the market, as much as the “enthusiastic” voices does. There seem to be a lack of reality understanding, and although this so-called reality might include a range of risks to asset, companies and researches tend to push towards greater development to create Big Data type of services that would result in high benefits. The investment stage whereby companies, network, platform and application suppliers position themselves on the value-chain to sustain a later full roll-out, is well underway, which rises the initial grade to 2.2.

Key market trends impact: When Big Data misses Big Data

Below is a trend-mapping analysis weighing the information impact and its correlation with Hadoop developments and market rise. It is based on a direct/indirect impact, positive/negative impact that enables to assess a relative weight in context. Information was collected from September 2014 to October 2014 through WAI Networks, and sources are listed below the graph.

1. Olivier Chicheportiche, ZDNet: Cost of cyber-attacks on French companies have risen by 20.5% over a year, 9.3% in the US, 200% in 5yrs. US costs: $12.7m per company per year
2. Bob Violino, Information Management: “BI Survey 14, BARC’s annual survey of more than 3,000 BI professionals worldwide, shows that 30% of BI users now consider themselves “power users,” higher than the previously assumed 20%.”
3. Jason Davison, Smarter Computing Blog: “In a recent ITG paper, VSC was shown to have approximately 72 percent better Total Cost of Ownership (TCO) compared to competitive solutions in large multi-petabyte infrastructures.”
4. Kerry Butters, XEN: “Jobs saw a huge 68% decline in results, Real estate fell by 63%, Movies dropped by 36%, Insurance saw an 11% decline “Pigeon is doing nothing to affect the quality of your traffic and is just giving you less of it,” wrote Adam Dorfman in a blog post.”
5. Barry Jaruzelski, Innovation Excellence: “The rate of growth in innovation expenditures for the Global Innovation 1000 slowed sharply in 2014, to just 1.4 percent—the slowest rate of growth in the past 10 years for the 1,000 global companies that spent the most on R&D.”
6. Serdar Yegulalp, InfoWorld: “One estimate puts its growth at nearly 60 percent year-over-year, with a market of $50 billion by 2020.”
7. Timothy Prickett Morgan, Enterprise Tech: The sweet spot for initial deployments is a starter cluster that costs on the order of $150,000 to $250,000. By the standards of large shared memory systems or commercial data warehouses, this is not a lot of money. A Hadoop distribution, depending on the features, costs roughly $7,000 per node for a software license, but a data warehouse with a big relational database sitting on it can cost $40,000 per TB and an in-memory database costs even more than that.”
8. Accenture: “they say that a lack of IT capability is their main impediment to setting up a dedicated claims analytics organization. (85% Insurance claims executives in Europe and Latin America).

Conclusion: The intermediate grade is lowered due to the lack of clearer proven market impact understanding of Big Data and Hadoop, which is mainly due to a careful approach to new development combined to a raising awareness of dysfunctions and limits of currently used strategies, tools and policies, carried unto new platforms. Although with indirect impact, the economic downturn now globalized and rise of security threats is preventing the core market to take ground and generate a first loop of the virtuous circle needed to spur Big Data. Strategies and development plans should take into account a tendency to “think twice” that might delay decision making for business stakeholders and put emphasis on messages shared with them which should reinsure and encourage new visions, new cultures to reach a Big Data and Hadoop sufficient level of readiness from market. This systemic lack of belief currently lowers the intermediate grade to 1.7.

Semantic Analysis: The power of enthusiasm

Below is a semantic analysis of “Hadoop”, listing and weighing all key words shared with the “Hadoop” term in articles shared on WAI networks from September to October 2014. It is based on a relative weight analyses in context and relates to sources as listed below the graph.

1. Zach Taiji, Smart Data Collective: “In layman’s terms, Hadoop is a framework for creating and supporting big data and large scale processing applications – something that a traditional software isn’t able to do.”
2. SAS Analytics: “With SAS Visual Analytics, organizations can seamlessly manipulate, analyze and view data of any size, helping them transform it into stronger customer relationships, increased sales and efficiency, even healthier patients”
3. Greg Satell, Forbes: “Yet big data and technology are clearly revolutionizing marketing.”
4. David S Linthicum, Cloud Cruiser: “Backing up a bit, enterprises move to hybrid- and multi-clouds for a variety of reasons:”
5. Lauris Veips, Cloud Tweaks: “APIs or application programming interfaces of popular services are used by numerous startups to bring valuable services to the general public.”
6. Paul DiMarzio, Smart Computing Blog: “Having an advanced Hadoop distribution on the mainframe is just part of the story; it’s also necessary to have a secure, easy to use, fast and low cost mechanism for populating System z Hadoop clusters with System z data.”
7. Sharon Gaudin, Computer World: “I actually do think this is a valid concern and it’s really an interesting one. It’s a remote, far future danger but sometime we’re going to have to think about it. If we’re at all close to building these super-intelligent, powerful machines, we should absolutely stop and figure out what we’re doing.“
8. Timothy Prickett Morgan, Enterprise Tech: “But the fact remains that the actual installed base of Hadoop clusters remains a lot smaller than many might expect given the amount of innovation that is going on around the platform”
9. CHRISTIAN LIENHARDT, Les Echos: “A partir d’un algorithme inédit prenant en compte 250 paramètres, Jérémy Wies, vingt-quatre ans, a élaboré un logiciel de gestion automatisée des flottes de véhicules pour le transport sanitaire.”
10. Julien Bonnet, L’Usine Nouvelle: “Indétrônable depuis 10 ans, Apple termine à la première place et Google regagne aux dépens du coréen Samsung la deuxième place perdue l’an dernier.”
11. John Ellett, Forbes: “We describe them as “digitally connected families that love to shop and demand great value.”
12. Ryan Goldman, Wall Street Technology Association: “firms are now able to not only fulfill increasingly stringent regulatory demands without the capital burden of specialized systems, but also take on more advanced workloads and realize new strategic benefits from the same data that they need to keep on-hand for compliance reporting.”

Conclusion: The final grade is largely supported by an overall shared enthusiasm about the assumed benefits of Hadoop use in companies. Although some constraints and reluctancy to create more systemic challenges, potentially ending, remain present in the overall conversation, the business benefits and operational efficiency as presented by analysts and experts currently offset the “pessimistic” views. The need for better prevalent models and contingency plans proving there is a secure model to roll-out for sustainable growth is at its paradigm, with words such as “danger” continuing to accompany the “value, benefits, seamless” positive attributes which make up for the vast majority of positive terms. The overall enthusiasm communicated by innovators therefore rises the final grade to 2.9.

In the next few days we will analyse the business model maturity and infrastructure and support impact to conclude on the general Innovation Index for Hadoop.