Table 2 Description of the Hadoop related projects/ecosystems

Avro

• Avro is a framework for performing remote procedure calls and data serialization.

http://avro.apache.org

Flume

• Flume is a tool for harvesting, aggregating and moving large amounts of log data in and out of Hadoop.

http://flume.apache.org

HBase

• Based on Google’s Bigtable, HBase is an open-source, distributed, versioned, column-oriented store that sits on top of HDFS. HBase is column-based rather than row-based, which enables high-speed execution of operations performed over similar values across massive datasets.

http://hbase.apache.org

HCatalog

• An incubator-level project at Apache, HCatalog is a metadata and table storage management service for HDFS.

http://Incubator.apache.org/hcatalog/

Hive

• Hive provides a warehouse structure and SQL-like access for data in HDFS and other Hadoop input sources

http://hive.apache.org

Mahout

• Mahout is a scalable machine-learning and data mining library.

http://mahout.apache.org

Oozie

• Oozie is a job coordinator and workflow manager for jobs executed in Hadoop, which can include non-MapReduce jobs.

http://oozie.apache.org

Pig

• Pig is a framework consisting of a high-level scripting language (Pig Latin) and a run-time environment that allows users to execute MapReduce on a Hadoop cluster.

http://pig.apache.org/docs/r0.7.0/piglatin_ref2.html

Sqoop

• Sqoop (SQL-to-Hadoop) is a tool which transfers data in both directions between relational systems and HDFS or other Hadoop data stores, e.g. Hive or HBase.

http://sqoop.apache.org

ZooKeeper

• ZooKeeper is a service for maintaining configuration information, naming, providing distributed synchronization and providing group services.

http://zookeeper.apache.org

YARN

• YARN is a resource-management platform responsible for managing compute resources in clusters and using them for scheduling of users’ applications.

http://hadoop.apache.org/docs/r2.3.0/hadoop-yarn/hadoop-yarn-site/YARN.html

Cascading

• Cascading is an alternative API to Hadoop MapReduce. Cascading now has support for reading and writing data to and from a HBase cluster.

http://wiki.apache.org/hadoop/Hbase/Cascading

Twitter Storm

• Twitter Storm is a free and open source distributed real time computation system.

http://storm.incubator.apache.org/

High performance computing cluster (HPCC)

• HPCC is an open source, data-intensive computing system platform developed by LexisNexis Risk Solutions

http://hpccsystems.com/

Dremel

• Dremel is a scalable, interactive ad-hoc query system for analysis of read-only nested data

http://research.google.com/pubs/pub36632.html