Apache Spark

Apache Spark(TM) is an open-source distributed general-purpose cluster computing framework with (mostly) in-memory data processing engine that can do ETL, analytics, machine learning and graph processing on large volumes of data at rest (batch processing) or in motion (streaming processing) with rich concise high-level APIs for the programming languages: Scala, Python, Java, R, and SQL.

In contrast to Hadoop’s two-stage disk-based MapReduce computation engine, Spark’s multi-stage (mostly) in-memory computing engine allows for running most computations in memory, and hence most of the time provides better performance for certain applications, e.g. iterative algorithms or interactive data mining.
- Mastering Apache Spark by Jacek Laskowski

See Apache Hadoop.

Libraries:

1. <strong>Spark SQL</strong> is Apache Spark’s module for working with structured data.
2. <strong>Spark Streaming</strong> makes it easy to build scalable fault-tolerant streaming applications.
3. <strong>MLlib</strong> is Apache Spark’s scalable machine learning library.
4. <strong>GraphX</strong> is Apache Spark’s API for graphs and graph-parallel computation.

News   |  Stack Overflow Q&amp;A   |  Community/Mailing Lists   |  Documentation   |  FAQ   |  IRC

• GENERAL