原文鏈接：『 Spark 』1. spark 簡介
寫在前面
本系列是綜合了自己在學習spark過程中的理解記錄 ＋ 對參考文章中的一些理解 ＋ 個人實踐spark過程中的一些心得而來。寫這樣一個系列僅僅是為了梳理個人學習spark的筆記記錄，并非為了做什么教程，所以一切以個人理解梳理為主，沒有必要的細節(jié)就不會記錄了。若想深入了解，最好閱讀參考文章和官方文檔。
其次，本系列是基于目前最新的 spark 1.6.0 系列開始的，spark 目前的更新速度很快，記錄一下版本好還是必要的。
最后，如果各位覺得內(nèi)容有誤，歡迎留言備注，所有留言 24 小時內(nèi)必定回復，非常感謝。
Tips: 如果插圖看起來不明顯，可以：1. 放大網(wǎng)頁；2. 新標簽中打開圖片，查看原圖哦。
1. 如何向別人介紹 spark
Apache Spark? is a fast and general engine for large-scale data processing.
Apache Spark is a fast and general-purpose cluster computing system.
It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs.
It also supports a rich set of higher-level tools including :
- Spark SQL for SQL and structured data processing, extends to DataFrames and DataSets
- MLlib for machine learning
- GraphX for graph processing
- Spark Streaming for stream data processing
2. spark 誕生的一些背景
Spark started in 2009, open sourced 2010, unlike the various specialized systems[hadoop, storm], Spark’s goal was to :
-
generalize MapReduce to support new apps within same engine
- it's perfectly compatible with hadoop, can run on Hadoop, Mesos, standalone, or in the cloud. It can access diverse data sources including HDFS, Cassandra, HBase, and S3.
-
speed up iteration computing over hadoop.
- use memory + disk instead of disk as data storage medium
- design a new programming modal, RDD, which make the data processing more graceful [RDD transformation, action, distributed jobs, stages and tasks]
3. 為何選用 spark
- designed, implemented and used as libs, instead of specialized systems;
- much more useful and maintainable
- from history, it is designed and improved upon hadoop and storm, it has perfect genes;
- documents, community, products and trends;
- it provides sql, dataframes, datasets, machine learning lib, graph computing lib and activitily growth 3-party lib, easy to use, cover lots of use cases in lots field;
- it provides ad-hoc exploring, which boost your data exploring and pre-processing and help you build your data ETL, processing job;
4. Next
下一篇，簡單介紹 spark 里必須深刻理解的基本概念。
