资源list：Github上关于大数据的开源项目、论文等合集-白红宇

资源list：Github上关于大数据的开源项目、论文等合集

阅读量：6307 次

发布时间：2019-06-22

本文共 24001 字，大约阅读时间需要 80 分钟。

Awesome Big Data

A curated list of awesome big data frameworks, resources and other awesomeness. Inspired by, , , & .

Your contributions are always welcome!

Frameworks

– framework for distributed processing. Integrates MapReduce (parallel processing)， YARN (job scheduling) and HDFS (distributed file system).

Distributed Programming

– distributed data processing and storage system originally developed at AddThis.

– run Spark on Hadoop MapReduce v1.

– a simple Java API for tasks like joining and data aggregation that are tedious to implement on plain MapReduce.

– collection of user-defined functions for Hadoop and Pig developed by LinkedIn.

– high-performance runtime, and automatic program optimization.

– framework for in-memory data model and persistence.

– BSP (Bulk Synchronous Parallel) computing framework.

– programming model for processing large data sets with a parallel, distributed algorithm on a cluster.

– high level language to express data analysis programs for Hadoop.

– framework for stream processing, implementation of S4.

– framework for in-memory cluster computing.

– framework for stream processing, part of Spark.

– framework for stream processing by Twitter also on YARN.

– application framework for executing a complex DAG (directed acyclic graph) of tasks, built on YARN.

– abstraction over YARN that reduces the complexity of developing distributed applications.

– data processing and querying library.

– High Performance, Custom Data Warehouse on Top of MapReduce.

– framework for data management/analytics on Hadoop.

– MapReduce library for Clojure.

– alternative MapReduce paradigm.

– real-time engine is designed to enable distributed, asynchronous, real time in-memory big-data computations in as unblocked a way as possible, with minimal overhead and impact on performance.

– Hadoop enhancement which removes single point of failure.

– Map Reduce framework.

– distributed in-memory datastore.

– create data pipelines to help themæingest, transform and analyze data.

– map reduce framework.

– fault tolerant stream processing framework.

– declarative programming language for working with structured, semi-structured and unstructured data.

– is a set of libraries, tools, examples, and documentation focused on making it easier to build systems on top of the Hadoop ecosystem.

– framework for real-time analysis of large datasets.

– map-reduce for Clojure whiche compiles to Apache Pig.

– MapReduce framework developed by Nokia.

– asynchronous job execution system.

– Python MapReduce and HDFS API for Hadoop.

– general purpose cluster computing framework.

– usefull for counting activities of event streams over different time windows and finding the most active one.

– Scala library for Map Reduce jobs, built on Cascading.

– Streaming MapReduce with Scalding and Storm, by Twitter.

– TimeSeries AggregatoR by Twitter.

Distributed Filesystem

– a way to store large files across multiple machines.

– formerly FhGFS, parallel distributed file system.

– software storage platform designed.

– object storage system.

– distributed filesystem (GFS2).

– distributed filesystem.

– scalable, highly available storage.

– GGFS, Hadoop compliant in-memory file system.

– high-performance distributed filesystem.

– open-source distributed file system.

– scale-out network-attached storage file system.

– reliable file sharing at memory speed across cluster frameworks.

Document Data Model

– commercial object-oriented database management systems .

– is an open source massively scalable data store. It requires zero administration.

– Facebook’s Paxos-like NoSQL database.

– document oriented datastore over Hadoop.

– horizontally scalable document-oriented NoSQL data store.

– Schema-agnostic Enterprise NoSQL database technology.

– Document-oriented database system.

– A transactional, open-source Document Database.

– document database that supports queries like table joins and group by.

Key Map Data Model

Note: There is some term confusion in the industry， and two different things are called “Columnar Databases”. Some, listed here, are distributed, persistent databases built around the “key-map” data model: all data has a (possibly composite) key， with which a map of key-value pairs is associated. In some systems, multiple such value maps can be associated with a key， and these maps are referred to as “column families” (with value map keys being referred to as “columns”).

Another group of technologies that can also be called “columnar databases” is distinguished by how it stores data, on disk or in memory — rather than storing data the traditional way， where all column values for a given key are stored next to each other, “row by row”， these systems store all columnvalues next to each other. So more work is needed to get all columns for a given key， but less work is needed to get all values for a given column.

The former group is referred to as “key map data model” here. The line between these and the stores is fairly blurry.

The latter, being more about the storage format than about the data model, is listed under .

You can read more about this distinction on Prof. Daniel Abadi’s blog: .

– distribuited key/value store, built on Hadoop.

– column-oriented distribuited datastore, inspired by BigTable.

– column-oriented distribuited datastore, inspired by BigTable.

– evolution of HBase made by Facebook.

– column-oriented distributed datastore.

– is a fully managed, schemaless database for storing non-relational data over BigTable.

– column-oriented distribuited datastore, inspired by BigTable.

– is accessed through a MySQL interface and use massive parallel processing to parallelize queries.

– improved version of HBase.

– Transactions for HBase.

– real-time, multi-tenant distributed database for Twitter scale.

Key-value Data Model

– NoSQL flash-optimized, in-memory. Open source and “Server code in ‘C’ (not Java or Erlang) precisely tuned to avoid context switching and memory copies.”

– distributed key/value store, implementation of Dynamo paper.

– is a protocol-compatible Server replacement for Redis.

– Distributed database specialized in exporting data from Hadoop.

– distributed time series database.

– is a simple persistent data store with very low latency and high throughput.

– distributed key/value storage system.

– distributed key-value database by Oracle Corporation.

– in memory key value datastore.

– a decentralized datastore.

– library to work with asynchronous key value stores, by Twitter.

– an efficient NoSQL database and a Lua application server.

– key-value store that’s replicated and sharded and provides atomic multirow writes.

Graph Data Model

– implementation of Pregel, based on Hadoop.

– implementation of Pregel, part of Spark.

– multi model distribuited database.

– TAO is the distributed data store that is widely used at facebook to store and serve the social graph.

– open-source graph database.

– graph processing framework.

– a core C++ GraphLab API and a collection of high-performance machine learning and data mining toolkits built on top of the GraphLab API.

– resilient Distributed Graph System on Spark.

– graph traversal Language.

– RDF-centric Map/Reduce framework.

– tools to construct large-scale graphs on top of Hadoop.

– Massively Parallel Graph processing on GPUs.

– graph database writting entirely in Java.

– document and graph database.

– framework for large scale graph processing.

– distributed graph database, built over Cassandra.

– distribuited graph database.

Columnar Databases

Note please read the note on section.

– an explanation of what columnar storage is and when you might want it.

– column-oriented analytic database.

– column oriented DBMS.

– column store database.

– columnar storage format for Hadoop.

– purpose-built, dedicated analytic data warehouse that offers a columnar engine as well as a traditional row-based one.

– is designed to manage large, fast-growing volumes of data and provide very fast query performance when used for data warehouses.

Google’s cloud offering backed by their pioneering work on Dremel.

Amazon’s cloud offering, also based on a columnar datastore backend.

NewSQL Databases

– commercially supported, open-source SQL relational database management system.

– data warehouse service, based on PostgreSQL.

– statistic oriented SQL database.

– Scalable, Geo-Replicated, Transactional Datastore.

– distributed database designed to enable scalable, flexible and intelligent applications.

– distributed database, inspired by F1.

– distributed SQL database built on Spanner.

– globally distributed semi-relational database.

– is an experimental main-memory， parallel database management system that is optimized for on-line transaction processing (OLTP) applications.

– linearly scalable multi-row， multi-table transaction library for HBase based on Percolator.

– NoSQL plugin for MySQL/MariaDB.

– infinity scalable RDBMS.

– in memory SQL database witho optimized columnar storage on flash.

– SQL/ACID compliant distributed database.

– object-relational database management system.

– in-memory， relational database management system with persistence and recoverability.

– Low-latency， in-memory， distributed SQL data store. Provides SQL interface to in-memory table data, persistable in HDFS.

– is an in-memory， column-oriented, relational database management system.

– distributed, realtime, semi-structured database.

– database used for flexible, high performance analysis of behavioral data.

– open source software for both file and database synchronization.

Time-Series Databases

– uses MongoDB to store time series data.

– distributed time series database.

– similar to OpenTSDB but allows for Cassandra.

– distributed time series database on top of HBase.

SQL-like processing

– high performance interactive SQL access to all Hadoop data.

– data warehouse system for Spark.

– framework for interactive analysis, inspired by Dremel.

– table and storage management layer for Hadoop.

– SQL-like data warehouse system for Hadoop.

– framework that allows efficient translation of queries involving heterogeneous and federated data.

– SQL skin over HBase.

– massively parallel, approximate query engine.

– framework for interactive analysis, Inspired by Dremel.

– SQL-like query language for Cascading.

– full SQL query engine for big datasets.

– distributed SQL query engine.

– framework for interactive analysis, implementation of Dremel.

– SQL-like data warehouse system for Hadoop.

– database for storing petabyte-scale volumes of structured and semi-structured data.

– is a Query Optimization Framework for Spark and Shark.

– Manipulating Structured Data Using Spark.

– a full-featured SQL-on-Hadoop RDBMS with ACID transactions.

– interactive query for Hive.

– distributed data warehouse system on Hadoop.

– enterprise-class SQL-on-HBase solution targeting big data transactional or operational workloads.

Integrated Development Environments

– IDE for R.

Data Ingestion

– real-time processing of streaming data at massive scale.

– data collection system.

– service to manage large amount of log data.

– distributed publish-subscribe messaging system.

– stream processing framework, based on Kafla and YARN.

– tool to transfer data between Hadoop and a structured datastore.

– framework that help ETL to Solr, HBase and HDFS.

– streamed log data aggregator.

– tool to collect events and logs.

– geographically distributed system for joining multiple continuously flowing streams of data in real-time with high scalability and low latency.

– open source stream processing software system.

– framework for connecting disparate data sources with Hadoop.

– distributed message queue system.

– stream of change capture events for a database.

– utility package for compressing sorted integer arrays.

– log aggregator and dashboard.

– a tool for managing events and logs.

– log agregattor like Storm and Samza based on Chukwa.

– is a service implementing Kafka log persistance.

Service Programming

– runtime for distributed, and fault tolerant event-driven applications on the JVM.

– data serialization system.

– Java libaries for Apache ZooKeeper.

– OSGi runtime that runs on top of any OSGi framework.

– framework to build binary protocols.

– centralized service for process management.

– a lock service for loosely-coupled distributed systems.

– cluster manager.

– message passing framework.

– decentralized solution for service discovery and orchestration.

– a Python package for building complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures, command line integration, and much more.

– distributed and extensible system for data ingestion, real time analytics, batch processing, and data export.

– libraries for working with LZOP-compressed data.

– asynchronous network stack for the JVM.

Scheduling

– is a service scheduler that runs on top of Apache Mesos.

– data management framework.

– workflow job scheduler.

– distributed and fault-tolerant scheduler.

– batch workflow job scheduler.

– scheduling platform.

Machine Learning

– machine learning library for Hadoop.

– Neural networks in JavaScript.

– real-time large-scale machine learning.

– machine learning library for Cascading.

– Deep Learning in Javascript. Train Convolutional Neural Networks (or ordinary ones) in your browser.

– Flexible and Extensible Machine Learning in Ruby.

– text classification with machine learning.

– scalable Machine Learning in Scalding.

– System for Large Scale Machine Learning at Google.

– statistical, machine learning and math runtime for Hadoop.

– distributed machine learning libraries for the BDAS stack.

– Fast multilayer perceptron neural network library for iOS and Mac OS X.

– Numenta Platform for Intelligent Computing: a brain-inspired machine intelligence platform, and biologically accurate neural network based on cortical learning algorithms.

– machine learning server buit on Hadoop, Mahout and Cascading.

– scikit-learn: machine learning in Python.

– a Spark implementation of some common machine learning (ML) functionality.

– learning system sponsored by Microsoft and Yahoo!.

– suite of machine learning software.

Benchmarking

– micro-benchmarks for testing Hadoop performances.

– real-world big data workload benchmark.

– a Hadoop benchmark suite.

– benchmark suite for MapReduce applications.

– Hadoop cluster benchmarking from Yahoo engineer team.

Security

– single point of secure access for Hadoop clusters.

– security module for data stored in Hadoop.

System Deployment

– operational framework for Hadoop mangement.

– system deployment framework for the Hadoop ecosystem.

– cluster management framework.

– cluster manager.

– is a YARN application to deploy existing distributed applications on YARN.

– set of libraries for running cloud services.

– Cluster manager.

– library that simplifies application deployment and management.

– Similar to Apache BigTop based on Groovy language.

– web application for interacting with Hadoop.

– multi datacenters replication system.

– job scheduling and monitoring system.

– job scheduling and monitoring system.

– application that can deploy HBase cluster on YARN.

– Mesos framework for long-running services.

Applications

– framework to collect and analyze data in real-time, based on HBase.

– open source web crawler.

– capturing, processing and sharing of data for NASA’s scientific archives.

– content analysis toolkit.

– Run, scale, share, and deploy models — without any infrastructure.

– Eclipse-based reporting system.

– open source event analytics platform.

– API for performing image processing tasks on Hadoop’s MapReduce.

– Splunk analytics for Hadoop.

– data-processing library of an RDBMS to analyze data.

– R on Pivotal HD / HAWQ and PostgreSQL.

– auto-scaling Hadoop cluster, built-in data connectors.

– Cloud Platform for Data Science and Big Data Analytics.

– enterprise-strength web and event analytics, powered by Hadoop, Kinesis, Redshift and Postgres.

– R frontend for Spark.

– analyzer for machine-generated date.

– unified open source environment for YARN, Hadoop, HBASE, Hive, HCatalog & Pig.

Search engine and framework

– Search engine library.

– Search platform for Apache Lucene.

– Search and analytics engine based on Apache Lucene.

– Freemium robust web application for exploring, filtering, analyzing, searching and exporting massive datasets scraped from across the Web.

– social graph search platform.

– continuous indexing system.

– continuous indexing system.

– large search index.

– implementation of Percolator, part of HBase.

– quickly and easily search for any content stored in HBase.

– is a Faceted Search implementation written purely in Java, an extension to Apache Lucene.

– is a flexible software library for enabling rapid development of partial, out-of-order and real-time typeahead search.

– search architecture at LinkedIn.

– is a realtime search/indexing system written in Java.

– fulltext search engine.

MySQL forks and evolutions

– MySQL databases in Amazon’s cloud.

– evolution of MySQL 6.0.

– MySQL databases in Google’s cloud.

– enhanced, drop-in replacement for MySQL.

– MySQL implementation using NDB Cluster storage engine.

– enhanced, drop-in replacement for MySQL.

– High Performance Proxy for MySQL.

– TokuDB is a storage engine for MySQL and MariaDB.

– is a collaboration among engineers from several companies that face similar challenges in running MySQL at scale.

PostgreSQL forks and evolutions

– hybrid of MapReduce and DBMS.

– high-performance data warehouse appliances.

– Scalable Open Source PostgreSQL-based Database Cluster.

– Open Source Recommendation Engine Built Entirely Inside PostgreSQL.

– open source MPP database system solely targeted at data warehousing and data mart applications.

– multi-peta-byte database / MPP derived by PostgreSQL.

Memcached forks and evolutions

– key/value cache for flash storage.

– fork of Memcache.

– A fast, light-weight proxy for memcached and redis.

– key/value cache for flash storage.

– fork of Memcache.

Embedded Databases

– ACID-compliant DBMS developed by Pervasive Software, optimized for embedding in applications.

– a software library that provides a high-performance embedded database for key/value data.

– Erlang LSM BTree Storage.

– a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.

– ultra-fast, ultra-compact key-value embedded data store developed by Symas.

– embeddable persistent key-value store for fast storage based on LevelDB.

Business Intelligence

– business intelligence platform in the cloud.

– lean business intelligence platform to visualize and explore your data.

– powerful business intelligence suite.

– customisable Business Intelligence platform.

– business intelligence software and platform.

– software platforms for business intelligence, mobile intelligence, and network applications.

– business intelligence platform.

– business intelligence and analytics platform.

– open source business intelligence platform.

– business intelligence platform.

– Big Data Analytics.

Data Visualization

– graph visualization library using web workers and jQuery.

– open-source or freemium hosting for geospatial databases with powerful front-end editing capabilities and a robust API.

– open source HTML5 Charts visualizations.

– JavaScript library for exploring large multivariate datasets in the browser. Works well with dc.js and d3.js.

– JavaScript library for time series visualization.

– JavaScript library for visualizing complex networks.

– Dimensional charting built to work natively with crossfilter rendered using d3.js. Excellent for connecting charts/additional metadata to hover events in D3.

– javaScript library for manipulating documents.

– dynamic HTML5 visualization.

– pen source real-time dashboard builder for IOT and other web mashups.

– An award-winning open-source platform for visualizing and manipulating large graphs and network connections. It’s like Photoshop, but for graphs. Available for Windows and Mac OS X.

– simple charting API.

– graphite dashboard frontend, editor and graph composer.

– scalable Realtime Graphing.

– simple and flexible charting API.

– provides a rich architecture for interactive computing.

– plotting with Python.

– chart components for d3.js.

– Progressive SVG bar, line and pie charts.

– Easy-to-use web service that allows for rapid creation of complex charts, from heatmaps to histograms. Upload data to create and style charts with Plotly’s online spreadsheet. Fork others’ plots.

– simple but powerful library for building data applications in pure Javascript and HTML.

– open-source platform to query and visualize data.

– JavaScript library dedicated to graph drawing.

– a visualization grammar.

Internet of things and sensor data

– Cloud-based sensor analytics.

Interesting Readings

– Benchmark of Redshift, Hive, Shark, Impala and Stiger/Tez.

– Cassandra vs MongoDB vs CouchDB vs Redis vs Riak vs HBase vs Couchbase vs Neo4j vs Hypertable vs ElasticSearch vs Accumulo vs VoltDB vs Scalaris comparison.

Interesting Papers

2013 – 2014