gearpump / gearpump-examples Goto Github PK

7.0 7.0 7.0 418 KB

License: Apache License 2.0

Scala 94.92% Shell 5.08%

gearpump-examples's Introduction

Gearpump

Gearpump is a lightweight real-time big data streaming engine. It is inspired by recent advances in the Akka framework and a desire to improve on existing streaming frameworks.

The name Gearpump is a reference to the engineering term "gear pump", which is a super simple pump that consists of only two gears, but is very powerful at streaming water.

We model streaming within the Akka actor hierarchy.

Per initial benchmarks we are able to process near 18 million messages/second (100 bytes per message) with a 8ms latency on a 4-node cluster.

For steps to reproduce the performance test, please check Performance benchmark.

Useful Resources

Read the Introduction on TypeSafe's Blog
Learn the Basic Concepts
How to Develop your first application
How to Submit your first application
Explore the Maven dependencies
Explore the Document site
Explore the User Mailing List
Report an issue

How to Build

1). Clone the Gearpump repository

  git clone https://github.com/gearpump/gearpump.git
  cd gearpump

2). Build package

  ## Please use scala 2.12
  ## The target package path: output/target/gearpump-${version}.zip
  sbt clean +assembly +packArchiveZip

After the build, there will be a package file gearpump-${version}.zip generated under output/target/ folder.

To build scala document, use

   ## Will generate the scala doc under target/scala_2.xx/unidoc/
   sbt unidoc

NOTE: The build requires network connection. If you are behind an enterprise proxy, make sure you have set the proxy in your env before running the build commands. For windows:

set HTTP_PROXY=http://host:port
set HTTPS_PROXY= http://host:port

For Linux:

export HTTP_PROXY=http://host:port
export HTTPS_PROXY=http://host:port

How to do style check before submitting a pull request?

Before submitting a PR, you should always run style check first:

  ## Run style check for compile, test, and integration test.
  sbt scalastyle test:scalastyle it:scalastyle

How to generate the license report to generate a list of all dependencies

  sbt dumpLicenseReport

How to generate dependencies by declared license

  sbt dependencyLicenseInfo

Contributors (time order)

License

Gearpump itself is licensed under the Apache License (2.0). For library it used, please see LICENSE.

Acknowledgement

The netty transport code work is based on Apache Storm. Thanks Apache Storm contributors.

The cgroup code work is based on JStorm. Thanks JStorm contributors.

Thanks to Jetbrains for providing a IntelliJ IDEA Free Open Source License.

gearpump-examples's People

Contributors

Stargazers

Watchers

Forkers

kkasravi skw1992 manuzhang huafengw darionyaphet ippromek

gearpump-examples's Issues

Add an ATK example

Story
Add an example that loads the ATK tar for something like kmeans

Provide a TAP application that can load a tap specific json file

Requirements

Should work with Dashboard
Should make it easy for an TAP application to use this functionality
Should read the conf file and pass relevant conf info to kafka, hbase, etc
Should work with kerberos

High Level Design

In order to work with the dashboard, the TAP application can look for a specific 'tap.json' file in the JAR and parse this file for relevant conf values. Otherwise it should allow for a specific CLI option where similar processing can take place. This utility / class should be easy to add to new TAP application that may be defining different types of DAG's.

[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: org.trustedanalytics.gearpump#config-tools;0.0.1-SNAPSHOT: not found
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[error] /Users/darion.johannes.yaphet/repository/gearpump-examples/kafka-hbase-pipeline/src/main/scala/io/gearpump/examples/kafka_hbase_pipeline/Processors.scala:173: overloaded method value apply with alternatives:
[error]   [T](userconfig: io.gearpump.cluster.UserConfig, tableName: String, configuration: org.apache.hadoop.conf.Configuration)io.gearpump.external.hbase.HBaseSink <and>
[error]   [T](userconfig: io.gearpump.cluster.UserConfig, tableName: String)io.gearpump.external.hbase.HBaseSink
[error]  cannot be applied to (String)
[error]       HBaseSink(hbaseTable)
[error]       ^
[error] /Users/darion.johannes.yaphet/repository/gearpump-examples/kafka-hbase-pipeline/src/main/scala/io/gearpump/examples/kafka_hbase_pipeline/Processors.scala:203: overloaded method value apply with alternatives:
[error]   [T](userconfig: io.gearpump.cluster.UserConfig, tableName: String, configuration: org.apache.hadoop.conf.Configuration)io.gearpump.external.hbase.HBaseSink <and>
[error]   [T](userconfig: io.gearpump.cluster.UserConfig, tableName: String)io.gearpump.external.hbase.HBaseSink
[error]  cannot be applied to (String)
[error]       HBaseSink(hbaseTable)
[error]       ^
[error] two errors found
[trace] Stack trace suppressed: run last gearpump-tap-pipeline/*:update for the full output.
[error] (gearpump-tap-pipeline/*:update) sbt.ResolveException: unresolved dependency: org.trustedanalytics.gearpump#config-tools;0.0.1-SNAPSHOT: not found
[error] (gearpump-kafka-hbase-pipeline/compile:compile) Compilation failed