Coder Social home page Coder Social logo

nicolasbaer / investigating-the-lambda-architecture Goto Github PK

View Code? Open in Web Editor NEW
3.0 2.0 1.0 36.34 MB

Master Thesis `Investigating the Lambda Architecture` University of Zurich: http://www.merlin.uzh.ch/publication/show/10128

JavaScript 0.37% Python 5.76% Shell 5.93% Java 85.84% PigLatin 0.30% Scala 1.80%

investigating-the-lambda-architecture's Introduction

Investigating the Lambda-Architecture

Master Thesis at the University of Zurich

# Requirements

  • Maven
  • Java 7
  • Pytohn 3

# Dependencies

  • Java dependencies are managed with maven
    • Samza and Pig were patched and are provided in the lib directory
  • Pyhton requirements can be found here: requirements.pip

Code Structure

The batch, speed and coordination layers are in their corresponding packages. The automatic_deployment comes in two flavors: (i) a local version and (ii) a distributed version that runs on SLURM or TORQUE. The data module includes common data structures such as objects to represent the data sets (DEBS and SRBench) and the esper queries that are used in the batch and speed layer. The utils package includes helper tools for example to partition the data or to inspect the Kafka topics.

# Local Deployment The local deployment script installs all services on the local node and can start and stop these services. The script grid.sh can be called to see all options. Note that the install functionality depends on the download links of these products and these links will change in the future. Therefore you may have to download some services manually.

Cluster Deployment

The automatic deployment script is designed to start experiments on a SLURM or TORQUE cluster. All scripts are located in the same directory execute. The reason to not further modularize the scripts is the burden to deal with multiple directory in a cluster environment. Tools such as pbsdsh or srun may impose strange constraints to switch between directories.

start_all is the script that encapsulates the logic to run experiments (e.g. sbatch start_all.sh). It is a good starting point to understand how the cluster setup is done and what parameters can be applied.

# Remarks Please note the directory /speed/src/main/java/ch/uzh/thesis/lambda_architecture/speed/spout/kafka includes code from the Storm repository. It was not possible to alter the behavior of the KafkaSpout through inheritance. Therefore the code was copied and modified. For a reference see (Storm)[https://github.com/apache/incubator-storm/tree/master/external/storm-kafka]

investigating-the-lambda-architecture's People

Contributors

nicolasbaer avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar

Forkers

mindis

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.