cloudmesh / classes

All class material

README

This repository contains class material. The class is published and updated on a regular basis at:

• https://cloudmesh.github.io/classes/

We are in the process of reorgannizing this directory and all of the class materials will be in docs/sources.

Working with Git

When working with git, do not forget to set it up properly. Often users seem to forget the following important commands to initialize it:

$ git config --global user.name "Albert Zweistein"
$ git config --global user.email [email protected]

Developing

Requirements: - python - pip - virtualenv

Preliminary steps:

1. clone the repository
2. create/activate a virtualenv
3. install the requirements:

$ pip install -r requirements.txt

Making changes:

Files are under docs/sources

Use make to build html.

Use make view to open the index.html in a browser window.

Submitting changes

Please only fork and work in the master branch.

Pull requests can be submitted against the master branch. Please synchronize against master before submitting a request.

Make sure you have this repository added as upstream:

$ git remote -v
origin    https://github.com/YOUR_USERNAME/YOUR_FORK.git (fetch)
origin    https://github.com/YOUR_USERNAME/YOUR_FORK.git (push)
upstream  https://github.com/ORIGINAL_OWNER/ORIGINAL_REPOSITORY.git (fetch)
upstream  https://github.com/ORIGINAL_OWNER/ORIGINAL_REPOSITORY.git (push)

To add the upstream repo:

$ git remote add upstream https://github.com/cloudmesh/classes

To synchronize your changes:

$ git fetch upstream
$ git rebase upstream/master

Push your changes and submit a pull request:

$ git push origin master

Please, do not work on gh-pages or try to create a merge from gh-pages into master. gh-pages is managed by us and is always overwritten and all changes will be lost from it.

Known External Repositores

These are older repositories:

• fall-2016: [email protected]:cloudmesh/fall2016.git
• spring-2016: [email protected]:cglmoocs/bdossspring2016.git

classes's Issues

verify fingerprint python example

Post AIs emails and IDs on class website/piazza

python lesson for big data (monday lecture)

matplotlib
pydal
numpy
scipy
mapreduce in python
...etc

Consolidate tips and questions in piazza

Continuation of #78

implement with cmd5 command into separate repositories

Should work on

• Chameleon
• Kilo
• AWS
• Azure

Publish lessons to students

Notify all students on CANVAS to join piazza

Separate virtualbox and ubuntu on virtualbox installation

There's a docs/source/lesson/linux/ubuntu.rst that documents how to install ubuntu on virtualbox.
The virtualbox-specific instructions should be separated into their own virtualbox.rst

Develop python assignment using cmd module (Monday Lecture)

Formulate homework assignment that:

Python module of the week: cmd

Create a python program that has

mycmd deploy: print deploy
mycmd kill: print kill
mycmd benchmark: print benchmark
mycmd report: ...

Prepare tutorial on Github for Feb. 6

I think this is duplicate of #71

Crash course on bash

• rst page for this on web page
• video

Flesh out git and github lessons

In addition to fleshing out the lessions, the content of github.rst has a lot of overlap with plain git and should be moved to git.rst.

docs/source/lesson/prg/git.rst
docs/source/lesson/prg/github.rst

citation for techlist 112 SAP HANA needs fixing

Python lesson for Monday

Republish website and update changelog

@laszewsk I can republish by tomorrow if you want, but you are better positioned to also update the changelog.

Todo

Top priority

• All: Change lectures table: Topic, Description, REsources, Length, ID

  • All: including what is in the unreleased lectures
  • All: in unreleased lectures, correct 'slides' to 'video' where appropriate

• (Jerome) Record paper1 video

• (Dimitar) prepare Python tutorial for Monday (link to eve as an example of what can be done in Python)

• (Dimitar) Email students about how to use the refs.bib file.

• (Dimitar) Email students that we can help them with committing their homework during Monday's lecture.

• (Vibhatha) Install Ubuntu on Virtual Box and Guest addition

• (Tony, Miao) prepare tutorial on Github for Monday

• Verify all the links in theory lectures go to youtube or correct box

• Publish lessons to students

• Miao & Jerome: fix slides in offixe mix, due Tue, Jan 17

• Badi copy all office mix slides into folder all of us have access to

• Dimitar: Create paper-1 assignment

• Miao & Jerome: class video for Monday (change and reapply theme: background & slide position)

• Tony and Dimitar: documentation and example on cluster command for Mark and other students (installing cloudmesh).

• Jerome: notify all students in CANVAS that are not on piazza on that class started they need to join piazza.

• Tony organizes

  • Who is expert for OSx: Tony
  • Who is expert for Windows 10: Vibatha and Jerome
  • Who is expert for Ubuntu: Dimitar, Miao, Tony, Vibatha

• Vibhatha: bash on Windows, 17, January.

  • Windows-bash rst file was added
  • Linking to web page

• Vibhatha Installed Cloudmesh

  • Install Hadoop using cloudmesh
  • Install Spark using cloudmesh

• Post AIs' emails and IDs on class website/Piazza

• Post office hour on class website/Piazza

• Cloudmesh lectures

• Crash course on Python

• Crash course on Bash

New

• complete bibtex for latex.rst:
  • LaTeX Users and Reference Guide, by Leslie Lamport ;
  • LaTeX an Introduction, by Helmut Kopka;
  • The LaTeX Companion, by Frank Mittelbach
  • Refernve manual
  • not so short introduction
    Priority

---

• badi: cleanup cluster command
• make sure the cluster command works on AWS and azure

Overdue

• Mon, Jan 16. Tony: Reclarification: A single TA collects all contact addresses of all TAS and sends single email to gregor. preferably this e-mail has an vcard attachment for each student so i can just click on the vcard so its added to my contact. Typically you can add yourself to your own contcats program and than export the vcard. On OSX it works like that.

• Fri, Jan 20. Tony verifies and organizes All Tas: do pull request of class and add/update resume and picture

• Fri, Jan 13. Identify 2 TAs that do Windows 10 support and run Windows 10 on their machines. Vibhatha & ?.

• Jan 10: TAs identify office hours. Add to calendar Dimitar created: due Jan 10

• Jerome: Jan 16: google spreadsheet, create scripts so we can merge spread sheets e.g. download
  CANVAS gradebook, download google spreadsheed compare. develop script such as

  i524-compare --canvas canvas.csv --google google.csv

  returns list such as

  Gregor von Laszewski, HID xyz, project is no longer found in CANVAS

  googlespread sheet gets column "enrolled" in which you mark all students that you found in
  canvas as enroled in spreadsheet. You must not remove them from google so we can identify how
  unenrolled and notify university, which is required from us.

  so we need option --update-enrollment so you do not have to do that by hand.
  use docopts to manage parameters

  due date Jan:10

  check into bin dir of classes.

TA test

On Friday 11am I will test all TAs for their setup of python on their Laptops. THis includes
proper setup of python 2.7.13, virtualenv, pip, coudmesh install, editor for python (pycharm)
We recommend for class ubuntu if linux. if windows, this is done in virtualbox.
PS: if you run anything else than ubuntu its yor buisiness, you need to help people in ubuntu. If students run anything else than ubuntu its their buisiness.
However we need 1 - 2 TAs that do support while running Windows 10.

Unassigned, Due Mon, Jan 16

• TAs Make sure you understand pip virtualenv
• TAs identify at least 10 hours after hour support
• TAs identify ansible project for you
• TAs decide if you like to contribute on a paper (non paid)
• TAs add more tasks here that were comming form initial meeting that Dimitar kept notes for
• looks like a company found this, so we may want to consider to see if something like this is useful for futuresystems. we may actually have this already: http://futuregrid.github.io/manual/horizon.html http://cloudmesh.github.io/introduction_to_cloud_computing/
• Everybody gets familiar with pull request on classes https://help.github.com/articles/creating-a-pull-request/
• work on https://cloudmesh.github.io/classes/todo.html

Jerome

• Jerome: creates Google spreadsheet with HIDs
• Jerome: replicates google quiz from I523 on canvas for I524 (deadline Tuesday). Do not make public before Gregor and badi have not reviewed.

Vibatha

• Vibatha: Learn how to do pull reuests
• Vibatha: Document pull request for technology homework. Talk to Gregor and Badi as needed

Miao

• Miao: get access to piazza
• Miao: catch up with the meeting we had Friday

Dimitar

• Dimitar: distribute meeting notes

Resolved

Move all resolved items after a week or so here so we have a record of it and keep the list above small

• Jan 10 Dimitar create split video code, check into bin dir of classes use docopts for parameters and manual

• Jan 10 Dimitar place videos in onedrive

• Jan 10 Dimitar create calendar of when you WORK on class, must cover 20 hours.
  2 hours of them need to be available for a weekly meeting all attend
  sometime between Tuesday - Friday 10 - 2pm

• Jan 10 Gregor get videos, post in youtube

• Jan 10 Gregor make announcement on piazza for online students. Latest Jan 11

• Jan 11: Gregor add posted lecture to lectures.rst

• Jan 10: Vibatha lead, all TAs create account on github. Must use git config, i think we have this as youtube and already in lessons. Vibhatha does it wrong. Jan 10

• Jan10: Gregor: post survey links on piazza

• Jerome: Generate HIDs.

• Gregor originally Jerome: Announce HIDs to students on Piazza, due Fri, Jan 13.

  • Gregor/Dimitar originally Jerome: make sure HIDs are explained on website. Due Fri, Jan 13.

• Dimitar: assignment of TechList <> HIDs, due Fri, Jan 13.

  • Dimitar: announce on Piazza, due Fri, Jan 13.

• Gregor originally Miao: video for TechList, due Mon, Jan 16.

• single meeting time this friday 11am. All TAs must be present

• weekly meeting time Tuesday all TAs must be present

• Vibhatha: doodle for fixed meeting times for online class support, due Fri, Jan 13

  • Vibhatha: post announcement on Piazza and update website

Cloudmesh virtual cluster tests

• hadoop
• spark

Meeting 2017-01-31

Topics

• Github
• Current status
• Milestone 1 https://github.com/cloudmesh/classes/milestone/1
• Milestone 2 https://github.com/cloudmesh/classes/milestone/2
• Milestone TBD https://github.com/cloudmesh/classes/milestone/3
• Lecture for monday
• Youtube

ansible

Provide instructions for installing Ubuntu 16.04 on Virtualbox on Windows 10

Stduent: Investigation of pull request 95

We need to investigate pull request:
#95

Test of techlist 1 document

@tonythomascn you should go though the documentation for techlist.1 as a student and find any bugs you can.

Cluster Command

• Test dev branch from the source

• Documentation and example on cluster command for Mark and other students

• Video by Vihbatha

Compile bibtex for latex.rst

• LaTeX Users and Reference Guide, by Leslie Lamport ;
• LaTeX an Introduction, by Helmut Kopka;
• The LaTeX Companion, by Frank Mittelbach
• Reference manual
• not so short introduction

Verify all links in theory lectures go to youtube or correct box

Possible duplicate of #75

Prepare linux lesson for monday

@laszewsk @jemitchell @dimitargnikolov @tonythomascn @vibhatha @miaozhan Who is doing this?

Add yaml template to paper setup script

deploy mongodb server on single vm

verify all links on the lectures page

Published and unpublished

• https://cloudmesh.github.io/classes/i524/lectures.html

• https://cloudmesh.github.io/classes/i524/lectures-unreleased.html#ref-unreleased

• formatting

• working

consensus algorithms (eg Zookeeper) lecture

Create LaTeX/BibTex FAQ for webpage

Cloudmesh lectures

Verify the new HID list on the webpage is correct

citation for techlist 162 HPX-5 needs fixing

What is the NEW_MIX folder on IU Box?

@jemitchell It looks like you created this. What's the difference between the "NEW_MIX" and previous "Mix" folders?

Convert slides for unreleased theory lectures

From here: https://cloudmesh.github.io/classes/i524/lectures-unreleased.html#lectures-unreleased-theory-track

• 7 Big Data Application Structure Slides
• 7.A NIST Big Data Sub Groups Slides
• 7.B Big Data Patterns - Sources of Parallelism Slides
• 7.C First and Second Set of Features Slides
• 7.D Machine Learning Aspect of Second Feature Set and the Third Set Slides
• 8 Aspects of Big Data Applications Slides
• 8.A Other sources of use cases and Classical Databases/SQL Solutions Video
• 8.B SQL Solutions - Machine Learning Example - and MapReduce Video
• 8.C Clouds vs HPC - Data Intensive vs. Simulation Problems Video
• 11 Big Data Applications and Generalizing their Structure Slides
• 11.A NIST UseCases and Image Based Applications Examples I Video
• 11.B Image Based Applications II Video
• 11.C Internet of Things Based Applications Video
• 11.D Big Data Patterns - the Ogres and their Facets I Video
• 11.E Facets of the Big Data Ogres II Video
• 12 More on Software Stack Slides
• 11.A More of Software Stack Video

Fix slides in office mix

Possibly related to #90, #111, #112

Verify links work for released lectures

check if all lectures and links to them as well as the slides on the “released” lectures are indeed working

Record Paper 1 video

Class video for Monday (change and reapply theme: background & slide position)

Originally for https://github.com/cloudmesh/classes/milestone/1

Consolidate tips and questions in piazza

@laszewsk @jemitchell @dimitargnikolov @tonythomascn @vibhatha @miaozhan Who is doing this?

Verify Techlist.1 on Ubuntu 16.04

Github lesson for monday

Create scripts to merge spreadsheets

Can be done by hand.

Something like (after manually downloading csv files of the canvas and google rosters):

i524-compare --canvas canvas.csv --google google.csv

returns a in the form:

Name, HID, status

We also need an --update-enrollment to automatically mark the enrollment status in the google spreadsheet.

Change lectures table: Topic, Description, Resources, Length, ID

• including what is in the unreleased lectures
• in unreleased lectures, correct 'slides' to 'video' where appropriate

Install Ubuntu on Virtual Box and Guest Addition

Post office hours on class website/piazza

hadoop lecture

cm virtual cluster
stack deployed to cluster
run program that needs compilation and deployment

Crash course on python

Add missing mix files to IU Box

Corollary for #90. Some lectures are out of date and not available on IU Box.
The current links point to outdated versions.
They need to be downloaded and added to Box, then updated as per #90