pzfreo / ox-clo

Materials for Oxford Software Engineering Programme CLO course

OX-CLO

https://github.com/pzfreo/ox-clo

Materials for Oxford Software Engineering Programme CLO course: Cloud Computing and Big Data See https://www.cs.ox.ac.uk/softeng/subjects/CLO.html A 5 day hands-on introduction to building scalable big data systems in the cloud

There are lectures and lab exercises. The lab exercises are usually done half on a VM and half in the Cloud on AWS.

The course is (c) Paul Fremantle 2015-2019 except where credited and is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License: https://creativecommons.org/licenses/by-nc-sa/4.0/

The VM for this class is available here: VMWare: https://oxclo.s3-eu-west-1.amazonaws.com/oxclo2019-vmware.zip VBox:

ox-clo's Issues

talk about docker and cloud security

load balancer lab

Check classic vs application load balancer

Workaround to avoid deleting ASG in order to "disable" it

Item 60 on page 9 of "ox-clo/lab-exercises/02-amazon-ec2-continued.pdf" currently reads:

...you can’t disable an ASG, only delete it.

One workaround might be to change the desired/min/max number of instances to zero. Could save the trouble of deleting and recreating the ASG

Exercise 08 page 6, typo in step 27

sudo yum install python2.7-pip -y

->

sudo yum install python27-pip -y

Spark on K8s update

Exercise 2, step 26 - port number does not match security group settings

The security group for the node-security-group does not allow port 3001, but step 26 says to try browsing to http://ww.xx.yy.zz:3001 so does not work. The step should probably say to browse to http://ww.xx.yy.zz:8080

ex13 hamstermap not working

Managed to use https://mapmakerapp.com/ instead

re-order to include docker earlier

add more about core theory into the final presentation

Update Dockerfile in container presentation

The Dockerfile example on slide 14 of the containers presentation uses the ADD command. It is recommended to use COPY instead, if neither unpacking, nor retrieving from an remote URL are required.

more on clustered containers (coreos, fleet, kubernetes, mesos, etc)

Virtual Machine Link Doesn't Work

The link provided in the README for downloading course's virtual machine doesn't work.

"The VM for this class is available here: VMWare: https://s3-eu-west-1.amazonaws.com/oxclo/oxclo2016vm.zip"

Returned data:

<Error>
<Code>NoSuchKey</Code>
<Message>The specified key does not exist.</Message>
<Key>oxclo2016vm.zip</Key>
<RequestId>1FCCCDEBD8D77F8E</RequestId>
<HostId>
Kfj60TTweyUZXC3vLv/RqY87WEDOlL3omXoMnVPewY7FwC4BkR+iFsZVLf468TD6zt1sdUYhDEs=
</HostId>
</Error>

show saveAsTextFile

Incorrect text in exercise 1

On the fourth page of https://github.com/pzfreo/ox-clo/blob/master/lab-exercises/01-amazon-ec2-getstarted.pdf, at step 14, it says 'Click Next: Tag Instance'. This should be 'Click Next: Add Tags'.

Also it might be handy to add some clarification at step 22. Clicking 'Download Key Pair' will prompt the user to either open or save, instead of saving by default.

Dirty Dozen

https://blog.acolyer.org/2017/09/25/a-dirty-dozen-twelve-common-metric-interpretation-pitfalls-in-online-controlled-experiments/

add zeppelin lab

Python print statement/function

Some of the Python code examples show print used as a statement (e.g. print 'foo'), and some examples show it used as a function (e.g. print('foo')).

With the movement to the print function in Python 3, I think it might be good to change all usages to the function version.

Side note: This can already be enforced by adding a from __future__ import print_function.

Terraform

install sublime text 2 on the vm

Create exercise midway between 6 and 6b

sample issue

Exercise 01 add detail to step 52 f.

Step 52 f., need to make clear to click on "Security credentials" tab.

Outdated VM image

The image (I only checked the VMWare one) still uses software that is outdated.

• Dockerfiles are pulling in Ubuntu 15.10, I would recommend to use the latest LTS: 16.04.
• The installed docker version is still the 1.X series, would recommend switching to the latest version.

Try Zeppelin again

need to terminate instance after part one of ex 2

add parquet

install pycharm on the machines

and figure out how to add packages to spark runner

add a cassandra read example and tie into realtime

Update Python string formatting in map reduce presentation

The mapper and reducer Python examples print formatted strings. The string formatter in the examples expect two strings: '%s\t%s', but receive a string and integer instead. Replace with '%s\t%d' instead.

Machine learning

https://www.google.com/amp/s/www.forbes.com/sites/johnkoetsier/2020/07/16/the-android-of-self-driving-cars-built-a-100000x-cheaper-way-to-train-ai-for-multiple-trillion-dollar-markets/amp/

Explain reduceByKey better in a slide

plus also slides on foreach, map, groupByKey, etc

show some java as well as python

Better explanation of the serving layer

replace siege with wrk

siege is pants, wrk is good.