berkeley-cs162 / ob2

An autograder and grade database for GitHub-based courses

ob2: An autograder and grade database for GitHub-based courses

Octobear 2 (ob2) is a framework for grading programming assignments, used at UC Berkeley. It comes with a dashboard for students, an administration interface for TA's, tools for running autograder scripts inside Docker, a transactional grade database with an audit log, a transactional email daemon, and a set of integrations with GitHub. It takes care of all the infrastructural details, so you can focus on writing great autograders.

Philosophy

Computer Science courses will always be TA'ed by talented computer programmers. Don't add any functionality that could be accomplished with a Google Form, SQL, or some Python scripting.

Quick-start guide

Here is the quickest way of getting started with ob2.

1. Download Vagrant and VirtualBox.
2. Download a copy of ob2 and run vagrant up inside the project root.
3. Use vagrant ssh to log in to the virtual machine.
4. Run sudo apt-get install linux-image-extra-$(uname -r) and then reboot the VM with vagrant reload. If apt-get reported that the kernel extras package is already installed, you don't need to reboot.
5. Create a new Python virtual environment with python3 -m venv ./env.
6. Activate the new virtual environment with source ./env/bin/activate.
7. The ob2 code is mounted inside the virtual machine at ~/src, so enter the directory with cd src/.
8. Install the necessary Python packages with pip3 install -r requirements.txt.
9. You can now start the ob2 server with python3 -m ob2.
10. When ob2 starts, it will try to initialize the database, so press 'y' and hit Enter to continue.
11. The Vagrantfile bundled with ob2 should have forwarded port 5002 (the default web port) for you, so go to localhost:5002 to access the ob2 web interface.

At this point, you should have a working instance of ob2, but it will not be very useful. You won't be able to log in without GitHub OAuth tokens, and there won't be any jobs configured.

See the article on Configuring ob2 to get started on setting up your own ob2 configuration.

If you want to use VMware Fusion instead of VirtualBox, see VMware Fusion support.

For more details about setting up ob2 in production environments, see Setting up ob2 (the hard way).

Contributing

For Python code, use PEP8 standards, but with 100-character line width.

For HTML, use 100-character line width and your own discretion.

For plaintext email templates, try to keep lines to 75 characters.

License

BSD 2-clause license (see LICENSE.txt)

ob2's Issues

Review/Revert github3.py workaround after upgrading github3.py

Once github3.py merges sigmavirus24/github3.py#911 and releases that fix, we can upgrade github3.py. That should fix another bug in github3.py that causes add_collaborator to return False even if adding a collaborator succeeds. Someone should review/revert commit 0013444 when we do upgrade, since that very ugly workaround probably won't be necessary anymore.

GitHub invites not being sent

Some students don't receive repo invites at all when they sign up for the autograder. Currently unsure why this is happening or what group of students this happens to.

The following is the log from journalctl that for a student that did not receive an invite.

Jan 18 17:12:56 cs162 run-autograder.sh[95713]: Traceback (most recent call last):
Jan 18 17:12:56 cs162 run-autograder.sh[95713]:   File "/work/cs162/ob2/ob2/util/resumable_queue.py", line 107, in run
Jan 18 17:12:56 cs162 run-autograder.sh[95713]:     self.process_job(operation, payload)
Jan 18 17:12:56 cs162 run-autograder.sh[95713]:   File "/work/cs162/ob2/ob2/repomanager/__init__.py", line 18, in process_job
Jan 18 17:12:56 cs162 run-autograder.sh[95713]:     _assign_repo(*payload)
Jan 18 17:12:56 cs162 run-autograder.sh[95713]:   File "/work/cs162/ob2/ob2/util/github_api.py", line 112, in _assign_repo
Jan 18 17:12:56 cs162 run-autograder.sh[95713]:     "Unable to add member %s to %s" % (repr(member), repr(fq_repo_name))
Jan 18 17:12:56 cs162 run-autograder.sh[95713]: AssertionError: Unable to add member 'awang193' to 'Berkeley-CS162/student376'

See https://piazza.com/class/kyb0uwsl9mj4oj?cid=16_f3 for students who are having this issue.

Static Slip Day Calculation

Currently, slip days are calculated statically, meaning the data is hard written into a database. This only gets updated when a student makes a new submission.

This is problematic when we want to retroactively give a student an extension (i.e. after they have already completed the assignment). From the student's perspective, it shows they've used slip days even though the deadlines are not like that.

A possible issue that might arise from this is calculating slip days for projects. Every semester, there are students who change project groups due to dropping group members, so this may affect that. A fool proof way of dynamic slip day calculation would be to take a student's most recent submission that is affiliated with them (i.e. instead of associating with a specific group).