submitty / submitty

Homework Submission, Automated Grading, and TA grading system.

Usage

System instructions located at https://submitty.org/

Background

Submitty is an open source course management, assignment submission, exam and grading system from the Rensselaer Center for Open Source Software (RCOS), Department of Computer Science at Rensselaer Polytechnic Institute.

The Submitty project is hosted on GitHub.

Our key features are described here: https://submitty.org/index/features

Contributing

To learn how to contribute to our project, please read:
How to Make a Pull Request
How to Review a Pull Request

Contact Us

If you have a question, please contact us through Zulip.

LICENSING

Submitty is licensed under BSD "3-Clause" which can be viewed in LICENSE
Licenses for the 3rd party components can be viewed in LICENSE-THIRD-PARTY

submitty's Issues

Make a survey

Get the opinion of the Data Structures students and CS1 Students.
Focus Opinion (Design, Sophistication of Algorithms, etc...)

Style Guide

Test case for to confirm we catch zip bomb & proper error message

• A zip bomb should first be tested in a non production, full installation server.

Then we should create automated tests of:

• zip file size > allowed
• uncompressed size of files (according to "table of contents", e.g. "unzip -l" ) > allowed
• zip bomb (uncompressed size of files appears smaller than it really will be), actual uncompressed size >> allowed

https://en.wikipedia.org/wiki/Zip_bomb

Write output from ses to JSON format

ifdef DEBUG statemtents

Logging in the validator currently has

#ifdef DEBUG
    {Some form of error checked std::cout statement}
#endif

As well as in the runner and config. To improve usability with the rest of the system, we should move extra print statements to these #ifdef DEBUG lines and take some of the error statements out of the DEBUG preprocessor statements so that errors are more clear.

remove status and error from url, esp for reloading

Add option to see contents of files uploaded

Mock up of student interface

Remove .gitignore from /site directory

More feedback on hidden test cases

Give Info on whether it produced any output, ran successfully, crashed, took too long to run, ext

Develop / agree on format for data sent to web page

Renaming

clean.h - contains functions that are not all related to cleaning.
STRutil.h - these functions can probably be moved to the clean.h module
metaData.h - this as well as the functions myersDiff uses in clean.h should be moved to a myersUtil module since 'metaData' is vague.
runDiff - Is this code even used? Can it be removed? What does this do?

Add read-me editor

Untrustred user script has no dummy version

Bash script has runner error "-bash: csci1200/bin/untrusted_runscript: No such file or directory"

Write bodies of diff modules

Fix compile errors in diff modules

Add link to assignment description in config

Wiki main page serving as table of contents

The main page of the wiki is currently the table of contents and contains no links, it should be on the sidebar which is editable and should be populated with links (even if the pages do not exist, they will be created when the link is clicked on).

give instructor option to remove hidden label on test cases (individually for output and expected output)

C++ JSON should use library

Unmaintainable methods for JSON generation within the C++ code, should be using a library.

Add diff output guide + JSON output

If the bash script fails to run properly, don't remove to_be_compiled entry- give users option to try for a re-grade?

Organize Repository

Parse through changes to find insertions/deletions/substitutions

Good error message when homework fails due to complication/running taking too long

Separate Academic integrity statement from main html

Change to included file

Add option to download entire assignment zip

Modules Reorganization

We should consider moving modules to .cpp files. They can also probably go into the same 'grading' folder as a sub directory.

Javascript diff viewer word ranges don't display correctly

homework days late calculation

In the pull down menu for different versions of a single homework, the days late calculation is incorrect, possibly only looking at the day of the month, not the month of the year?

JSON test cases - need both instructor and student files

I need the two files that were diff'd, right now there's only resultsX.txt and resultsX_out.json, but I need what resultsX.txt was compared to to write the web interface.

TestCase recompile

TestCase.h should not be aware of recompile. New TestCase schema in the wiki soon.

files are not unzipping on local version of server- provide dummy script

Fix Branch Structures

Update JSON output to include secondary diff

Test Suite

Conform Styles of immediately commited code

change alert() to a div that is hidden unless used

User Interface Changes to HWTemplate.h

HW_Num is set in beginning; Since directories rely on this, can we replace string constants to use that number so we don't need to change it everywhere?

All Directory string constants have the same "CSCI1200/HWX/User" root. Can we make this an argument for the validator to make the string constants more versatile and support any user?

I also noticed in the perl script a flag that is partial credit: "fraction" / "none".

Is the program recompiled for each test case? I was a little confused when looking at it...

Pick a unit testing framework or develop a system for testing

Too many issues

Really now.

Setup script has user-specific paths

Output from running setup script

/home/seve/tmp/tmp
Developer? y
Linux (0) or Mac(1)? 0
What is your class name? csci1200
Remove previous directories? n
Creating new directory structure
[sudo] password for seve: 
chown: invalid user: ‘www_data’
chgrp: invalid group: ‘everyone’
cloning develop
Cloning into 'csci1200/code/rcos'...
remote: Counting objects: 4019, done.
remote: Compressing objects: 100% (53/53), done.
remote: Total 4019 (delta 25), reused 0 (delta 0)
Receiving objects: 100% (4019/4019), 3.77 MiB | 1.42 MiB/s, done.
Resolving deltas: 100% (2173/2173), done.
Checking connectivity... done
Branch develop set up to track remote branch develop from origin.
Switched to a new branch 'develop'
Adding development php auth
Move (m), Copy (c), or Link (l) site? (or do (n)othing) n
creating class.json and site_path.txt
cp: cannot create regular file ‘public/view/csci1200_container.php’: No such file or directory
Creating sample assignment
test -d /Users/kiana/GIT/HWserver-site/csci1200/bin/hw1 || mkdir /Users/kiana/GIT/HWserver-site/csci1200/bin/hw1
mkdir: cannot create directory ‘/Users/kiana/GIT/HWserver-site/csci1200/bin/hw1’: No such file or directory
make: *** [assignment] Error 1
Add symlink to /var/www? (for apache) 
./run.sh: line 137: [: =: unary operator expected
Done setting up csci1200
/home/seve/tmp/tmp
Would you like to start the bash grader script? y
Grade all submissions in /home/seve/tmp/tmp/to_be_graded/
error: user name does not exist

Usage:
 ps [options]

 Try 'ps --help <simple|list|output|threads|misc|all>'
  or 'ps --help <s|l|o|t|m|a>'
 for additional help text.

For more details see ps(1).
pgrep: invalid user name: hwcron
hwcron is running 0 parallel scripts
sleep iter 1: no work

Output from grade formatter is different than original formatted output (.submit.grade stuff)

Some of the original fields may not be necessary, and some cannot be done at the moment because of the way we're handling test cases. This is mainly a reminder so I don't forget to add the additional fields.

Original .submit grade

Grade for: <NAME>
 submission #:35
  Penalty for excessive submissions:              -1 points
  Points for README.txt:                          2 / 2
  Points for Compilation:                         20 / 20
  Points for Compilation (hidden):                50 / 60
Automatic Extra Credit (w/o hidden):              +0 points
Automatic Grading Total (w/o hidden):             51 / 52
Max possible hidden automatic grading points:     0
Automatic extra credit:                           +0 points
Automatic Grading Total:                          101 / 112
Remaining points to be graded by TA:              0
Max points for assignment (excluding extra credit): 101

the .submit.grade currently returned by the grade formatter now

Grade for <NAME>
  submission :#16
    Penalty for excessive submissions:                              -2 points
    Readme                                                          2 / 2
    Compilation                                                     5 / 5
    Test 1                                                          8 / 8
    Test 2                                                          5 / 5
    Test 3                                                          5 / 5
Automatic Extra Credit:                                             0 points
Automatic Grading Total:                                            23 / 25
Remaining points to be graded by TA:                                0 points 

Delete unused branches

Can we remove some of the clutter? Also, origin/Master-both is 399 commits behind. Is it safe to merge master into it so the revision history before the last 20 or so commits are available?

Run secondary diff on substitutions

Rewrite urls to look better

Typical url:
/index.php?course=csci1200&assignment=hw01
Could be presented as
/csci1200/hw01

Research open source algorithms & implement an STL

Change Function Pointer Return type for easier validator output

Javascript insert blank line doesn't work with multiple inserts

-