ucsdscheduleplanner / ucsd-schedule-planner

A project to help UCSD students plan their schedules quickly and easily.

UCSD Planner Helper

Live at sdschedule.com! Designed for and by UCSD students. Use this tool to automatically generate your optimal class schedule without relying on WebReg.

Details

UCSD's class registration system, WebReg, is notoriously slow and clunky. During peak hours, students trying to plan their classes on WebReg might find themselves unable to proceed, stuck waiting for a simple search query to load. Everything about WebReg is extremely dated, from its outmoded user interface to the bare minimum of features that it supports. Most importantly, WebReg lacks the ability to generate schedules automatically, requiring users to manually add each of their prospective classes themselves.

In this project, we attempted to address these concerns. Our planning tool allows UCSD students to automatically generate schedules given a list of classes that they are interested in. Students can indicate at what times they would prefer certain classes and can specify class priorities. All of the schedule generation legwork is handled by our system. Additionally, we boast extremely fast load times, a clean, modern UI and near 100% uptime. This schedule generation tool also integrates with calendar frameworks like Google Calendar and Apple Calendar, enabling students to export their new schedule to the device of their choice.

_{^{Note that the user must manually enroll in classes on WebReg itself; we only provide schedule generation utilities, not enrollment features.}}

Primary Technologies

This project uses the following frameworks:

• ReactJS and Redux _{frontend logic}
• React Storybook _{component testing}
• Primereact _{UI components}
• Flask _{backend server}
• Selenium _{web scraping}
• Cheerio _{HTML parsing}
• MySQL _{database for scraped results}
• Docker _{container management}

The frontend is mostly written in JS, by virtue of React, and the backend is mostly written in Python.

System Requirements

The following utilities must be present for installation purposes:

• docker
• docker-compose

Installation

This project is designed to be portable. We use docker to ensure that there is clear separation between our tool's runtime environment and the rest of your system.

To clone, use the following command:

git clone https://github.com/ctrando/UCSD-Planner-Helper
cd UCSD-Planner-Helper

Run this app using a simple shell script:

• To download the data, add "-d" flag. Note the scraping might take a while depends on your network environment.
• The app can be either run in development mode or production mode. Production mode uses Nginx.
• If any changes made to the files (e.g. update), add "-b" flag to tell docker to rebuild. Otherwise docker will always use the images existing.
• "-s" flag is used to stop the detached services in production mode.

./scripts/run.sh [-h|--help] [-d|--download] [-p|--production] [-b|--build] [-s|--stop] [-c|--cert <email>]

Usage:

./scripts/run.sh -h                 # display help message

./scripts/run.sh                    # run in development mode without reloading or rebuilding
./scripts/run.sh -d                 # run in development mode and download the data
./scripts/run.sh -p                 # run the app in production mode
./scripts/run.sh -p -d              # run the app in production mode, and download data
./scripts/run.sh -b -p -d           # rebuild, run the app in production mode, and download data

./scripts/run.sh -s                 # stop the running production mode services
                                    # give error if production services not started properly
./scripts/run.sh -c [email protected] # FIRST FIRST PRODUCTION RUN ONLY
                                    # generate a letsencrypt cert using given email

The development server will be live at http://localhost:3000. Make sure the ports 3000 and 5000 are not used on your machine.

The production server will listen to 80 and 443 ports (ssl enabled by default).

Production Mode

In production, the docker services are run as detached daemons.

Check status:

docker ps
docker logs <service-name>

Service name can be:

sdschedule-backend   sdschedule-certbot   sdschedule-database  sdschedule-frontend  sdschedule-web

Graceful stop detached services:

• ./scripts/run.sh -s is just a shortcut for this

docker-compose -f docker-compose-production.yml down

SSL

Add https (Let's Encrypt):

./scripts/run.sh -p                 # make sure the production services are up
                                    # recommend wait for a minute or so
./scripts/run.sh -c <your-email>    # give an email address

If you don't want to use https, change the nginx config file, UCSD-Planner-Helper\web\nginx\sites\sdschedule.conf, as specified in that file.

Contributing

If you wish to contribute, please speak to @ctrando. We emphasize good testing practices as well as maintainable and well-documented code. Contributing guidelines will be posted in the near future.

#Credits SVGs taken from Heroicons UI: https://github.com/sschoger/heroicons-ui

ucsd-schedule-planner's Issues

Refactor structure to separate into backend and frontend

Everything is really all over the place

Docker support

Integrate docker to make it easier to setup the dev environment

Upper-limit or warning for too many courses added at one time

Currently it seems that there is no restriction for the number of courses that a user can add.
So a user can add all the courses, which would be one heavy load for our server (it is meaningless but our server still processes it loyally).

It can be assumed that a user may not do so.

But it is very common for some students to add additional courses when planning their schedule even there are time conflicts.

So it can be expected that a lot of users will click the generate button and see an empty result (with warning) on their first few tries.

My suggestion is to include the conflicts as webreg does. Because in real life, students can use things like podcasts.

And to display a small warning about the large number of courses.

Location Preference

Have an option that would optimize for location preferences using maps

Add scripts to make sshing into the docker containers easier

Add edit and remove functionality

Make it so you can edit the classes you put and remove them as well.

Speed up course scraping with more multithreading

Readd download button and support GCalendar, Outlook, ICal, etc.

Preferably we should edit the existing add-to-calendar library and make it so that we can add multiple events, but the user should be able to hit add to google calendar and get redirected (works for mobile too), and all the other options to download the schedule.

Make onboarding an easier and quicker experience

^^

Make app more mobile friendly

Add media queries to each stage to make the app transition better when shrinking to mobile devices.

Add better README

The current README was written several months ago. It should be updated to include more detailed guidelines for setting up development environments and contributing. We might consider using the wiki if there is too much in the README.

Add docker to react storybook for quick setup

Add CAPES scraping abilities & separate table for storing results

We should consider integrating some form of CAPES/RateMyProfessor into schedule generation, so that users can get a glimpse of their proposed classes' grade distributions and professor ratings. This task includes making a webscraper for CAPES and then storing the results in a table in the classes database. Might need to adjust the existing schema.

@CTrando Thoughts?

Adjust generation algorithm to always work

Use the bruteforce with DP algorithm to get the schedule generation to always work.

Add React Router to Pages

For google analytics and to see if people are actually generating schedules, have more pages so that when a user hits generate it will redirect them to a new page.

Add better logging

Currently everything uses console.log, probably want to switch that to an actual logging framework with levels

Speed up web scraping process by adding multi-threading

See title. There might be two parts to this: scraping in parallel, parsing in parallel.

Interactive Mode

Create interactive mode so users can see the spots on the calendar and select which classes they want.

Add react storybook to test rolling new components

^^

Look into avoiding wasteful rerenders

After fixing some bugs, I noticed that some components were rerendering pointlessly - ideally should fix that, look into https://github.com/maicki/why-did-you-update

Install procedure on README does not work

The current way to decide if the backend will start scraping or not is based on a 'global' environment variable. Perhaps we can make it built-in via docker files.

And therefore the command on readme does not work. It will not export SDSCHEDULE_SCRAPE=1. User needs to run the scripts with . to keep the variable.

Reference

Ask for confirmation on 'Add to Google Calendar' button

^^

Package together redux updates so it updates don't cause unnecessary rerenders

Currently, in order to make the code more declarative, we use multiple dispatches when realistically it could be grouped together in one big dispatch. However, declarative code makes it easier to understand, so we should not make one big dispatch, rather we should look into middleware that can batch redux updates together to avoid rerendering components every time dispatch is called.

On mobile safari, when clicking Class Input Panel, the browser will always scroll to the top

Remove close icon from preference sidebar

Close icon is there but hidden a lot so if you hover over the icon it will show up and look weird

Analyze the performance of different DBMS by load testing

We are trying to analyze the performance of different DBMS for our app.

Only changing application.py and backend.py is not enough since we need to get data first.

Based on my understanding of the codes. what our backend is doing right now is:

1. call webreg_scrape_upload
2. scrape and save data to sqlite
3. process data
4. export to mysql, another service in our docker-compose file

To analyze and compare performance of different DBMS, my plan is to do the same load test under the same environment and see the metrics.

The JMeter test script is done. Now I'm trying to write similar functions like export_to_mysql() to store the scraped data to other DBMS.

Am I doing it in the correct way?

Fix issues with PrimeReact

Such issues include the TimePicker for calendar and the whole not updating the component input when autocomplete has a new state

When looking at two popovers and switching between them quickly, the mouseenter event can fail to trigger

Time Widget Preference

Fix up time widget preference so looks consistent with theming and thoroughly test it

When user has node_modules locally in storybook folder, running node in docker container fails

This is due to the frontend dockerfile copying the whole frontend directory from local to container, which overwrites the node_modules in the container.

Component for instructor pref has broken style

Use different container for backend and scraping

Using separated containers for uwsgi and scrapers seems more clean, and it does not make sense to block the uwsgi by scraper

Custom autocomplete component

Ideally want to reduce the dependency on prime react cause they changed a lot of things in their last update and that hurt us a little. Could be a fun project too to learn React Storybook.

Move backend to GraphQL

^^^

Create Custom Button Component

Redo MySQL tables to support linking with subclasses better

This is to try to accommodate WebReg's funky organization of classes more easily.

Redesign the website

Make the website more easy to use and intuitive and make it scalable for future features such as CAPES integration and course catalog integration.

Prevent users from unreasonable courses of cancelled instructors

Cancelled courses or sections are rare but it is still common for users to encounter them.

Aside from updating our database more frequently, it might be useful if we check if the schedule is legal when scraping since some entries of cancelled courses will not be removed by the school.

Possibly duplicated api_classes http requests

When the user types a department, there is a chance that the browser will send duplicated api_calsses requests. It happens more frequently when the user did not type everything in uppercase.

Steps to reproduce:

1. Open https://sdschedule.com/
2. Click the blank for Department
3. Type a department name and click the autocomplete
4. Two requests will be sent:

• One when the user finishes typing
• Another one when the user clicks the autocomplete

Don't know if this would be a concern.

But it might be better to have a local cache for the data fetched from server. It would be faster if next time user selects the same department, the browser can use the local cache directly.

Scrape course catalog data to display to users

On timeout exception scraping thread fails.

File "/usr/local/lib/python3.7/threading.py", line 917, in _bootstrap_inner
sdschedule-backend | self.run()
sdschedule-backend | File "/usr/local/lib/python3.7/threading.py", line 865, in run
sdschedule-backend | self._target(*self._args, **self._kwargs)
sdschedule-backend | File "/app/scraper/scraper.py", line 117, in iter_departments_by_thread
sdschedule-backend | self.iter_pages(department, browser)
sdschedule-backend | File "/app/scraper/scraper.py", line 142, in iter_pages
sdschedule-backend | browser.get(current_url)
sdschedule-backend | File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 333, in get
sdschedule-backend | self.execute(Command.GET, {'url': url})
sdschedule-backend | File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
sdschedule-backend | self.error_handler.check_response(response)
sdschedule-backend | File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
sdschedule-backend | raise exception_class(message, screen, stacktrace)
sdschedule-backend | selenium.common.exceptions.TimeoutException: Message: timeout
sdschedule-backend | (Session info: headless chrome=67.0.3396.99)
sdschedule-backend | (Driver info: chromedriver=2.40.565383 (76257d1ab79276b2d53ee976b2c3e3b9f335cde7),platform=Linux 4.15.0-29-generic x86_64)
sdschedule-backend |

increase the timeout or maybe come up with a way that handles timeouts gracefully.

Rework frontend so passing around data is easier

Make webapp load faster

Possibly use Preact or experiment with gzipping to get the webapp to load faster

Add google calendar integration

Add google calendar integration such that users can save their calendars to their google calendars.

There should be a button next to download calendar, that says export to calendar and such, it will ask them how many weeks they want to put on their calendar and then ask for google authentication and then correctly export.

Add Nginx to docker to support complicated web config

Edit docker-compose.yml to include nginx container listening to port 80 and port 443.

Note that the container of letsencrypt should not be included here since people don't want to generate/renew ssl key each time when docker-compose up. It should be renewed using crontab

When we decided to use letsencrypt, we can add the necessary conf. The ssl of letsencrypt needs to be verified using the domain name so currently there is not way for me to do this.

Also we might be able to "hide" the ports of backend and frontend containers.

(For Nginx, I used user nginx instead of www-data)

Come up with global system to do popup notifications

Using primereact's growl component preferably.

Allow users to click on events on Calendar and see info

This info would be things such as the instructor, the course number, and the building

Remove being able to change the class department and course num while in edit mode

This feature is really difficult to maintain and it makes adding new features hard, and it's barely used, so I'll try and remove it.

Web Animations on Add Class

Use the WAAPI to add more animations to the app as we add classes

Scraper breaks with latest python packages

With the latest versions of the packages in requirements.txt, the backend container fails to start the web scraper with the error

ConnectionResetError: [Errno 104] Connection reset by peer

One tested solution: manually downgrade some packages.