hackclub / api

[DEPRECATED] The backend (that used to) power Hack Club.

⚠️ This repo is deprecated

Looking for public data from Hack Club? See AirBridge or Hack Club Bank API

API

The backend powering https://hackclub.com. Illustrated above by @maxwofford.

Getting Started

Install rbenv

brew install rbenv

Install bundler

gem install bundler -v 1.17.3

Run bundler

bundle install

Copy .env.example to .env

cp .env.example .env

Create and migrate database

bundle exec rake db:drop db:create db:migrate

Run the application

bin/rails s

Browse to localhost:3000

Alternative with Docker

Copy .env file

cp .env.docker.example .env.docker

Run Docker

docker-compose build
docker-compose run web bundle install
#docker-compose run web yarn install --check-files
docker-compose run web bundle exec rails db:drop db:create db:migrate
docker-compose run --service-ports web bundle exec rails s -b 0.0.0.0 -p 3000

Other Development Setup

Setting up the integrated Slack bot

1. Create a new Slack app on Slack
2. Create one (and only one) bot user and set "Always Show My Bot as Online" to "On"
3. Click "Event Subscriptions" on the sidebar in the left and set the request URL to HOSTNAME/v1/hackbot/webhooks/events, replacing HOSTNAME with your actual hostname.
4. Subscribe to the following bot events: message.channels, message.im, message.groups, message.mpim
5. Click "Interactive Messages" on the left sidebar and set the request URL to HOSTNAME/v1/hackbot/webhooks/interactive_messages, replacing HOSTNAME with your actual hostname.
6. Manually go through the Oauth flow and POST code to /v1/hackbot/auth

Production Setup

Scheduled Jobs

This application depends on a few jobs running periodically in the background. Set this up using cron or a similar scheduler on your deployment of the application -- we use Heroku's scheduler in production.

• rails heroku_scheduler:queue_update_hackbot_slack_username_job hourly
• rails heroku_scheduler:queue_record_slack_stats_job daily
• rails heroku_scheduler:queue_activate_clubs_job daily
• rails heroku_scheduler:queue_collect_projects_shipped_job daily
• rails heroku_scheduler:queue_schedule_leader_check_ins_job daily
• rails heroku_scheduler:queue_handle_spam_club_applications_job every 10 minutes
• rails heroku_scheduler:queue_update_from_streak_job hourly
• rails heroku_scheduler:queue_close_check_ins_job daily

Deployment on Heroku

We use Heroku for managing our deployment of this project and that brings along some special caveats. Specifically, we rely on multiple buildpacks.

Here are the buildpacks that need to be configured (they must be in the given order):

https://github.com/heroku/heroku-buildpack-activestorage-preview
https://github.com/heroku/heroku-buildpack-apt
heroku/ruby

Refer to https://devcenter.heroku.com/articles/buildpacks for instructions on configuring buildpacks.

Profiling

We use Skylight to profile the performance of our backend in production. To use it, you must set SKYLIGHT_AUTHENTICATION in the environment to the value that Skylight gives you.

api's Issues

As a user on our website, I want to see download as PDF and print buttons on every workshop page

As part of our trademark application, we need to add a "Download as PDF" and a "Print" button to our workshop pages to show that our workshops fall within three trademark categories: (1) online curriculum materials, (2) downloadable curriculum materials, and (3) printable physical curriculum materials.

---

As a user, when I go to https://hackclub.com/workshops/personal_website/ or any other workshop on our website, I want to see a "Download as PDF" and a "Print" button somewhere on the page.

Both buttons should be functional and work on all modern browsers. We don't know what browser the person at the trademark office will be using, so we should make sure that it'll definitely work. I have a Browserstack account I can give you access to for testing on different browsers.

Rough mockup of what this could potentially look like:

This can be resolved once both buttons are live and functional on https://hackclub.com.

Set up Hakiri

Currently we use Hakiri to alert for vulnerabilities in https://github.com/hackclub/api. We need to migrate this to https://github.com/hackclub/monolith.

SMTP_ADDRESS is not documented

Just like #53, the API requires SMTP_ADDRESS to be set to boot up, but it's not documented anywhere in the README.

hackclub.com needs a description in Google search results

This results is super bare ^

Setup instructions in README fail on a new clone

The setup instructions in README.md fail when you create a clean clone. frontend fails to launch because react-scripts is missing.

Steps to reproduce:

1. Clone monolith

2. Add .env files to frontend/ and api/ with appropriate contents

3. Run the following commands:

   1. docker-compose build
   2. docker-compose run api bundle
   3. docker-compose run api rails db:create db:migrate
   4. docker-compose up

4. Notice that the frontend service fails to launch.

Fix incorrect club application start dates

The 'Start date' field in our application form is being messed up somewhere along the way to being saved in the database and sent out in the club application confirmation email.

Example:

To recreate:

• Submit an application from /apply
• Choose a start date
• Submit application
• Notice how the start date is not what you chose on the application page.

Set up Snyk

Currently we use Snyk to alert for vulnerabilities in https://github.com/hackclub/api. Now that api is no longer the root folder, we need to indicate to Snyk that it should only test the contents of the api directory.

Club time of death is set to 1969

All null club death dates are being set to 1969:

This is not a problem for death dates that were already set:

We want to fix this because it's incorrect data and messes up our records.

Workshop links render incorrectly if entering the website at hackclub.com/workshops

Submit buttons on the frontend shouldn't submit when loading

Right now clicking on the submit button on https://hackclub.com/apply while it's loading submits multiple applications.

SLACK_ADMIN_ACCESS_TOKEN is not documented

Trying to configure a fresh install from instructions in the README.

[zrl@charmander monolith]$ docker-compose run api rails db:create db:migrate   
Starting monolith_bundle_1 ...         
Starting monolith_cache_1 ...          
Starting monolith_bundle_1             
Starting monolith_bundle_1 ... done    
Database 'app_development' already exists                                      
Database 'app_test' already exists     
rails aborted!                         
KeyError: key not found: "SLACK_ADMIN_ACCESS_TOKEN"                            
(erb):50:in `fetch'                    
(erb):50:in `<main>'                   
/bundle/gems/railties-5.0.3/lib/rails/application.rb:391:in `secrets'          
/usr/src/app/config/initializers/cloud9_client.rb:1:in `<top (required)>'      
/bundle/gems/activesupport-5.0.3/lib/active_support/dependencies.rb:287:in `load'                                                                              
/bundle/gems/activesupport-5.0.3/lib/active_support/dependencies.rb:287:in `block in load'                                                                     
/bundle/gems/activesupport-5.0.3/lib/active_support/dependencies.rb:259:in `load_dependency'                                                                   
/bundle/gems/activesupport-5.0.3/lib/active_support/dependencies.rb:287:in `load'                                                                              
/bundle/gems/railties-5.0.3/lib/rails/engine.rb:648:in `block in load_config_initializer'                                                                      
/bundle/gems/activesupport-5.0.3/lib/active_support/notifications.rb:166:in `instrument'                                                                       
/bundle/gems/railties-5.0.3/lib/rails/engine.rb:647:in `load_config_initializer'                                                                               
/bundle/gems/railties-5.0.3/lib/rails/engine.rb:612:in `block (2 levels) in <class:Engine>'                                                                    
/bundle/gems/railties-5.0.3/lib/rails/engine.rb:611:in `each'                  
/bundle/gems/railties-5.0.3/lib/rails/engine.rb:611:in `block in <class:Engine>'                                                                               
/bundle/gems/railties-5.0.3/lib/rails/initializable.rb:30:in `instance_exec'   
/bundle/gems/railties-5.0.3/lib/rails/initializable.rb:30:in `run'             
/bundle/gems/railties-5.0.3/lib/rails/initializable.rb:55:in `block in run_initializers'                                                                       
/bundle/gems/railties-5.0.3/lib/rails/initializable.rb:44:in `each'            
/bundle/gems/railties-5.0.3/lib/rails/initializable.rb:44:in `tsort_each_child'                                                                                
/bundle/gems/railties-5.0.3/lib/rails/initializable.rb:54:in `run_initializers'                                                                                
/bundle/gems/railties-5.0.3/lib/rails/application.rb:352:in `initialize!'      
/usr/src/app/config/environment.rb:5:in `<top (required)>'                     
/bundle/gems/activesupport-5.0.3/lib/active_support/dependencies.rb:293:in `require'                                                                           
/bundle/gems/activesupport-5.0.3/lib/active_support/dependencies.rb:293:in `block in require'                                                                  
/bundle/gems/activesupport-5.0.3/lib/active_support/dependencies.rb:259:in `load_dependency'                                                                   
/bundle/gems/activesupport-5.0.3/lib/active_support/dependencies.rb:293:in `require'                                                                           
/bundle/gems/railties-5.0.3/lib/rails/application.rb:328:in `require_environment!'                                                                             
/bundle/gems/railties-5.0.3/lib/rails/application.rb:448:in `block in run_tasks_blocks'                                                                        
/bundle/gems/railties-5.0.3/lib/rails/commands/rake_proxy.rb:14:in `block in run_rake_task'                                                                    
/bundle/gems/railties-5.0.3/lib/rails/commands/rake_proxy.rb:11:in `run_rake_task'                                                                             
/bundle/gems/railties-5.0.3/lib/rails/commands/commands_tasks.rb:51:in `run_command!'                                                                          
/bundle/gems/railties-5.0.3/lib/rails/commands.rb:18:in `<top (required)>'     
bin/rails:4:in `require'               
bin/rails:4:in `<main>'                
Tasks: TOP => db:migrate => environment                                        
(See full trace by running task with --trace)                                  

README currently doesn't mention anything about SLACK_ADMIN_ACCESS_TOKEN.

Sync start date form Streak

Currently Streakable handles saving dates to Streak pretty well, but it fails dismally at reading them in.

#63 should be reverted once this is added in.

Set up testing & automated deployment with Circle-CI

Currently our two separate repositories (https://github.com/hackclub/api and https://github.com/hackclub/frontend) have individual Circle-CI test suites.

This repository needs to be able to run BOTH of those whenever a commit is pushed.

set-poc should assign letter boxes to the person who runs the command

warning: already initialized constant Module::USER_AGENT

This warning occurs whenever I run UpdateFromStreakJob.perform_now

irb(main):016:0> UpdateFromStreakJob.perform_now
Performing UpdateFromStreakJob from DelayedJob(default)
/usr/src/app/lib/cloud9_client.rb:66: warning: already initialized constant Module::USER_AGENT
/usr/src/app/lib/cloud9_client.rb:66: warning: previous definition of USER_AGENT was here

STREAK_CLUB_APPLICATIONS_PIPELINE_KEY is not documented

Same deal as #53 and #54. Tried setting up monolith from scratch and needed to set STREAK_CLUB_APPLICATIONS_PIPELINE_KEY, which is not documented in the README.

Authentication not required when accessing some major API endpoints

v1/clubs

Summary

This endpoint shows sensitive, internal information about all clubs

Reproduce

curl -X GET https://hackclub.com/v1/clubs

v1/clubs

Summary

Submitting a POST request can alter entries and create entries

Examples

Look at club.id 114,1063,1064

I'll be looking at a bit more later, but these are the most fearing from the bunch.

Credits:
@maxwofford For confirming results
@paked For egging me on (and backing up the db)

Read only Streak fields

Context

Creating this issue so I remember to flesh it out in a couple of hours time.

Fix Google analytics

@zachlatta mentioned we were getting impossible results because the analytics code might still be in our iframe'd team and home pages.

Set up continuous deployment with Heroku

Consolidate stats services

A big part of the work which has been done on Hack Club over the past couple of months has been making it easier for us to collect statistics on things. While this has been a positive change for external forces, the way everything has been implemented internally is... well, it's shit. We're at a point where we have upwards of 3 different statistics related 'services' in our API codebase. Each serving a similar role, calculating the same things in different ways, and all around just not being consistent.

I'd like to consolidate CheckInReportService, ClubStatsService, and StatsService into a single 'Stats' service. With a consistent API.

Duplicate applications are threaded in Gmail

Right now if someone submits their application twice it gets attached to only one of the club application boxes on Streak because both emails have the same subject and Gmail threads emails with the same subject. We can fix this by adding their club application ID to the email subject.

We'll also want to remove the club application ID added to the confirmation email message we currently have here: https://github.com/hackclub/monolith/blob/master/api/app/views/club_application_mailer/admin_notification.text.erb#L10

UnhandledPromiseRejectionWarning

I'm getting the following error when I start up the container.

frontend_1  | (node:33) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: Exited with code 3
frontend_1  | (node:33) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections thatare not handled will terminate the Node.js process with a non-zero exit co

Application start date in 2000

The application API isn't handling the application start date correctly. Here's the network request for an application that should be set to today:

Here's the start date on the application confirmation email:

We need this fixed because 1. it looks weird to applicants when they receive the confirmation email and 2. we're collecting incorrect information about clubs that might cause issues in our ops process.

Frontend doesn't have a robots.txt

Implement accountability bot for operations processes

Here's our whiteboard:

Accountability bot stuff in black and on the right.

Application year is set to the streak dropdown key in confirmation emails

Check out the application confirmation emails to see an example of this ^

Club leaders in streak should have the same assignee as their club

No link on 'click here' text on the application page

As CircleCI, I want to cache `bundle install`

Currently, CircleCI runs bundle install on every build without any caching, which takes ~2 minutes and slows down the overall build.

You can see https://circleci.com/gh/hackclub/monolith/382 for an example of what this looks like:

This issue can be resolved by having CircleCI cache the installed gems so we don't have to start from scratch on every build.

Hide iframe links on Firefox

From @BUSH1DO on August 14, 2017 22:35

Not sure it's just an issue on my browser but these buttons the Hack Club front page aren't functional on Firefox.

Copied from original issue: hackclub/hackclub#1036

Error: Not Found: https://hackclub.com/workshops/that_was_easy/sounds

https://sentry.io/hack-club/Frontend/issues/336282098/

Error: Not Found: https://hackclub.com/workshops/that_was_easy/sounds
  at callbacks (containers/NotFound/NotFound.js:31:34)
  at this (../node_modules/react-dom/lib/CallbackQueue.js:76)
  at wrapper (../node_modules/react-dom/lib/ReactReconcileTransaction.js:80)
  at this (../node_modules/react-dom/lib/Transaction.js:209)
  at method (../node_modules/react-dom/lib/Transaction.js:156)
...
(12 additional frame(s) were not displayed)

TypeError: Cannot read property 'document' of null

https://sentry.io/hack-club/Frontend/issues/327052070/

TypeError: Cannot read property 'document' of null
  at document (components/FullPageIframe/FullPageIframe.js:17:54)

Fix read only Streak values being evaluated at the wrong time

There is currently a bug with our Streak syncing code that will evaluate read-only values before everything else.

This is a problem because often these read-only values require the latest information in order to parse their results, and in an environment where this information is often assumed to not be nil this can be very bad.

Example: We're loading a bunch of new data into the database, but Streak because read only field X relies on Z, and Z has not been set from in the DB yet X will throw a nil value error.

Emoji do not properly render on workshops pages

When I go to https://hackclub.com/workshops, I see :smile: instead of the 😊 emoji rendered.

This can be fixed by properly rendering emoji tags on all workshop pages.

As a teacher, I want to have an explanation for why I can't start a club myself

Consider the following:

1. I'm a teacher and hear about starting a Hack Club and am interested
2. I go to https://hackclub.com, click "Start a Hack Club", and am brought to https://hackclub.com/start
3. I read through the page, decide I want to start my application, and say I'm a teacher.
4. When I click through, I'm told that Hack Club only accepts applications from students, but not why that's the case or what I can do to get one started at my high school as a teacher. See screenshots below.

When we reject teachers without a proper explanation, we make it harder for people to start Hack Clubs and establish ourselves as an organization that doesn't care about explaining itself.

To resolve this, we should replace the current text displayed to teachers with a proper explanation for why we believe all Hack Clubs need to be led by students, for students and what they can do as a teacher.

This story can be resolved once new copy has been approved by me and deployed to https://hackclub.com. Ping me on Slack if I can be helpful figuring out what to write here.

Store information of address in separate DB collumns

At the moment we store the addresses of our leaders, clubs and letters as strings and (sometimes) lat/long pairings.

In order for @zachlatta to get proper demographic data about our leaders, we need to have a broken down set of columns containing segments of the addresses which he can query.

The important fields are:

1. Zip/postal code
2. Country
3. State

But getting as many of the other fields as possible is also helpful.

Feature request: tie club leader assignee to club assignee

If I change the assignee of a club, I'd like the backend to change the assignee of all of the leaders to the club's assignee.

If I change the assignee of a leader, I'd like the backend to change it back to the club's assignee and send me a notification letting me know that if I want to change a leader's assignee, I need to change the club's assignee directly.

As social media sites or search engines, I want to get real metadata for workshops

Currently, if you go to https://cards-dev.twitter.com/validator and enter https://hackclub.com/workshops/personal_website/, you see generic metadata for the homepage.

As a social media sites or search engines, I want to see custom metadata for each workshop page. For personal website, I'll want to see "Personal Website" as the title and either a short description of the workshop (ex. "Learn to build your first website!") or the first few words of text from the workshop (ex. "Prophet Orpheus, our mascot, is here to guide you through making your own personal website. It will look...").

This story can be resolved once https://cards-dev.twitter.com/validator and https://developers.facebook.com/tools/debug/sharing/ both get custom metadata for every workshop on https://hackclub.com.

Rename "frontend" to "web"

Right now the division between codebases isn't clear because "api" contains hackbot's logic, which is arguably part of "frontend". This could get confusing to newcomers.

I think we can make a better distinction by renaming "frontend" to "web".

Assign clubs, leaders, letters, and tasks to their appropriate Hack Club staff member

Currently when any of these Streakable items are created they are assigned to API User. This has worked OK up until now, but now that both Max and I are working it is becoming annoying to have to constantly assign these items to each other to various boxes -- and when this 'system' breaks down, it becomes difficult to tell who belongs to who.

So, instead of having humans do things (boo) we should make robots to do them instead (yay!). Whenever a box (of type club, letter, leader OR a task) is created they should be assigned to someone by the following strategies:

• Clubs: Randomly assign between @maxwofford and I
• Letters: Use leader's assignee
• Leaders: Use club's assignee
• Tasks: Use leader's assignee

Use https://papertrail.com instead of Heroku's embedded Papertrail service

We're currently using the Papertrail Heroku addon for our logs, which is significantly more expensive than the service available directly from https://papertrail.com – 8GB of logs on https://papertrail.com is $75/m, where 10GB of logs from the Heroku addon is $425/m.

Our primary hosting expense today is Papertrail and as part of our cost cutting, we want to move our logs to the service directly available on https://papertrail.com.

This issue can be resolved once all Heroku applications on the hackclub team are using the service available directly from https://papertrail.com. Ping me on Slack if you need billing info.

Sync year

Every single Club Application should have a year field associated with it.

The validation for this was disabled with #65 because of a failure on my behalf with the migration code.

Monthly contributions checkbox on donations page is broken

When loading the page, it acts like the "monthly contribution?" box is checked when it's not.

Add deploy instructions to README

Workshop link in navbar is broken when looking at workshops

1. You can test this by going to https://hackclub.com/workshops/personal_website/:

2. Then click the "workshops" link in the navbar:

3. The url updates to https://hackclub.com/workshops/, but the page content doesn't change:

Submodules must be initialized and updated before app will run

As part of setting up monolith for the first time, you must run git submodule init and git submodule update, but neither commands are currently documented in the README.

Enforce HTTPS for hackclub.com

http://hackclub.com should redirect to https://hackclub.com

set-poc command sets letter assignee to leader assignee

Right now letter boxes created by the set-poc command are automatically assigned to "API User":

This means I have to manually self-assign letters sent to club leaders created by the set-poc command. It would be great if it automatically assigned me to the letter (because I'm already the leader's assignee).

Rewrite home page in React

Right now our home page is sitting in an iframe. This sucks for numerous reasons (that I won't get into too much detail on here). I'm making this issue to track a new home page I'm making with React components. Still figuring out all the details to this, so I'll edit the description when I know more.