hackforla / jobs-for-hope

Aggregate job opportunities for homeless service providers so that people can search for a job in the non-profit sector that fits their skillset.

Jobs for Hope

Aggregate job opportunities with homeless service providers so that people can search for a job that fits their skillset.

Prerequisites

1. Git for working with Github source code
2. Node and npm for running the web app
3. Python 2 and pip for running the web scraper

Complete beginner instructions

click here to show

Windows

show

1. Install chocolatey (https://chocolatey.org/install)
   1. Install chocolatey gui (optional)

      choco install chocolateygui -y
      

2. Install git, chromium, chromedriver, vscode, python2, dbeaver, nodejs

   choco install git chromium chromedriver vscode python2 dbeaver nodejs -y
   

3. Install postgresql and set postgres user password

   choco install postgresql --params '/Password:password' --params-global -y
   

Windows 10

show

1. Enable Windows Subsystem for Linux (reference: https://docs.microsoft.com/en-us/windows/wsl/install-win10 https://lifehacker.com/how-to-get-started-with-the-windows-sybsystem-for-linux-1828952698)
   1. In the search bar, type "turn windows features on or off" and choose the correct item
   2. Scroll down and check the box for Windows Subsystem for Linux
   3. Windows will restart to complete the installation
2. Install Ubuntu Linux
   1. Open the Microsoft Store and search for "Run Linux on Windows"
   2. Install and launch Ubuntu
   3. Set up a new linux user account when running for the first time
   4. Update and upgrade all packages
      1. In a terminal, run (you will need to type in your user password when running sudo)

         sudo apt update && sudo apt upgrade -y
         

3. Install ChomeDriver
   1. Install chocolatey (https://chocolatey.org/install)
   2. Install ChomeDriver

      choco install chromedriver -y
      

   3. Add a file chromedriver to the project directory with this content

      #!/bin/sh
      chromedriver.exe "$@"
      

4. Continue to Linux instructions

Linux

show

1. Install Homebrew on Linux (reference: https://docs.brew.sh/Homebrew-on-Linux)
   1. Open a Linux terminal
   2. Install dependencies
      • Debian-based (Ubuntu)

        sudo apt install build-essential curl file git
        

      • Fedora-based

        sudo yum groupinstall 'Development Tools' && sudo yum install curl file git
        

   3. Install homebrew

      sh -c "$(curl -fsSL https://raw.githubusercontent.com/Linuxbrew/install/master/install.sh)"
      

2. Install packages brew install python@2 postgresql sudo apt install chromium-chromedriver -y

macOS

show

1. Install homebrew (https://brew.sh/)
2. Install packages brew install git python@2 postgresql brew cask install chromedriver

Common Tools for all OSes

show

1. Install Visual Studios Code (https://code.visualstudio.com/)

   1. Install Prettier - Code formatter extension

2. Install DBeaver (https://dbeaver.io/), Community Edition

3. Install nvm, node, and npm (reference: https://gist.github.com/d2s/372b5943bce17b964a79)

   1. Install nvm

      curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.11/install.sh | bash
      

   2. Install the latest LTS Node.js (https://nodejs.org/en/)

      nvm install v10.15.3
      

4. Download and install Chrome (https://www.google.com/chrome/)

5. Setup Postgresql (reference: https://github.com/michaeltreat/Windows-Subsystem-For-Linux-Setup-Guide/blob/master/readmes/installs/PostgreSQL.md)

   1. Start the postgres service

      1. Start a terminal app
         1. linux (and WSL)
            1. Start the service

               sudo service postgresql start
               

            2. Setup the postgres user (linux)
               1. Start a terminal app
               2. Set the password

                  sudo passwd postgres
                  

               3. Type in the password and confirmation
               4. Close the terminal
            3. Connect to postgres
               1. Start a terminal app
               2. Switch to the postgres user and start the psql prompt

                  sudo -u postgres psql
                  

               3. If the above doesn't work, do this instead

                  su - postgres
                  psql
                  

            4. Troubleshooting postgres on WSL (reference: microsoft/WSL#3863)
               1. Append this at the end of /etc/postgresql/10/main/postgresql.conf

                  data_sync_retry = true
                  

         2. macOS
            1. Start the service

               brew services start postgresql
               

            2. Connect to postgres
               1. Start a terminal app
               2. enter the psql prompt

                  psql postgres
                  

   2. Create the database (reference: https://www.techrepublic.com/blog/diy-it-guy/diy-a-postgresql-database-server-setup-anyone-can-handle/)

      1. Start the psql prompt
      2. Issue the command

         create database jobsforhope;
         

      3. Create the user
         1. Start the psql prompt
         2. Issue the command

            create user jobsforhope;
            

         3. Check that the user was created

            \du
            

      4. Grant user privilege 1. Start the psql prompt 1. Issue the command grant all privileges on database jobsforhope to jobsforhope;

Full-Stack React/Node Application Installation

1. Clone this repo to your local drive.

   details

   1. Start a terminal app, such as Ubuntu for Windows Subsystem for Linux
   2. Create a src directory in the user's home directory and go in it

      cd && mkdir src && cd src
      

   3. Clone the repository

      git clone https://github.com/hackforla/jobs-for-hope
      

2. Change to the jobs-for-hope directory:

   cd jobs-for-hope
   

3. Install the node server npm depedencies:

   npm install
   

4. Obtain the .env file from the slack channel and place it in this directory. It contains private info (i.e., the production database connection string) that we cannot put in this public GitHub repo.

5. Change to the client directory:

   cd client
   

6. Install the client (React) dependencies:

   npm install
   

To Run the React/Node Application

1. Run npm start from the jobs-for-hope directory to start the node server.
2. Run npm start from the jobs-for-hope/client directory to start the react app and open the browser.

Backend Python Scraper Setup

1. Virtualenv (optional, but will be useful if you eventually want to have multiple python projects)

   show

   1. Set up virtualenv

      1. Install virtualenv using pip

         pip install virtualenv virtualenvwrapper
         

      2. cd into project directory

         cd jobs-for-hope/backend
         

      3. create the virtualenv

         virtualenv venv
         

         Alternative for systems where python 2.7 is not the default

         1. Specify the python location when creating the virtualenv

            virtualenv -p /usr/local/bin/path/to/python2.7 venv # use this if the system default is python3
            

      alternative setup using virtualenvwrapper

      1. Install virtualenv using pip

         pip install virtualenv virtualenvwrapper
         

      2. Create directory to hold virtual environments

         mkdir $HOME/.virtualenvs
         

      3. Find out where virtualenvwrapper.sh is located for next step

         which virtualenvwrapper.sh
         

      4. Make .bash_profile call .bashrc (reference:http://www.joshstaiger.org/archives/2005/07/bash_profile_vs.html)

         1. Add this to .bash_profile

            if [ -f ~/.bashrc ]; then
              source ~/.bashrc
            fi
            

      5. Add to .bashrc

         export WORKON_HOME=$HOME/.virtualenvs
         export PATH="/path/to/virtualenvwrapper:$PATH"
         source virtualenvwrapper.sh
         

      6. Start a new terminal session or call .bashrc

         source ~/.bashrc        # activate virtualenvwrapper.sh, just for the first time
         

      7. Create the virtualenv

         mkvirtualenv jobs-for-hope
         

         Alternative for systems where python 2.7 is not the default

         1. Specify the python location when creating the virtualenv mkvirtualenv -p /usr/local/bin/path/to/python2.7 jobs-for-hope # use this if the system default is python3

   2. Activate the virtualenv

      source venv/bin/activate
      

      for virtualenvwrapper

      workon  // list the existing virtual environments, blank if none is created
      workon jobs-for-hope  // activate virtual environment, not needed when first creating the virtualenv
      

   3. Do work and run python within the virtualenv

   4. Deactivate the virtualenv

      deactivate // switch back to system python
      

   1. All python commands should be run inside the jobs-for-hope virtualenv if you choose to set it up. You will have to make sure the virtualenv is activated before following the steps.

2. Install project dependencies

   cd backend
   pip install -r ./requirements.txt
   

3. Setup for the scrapers

   1. Download the database.ini file from the slack channel into the backend directory
   2. Dump the aws database to a file (reference database.ini for values)

      pg_dump -d <database> -h <host> -U <user> -C -f jobsforhope.sql
      

   3. restore to the local database

      psql -h localhost -U postgres -d jobsforhope -f jobsforhope.sql
      

To run the scraper

1. Switch to the backend directory

   cd backend
   

2. To run all the scrapers

   python scraper_runner.py
   

3. To run a single scraper

   python scraper_runner.py scrapers/prototypes.py
   

Checking the database for scraped data

1. Use DBeaver
2. Add a postgres database connection with the .env file credentials from the slack channel.
3. The jobs are in the jobs table.

jobs-for-hope's Issues

Modify the Organization Form to record zero or more SPAs for each Organization

'view more' modal in the job posting card in the Main page

When the user clicks on the "View More" button, want the detailed information to come up as pop up modal. Should also have an apply button in the modal that takes the user to the organizations career section. There is a rough mock up for this, it's located in the HackForLa slack workspace in the jobs-for-hope channel. I can give more details in Slack.

Organization logos

Fix job posting card layout

fix the margins and paddings for the job posting cards now that we're collecting dynamic data.

Organization Detail page

This is a read-only page that displays detailed organization info. Need to wait til we have some more fields in the organization table to do this.

Organizations Page - Make Job Count Link Populate with Correct Count.

The link should also navigate to the Jobs page with the query set up to filter by that organization.

Implement Job Posting Entry and Maintenance Form

Scrapers successful timestamp

Store a timestamp in the organizations table on successful scraping

fix: convert css files to Sass

We're switching to Sass, so css files need to be converted over to Sass(.scss) files with updated Sass configurations.

Filter jobs based on regions

Load job page inside the job "show more" modal

Right now, the modal shows the job description and a link to the job page. I would be better to load the actual page without leaving the site.

Some agency job boards don't provide a unique link to the jobs. Maybe do something different for those.

Show all jobs initially without needing to apply a filter

Currently, jobs are hidden until a filter is applied.

Reduce file size of large images - Some are over 3MB

Run scraper automatically

Need to figure out how to run the web scraper automatically.

Implement uploading Organization logo images to AWS

2018-07-31 Meeting Notes

Attendees: Nina, Bobby, Tinnie, Samantha, April

KEY NOTES

Goal:
Demonstrating the value of the website
Determining the function of the website from LA County's perspective
Have government realistically see us as a resource to build projects on a long-term basis
To improve the way government internally works technically, so they're not wasting money on contractors
So the government can see value in hiring talent
To increase community engagement with how the government operations (transparency)

NEXT STEPS:
Finalize presentation and refine prototype for meeting tomorrow
Nina's Meeting tomorrow at 1:30 pm!
UX team will present findings
Nina will assist with all non-UX related issues

Delete Organizations

Scrapers failure count

Store scraper failures in the organizations table.
Increment each time the scraper fails so that we can disable the scraper later if it fails too many times without successing.
Clear the failure count on successful scraping.

Filter Organizations Based on Regions

separate style from html files and put in scss file

Need to separate out the styles from all the html files (except index.html as that has already been done) and save them as a scss file and placed in the scss folder (located in the static directory within the templates directory).

About page

Need to create responsive About section(component).
design guide: https://projects.invisionapp.com/share/BTN6GLIJNUY#/screens/311697394

Guidelines:
-No need for the banner, as it will be a component.
-If anything, just the shell of the page would be great, we could have someone take care of the content later.
-here is where you can get the county seal (you can just copy/paste this in src: "https://upload.wikimedia.org/wikipedia/commons/thumb/6/65/Seal_of_Los_Angeles_County%2C_California.svg/215px-Seal_of_Los_Angeles_County%2C_California.svg.png"

Notes-you'll need to use Sass (particularly the variables from -variables.scss) for the About page

Create Regular Users (Job Seekers)/Registration Form

Ability to sign up as a regular user/job seeker

Re-organize filters on Jobs Page

python .venv instructions in README

We currently have instructions for virtualenvwrapper, which works but maybe .venv/ works more like node_modules/ for npm. If so, we need instructions to set that up. I expect it to be easier that what we have currently.

fix: gulp set-up

Sass file is not being tracked.

Redo organization view

Convert scrapers into plugins

Currently all scrapers are in a single file and the problem is one failure will stop all scrapers after it.

The advantage of converting them into plugins is:

• continue running other scrapers if one fails

It will make these future enhancements possible:

• record in organizations table which scraper failed
• record the last successful scraped timestamp
• delete jobs related to organization before running the scraper

Update scrapers to python 3

Need to make sure code still runs and requirements can all run on python 3.
This should be worked on after #122

Convert organization spreadsheet to db table

Convert the organizations spreadsheet into a table.

• have organization id as a foreign key in jobs table.
  Update the scraper to use the foreign key.

Add some way for an employer to add second and third organizations from account page

would need a backend route to add email, org to emails_to_orgs table and org to organizations table. if you have questions ask me on slack :)

Add paging to Jobs Page

Allow user to select how many job listings to show on each page, and allow them to navigate between pages.

Remove old scrapers

Remove the code.
Update the README.
Maybe reorganize files in the directory structure.

responsive design

running list of responsive design things (i think coco has a list too):

• making the top navbar a hamburger menu on mobile

• footer margin is nonexistent

• county of la seal on about page throws everything off

2018-07-17 meeting notes

Who was here?

Nina
Ryan
Katie
Laura
Hayden
Anthony
Bobby
April
Tinnie
Sery

What did we do?

Strategy Team Updates

Update new folks on the status of the project - Nina had a call with person from LA County Homeless Initiative. She's definitely interested, Nina proposed that we present to her a prototype at an in-person meeting in 2 weeks.

• We have a 2 week deadline to present a proof of concept/prototype.
• Focus on a standalone site for the prototype design.

Design Team Updates:

Bobby - July 17, 2018 - 11:00pm:

• We welcomed the new additions (Katie, Laura, Hayden, Anthony and Sery) to the team — got them set up on Slack, GDrive and Github.
• We discussed creating a presentation for Nina for her meeting in 2 weeks. Should be able to show the research, redesign, and possibly add a business aspect to it.
• Created multiple sketches to showcase some possible ideas. Sketches can be found below with the other images.
• Also discussed the idea of having this be a standalone site. This could be beyond the scope, but more research needs to be done. - We discussed what the navigation would be like. Really interesting possibilities.

Primarily We Focused on:

• We started a competitor analysis on other sites and were able to get clearer ideas of features that are going to be essential.
• April and Tinnie were able to do a feature prioritization. Pictured below with other images.
  We discussed adding the following features:

• Dedicated Search for Jobs
• Filters: (distance, keyword, etc. Feature prioritization image is referenced below and can also be found on our GDrive folder).
• A "You May Also Be Interested In..." feature to show other potential jobs the user might also want to see.
• A Possible Login Feature (Not likely) But would also allow us to have the following feature:
• Ability to Favorite or Save a job listing.
• Post A Job (would require Login to work?)

Dev Team Updates

• Nina and Bobby assigned to front end design (CSS, HTML, and JS).

What do we want to do?

• Continue research and begin user testing. Document research and design along the way.
• Begin to think of presentation content.

responsive footer

Need to make the footer responsive in index.html

Organization detail data

Go to each agency and copy down several pieces of information into a spreadsheet

• name
• mission statement
• description
• location
• website
• logo url

If user attempts to edit Organization they are not assocated with, redirect them to the Organizations Page

Add a login/registration system

To get set up for the application to allow data entry and maintenance of organizations and jobs, we need to allow users to register and login (anonymous users will still be able to search read-only information about jobs and organizations). They will need to provide an email as a username and a password, stored in the postgres database in a logins table. There needs to be two kinds of registered users. System administrators will be able to enter/edit all organizations and jobs. Organization Representatives will only be able to enter/edit their own organization(s) and the associated jobs. I guess an organization rep might be assocated with more than one organization.

Once all this is set up, then we can lock down the Organization and Job create/edit forms so that only authorized users can see and run them.

Scraper code autoformatting

There's autopep8, yapf, and black. I'm leaning towards black but that would depend on #123. Definitely do this after #122.

2018-07-24 Notes

Attendees: Bobby, Nina, Ryan, Katie

KEY NOTES

• Goal: to get a buy-in from the government organization
• Prototyping with Envision
• Interactive prototype to demonstrate the tool itself

7/31 DEADLINES

• Have a working model of the search interface and specific jobs page
• Set up a meeting with the contact at the LA County
• Update the team about plans moving forward following the meeting

TEAMS
Graphic Design Team (Sery, Katie)

• Create the additional pages:
  About Us page, Organization profile, LA County Homeless Initiative about page, Contact form

UI Experience Team (Tinnie, Bobby, Sery, April)

• Finish about us page, organization profile, contact page, home/search page, detailed job page
• Finalize the pages on envision

Tech Team (Anthony, Hayden, Ryan)

• Create the repository with the project structure
• Create a read me page that serves as an onboarding document for developers
• Develop job search and specific job page

Administration (Nina, Katie)

• Prepare key talking points for the presentation
• Prepare a set of suggestions for our collaboration
• Create a presentation deck outlining the solution to the problem along with current and future progress, end goals, and desired outcome
• Outline:
  Introduction to the website, problem statement, solution, demo of solution, next steps

2018-07-10 Notes

Who's here

• Bobby
• Ryan
• Nicole
• Nina
• April
• Monica
• Tinnie

Notes

Strategy:

• Manually gather the data because much of it is unstructured.
• Beyond the MVP - utilize various strategies to get structured data so we can scrape the pages and get automatic updates.

What problem are we trying to solve? Problem Statement:

• We are trying to find jobs for individuals in a quick, efficient, and easy way, and this brings value to the individual because it enables community members to get paid for meaningful work.

What's our end goal for the MVP?

• A user looking to find work with homeless service providers will be able to find a job matching their skillset in order to help reduce homelessness.

Features

• Search functionality
• The nice thing is the sites don't have a login wall.

Action items & notes

• Create a schema for the data we want to collect.
  • Look at the listings and come up with ideas on data trends to see what data we can collect.
• Scrape linked pages and aggregate content onto a single page so that we get one source of all listings?
• Have MVP each person get data from 5-6 links?
• Focus on "Why it matters" in our presentation of the MVP? What is the satisfaction/benefit the user will have by getting these jobs (focus on how they'll feel)?

1. Create Google Form for data entry into Google Sheet.
2. Assign groups of links to group members.
3. Enter data into form (note details about listings).

Schema Notes

• location
• job title
• skills
• responsibilities
• duration
• salary

Assignments for Data Entry (6 job listings)

• Bobby: Antelope Valley
• Ryan: San Fernando Valley
• Nicole: San Gabriel Valley
• Nina: Metro, South Bay
• April: West
• Monica: South
• Tinnie: East

make sure we're following auth best practices

this is my first time building an auth system so if you have exp in this area I would appreciate a set of fresh eyes to make sure I'm doing everything correctly!

Styled, Responsive Header. Move SearchBox to Main

Need to:
Header:
-Style and make header fully responsive.
-Move react router 'NavLinks' to Navbar.

SearchBox:
-move SearchBox to appropriate page (Main page)

Account Page

Basically add stuff to the account page which is very barebones right now

for admin

• add an orgs list with options to edit org info / delete organizations

for employers

• add a my orgs widget that also links to editing org info
• add a my jobs widget that also links to create new job, edit job, delete job
• add a way to add second and third associated organizations to account

if you think of more stuff that you think would be cool feel free to add to this

Banner component

Need a Banner component.
design guide: https://projects.invisionapp.com/share/BTN6GLIJNUY#/screens/311697394

Guidelines:
-to be used in the Organizations, About, Contact pages
-should be responsive
-hardcoded contact banner is up (can use this as a reference)
-state handling should be done using Redux
-styling should be done in Sass(.scss file)
-need images for Org, and About banner-you can choose just use similar photos as in the design guide(subject to approval)-can use any free stock photos (ex. pexels.com or unsplash.com)

find banner images for login / register

Create bookmark functionality for job seekers

Allow users to save jobs. Have them saved in Accounts Page.

attach mailto for contact form

this is on hold until we get an email to send information to.

Use RESTful Routes for Organizations

Show all orgs:
/organizations

Show individual orgs:
/organizationView --> organizations/:id

Edit Individual orgs:
/organizations/:id/edit

Add organization flag to switch between scraping and manual entry