apophis981 / shackscout

shackscout

Service for identifying new craigslist listings that are desirable and notifying when they become listed.

Grabs the latest housing urls from craigslist rss feed, scrapes data for each new listing, and saves it to a local database. When discovering a new listing that fits the user parameters it notifies the user.

Setup:

pip install -r requirements.txt

Before running script start Mongdb:

sudo service mongod start

Usage:

For help run python3 shackscout.py -h

Input search info as command arguments in the format shown below

Arguments in [] are optional, arguments outside of brackets are mandatory

python3 shackscout.py [positional args] [optional args]

Positional args:

distance geo geo pricemin pricemax

Optional args

[-h] [--bedrooms] [--spacemin] [--spacemax] [--cats] [--dogs] [--wd]

Example:

python3 shackscout.py 5 37.557516 -122.287266 2000 3000 --bedrooms 1 --cats

shackscout's Issues

Add ability to search more than one zip code

Need to research the best way to search for listings in the areas that are important to the user
zillow allows you to make a rectangle
example: 37.582439,-122.225905,37.526643,-122.31371_rect
perhaps we can search for a square which is a certain distance away from their work and draw a square

Craigslist can search by miles from zip which is ideal:
https://sfbay.craigslist.org/search/apa?search_distance=5&postal=94404&availabilityMode=0&sale_date=all+dates

Allow user to specify the type of residence they're interested in

Zillow allows any combination of
Houses
Apartments
Townhomes
Condos/co-ops
Manufactured
Lots/Land

The url builder will have to construct a comma separated list of types based on user input
example:
/house,condo,apartment_duplex,mobile,land_type/
If everything is checked, the field is removed from the url

Create a better database

The database needs the following functionality:

1. Store the following data:
   'address':address,
   'city':city,
   'state':state,
   'postal_code':postal_code,
   'price':price,
   'facts and features':info,
   'real estate provider':broker,
   'url':property_url,
   'title':title
   rank?
2. The database needs to be mutable so that after evaluating each result we can rank them.
3. It should be able to check if newly scraped results exist in the database already and if not add them

Add GPS coordinate house info to zillow scraper

Currently the zillow scraper grabs the following information about homes:
address,
city,
state,
postal_code,
price,
info,
broker,
property_url,
title

The gps coordinates of the house should also be available somewhere.

Add proper url building

Use a python url building library to take user input for zipcode, price range, and type of home and construct a zillow url to search for it.
Probably urljoin() from https://docs.python.org/3/library/urllib.parse.html

Use postgresql database instead of MongoDB