schollz / websitechanges

Alerts you via email about a website change.

websitechanges

Alerts you via email about a website change, with an image of the change.

After writing a website change script in Python, I realized I could make it much better using Go+Node. This code was very helpful for me to get registered for a high-demand pottery class.

In this version I'm using puppeteer to do the capturing. The puppeteer code first filters out any ads (to prevent new things appearing in screen) and then captures an image of a CSS-selected content. The change-detection happens in Go and sends an email with an image of the change when any is detected.

Example

Here is an example of a change detected from the New York times headlines.

The previous state is shown in red and the new state is shown in green.

The config.json used was (note the folder key is optional and is used to customize the folder name of images for easier recognition.):

{
    "watchers": [
    {
        "url": "https://www.nytimes.com",
        "css": "h2 > span",
        "folder": "nytimes"
    }]
}

Usage

First make sure you have node and go installed on your system.

Then download and build:

> git clone https://github.com/schollz/websitechanges
> cd websitechanges
> go build -v

Now copy the config file and fill it in (you don't have to include email, just erase that section if you don't need it):

> cp config-example.json config.json
> vim config.json

Now run. The first time you run it will download a HOSTS file that is used for filtering adds and it will install puppeteer if not installed.

> ./websitechanges

By default this will run every 5 minutes. You can adjust the delay by passing the -delay 10 where 10 is the number of minutes to wait before the next run.

> ./websitechanges -delay 10

It automatically generates diff images everytime it encounters a change.

Contributing

Pull requests are welcome. Feel free to...

• Revise documentation
• Add new features
• Fix bugs
• Suggest improvements

License

MIT

websitechanges's Issues

Will this work on Amazon price checking?

I'm looking to track pricing on Amazon pages.

Feature Request: Add option to adjust delay from commandline

Greetings,

I stumbled on this repo when trying to figure out a way to make my job (and the life of my PM) a little easier by automatically finding issues with sites we work on. I love the simplicity and out of the box it has most of what I need. I did have 2 simple requests.

1. Add the ability to pass a delay flag on the command line instead of having it hardcoded in main.go.
2. Have a way to customize the folder name for images to make it easier to recognize.

Already created a PR for this here: #2

images