wazbat / neos-website-archive

Archive of the neos.com website

Neos Website Archive

An archive of the neos.com website. Running automatically every 3rd hour

Last scrape status

How to use

A local viewing friendly scrape output is located inside the neos-local folder. If you wish to browse a specific historical revision you can use the commit history to find the revision you want. This folder should also contain any PDFs or images that were uploaded to the page.

There is also a raw unmodified version of the website html in the neos-raw folder, should httrack break the website when scraping it.

If you wish can clone the repo locally and navigate to a specific commit hash using the following command:

git checkout <commit hash>

When trying to view the neos-local website, you might run into some CORS errors, causing jQuery to not load, and subsequently the fancy scrolling text to not show. To fix that, just edit the neos-local/neos.com/index.html file and remove the following properties from the jQuery script tag.

integrity="sha256-9/aliU8dGd2tb6OSsuzixeV4y/faTqgFtohetphbbj0=" crossorigin="anonymous"

(Just ctrl + f for the word jquery and you'll find it)

Then just save and reload the page and it should display

PRs welcome!

neos-website-archive's Issues

HTML is minified in the local scrape output

The local and raw scrapes are both minified. This makes undersanding the git diff's rather tedious. Also, httrack inserts a timestamp into the resulting html file which means that the file is updated with every single scrape.

I'm not experienced with httrack, but if possible we should use some kind of command line option to prevent those comments being added.

Also, if possible, the HTML should be prettified. I've experimented a little and it looks like it'd be relatively easy with a basic python script (should httrack not support this natively)

from bs4 import BeautifulSoup as bs
import codecs
file=codecs.open("neos-local/neos.com/index.html", 'r', 'utf-8')
raw_html = file.read()
soup = bs(raw_html)
prettyHTML = soup.prettify()
print(prettyHTML)
# The above should instead be saved to a file

The above script could be ran as a step inside the github actions local scrape job and just save the output in a new file next to it. Maybe also remove the comments too

Local friendly version of the website scrape takes too long

For some reason one of the scrape jobs takes ~17 minutes to complete, while the raw html scrape takes 15 seconds. Not sure what's causing this, but it's likely part of the httrack command. Might be a download speed limit, or it's scraping too deep?