Containerized nvidia-snatcher about streetmerchant

[matthewLee711]

As a quick heads up everyone, my nvidia-snatcher on AWS got blocked. I don't have much experience when it comes to scraping pages, but I recently found this guide to help your bot run a little while longer.

https://www.reddit.com/r/programming/comments/ecvc42/a_guide_to_web_scraping_without_getting_blocked/fbei5dp?utm_source=share&utm_medium=web2x&context=3

I do want to mentioned, the easiest way prevent getting blocked, is by limiting your rate. Change your request interval to 30+ seconds.

[jef]

I think getting this to not use puppeteer and separating logic would help with containerization. Related to #113

[ljmerza]

puppeteer needs a display to work. xvfb will do the trick

 RUN apt-get update && \ 
     apt-get install -yq --no-install-recommends \ 
     libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 \ 
     libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 \ 
     libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 \ 
     libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 \ 
     libnss3 

# Start script on Xvfb
CMD Xvfb :99 -screen 0 1024x768x16 & npm start

I have a docker container for nodejs and xvfb as well: lmerza/xvfb

[geman220]

Do you mind if we close this out?

[matthewLee711]

I recommend we keep it open, as I feel the information/discussion is helpful for others going down the path of containerization and web scraping on cloud providers. However, if you feel it is necessary to close, I'm 100% fine with you going forward on it.

[geman220]

It will still be accessible, just closed. Or you can make a Wiki page https://github.com/jef/nvidia-snatcher/wiki . I'm just not sure what to do with this in Issue tracking.

[jef]

Related: #174

[jef]

Not sure what I want do with Docker yet. There are some hurdles and hacks and I don't know if I want to put anything inside the repository.

Feel free to create a Wiki page with instructions on installation. I don't know if I want to support this yet.

[geman220]

Closing due to the merge of #209 there is also #174. Feel free to create a Wiki page for instructions on docker.