pixep / crowlet Goto Github PK
View Code? Open in Web Editor NEWTiny sitemap crawler for cache warming, and website status monitoring
License: Apache License 2.0
Tiny sitemap crawler for cache warming, and website status monitoring
License: Apache License 2.0
How can I use this option? It seems to be ignored.
crawler seem to be using a lot of memory.
I have a website with ~22K URLs that I need to crawl. I was running crawler on AWS EC2 t2.micro
, which has ~1 GB of memory.
Crawler was leading this instance to freeze.
I was testing locally with 1GB memory limit, crawler hits it.
I think 1GB of memory is too much for simple crawler.
I'm not a Go expert and I'm not able to identify memory leak in the code.
I assume that it is about not releasing/cleaning the response results for each crawled URL.
I used this:
docker run -it --rm aleravat/crowlet --forever --crawl-hyperlinks --summary-only --debug --wait-interval 1800 https://webapplicationconsultant.com/sitemap.xml
It didn't exit when I pressed Ctrl+c It happens ONLY when it is waiting for 1800 seconds to be over - so when the first crawl is done.
hello
docker run -it --rm aleravat/crowlet --forever --wait-interval 5 https://website.com/sitemap.xml
I am running this command, 5 seconds for testing obviously, however the console is not displaying new crawls after the first batch.
Any ideas? Thanks :)
Any internal link, including:
Not sure if i am doing anything wrong but regardless of any crawl i perform i just get 9223372036854 as a time :-/
using aleravat/crowlet@sha256:1c23f62c5328f5e79c5b347e0eb4bf10223ebca0bbff80bc38d83206ef79141e
I have a quick question, does it, or can it be implemented to recursively crawl sitemaps/xml files? Or is it only first-level links in the Sitemaps? I'd like to create a Public Sitemap of Sitemaps so I only need to update the container once - if that makes sense.
Thanks!
subj :)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.