sixtoad / poc-sitemap-crawler

= Spring Boot POF of sitemap crawler

The crawler is limited to one domain. Given a starting URL it will visit all pages within the domain and subdomains, but not follow the links to external sites such as Google or Twitter.

The output is a simple site map, showing links to other pages under the same domain, links to static content such as images, and to external URLs.

== Description

This project is using JSoup internally to analyze pages.

Main tradeoffs:

• Has a very basic control of crawler blocks, only retry after 1 second.
• It is not smart and can go into infinitive loop in case timeout in a page, eventually it will be abe to retrieve it.
• Even recognize moved pages, will show up this links in the sitemap
• Because the size of sitemap can be huge, the server flush the buffer to ensure there is not a timeout in the browser meanwhile is crawling.
• Not very good control of errors during the crawling, can be better.
• Link parsing has issues with some characters, so this links are lost in the sitemap.
• It is not multithread so the crawling in websites not protected against crawling will take longeer than could

=== Test the system

Follow these instructions to execute application using embedded tomcat server:

• Generate project distribution using mvn clean install command on Parent module.
• Execute mvn spring-boot:run command on rest-api module (contains .war)
• Open browser on http://localhost:8080
• Test http://localhost:8080/sitemap?url=http://yourdomain.tld. It will send back a sitemap.xml
• If you are in a Linux based shell use curl. You will see how the service is sending all the links t the caller.

 curl http://localhost:8080/sitemap?url=

=== SONARQUBE support

This project is integrated with sonarqube. This trigger a sonar check in verify phase, to use it you have two options.

=== Existing sonar server If you have already a sonar server you only need to modify the sonar.host.url property in dev-sonar profile and then run maven with this profile:

mvn verify -P dev-sonar

The project is configured to do not send the violations to Sonar, so it will not create a project and sonar will not store the statistics. I assume you have in your settings.xml configured a valid user for your sonar server, please review sonar documentaion if your sonar server is not configured to allow Anonymous user and you do not have a valid user configured.

*** Not sonar server available *** If you want to run sonar checks but you do not have a SonarQube server available, you can install docker machine in your computer. After it is installed you only need to run:

mvn compile -P sonarqube

And it will start a container with a default sonar.

[INFO] --- docker-maven-plugin:0.18.1:start (prepare-sonar) @ pom ---
[INFO] DOCKER> Docker machine "default" is running
[INFO] DOCKER> DOCKER_HOST from docker-machine "default" : tcp://192.168.99.100:2376
[INFO] DOCKER> [sonarqube:6.2] "sonarqube": Start container 732454c3e89b
[INFO] DOCKER> [sonarqube:6.2] "sonarqube": Waited on log out 'SonarQube is up' 41801 ms

Use Kitematic to see what port has been assigned to the container and update sonar.host.url property in dev-sonar profile.

To get the sonar checks you only need to run mvn verify -P dev-sonar.