Welcome to Dourado! This project is designed to bring info about a website sitemaps.
- robots_exists(website_url): β
Checks if
robots.txt
exists in the website root and returns a boolean. - valid_sitemaps_robots(website_url): π Verifies if there is at least one available and valid sitemap in
robots.txt
and returns a boolean. - website_sitemaps(website_url): π Retrieves and returns a list of all sitemaps from the given website.
- pages_from_sitemaps(website_url): π Extracts and returns a list of page URLs from the sitemaps of the given website.
- sitemap_indicated_on_robots(website_url): π Checks if at least one sitemap is indicated in the
robots.txt
file of the given website and returns a boolean.
You can install the package via pip:
pip install GIT+https://github.com/alexruco/dourado
Alternatively, you can clone the repository and install the dependencies from requirements.txt:
git clone https://github.com/alexruco/dourado.git
cd dourado
pip install -r requirements.txt
Here's a quick example to get you started:
from dourado import dourado
# Example usage
website = 'https://mysitefaster.com/'
print(f"valid_sitemaps_robots: {dourado.valid_sitemaps_robots(website_url=website)}")
print(f"website_sitemaps: {dourado.website_sitemaps(website_url=website)}")
print(f"pages_from_sitemaps: {dourado.pages_from_sitemaps(website_url=website)}")
print(f"robots_exists: {dourado.robots_exists(website_url=website)}")
print(f"sitemap_indicated_on_robots: {dourado.sitemap_indicated_on_robots(website_url=website)}")
## Contributing π€
We welcome contributions from the community! Hereβs how you can get involved:
1. **Report Bugs**: If you find a bug, please open an issue [here](https://github.com/alexruco/dourado/issues).
2. **Suggest Features**: Weβd love to hear your ideas! Suggest new features by opening an issue.
3. **Submit Pull Requests**: Ready to contribute? Fork the repo, make your changes, and submit a pull request. Please ensure your code follows our coding standards and is well-documented.
4. **Improve Documentation**: Help us improve our documentation. Feel free to make edits or add new content.
### How to Submit a Pull Request
1. Fork the repository.
2. Create a new branch: `git checkout -b my-feature-branch`.
3. Make your changes and commit them: `git commit -m 'Add some feature'`.
4. Push to the branch: `git push origin my-feature-branch`.
5. Open a pull request on the original repository.
## Contact information
For any questions or suggestions, please contact:
Alex Ruco: alex@ruco.pt
GitHub: https://github.com/alexruco
## Honor
Named in the honor of FernΓ£o Dourado, a famous Portuguese cartographer of the sixteenth century
## License π
This project is licensed under the MIT License. Feel free to use, modify, and distribute this software in accordance with the terms outlined in the [LICENSE](LICENSE) file.