Comments (7)
Keeping it static and less complex is also in my favor. I also don't see the use case?
An elaboration of the intended workflow where a different folder name is useful would be great!
from website-stalker.
The downside are many pages on different domains but I think most people have not that many different domains? My public example would have 21 folders which seems fine.
Git handles renames kinda well so I don't think the breaking change would be that annoying.
what's interesting are domains without a path. https://edjopato.de/
for example. Maybe use index for that? with a editor it could also know the extension (html in this case) de-edjopato/index.html
.
Also @Teufelchen1 I would remove the sites folder and put everything plain into the main repo. Otherwise there are only .github
, README.md
, sites
and website-stalker.yaml
which isn't that much. having it with less directory depth seems better to me?
from website-stalker.
The main filename (website-stalker.yaml
) is also not changeable currently. It has benefits to the complexity with the git handling and so on so I like that its a bit more static.
What would be your use case of having a different folder name there?
from website-stalker.
I also discussed it with @Teufelchen1 so maybe there are some different thoughts?
from website-stalker.
@EdJoPaTo @Teufelchen1 thanks for considering this.
I already have my own hand-rolled version of this, where I scrape hundreds of government and parliament sites worldwide, extract lists of people from them, and then compare that against information in Wikidata. Currently each source is scoped to its own directory, which keeps things cleaner and more manageable, and makes it easier to build reporting etc.
I would like to replace large chunks of this workflow with website-stalker
, as you have a lot of very nice functionality that my much simpler curl-and-commit version doesn't have, but I would like to maintain the one-directory-per-source structure as much as possible.
from website-stalker.
We discussed your idea and concluded it would be a nice to sort the output in folders per domain. Thank you for the additional description!
E.g.:
├── sites
│ └── de-edjopato
│ ├── post.html
│ └── robots.txt
Would that work for you?
However, this would be a breaking change, so care must be taken (git history?). We would also like to keep it as simple as possible; If the folder structure shown (one folder per TLD) is non configurable, would it still fulfill your feature request?
from website-stalker.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
from website-stalker.
Related Issues (20)
- consider switching to `git-repository` crate HOT 2
- Current main can't parse it's own example config HOT 1
- Replace integrated git support by usage of the existing git executable HOT 1
- Overriding filename? HOT 7
- Check robots.txt HOT 2
- Commit per Domain or all HOT 1
- Create issue for the changes with link for the commit HOT 2
- Latest version breaks Github Action HOT 8
- Sorting/Ordering is changed, but no change in content HOT 11
- One commit per host HOT 1
- Remove `check` sub-command HOT 2
- command editor
- Self-signed certificate not recognized HOT 6
- Commit message based on changed site
- Allow grouping of links HOT 2
- Use environment variable for `from` HOT 2
- Default extension HOT 1
- html_prettify: sort class and style attributes HOT 1
- Select parts of a json
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from website-stalker.