Comments (5)
[deleted comment]
from abot.
I love the idea of having more information about the links but am hesitant on
adding anymore parsing than needs to happen since most people wouldn't need the
link text or need to know if the link was an image. Can you first attach the
impl that actually fills/returns the list of ILinkInfo object so I can take a
quick look?
Thank you for offering your code!!!!
Original comment by [email protected]
on 18 Dec 2013 at 9:40
- Changed state: Accepted
- Added labels: ****
- Removed labels: ****
from abot.
My code is here, I have changed the HyperLinkParser to return a list of
ILinkInfo, instead of the Uri-list that is returned now. In addition I have
changed the interface for the PageRequester to crawl PageToCrawl objects
directly instead of the Uri-object it currently accepts. When I want to crawl
extra metadata I can then subclass ILinkInfo and update my own HyperLinkParser
accordingly. The only remaining implementation would be to implement something
like a PageToCrawl.Bag for storing the metadata. I have done this the ugly way
locally (By just modifying the PageToCrawl class), so I am not sharing that
code. Also, I have not updated the CsQueryHyperLinkParser, as I am using the
HAP-parser:)
I dont know if this is the best way of implementing the described
functionality, but I have made an attempt at least, so just let me know if you
like it :) I havent tested it, but I assume it will work just fine :)
Modified files are attached.
Original comment by [email protected]
on 19 Dec 2013 at 7:53
- Added labels: ****
- Removed labels: ****
Attachments:
from abot.
fyi, v1.2.3 already has a PageToCrawl.PageBag of dynamic expando type.
I'll take a look at your impl and get back to you. Thanks again.
Original comment by [email protected]
on 19 Dec 2013 at 6:04
- Added labels: ****
- Removed labels: ****
from abot.
As of right now, i don't think I will pull your changes into the product due to
the reasons I stated above. However, i may change my position in the future.
Thanks for offering your implementation. Your time is appreciated.
Original comment by [email protected]
on 30 Dec 2013 at 3:12
- Changed state: WontFix
- Added labels: ****
- Removed labels: ****
from abot.
Related Issues (20)
- When set to not crawl external links, They are still added to scheduler to be crawled and later skipped HOT 9
- Create an auto encoding solution that detects the decoding of each page based on headers HOT 9
- Converting RawContent to a byte stream and writing to disk corrupts the data HOT 9
- It's great but could not save session (it's very important) HOT 7
- Abot.SiteSimulator.casproj can not be opened HOT 4
- How to download the crawled web pages HOT 1
- Incorrectly parsing robots.txt HOT 13
- RobotsDotText implementation disallows all external pages HOT 2
- MaxPagesToCrawl is broken HOT 2
- Remove all marked with [Obsolete] attribute
- Very slow HOT 7
- Encoding issue HOT 1
- Retry crawl on webexception because of a bad proxy HOT 1
- Possibility to stop crawling (and start crawling async) HOT 1
- Github bugs/issues HOT 1
- Using ampersand in URLs causes "Page has no content" HOT 1
- 'Abot.Poco.CrawlResult' does not contain a definition for 'ErrorMessage' HOT 1
- AutoMapper MappingException after update HOT 8
- I can't crawl all the pages of a site HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from abot.