cdimascio / essence Goto Github PK
View Code? Open in Web Editor NEWAutomatically extract the main text content (and more) from an HTML document
License: Apache License 2.0
Automatically extract the main text content (and more) from an HTML document
License: Apache License 2.0
in the readme the link https://essence.mybluemix.net/ is mentioned for a "Try out the demo", this link is not working any more.
Is essence still maintained?
It looks like essence has a problem with the content of the following pages:
It looks like these are specifically problems with these pages?
At least, that's the only place I've had a problem so far. But of course, I'm only using a few feeds so far.
In general essence is really good and does exactly what it is supposed to. Ingenious!
If the project is still being maintained, it would be great to have a look and see if you can fix the problem or if there is a workaround.
Hi,
I was testing it with the Yahoo finance website, however, it's unable to get the text. For example, this post data is not parsed and I get the empty text: https://finance.yahoo.com/news/3-airline-stocks-ready-takeoff-080507562.html
Not sure whether I get it right. in the DocumentScorer.kt, I think the code here is using wrong judgement:
class DocumentScorer(private val stopWords: StopWords) : Scorer {
override fun score(doc: Document): ScoredElement? {
val nodesWithText = mutableListOf<Element>()
val nodesToCheck = doc.select("p, pre, td")
nodesToCheck.forEach { node ->
val text = node.text()
val wordStats = stopWords.statistics(text)
val hasHighLinkDensity = NodeHeuristics.hasHighLinkDensity(node)
// if stopWords.size is bigger than 2, this node should be ignored, rather than added to nodesWithText?
// this should be changed to: wordStats.stopWords.size <= 2
if (wordStats.stopWords.size > 2 && !hasHighLinkDensity) {
nodesWithText.add(node)
}
}
......
}
}
I think we meant to find the the nodes with good text, and not containing a lot of stopwords, right?
The tokenization logic should be more generic.
Could use something like: https://www.atilika.org/ , to tokenize Japanese.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.