Coder Social home page Coder Social logo

yahooarchive / anthelion Goto Github PK

View Code? Open in Web Editor NEW
2.8K 2.8K 668.0 34.5 MB

Anthelion is a plugin for Apache Nutch to crawl semantic annotations within HTML pages.

Home Page: https://labs.yahoo.com/publications/6702/focused-crawling-structured-data

License: Apache License 2.0

Java 96.64% XSLT 0.08% Shell 0.65% HTML 2.63%

anthelion's People

Contributors

aaroncritchley avatar petarr avatar robertmeusel avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

anthelion's Issues

Propose Anthelion for Nutch Trunk

Hi Folks,
Are you interested in proposing Anthelion for integration into the Nutch trunk source code?
I think I've spoken with a few of you over on the Any23 ML and I am very glad to see you publish the outcome of the work as source code.
What do you guys think? I am one of the main developers of Nutch and Any23 so I am very interested in seeing this plugin available for more to use.

Add Desc

"Anthelion is a Nutch plugin for focused crawling of semantic data."

nutch 2.2 not compatible

when I follow the instructions 1-9 for nutch 2.2, some files in not found and is in nutch 1.6 library:

compile:
[echo] Compiling plugin: parse-anth
[javac] Compiling 13 source files to NUTCH2.2_ROOT/build/parse-anth/classes
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.6
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/indexing/TripleExtractor.java:23: error: cannot find symbol
[javac] import org.apache.nutch.crawl.CrawlDatum;
[javac] ^
[javac] symbol: class CrawlDatum
[javac] location: package org.apache.nutch.crawl
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/indexing/TripleExtractor.java:24: error: cannot find symbol
[javac] import org.apache.nutch.crawl.Inlinks;
[javac] ^
[javac] symbol: class Inlinks
[javac] location: package org.apache.nutch.crawl
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/indexing/TripleExtractor.java:45: error: cannot find symbol
[javac] public NutchDocument filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) {
[javac] ^
[javac] symbol: class CrawlDatum
[javac] location: class TripleExtractor
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/indexing/TripleExtractor.java:45: error: cannot find symbol
[javac] public NutchDocument filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) {
[javac] ^
[javac] symbol: class Inlinks
[javac] location: class TripleExtractor
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/parsing/WdcParser.java:42: error: cannot find symbol
[javac] import org.apache.nutch.parse.HtmlParseFilters;
[javac] ^
[javac] symbol: class HtmlParseFilters
[javac] location: package org.apache.nutch.parse
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/parsing/WdcParser.java:45: error: cannot find symbol
[javac] import org.apache.nutch.parse.ParseData;
[javac] ^
[javac] symbol: class ParseData
[javac] location: package org.apache.nutch.parse
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/parsing/WdcParser.java:46: error: cannot find symbol
[javac] import org.apache.nutch.parse.ParseImpl;
[javac] ^
[javac] symbol: class ParseImpl
[javac] location: package org.apache.nutch.parse
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/parsing/WdcParser.java:47: error: cannot find symbol
[javac] import org.apache.nutch.parse.ParseResult;
[javac] ^
[javac] symbol: class ParseResult
[javac] location: package org.apache.nutch.parse
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/parsing/WdcParser.java:48: error: cannot find symbol
[javac] import org.apache.nutch.parse.ParseStatus;
[javac] ^
[javac] symbol: class ParseStatus
[javac] location: package org.apache.nutch.parse
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/parsing/WdcParser.java:143: error: cannot find symbol
[javac] private HtmlParseFilters htmlParseFilters;
[javac] ^
[javac] symbol: class HtmlParseFilters
[javac] location: class WdcParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/parsing/WdcParser.java:177: error: cannot find symbol
[javac] public ParseResult getParse(Content content) {
[javac] ^
[javac] symbol: class ParseResult
[javac] location: class WdcParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/scoring/AnthelionScoringFilter.java:30: error: cannot find symbol
[javac] import org.apache.nutch.crawl.CrawlDatum;
[javac] ^
[javac] symbol: class CrawlDatum
[javac] location: package org.apache.nutch.crawl
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/scoring/AnthelionScoringFilter.java:31: error: cannot find symbol
[javac] import org.apache.nutch.crawl.Inlinks;
[javac] ^
[javac] symbol: class Inlinks
[javac] location: package org.apache.nutch.crawl
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/scoring/AnthelionScoringFilter.java:34: error: cannot find symbol
[javac] import org.apache.nutch.parse.ParseData;
[javac] ^
[javac] symbol: class ParseData
[javac] location: package org.apache.nutch.parse
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/scoring/AnthelionScoringFilter.java:139: error: cannot find symbol
[javac] public CrawlDatum distributeScoreToOutlinks(Text fromUrl, ParseData parseData,
[javac] ^
[javac] symbol: class ParseData
[javac] location: class AnthelionScoringFilter
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/scoring/AnthelionScoringFilter.java:140: error: cannot find symbol
[javac] Collection<Entry<Text, CrawlDatum>> targets, CrawlDatum adjust, int allCount)
[javac] ^
[javac] symbol: class CrawlDatum
[javac] location: class AnthelionScoringFilter
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/scoring/AnthelionScoringFilter.java:140: error: cannot find symbol
[javac] Collection<Entry<Text, CrawlDatum>> targets, CrawlDatum adjust, int allCount)
[javac] ^
[javac] symbol: class CrawlDatum
[javac] location: class AnthelionScoringFilter
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/scoring/AnthelionScoringFilter.java:139: error: cannot find symbol
[javac] public CrawlDatum distributeScoreToOutlinks(Text fromUrl, ParseData parseData,
[javac] ^
[javac] symbol: class CrawlDatum
[javac] location: class AnthelionScoringFilter
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/scoring/AnthelionScoringFilter.java:182: error: cannot find symbol
[javac] public void injectedScore(Text url, CrawlDatum datum) throws ScoringFilterException {
[javac] ^
[javac] symbol: class CrawlDatum
[javac] location: class AnthelionScoringFilter
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/scoring/AnthelionScoringFilter.java:188: error: cannot find symbol
[javac] public void initialScore(Text url, CrawlDatum datum) throws ScoringFilterException {
[javac] ^
[javac] symbol: class CrawlDatum
[javac] location: class AnthelionScoringFilter
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/scoring/AnthelionScoringFilter.java:199: error: cannot find symbol
[javac] public float generatorSortValue(Text url, CrawlDatum datum, float initSort) throws ScoringFilterException {
[javac] ^
[javac] symbol: class CrawlDatum
[javac] location: class AnthelionScoringFilter
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/scoring/AnthelionScoringFilter.java:205: error: cannot find symbol
[javac] public void passScoreBeforeParsing(Text url, CrawlDatum datum, Content content) throws ScoringFilterException {
[javac] ^
[javac] symbol: class CrawlDatum
[javac] location: class AnthelionScoringFilter
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/scoring/AnthelionScoringFilter.java:211: error: cannot find symbol
[javac] public void updateDbScore(Text url, CrawlDatum old, CrawlDatum datum, List inlinked)
[javac] ^
[javac] symbol: class CrawlDatum
[javac] location: class AnthelionScoringFilter
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/scoring/AnthelionScoringFilter.java:211: error: cannot find symbol
[javac] public void updateDbScore(Text url, CrawlDatum old, CrawlDatum datum, List inlinked)
[javac] ^
[javac] symbol: class CrawlDatum
[javac] location: class AnthelionScoringFilter
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/scoring/AnthelionScoringFilter.java:211: error: cannot find symbol
[javac] public void updateDbScore(Text url, CrawlDatum old, CrawlDatum datum, List inlinked)
[javac] ^
[javac] symbol: class CrawlDatum
[javac] location: class AnthelionScoringFilter
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/scoring/AnthelionScoringFilter.java:218: error: cannot find symbol
[javac] public float indexerScore(Text url, NutchDocument doc, CrawlDatum dbDatum, CrawlDatum fetchDatum, Parse parse,
[javac] ^
[javac] symbol: class CrawlDatum
[javac] location: class AnthelionScoringFilter
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/scoring/AnthelionScoringFilter.java:218: error: cannot find symbol
[javac] public float indexerScore(Text url, NutchDocument doc, CrawlDatum dbDatum, CrawlDatum fetchDatum, Parse parse,
[javac] ^
[javac] symbol: class CrawlDatum
[javac] location: class AnthelionScoringFilter
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/scoring/AnthelionScoringFilter.java:219: error: cannot find symbol
[javac] Inlinks inlinks, float initScore) throws ScoringFilterException {
[javac] ^
[javac] symbol: class Inlinks
[javac] location: class AnthelionScoringFilter
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/org/apache/nutch/parse/html/HtmlParser.java:112: error: cannot find symbol
[javac] private HtmlParseFilters htmlParseFilters;
[javac] ^
[javac] symbol: class HtmlParseFilters
[javac] location: class HtmlParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/org/apache/nutch/parse/html/HtmlParser.java:116: error: cannot find symbol
[javac] public ParseResult getParse(Content content) {
[javac] ^
[javac] symbol: class ParseResult
[javac] location: class HtmlParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/indexing/TripleExtractor.java:38: error: TripleExtractor is not abstract and does not override abstract method filter(NutchDocument,String,WebPage) in IndexingFilter
[javac] public class TripleExtractor implements IndexingFilter {
[javac] ^
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/indexing/TripleExtractor.java:50: error: cannot find symbol
[javac] containsSem = parse.getData().getMeta(WdcParser.META_CONTAINS_SEM);
[javac] ^
[javac] symbol: method getData()
[javac] location: variable parse of type Parse
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/parsing/WdcParser.java:74: error: WdcParser is not abstract and does not override abstract method getParse(String,WebPage) in Parser
[javac] public class WdcParser implements Parser {
[javac] ^
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/parsing/WdcParser.java:194: error: cannot find symbol
[javac] return new ParseStatus(e).getEmptyParseResult(content.getUrl(), getConf());
[javac] ^
[javac] symbol: class ParseStatus
[javac] location: class WdcParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/parsing/WdcParser.java:210: error: no suitable method found for autoDetectClues(Content,boolean)
[javac] detector.autoDetectClues(content, true);
[javac] ^
[javac] method EncodingDetector.autoDetectClues(ByteBuffer,Utf8,String,boolean) is not applicable
[javac](actual and formal argument lists differ in length)
[javac] method EncodingDetector.autoDetectClues(WebPage,boolean) is not applicable
[javac](actual argument Content cannot be converted to WebPage by method invocation conversion)
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/parsing/WdcParser.java:212: error: no suitable method found for guessEncoding(Content,String)
[javac] String encoding = detector.guessEncoding(content, defaultCharEncoding);
[javac] ^
[javac] method EncodingDetector.guessEncoding(String,String) is not applicable
[javac](actual argument Content cannot be converted to String by method invocation conversion)
[javac] method EncodingDetector.guessEncoding(WebPage,String) is not applicable
[javac](actual argument Content cannot be converted to WebPage by method invocation conversion)
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/parsing/WdcParser.java:223: error: cannot find symbol
[javac] return new ParseStatus(e).getEmptyParseResult(content.getUrl(), getConf());
[javac] ^
[javac] symbol: class ParseStatus
[javac] location: class WdcParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/parsing/WdcParser.java:225: error: cannot find symbol
[javac] return new ParseStatus(e).getEmptyParseResult(content.getUrl(), getConf());
[javac] ^
[javac] symbol: class ParseStatus
[javac] location: class WdcParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/parsing/WdcParser.java:227: error: cannot find symbol
[javac] return new ParseStatus(e).getEmptyParseResult(content.getUrl(), getConf());
[javac] ^
[javac] symbol: class ParseStatus
[javac] location: class WdcParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/parsing/WdcParser.java:230: error: cannot find symbol
[javac] return new ParseStatus(e).getEmptyParseResult(content.getUrl(), getConf());
[javac] ^
[javac] symbol: class ParseStatus
[javac] location: class WdcParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/parsing/WdcParser.java:282: error: cannot find symbol
[javac] ParseStatus status = new ParseStatus(ParseStatus.SUCCESS);
[javac] ^
[javac] symbol: class ParseStatus
[javac] location: class WdcParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/parsing/WdcParser.java:282: error: cannot find symbol
[javac] ParseStatus status = new ParseStatus(ParseStatus.SUCCESS);
[javac] ^
[javac] symbol: class ParseStatus
[javac] location: class WdcParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/parsing/WdcParser.java:282: error: cannot find symbol
[javac] ParseStatus status = new ParseStatus(ParseStatus.SUCCESS);
[javac] ^
[javac] symbol: variable ParseStatus
[javac] location: class WdcParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/parsing/WdcParser.java:284: error: cannot find symbol
[javac] status.setMinorCode(ParseStatus.SUCCESS_REDIRECT);
[javac] ^
[javac] symbol: variable ParseStatus
[javac] location: class WdcParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/parsing/WdcParser.java:288: error: cannot find symbol
[javac] ParseData parseData = new ParseData(status, title, outlinks, content.getMetadata(), metadata);
[javac] ^
[javac] symbol: class ParseData
[javac] location: class WdcParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/parsing/WdcParser.java:288: error: cannot find symbol
[javac] ParseData parseData = new ParseData(status, title, outlinks, content.getMetadata(), metadata);
[javac] ^
[javac] symbol: class ParseData
[javac] location: class WdcParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/parsing/WdcParser.java:289: error: cannot find symbol
[javac] ParseResult parseResult = ParseResult.createParseResult(content.getUrl(), new ParseImpl(text, parseData));
[javac] ^
[javac] symbol: class ParseResult
[javac] location: class WdcParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/parsing/WdcParser.java:289: error: cannot find symbol
[javac] ParseResult parseResult = ParseResult.createParseResult(content.getUrl(), new ParseImpl(text, parseData));
[javac] ^
[javac] symbol: class ParseImpl
[javac] location: class WdcParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/parsing/WdcParser.java:289: error: cannot find symbol
[javac] ParseResult parseResult = ParseResult.createParseResult(content.getUrl(), new ParseImpl(text, parseData));
[javac] ^
[javac] symbol: variable ParseResult
[javac] location: class WdcParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/parsing/WdcParser.java:292: error: cannot find symbol
[javac] parse.getData().getContentMeta().set(META_CONTAINS_SEM, Boolean.toString(containsSem));
[javac] ^
[javac] symbol: method getData()
[javac] location: variable parse of type Parse
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/parsing/WdcParser.java:298: error: cannot find symbol
[javac] entry.getValue().getData().getParseMeta().set(Nutch.CACHING_FORBIDDEN_KEY, cachingPolicy);
[javac] ^
[javac] symbol: method getData()
[javac] location: class Parse
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/parsing/WdcParser.java:372: error: cannot find symbol
[javac] System.out.println("data: " + parse.getData());
[javac] ^
[javac] symbol: method getData()
[javac] location: variable parse of type Parse
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/parsing/WdcParser.java:376: error: cannot find symbol
[javac] String contains = parse.getData().getMeta(META_CONTAINS_SEM);
[javac] ^
[javac] symbol: method getData()
[javac] location: variable parse of type Parse
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/parsing/WdcParser.java:420: error: cannot find symbol
[javac] this.htmlParseFilters = new HtmlParseFilters(getConf());
[javac] ^
[javac] symbol: class HtmlParseFilters
[javac] location: class WdcParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/org/apache/nutch/parse/html/DOMContentUtils.java:388: error: cannot find symbol
[javac] URL url = URLUtil.resolveURL(base, target);
[javac] ^
[javac] symbol: method resolveURL(URL,String)
[javac] location: class URLUtil
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/scoring/AnthelionScoringFilter.java:54: error: AnthelionScoringFilter is not abstract and does not override abstract method indexerScore(String,NutchDocument,WebPage,float) in ScoringFilter
[javac] public class AnthelionScoringFilter implements ScoringFilter {
[javac] ^
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/scoring/AnthelionScoringFilter.java:110: error: cannot find symbol
[javac] containsSem = Boolean.parseBoolean(parse.getData().getMeta(WdcParser.META_CONTAINS_SEM));
[javac] ^
[javac] symbol: method getData()
[javac] location: variable parse of type Parse
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/scoring/AnthelionScoringFilter.java:112: error: cannot find symbol
[javac] semFather = Boolean.parseBoolean(parse.getData().getMeta(WdcParser.META_CONTAINS_SEM_FATHER));
[javac] ^
[javac] symbol: method getData()
[javac] location: variable parse of type Parse
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/scoring/AnthelionScoringFilter.java:129: error: cannot find symbol
[javac] parse.getData().getContentMeta().set(WdcParser.META_CONTAINS_SEM_FATHER_FOR_SUB, Boolean.toString(containsSem));
[javac] ^
[javac] symbol: method getData()
[javac] location: variable parse of type Parse
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/scoring/AnthelionScoringFilter.java:98: error: method does not override or implement a method from a supertype
[javac] @OverRide
[javac] ^
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/scoring/AnthelionScoringFilter.java:146: error: cannot find symbol
[javac] for (Entry<Text, CrawlDatum> target : targets) {
[javac] ^
[javac] symbol: class CrawlDatum
[javac] location: class AnthelionScoringFilter
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/scoring/AnthelionScoringFilter.java:138: error: method does not override or implement a method from a supertype
[javac] @OverRide
[javac] ^
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/scoring/AnthelionScoringFilter.java:181: error: method does not override or implement a method from a supertype
[javac] @OverRide
[javac] ^
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/scoring/AnthelionScoringFilter.java:187: error: method does not override or implement a method from a supertype
[javac] @OverRide
[javac] ^
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/scoring/AnthelionScoringFilter.java:198: error: method does not override or implement a method from a supertype
[javac] @OverRide
[javac] ^
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/scoring/AnthelionScoringFilter.java:204: error: method does not override or implement a method from a supertype
[javac] @OverRide
[javac] ^
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/scoring/AnthelionScoringFilter.java:210: error: method does not override or implement a method from a supertype
[javac] @OverRide
[javac] ^
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/com/yahoo/research/scoring/AnthelionScoringFilter.java:217: error: method does not override or implement a method from a supertype
[javac] @OverRide
[javac] ^
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/org/apache/nutch/parse/html/HtmlParser.java:44: error: HtmlParser is not abstract and does not override abstract method getParse(String,WebPage) in Parser
[javac] public class HtmlParser implements Parser {
[javac] ^
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/org/apache/nutch/parse/html/HtmlParser.java:123: error: cannot find symbol
[javac] return new ParseStatus(e).getEmptyParseResult(content.getUrl(), getConf());
[javac] ^
[javac] symbol: class ParseStatus
[javac] location: class HtmlParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/org/apache/nutch/parse/html/HtmlParser.java:138: error: no suitable method found for autoDetectClues(Content,boolean)
[javac] detector.autoDetectClues(content, true);
[javac] ^
[javac] method EncodingDetector.autoDetectClues(ByteBuffer,Utf8,String,boolean) is not applicable
[javac](actual and formal argument lists differ in length)
[javac] method EncodingDetector.autoDetectClues(WebPage,boolean) is not applicable
[javac](actual argument Content cannot be converted to WebPage by method invocation conversion)
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/org/apache/nutch/parse/html/HtmlParser.java:140: error: no suitable method found for guessEncoding(Content,String)
[javac] String encoding = detector.guessEncoding(content, defaultCharEncoding);
[javac] ^
[javac] method EncodingDetector.guessEncoding(String,String) is not applicable
[javac](actual argument Content cannot be converted to String by method invocation conversion)
[javac] method EncodingDetector.guessEncoding(WebPage,String) is not applicable
[javac](actual argument Content cannot be converted to WebPage by method invocation conversion)
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/org/apache/nutch/parse/html/HtmlParser.java:149: error: cannot find symbol
[javac] return new ParseStatus(e).getEmptyParseResult(content.getUrl(), getConf());
[javac] ^
[javac] symbol: class ParseStatus
[javac] location: class HtmlParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/org/apache/nutch/parse/html/HtmlParser.java:151: error: cannot find symbol
[javac] return new ParseStatus(e).getEmptyParseResult(content.getUrl(), getConf());
[javac] ^
[javac] symbol: class ParseStatus
[javac] location: class HtmlParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/org/apache/nutch/parse/html/HtmlParser.java:153: error: cannot find symbol
[javac] return new ParseStatus(e).getEmptyParseResult(content.getUrl(), getConf());
[javac] ^
[javac] symbol: class ParseStatus
[javac] location: class HtmlParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/org/apache/nutch/parse/html/HtmlParser.java:156: error: cannot find symbol
[javac] return new ParseStatus(e).getEmptyParseResult(content.getUrl(), getConf());
[javac] ^
[javac] symbol: class ParseStatus
[javac] location: class HtmlParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/org/apache/nutch/parse/html/HtmlParser.java:187: error: cannot find symbol
[javac] ParseStatus status = new ParseStatus(ParseStatus.SUCCESS);
[javac] ^
[javac] symbol: class ParseStatus
[javac] location: class HtmlParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/org/apache/nutch/parse/html/HtmlParser.java:187: error: cannot find symbol
[javac] ParseStatus status = new ParseStatus(ParseStatus.SUCCESS);
[javac] ^
[javac] symbol: class ParseStatus
[javac] location: class HtmlParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/org/apache/nutch/parse/html/HtmlParser.java:187: error: cannot find symbol
[javac] ParseStatus status = new ParseStatus(ParseStatus.SUCCESS);
[javac] ^
[javac] symbol: variable ParseStatus
[javac] location: class HtmlParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/org/apache/nutch/parse/html/HtmlParser.java:189: error: cannot find symbol
[javac] status.setMinorCode(ParseStatus.SUCCESS_REDIRECT);
[javac] ^
[javac] symbol: variable ParseStatus
[javac] location: class HtmlParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/org/apache/nutch/parse/html/HtmlParser.java:193: error: cannot find symbol
[javac] ParseData parseData = new ParseData(status, title, outlinks,
[javac] ^
[javac] symbol: class ParseData
[javac] location: class HtmlParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/org/apache/nutch/parse/html/HtmlParser.java:193: error: cannot find symbol
[javac] ParseData parseData = new ParseData(status, title, outlinks,
[javac] ^
[javac] symbol: class ParseData
[javac] location: class HtmlParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/org/apache/nutch/parse/html/HtmlParser.java:195: error: cannot find symbol
[javac] ParseResult parseResult = ParseResult.createParseResult(content.getUrl(),
[javac] ^
[javac] symbol: class ParseResult
[javac] location: class HtmlParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/org/apache/nutch/parse/html/HtmlParser.java:196: error: cannot find symbol
[javac] new ParseImpl(text, parseData));
[javac] ^
[javac] symbol: class ParseImpl
[javac] location: class HtmlParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/org/apache/nutch/parse/html/HtmlParser.java:195: error: cannot find symbol
[javac] ParseResult parseResult = ParseResult.createParseResult(content.getUrl(),
[javac] ^
[javac] symbol: variable ParseResult
[javac] location: class HtmlParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/org/apache/nutch/parse/html/HtmlParser.java:199: error: cannot find symbol
[javac] ParseResult filteredParse = this.htmlParseFilters.filter(content, parseResult,
[javac] ^
[javac] symbol: class ParseResult
[javac] location: class HtmlParser
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/org/apache/nutch/parse/html/HtmlParser.java:203: error: cannot find symbol
[javac] entry.getValue().getData().getParseMeta().set(Nutch.CACHING_FORBIDDEN_KEY,
[javac] ^
[javac] symbol: method getData()
[javac] location: class Parse
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/org/apache/nutch/parse/html/HtmlParser.java:281: error: cannot find symbol
[javac] System.out.println("data: "+parse.getData());
[javac] ^
[javac] symbol: method getData()
[javac] location: variable parse of type Parse
[javac] NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/org/apache/nutch/parse/html/HtmlParser.java:289: error: cannot find symbol
[javac] this.htmlParseFilters = new HtmlParseFilters(getConf());
[javac] ^
[javac] symbol: class HtmlParseFilters
[javac] location: class HtmlParser
[javac] Note: NUTCH2.2_ROOT/src/plugin/parse-anth/src/java/org/apache/nutch/parse/html/DOMBuilder.java uses unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 89 errors
[javac] 1 warning

BUILD FAILED
NUTCH2.2_ROOT/build.xml:108: The following error occurred while executing this line:
NUTCH2.2_ROOT/src/plugin/build.xml:29: The following error occurred while executing this line:
NUTCH2.2_ROOT/src/plugin/build-plugin.xml:117: Compile failed; see the compiler error output for details.

Hi

image
what's the meaning

Seems the HDFS file path contains ':' colon will throw exception

I had download this whole source code and built it successfully. When i tried to run a crawl test:
bin/crawl urls/ TestCrawl/ http://localhost:8983/solr/nutch 2
I run into this URI path name issue.
hadoop.log.zip

i have this log file attached. It seems the HDFS file path name special characters issue is still there?

2016-01-03 13:27:08,405 INFO fetcher.Fetcher - Fetcher: starting at 2016-01-03 13:27:08
2016-01-03 13:27:08,405 INFO fetcher.Fetcher - Fetcher: segment: TestCrawl/segments/drwxr-xr-xnn4nstevennstaffnn136nJannn3n13:24n20160103090925
2016-01-03 13:27:08,406 INFO fetcher.Fetcher - Fetcher Timelimit set for : 1451809628406
2016-01-03 13:27:08,631 WARN util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-01-03 13:27:08,677 ERROR fetcher.Fetcher - Fetcher: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: drwxr-xr-xnn4nstevennstaffnn136nJannn3n13:24n20160103090925
at org.apache.hadoop.fs.Path.initialize(Path.java:148)
at org.apache.hadoop.fs.Path.(Path.java:126)
at org.apache.hadoop.fs.Path.(Path.java:50)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.