hxx97 / edgqa Goto Github PK
View Code? Open in Web Editor NEWCodes for "EDG-based Question Decomposition for Complex Question Answering over Knowledge Bases"
License: GNU General Public License v3.0
Codes for "EDG-based Question Decomposition for Complex Question Answering over Knowledge Bases"
License: GNU General Public License v3.0
In the file "src/main/java/cn/edu/nju/ws/edgqa/main/EDGQA.java", in the function "private static void computeMetrics(long startTime, CumulativeIRMetrics cumulativeIRMetrics, CumulativeIRMetrics QALDcumulativeIRMetrics, int quesIdx, long questionStartTime, int sparqlTemplateId, String goldenSparql, EDG edg, IRMetrics localIRMetrics, IRMetrics localQALDIRMetrics)", when show the information of the time for solving the current question, it's better to use phase "time for solving the current question" instead of "current time". Otherwise, some people may regard the "current time" as "time from the start of the system to the present" or "current time(time and date)".
I feel confused about the correspondence between macro F1 and getMicroF1() in both function toString and toQALDString in file CumulativeIRMetrics.java. Maybe there is a small mistake?
public String toString() { return "sample: " + numSample + ", P: " + decimalFormat.format(getPrecision()) + ", R: " + decimalFormat.format(getRecall()) + ", macro F1: " + decimalFormat.format(getMicroF1()) + ", macro F1*: " + decimalFormat.format(getMacroF1()); }
public String toQALDString() { return "sample: " + numSample + ", P: " + decimalFormat.format(getPrecision()) + ", R: " + decimalFormat.format(getRecall()) + ", macro F1: " + decimalFormat.format(getMicroF1()) + ", QALD macro F1: " + decimalFormat.format(getMacroF1()); } }
When I run the EDGQA system in the IDEA, I can reproduce the results in the paper, and even get a little better results, both LC-QuAD and QALD-9. However, when I try to run the EDGQA system in the eclipse, I can't get such good results. I have run the test for QALD-9 for two times in eclipse, and the results are as follows:
[INFO] Cumulative metrics, sample: 150, P: 0.277, R: 0.367, macro F1: 0.286, macro F1*: 0.316
[INFO] QALD Cumulative metrics, sample: 150, P: 0.557, R: 0.367, macro F1: 0.286, QALD macro F1: 0.442
and
[INFO] Cumulative metrics, sample: 150, P: 0.298, R: 0.387, macro F1: 0.306, macro F1*: 0.337
[INFO] QALD Cumulative metrics, sample: 150, P: 0.591, R: 0.387, macro F1: 0.306, QALD macro F1: 0.468
And the test results for QALD-9 in IDEA are as follows:
[INFO] Cumulative metrics, sample: 150, P: 0.319, R: 0.409, macro F1: 0.326, macro F1*: 0.359
[INFO] QALD Cumulative metrics, sample: 150, P: 0.546, R: 0.409, macro F1: 0.326, QALD macro F1: 0.468
I've applied the same settings in both eclipse and IDEA. And As far as I'm concerned, the results for QALD-9 is ought to be independent of the IDE I use. I wonder why the above difference between eclipse and IDEA exists.
On the other hand, I've been running experiments of LC-quad datasets in eclipse to further explore the doubts above.
In the file "main/src/main/java/cn/edu/nju/ws/edgqa/utils/LogUtil.java", in the function "public static void print(String writer, String str, LogType logType)", it's better to use "withStdOutMap.containsKey(writer) && withStdOutMap.get(writer)" instead of only "withStdOutMap.containsKey(writer)" to decide whether to execute the code "System.out.print(str)". Otherwise, the variable "withStdOut" doesn't play its due role and seems redundant.
When running the EDGQA system, are the following warnings imperative?
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
I have noticed that when EDGQA deals with the QALD dataset, before using Query Factory to create the goldenQuery, there is a step which use "Preprocessor.rewriteQALDSparql" to rewrite the SPARQL for QALD. I wonder why this step is in need.
goldenSparql = Preprocessor.rewriteQALDSparql(goldenSparql); // rewrite sparql for qald
Query goldenQuery = QueryFactory.create(goldenSparql, Syntax.syntaxARQ);
goldenAnswer.addAll(KBUtil.getQueryStringResult(goldenQuery)); // the golden answers
Can you please let me know which files from the DBpedia dump did you load in virtuoso from DBpedia 1604 and DBpedia 1610 in order to achieve the benchmark results?
I tried to run the System on Qald 9 dataset as specified in the Read.me file, and it worked correctly. Then I tried to run it on Lc-Quad with the following parameters "-d lc-quad -uc true" but for all questions I am getting empty Golden Answer and Predicted Answer.
This can be seen in the following example for question 5:
[DEBUG] Golden sparql: SELECT DISTINCT ?uri WHERE { <http://dbpedia.org/resource/Joe_Pass http://dbpedia.org/ontology/associatedBand ?uri. http://dbpedia.org/resource/Dream_Dancing_(album) http://dbpedia.org/property/artist ?uri . }
[INFO] SparqlGenerator list: []
[INFO] Golden Answer: []
[INFO] Predicted Answer: []
But in the log that you provide here it show non-empty Golden answer and Predicted answer lists as shown:
[DEBUG] Golden sparql: SELECT DISTINCT ?uri WHERE { http://dbpedia.org/resource/Joe_Pass http://dbpedia.org/ontology/associatedBand ?uri. http://dbpedia.org/resource/Dream_Dancing_(album) http://dbpedia.org/property/artist ?uri . }
[INFO] SparqlGenerator list: [SELECT DISTINCT ?e0 WHERE { {?e0 http://dbpedia.org/ontology/associatedBand http://dbpedia.org/resource/Joe_Pass} UNION {http://dbpedia.org/resource/Joe_Pass http://dbpedia.org/ontology/associatedBand ?e0} . {?e0 http://dbpedia.org/property/artist http://dbpedia.org/resource/Dream_Dancing_(album)} UNION {http://dbpedia.org/resource/Dream_Dancing_(album) http://dbpedia.org/property/artist ?e0} . }, SELECT DISTINCT ?e0 WHERE { {?e0 http://dbpedia.org/ontology/associatedBand http://dbpedia.org/resource/Joe_Pass} UNION {http://dbpedia.org/resource/Joe_Pass http://dbpedia.org/ontology/associatedBand ?e0} . {?e0 http://dbpedia.org/ontology/artist http://dbpedia.org/resource/Dream_Dancing_(album)} UNION {http://dbpedia.org/resource/Dream_Dancing_(album) http://dbpedia.org/ontology/artist ?e0} . }]
[INFO] Golden Answer: [http://dbpedia.org/resource/Ella_Fitzgerald]
[INFO] Predicted Answer: [http://dbpedia.org/resource/Ella_Fitzgerald]
I ran the Golden SPARQL query against the Virtuoso endpoint to make sure that it is not the cause of the error, and it worked and returned the correct result, but during the running of the system, I get the explained output.
Would you please help me in solving this error?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.