hxx97 / edgqa Goto Github PK

In the file "src/main/java/cn/edu/nju/ws/edgqa/main/EDGQA.java", in the function "private static void computeMetrics(long startTime, CumulativeIRMetrics cumulativeIRMetrics, CumulativeIRMetrics QALDcumulativeIRMetrics, int quesIdx, long questionStartTime, int sparqlTemplateId, String goldenSparql, EDG edg, IRMetrics localIRMetrics, IRMetrics localQALDIRMetrics)", when show the information of the time for solving the current question, it's better to use phase "time for solving the current question" instead of "current time". Otherwise, some people may regard the "current time" as "time from the start of the system to the present" or "current time(time and date)".

I feel confused about the correspondence between macro F1 and getMicroF1() in both function toString and toQALDString in file CumulativeIRMetrics.java. Maybe there is a small mistake?
public String toString() { return "sample: " + numSample + ", P: " + decimalFormat.format(getPrecision()) + ", R: " + decimalFormat.format(getRecall()) + ", macro F1: " + decimalFormat.format(getMicroF1()) + ", macro F1*: " + decimalFormat.format(getMacroF1()); }
public String toQALDString() { return "sample: " + numSample + ", P: " + decimalFormat.format(getPrecision()) + ", R: " + decimalFormat.format(getRecall()) + ", macro F1: " + decimalFormat.format(getMicroF1()) + ", QALD macro F1: " + decimalFormat.format(getMacroF1()); } }

When I run the EDGQA system in the IDEA, I can reproduce the results in the paper, and even get a little better results, both LC-QuAD and QALD-9. However, when I try to run the EDGQA system in the eclipse, I can't get such good results. I have run the test for QALD-9 for two times in eclipse, and the results are as follows:
[INFO] Cumulative metrics, sample: 150, P: 0.277, R: 0.367, macro F1: 0.286, macro F1*: 0.316
[INFO] QALD Cumulative metrics, sample: 150, P: 0.557, R: 0.367, macro F1: 0.286, QALD macro F1: 0.442
and
[INFO] Cumulative metrics, sample: 150, P: 0.298, R: 0.387, macro F1: 0.306, macro F1*: 0.337
[INFO] QALD Cumulative metrics, sample: 150, P: 0.591, R: 0.387, macro F1: 0.306, QALD macro F1: 0.468
And the test results for QALD-9 in IDEA are as follows:
[INFO] Cumulative metrics, sample: 150, P: 0.319, R: 0.409, macro F1: 0.326, macro F1*: 0.359
[INFO] QALD Cumulative metrics, sample: 150, P: 0.546, R: 0.409, macro F1: 0.326, QALD macro F1: 0.468
I've applied the same settings in both eclipse and IDEA. And As far as I'm concerned, the results for QALD-9 is ought to be independent of the IDE I use. I wonder why the above difference between eclipse and IDEA exists.
On the other hand, I've been running experiments of LC-quad datasets in eclipse to further explore the doubts above.

In the file "main/src/main/java/cn/edu/nju/ws/edgqa/utils/LogUtil.java", in the function "public static void print(String writer, String str, LogType logType)", it's better to use "withStdOutMap.containsKey(writer) && withStdOutMap.get(writer)" instead of only "withStdOutMap.containsKey(writer)" to decide whether to execute the code "System.out.print(str)". Otherwise, the variable "withStdOut" doesn't play its due role and seems redundant.

When running the EDGQA system, are the following warnings imperative?

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

I have noticed that when EDGQA deals with the QALD dataset, before using Query Factory to create the goldenQuery, there is a step which use "Preprocessor.rewriteQALDSparql" to rewrite the SPARQL for QALD. I wonder why this step is in need.
goldenSparql = Preprocessor.rewriteQALDSparql(goldenSparql); // rewrite sparql for qald
Query goldenQuery = QueryFactory.create(goldenSparql, Syntax.syntaxARQ);
goldenAnswer.addAll(KBUtil.getQueryStringResult(goldenQuery)); // the golden answers

Can you please let me know which files from the DBpedia dump did you load in virtuoso from DBpedia 1604 and DBpedia 1610 in order to achieve the benchmark results?

What version of Virtuoso is required to run EDGQA? I get an error with KBUtil initialising.

I tried to run the System on Qald 9 dataset as specified in the Read.me file, and it worked correctly. Then I tried to run it on Lc-Quad with the following parameters "-d lc-quad -uc true" but for all questions I am getting empty Golden Answer and Predicted Answer.

This can be seen in the following example for question 5:

[DEBUG] Golden sparql: SELECT DISTINCT ?uri WHERE { <http://dbpedia.org/resource/Joe_Pass http://dbpedia.org/ontology/associatedBand ?uri. http://dbpedia.org/resource/Dream_Dancing_(album) http://dbpedia.org/property/artist ?uri . }
[INFO] SparqlGenerator list: []
[INFO] Golden Answer: []
[INFO] Predicted Answer: []

But in the log that you provide here it show non-empty Golden answer and Predicted answer lists as shown:

[DEBUG] Golden sparql: SELECT DISTINCT ?uri WHERE { http://dbpedia.org/resource/Joe_Pass http://dbpedia.org/ontology/associatedBand ?uri. http://dbpedia.org/resource/Dream_Dancing_(album) http://dbpedia.org/property/artist ?uri . }
[INFO] SparqlGenerator list: [SELECT DISTINCT ?e0 WHERE { {?e0 http://dbpedia.org/ontology/associatedBand http://dbpedia.org/resource/Joe_Pass} UNION {http://dbpedia.org/resource/Joe_Pass http://dbpedia.org/ontology/associatedBand ?e0} . {?e0 http://dbpedia.org/property/artist http://dbpedia.org/resource/Dream_Dancing_(album)} UNION {http://dbpedia.org/resource/Dream_Dancing_(album) http://dbpedia.org/property/artist ?e0} . }, SELECT DISTINCT ?e0 WHERE { {?e0 http://dbpedia.org/ontology/associatedBand http://dbpedia.org/resource/Joe_Pass} UNION {http://dbpedia.org/resource/Joe_Pass http://dbpedia.org/ontology/associatedBand ?e0} . {?e0 http://dbpedia.org/ontology/artist http://dbpedia.org/resource/Dream_Dancing_(album)} UNION {http://dbpedia.org/resource/Dream_Dancing_(album) http://dbpedia.org/ontology/artist ?e0} . }]
[INFO] Golden Answer: [http://dbpedia.org/resource/Ella_Fitzgerald]
[INFO] Predicted Answer: [http://dbpedia.org/resource/Ella_Fitzgerald]

I ran the Golden SPARQL query against the Virtuoso endpoint to make sure that it is not the cause of the error, and it worked and returned the correct result, but during the running of the system, I get the explained output.

Would you please help me in solving this error?