rescience / mlrc

Editorial venue for ML Reproducibility Challenge Accepted papers

The ReScience Journal

ReScience is a platinum open-access peer-reviewed journal that targets computational research and encourages the explicit replication of already published research promoting new and open-source implementations in order to ensure the original research is reproducible. To achieve such a goal, the whole editing chain is radically different from any other traditional scientific journal. ReScience lives on github where each new implementation is made available together with the comments, explanations and tests. Each submission takes the form of a pull request that is publicly reviewed and tested in order to guarantee any researcher can re-use it. If you ever replicated a computational result from the literature, ReScience is the perfect place to publish this new implementation. Reproducible Science is Good. Replicated Science is better.

The Editorial Board

Note: This repository contains the first volumes of ReScience. Articles were submitted as pull requests to ReScience-submissions, which were merged upon acceptance with a reference added to this repository. The new ReScience C workflow is a bit different and based on two new repositories: one for submissions and another one containing the accepted articles. The entry point for ReScience remains its Web site.

Archives

• Volume 4 - Issue 1
• Volume 3 - Issue 1
• Volume 2 - Issue 1
• Volume 1 - Issue 1

mlrc's Issues

Questions about reformatting

Hi , I just finished a first round of reformatting.
Some questions:

• As the margins in the RC-2020 are much wider, all of the tables in the document had to be reformatted, some of them split into two parts. This takes (much) more space --> I hope this will not be a problem.
• We used bold, italic and some colors in our text (e.g. to highlight notable differences in tables). But in the RC-2020 format the codes don't work (e.g. \textit, \textbf) --> how can I include these.
• If you have suggestions about the other warnings in the build.log, please let me know!
  Thank you, Frank

build.log

Makefile Error

Latexmk: Non-existent bbl file 'article.bbl'
No file article.bbl.
Latexmk: References changed.
Latexmk: Log file says output to 'article.pdf'
Latexmk: List of undefined refs and citations:
Citation 'r1' on page 1 undefined on input line 4
Citation 'r10' on page 5 undefined on input line 107
Citation 'r10' on page 7 undefined on input line 348
Citation 'r11' on page 4 undefined on input line 104
Citation 'r11' on page 5 undefined on input line 107
Citation 'r12' on page 5 undefined on input line 107
Citation 'r3' on page 9 undefined on input line 453
Citation 'r6' on page 3 undefined on input line 73
Citation 'r7' on page 4 undefined on input line 85
Citation 'r8' on page 4 undefined on input line 104
Citation 'r8' on page 5 undefined on input line 109
Citation 'r8' on page 6 undefined on input line 300
Citation 'r8' on page 6 undefined on input line 311
Citation 'r9' on page 5 undefined on input line 107
Citation 'r9' on page 7 undefined on input line 348
Reference alg1' on page 8 undefined on input line 440 Reference fig3' on page 7 undefined on input line 348
Reference fig4' on page 7 undefined on input line 348 Reference fig5' on page 7 undefined on input line 391
Reference table1' on page 5 undefined on input line 121 Reference table1' on page 5 undefined on input line 189
Reference table1' on page 7 undefined on input line 391 Reference table5' on page 7 undefined on input line 391
Latexmk: 'pdflatex': source file 'article.bbl' doesn't exist. I'll try making it...

Running 'make "article.bbl"'

make[1]: Entering directory '/home/stark/Projects/MLRC2020/openreview/last_attempt/RC-2020/garg2021re/journal'
make[1]: *** No rule to make target 'article.bbl'. Stop.
make[1]: Leaving directory '/home/stark/Projects/MLRC2020/openreview/last_attempt/RC-2020/garg2021re/journal'
Latexmk: Summary of warnings:
Latex failed to resolve 8 reference(s)
Latex failed to resolve 16 citation(s)
Latexmk: Errors, so I did not complete making targets
Collected error summary (may duplicate other messages):
pdflatex: Command for 'pdflatex' gave return code 256
Latexmk: Use the -f option to force complete processing,
unless error was exceeding maximum runs of latex/pdflatex.
Makefile:30: recipe for target 'article.pdf' failed
make: *** [article.pdf] Error 12

Citation and table, figure references are not getting recognized. Header misaligned.

PR: #103

lenovo@lenovo-Lenovo-ideapad-330-15IKB:~/MLRC/2021$ ./build_all.sh
Building ahmed2022re/
/home/lenovo/MLRC/2021/ahmed2022re/journal
Latexmk: This is Latexmk, John Collins, 26 Dec. 2019, version: 4.67.
Latexmk: This is Latexmk, John Collins, 26 Dec. 2019, version: 4.67.
Rule 'pdflatex': The following rules & subrules became out-of-date:
      'pdflatex'
------------
Run number 1 of rule 'pdflatex'
------------
------------
Running 'xelatex -interaction=nonstopmode  -recorder  "article.tex"'
------------
Latexmk: References changed.
Latexmk: Non-existent bbl file 'article.bbl'
 No file article.bbl.
Latexmk: Missing input file: 'Capture.png' from line
  'LaTeX Warning: File `Capture.png' not found on input line 99.'
Latexmk: Missing input file: 'Capture.png' from line
  '! Unable to load picture or PDF file 'Capture.png'.'
Latexmk: Missing input file: 'graph_1.png' from line
  'LaTeX Warning: File `graph_1.png' not found on input line 311.'
Latexmk: Missing input file: 'graph_1.png' from line
  '! Unable to load picture or PDF file 'graph_1.png'.'
Latexmk: References changed.
Latexmk: References changed.
Latexmk: Log file says output to 'article.pdf'
Latexmk: List of undefined refs and citations:
  Citation 'hartley2020dtoolai' on page 3 undefined on input line 34
  Citation 'he2016deep' on page 3 undefined on input line 44
  Citation 'he2016deep' on page 5 undefined on input line 67
  Citation 'he2016deep' on page 5 undefined on input line 69
  Citation 'heumos2021mlf' on page 3 undefined on input line 34
  Citation 'kornblith2019similarity' on page 7 undefined on input line 129
  Citation 'krizhevsky2009learning' on page 3 undefined on input line 44
  Citation 'loshchilov2016sgdr' on page 6 undefined on input line 122
  Citation 'paszke2019pytorch' on page 5 undefined on input line 67
  Citation 'renard2020variability' on page 3 undefined on input line 34
  Citation 'summers2021nondeterminism' on page 1 undefined on input line 3
  Citation 'summers2021nondeterminism' on page 1 undefined on input line 8
  Citation 'summers2021nondeterminism' on page 11 undefined on input line 342
  Citation 'summers2021nondeterminism' on page 9 undefined on input line 170
  Citation 'summers2021nondeterminism' on page 9 undefined on input line 177
  Citation 'szegedy2015going' on page 1 undefined on input line 3
  Citation 'wen2020batchensemble' on page 1 undefined on input line 3
  Citation 'zhuang2021randomness' on page 3 undefined on input line 34
  Citation 'zhuang2021randomness' on page 3 undefined on input line 47
  Reference `fig:pic0' on page 5 undefined on input line 69
  Reference `fig:pic1' on page 11 undefined on input line 276
  Reference `sec:AdditionalResults' on page 5 undefined on input line 58
  Reference `sec:Results' on page 12 undefined on input line 363
  Reference `table:table1' on page 5 undefined on input line 69
  Reference `table:table10' on page 11 undefined on input line 342
  Reference `table:table2' on page 5 undefined on input line 106
  Reference `table:table3' on page 7 undefined on input line 143
  Reference `table:table4' on page 9 undefined on input line 177
  Reference `table:table5' on page 9 undefined on input line 200
  Reference `table:table6' on page 10 undefined on input line 217
  Reference `table:table7' on page 10 undefined on input line 217
  Reference `table:table7' on page 10 undefined on input line 219
  Reference `table:table8' on page 11 undefined on input line 276
  Reference `table:table9' on page 11 undefined on input line 316
------------
Running 'make "Capture.png"'
------------
make[1]: *** No rule to make target 'Capture.png'.  Stop.
------------
Running 'make "article.bbl"'
------------
make[1]: *** No rule to make target 'article.bbl'.  Stop.
------------
Running 'make "graph_1.png"'
------------
make[1]: *** No rule to make target 'graph_1.png'.  Stop.
Latexmk: Summary of warnings from last run of (pdf)latex:
  Latex failed to resolve 21 reference(s)
  Latex failed to resolve 23 citation(s)
Collected error summary (may duplicate other messages):
  pdflatex: Command for 'pdflatex' gave return code 1
      Refer to 'article.log' for details
Latexmk: Use the -f option to force complete processing,
 unless error was exceeding maximum runs, or warnings treated as errors.
make: *** [Makefile:30: article.pdf] Error 12
Build complete
rm: cannot remove 'article.bbl': No such file or directory
rm: cannot remove 'article.blg': No such file or directory
Building editorial/
/home/lenovo/MLRC/2021/editorial/journal
Latexmk: This is Latexmk, John Collins, 26 Dec. 2019, version: 4.67.
Latexmk: This is Latexmk, John Collins, 26 Dec. 2019, version: 4.67.
Rule 'pdflatex': The following rules & subrules became out-of-date:
      'pdflatex'
------------
Run number 1 of rule 'pdflatex'
------------
------------
Running 'xelatex -interaction=nonstopmode  -recorder  "article.tex"'
------------
Latexmk: References changed.
Latexmk: Non-existent bbl file 'article.bbl'
 No file article.bbl.
Latexmk: References changed.
Latexmk: References changed.
Latexmk: Log file says output to 'article.pdf'
Latexmk: List of undefined refs and citations:
  Citation 'Roboto:2011' on page 2 undefined on input line 74
  Citation 'Rougier:2017' on page 1 undefined on input line 12
  Citation 'Rougier:2017' on page 1 undefined on input line 26
  Citation 'Rougier:2018' on page 1 undefined on input line 13
  Citation 'Science:2018' on page 1 undefined on input line 11
  Citation 'SourceCodePro:2012' on page 2 undefined on input line 74
  Citation 'SourceSerifPro:2014' on page 2 undefined on input line 74
  Citation 'Topalidou:2015' on page 1 undefined on input line 4
------------
Running 'make "article.bbl"'
------------
make[1]: *** No rule to make target 'article.bbl'.  Stop.
Rule 'biber article': File changes, etc:
   Changed files, or newly in use since previous run(s):
      'article.bcf'
------------
Run number 1 of rule 'biber article'
------------
------------
Running 'biber  "article"'
------------
Latexmk: Found biber source file(s) [article.bcf bibliography.bib]
Rule 'pdflatex': File changes, etc:
   Changed files, or newly in use since previous run(s):
      'article.aux'
      'article.bbl'
      'article.out'
------------
Run number 2 of rule 'pdflatex'
------------
------------
Running 'xelatex -interaction=nonstopmode  -recorder  "article.tex"'
------------
Latexmk: Found input bbl file 'article.bbl'
Latexmk: Log file says output to 'article.pdf'
Latexmk: Found biber source file(s) [article.bcf bibliography.bib]
Rule 'pdflatex': File changes, etc:
   Changed files, or newly in use since previous run(s):
      'article.aux'
      'article.run.xml'
------------
Run number 3 of rule 'pdflatex'
------------
------------
Running 'xelatex -interaction=nonstopmode  -recorder  "article.tex"'
------------
Latexmk: Found input bbl file 'article.bbl'
Latexmk: Log file says output to 'article.pdf'
Latexmk: Found biber source file(s) [article.bcf bibliography.bib]
Rule 'pdflatex': File changes, etc:
   Changed files, or newly in use since previous run(s):
      'article.aux'
      'article.run.xml'
------------
Run number 4 of rule 'pdflatex'
------------
------------
Running 'xelatex -interaction=nonstopmode  -recorder  "article.tex"'
------------
Latexmk: Found input bbl file 'article.bbl'
Latexmk: Log file says output to 'article.pdf'
Latexmk: Found biber source file(s) [article.bcf bibliography.bib]
Build complete

Unable to load references

Hi, we're having trouble with loading our references. We tried many ways to do this, but are unable to find where the problem lies. Can you please help us?

Below you can find the .log file that is created on the latest run of 'make'.
article.log

All the files in our directory can be find under PR #24

Permission denied while cloning the repo

Following is the message shown after cloning

Please make sure you have the correct access rights and the repository exists.
fatal: clone of '[email protected]:ReScience/template.git' into submodule path 'D:/ MLRC Submission/Camera Ready/RC-2020/template' failed Failed to clone 'template'. Retry scheduled Cloning into 'D:/MLRC Submission/Camera Ready/RC-2020/template'...
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights and the repository exists.
fatal: clone of '[email protected]:ReScience/template.git' into submodule path 'D:/ MLRC Submission/Camera Ready/RC2020/template' failed
Failed to clone 'template' a second time, aborting

[togt2022badder] Some questions

Hi, we are going through the process of preparing our pull requests. Some questions:

1. There is a typo in our title metadata (link) (I think inherited from openreview, this is our fault, sorry): there is a missing space between "Methods" and "for", resulting in "Methodsfor". I can fix this as part of my pull request of course. Would that be ok?
2. On the topic of the title, i see from the checklist that the title should start with "[RE]". We weren't aware of this requirement in the openreview process and as a result our title is different. Should we change it to include "[RE]"? I.e. is this a hard constraint? If so, I suppose we should also definitely change it in the accepted.bib file? Or will that mess with the key?

Thanks

Help needed to understand the whole process.

Since i am lacking in basics, and this is my first ever paper. i would like someone to guide me through the process as written guidelines bit confusing to me. i am really looking forward to get your help in order to timely submission of camera ready version.

regards
Waqas Ahmed

not able to clone

following file has some bug in it or what? i cant clone it untill i delet this file.

Waqas Ahmed@DESKTOP-4APMTJQ MINGW64 ~/Divya
$ git clone --recursive https://github.com/Divyasha02/MLRC.git
Cloning into 'MLRC'...
remote: Enumerating objects: 6689, done.
remote: Total 6689 (delta 0), reused 0 (delta 0), pack-reused 6689
Receiving objects: 100% (6689/6689), 151.20 MiB | 2.47 MiB/s, done.
Resolving deltas: 100% (2692/2692), done.
error: invalid path '2020/sundar2021reproducibility/journal/Icon?'
fatal: unable to checkout working tree
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'

brew install

Hi,

This is a small detail but brew cask doesn't work anymore. The command to install mactex is
brew install --cask mactex

\citeauthor{}

Hi,

In our original code we use \citeauthor{} to cite the authors of the corresponding paper instead of referencing just the paper. This does compile correctly in the openreview folder, but not in the journal folder.

Openreview version:

Journal version:

Can someone help with this issue?

Error on yaml_to_latex conversion

Hi, while running the make command, I am getting the following error. What could possibly be the reason?
/yaml-to-latex.py -i metadata.yaml -o metadata.tex Traceback (most recent call last): File "./yaml-to-latex.py", line 93, in <module> content = generate_latex_metadata(filename_in, article) File "./yaml-to-latex.py", line 50, in generate_latex_metadata "".format(filename=filename, _=article, abstract=abstract)) AttributeError: 'str' object has no attribute 'name' Makefile:18: recipe for target 'metadata.tex' failed make: *** [metadata.tex] Error 1

Make does not compile bibliography, references and header correctly

Referring to our code in PR #9.

When trying to run the command make from the journal folder, we are having some trouble.
Here a list of the issues I can see from the make-build.log and article.log:

Bibliography compilation fails due to missing article.bbl

(from article.log)

I was assuming the Makefile automatically also runs bibtex, but according to the logs it does not. Do we manually have to run the Latex compile / bibtex pipeline before we run make?

ackage biblatex Info: XeTeX detected.
(biblatex)             Assuming input encoding 'utf8'.
Package biblatex Info: Automatic encoding selection.
(biblatex)             Assuming data encoding 'utf8'.
Package biblatex Info: Document encoding is UTF8 ....
Package biblatex Info: ... and expl3
(biblatex)             2021-02-18 L3 programming layer (loader) 
(biblatex)             is new enough (at least 2020/04/06),
(biblatex)             setting 'casechanger=expl3'.

(/usr/local/texlive/2021/texmf-dist/tex/latex/biblatex/blx-case-expl3.sty
Package: blx-case-expl3 2020/12/31 v3.16 expl3 case changing code for biblatex
)
\openout6 = `article.bcf'.

Package biblatex Info: Trying to load bibliographic data...
Package biblatex Info: ... file 'article.bbl' not found.

No file article.bbl.
Package biblatex Info: Reference section=0 on input line 5.
Package biblatex Info: Reference segment=0 on input line 5.
(./header.tex

References are undefined

I am unaware why this might be happening.

LaTeX Warning: Reference `tab:classification results' on page 7 undefined on in
put line 328.

[7] [8]

LaTeX Warning: Reference `tab:Authors seq-to-seq synth results' on page 9 undef
ined on input line 331.

... 

LaTeX Warning: There were undefined references.

Header not formatted correctly

I also added the currently compiled article.pdf to the repo, so that you can also have a look at the visual issues. Thanks for your help.

Undefined refs and citations, article.bbl not found

I submitted a pull request #90 that has build issue - undefined refs and citations and article.bbl file is not found.

Log when errors begin:

Running 'xelatex -interaction=nonstopmode  -recorder  "article.tex"'
------------
Latexmk: Found input bbl file 'article.bbl'
Latexmk: Log file says output to 'article.pdf'
Latexmk: Found biber source file(s) [article.bcf bibliography.bib]
Build complete
Building zrimsek2022learning/
/home/andraz/MLRC/2021/zrimsek2022learning/journal
Latexmk: This is Latexmk, John Collins, 26 Dec. 2019, version: 4.67.
Latexmk: This is Latexmk, John Collins, 26 Dec. 2019, version: 4.67.
Rule 'pdflatex': The following rules & subrules became out-of-date:
      'pdflatex'
------------
Run number 1 of rule 'pdflatex'
------------
------------
Running 'xelatex -interaction=nonstopmode  -recorder  "article.tex"'
------------
Latexmk: References changed.
Latexmk: Non-existent bbl file 'article.bbl'
 No file article.bbl.
Latexmk: References changed.
Latexmk: References changed.
Latexmk: Log file says output to 'article.pdf'
Latexmk: List of undefined refs and citations:
  Citation 'chen2019pipr' on page 1 undefined on input line 9
  Citation 'chen2019pipr' on page 3 undefined on input line 50
  Citation 'gui2019dnn' on page 3 undefined on input line 50
  Citation 'hashemifar2018predicting' on page 3 undefined on input line 50
  Citation 'lv2021learning' on page 1 undefined on input line 3
  Reference `generalize' on page 7 undefined on input line 184
  Reference `graph' on page 7 undefined on input line 205
  Reference `graph' on page 9 undefined on input line 240
  Reference `subsets' on page 5 undefined on input line 104
  Reference `subsets' on page 6 undefined on input line 140
  Reference `subsets' on page 8 undefined on input line 231
  Reference `times' on page 6 undefined on input line 115
------------
Running 'make "article.bbl"'
------------
make[1]: *** No rule to make target 'article.bbl'.  Stop.
Latexmk: Summary of warnings from last run of (pdf)latex:
  Latex failed to resolve 7 reference(s)
  Latex failed to resolve 5 citation(s)
Collected error summary (may duplicate other messages):
  pdflatex: Command for 'pdflatex' gave return code 1
      Refer to 'article.log' for details
Latexmk: Use the -f option to force complete processing,
 unless error was exceeding maximum runs, or warnings treated as errors.
make: *** [Makefile:30: article.pdf] Error 12
Build complete
rm: cannot remove 'article.bbl': No such file or directory
rm: cannot remove 'article.blg': No such file or directory

[buvanesh2022re] Unresolved citation and improper format for header

Referring to our code in PR #47

We are facing the following issues:

Improper format for header (header overlaps with main content)

Citation references undefined

We see errors the following errors during build:

LaTeX Warning: Citation 'zhuang_adabelief_2020' on page 3 undefined on input li
ne 62.
 
 
LaTeX Warning: Citation 'SGD' on page 3 undefined on input line 73.
 
 
LaTeX Warning: Citation 'Momentum' on page 3 undefined on input line 73.
 
 
LaTeX Warning: Citation 'NAG' on page 3 undefined on input line 73.
 
 
LaTeX Warning: Citation 'Adam' on page 3 undefined on input line 73.
 
 
LaTeX Warning: Citation 'liu_variance_2020' on page 3 undefined on input line 7
3.

References to tables, images, sections are not resolved

Whatever is referred using \ref command appears as ?? in the pdf

make[1]: *** No rule to make target `article.bbl'

STDOUT Logs of build-all.sh

build.log file

Thanks for your help.

Bibliography not found when compiling

Hello,
We are attempting to compile the camera ready version of our paper but we have encountered a few errors (see https://pastebin.com/7anJ8MKR). Our code can be found in #18.

The citations are missing in the compiled pdf and tex does not seem to find the .bib file.
We have changed the path in .yaml and tried both ../openreview/bibliography.bib and bibliography.bib
but neither seem to work

Thank you

check_title function

Hi,
Our paper got accepted but we failed to reproduce the results.
As suggested in the metadata.yaml we now have the following title:
"[¬Re] Reproducing 'Fair Selective Classification via Sufficiency'"
However, the check_yaml.py won't accept this as a correct title.
The check_title function only passes for "[Re] " but does not consider our case.
Could I suggest changing the code to:
def check_title(meta): check_ne(meta, "title", "Metadata") assert meta["title"].startswith( "[Re] " ) or meta["title"].startswith( "[¬Re] " ), "title in metadata should start with '[Re]' or '[¬Re]'"

Or is [Re] correct in our situation?
Thanks

build_all.sh is not working (not able to generate article.pdf)

I have completed all my files editing but i am struggling with this issue. what might be the possible issue? metadata.tex is not foung or generated.

$ ./build_all.sh
Building ahmed2022re/
/c/Users/Waqas Ahmed/MLRC_2/MLRC/2021/ahmed2022re/journal
Reverting Windows console CPs to (in,out) = (65001,65001)
Win CP console initial and current in/out Win: (65001, 65001), (1252, 1252)
Coding system for system and terminal: 'CP1252'

Rc files read:
NONE
Latexmk: This is Latexmk, John Collins, 17 Mar. 2022. Version 4.77, version: 4.77.
No specific requests made, so using default for latexmk.
Latexmk: Doing main (small) clean up for 'article.tex'
Latexmk: Doing extra clean up (of final output files) for 'article.tex'
Latexmk: Doing main (small) clean up for 'content.tex'
Latexmk: Doing extra clean up (of final output files) for 'content.tex'
Latexmk: Doing main (small) clean up for 'header.tex'
Latexmk: Doing extra clean up (of final output files) for 'header.tex'
rm: cannot remove 'metadata.tex': No such file or directory
C:\Users\Waqas Ahmed\AppData\Local\Microsoft\WindowsApps\python3.exe: can't open file 'C:\Users\Waqas': [Errno 2] No such file or directory
make: *** [Makefile:18: metadata.tex] Error 2
Build complete
rm: cannot remove 'article.aux': No such file or directory
rm: cannot remove 'article.bbl': No such file or directory
rm: cannot remove 'article.blg': No such file or directory
rm: cannot remove 'article.fdb_latexmk': No such file or directory
rm: cannot remove 'article.fls': No such file or directory
rm: cannot remove 'article.out': No such file or directory
rm: cannot remove 'article.log': No such file or directory
rm: cannot remove 'article.run.xml': No such file or directory
rm: cannot remove 'article.bcf': No such file or directory
Building editorial/
/c/Users/Waqas Ahmed/MLRC_2/MLRC/2021/editorial/journal
Makefile:1: *** missing separator. Stop.
rm: cannot remove 'metadata.tex': No such file or directory
Makefile:1: *** missing separator. Stop.
Build complete
rm: cannot remove 'article.aux': No such file or directory
rm: cannot remove 'article.bbl': No such file or directory
rm: cannot remove 'article.blg': No such file or directory
rm: cannot remove 'article.fdb_latexmk': No such file or directory
rm: cannot remove 'article.fls': No such file or directory
rm: cannot remove 'article.out': No such file or directory
rm: cannot remove 'article.log': No such file or directory
rm: cannot remove 'article.run.xml': No such file or directory
rm: cannot remove 'article.bcf': No such file or directory

Citations not getting recognized and header is misaligned

The header which contains "Edited by..." is overlapping with the reproducibility summary and the citations are getting boldened and not recognized.

build_all.sh fails

Hi, I am running build_all.sh after successfully passing the check_yaml.py test suite and organizing my content. I am getting the following output:

Building editorial/
/Users/thesofakillers/repos/MLRC/2021/editorial/journal
Rc files read:
  NONE
Latexmk: This is Latexmk, John Collins, 17 Mar. 2022. Version 4.77, version: 4.77.
No specific requests made, so using default for latexmk.
Latexmk: Doing main (small) clean up for 'article.tex'
Latexmk: Doing extra clean up (of final output files) for 'article.tex'
Latexmk: Doing main (small) clean up for 'content.tex'
Latexmk: Doing extra clean up (of final output files) for 'content.tex'
Latexmk: Doing main (small) clean up for 'header.tex'
Latexmk: Doing extra clean up (of final output files) for 'header.tex'
Latexmk: Doing main (small) clean up for 'metadata.tex'
Latexmk: Doing extra clean up (of final output files) for 'metadata.tex'
Latexmk: Missing bbl file 'article.bbl' in following:
 No file article.bbl.
make[1]: *** No rule to make target `article.bbl'.  Stop.
Build complete
Building togt2022badder/
/Users/thesofakillers/repos/MLRC/2021/togt2022badder/journal
Rc files read:
  NONE
Latexmk: This is Latexmk, John Collins, 17 Mar. 2022. Version 4.77, version: 4.77.
No specific requests made, so using default for latexmk.
Latexmk: Doing main (small) clean up for 'article.tex'
Latexmk: Doing extra clean up (of final output files) for 'article.tex'
Latexmk: Doing main (small) clean up for 'content.tex'
Latexmk: Doing extra clean up (of final output files) for 'content.tex'
Latexmk: Doing main (small) clean up for 'header.tex'
Latexmk: Doing extra clean up (of final output files) for 'header.tex'
rm: metadata.tex: No such file or directory
Traceback (most recent call last):
  File "/Users/thesofakillers/repos/MLRC/2021/togt2022badder/journal/./yaml-to-latex.py", line 93, in <module>
    content = generate_latex_metadata(filename_in, article)
  File "/Users/thesofakillers/repos/MLRC/2021/togt2022badder/journal/./yaml-to-latex.py", line 10, in generate_latex_metadata
    "% DO NOT EDIT - automatically generated from {filename}\n\n"
IndexError: list index out of range
make: *** [metadata.tex] Error 1
Build complete
rm: article.aux: No such file or directory
rm: article.bbl: No such file or directory
rm: article.blg: No such file or directory
rm: article.fdb_latexmk: No such file or directory
rm: article.fls: No such file or directory
rm: article.out: No such file or directory
rm: article.log: No such file or directory
rm: article.run.xml: No such file or directory
rm: article.bcf: No such file or directory

I am not sure what I should do about the missing article.bbl and the metadata.tex IndexError. Please let me know how I should debug this. Thank you

Make fails due to file not found (relative path issues)

Referring to our code in PR #5.

When trying to run the command make from the journal folder, we get errors due to pdflatex not being able to find the included files. For example, the line \input{intro.tex} in the file openreview/content.tex will not work:

Latexmk: This is Latexmk, John Collins, 1 January 2015, version: 4.41.
Rule 'pdflatex': Rules & subrules not known to be previously run:
   pdflatex
Rule 'pdflatex': The following rules & subrules became out-of-date:
      'pdflatex'
------------
Run number 1 of rule 'pdflatex'
------------
------------
Running 'xelatex -interaction=nonstopmode  -recorder  "article.tex"'
------------
Latexmk: References changed.
Latexmk: Non-existent bbl file 'article.bbl'
 No file article.bbl.
Latexmk: Missing input file: 'intro.tex' from line
  '! LaTeX Error: File `intro.tex' not found.'
Latexmk: Log file says output to 'article.pdf'
Latexmk: List of undefined refs and citations:
  Citation `beta-vae' on page 2 undefined on input line 34
  Citation `geco' on page 2 undefined on input line 33
  Citation `pytorch' on page 1 undefined on input line 10
------------
Running 'make "article.bbl"'
------------
make[1]: *** No rule to make target 'article.bbl'.  Stop.
------------
Running 'make "intro.tex"'
------------
make[1]: *** No rule to make target 'intro.tex'.  Stop.
Latexmk: Summary of warnings:
  Latex failed to resolve 3 citation(s)
Collected error summary (may duplicate other messages):
  pdflatex: Command for 'pdflatex' gave return code 256
Latexmk: Use the -f option to force complete processing,
 unless error was exceeding maximum runs of latex/pdflatex.
make: *** [article.pdf] Error 12

The compiler cannot find the intro.tex file.

However, if we modify the include statement as \input{../openreview/content.tex} it will work, and return an error to the next \input statement whose path is not relative to the journal folder. This happens also for included images, so to fix it we would need to add ../openreview to all the figures and inputs.

Here is the link to the full output of make.

I opened the issue because to me this looks related to some issue in the makefiles and it might be helpful to others.