Comments (4)
"DeepAll is the baseline number that should have been the same across different methods had the dataset, implementation are standardized, is that correct?"
Yes, it is correct.
About your question: there could be several differences among the way in which each method is implemented that lead to a different DeepAll, also if all the methods use the same backbone as starting point. It could be used a different learning rate, batch size, a different data augmentation. So, in an ideal world each method that use the same backbone should have the same DeepAll, but since it is not possible (also because not all the algorithms provide the code, so it is not always possible to see in details the implementation choices) we think that it is more fair to report the DeepAll for each method.
Furthermore, from Table 1 and 2 of our work you can see that our DeepAll, in almost all cases, is higher than the others: we tried to compare our method with the more powerful version of DeepAll in order to see the actual gain. These methods in literature (with which we compare) show you what happens in settings where the DeepAll is quite low, but you don't know if those methods will actually work once you have raised DeepAll.
I hope that I have answered to your questions!
from jigendg.
Thanks much for the quick response.
I highly appreciate that you report DeepAll for all the methods. This brought much needed clarity since many other DG papers which use these datasets directly compare without any indication that some/much of the improvement is obtained from better implementation.
Just a quick follow up question:
These methods in literature (with which we compare) show you what happens in settings where the DeepAll is quite low, but you don't know if those methods will actually work once you have raised DeepAll.
Although, I see your point I feel we cannot be sure of it. MLDG and DeepC (referring to table 1 of your paper) improve over DeepAll by 65.27->69.26 (+3.99) and 67.24->70.01 (+2.77) respectively compared to 71.52->73.38 (+1.86) of JiGen. The improvements of MLDG and DeepC may suffer when using better implementation of JiGen but it is hard to answer if it would be better or worse than JiGen. What are your comments?
I find this problem quite unsettling, so in our paper we refused to compare beyond the method whose implementation we used: JiGen -- https://arxiv.org/abs/2003.12815.
from jigendg.
"The improvements of MLDG and DeepC may suffer when using better implementation of JiGen but it is hard to answer if it would be better or worse than JiGen."
Yes, I get your point but it is infeasible to mount all the methods over our baseline implementation.
So, I think it's enough to report the DeepAll for all the methods for a fair comparison.
from jigendg.
Thanks, that answers my questions.
from jigendg.
Related Issues (20)
- About requisite of this repository HOT 1
- About the performance on VLCS dataset HOT 4
- Hi, Could you give some details about the generation of permutations based on Hamming distance? HOT 1
- Paper Question HOT 1
- Questionable implementation of parsing boolean args HOT 1
- The performance in paper about DeepAll HOT 20
- PyTorch version of this code? HOT 1
- Python Version HOT 1
- TypeError: alexnet() got an unexpected keyword argument 'jigsaw_classes' HOT 1
- Error with argparser HOT 3
- what is the meanin of patch_based? HOT 1
- Doubt about experimental results HOT 1
- Office home dataset HOT 4
- Why is the input data composed with 9 grid? HOT 2
- python version HOT 2
- evaluate on VLCS dataset HOT 5
- Train on different datasets HOT 2
- The question about the classifier
- understanding bias_whole_image
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from jigendg.