Comments (8)
Sure. test and valid unseen. Test data and valid unseen data have been carefully removed from these aug paths. For these experiments, I actually only allow the agent to explore the test environment but not give it testing instructions (as in RCM).
I have not tested the performance with new PyTorch version but the result should be reproducible by replacing --aug tasks/R2R/data/aug_paths.json
in bt_envdrop.bash
with appropriate json files (especially on unseen valid because you could see the local evaluation). I would update github after verification. Could you please leave this issue open until then?
from r2r-envdrop.
Thanks for the timely response, I will leave this issue open. By "not give it testing instructions", you mean the instructions will be generated by the trained speaker model with environmental dropout, right? Thanks.
from r2r-envdrop.
Yep. And it would never touch the paths/instructions in testing data and validation unseen data.
from r2r-envdrop.
May I know if there are some rules you use for generating augmented paths? Since I want to sample more paths to see the upper bound of the pre-exploration method. Currently, I randomly sample two viewpoints with their distance larger than 5 and a start heading angle. Then I add them to the original aug_path file, but the result gets worse. Do you have any suggestion? Thanks.
from r2r-envdrop.
Sorry for late replying (due to ACL).
The file is generated by exploiting all viewpoint pairs which have action length from 4~6. I also exclude the val/test data. Thus it is a complete set of all available short-distance paths. The initial heading angle are randomly sampled. I visualize the headings in training data and I believe that the initial headings in training are uniformly sampled.
If you want to verify the upper bound, I suggest to try:
- Sampling more initial headings.
- Generate more instructions for each paths. (currently one for each)
I am also notified (by Peter) that the speaker model trained with PyTorch 1.0 might be weaker than with PyTorch 0.4. Since the pre-exploring results highly depends on the performance of speaker, I doubt whether the results are still the same. Have you achieved any similar result by adding pre-exploring paths? If not, I would definitely take more time on fixing the speaker issue.
from r2r-envdrop.
Sure. test and valid unseen. Test data and valid unseen data have been carefully removed from these aug paths. For these experiments, I actually only allow the agent to explore the test environment but not give it testing instructions (as in RCM).
I have not tested the performance with new PyTorch version but the result should be reproducible by replacing
--aug tasks/R2R/data/aug_paths.json
inbt_envdrop.bash
with appropriate json files (especially on unseen valid because you could see the local evaluation). I would update github after verification. Could you please leave this issue open until then?
May I know how to allow the agent to explore the test environment but not give it testing instructions?
from r2r-envdrop.
It is completed via the trick of back-translation. The paths are first randomly sampled and the instructions are then generated from the speaker.
from r2r-envdrop.
@airsplay Does aug_paths.json
also contain original R2R train + val seen data?
from r2r-envdrop.
Related Issues (20)
- Retrained model does not get the same SPL on val unseen as reported in paper HOT 4
- Help, error about the code HOT 4
- Test Question HOT 2
- About the back-translation performance and trigger of pre-exploration HOT 3
- Can this repo reproduce the reported result ?
- One possiable bug in agent._teacher_action() HOT 3
- Question about the valid seen/unseen performance report on the paper HOT 2
- About make_action when feedback method is 'sample' or 'argmax' HOT 4
- Beam Search setting HOT 2
- Why the loc_elevation is not updated in the env? HOT 1
- How to run with multiple gpus? HOT 2
- Picking the best checkpoint in the pre-exploration setting. HOT 2
- Beam search validation HOT 4
- Vanilla Listener (Follower) Training without RL (Student Forcing + Teacher Forcing?) HOT 2
- Difference between aug_paths and Speaker-Follower (Literal) Paths? HOT 2
- environment set up problem HOT 1
- Missing files HOT 1
- Multiple GPUs HOT 1
- Import MatterSim error HOT 3
- Request code to generate R2R_<partition>.json
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from r2r-envdrop.