Comments (9)
I remeber that .mean(1) is equal to reduction='batch_mean‘ ?
from ld.
I remeber that .mean(1) is equal to reduction='batch_mean‘ ?
Here is the source code of F.kl_div
:
https://github.com/pytorch/pytorch/blob/defa0d3a2d230e5d731d5c443c1b9beda2e7fd93/torch/nn/functional.py#L2949-L2958
And the problem here is that the kd_loss
is subsequently averaged by @weighted_loss
wrapper.
from ld.
So batch_mean equals .mean(0)?
from ld.
So batch_mean equals .mean(0)?
No. "batchmean" means .sum()/batch_size, i.e., .sum(1).mean()
from ld.
OK, I get your point, you mean mathmatically .sum(1) is the correct implementation and .mean(1)=.sum(1)/16
That's true, but how is it related to batchmean?
from ld.
OK, I get your point, you mean mathmatically .sum(1) is the correct implementation and .mean(1)=.sum(1)/16 That's true, but how is it related to batchmean?
BTW, I also found that loss_ld
used weighted sum and was not divided by avg_factor
(i.e. sum of weights). Is this a typo or intended behavior for not using normalization?
from ld.
FYI: I record the factor ratio avg_factor/(self.reg_max+1)
during the training. Maybe it will help this discussion.
![image](https://private-user-images.githubusercontent.com/20830726/275184910-ec7f45ed-d476-49bb-a055-c9a57cca224d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDk0NjMxMzksIm5iZiI6MTcwOTQ2MjgzOSwicGF0aCI6Ii8yMDgzMDcyNi8yNzUxODQ5MTAtZWM3ZjQ1ZWQtZDQ3Ni00OWJiLWEwNTUtYzlhNTdjY2EyMjRkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAzMDMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMzAzVDEwNDcxOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWQzMGYyYzlmMzA5YjBkYTJiODNkZjhkNDVlZDQyMGQ4NWJkOTQzNGY5MjRjNmY1MzQ5ZWYwZjJmZDM5ZGQ2NzAmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.2tdFaJcEogVv2COMIqXD0aRSUWk86BtYsbBm7hsLVDM)
from ld.
It's a intended behavior because experiment shows not dividing is better. Don't know the theory behind this though
from ld.
It's a intended behavior because experiment shows not dividing is better. Don't know the theory behind this though
I see, thanks for the reply.
from ld.
Related Issues (20)
- 请问如果使用自己的feature-imitation方法应该在哪里添加,您的feature-imitation代码写在哪里了,我没有找到 HOT 6
- I reimplement your model (Teacher: R101, Student: R50, Detection head: Foveabox) and got RuntimeError: shape '[-1, 68]' is invalid for input of size 115200 HOT 5
- Welcome update to OpenMMLab 2.0 HOT 2
- 请问下代码里的教师模型是跟着学生模型一起训练的吗 HOT 2
- If I set output_feature = False , Code don't work HOT 1
- 参考install.md 里面配置的环境无法找到cuda HOT 13
- 简易版本复现问题 HOT 5
- 将DKD与LD结合的问题 HOT 8
- Implement LD on VisDrone dataset HOT 5
- 学生网络与教师网络性能差距较大
- mmpycocotools HOT 1
- 请问下是否可以迁移到centernet呢 HOT 2
- 知识蒸馏 HOT 10
- 请问是使用 ldv2/ld_r50_gflv2_r101_fpn_1x.py 这个文件来复现 Temperature τ in LD 这个实验吗? HOT 1
- 关于数据集minicoco HOT 4
- About Adaptive Layer HOT 2
- Can we pass different set of image to teacher and student. HOT 6
- samples_per_gpu HOT 1
- loadcheckpoint出现问题 HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ld.