Hey there, this is the implementation of I1 (heatmap loss + integral loss) in this paper. Training is on H36M exclusively with 2-stage coarse to fine Hourglass. Kindly jump to Training procedure below if you are not using C++.
${POSE_ROOT}
+-- caffe_code
+-- data
+-- figs
+-- models
+-- training
+-- testing
+-- README.md
I1 with group normalization, batch size = 1
Please refer to the "caffe_code/" folder and "Installation" section in readme of this repo
This is for Caffe Ubuntu with 12 GB Titan card.
Same as "data/". Note this is different from commonly used data like effective baseline. Major difference is my training(testing) set has 1559571(548999) images.
d2 | MPJPE(mm) of this repo (I1) | Caffe Model | (For reference) MPJPE(mm) of corresponding H1 repo |
---|---|---|---|
16 | 71.0 | net_iter_526322.caffemodel | 73.6 |
32 | 55.9 | net_iter_567946.caffemodel | 68.6 |
64 | 65.9 | net_iter_671782.caffemodel | 67.1 |
Note that in order to restrain one from tuning loss weight of 2D/3D HM/integral to the best of one's ability, adaptive euclidean loss weight balancing technique (as detailed below) is initialized insofar as it does not degrade performance.
I have a deep skepticsm concerning the bearing of different training technique on final number. And so the point I would like to make here is that what matters most is the algorithm itself rather than the superiority or inferiority of number.
As to depth dimension d2, the original paper uses 64. This paper uses 16. This repo prefers 32.
Piece of advice here, insofar as d2 is at least 16, it's okay.
First off, THERE IS NO GUARANTEE THAT OTHER TRAINING STRATEGY WOULD NOT YIELD SAME/SIMILAR OR BETTER RESULTS. THIS IS JUST FOR REFERENCE.
Having said that, I WOULD BE MORE THAN HAPPY TO KNOW IF ANYONE HAS A MORE ELEGANT WAY TO TRAIN THIS WACKY STUFF. MUCH APPRECIATED. THANKS!
Back to our topic,
The starting point is MPII pretrained model. See head of training section .
Note 1 [heatmap2 init std]: The init of layer "heatmap2" (which reduces dimension to generate final 3d heatmap output) is gaussian with standard deviation of a hyper param.
Note 2 [heatmap2_flat_scale]: Before softmax normalization, hereinafter a scale 30.0 is multiplied by 3d heatmap output for numerical reason. The semantic of heatmap no longer preserves any more, as alluded in this issue. Otherwise I have no idea how to train I1. CAN ANYONE ENLIGHTEN ME?
Note 3 [loss]: Adaptive weight balancing is employed. Details here. Found this paper several weeks after writing the AdaptiveWeightEucLossLayer
-
Adaptive H1 adaptively computes gradient magnitude of 2D/3D heatmap w.r.t neuron, and tries to balance gradients flowing from 2D and 3D heatmap. Search AdaptiveWeightEucLoss in this pdf for expatiation.
-
Adaptive I1 adaptively balances the gradient of 2D/3D heatmap/integral loss w.r.t neuron, whereupon restraining one from tuning weight ratio between heatmap euclidean loss and integral regression loss.
-
Manual sets loss ratio of heatmap:integral manually after Adaptive I1 warm-up.
The following is sorted by d2 (depth dimension of 3d heatmap) in increasing order.
-
d2 = 4
cd training/d2=4 $CAFFE_ROOT/build/tools/caffe train --solver=solver_d4_ada.prototxt --weights=improved-hourglass_iter_640000.caffemodel
d2 lr "heatmap2" init std loss Caffe Model Solver State 4 2.5e-5 0.01 Adaptive H1 net_iter_160533.caffemodel net_iter_160533.solverstate -
d2 = 8
- Adaptive I1
cd ../../training/d2=8 $CAFFE_ROOT/build/tools/caffe train --solver=solver_d8_ada.prototxt --snapshot=net_iter_160533.solverstate
d2 lr "heatmap2" init std loss Caffe Model Solver State 8 2.5e-5 0.01 Adaptive I1 net_iter_265747.caffemodel net_iter_265747.solverstate - Manual
$CAFFE_ROOT/build/tools/caffe train --solver=solver_d8_ada_manual.prototxt --snapshot=net_iter_265747.solverstate
d2 lr loss ratio of 2D HM:3D HM:integral loss Caffe Model Solver State 8 5e-6 1:0.1:1 Manual net_iter_350433.caffemodel net_iter_350433.solverstate net_iter_350433.caffemodel train around 71 mm, test around 83 mm.
-
d2 = 16
- Adaptive I1
cd ../../training/d2=16 $CAFFE_ROOT/build/tools/caffe train --solver=solver_16_ada.prototxt --snapshot=net_iter_350433.solverstate
d2 lr "heatmap2" init std loss Caffe Model Solver State 16 2.5e-5 0.002 Adaptive I1 net_iter_434657.caffemodel net_iter_434657.solverstate net_iter_434657.caffemodel train around 50 mm, test around 73 mm.
- Manual
$CAFFE_ROOT/build/tools/caffe train --solver=solver_16_ada_manual.prototxt --snapshot=net_iter_434657.solverstate
d2 lr loss ratio of 2D HM:3D HM:integral loss Caffe Model Solver State 16 5e-6 1:0.3:1 Manual net_iter_526322.caffemodel net_iter_526322.solverstate net_iter_526322.caffemodel train around 47 mm, test around 71 mm (click this for screenshot).
-
d2 = 32
- Adaptive I1
cd ../../training/d2=32 $CAFFE_ROOT/build/tools/caffe train --solver=solver_32_ada.prototxt --snapshot=net_iter_526322.solverstate
d2 lr "heatmap2" init std loss Caffe Model Solver State 32 2.5e-5 0.002 Adaptive I1 net_iter_567946.caffemodel net_iter_567946.solverstate net_iter_567946.caffemodel train around 49 mm test around 56 mm (click this for screenshot).
-
d2 = 64
- Adaptive I1
cd ../../training/d2=64 $CAFFE_ROOT/build/tools/caffe train --solver=solver_64_ada.prototxt --snapshot=net_iter_567946.solverstate
d2 lr "heatmap2" init std loss Caffe Model Solver State 64 2.5e-5 0.002 Adaptive I1 net_iter_600890.caffemodel net_iter_600890.solverstate net_iter_600890.caffemodel train around 43 mm, test around 68 mm.
- Manual
$CAFFE_ROOT/build/tools/caffe train --solver=solver_64_ada_manual.prototxt --snapshot=net_iter_600890.solverstate
d2 lr loss ratio of 2D HM:3D HM:integral loss Caffe Model Solver State 64 5e-6 1:0.08:1 Manual net_iter_671782.caffemodel net_iter_671782.solverstate net_iter_671782.caffemodel train around 40 mm, test around 66 mm (click this for screenshot).
Put corresponding trained models under folder models. (Simply scroll up to MPJPE Performance for names of models)
d2 = 32
cd testing/d2=32
$CAFFE_ROOT/build/tools/caffe test -model test_d32_statsfalse.prototxt -weights models/net_iter_567946.caffemodel -iterations 274500
Denouement: 55.9 mm
d2 = 64
cd testing/d2=64
$CAFFE_ROOT/build/tools/caffe test -model test_d64_statsfalse.prototxt -weights models/net_iter_671782.caffemodel -iterations 274500
Denouement: 65.9 mm
d2 = 16
cd testing/d2=16
$CAFFE_ROOT/build/tools/caffe test -model test_d16_statsfalse.prototxt -weights models/net_iter_526322.caffemodel -iterations 274500
Denouement: 71.0 mm
Glad to answer any concern. Just email [email protected]. Thanks!