Training config:
- ~ 75% of 30k / class ( like ~23k / class)
- size 128 x 128
- batch size 128 (can be a bit more larger)
- 500, steps every epoch, so every epoch train with 128 * 500 images
- Adam, lr start at 2e-4, use cyclic LR with cos annealing
def cos_annealing_lr(initial_lr, cur_epoch, epoch_per_cycle):
return initial_lr * (np.cos(np.pi * cur_epoch / epoch_per_cycle) + 1) / 2
- 6 cycles, each cycle 25 epochs. By checking the training log I think this schedule should be improved further)
Model used (with best score):
- MobileNet 0.913 (by hflip TTA)
- ResNet18 0.919
- Xception 0.925 (not with the current 3 channel information, I think the current one can give a better result)
- SE-ResNeXt50 (32 x 4d) 0.930
Ideas used:
- imagenet-pretrained weights
- larger images I tried Xception with 224 x 224 image, by using about 20k/class, and just ReduceOnPlaque can get 0.925, I think larger --> better
- grayscale strokes to ordered strokes
- country code information encoding
c_value = country_mapping.get(df.iloc[i]['countrycode'], 40)
x[i, :, :, 0] = img + mask * c_value
- highlight first and last stokes (may drop the last and retry)
if i == 0 or i == len(stroke[0]) - 2:
color = 255
_ = cv2.line(img, (stroke[0][i], stroke[1][i]),
(stroke[0][i + 1], stroke[1][i + 1]), color, lw)
- hflip TTA (good when LB < 0.920, bad when LB >= 0.920)
- last few layers in the model, with dropout rate 0.075 (prob not good)
def forward(self, x):
x = self.features(x)
x = self.relu(x)
x = self.avg_pool(x)
x = x.view(x.size(0), -1)
x = self.dropout(x)
x = self.last_linear(x)
Possible Models:
Size 1:
- Xception
- ResNet50
- SE ResNeXt50
Size 2:
- SE ResNeXt101
- DPN?
- InceptionResV2
TODO:
- better LR scheduler (this seems to be the most important one)
- update network after several batches to increase the valid batch size
- finetune with different optimizers?
- larger batches
- lstm
- bi-lstm
- gru?
- cnn as feature extractor and rnn accept result from global pooling?
- label denoising