why treating Alexnet/VGG differently in ImageNet example? about examples

[soumith]

yes

[colesbury]

@cheerss see https://arxiv.org/abs/1404.5997 [pdf] (esp. sections 1-3):

... data parallelism is efficient when the amount of computation per weight is high (because
the weight is the unit being communicated)

It's not efficient to use DataParallel with the fully connected layers because they have lots of parameters, but relatively little computation. In theory, the fully connected layers are well suited for model parallelism, but it's kind of a pain to implement and the performance gain is small.

[soumith]

it is because they have very big fully connected layers at the end that are not suitable to DataParallel

[zym1010]

@soumith thanks. Are you suggesting that fc layers for these networks are all stored on the same card?

[cheerss]

Is there anyone know why "big fully connected layers at the end are not suitable to DataParallel"?

[cheerss]

@colesbury Thanks a lot !!