berivanisik / sparse-random-networks

Implementation of the FedPM framework by the authors of the ICLR 2023 paper "Sparse Random Networks for Communication-Efficient Federated Learning".

Sparse-Random-Networks-for-Communication-Efficient-Federated-Learning

PyTorch implementation of the FedPM framework by the authors of the ICLR 2023 paper "Sparse Random Networks for Communication-Efficient Federated Learning".

Sparse Random Networks for Communication-Efficient Federated Learning
Berivan Isik, Francesco Pase, Deniz Gunduz, Tsachy Weissman, Michele Zorzi
International Conference on Learning Representations (ICLR), 2023.

Environment setup:

Packages can be found in fedpm.yml.

Training:

Set the params_path in main.py to the the path of the {}.yaml file with the desired model and dataset. The default parameters can be found in the provided {}.yaml files. To train the model, run:

python3 main.py

References

If you find this work useful in your research, please consider citing our paper:

@inproceedings{
isik2023sparse,
title={Sparse Random Networks for Communication-Efficient Federated Learning},
author={Berivan Isik and Francesco Pase and Deniz Gunduz and Tsachy Weissman and Zorzi Michele},
booktitle={The Eleventh International Conference on Learning Representations },
year={2023},
url={https://openreview.net/forum?id=k1FHgri5y3-}
}

sparse-random-networks's Issues

upload_mask

Why use "updates_s = np.where(updates_s == 0, self.epsilon, updates_s) " and "updates_s = np.where(updates_s == 1, 1-self.epsilon, updates_s)"?
It may be not binary mask upload from client to server ,which causes the same communication cost as standard FL

Experiments with ResNet-18 on CIFAR-10 and CIFAR-100

Could you release the code about experiments with ResNet-18 on CIFAR-10 and CIFAR-100?

Confusing

mark

Question Regarding the Use of Inverse Sigmoid Function in Mask Aggregation

Hi,

Firstly, I want to express my gratitude for sharing your code. It has been incredibly beneficial to my work, and I appreciate the opportunity to learn from your approach.

I have a question about a specific part of the implementation, particularly regarding the aggregation of masks. I noticed that you introduced an inverse sigmoid function (logit) during the mask aggregation process.

sparse-random-networks/src/fedssm/server.py

Line 107 in cf6b549

torch.log(avg_p / (1 - avg_p)),

Could you kindly explain the rationale behind this choice? I'm interested in understanding the advantages or the specific problems it addresses in the context of federated learning or the particular application you designed it for.

Understanding your reasoning would be immensely helpful for me to grasp the full scope and depth of your methodology.

Thank you for your time and for sharing your valuable work.

Best regards,

Ken