About the experiment setting about fedntd

[Lee-Gihun]

Here I'm posting my email response here to make it publicly accessible:

---

Indeed, the performance of FL methods can vary significantly across different scenarios. While FedNTD generally demonstrates effectiveness, it may underperform compared to FedAvg in certain settings.

Regarding your setup, a plausible explanation for the limited effectiveness of FedNTD concerns its knowledge preservation loss. This loss is designed to maintain feature similarity with respect to non-target classes. However, conducting too many optimization steps for each client can inadvertently lead to overconfidence in predictions for the true-target class. Essentially, this overconfidence arises in an effort to minimize Cross-Entropy loss, particularly when dealing with logits for non-target classes that are intended to be preserved. Such overemphasis on the true-target class can diminish FedNTD's effectiveness in settings with a small number of clients.

Very recently, I conducted an experiment using ResNet-18 with ImageNet-100 (13k samples, 224x224 resolution) in two different setups. In these experiments, FedNTD surpassed FedAvg in one setup but underperformed in the other:

• ResNet18, batch size 128, LDA (alpha=0.1), 200 clients, sampling ratio 0.05, 300 rounds -- FedAvg: 0.2899, FedNTD: 0.2706
• ResNet18, batch size 128, LDA (alpha=0.1), 100 clients, sampling ratio 0.1, 300 rounds -- FedAvg: 0.4346, FedNTD: 0.4418

I've also found that its performance is highly dependent on the model architecture, particularly in terms of model sizes, when using CIFAR-100 datasets:

FedAvgNet-CIFAR, batchsize 50, LDA (alpha=0.1), 100 clients, sampling ratio 0.1, 300 rounds -- FedAvg: 0.3985, FedNTD: 0.4128
ResNet18, batchsize 50, LDA (alpha=0.1), 100 clients, sampling ratio 0.1, 300 rounds -- FedAvg: 0.5470, FedNTD 0.5442

I hope this information provides you with more insight.