minaremeli / moon Goto Github PK

Currently all new results overwrite old ones. Need a solution for saving results in different locations.

We want to use Docker to deploy and run experiments on any cloud platform quickly, and seamlessly.

Running the model with the following parameters does not learn even after 100 rounds of training (this code ran for a total of 1.5 hours on dev-gpu-0.cl.cam.ac.uk ❗ I think this is less than expected).

python main.py --num_rounds 100 --sample_fraction_fit 1.0 --num_clients 10 --strategy fedAvg --beta 5

Very small changes (1e-3 and smaller) can be observed over time on the loss (CrossEntropyLoss), but accuracy almost always stays a constant 0.1. This is true for both local and global accuracies. I experienced the same when running for fewer rounds / different beta / num_clients / sample_fraction_fit / etc.

Possible causes:

• learning rate is too small
• weights don't get updated
• gradient vanishing (?)

I might need some help figuring this out @VasundharaAgarwal 🙏

So far only accuracy has been calculated on evaluation (both centralized and distributed). Additionally, we could calculate the supervised loss (CrossEntropyLoss) for debugging and evaluation purposes.

I have thought about calculating the contrastive loss in case of moon strategy, but have decided against it. 1) The implementation would be messy in my opinion 2) Contrastive loss does not make sense to me when the global and the local model are the same 3) Logically CrossEntropyLoss should decrease as well if implementation works

In each round we fit on fraction * num_clients.

In each round we also evaluate on client's local dataset. Currently evaluation is calculated on a fix number of 2 clients.

Expected behaviour is not clear, and there are multiple options:

• always evaluate on the full set of clients
• evaluate on fraction * num_clients (random sampling)
• evaluate on fraction * num_clients (SAME clients as before, when we fit the model)

@VasundharaAgarwal what do you think?

We use Flower to simulate Federated Learning between clients. However, we don't want to start clients by hand. We will look into solutions for simulating these clients automatically. (For example, Ray.)

After running an experiment, accuracies calculated on centralised and local data should be saved.

The corresponding files get created:

• data/accs_centralized.npy
• data/accs_distributed.npy

However, when loading them using np.load(), we get an empty array array([], dtype=float64).

When running main.py --to_partition False the boolean value is not set to False, but to True, because non-empty strings are True in Python.

Task scope

We recommend you focus on the central claim of the paper. For example, if a paper introduces a new RL learning algorithm that performs better in sparse-reward environments, verify that you can re-implement the algorithm, run it on the same benchmarks and get results that are close to those in the original paper (exact reproducibility is in most cases very difficult due to minor implementation details). You do not need to reproduce all experiments in your selected paper, but only those that you feel are sufficient for you to verify the validity of the central claim.

More information on the task here.

ToDo's

• Briefly define the claim of the paper that you intend to tackle in this report.
• Choose experiments to reproduce.
• Review possible extension of the paper w/ additional experiments. (optional)

For reproducibility purposes, the code should execute the same with a fixed seed.