smilelab-fl / fedpetuning Goto Github PK

您好。我发现您们的代码似乎只能在3个gpu的单机运行才行。否则出错
例如我在google colab上运行
!python fed_seed_run.py /content/drive/MyDrive/workspace/ fedavg mrpc adapter 1100 0，1，3

训练相当长时间，报错收不到客户消息超时退出。想改成!python fed_seed_run.py /content/drive/MyDrive/workspace/ fedavg mrpc adapter 1100 0 运行也报错
因为colab只能有一个gpu,不知您有没有办法让他在单gpu机器上运行？（或哪里可找到多gpu的云计算平台）

11-13/23:28:18|INFO |base_client.py:292|MRPC Train, Client:14, Loss:0.318, Accuracy:0.939
11-13/23:28:36|INFO |base_client.py:292|MRPC Train, Client:3, Loss:0.313, Accuracy:0.970
11-13/23:28:54|INFO |base_client.py:292|MRPC Train, Client:35, Loss:0.324, Accuracy:0.939
11-13/23:29:13|INFO |base_client.py:292|MRPC Train, Client:31, Loss:0.309, Accuracy:0.970
Traceback (most recent call last):
File "/content/drive/MyDrive/workspace/code/FedETuning/main.py", line 20, in
main()
File "/content/drive/MyDrive/workspace/code/FedETuning/main.py", line 16, in main
trainer.train()
File "/content/drive/MyDrive/workspace/code/FedETuning/trainers/FedBaseTrainer.py", line 91, in train
self.server_manger.run()
File "/content/drive/MyDrive/workspace/code/FedETuning/fedlab/core/network_manager.py", line 38, in run
self.main_loop()
File "/content/drive/MyDrive/workspace/code/FedETuning/trainers/BaseServer/base_server.py", line 253, in main_loop
sender_rank, message_code, payload = self._network.recv()
File "/content/drive/MyDrive/workspace/code/FedETuning/fedlab/core/network.py", line 102, in recv
sender_rank, message_code, content = PackageProcessor.recv_package(
File "/content/drive/MyDrive/workspace/code/FedETuning/fedlab/core/communicator/processor.py", line 118, in recv_package
sender_rank, _, slices_size, message_code, data_type = recv_header(
File "/content/drive/MyDrive/workspace/code/FedETuning/fedlab/core/communicator/processor.py", line 96, in recv_header
dist.recv(buffer, src=src)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 1632, in recv
work.wait()
RuntimeError: [../third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:81] Timed out waiting 18000000ms for recv operation to complete
sh: 0: getcwd() failed: Transport endpoint is not connected
python3: can't open file 'main.py': [Errno 107] Transport endpoint is not connected
python3: can't open file 'main.py': [Errno 107] Transport endpoint is not connected
python3: can't open file 'main.py': [Errno 107] Transport endpoint is not connected
sh: 0: getcwd() failed: Transport endpoint is not connected
python3: can't open file 'main.py': [Errno 107] Transport endpoint is not connected
python3: can't open file 'main.py': [Errno 107] Transport endpoint is not connected
python3: can't open file 'main.py': [Errno 107] Transport endpoint is not connected
sh: 0: getcwd() failed: Transport endpoint is not connected
python3: can't open file 'main.py': [Errno 107] Transport endpoint is not connected
python3: can't open file 'main.py': [Errno 107] Transport endpoint is not connected
python3: can't open file 'main.py': [Errno 107] Transport endpoint is not connected
sh: 0: getcwd() failed: Transport endpoint is not connected
python3: can't open file 'main.py': [Errno 107] Transport endpoint is not connected
python3: can't open file 'main.py': [Errno 107] Transport endpoint is not connected
python3: can't open file 'main.py': [Errno 107] Transport endpoint is not connected
Exception ignored in: <function Pool.del at 0x7cf9930e17e0>
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/pool.py", line 271, in del
File "/usr/lib/python3.10/multiprocessing/queues.py", line 371, in put
AttributeError: 'NoneType' object has no attribute 'dumps'

Hi, thank you for providing the initial code.
I have thoroughly examined the code and find myself intrigued by the section concerning "reconstruction attacks." I am eager to understand the implementation details further. Does this pertain to the creation of an MNLi attack dataset, followed by the computation of training metrics such as F1 and Precision?
If so, could you kindly elucidate on the methodology employed for constructing the MNLi dataset? Your response would be greatly appreciated.

When I try to reproduce the result, I encounter a bug. I post some information below. Please contact me if anyone knows how to deal with it. Any help from you will be gratitude.

+++
-environment:
Ubuntu 22.04.3
python 3.10.12
CUDA 12.1
PyTorch 2.1.1
GPU V100 * 1
-Command:
python fed_seed_run.py /root/workspace fedavg rte fine-tuning 1000 0,0,0
-Log:
Traceback (most recent call last):
File "/root/workspace/code/FedETuning/main.py", line 20, in
main()
File "/root/workspace/code/FedETuning/main.py", line 16, in main
trainer.train()
File "/root/workspace/code/FedETuning/trainers/FedBaseTrainer.py", line 96, in train
self.client_manager.run()
File "/root/workspace/code/FedETuning/fedlab/core/network_manager.py", line 38, in run
self.main_loop()
File "/root/workspace/code/FedETuning/trainers/BaseClient/base_client.py", line 344, in main_loop
sender_rank, message_code, payload = self._network.recv(src=0)
File "/root/workspace/code/FedETuning/fedlab/core/network.py", line 102, in recv
sender_rank, message_code, content = PackageProcessor.recv_package(
File "/root/workspace/code/FedETuning/fedlab/core/communicator/processor.py", line 118, in recv_package
sender_rank, _, slices_size, message_code, data_type = recv_header(
File "/root/workspace/code/FedETuning/fedlab/core/communicator/processor.py", line 96, in recv_header
dist.recv(buffer, src=src)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 1640, in recv
pg.recv([tensor], src, tag).wait()
RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [172.17.0.5]:38114
Traceback (most recent call last):
File "/root/workspace/code/FedETuning/main.py", line 20, in
main()
File "/root/workspace/code/FedETuning/main.py", line 16, in main
trainer.train()
File "/root/workspace/code/FedETuning/trainers/FedBaseTrainer.py", line 96, in train
self.client_manager.run()
File "/root/workspace/code/FedETuning/fedlab/core/network_manager.py", line 38, in run
self.main_loop()
File "/root/workspace/code/FedETuning/trainers/BaseClient/base_client.py", line 368, in main_loop
self.synchronize()
File "/root/workspace/code/FedETuning/trainers/BaseClient/base_client.py", line 376, in synchronize
self._network.send(
File "/root/workspace/code/FedETuning/fedlab/core/network.py", line 90, in send
PackageProcessor.send_package(pack, dst=dst)
File "/root/workspace/code/FedETuning/fedlab/core/communicator/processor.py", line 74, in send_package
send_content(content=package.content, dst=dst)
File "/root/workspace/code/FedETuning/fedlab/core/communicator/processor.py", line 61, in send_content
dist.send(content, dst=dst)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 1597, in send
default_pg.send([tensor], dst, tag).wait()
RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [172.17.0.5]:41358
Exception ignored in: <function Pool.del at 0x7f0097291990>
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/pool.py", line 271, in del
File "/usr/lib/python3.10/multiprocessing/queues.py", line 371, in put
AttributeError: 'NoneType' object has no attribute 'dumps'
+++

According to figure3.py, the communication budgets are plotted using the data in the list parameter_num. I'm curious about the computation of the values within parameter_num and would like to understand why they are constants.