smilelab-fl / fedpetuning Goto Github PK
View Code? Open in Web Editor NEWLicense: Other
License: Other
您好。我发现您们的代码似乎只能在3个gpu的单机运行才行。否则出错
例如我在google colab上运行
!python fed_seed_run.py /content/drive/MyDrive/workspace/ fedavg mrpc adapter 1100 0,1,3
训练相当长时间,报错收不到客户消息超时退出。想改成!python fed_seed_run.py /content/drive/MyDrive/workspace/ fedavg mrpc adapter 1100 0 运行也报错
因为colab只能有一个gpu,不知您有没有办法让他在单gpu机器上运行?(或哪里可找到多gpu的云计算平台)
11-13/23:28:18|INFO |base_client.py:292|MRPC Train, Client:14, Loss:0.318, Accuracy:0.939
11-13/23:28:36|INFO |base_client.py:292|MRPC Train, Client:3, Loss:0.313, Accuracy:0.970
11-13/23:28:54|INFO |base_client.py:292|MRPC Train, Client:35, Loss:0.324, Accuracy:0.939
11-13/23:29:13|INFO |base_client.py:292|MRPC Train, Client:31, Loss:0.309, Accuracy:0.970
Traceback (most recent call last):
File "/content/drive/MyDrive/workspace/code/FedETuning/main.py", line 20, in
main()
File "/content/drive/MyDrive/workspace/code/FedETuning/main.py", line 16, in main
trainer.train()
File "/content/drive/MyDrive/workspace/code/FedETuning/trainers/FedBaseTrainer.py", line 91, in train
self.server_manger.run()
File "/content/drive/MyDrive/workspace/code/FedETuning/fedlab/core/network_manager.py", line 38, in run
self.main_loop()
File "/content/drive/MyDrive/workspace/code/FedETuning/trainers/BaseServer/base_server.py", line 253, in main_loop
sender_rank, message_code, payload = self._network.recv()
File "/content/drive/MyDrive/workspace/code/FedETuning/fedlab/core/network.py", line 102, in recv
sender_rank, message_code, content = PackageProcessor.recv_package(
File "/content/drive/MyDrive/workspace/code/FedETuning/fedlab/core/communicator/processor.py", line 118, in recv_package
sender_rank, _, slices_size, message_code, data_type = recv_header(
File "/content/drive/MyDrive/workspace/code/FedETuning/fedlab/core/communicator/processor.py", line 96, in recv_header
dist.recv(buffer, src=src)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 1632, in recv
work.wait()
RuntimeError: [../third_party/gloo/gloo/transport/tcp/unbound_buffer.cc:81] Timed out waiting 18000000ms for recv operation to complete
sh: 0: getcwd() failed: Transport endpoint is not connected
python3: can't open file 'main.py': [Errno 107] Transport endpoint is not connected
python3: can't open file 'main.py': [Errno 107] Transport endpoint is not connected
python3: can't open file 'main.py': [Errno 107] Transport endpoint is not connected
sh: 0: getcwd() failed: Transport endpoint is not connected
python3: can't open file 'main.py': [Errno 107] Transport endpoint is not connected
python3: can't open file 'main.py': [Errno 107] Transport endpoint is not connected
python3: can't open file 'main.py': [Errno 107] Transport endpoint is not connected
sh: 0: getcwd() failed: Transport endpoint is not connected
python3: can't open file 'main.py': [Errno 107] Transport endpoint is not connected
python3: can't open file 'main.py': [Errno 107] Transport endpoint is not connected
python3: can't open file 'main.py': [Errno 107] Transport endpoint is not connected
sh: 0: getcwd() failed: Transport endpoint is not connected
python3: can't open file 'main.py': [Errno 107] Transport endpoint is not connected
python3: can't open file 'main.py': [Errno 107] Transport endpoint is not connected
python3: can't open file 'main.py': [Errno 107] Transport endpoint is not connected
Exception ignored in: <function Pool.del at 0x7cf9930e17e0>
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/pool.py", line 271, in del
File "/usr/lib/python3.10/multiprocessing/queues.py", line 371, in put
AttributeError: 'NoneType' object has no attribute 'dumps'
Hi, thank you for providing the initial code.
I have thoroughly examined the code and find myself intrigued by the section concerning "reconstruction attacks." I am eager to understand the implementation details further. Does this pertain to the creation of an MNLi attack dataset, followed by the computation of training metrics such as F1 and Precision?
If so, could you kindly elucidate on the methodology employed for constructing the MNLi dataset? Your response would be greatly appreciated.
When I try to reproduce the result, I encounter a bug. I post some information below. Please contact me if anyone knows how to deal with it. Any help from you will be gratitude.
+++
-environment:
Ubuntu 22.04.3
python 3.10.12
CUDA 12.1
PyTorch 2.1.1
GPU V100 * 1
-Command:
python fed_seed_run.py /root/workspace fedavg rte fine-tuning 1000 0,0,0
-Log:
Traceback (most recent call last):
File "/root/workspace/code/FedETuning/main.py", line 20, in
main()
File "/root/workspace/code/FedETuning/main.py", line 16, in main
trainer.train()
File "/root/workspace/code/FedETuning/trainers/FedBaseTrainer.py", line 96, in train
self.client_manager.run()
File "/root/workspace/code/FedETuning/fedlab/core/network_manager.py", line 38, in run
self.main_loop()
File "/root/workspace/code/FedETuning/trainers/BaseClient/base_client.py", line 344, in main_loop
sender_rank, message_code, payload = self._network.recv(src=0)
File "/root/workspace/code/FedETuning/fedlab/core/network.py", line 102, in recv
sender_rank, message_code, content = PackageProcessor.recv_package(
File "/root/workspace/code/FedETuning/fedlab/core/communicator/processor.py", line 118, in recv_package
sender_rank, _, slices_size, message_code, data_type = recv_header(
File "/root/workspace/code/FedETuning/fedlab/core/communicator/processor.py", line 96, in recv_header
dist.recv(buffer, src=src)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 1640, in recv
pg.recv([tensor], src, tag).wait()
RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [172.17.0.5]:38114
Traceback (most recent call last):
File "/root/workspace/code/FedETuning/main.py", line 20, in
main()
File "/root/workspace/code/FedETuning/main.py", line 16, in main
trainer.train()
File "/root/workspace/code/FedETuning/trainers/FedBaseTrainer.py", line 96, in train
self.client_manager.run()
File "/root/workspace/code/FedETuning/fedlab/core/network_manager.py", line 38, in run
self.main_loop()
File "/root/workspace/code/FedETuning/trainers/BaseClient/base_client.py", line 368, in main_loop
self.synchronize()
File "/root/workspace/code/FedETuning/trainers/BaseClient/base_client.py", line 376, in synchronize
self._network.send(
File "/root/workspace/code/FedETuning/fedlab/core/network.py", line 90, in send
PackageProcessor.send_package(pack, dst=dst)
File "/root/workspace/code/FedETuning/fedlab/core/communicator/processor.py", line 74, in send_package
send_content(content=package.content, dst=dst)
File "/root/workspace/code/FedETuning/fedlab/core/communicator/processor.py", line 61, in send_content
dist.send(content, dst=dst)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 1597, in send
default_pg.send([tensor], dst, tag).wait()
RuntimeError: [../third_party/gloo/gloo/transport/tcp/pair.cc:534] Connection closed by peer [172.17.0.5]:41358
Exception ignored in: <function Pool.del at 0x7f0097291990>
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/pool.py", line 271, in del
File "/usr/lib/python3.10/multiprocessing/queues.py", line 371, in put
AttributeError: 'NoneType' object has no attribute 'dumps'
+++
According to figure3.py
, the communication budgets are plotted using the data in the list parameter_num
. I'm curious about the computation of the values within parameter_num
and would like to understand why they are constants.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.