重启shell后训练报错 about xtuner

[LZHgrla]

在这一行报错

from peft import get_peft_model, prepare_model_for_kbit_training

看起来是环境有问题

[LumenScope]

卸载deepspeed，之后重装xtuner以安装适配版本的deepspeed后报错更改：

Traceback (most recent call last):
  File "/root/miniconda3/envs/xtuner/lib/python3.10/site-packages/xtuner/tools/train.py", line 299, in <module>
    main()
  File "/root/miniconda3/envs/xtuner/lib/python3.10/site-packages/xtuner/tools/train.py", line 205, in main
    if digit_version(deepspeed.__version__) < digit_version('0.12.3'):
AttributeError: module 'deepspeed' has no attribute '__version__'
Traceback (most recent call last):
  File "/root/miniconda3/envs/xtuner/lib/python3.10/site-packages/xtuner/tools/train.py", line 299, in <module>
    main()
  File "/root/miniconda3/envs/xtuner/lib/python3.10/site-packages/xtuner/tools/train.py", line 205, in main
    if digit_version(deepspeed.__version__) < digit_version('0.12.3'):
AttributeError: module 'deepspeed' has no attribute '__version__'
Traceback (most recent call last):
Traceback (most recent call last):
  File "/root/miniconda3/envs/xtuner/lib/python3.10/site-packages/xtuner/tools/train.py", line 299, in <module>
  File "/root/miniconda3/envs/xtuner/lib/python3.10/site-packages/xtuner/tools/train.py", line 299, in <module>
Traceback (most recent call last):
  File "/root/miniconda3/envs/xtuner/lib/python3.10/site-packages/xtuner/tools/train.py", line 299, in <module>
        main()main()

  File "/root/miniconda3/envs/xtuner/lib/python3.10/site-packages/xtuner/tools/train.py", line 205, in main
  File "/root/miniconda3/envs/xtuner/lib/python3.10/site-packages/xtuner/tools/train.py", line 205, in main
    main()
  File "/root/miniconda3/envs/xtuner/lib/python3.10/site-packages/xtuner/tools/train.py", line 205, in main
        if digit_version(deepspeed.__version__) < digit_version('0.12.3'):if digit_version(deepspeed.__version__) < digit_version('0.12.3'):

AttributeErrorAttributeError: : module 'deepspeed' has no attribute '__version__'module 'deepspeed' has no attribute '__version__'

    if digit_version(deepspeed.__version__) < digit_version('0.12.3'):
AttributeError: module 'deepspeed' has no attribute '__version__'
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 133292 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 133293 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 133294 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 133295 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 133291) of binary: /root/usr/bin/python3.10
Traceback (most recent call last):
  File "/root/usr/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/root/usr/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/root/usr/lib/python3.10/site-packages/torch/distributed/run.py", line 794, in main
    run(args)
  File "/root/usr/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/root/usr/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/root/usr/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/root/miniconda3/envs/xtuner/lib/python3.10/site-packages/xtuner/tools/train.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-02-01_17:24:37
  host      : VM-0-12-centos
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 133291)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

[LumenScope]

在这一行报错

from peft import get_peft_model, prepare_model_for_kbit_training

看起来是环境有问题

好离谱啊，我就只重启了shellQAQ

[LZHgrla]

上游库的环境问题:(

[LumenScope]

上游库的环境问题:(

可能是deepspeed出问题了？我从conda环境全部重装了还是不行，甚至 accelerate config也不行了：

(xtuner) [root@VM-0-12-centos tzz]# accelerate config
[2024-02-01 17:31:56,647] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Traceback (most recent call last):
  File "/root/usr/bin/accelerate", line 5, in <module>
    from accelerate.commands.accelerate_cli import main
  File "/root/usr/lib/python3.10/site-packages/accelerate/__init__.py", line 3, in <module>
    from .accelerator import Accelerator
  File "/root/usr/lib/python3.10/site-packages/accelerate/accelerator.py", line 35, in <module>
    from .checkpointing import load_accelerator_state, load_custom_state, save_accelerator_state, save_custom_state
  File "/root/usr/lib/python3.10/site-packages/accelerate/checkpointing.py", line 24, in <module>
    from .utils import (
  File "/root/usr/lib/python3.10/site-packages/accelerate/utils/__init__.py", line 133, in <module>
    from .launch import (
  File "/root/usr/lib/python3.10/site-packages/accelerate/utils/launch.py", line 33, in <module>
    from ..utils.other import is_port_in_use, merge_dicts
  File "/root/usr/lib/python3.10/site-packages/accelerate/utils/other.py", line 30, in <module>
    from deepspeed import DeepSpeedEngine
  File "/root/usr/lib/python3.10/site-packages/deepspeed/__init__.py", line 22, in <module>
    from . import module_inject
  File "/root/usr/lib/python3.10/site-packages/deepspeed/module_inject/__init__.py", line 6, in <module>
    from .replace_module import replace_transformer_layer, revert_transformer_layer, ReplaceWithTensorSlicing, GroupQuantizer, generic_injection
  File "/root/usr/lib/python3.10/site-packages/deepspeed/module_inject/replace_module.py", line 607, in <module>
    from ..pipe import PipelineModule
  File "/root/usr/lib/python3.10/site-packages/deepspeed/pipe/__init__.py", line 6, in <module>
    from ..runtime.pipe import PipelineModule, LayerSpec, TiedLayerSpec
  File "/root/usr/lib/python3.10/site-packages/deepspeed/runtime/pipe/__init__.py", line 6, in <module>
    from .module import PipelineModule, LayerSpec, TiedLayerSpec
  File "/root/usr/lib/python3.10/site-packages/deepspeed/runtime/pipe/module.py", line 19, in <module>
    from ..activation_checkpointing import checkpointing
  File "/root/usr/lib/python3.10/site-packages/deepspeed/runtime/activation_checkpointing/checkpointing.py", line 26, in <module>
    from deepspeed.runtime.config import DeepSpeedConfig
  File "/root/usr/lib/python3.10/site-packages/deepspeed/runtime/config.py", line 24, in <module>
    from .config_utils import (
  File "/root/usr/lib/python3.10/site-packages/deepspeed/runtime/config_utils.py", line 12, in <module>
    from deepspeed.pydantic_v1 import BaseModel
ImportError: cannot import name 'BaseModel' from 'deepspeed.pydantic_v1' (/root/usr/lib/python3.10/site-packages/deepspeed/pydantic_v1.py)

[LZHgrla]

python 环境有问题，看起来走的不是anaconda的虚拟环境，而是/root/usr/lib/python3.10/site-packages