ubuntu2204,cuda12.1
CUDA available: True
Number of CUDA devices: 1
Device 0: NVIDIA GeForce GTX 1080 Ti
Current device: 0
Current device name: NVIDIA GeForce GTX 1080 Ti
CUDA version: 12.1
CUDNN version: 8902
(NetMamba) clb@MS-7C82:~/桌面/NetMamba$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Tue_Feb__7_19:32:13_PST_2023
Cuda compilation tools, release 12.1, V12.1.66
Build cuda_12.1.r12.1/compiler.32415258_0
(NetMamba) clb@MS-7C82:~/桌面/NetMamba$ CUDA_VISIBLE_DEVICES=0 CUDA_LAUNCH_BLOCKING=1 python src/pre-train.py --batch_size 32 --blr 1e-3 --steps 150000 --mask_ratio 0.9 --data_path dataset/root/Vim/dataset/ISCXVPN2016/dataset_sampled --model net_mamba_pretrain --no_amp --num_workers 0
Not using distributed mode
[11:13:47.785222] job dir: /home/clb/桌面/NetMamba/src
[11:13:47.785270] Namespace(batch_size=32,
steps=150000,
epochs=400,
save_steps_freq=10000,
accum_iter=1,
model='net_mamba_pretrain',
input_size=40,
mask_ratio=0.9,
norm_pix_loss=False,
weight_decay=0.05,
lr=None,
blr=0.001,
min_lr=0.0,
warmup_epochs=25,
data_path='dataset/root/Vim/dataset/ISCXVPN2016/dataset_sampled',
output_dir='./output/pretrain',
log_dir='./output/pretrain',
device='cuda',
seed=0,
resume='',
num_workers=0,
pin_mem=True,
world_size=1,
local_rank=-1,
dist_on_itp=False,
dist_url='env://',
if_amp=False,
pop=False,
pop_loss_weight=0.01,
distributed=False)
[11:13:47.811319] Dataset ImageFolder
Number of datapoints: 13281
Root location: dataset/root/Vim/dataset/ISCXVPN2016/dataset_sampled/train
StandardTransform
Transform: Compose(
Grayscale(num_output_channels=1)
ToTensor()
Normalize(mean=[0.5], std=[0.5])
)
[11:13:47.811398] Sampler_train = <torch.utils.data.distributed.DistributedSampler object at 0x7253e03cb310>
[11:13:47.927809] Model = NetMamba(
(patch_embed): StrideEmbed(
(proj): Conv1d(1, 256, kernel_size=(4,), stride=(4,))
)
(drop_path): DropPath()
(blocks): ModuleList(
(0-1): 2 x Block(
(mixer): Mamba(
(in_proj): Linear(in_features=256, out_features=1024, bias=False)
(conv1d): Conv1d(512, 512, kernel_size=(4,), stride=(1,), padding=(3,), groups=512)
(act): SiLU()
(x_proj): Linear(in_features=512, out_features=48, bias=False)
(dt_proj): Linear(in_features=16, out_features=512, bias=True)
(out_proj): Linear(in_features=512, out_features=256, bias=False)
)
(norm): RMSNorm()
(drop_path): Identity()
)
(2-3): 2 x Block(
(mixer): Mamba(
(in_proj): Linear(in_features=256, out_features=1024, bias=False)
(conv1d): Conv1d(512, 512, kernel_size=(4,), stride=(1,), padding=(3,), groups=512)
(act): SiLU()
(x_proj): Linear(in_features=512, out_features=48, bias=False)
(dt_proj): Linear(in_features=16, out_features=512, bias=True)
(out_proj): Linear(in_features=512, out_features=256, bias=False)
)
(norm): RMSNorm()
(drop_path): DropPath()
)
)
(norm_f): RMSNorm()
(decoder_embed): Linear(in_features=256, out_features=128, bias=True)
(decoder_blocks): ModuleList(
(0-1): 2 x Block(
(mixer): Mamba(
(in_proj): Linear(in_features=128, out_features=512, bias=False)
(conv1d): Conv1d(256, 256, kernel_size=(4,), stride=(1,), padding=(3,), groups=256)
(act): SiLU()
(x_proj): Linear(in_features=256, out_features=40, bias=False)
(dt_proj): Linear(in_features=8, out_features=256, bias=True)
(out_proj): Linear(in_features=256, out_features=128, bias=False)
)
(norm): RMSNorm()
(drop_path): Identity()
)
)
(decoder_norm_f): RMSNorm()
(decoder_pred): Linear(in_features=128, out_features=4, bias=True)
)
[11:13:47.928055] trainable params: 2174724 || all params: 2174724 || trainable%: 100.0000
[11:13:47.928089] base lr: 1.00e-03
[11:13:47.928098] actual lr: 1.25e-04
[11:13:47.928104] accumulate grad iterations: 1
[11:13:47.928110] effective batch size: 32
[11:13:47.928466] AdamW (
Parameter Group 0
amsgrad: False
betas: (0.9, 0.95)
capturable: False
differentiable: False
eps: 1e-08
foreach: None
fused: None
lr: 0.000125
maximize: False
weight_decay: 0.0
Parameter Group 1
amsgrad: False
betas: (0.9, 0.95)
capturable: False
differentiable: False
eps: 1e-08
foreach: None
fused: None
lr: 0.000125
maximize: False
weight_decay: 0.05
)
[11:13:47.928530] Start training for 150000 steps
[11:13:47.928979] log_dir: ./output/pretrain
Traceback (most recent call last):
File "/home/clb/桌面/NetMamba/src/pre-train.py", line 236, in
main(args)
File "/home/clb/桌面/NetMamba/src/pre-train.py", line 205, in main
train_stats = pretrain_one_epoch(
File "/home/clb/桌面/NetMamba/src/engine.py", line 48, in pretrain_one_epoch
loss, _, _ = model(samples, mask_ratio=args.mask_ratio)
File "/home/clb/anaconda3/envs/NetMamba/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/clb/anaconda3/envs/NetMamba/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/clb/桌面/NetMamba/src/models_net_mamba.py", line 264, in forward
latent, mask, ids_restore = self.forward_encoder(imgs,
File "/home/clb/桌面/NetMamba/src/models_net_mamba.py", line 188, in forward_encoder
hidden_states, residual = blk(hidden_states, residual)
File "/home/clb/anaconda3/envs/NetMamba/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/clb/anaconda3/envs/NetMamba/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/clb/桌面/NetMamba/src/models_mamba.py", line 94, in forward
hidden_states = self.mixer(hidden_states, inference_params=inference_params)
File "/home/clb/anaconda3/envs/NetMamba/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/clb/anaconda3/envs/NetMamba/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/clb/桌面/NetMamba/mamba-1p1p1/mamba_ssm/modules/mamba_simple.py", line 249, in forward
out = mamba_inner_fn(
File "/home/clb/桌面/NetMamba/mamba-1p1p1/mamba_ssm/ops/selective_scan_interface.py", line 612, in mamba_inner_fn
return MambaInnerFn.apply(xz, conv1d_weight, conv1d_bias, x_proj_weight, delta_proj_weight,
File "/home/clb/anaconda3/envs/NetMamba/lib/python3.10/site-packages/torch/autograd/function.py", line 539, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
File "/home/clb/anaconda3/envs/NetMamba/lib/python3.10/site-packages/torch/cuda/amp/autocast_mode.py", line 113, in decorate_fwd
return fwd(*args, **kwargs)
File "/home/clb/桌面/NetMamba/mamba-1p1p1/mamba_ssm/ops/selective_scan_interface.py", line 318, in forward
conv1d_out = causal_conv1d_cuda.causal_conv1d_fwd(x, conv1d_weight, conv1d_bias, None, True)
RuntimeError: CUDA error: no kernel image is available for execution on the device
Compile with TORCH_USE_CUDA_DSA
to enable device-side assertions.
请问一下这个是什么原因导致的呢