Comments (3)
"""
Traceback (most recent call last):
File "D:\code\LoRA_LLM\env\lib\site-packages\multiprocess\pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "D:\code\LoRA_LLM\env\lib\site-packages\datasets\utils\py_utils.py", line 1328, in _write_generator_to_queue
for i, result in enumerate(func(**kwargs)):
File "D:\code\LoRA_LLM\env\lib\site-packages\datasets\arrow_dataset.py", line 3463, in _map_single
batch = apply_function_on_filtered_inputs(
File "D:\code\LoRA_LLM\env\lib\site-packages\datasets\arrow_dataset.py", line 3344, in apply_function_on_filtered_inputs
processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
File "C:\Users\Administrator\AppData\Local\Temp\ipykernel_25996\2492540495.py", line 4, in preprocess
NameError: name 'cfg' is not defined
"""
The above exception was the direct cause of the following exception:
NameError Traceback (most recent call last)
Cell In[15], line 1
----> 1 ds_train = ds_train_raw.map(
2 preprocess,
3 batched=True,
4 num_proc=4,
5 remove_columns=ds_train_raw.column_names
6 )
8 ds_val = ds_val_raw.map(
9 preprocess,
10 batched=True,
11 num_proc=4,
12 remove_columns=ds_val_raw.column_names
13 )
File D:\code\LoRA_LLM\env\lib\site-packages\datasets\arrow_dataset.py:580, in transmit_tasks..wrapper(*args, **kwargs)
578 self: "Dataset" = kwargs.pop("self")
579 # apply actual function
--> 580 out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
581 datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
582 for dataset in datasets:
583 # Remove task templates if a column mapping of the template is no longer valid
File D:\code\LoRA_LLM\env\lib\site-packages\datasets\arrow_dataset.py:545, in transmit_format..wrapper(*args, **kwargs)
538 self_format = {
539 "type": self._format_type,
540 "format_kwargs": self._format_kwargs,
541 "columns": self._format_columns,
542 "output_all_columns": self._output_all_columns,
543 }
544 # apply actual function
--> 545 out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
546 datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
547 # re-apply format to the output
File D:\code\LoRA_LLM\env\lib\site-packages\datasets\arrow_dataset.py:3180, in Dataset.map(self, function, with_indices, with_rank, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
3172 logger.info(f"Spawning {num_proc} processes")
3173 with logging.tqdm(
3174 disable=not logging.is_progress_bar_enabled(),
3175 unit=" examples",
(...)
3178 desc=(desc or "Map") + f" (num_proc={num_proc})",
3179 ) as pbar:
-> 3180 for rank, done, content in iflatmap_unordered(
3181 pool, Dataset._map_single, kwargs_iterable=kwargs_per_job
3182 ):
3183 if done:
3184 shards_done += 1
File D:\code\LoRA_LLM\env\lib\site-packages\datasets\utils\py_utils.py:1354, in iflatmap_unordered(pool, func, kwargs_iterable)
1351 break
1352 finally:
1353 # we get the result in case there's an error to raise
-> 1354 [async_result.get(timeout=0.05) for async_result in async_results]
File D:\code\LoRA_LLM\env\lib\site-packages\datasets\utils\py_utils.py:1354, in (.0)
1351 break
1352 finally:
1353 # we get the result in case there's an error to raise
-> 1354 [async_result.get(timeout=0.05) for async_result in async_results]
File D:\code\LoRA_LLM\env\lib\site-packages\multiprocess\pool.py:774, in ApplyResult.get(self, timeout)
772 return self._value
773 else:
--> 774 raise self._value
NameError: name 'cfg' is not defined
from torchkeras.
往里面传全局参数:
def preprocess(cfg,tokenizer, examples):
from functools import partial
new_preprocess = partial(preprocess, cfg,tokenizer)
然后再将preprocess改成new_preprocess。
from torchkeras.
修改这里即可 num_proc=1。
from torchkeras.
Related Issues (20)
- 国内镜像源还没有3.9.4版本 HOT 1
- 怎么保存指定帧的参数?
- AttributeError: 'ChatGLM' object has no attribute 'top_p' HOT 4
- BaiChuan2_13B微调范例,报错“NotImplementedError: Cannot copy out of meta tensor; no data!” HOT 1
- AttributeError: 'Parameter' object has no attribute 'weight' HOT 1
- 你好,我再vscode中进行使用,好像无法支持vsocde,您有别的方案推荐么,使用wandb,服务器无法连接翻墙
- 建议增加参数
- python3.7版本使用accelerate报错
- 老铁,能不能把代码的注释加上,代码的参数用途都不晓得,读起来需要根据示例看,太费劲了~~。
- FileNotFoundError: [Errno 2] No such file or directory: 'qwen7b_multirounds/adapter_model.bin' 如何解决 HOT 8
- 大佬,能不能增加一下注释,看着有些混乱
- #
- 能否写一个微调VL-Qwen模型的例子
- 模型好像更新了,无法调用 HOT 2
- ValueError: Can't find 'adapter_config.json'
- ValueError: Attempting to unscale FP16 gradients. HOT 2
- list index out of range HOT 1
- 调度器学习率的更新是否该在EpochRunner中进行
- 请问下 pt 模型文件怎么转化成 transformers 文件结构
- 关于断点重新训练后先前的记录图像会覆盖掉,不能再次基础上进行记录
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from torchkeras.