数据预处理报错 about chatglm-tuning

[zxcvbn114514]

兄弟可以看我的issue,有大佬解答了

[c2j]

Same Error in the first step:
(GPTENV) c2j@Goku:~/workspace/GPT/ChatGLM-Tuning$ python tokenize_dataset_rows.py --jsonl_path data/alpaca_data.jsonl --save_path data/alpaca --max_seq_length 512
Explicitly passing a revision is encouraged when loading a model with custom code to ensure no malicious code has been contributed in a newer revision.
52002it [00:20, 2508.60it/s]
Traceback (most recent call last):
File "tokenize_dataset_rows.py", line 45, in
main()
File "tokenize_dataset_rows.py", line 38, in main
arr = np.array(all_tokenized)
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (52002,) + inhomogeneous part.

[MEIZU16]

兄弟可以看我的issue,有大佬解答了

ok