deepseek-ai / deepseek-moe Goto Github PK

Detected kernel version 4.19.118, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
Traceback (most recent call last):
  File "/data/share_user//training/fintune.py", line 332, in <module>
    train()
  File "/data/share_user/training/fintune.py", line 322, in train
    trainer = Trainer(model=model, tokenizer=tokenizer, args=training_args, **data_module)
  File "/root/miniconda3/envs/moe-env/lib/python3.10/site-packages/transformers/trainer.py", line 408, in __init__
    raise ValueError(
ValueError: You cannot perform fine-tuning on purely quantized models. Please attach trainable adapters on top of the quantized model to correctly perform fine-tuning. Please see: https://huggingface.co/docs/transformers/peft for more details

DATA_PATH="garage-bAInd/Open-Platypus"
OUTPUT_PATH="/test"
MODEL_PATH="deepseek-ai/deepseek-moe-16b-base"

deepspeed training/finetune.py \
    --model_name_or_path $MODEL_PATH \
    --data_path $DATA_PATH \
    --output_dir $OUTPUT_PATH \
    --cache_dir ./cache \
    --num_train_epochs 1 \
    --model_max_length 1024 \
    --per_device_train_batch_size 16 \
    --per_device_eval_batch_size 1 \
    --gradient_accumulation_steps 4 \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 100 \
    --save_total_limit 2 \
    --learning_rate 2e-5 \
    --warmup_steps 10 \
    --logging_steps 1 \
    --lr_scheduler_type "cosine" \
    --gradient_checkpointing True \
    --report_to "tensorboard" \
    --deepspeed training/configs/ds_config_zero3.json \
    --bf16 True 

deepseek-moe-16b 14.73 tokens/s
baichuan-13b 22.00 tokens/s

model.eval()
    tokenizer = AutoTokenizer.from_pretrained(tokenizer_path)
    messages = [
        {"role": "user", "content": "How to learn English?"},
    ]
    input_tensor = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")

    
    outputs = model.generate(input_tensor.to(model.device), max_new_tokens=max_new_tokens)
    result = tokenizer.decode(outputs[0][input_tensor.shape[1]:], skip_special_tokens=True)
    print(result)

1. Reading: Reading is the first step in learning English. It helps to develop vocabulary, grammar, and pronunciation skills. Choose a picture book or a good friend to read on.

2. Writing: Writing is a great way to practice grammar and sentence structure. Write short stories or practice writing on a computer or journal.

3. Listening and Speaking: Listening to English and speaking in front of an English audience can help to improve pronunciation and listening skills.

4. Reading books and games: Reading books and games with English can help to develop vocabulary, reading style, and comprehension skills. Choose books and games that have interactive elements and that challenge you.

5. Speaking with an English tutor or tutor in a classroom: Having a trained English tutor or tutor in a classroom can provide you with real-life practice and help you improve grammar and vocabulary skills.

6. Watching movies and TV shows: Watching movies and TV shows with subtitles or in international English can help to improve your listening skills and vocabulary. Choose movies and shows that have subtitles or that feature English characters.

7. Playing with a native speaker: Playing with a native speaker of the language can help to improve pronunciation and vocabulary skills. Choose a friend to play with and practice in a group.


8. Speaking with a real-life English person: Speaking with a real-life English person can help to improve grammar, vocabulary, and listening skills. Choose a friend to practice with a trained English tutor or tutor.
<|EOT|>OT|>2. Playing with all of these methods above can help to improve your English skills.
<||EOT|>||
<|EOT|>|
<|EOT|>|
<|EOT|>
<|EOT|>
<|EOT|>
<|EOT|>
<|EOT|>
<EOT>
<|EOT|>
<|EOT>
<|EOT>
<|EOT>
<|EOT>
<EOT>
<EOT>
<|EOT>
<EOT>
<EOT>
<|EOT>
<EOT>
<EOT>
<|EOT>
<EOT>
<EOT>
<EOT>
<EOT>
<EOT>
<EOT>
<EOT>
<EOT>
<EOT>
<EOT>
<EOT>
<EOT>
<EOT>
<EOT>
<EOT>
<EOT>
<EOT>
<EOT>
<EOT>
<EOT>
<EOT>
<EOT>
<EOT>
<EOT>
<EOT>
<EOT>
<EOT>
<EOT>
<EOTOT>
<EOT>
<EOT>
<EOTOT>
<EOT>
<EOT>
<EOTOT>
<EOT>
<EOT>
<EOTOT>
<EOT>
<EOT>
<EOTOT>
<EOT>
<EOT>
<EOTOT>
<EOT>
<EOT>
<EOTOT>
<EOT>
<EOT>
<EOTOT>
<EOT>
<EOT>
<EOTOT>
<EOT>
<EOT>
<EOTOT>
<EOT>
<EOT>
<EOTOT>
<EOTOT>
<EOT>
<EOT>
<EOT>
<EOTOT>
<EOT>
<<EOTOT>
<EOT>
<EOT>
<EOT>EOT>
<EOTOT>
<EOT>
<EOT>
<EOTOTEOT>
<EOTOTEOT>
<EOTOTEOT>
<EOTOTEOT>
<EOTOTOT>
<EOTOT>
<EOT>
<EOT>
<EOTOT>
<EOT>
<EOT>
<EOTOT>
<EOT>
<EOT>
<EOTOT>
<EOT>
<EOT>
<EOTOT>
<EOT>
<EOT>
<EOTOT>
<EOT>
<EOT>
<EOTOT>
<EOT>
<EOT>
<EOTOT>
<EOT>
<EOTOTOT
<EOTOTOT

lm_eval --model hf \  
    --model_args pretrained=deepseek-ai/deepseek-moe-16b-chat,dtype="bfloat16",parallelize=True \  
    --tasks drop \
    --batch_size 8 \
    --num_fewshot 1 \
    --output_path deepseek-moe-16b-chat-drop.json \
    --trust_remote_code

{
  "results": {
    "drop": {
      "em,none": 0.011954697986577181,
      "em_stderr,none": 0.001113005689885913,
      "f1,none": 0.044237625838926285,
      "f1_stderr,none": 0.0014818249593304353,
      "alias": "drop"
    }
  },
  "group_subtasks": {
    "drop": []
  },
  "configs": {
    "drop": {
      "task": "drop",
      "dataset_path": "benchmark_datasets/drop",
      "dataset_kwargs": {
        "trust_remote_code": true
      },
      "training_split": "train",
      "validation_split": "validation",
      "process_docs": "def process_docs(dataset):\n    def _process(doc):\n        return {\n            \"id\": doc[\"query_id\"],\n            \"passage\": doc[\"passage\"],\n            \"question\": doc[\"question\"],\n            \"answers\": get_answers(doc),\n        }\n\n    return dataset.map(_process)\n",
      "doc_to_text": "{{passage}} {{question}}",
      "doc_to_target": "{{ answer|join(',')}}",
      "process_results": "def process_results(doc, results):\n    preds, golds = results, doc[\"answers\"]\n    max_em = 0\n    max_f1 = 0\n    for gold_answer in golds:\n        exact_match, f1_score = get_metrics(preds, gold_answer)\n        if gold_answer[0].strip():\n            max_em = max(max_em, exact_match)\n            max_f1 = max(max_f1, f1_score)\n    return {\"em\": max_em, \"f1\": max_f1}\n",
      "description": "",
      "target_delimiter": "",
      "fewshot_delimiter": "\n\n",
      "num_fewshot": 1,
      "metric_list": [
        {
          "metric": "em",
          "aggregation": "mean",
          "higher_is_better": true
        },
        {
          "metric": "f1",
          "aggregation": "mean",
          "higher_is_better": true
        }
      ],
      "output_type": "generate_until",
      "generation_kwargs": {
        "until": [
          "."
        ]
      },
      "repeats": 1,
      "should_decontaminate": true,
      "doc_to_decontamination_query": "{{passage}} {{question}}",
      "metadata": {
        "version": 3.0
      }
    }
  },
  "versions": {
    "drop": 3.0
  },
  "n-shot": {
    "drop": 1
  },
  "config": {
    "model": "hf",
    "model_args": "pretrained=deepseek-ai/deepseek-moe-16b-chat,dtype=bfloat16,parallelize=True,trust_remote_code=True",
    "batch_size": "8",
    "batch_sizes": [],
    "device": null,
    "use_cache": null,
    "limit": null,
    "bootstrap_iters": 100000,
    "gen_kwargs": null
  },
  "git_hash": "2dafddf",
  "transformers_version": "4.38.1",
  "upper_git_hash": null
}

lm_eval --model hf \  
    --model_args pretrained=deepseek-ai/deepseek-moe-16b-chat,dtype="bfloat16",parallelize=True \  
    --tasks gsm8k \
    --batch_size 8 \
    --num_fewshot 0 \
    --output_path deepseek-moe-16b-chat-gsm8k.json \
    --trust_remote_code

{
  "results": {
    "gsm8k": {
      "exact_match,strict-match": 0.0,
      "exact_match_stderr,strict-match": 0.0,
      "exact_match,flexible-extract": 0.33358605003790753,
      "exact_match_stderr,flexible-extract": 0.012987282131410809,
      "alias": "gsm8k"
    }
  },
  "group_subtasks": {
    "gsm8k": []
  },
  "configs": {
    "gsm8k": {
      "task": "gsm8k",
      "group": [
        "math_word_problems"
      ],
      "dataset_path": "benchmark_datasets/gsm8k",
      "dataset_name": "main",
      "training_split": "train",
      "test_split": "test",
      "fewshot_split": "train",
      "doc_to_text": "Question: {{question}}\nAnswer:",
      "doc_to_target": "{{answer}}",
      "description": "",
      "target_delimiter": " ",
      "fewshot_delimiter": "\n\n",
      "num_fewshot": 0,
      "metric_list": [
        {
          "metric": "exact_match",
          "aggregation": "mean",
          "higher_is_better": true,
          "ignore_case": true,
          "ignore_punctuation": false,
          "regexes_to_ignore": [
            ",",
            "\\$",
            "(?s).*#### ",
            "\\.$"
          ]
        }
      ],
      "output_type": "generate_until",
      "generation_kwargs": {
        "until": [
          "Question:",
          "</s>",
          "<|im_end|>"
        ],
        "do_sample": false,
        "temperature": 0.0
      },
      "repeats": 1,
      "filter_list": [
        {
          "name": "strict-match",
          "filter": [
            {
              "function": "regex",
              "regex_pattern": "#### (\\-?[0-9\\.\\,]+)"
            },
            {
              "function": "take_first"
            }
          ]
        },
        {
          "name": "flexible-extract",
          "filter": [
            {
              "function": "regex",
              "group_select": -1,
              "regex_pattern": "(-?[$0-9.,]{2,})|(-?[0-9]+)"
            },
            {
              "function": "take_first"
            }
          ]
        }
      ],
      "should_decontaminate": false,
      "metadata": {
        "version": 3.0
      }
    }
  },
  "versions": {
    "gsm8k": 3.0
  },
  "n-shot": {
    "gsm8k": 0
  },
  "config": {
    "model": "hf",
    "model_args": "pretrained=deepseek-ai/deepseek-moe-16b-chat,dtype=bfloat16,parallelize=True,trust_remote_code=True",
    "batch_size": "8",
    "batch_sizes": [],
    "device": null,
    "use_cache": null,
    "limit": null,
    "bootstrap_iters": 100000,
    "gen_kwargs": null
  },
  "git_hash": "2c017c1",
  "transformers_version": "4.38.1",
  "upper_git_hash": null
}