Comments (3)
I replaced the PAD token from 0 to '0' and solved this issue. Due to the resourse restricted, I replaced the basemodel to Llama2-13B to train and evaluate the E4SRec, but the gained result is very poor.
The finetune script is
torchrun --nproc_per_node=8 --master_port=1907 finetune.py \
--base_model /data06/Llama-2-13b-hf \
--data_path Beauty \
--task_type sequential \
--output_dir ./LLM4Rec \
--batch_size 16 \
--micro_batch_size 1 \
--num_epochs 1 \
--learning_rate 0.0003 \
--cutoff_len 4096 \
--val_set_size 0 \
--lora_r 16 \
--lora_alpha 16 \
--lora_dropout 0.05 \
--lora_target_modules '[gate_proj, down_proj, up_proj]' \
--train_on_inputs False \
--add_eos_token False \
--group_by_length False \
--prompt_template_name alpaca \
--lr_scheduler 'cosine' \
--warmup_steps 100 \
--wandb_project E4SRec
The infer script is
torchrun --nproc_per_node=8 --master_port=1234 inference.py \
--base_model /data06/Llama-2-13b-hf \
--data_path Beauty \
--task_type sequential \
--checkpoint_dir ./LLM4Rec/ \
--cache_dir cache_dir/ \
--output_dir ./LLM4Rec/ \
--batch_size 16 \
--micro_batch_size 1
The reported results is that
Evaluation for User:
Precision@1: 8.9433439162903e-05
Recall@1: 8.9433439162903e-05
MRR@1: 8.9433439162903e-05
MAP@1: 8.9433439162903e-05
NDCG@1: 8.9433439162903e-05
Precision@5: 8.9433439162903e-05
Recall@5: 0.00044716719581451504
MRR@5: 0.0001974988448180775
MAP@5: 0.0001974988448180775
NDCG@5: 0.0002586554819987668
Precision@10: 8.943343916290303e-05
Recall@10: 0.0008943343916290301
MRR@10: 0.00025114116358582666
MAP@10: 0.00025114116358582666
NDCG@10: 0.0003971278706389283
Precision@20: 7.825425926754018e-05
Recall@20: 0.0015650851853508028
MRR@20: 0.0002965062025846168
MAP@20: 0.0002965062025846168
NDCG@20: 0.0005650891080215324
Precision@100: 7.154675133032246e-05
Recall@100: 0.007154675133032241
MRR@100: 0.00040016219874475093
MAP@100: 0.00040016219874475093
NDCG@100: 0.0015212688144377391
from e4srec.
The similar issues here.I would like to ask if there are any issues with the evaluation? There are too many decimals,like 0.000.
from e4srec.
If you want to use a smaller base model, please use garage-bAInd/Platypus2-13B or garage-bAInd/Platypus2-7B instead.
Original LLaMA2 is not instruction-tuned and can not get desirable results.
from e4srec.
Related Issues (13)
- The performance gap compared to P5 HOT 1
- Acceptance of WWW HOT 1
- baseline负样本数量 HOT 4
- ValueError: Target modules [gate_proj, down_proj, up_proj] not found in the base model. Please check the target modules and try again. HOT 2
- Hyperparameters on Yelp HOT 1
- Details of experiments and Reproducibility issue
- user embedding HOT 1
- RuntimeError: expected mat1 and mat2 to have the same dtype, but got: c10::Half != float HOT 3
- files missed HOT 1
- \ ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: =====\
- 数据集问题请教
- 关于evaluate data,想咨询一下您,是使用什么数据集进行evaluate的?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from e4srec.