Are these normal results？ about diffusion-lm

[ChorlingLau]

@ChorlingLau were you able to get better results? My unconditional generation with either dataset still seems repetitive and incoherent.

I got coherent sentences in unconditional generation. And after pulling the newest code I get better results in task 'control_pos' but 'control_tree' still not.

unconditional:

["star value food , Midsummer House . Not far from All Bar One . \n END START near Caf\u00e9 Rouge , is a five star restaurant called The Golden Curry , good for families . \n END START There is a coffee shop close to the river called Fitzbillies . It is a family place . \n END START For cheap Fast food in"]
["You can have UNK a meal , but is not not recommended for kids . \n END START Fitzbillies near the Express by Holiday Inn is cheap and not family friendly with an average rating \n END START The high priced Chinese pub Wildwood holds a high customer rating \n END START The Punter is a family - friendly coffee shop that served Japanese"]
...

control_pos:

{('START', 'ADV', 'NOUN', 'ADV', 'ADP', 'PROPN', 'PROPN', 'PUNCT', 'DET', 'PROPN', 'NOUN', 'NOUN', 'VERB', 'ADJ', 'NOUN', 'NOUN', 'CCONJ', 'AUX', 'PART', 'VERB', 'NOUN', 'NOUN', 'PUNCT', 'PROPN', 'END'): [
  'the city centre area near Café Sicilia , The Cambridge customer rating has low customer ratings and is not rated coffee shop . \n END START The Eagle , is a UNK Bar with a UNK UNK UNK UNK you UNK UNK of UNK UNK UNK UNK UNK UNK will stop now of wine UNK such and UNK UNK UNK UNK UNK at a',
  'START Located in riverside near Clare Hall , a cheap coffee shop serving fast food restaurant and is not a coffee shop . Cocum has a customer rating of 5 out of 5 . \n END START The Waterman is a children friendly place in the UNK UNK UNK of over £ 30 , with a UNK 5 out of 5 , UNK a',
  'START A coffee shop near Café Sicilia , The Dumpling Tree shop serves moderate price range and is not the city centre . It is not kid friendly . \n END START The Waterman is a restaurant on are UNK UNK UNK , UNK of UNK fresh UNK UNK , while have UNK UNK UNK . They do UNK you UNK UNK UNK UNK',
  'START Moderately priced highly near Café Sicilia , The Punter coffee shop has low customer ratings and is not family - shop . \n END START The Golden Curry is a just UNK the UNK of Cambridge UNK will UNK with the UNK . your children will UNK UNK . UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK',
...

[ChorlingLau]

感谢您的回复！ 也许是因为我在训练分类器时删除了“--learn_emb yes”。 期待允许程序“学习 emb”的最新提交。(´▽｀)ノ♪

@ChorlingLau你能得到更好的结果吗？我使用任一数据集的无条件生成似乎仍然重复且不连贯。

我在无条件生成中得到了连贯的句子。在提取最新代码后，我在任务“control_pos”中得到了更好的结果，但“control_tree”仍然没有。
无条件：

["star value food , Midsummer House . Not far from All Bar One . \n END START near Caf\u00e9 Rouge , is a five star restaurant called The Golden Curry , good for families . \n END START There is a coffee shop close to the river called Fitzbillies . It is a family place . \n END START For cheap Fast food in"]
["You can have UNK a meal , but is not not recommended for kids . \n END START Fitzbillies near the Express by Holiday Inn is cheap and not family friendly with an average rating \n END START The high priced Chinese pub Wildwood holds a high customer rating \n END START The Punter is a family - friendly coffee shop that served Japanese"]
...

控制位置：

{('START', 'ADV', 'NOUN', 'ADV', 'ADP', 'PROPN', 'PROPN', 'PUNCT', 'DET', 'PROPN', 'NOUN', 'NOUN', 'VERB', 'ADJ', 'NOUN', 'NOUN', 'CCONJ', 'AUX', 'PART', 'VERB', 'NOUN', 'NOUN', 'PUNCT', 'PROPN', 'END'): [
  'the city centre area near Café Sicilia , The Cambridge customer rating has low customer ratings and is not rated coffee shop . \n END START The Eagle , is a UNK Bar with a UNK UNK UNK UNK you UNK UNK of UNK UNK UNK UNK UNK UNK will stop now of wine UNK such and UNK UNK UNK UNK UNK at a',
  'START Located in riverside near Clare Hall , a cheap coffee shop serving fast food restaurant and is not a coffee shop . Cocum has a customer rating of 5 out of 5 . \n END START The Waterman is a children friendly place in the UNK UNK UNK of over £ 30 , with a UNK 5 out of 5 , UNK a',
  'START A coffee shop near Café Sicilia , The Dumpling Tree shop serves moderate price range and is not the city centre . It is not kid friendly . \n END START The Waterman is a restaurant on are UNK UNK UNK , UNK of UNK fresh UNK UNK , while have UNK UNK UNK . They do UNK you UNK UNK UNK UNK',
  'START Moderately priced highly near Café Sicilia , The Punter coffee shop has low customer ratings and is not family - shop . \n END START The Golden Curry is a just UNK the UNK of Cambridge UNK will UNK with the UNK . your children will UNK UNK . UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK',
...

I want to know your detailed commands in the last step of running scripts/infill.py and how to generate unconditionally. Thank you very much

# unconditional
scripts/batch_decode.py [diffusion model path] -1.0 ema

# controlled
scripts/infill.py --model_path [path]/model200000.pt --eval_task_ 'control_pos' --use_ddim True --notes "pos_adagrad" --eta 1. --verbose pipe

[XiangLi1999]

No... these are not normal results...

Some tips for debugging:
when you generate unconditionally, does it look fluent? If it's not fluent, it's probably the Diffusion-LM is not trained well.
run the classifier/parser/tagger on some normal sentence, if it seems obviously wrong, then it's the classifier/parser/tagger's problem.

[XiangLi1999]

actually, this reminds me of #21 -- might be worth checking.

[ChorlingLau]

Thanks for your reply!
Perhaps it's because I removed "--learn_emb yes" when training classifier.
Looking forward to the latest commit which allows the program to 'learn emb'. (´▽｀)ノ♪

[a-antoniades]

@ChorlingLau were you able to get better results? My unconditional generation with either dataset still seems repetitive and incoherent.

[a-antoniades]

@ChorlingLau hmm, to me that seems like "barely coherent" - but admittedly better than what I'm getting. This is using the commands in the readme for training and decoding right?

@XiangLi1999 any thoughts on the quality of the outputs we've been getting? Seems like it's way off from your results, what could we be doing wrong? Have you used this actual codebase/data to train a model yourself? If so, what are your results/hyperparams? Thanks a lot.

[a-antoniades]

update: i was able to train a much better model that generates nice text, using much larger batch sizes (640).

["area . It is friendly for families . \n END START The Rice Boat serves English food near Express by Holiday Inn in city centre it is family - friendly with a high rating . \n END START There is a restaurant called Aromi that serves Chinese food and is located in the city centre . \n END START For a kid friendly Chinese"]
["Waterman located near Crowne Plaza Hotel . It is not a family friendly restaurant . \n END START Fitzbillies is near Express by Holiday Inn . It is not family - friendly and has a more than \u00a3 30 price range and a 5 out of 5 customer rating \n END START The Punter is a family - friendly coffee shop offering Italian food"]
["riverside . \n END START Fitzbillies is a best cheap Indian restaurant near The Six Bells . \n END START The Waterman is a non family - friendly fast food restaurant with a low customer rating and price range less than 20 euros and is in the city centre . \n END START There is a coffee shop called Fitzbillies located in the city"]

[lcy5058]

感谢您的回复！ 也许是因为我在训练分类器时删除了“--learn_emb yes”。 期待允许程序“学习 emb”的最新提交。(´▽｀)ノ♪

@ChorlingLau你能得到更好的结果吗？我使用任一数据集的无条件生成似乎仍然重复且不连贯。

我在无条件生成中得到了连贯的句子。在提取最新代码后，我在任务“control_pos”中得到了更好的结果，但“control_tree”仍然没有。

无条件：

["star value food , Midsummer House . Not far from All Bar One . \n END START near Caf\u00e9 Rouge , is a five star restaurant called The Golden Curry , good for families . \n END START There is a coffee shop close to the river called Fitzbillies . It is a family place . \n END START For cheap Fast food in"]
["You can have UNK a meal , but is not not recommended for kids . \n END START Fitzbillies near the Express by Holiday Inn is cheap and not family friendly with an average rating \n END START The high priced Chinese pub Wildwood holds a high customer rating \n END START The Punter is a family - friendly coffee shop that served Japanese"]
...

控制位置：

{('START', 'ADV', 'NOUN', 'ADV', 'ADP', 'PROPN', 'PROPN', 'PUNCT', 'DET', 'PROPN', 'NOUN', 'NOUN', 'VERB', 'ADJ', 'NOUN', 'NOUN', 'CCONJ', 'AUX', 'PART', 'VERB', 'NOUN', 'NOUN', 'PUNCT', 'PROPN', 'END'): [
  'the city centre area near Café Sicilia , The Cambridge customer rating has low customer ratings and is not rated coffee shop . \n END START The Eagle , is a UNK Bar with a UNK UNK UNK UNK you UNK UNK of UNK UNK UNK UNK UNK UNK will stop now of wine UNK such and UNK UNK UNK UNK UNK at a',
  'START Located in riverside near Clare Hall , a cheap coffee shop serving fast food restaurant and is not a coffee shop . Cocum has a customer rating of 5 out of 5 . \n END START The Waterman is a children friendly place in the UNK UNK UNK of over £ 30 , with a UNK 5 out of 5 , UNK a',
  'START A coffee shop near Café Sicilia , The Dumpling Tree shop serves moderate price range and is not the city centre . It is not kid friendly . \n END START The Waterman is a restaurant on are UNK UNK UNK , UNK of UNK fresh UNK UNK , while have UNK UNK UNK . They do UNK you UNK UNK UNK UNK',
  'START Moderately priced highly near Café Sicilia , The Punter coffee shop has low customer ratings and is not family - shop . \n END START The Golden Curry is a just UNK the UNK of Cambridge UNK will UNK with the UNK . your children will UNK UNK . UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK',
...

I want to know your detailed commands in the last step of running scripts/infill.py and how to generate unconditionally. Thank you very much

[XiangLi1999]

I think one important detail is to train a PAD based model via "--padding_mode pad ". Essentially, this makes a sentence start from [START] [sentence content] [END] [PAD*n]. Training by block looks fine for unconditional generation, but for controls, I think there are too much noise when learning the classifier.

[lcy5058]

感谢您的回复！ 也许是因为我在训练分类器时删除了“--learn_emb yes”。 期待允许程序“学习 emb”的最新提交。(´▽｀)ノ♪

@ChorlingLau你能得到更好的结果吗？我使用任一数据集的无条件生成似乎仍然重复且不连贯。

我在无条件生成中得到了连贯的句子。在提取最新代码后，我在任务“control_pos”中得到了更好的结果，但“control_tree”仍然没有。
无条件：

["star value food , Midsummer House . Not far from All Bar One . \n END START near Caf\u00e9 Rouge , is a five star restaurant called The Golden Curry , good for families . \n END START There is a coffee shop close to the river called Fitzbillies . It is a family place . \n END START For cheap Fast food in"]
["You can have UNK a meal , but is not not recommended for kids . \n END START Fitzbillies near the Express by Holiday Inn is cheap and not family friendly with an average rating \n END START The high priced Chinese pub Wildwood holds a high customer rating \n END START The Punter is a family - friendly coffee shop that served Japanese"]
...

控制位置：

{('START', 'ADV', 'NOUN', 'ADV', 'ADP', 'PROPN', 'PROPN', 'PUNCT', 'DET', 'PROPN', 'NOUN', 'NOUN', 'VERB', 'ADJ', 'NOUN', 'NOUN', 'CCONJ', 'AUX', 'PART', 'VERB', 'NOUN', 'NOUN', 'PUNCT', 'PROPN', 'END'): [
  'the city centre area near Café Sicilia , The Cambridge customer rating has low customer ratings and is not rated coffee shop . \n END START The Eagle , is a UNK Bar with a UNK UNK UNK UNK you UNK UNK of UNK UNK UNK UNK UNK UNK will stop now of wine UNK such and UNK UNK UNK UNK UNK at a',
  'START Located in riverside near Clare Hall , a cheap coffee shop serving fast food restaurant and is not a coffee shop . Cocum has a customer rating of 5 out of 5 . \n END START The Waterman is a children friendly place in the UNK UNK UNK of over £ 30 , with a UNK 5 out of 5 , UNK a',
  'START A coffee shop near Café Sicilia , The Dumpling Tree shop serves moderate price range and is not the city centre . It is not kid friendly . \n END START The Waterman is a restaurant on are UNK UNK UNK , UNK of UNK fresh UNK UNK , while have UNK UNK UNK . They do UNK you UNK UNK UNK UNK',
  'START Moderately priced highly near Café Sicilia , The Punter coffee shop has low customer ratings and is not family - shop . \n END START The Golden Curry is a just UNK the UNK of Cambridge UNK will UNK with the UNK . your children will UNK UNK . UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK UNK',
...

I want to know your detailed commands in the last step of running scripts/infill.py and how to generate unconditionally. Thank you very much

# unconditional
scripts/batch_decode.py [diffusion model path] -1.0 ema

# controlled
scripts/infill.py --model_path [path]/model200000.pt --eval_task_ 'control_pos' --use_ddim True --notes "pos_adagrad" --eta 1. --verbose pipe

Thanks,When finally generating controllable text, python scripts/infill.py --model_path {path-to-diffusion-lm} command, which path does "path-to-diffusion-lm" refer to, the first step model path, or the last but one model generated from the classifier?
"improved diffusion/diffusion_models/diff_e2e-tgt_block_rand16_transformer_lr0.0001_0.0_2000_sqrt_Lsimple_h128_s2_d0.1_sd102_xstart_e2e" or "Classifier_models/e2e-tgt-tree_e=6_b=10_m=bert base uncased_wikitext-103-raw-v1_101_wp_None"

[ChorlingLau]

@Licy1999 It's 1st step model. The classifier path should be modified in code file infill.py

[lcy5058]

@Licy1999 It's 1st step model. The classifier path should be modified in code file infill.py

hello,I am filling as you said infill.py changes the training path of the classifier model, code:model_control = Classifier_POS.from_pretrained('../classifier_models/e2e-tgt-tree_e=6_b=10_m=bert-base-uncased_wikitext-103-raw-v1_101_wp_None').cuda(),and I confirm the file of oil training classifier in this folder.but it seems that the loading fails. An error like this is reported. Can you tell me how to handle it.
load the partial sequences
None
Traceback (most recent call last):
File "scripts/infill.py", line 764, in
args = main()
File "scripts/infill.py", line 241, in main
model_control = Classifier_POS.from_pretrained('../classifier_models/e2e-tgt-tree_e=6_b=10_m=bert-base-uncased_wikitext-103-raw-v1_101_wp_None').cuda()
File "/tmp/pycharm_project_305/transformers/src/transformers/modeling_utils.py", line 1496, in from_pretrained
model = cls(config, *model_args, **model_kwargs)
File "/tmp/pycharm_project_25/improved-diffusion/../transformers/examples/pytorch/language-modeling/custom_trainer.py", line 1444, in init
self.lm_head2 = nn.Linear(config.hidden_size, config.pos_vocab_size, bias=False)
File "/tmp/pycharm_project_305/transformers/src/transformers/configuration_utils.py", line 252, in getattribute
return super().getattribute(key)
AttributeError: 'BertConfig' object has no attribute 'pos_vocab_size'