error running example.py about google-research

[uricohen]

It seems that tensor2tensor's problem.dataset() has changed and now require the second argument data_dir not to be left empty.

I could solve the problem by changing demogen/data_util.py as follows:

  def input_data():
    """Input function to be returned."""
    prob = problems.problem(problem_name)
    data_dir = '<<<MY_HOME_DIRECTORY>>>/t2t_data/';
    if data == 'image_cifar100':
      dataset = prob.dataset(mode, data_dir, preprocess=augmented)
      if not augmented: dataset = dataset.map(map_func=standardization)
    else:
      dataset = prob.dataset(mode, data_dir)
      dataset = dataset.map(map_func=standardization)

and I have previously downloaded the data into this directory using:

  t2t-datagen --generate_data --data_dir=~/t2t_data --problem=image_cifar100
  t2t-datagen --generate_data --data_dir=~/t2t_data --problem=image_cifar10

[uricohen]

Same for me!

Perhaps this is a setup issue of tensorflow-datasets? I didn't configure it to download CIFAR, etc.

[yidingjiang]

Thank you for working this out. I am working on potentially easier way to get the dataset for external users.

[uricohen]

Thanks, @yidinggiang! But are you sure the data split into training/validation/test set is determenistic?
That is, that the CIFAR10/100 training set I got is the same as the one used to train the models?

[yidingjiang]

CIFAR10 and 100 have fixed training and test sets, so I don't see any problems. (There have been some changes to the data pipeline, please pull to get them)