Performance issues in /model/train.py (by P3) about cam2bev

[lreiher]

Hi, have you actually tried this and noticed a performance difference? If so, feel free to propose a PR with some information on what speed-up this gives on your system.

[lreiher]

I have tried to swap the order of the pipeline calls, i.e. batch first and then map, by adjusting the mapped parse_sample function like so:

# build dataset pipeline parsing functions
def parse_sample(batched_input_files, batched_label_file):
    # parse and process input images

    # define unbatched parsing operations
    def parse_input(i):
        i = utils.load_image_op(i)
        i = utils.resize_image_op(i, image_shape_original_input, conf.image_shape, interpolation=tf.image.ResizeMethod.NEAREST_NEIGHBOR)
        i = utils.one_hot_encode_image_op(i, conf.one_hot_palette_input)
        return i
    def parse_label(l):
        l = utils.load_image_op(l)
        l = utils.resize_image_op(l, image_shape_original_label, conf.image_shape, interpolation=tf.image.ResizeMethod.NEAREST_NEIGHBOR)
        l = utils.one_hot_encode_image_op(l, conf.one_hot_palette_label)
        return l

    # parse and process input images
    inputs = []
    for inp in batched_input_files:
        inputs.append(tf.map_fn(parse_input, inp, fn_output_signature=tf.float32))
    inputs = inputs[0] if n_inputs == 1 else tuple(inputs)

    # parse and process label image
    label = tf.map_fn(parse_label, batched_label_file, fn_output_signature=tf.float32)

    return inputs, label

I did not find a way to vectorize the parsing across the entire batch. Vectorization is why the linked TensorFlow doc recommends batch first, then map. Without vectorization, this version actually had a worse runtime than before. My runtimes are included for reference:

• config.1_FRLR.unetxst.yml
  • max-samples-training: 1000
  • max-samples-validation: 100
  • epochs: 20
  • save_interval: 10
• GPU: RTX 3090
• before: 16m 40s
• after: 18m 30s

@bastilam: I propose to close the issue. MRs with documented runtime improvements are still happily accepted.