Use all the CPU available on a single node for `generate_statistics_from_tfrecord` about data-validation

[singhniraj08]

@zippeurfou,

If you can load the whole data into memory, you can use tfdv.generate_statistics_from_dataframe with n_jobs argument set to -1 to utilize all the CPU cores.

Can you share us an example graphs where it ignores the subset of data? I am unable to replicate the issue with below code. Also, make sure you are using Apache beam >= 2.22.0 as direct_num_workers = 0 support was introduced in 2.22.0.

stats = tfdv.generate_statistics_from_tfrecord(data_location=train_tf_file,
                                              #  stats_options=tfdv.StatsOptions(enable_semantic_domain_stats=True),
                                               pipeline_options=PipelineOptions(['--runner','DirectRunner','--direct_num_workers', '0', '--direct_running_mode', 'multi_processing']),
                                               output_path=new_stats_location,
                                               )

Thank you!

[zippeurfou]

Thank you for the reply.
I would like to avoid using the data frame one.
I will share an example when I can.
I think the main difference most likely is that I am working with a folders that has multiple tfrecord files compressed.

[singhniraj08]

@zippeurfou,

generate_statistics_from_tfrecord by default will pick multiple tfrecord files from given data folder(location). Make sure the given path have all the tfrecord files and avoid sub folders. Please share us the example code to replicate this issue on our end. Thanks.

[singhniraj08]

Closing this due to inactivity. Please take a look into the answers provided above, feel free to reopen and post your comments(if you still have queries on this). Thank you!