boto3 credentials error when running CCSparkJob with ~100 S3 warc paths as input, but works with <10 S3 warc paths as input about cc-pyspark

[sebastian-nagel]

Hi @praveenr019, given the error message "botocore.exceptions.NoCredentialsError: Unable to locate credentials": is the job run on a Spark cluster or on a single instance? If on a cluster: how are the credentials deployed to the cluster instances (eg. via IAM roles)?

• see https://github.com/commoncrawl/cc-pyspark#authenticated-s3-access-or-access-via-http
• if not running on AWS: use --input_base_url https://data.commoncrawl.org/

[sebastian-nagel]

If on a single instance: I haven't seen a credential error just because of processing more data. How are the credentials configured?

[praveenr019]

Thanks for the reply @sebastian-nagel. Yes, the job is run on a Spark cluster in AWS and the credentials are setup using IAM roles.

[sebastian-nagel]

No glue what could be the reason. And never seen this.

My assumption is that in cluster mode, every Python runner is a separate process. This would exclude any concurrency issues while fetching the credentials (for example here).

To address the problem, I'd catch the NoCredentialsError along the ClientError (sparkcc.py, line 283), log the error, re-instantiate the S3 client and try the download a second time. Let me know if you need help to implement this. Otherwise, would be interesting to hear whether this solves the problem.

[sebastian-nagel]

Closing for now. @praveenr019 let me know if this is still an issue!