Training Failed about arcee-python

[Jacobsolawetz]

Hello @beddows! Great to hear from you. We have discovered a non-deterministic bug in our upload flow that is causing this. We will keep you posted on the fix

[Jacobsolawetz]

@beddows confirming here that this is an issue with our upload APIs getting overwhelmed. Will keep you posted

[beddows]

Ok, thanks for letting me know @Jacobsolawetz

[Jacobsolawetz]

@beddows we have healed these APIs - wondering, about how large is the context you are uploading?

You should have a green light to continue below 500K or so. We were getting some large uploads last week which overwhelmed upload infra

[beddows]

Thanks @Jacobsolawetz , will check it out! The context file I uploaded was no more than 100k with under 100 rows.

[Jacobsolawetz]

@beddows let me know! And for 100K wait perhaps 10-20min after upload for it to drain through before train kickoff - we'll have better fully ACK'd upload status soon