Deliver action keep in Waiting for all the screenshots to finish being processed forever about fastlane

[thisisthekap]

@ralfebert Are you considering creating a PR for your fix?

[mikhailmaslo]

@vhartikainen @efraespada @assembleMHN @jjochen @vanniktech @rocketnik @AlkanV

I've made a PR that addresses the issue with screenshots uploading #21693.

Could you check it with your setup and give the feedback? I would be grateful.

You can adjust your Gemfile in this way:

gem "fastlane", git: "https://github.com/mikhailmaslo/fastlane",  branch: "feature/introduce-timeout-for-screenshots-upload"

Then, you can use deliver to upload screenshots

deliver(
	api_key: api_key,
	skip_binary_upload: true,
	skip_screenshots: false,
	skip_metadata: true,
	skip_app_version_update: true,
	force: true,
	overwrite_screenshots: true,
	run_precheck_before_submit: false,
	screenshot_processing_timeout: 120
)

Note the screenshot_processing_timeout parameter, which defines the timeout for screenshots processing awaiting. In this example, it's set to 2 minutes. All unprocessed screenshots are deleted from App Store Connect and then re-uploaded again with the same timeout of screenshot_processing_timeout

In my experience, one re-uploading loop is sufficient to upload all screenshots. I've tested with ~200 screenshots.

[Ido-Levi]

Same issue here, any workaround for the meantime?

[EmDee]

Today has been such a day again. Our runners have been stuck the last 5h waiting for the processing to finish.

[daidongon]

I tried again just now and it seems to have worked with same script !

[fhperuchi]

We were already using the 2.209.1 fastlane version, and we did not changed anything on our fastlane scripts. It started to work again today for us.

$ fastlane --version
fastlane installation at path:
/Users/fhperuchi/.rvm/gems/ruby-3.1.2/gems/fastlane-2.209.1/bin/fastlane
-----------------------------
[✔] 🚀 
fastlane 2.209.1

[ralfebert]

This change seems to do the trick of not waiting endlessly for the uploads to complete – all the logic to re-upload failed screenshots is already there:
ralfebert@6168384

[assembleMHN]

this has never gone away for me, and still happnes. Im looking at one terminal right now "waiting" for the screenshots to finish being processed... spoiler it will never finish

[jjochen]

I tested it, but in my first couple of tries, the upload didn't get stuck, but worked on the first try 🙄
Will try some more.

[hakburi]

Same issue.

[⠋] Waiting for all the screenshots to finish being processed... DEBUG [2022-09-06 01:34:51.31]: There are still incomplete screenshots - {"COMPLETE"=>2, "UPLOAD_COMPLETE"=>4}

• Fastfile

    upload_to_app_store(
      force: true,
      overwrite_screenshots: true,
      skip_binary_upload: true,
    )

[sonjz]

is a reoccurring issue with apple i've noticed over at least the past 4 years.
my guess has been, apple rolling in server changes to the fleet, and some get caught with the old code and can't resolve. i recall it normally taking a day up to a week for the issue to disappear on its own.

[Westacular]

This does seem to be an Apple issue, when I logged into App Store Connect 10 hours later, some of the screenshoots were still showing a blank placeholder with the message "This file has been uploaded and is processing."

I can remove the placeholders and manually upload the missing screenshots directly in App Store Connect, and that doesn't seem to incur this processing delay. But after doing so, when I try to submit to app review, it fails with an error message saying "There are still screenshot uploads in progress."

Ugh.

[jakub-przy]

It seems to be working again for us as well! Oh Apple 🍏

[sonjz]

@saggio89 i received the same but think it cleared up last night.
i was able to save it as Free and rerelease without issue.

[ppamorim]

Still happening for me. I believe fastlane should offer a timeout option to not keep the process stuck.

[sonjz]

Still happening for me. I believe fastlane should offer a timeout option to not keep the process stuck.

just use a timeout from bash or your scripting language and wrap it.
bash has timeout and ruby has Timeout

check the error it throws and respond appropriately...

[EmDee]

1 Question for those who faced this error.

If I am stuck for 2 hours in this step, Is it better to keep the process running, or restart the process?

I mean was the issue fixed for anyone just by restarting the fastlane process for the same app? Or we should just keep waiting?

Waiting never worked for us. When you go to ASC you can see that the resources are just stuck in the processing loop forever.

The only solution for us was either manually uploading the screenshots with ASC's media manager or restarting the job.

[ralfebert]

Has anyone a workaround for this issue?

Here is my routine for uploading screenshots which works but is still painful:

I always upload all screenshots (overwrite_screenshots: true).

Most of the time this will result in an endless loop like this:

[⠇] Waiting for all the screenshots to finish being processed... DEBUG [2023-05-02 16:35:00.45]:
     There are still incomplete screenshots - {"COMPLETE"=>538, "UPLOAD_COMPLETE"=>6, "FAILED"=>15}
[⠇] Waiting for all the screenshots to finish being processed... DEBUG [2023-05-02 16:35:00.45]:
    There are still incomplete screenshots - {"COMPLETE"=>538, "UPLOAD_COMPLETE"=>6, "FAILED"=>15}
[⠇] Waiting for all the screenshots to finish being processed... DEBUG [2023-05-02 16:35:00.45]:
    There are still incomplete screenshots - {"COMPLETE"=>538, "UPLOAD_COMPLETE"=>6, "FAILED"=>15}

The FAILED ones are not a problem, fastlane will re-upload them (in App Store connect those will have a red exclamation mark).

But the UPLOAD_COMPLETE ones are just stuck. So what I do is I go to App Store Connect, click through all the locales and find those (6 in this case) that are stuck (they will have a gray cloud symbol). I delete those. Then fastlane will continue and upload the missing and failed ones. It might get stuck a second time, same procedure.

I tried using the new sync_screenshots, this worked the first time but this failed to do the update the second time...

It seems the screenshot upload needs a timeout for the images in state UPLOAD_COMPLETE...

[rocketnik]

We are uploading 100 images at the moment and usually around 5 keep hanging in "processing" state on the AppStore Connect page. Cancelling fastlane deliver and then uploading the missing ones by hand is one option.

Another option for automatic uploads for me was limiting the upload bandwidth to 2MBit using the Network Link Conditioner preferences pane. The upload takes long, but it seems, that Apple's server can keep up with processing the images - just a thought, no evidence.

For fastlane maybe it would help to not upload the images in parallel. It saves more time, if the upload goes through after 1 minutes instead of 30 seconds, than if we spend 10 minutes figuring out, why the upload failed and searching for alternatives.

[vanniktech]

The best 'solution', I have is to go to the Media Manager, here's an url for my app: https://appstoreconnect.apple.com/apps/6453164382/appstore/ios/version/inflight/media-manager (simply swap the app id and it should work for you). Force refresh the site without a cache and go through each language and then you can find the unprocessed screenshots, remove them yourself and reupload the screenshots yourself. Fastlane is able to pick it up again:

[19:31:27]: Uploaded './fastlane/screenshots/cs/iPhone 8 Plus-06.png'... (3.93246 secs)
[✔] Waiting for all the screenshots to finish being processed...
[19:47:41]: ./fastlane/screenshots/sv/iPhone 14 Pro Max-02.png is missing on App Store Connect.

As you can see this took me more than 15 minutes, since I have 3 different sizes with 25 languages.

In the meantime - fastlane must know this - would it be possible to print out which exact screenshots (file path is suffice) are missing?

[jjochen]

Lately this is happening so much again.

[fastlane-bot]

It seems like you have not included the output of fastlane env
To make it easier for us help you resolve this issue, please update the issue to include the output of fastlane env 👍

[OmniaMahmoud]

It seems like you have not included the output of fastlane env To make it easier for us help you resolve this issue, please update the issue to include the output of fastlane env 👍

done

[mjovkovic]

Same here

[OmniaMahmoud]

there is a part of my images uploaded and appear on app store and other not appear with this message
"this file has been uploaded and is processing"

[markmuniz]

We are having the same issue.

[henry-wbs]

Fastlane continued for me after cancelling the images with "this file has been uploaded and is processing" and uploading them manually.

[jmif]

Having the same issue, started occurring ~4 days ago. Manual upload of same images also works.

[manumax]

Same issue. Not sure if it is related and may help, but if I set the option overwrite_screenshots to true, fastlane fails with the error Failed verification of all screenshots deleted....

[development-incapptic]

We are also facing the same issue
There are still incomplete screenshots - {"UPLOAD_COMPLETE"=>5, "COMPLETE"=>3}

[fhperuchi]

I am having the same issue.

[calinraducalin]

I am facing the same issue

[Thangaraj-M]

facing the same issue

[nachocodoner]

Recently we are facing the exact same issue.

[muehlemann]

Facing the same issue

[samueldusek]

I am facing the same issue. Is there any workaround? 🙂

[galacticdisassembler]

We're facing the same issue for quite some time already.
Is there some kind of a workaround?

[TalRandi1]

Same issue here

[development-incapptic]

Hi Team, Do you have any workaround or can you please let us know about any ETA?

[jakub-przy]

Experiencing the same issue (as per my comment in: #20625)

[michaelforrest]

Happening to me too. Would love a workaround or ETA for a fix...

[BDDev]

We noticed our builds getting stuck on "Waiting for screenshots to finish uploading" September 1st. I suspect the same or similar as #20619 (comment) and hope it is an issue on Apple's side that will resolve any day now.

(OP mentioned app_store_connect_api_key, we can rule that out because we use FASTLANE_SESSION)

[daidongon]

Facing the same issue too

deliver(
    submit_for_review: false,
    automatic_release: false,
    force: true,
    skip_binary_upload: true,
    skip_metadata: false,
    skip_screenshots: false,
    overwrite_screenshots: true,
    precheck_include_in_app_purchases: false,
)

[Ido-Levi]

for now we have just skipped uploading screenshots using fastlane and doing it manually which sucks, but couldn't find a different way

[OmniaMahmoud]

Yes working
Thank you all

[murataksuu]

not working for me

[sonjz]

hit and miss, must be still rolling out slowly.
(1 for 3 today)

noticed the issue is resolved, i'm hitting next issue with new app, the pricing can't be set (getting 500 issue on appstoreconnect GUI too)

[13:34:05]: Successfully uploaded screenshots to App Store Connect
[13:34:05]: App has no prices yet... Enabling all countries in App Store Connect


[!] The request could not be completed because:

        Server error got 500

[sonjz]

noticed that i hadn't updated to fastlane 2.209.1 yet, retesting, the issue had passed the screenshot processing.

[saggio89]

@sonjz I think this is a problem with Apple -- I manually uploaded the build via xcode & when I try to update the pricing on AppStore Connect, it throws an error

[saggio89]

@sonjz same! I submitted an issue to Apple yesterday, looks like they fixed it since then. Happy coding!

[samueldusek]

I tried it again today and everything seems to be fine. I was finally able to realease the app using the same fastlane script as always. Looks like apple has solved the issue. 🙂

[markmuniz]

Is this happening again for anyone else?

[samueldusek]

@markmuniz

Is this happening again for anyone else?

No. Everything seems to be working fine. 👍

[jjochen]

This is still happening for me...

[jjochen]

This is still happening for me...

Worked after deleting all existing screenshots from App Store Connect.

[skarol]

Is this happening again for anyone else?

After some time when working fine, started appearing again today.

[Westacular]

I've seen a couple random recurrences of this. It definitely seems like there's something flaky in whatever queue Apple's using to process these uploads? Not sure there's much Fastlane can do to work around that.

Our CI process for submitting builds is granular enough that we have a specific job just for uploading screenshots. The times we've experienced this issue in the last couple of weeks, retrying just that job has always succeeded, so ... at least the problem isn't as big or consistent as it was when this all started.

[EmDee]

Just happened on our end again.

[ralfebert]

Same here, it seems to be stuck in a loop with the screenshots never finishing:

[⠹] Waiting for all the screenshots to finish being processed... DEBUG [2022-10-07 14:29:45.85]: There are still incomplete screenshots - {"COMPLETE"=>439, "FAILED"=>17, "UPLOAD_COMPLETE"=>36}
[⠸] Waiting for all the screenshots to finish being processed... DEBUG [2022-10-07 14:29:52.12]: There are still incomplete screenshots - {"COMPLETE"=>439, "FAILED"=>17, "UPLOAD_COMPLETE"=>36}
[⠹] Waiting for all the screenshots to finish being processed... DEBUG [2022-10-07 14:29:58.31]: There are still incomplete screenshots - {"COMPLETE"=>439, "FAILED"=>17, "UPLOAD_COMPLETE"=>36}

[jakub-przy]

It still happens to my team from time to time, but just singular instances - nothing similar to that massive outage (which lasted for over 2 weeks) in September 🤞

[jakub-przy]

In regards to the timeout ⬆️ I recommend doing this: https://support.circleci.com/hc/en-us/articles/4411204604059-Automatically-cancel-build-after-set-amount-of-time

This is a CircleCI specific doc, but this should be doable elsewhere.

[sonjz]

In regards to the timeout ⬆️ I recommend doing this: https://support.circleci.com/hc/en-us/articles/4411204604059-Automatically-cancel-build-after-set-amount-of-time

This is a CircleCI specific doc, but this should be doable elsewhere.

to each their own, but if i already knew my timeout time, i would rather wrap the process in a timeout vs creating another job to poll it and send the kill signal.

[mrazam110]

Its still happening :(

[EmDee]

Apple seems to have trouble in regard to the screenshot processing lately.

What we did is separate the deliver job for screenshots from the submission one. This helped tremendously as we can just retry the screenshot upload as often as we need to.

[RaymondGuirguis]

1 Question for those who faced this error.
If I am stuck for 2 hours in this step, Is it better to keep the process running, or restart the process?
I mean was the issue fixed for anyone just by restarting the fastlane process for the same app? Or we should just keep waiting?

[mregnauld]

I experienced the same problem today.
Relaunching the same lane without changing anything worked out for me.

[bvanderplanck]

We are facing the issue as I'm writing this comment, am I the only one having this issue at the moment ?

[jjochen]

No, also happened to us recently.

[AlkanV]

hello, any news on this? i am facing same issue.

[efraespada]

Stills happening, with this error at the end:

[13:40:22]: Uploaded './fastlane/screenshots/tr/2_APP_IPHONE_65_2.png'... (3.193515 secs)
[⠏] Waiting for all the screenshots to finish being processed... #<Thread:0x00007fb8702b19e0@/Library/Ruby/Gems/2.6.0/gems/fastlane-2.216.0/deliver/lib/deliver/app_screenshot_iterator.rb:24 run> terminated with exception (report_on_exception is true):
/Library/Ruby/Gems/2.6.0/gems/fastlane-2.216.0/spaceship/lib/spaceship/connect_api/models/app_store_version_localization.rb:103:in `rescue in get_app_screenshot_sets': it (Spaceship::AppStoreScreenshotError)
        from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.216.0/spaceship/lib/spaceship/connect_api/models/app_store_version_localization.rb:99:in `get_app_screenshot_sets'
        from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.216.0/deliver/lib/deliver/app_screenshot_iterator.rb:25:in `block (3 levels) in each_app_screenshot_set'
/Library/Ruby/Gems/2.6.0/gems/fastlane-2.216.0/spaceship/lib/spaceship/connect_api/api_client.rb:188:in `rescue in with_asc_retry': undefined method `refresh!' for nil:NilClass (NoMethodError)
        from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.216.0/spaceship/lib/spaceship/connect_api/api_client.rb:165:in `with_asc_retry'
        from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.216.0/spaceship/lib/spaceship/connect_api/api_client.rb:105:in `get'
        from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.216.0/spaceship/lib/spaceship/connect_api/tunes/tunes.rb:493:in `get_app_screenshot_sets'
        from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/forwardable.rb:230:in `get_app_screenshot_sets'
        from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.216.0/spaceship/lib/spaceship/connect_api/models/app_screenshot_set.rb:129:in `all'
        from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.216.0/spaceship/lib/spaceship/connect_api/models/app_store_version_localization.rb:101:in `get_app_screenshot_sets'
        from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.216.0/deliver/lib/deliver/app_screenshot_iterator.rb:25:in `block (3 levels) in each_app_screenshot_set'
/Library/Ruby/Gems/2.6.0/gems/fastlane-2.216.0/spaceship/lib/spaceship/connect_api/api_client.rb:244:in `handle_error': Authentication credentials are missing or invalid. - No valid credentials found in the request. (Spaceship::UnauthorizedAccessError)
        from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.216.0/spaceship/lib/spaceship/client.rb:932:in `block in send_request'
        from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.216.0/spaceship/lib/spaceship/client.rb:687:in `with_retry'
        from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.216.0/spaceship/lib/spaceship/client.rb:928:in `send_request'
        from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.216.0/spaceship/lib/spaceship/client.rb:772:in `request'
        from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.216.0/spaceship/lib/spaceship/connect_api/api_client.rb:106:in `block in get'
        from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.216.0/spaceship/lib/spaceship/connect_api/api_client.rb:166:in `with_asc_retry'
        from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.216.0/spaceship/lib/spaceship/connect_api/api_client.rb:105:in `get'
        from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.216.0/spaceship/lib/spaceship/connect_api/tunes/tunes.rb:493:in `get_app_screenshot_sets'
        from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/forwardable.rb:230:in `get_app_screenshot_sets'
        from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.216.0/spaceship/lib/spaceship/connect_api/models/app_screenshot_set.rb:129:in `all'
        from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.216.0/spaceship/lib/spaceship/connect_api/models/app_store_version_localization.rb:101:in `get_app_screenshot_sets'
        from /Library/Ruby/Gems/2.6.0/gems/fastlane-2.216.0/deliver/lib/deliver/app_screenshot_iterator.rb:25:in `block (3 levels) in each_app_screenshot_set'
+------------------------------------+
|            Lane Context            |
+------------------+-----------------+
| DEFAULT_PLATFORM | ios             |
| PLATFORM_NAME    | ios             |
| LANE_NAME        | ios deliver_ios |
+------------------+-----------------+
[15:07:54]: it

+---------------------------------------+
|           fastlane summary            |
+------+------------------+-------------+
| Step | Action           | Time (in s) |
+------+------------------+-------------+
| 1    | default_platform | 0           |
| 💥   | deliver          | 5438        |
+------+------------------+-------------+

[15:07:54]: fastlane finished with errors

[!] An exception occurred for locale: it. Check the screenshot requirements here: https://help.apple.com/app-store-connect/en.lproj/static.html#devd274dd925

[rogerluan]

Thanks for your PR @mikhailmaslo ! 🙏

I'd also like to ask as many people as possible to test @mikhailmaslo 's PR, here: #21693

There are testing instructions at the bottom of the PR description to help you test this 😊 thank you everyone! ❤️