Activate task becomes very slow when you have a lot of deployed items about ember-cli-deploy-s3-index

[lukemelia]

I don't see any harm in having a config flag for this plugin that allows you to opt into the behavior you're describing. Then you wouldn't have the hassle of maintaining a fork.

[lukemelia]

@vitch I understand the issue and it seems from our discussions at the time that we had some concerns about this eventuality but decided the benefits outweighed the costs. The contract of the hooks is that they will fetch all the revisions so I am skeptical that general solutions that don't do this would be guaranteed to be backward compatible user's deployment configurations.

As a low-tech solution, have you considered deleting or archiving your 59,000 oldest revisions?

[vitch]

Yeah - I meant to mention that we'd considered that. The same folders are used to power our special historical URLs and I wanted to avoid adding special casing there...

I agree that it would be hard to do this in a backward compatible way which is why I split it out from #119.

I guess one other option would be to have a config flag to allow people to opt in to a new behaviour (where fetchRevisions returned nothing and fetchInitialRevisions returned only the currently active revision) if they were running into performance issues.

I'm essentially going to build that out anyway and test it in our environment to see if there are any unforeseen consequences. I can put up a PR for it and we can discuss there - if you don't think it's a good match for this plugin then I'm happy to maintain a forked version for our needs...

If you have any concrete examples of fetchRevisions or fetchInitialRevisions which need access to more than I describe above then let me know and it might help me to come up with a flexible solution...

[vitch]

Cool. Any preferences on a name? optimisedFetchRevisions? fetchOnlyRelevantRevisions? speedOverAccuracy?

(naming things is hard - those are more in the vein of brainstorming than actual suggestions)

[vitch]

In fact, it looks like fetchRevisions is used by deploy:list as well - I guess we don't want to break it there...

Looking at the hooks documentation and re-reading the threads above, I'm still not too clear on why fetchRevisions is called after activate. Can you clarify the use-case for me?

(it looks like I can get another ~20% speed up by changing fetchInitialRevisions to only loop over s3 until it finds the active revision but that percentage will change depending on where the active revision lies and I will be able to get a much bigger speedup by noop-ing the fetchRevisions call but I can't figure out how to do that safely)

[vitch]

Another possible approach is to cache the result of fetchInitialRevisions and manually update it post-activation rather than looping over s3 again in fetchRevisions...

That (in combination with #119) means we're loading the list of all deployed versions once instead of three times when deploy:activate is called... I'd prefer an option for zero times but once would definitely be a step in the right direction ;)