Possible asset issue during deploy about kamal

[SergeyBurtsev]

If each instance has it's own version of precompiled assets within it's container, there should be no problem, no?

[peterberkenbosch]

I think each container serves the assets from its own precompiled version, some can be outdated. Not sure how this would work when using a single CDN though without keeping some older refs around.

[edwinv]

If each instance has it's own version of precompiled assets within it's container, there should be no problem, no?

Then you make the assumption that a request for a (new) asset lands on the same server. When using a load balancer you should never assume that is the case.

I think each container serves the assets from its own precompiled version, some can be outdated. Not sure how this would work when using a single CDN though without keeping some older refs around.

That’s why we use an s3 bucket with all versions. We remove the older versions on each deploy.

[SergeyBurtsev]

Now I understand. We need public/assets folder shared between multiple instances, connected using volumes and containing old and new versions of assets...

[dhh]

I'd solve that by letting your CDN be the cache of the old assets. Not going to introduce the complication of sharing assets between containers into MRSK. Assets are set to live for a year with no need to check the origin. So assets from old versions will continue to live in the CDN.

[dhh]

Actually, I see what you mean. That you serve HTML from a new version of the app, which then triggers a request for a new asset, which hits the CDN, the CDN then hits the app, which then potentially hits an old version that still hasn't been restarted. I think that window is pretty small with these deploys, but if we start doing rolling deploys, it could increase.

One way of dealing with it is using an asset host, then ensuring the asset host points to the first container instance that will be restarted first. But need to document that.

[Verdoso]

At scale, having an asset host that is restarted first would move the issue from the new version of the app to the old versions of the app. A user hits an old host that has not been restarted, gets an asset from the new app and things might break (old html with new styles, new Javascript referencing non existing fields).
Hence @edwinv solution of having them both available while the deploy is going on, and assuming you have implemented the proper naming/versioning so each version requests their own assets.

I don't see a general solution that mrsk can provide, but at allowing users to add their "custom things" at the proper moments might be enough, so maybe adding some kind of pre-/post-deployment hooks to add custom code to prepar/clean up assets? If generalised, these hooks might even be used for other interesting things.

[dhh]

I don't think there's an issue in the other direction when you use a CDN that'll cache all old assets, and you ensure that the CDN uses the first-to-restart app as the origin for new assets.

But yeah, would be happy to see general callbacks explored. The audit broadcast callback is one such specialized example.

[AxelTheGerman]

when you use a CDN

and

when using a single CDN though without keeping some older refs around

I personally agree that the CDN should keep old assets around at least long enough for a deploy - if someone has humongous assets that are too expensive too keep they might need a special strategy there to purge as soon as it's safe.

I first thought assuming that folks use a CDN at all might be wrong, but when we're talking about a large deployment like this (not single containers where there is no issue) then I think documenting this somewhere might be good enough?

[dhh]

Yes, that's my view. This is a problem better handled at the CDN. This is how we keep MRSK small and light. Realizing that it's NOT a full-stack deployment framework, if you will. We're not trying to solve all possible deployment scenarios in this box. That's the pursuit for the likes of k8s or others. MRSK will stay lean by leaning on other modern, solutions that work with it.

So going to close this. Feel free to open a PR with documentation on this 👍

[shakalandy]

I think there is a misunderstanding of how CDNs work.
Just because you configure proper cache headers doesn't mean that CDNs will keep them for the time configured.
The time is not guaranteed, as unpopular assets will be evicted from cache and will require a repopulation from origin.

One would need a push-CDN for the described use-case, where you pay for cache storage size and get that space guaranteed - CDN will then take care about population of the assets - e.g. BunnyCDN offers that with "perma-cache".

An easier solution would be to provide an "asset" volume which has a mechanism in place to keep several versioned old assets.

I think @dhh used Cloudflare in the example on YouTube, which is a Pull CDN which will not behave the same way as e.g. Cloudfront/S3.

[claasz]

Regardless whether your CDN supports a perma cache or not (or maybe you don't use a CDN at all), you could always put your app behind a standard HTTP caching server like Varnish, nginx, Apache+mod_cache.