django-assets should support Django's storage backends about webassets

[miracle2k]

Since support for this would probably have to be added to webassets core anyway, using PyFilesystem might be an option.

[nsb]

+1

If you are not already working on this I could potentially give it a go. Let me know if you are interested.

[miracle2k]

Hey Niels, missed your comment. I'm not currently working on it.

This isn't that hard a thing to implement in general, but the devil is in the details. Some thoughts.

• Should any support for Django storages be based on PyFilesystem integration in webassets core? (i.e. wrapping Django storages in PyFilesystem storages?) I would say no. It would add additional complexity, make PyFilesystem a dependency, and also be more work, thus holding the whole thing up.

• Obviously, all filesystem interactions of webassets need to be wrapped.

• There is a very new Resolver class used to "find" bundle contents, which will be a lot of help here. For example, it could get a new open() method that returns a file-like object, and then django-assets could overwrite this method to open files from a storage.

• Besides reading and writing files, in a number of places webassets looks at timestamps. This might be the hardest design challenge, because a) not all storages may support getting the last modified time, and b) in some cases this may be too slow, thus defeating the purpose. For example, with auto_build the timestamp of the output file is checked on every request, but if the output file is on S3, you probably wouldn't want that.

  Of course, for output files an obvious solution is caching the last known modified timestamp. For input files, the solution might be to simply not check the timestamp and thus not recognizing changes for files that are on remote storages like S3.

[nsb]

Thanks for getting back to me. I'm not using webassets for my project anymore, so I'm not motivated to work on it currently. Sorry.

[jdp]

Having remote options would be great for deploying Python applications to hosts like Heroku, that allow applications to write to /tmp but don't guarantee that they will still be available on the next request. Being able to have bundle outputs stored on S3 to be accessed through Cloudfront or some other CDN that would be awesome.

[michaelmior]

One other alternative for consideration. After browsing through the code I think this would be a fairly quick implementation.

1. Enable the Django staticfiles resolver even if DEBUG is True.
2. Add an ASSETS_OUTPUT_PATH to the settings so that when assets are built, they all go to this directory.
3. Add a staticfiles finder which searches ASSETS_OUTPUT_PATH so these files are picked up when collectstatic is run.

Under this setup, you would just configure your storage backend for staticfiles, and collectstatic would pick up all the built assets. I'm sure there's some caveats I'm not seeing, but these seems much simpler than making the library storage-aware.

[miracle2k]

Note that bad an idea, Michael. It's not exactly "nice" having to use yet another directory for the webassets output, but it would be easy.

It might even be already possible to configure it along those lines: ASSETS_ROOT should work for a custom output directory (staticfiles would still be used for source files).

Do people push files to S3 even during development?

[michaelmior]

Thinking about it a bit more, I think we could just use STATIC_ROOT with a subfolder since that folder shouldn't be used for external storages. You might be right about things already working, the only piece that's missing then would be to enable the finder when DEBUG is disabled.

I personally don't push to S3 during development, but I do in our staging environment.

[michaelmior]

I just got back to this and I found a solution that's working great for me. I have a dummy app webassets that just contains __init__.py and a .gitignore to ignore .webassets-cache and static. Then I added this to INSTALLED_APPS and set ASSETS_ROOT to webassets/static.

First running assets build will build the assets and place them in webassets/static. Then when I run collectstatic, it will pick up those files along with all others. This simply requires use_staticfiles in DjangoResolver to return True. Perhaps this could be a setting?

For me just having a setting to enable this and some documentation similar to what I described above completely solves the issue for me.

[michaelmior]

Just an update that I realized using webassets for the dummy app name has the potential to cause naming conflicts in certain scenarios. Really any name can be used here. I decided to just prepend the project name.

[giacomocariello]

I'm not currently developing under Django, but I'd really have webassets modified to allow using PyFilesystem to access storage. I suppose the main modification would to abstract filesystem access routine by replacing os.*, FileHunk and open() built-ins with an extensible interface. I'm willing to help. Would you consider evaluating a patch? Any advice?

[Koed00]

I'm also having a problem with this. Django-assets keeps looking for assets in STATIC_ROOT but on the local filesystem instead of the actual filesytem (S3 in my case). Everything else seems to work.

Wouldn't it suffice then to change the AssetsFileStorage base class from FileSystemStorage to DefaultStorage ?

[michaelmior]

IMO, you still want django-assets looking on the local filesystem. No need to download from S3 to generate static assets when you should have the files sitting right there. I'm just using the AppDirectoriesFinder and not the AssetsFinder at all. I haven't experienced any issues thus far.

[Koed00]

I prefer to not have any static files on my heroku slugs, but I could live with a small assets folder for this. I understand this might not be ideal for everyone. Making the storage backend configurable via ASSETS_STORAGE for instance would work.

If only that would solve my problem.

Even though I have ASSETS_AUTO_BUILD = False , I still get an bundle error stating that the compiled files can't be found . The path shown is the ASSETS_ROOT. I'm guessing this is to check the md5 checksum of the compiled files. WIth heroku, I can't have a compiled file in the /static/ directory since the webservers keep cloning themselves from git. So I really need it to be checked on my storage backend.

[michaelmior]

I haven't gotten a solution I'm happy with for Heroku yet either. Are you using the %(version)s placeholder in your output filenames? If you're not, then you should be fine. You could also just commit your compiled static files to the repository, but that's obviously not ideal.

[miracle2k]

@Koed00 A traceback would be illuminating. ASSETS_AUTO_BUILD should indeed disable any file system access. Obviously that means that it needs to be able to construct the url to the files with checking any files, but that is only a problem if you want the url to contain a version. The error you should be seeing in such a case is BundleError("Cannot find version of ...")

[Koed00]

I gisted a stacktrace https://gist.github.com/Koed00/6842666

I gather the following happens; Webassets will look for the version number in the .webassets-cache folder This folder is not found on the local filesystem on Heroku since it lives in the STATIC_ROOT on S3 , so webassets looks for the bundles output file to determine the hash or file date,which is of course not found either.

I'll try ASSETS_VERSIONS = False on the next deploy

[miracle2k]

That would make it worse :) You need to either

1. disable ASSETS_URL_EXPIRE and make sure that no %(version)s placeholder is used in output files, or

2. Upload a manifest file that contains the version (ASSETS_VERSION='manifest'). I've tried to write up the issue here.

It would probably be useful though if webassets could get the version from the files on S3 by itself initially (and then maybe store them somewhere for speed), so this ticket does relate.

[Koed00]

@miracle2k works now without the versioning. Manifest file didn't work, cause it looks for it on the local filesystem. This would mean I'd have to commit the manifest to git and manually upload the compiled bundles. This is far from ideal with the dozen bundles I've already got. I'll do without versioning for now.

A solution I've seen in other django packages is to allow the user to select a different storage handler for the django-assets part via a setting. This would probably make things work, as long as Auto Build is off , so there's no webassets code used.

I'm willing to give it a go, just be aware that I've only been using Python and Django for about 3 months ;)

[osdiab]

Is this still in the pipeline? And does it extend to non-django projects? I'm using webassets standalone and can use this!