abashev / vfs-s3 Goto Github PK

View Code? Open in Web Editor NEW

92.0 7.0 50.0 5.35 MB

Amazon S3 driver for Apache commons-vfs (Virtual File System) project

License: Apache License 2.0

Java 100.00%

vfs-s3's People

Contributors

Stargazers

Watchers

Forkers

easel scoyo msiuts epugh alza-bitz ebrevdo quilombodigital mccllc mariogarcia svella flicken ms-tg waseemtaj dejlek roarking geoffhendrey themartynroberts ljy201992 eepstein edgardleal activeeon soxford etsangsplk yaraslaubarysenka fixitcode vraptorche sergey-derugo ptahchiev tool-recommender-bot voyagersearch vijayeluri kuangjz ofaizulin smartsquaregmbh schaefer-thomas maciejem mahifx danzelziggy zipleen bestyond radixisatidis yswang0927 44dw uabajwa jahirjava myz525

vfs-s3's Issues

Update AWS-S3 Version

Hello,
I hit the exact same error as these guys here:

aws/aws-sdk-java#2213

This seems to be a know bug which was fixed in version 1.11.925 as per the last comment:

We have updated the AWS SDK for Java, version 1.11.925 and newer, to correct instances where the Apache Xerces library was used incorrectly when parsing XML documents.

Can you please update the version of the AWS SDK. It is blocking me now to use the library.

java.lang.NoClassDefFoundError: Could not initialize class org.codehaus.groovy.vmplugin.v7.Java7

Hi, thx for this library, unfortunately I'm running into this error under openjdk 14 on Mac any hint?

[2020-09-20 17:20:19,500]-[Hotswap] INFO  io.be1.circular.filemanager.FileSystem - Starting Filesystem name=usercontent type=mac props={createpolicy=createandcopy, name=usercontent, rootdir=s3://circular-test.oc3.be1.io.ams3.digitaloceanspaces.com/usercontent/}
java.lang.NoClassDefFoundError: Could not initialize class org.codehaus.groovy.vmplugin.v7.Java7
	at io.be1.circular.my-app//org.codehaus.groovy.vmplugin.VMPluginFactory.<clinit>(VMPluginFactory.java:39)
	at io.be1.circular.my-app//org.codehaus.groovy.reflection.GroovyClassValueFactory.<clinit>(GroovyClassValueFactory.java:35)
	at io.be1.circular.my-app//org.codehaus.groovy.reflection.ClassInfo.<clinit>(ClassInfo.java:109)
	at io.be1.circular.my-app//org.codehaus.groovy.reflection.ReflectionCache.getCachedClass(ReflectionCache.java:95)
	at io.be1.circular.my-app//org.codehaus.groovy.reflection.ReflectionCache.<clinit>(ReflectionCache.java:39)
	at io.be1.circular.my-app//org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.registerMethods(MetaClassRegistryImpl.java:207)
	at io.be1.circular.my-app//org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.<init>(MetaClassRegistryImpl.java:105)
	at io.be1.circular.my-app//org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.<init>(MetaClassRegistryImpl.java:83)
	at io.be1.circular.my-app//groovy.lang.GroovySystem.<clinit>(GroovySystem.java:36)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:340)
	at io.be1.circular.my-app//com.amazonaws.util.VersionInfoUtils.languageVersion(VersionInfoUtils.java:279)
	at io.be1.circular.my-app//com.amazonaws.util.VersionInfoUtils.groovyVersion(VersionInfoUtils.java:206)
	at io.be1.circular.my-app//com.amazonaws.util.VersionInfoUtils.getAdditionalJvmLanguages(VersionInfoUtils.java:190)
	at io.be1.circular.my-app//com.amazonaws.util.VersionInfoUtils.userAgent(VersionInfoUtils.java:159)
	at io.be1.circular.my-app//com.amazonaws.util.VersionInfoUtils.initializeUserAgent(VersionInfoUtils.java:137)
	at io.be1.circular.my-app//com.amazonaws.util.VersionInfoUtils.getUserAgent(VersionInfoUtils.java:100)
	at io.be1.circular.my-app//com.amazonaws.ClientConfiguration.<clinit>(ClientConfiguration.java:70)
	at io.be1.circular.my-app//com.github.vfss3.S3FileSystemConfigBuilder.getClientConfiguration(S3FileSystemConfigBuilder.java:95)
	at io.be1.circular.my-app//com.github.vfss3.S3FileSystemOptions.getClientConfiguration(S3FileSystemOptions.java:83)
	at io.be1.circular.my-app//com.github.vfss3.S3FileProvider.doCreateFileSystem(S3FileProvider.java:60)
	at io.be1.circular.my-app//com.github.vfss3.CachingFileProvider.findFile(CachingFileProvider.java:121)
	at io.be1.circular.my-app//com.github.vfss3.CachingFileProvider.findFile(CachingFileProvider.java:91)
	at io.be1.circular.my-app//com.github.vfss3.S3FileProvider.findFile(S3FileProvider.java:26)
	at io.be1.circular.my-app//org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:711)
	at io.be1.circular.my-app//org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:648)
	at io.be1.circular.my-app//io.be1.circular.filemanager.FileSystem.startFileSystem(FileSystem.java:79)
	at io.be1.circular.my-app//io.be1.circular.filemanager.FileManager.startFileSystem(FileManager.java:1417)
	at io.be1.circular.my-app//io.be1.circular.filemanager.FileManager.lambda$configure$0(FileManager.java:2340)
	at io.be1.circular.my-app//org.jooby.funzy.Throwing$Consumer.lambda$accept$0(Throwing.java:273)
	at io.be1.circular.my-app//org.jooby.funzy.Throwing.runAction(Throwing.java:2415)
	at io.be1.circular.my-app//org.jooby.funzy.Throwing.access$000(Throwing.java:39)
	at io.be1.circular.my-app//org.jooby.funzy.Throwing$Consumer.accept(Throwing.java:273)
	at io.be1.circular.my-app//org.jooby.Jooby.start(Jooby.java:2219)
	at io.be1.circular.my-app//org.jooby.Jooby.start(Jooby.java:2178)
	at io.be1.circular.my-app//org.jooby.Jooby.run(Jooby.java:2107)
	at io.be1.circular.my-app//io.be1.circular.App.main(App.java:624)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at org.jooby.run.Main.lambda$startApp$3(Main.java:444)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
	at java.base/java.lang.Thread.run(Thread.java:832)

S3FileObject/S3OutputStream onClose() should setContentLength()

Problem observed:

Truncated uploads w/ non-trivial objects (e.g. missing bytes at the end of a ~230MB object). This has sometimes been but is not always associated with a timeout.

We've worked around this by not using vfs-s3 to upload and instead using jets3t 0.9 directly, but we're not doing much different than vfs-s3 other than specifying the temporary File instead of using a FileChannel and explicitly setting the content length.

Observation -- S3FileObject does not set the content length when you close it (after writing directly to the OutputStream from the FileObject), but the jets3t docs suggest that you must.

http://jets3t.s3.amazonaws.com/toolkit/code-samples.html#uploading

If your data isn't a File or String you can use any input stream as a data source, but you must manually set the Content-Length.

Upgrade aws-java-sdk

The aws-java-sdk 1.6.3 version has a dependency graph that depends on a number of older Apache commons libraries (e.g. codec) which is incompatible with the dependency graph of newer Apache commons libraries.

I will be associating a pull request with this issue as my suggested solution. Another solution may be to shade the entire jar in order to avoid the dependency issues altogether.

Need to document that you need to tweak pom to run tests.

Had to manually change to true the pom:

  <configuration>
                <skipTests>false</skipTests>
            </configuration>

Upgrade to aws-sdk-java-v2

https://github.com/aws/aws-sdk-java-v2

Update to commons-vfs 2.2

Commons-vfs 2.2 was released on 2017-10-06. Are there plans to update the custom version used by vfs-s3 to 2.2? Additionally, what exactly is the issue that necessitates a custom version (README only says "some concurrency issues")?

Update the samples.

The S3Shell in the samples module has this code:

        FileSystemOptions opts = S3FileProvider.getDefaultFileSystemOptions();

however this method is now missing in S3FileProvider. Please update the samples - how are we supposed to know what has replaced this method.

S3FileObject.exists returns true even AmazonS3 returns 403 forbidden

How to repeat:
Connect to bucket without access rights to it (Bucket is valid, keys valid (or IAM role), but no access rights)
Resolve fileObject and check it exists using
fileObject.exists()

S3FileObject.exists just checks getType()!=IMAGANARY

getType() tries to do attach.

During doAttach it fails with AmazonS3Exception e, e.getStatusCode() == 403
Then it performs doAttach with empty Matadata

doAttach(FILE, new ObjectMetadataHolder());

As the result fileObject.type == FILE and exists() will return true

Expected result:
exists must throw FileSystemException in case access is Forbidden

parent pom of commons-vfs 2.2-20181130 does not exist

com.github.abashev commons-vfs2 2.2-20181130

This will pull in parent pom but cannot find it in maven repo.

us-east-1 is hardcoded and no other region works

Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-central-1' (Service: Amazon S3; Status Code: 400; Error Code: AuthorizationHeaderMalformed; Request ID: ...; S3 Extended Request ID: ...)

When I have s3://s3.eu-central-1.amazonaws.com/bucket-name as a URL. I believe the issue is here.

Also, is there any way to make vfs-s3 work with the original VFS code not your patched one? As we depend on their latest fixes...

Support for AWS_WEB_IDENTITY_TOKEN_FILE for container running on AWS EKS cluster

Hey guys,

We have a microservice that uses vfs-s3. We run it in a container running on an AWS EKS cluster.
For security reasons, we do not want to use explicit credentials, but we set up the service account of the pod to have the necessary IAM role to access our S3 bucket.
For that EKS then dynamically provides the access token in a file which is offered via the environment variable AWS_WEB_IDENTITY_TOKEN_FILE.

Unfortunately, we are not able to get it to work, the library does not pick up that secret.
It would be supported in the DefaultAWSCredentialsProviderChain, but it looks like that this is not being picked up.

Can you guide us, on how to get it to use that Web Identity Token credentials for the authentication?

Thanks and regards,
Matthias

Temporary files (/tmp/scalr.*.s3)

We're using the vfs-s3 library and are seeing a large number of files in the temp directory following the form of scalr.*.s3 that appear to be created by the vfs-s3 code and are never cleaned up. Looking at the code for S3FileObject, it appears these are cache files (see getCacheFileChannel()) created using File.createTempFile(). Our application runs for an extended period of time so these files will not be removed as the VM isn't shut down. It might be worth utilizing something like org.apache.commons.io.FileCleaner which provides a way to clean up files when GC occurs.

Incorrect S3 metadata for content type, last accessed time and content length.

All uploaded files have content-type = application/octet-stream. Mime type is not properly set before uploading in ObjectMetadataHolder.

A quick patch fixes the problem, see attached file.

Use_correct_ObjectMetadata_instance_in_withContentType,_withLastModifiedNow_and_withConten.patch.zip

Vfs-S3 v3.0 is not available for download

According to https://github.com/abashev/vfs-s3 the VFS plugin v3.0 should be available in the repository http://dl.bintray.com/content/abashev/vfs-s3 but currently it is not available there

Multiple providers registered for URL scheme "s3"

Hello,

I have a spring-boot project that uses org.reflections:

https://github.com/ronmamo/reflections/

It all works fine when I run the project with the maven plugin mvn spring-boot:run. However, when I produce a fat war file and run it with java -jar target/storefront.war I get the following exception:

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'localFileStorageService': Injection of resource dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'defaultFileSystemManager' defined in class path resource [com/nemesis/platform/config/PlatformCoreConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.commons.vfs2.FileSystemManager]: Factory method 'defaultFileSystemManager' threw exception; nested exception is org.apache.commons.vfs2.FileSystemException: Could not create a file system manager of class "org.apache.commons.vfs2.impl.StandardFileSystemManager".
    at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.postProcessPropertyValues(CommonAnnotationBeanPostProcessor.java:308)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1210)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:537)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1120)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1044)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:942)
    at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:813)
    at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:741)
    ... 39 more
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'defaultFileSystemManager' defined in class path resource [com/nemesis/platform/config/PlatformCoreConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.commons.vfs2.FileSystemManager]: Factory method 'defaultFileSystemManager' threw exception; nested exception is org.apache.commons.vfs2.FileSystemException: Could not create a file system manager of class "org.apache.commons.vfs2.impl.StandardFileSystemManager".
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:599)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1119)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1014)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:504)
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
    at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.autowireResource(CommonAnnotationBeanPostProcessor.java:461)
    at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.getResource(CommonAnnotationBeanPostProcessor.java:435)
    at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor$ResourceElement.getResourceToInject(CommonAnnotationBeanPostProcessor.java:559)
    at org.springframework.beans.factory.annotation.InjectionMetadata$InjectedElement.inject(InjectionMetadata.java:169)
    at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
    at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.postProcessPropertyValues(CommonAnnotationBeanPostProcessor.java:305)
    ... 51 more
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.apache.commons.vfs2.FileSystemManager]: Factory method 'defaultFileSystemManager' threw exception; nested exception is org.apache.commons.vfs2.FileSystemException: Could not create a file system manager of class "org.apache.commons.vfs2.impl.StandardFileSystemManager".
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189)
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
    ... 65 more
Caused by: org.apache.commons.vfs2.FileSystemException: Could not create a file system manager of class "org.apache.commons.vfs2.impl.StandardFileSystemManager".
    at org.apache.commons.vfs2.VFS.createManager(VFS.java:89)
    at org.apache.commons.vfs2.VFS.getManager(VFS.java:52)
    at com.nemesis.platform.config.PlatformCoreConfig.defaultFileSystemManager(PlatformCoreConfig.java:150)
    at com.nemesis.platform.config.PlatformCoreConfig$$EnhancerBySpringCGLIB$$424fad06.CGLIB$defaultFileSystemManager$9(<generated>)
    at com.nemesis.platform.config.PlatformCoreConfig$$EnhancerBySpringCGLIB$$424fad06$$FastClassBySpringCGLIB$$2bf4a0e0.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
    at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:309)
    at com.nemesis.platform.config.PlatformCoreConfig$$EnhancerBySpringCGLIB$$424fad06.defaultFileSystemManager(<generated>)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
    ... 66 more
Caused by: org.apache.commons.vfs2.FileSystemException: Could not load VFS configuration from "jar:file:/tmp/tomcat.6453335118692475459.8112/work/Tomcat/localhost/storefront/WEB-INF/lib/vfs-s3-2.3.1.jar!/META-INF/vfs-providers.xml".
    at org.apache.commons.vfs2.impl.StandardFileSystemManager.configure(StandardFileSystemManager.java:194)
    at org.apache.commons.vfs2.impl.StandardFileSystemManager.configurePlugins(StandardFileSystemManager.java:148)
    at org.apache.commons.vfs2.impl.StandardFileSystemManager.init(StandardFileSystemManager.java:118)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.commons.vfs2.VFS.createManager(VFS.java:76)
    ... 78 more
Caused by: org.apache.commons.vfs2.FileSystemException: Multiple providers registered for URL scheme "s3".
    at org.apache.commons.vfs2.impl.DefaultFileSystemManager.addProvider(DefaultFileSystemManager.java:208)
    at org.apache.commons.vfs2.impl.StandardFileSystemManager.addProvider(StandardFileSystemManager.java:374)
    at org.apache.commons.vfs2.impl.StandardFileSystemManager.configure(StandardFileSystemManager.java:268)
    at org.apache.commons.vfs2.impl.StandardFileSystemManager.configure(StandardFileSystemManager.java:190)
    ... 85 more

I stopped with a breakpoint in DefaultFileSystemManager:addProvider and indeed the s3 is registered twice. This makes sense because it is declared twice in the vfs-providers.xml:

<?xml version="1.0" encoding="UTF-8"?>
<providers>
    <provider class-name="com.intridea.io.vfs.provider.s3.S3FileProvider">
        <scheme name="s3"/>
    </provider>
  <operationProvider class-name="com.intridea.io.vfs.provider.s3.operations.S3FileOperationsProvider">
    <scheme name="s3" />
  </operationProvider>
</providers>

And when I run it from the maven plugin it is only registered once! What is the difference between operationProvider and provider? Why running it from a war file registers both providers, and running it from the webapps folder registers only one? How can I avoid this exception?

S3 Metadata

The API doesn't seem to permit for modification of S3 metadata, either through operations or file attributes. If there exists a way, I can't find it in code samples.

[4.0.0] AbstractFileSystem and AbstractVfsComponent are custom

I saw the following error in our logs:

class com.github.vfss3.S3FileSystem cannot be cast to class org.apache.commons.vfs2.provider.AbstractFileSystem

Coming from here:

commons-vfs2-2.4.1.jar!/org/apache/commons/vfs2/impl/DefaultFileMonitor$FileMonitorAgent.class:423→ fireAllCreate

And I saw that in version 4.0.0 of vfs-s3 there is a custom class AbstractFileSystem and AbstractVfsComponent (at least these two, maybe there are more). This makes vfs-s3 incompatible with utilities provided by commons-vfs like the DefaultFileMonitor.

S3 files are copied to local tmp dir then streamed through VFS

Although the VFS abstraction works with streams this library appears to use an intermediary file stored locally in the temp directory. For example when obtaining an input stream, the S3 file is first downloaded into the temp directory then an input stream on that temp file is opened. See S3FileObject.doGetInputStream() which in turn calls S3FileObject.downloadOnce().

Is there any plan to change this such that files are streamed to and from S3 directly without the temp file step?

Direct streaming is what we have in the VFS http/https providers, see HttpFileObject.doGetInputStream(). Without this there are issues with handling large files.

The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256

I get this error when I try to list a file in my bucket:

ERROR: Cannot load image: /solarapparel/content/banner/solar-your-better-half.png because of Unknown message with code "Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: 13180FE7FC5D411F)"

When I try to open the bucket with S3Browser I get the following error:

The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256

According to this: http://stackoverflow.com/questions/26533245/the-authorization-mechanism-you-have-provided-is-not-supported-please-use-aws4
the Frankfurt region only supports the V4 method of authentication. Please make sure the vfs-s3 supports the V4 too.

Can't use custom AmazonS3Client

You can not manually create the AmazonS3Client and set it on the configuration without also setting credentials. If you already configured the client, then the credentials should not be required.

I'd advise moving https://github.com/abashev/vfs-s3/blob/branch-2.3.x/src/main/java/com/intridea/io/vfs/provider/s3/S3FileProvider.java#L78 down into the "if(service == null)" block.

Cannot resolve files with spaces in the name

Calling VFS.getManager().resolveFile(...) with a file name that has a space inside fails because S3FileNameParser does a new URI on line 50 which blows up with java.net.URISyntaxException: Illegal character in path at index 46:.

Other VFS providers work fine with such filenames (I guess by encoding the URL or something - you can check LocalFileNameParser for example) so the problem is in vfs-s3.

java.lang.ClassNotFoundException: org.slf4j.LoggerFactory and why slf4j dependency.

A simple project with these dependencies (aws libraries are transitive dependencies so no included):

  <dependencies>
    <dependency>
      <groupId>org.apache.commons</groupId>
      <artifactId>commons-vfs2</artifactId>
      <version>2.4.1</version>
    </dependency>
    <dependency>
      <groupId>com.github.abashev</groupId>
      <artifactId>vfs-s3</artifactId>
      <version>4.0.0</version>
    </dependency>
  </dependencies>

And this simple code:

    public static void main( String[] args ) throws FileSystemException {
        FileSystemManager vfs = VFS.getManager();
    }

Throws this exception:

Caused by: java.lang.ClassNotFoundException: org.slf4j.LoggerFactory

So two questions:

Why in your pom, the slf4j dependency scope is provided?

        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>jcl-over-slf4j</artifactId>
            <version>1.7.21</version>
            <scope>provided</scope>
        </dependency>

Why are not you using commons-logging like aws-sdk libraries and commons-vfs2 are doing? Both libraries are forced dependency in your case, so why to add a new dependency that it is not necessary at all.

Update the aws-java-sdk-s3 library

looks like (at least) eu-north-1 is missing from it

Copying from S3 leaves a temp file

I copy files from an S3 bucket to the local file system on Linux with copyFrom, and this leaves a temp file in /tmp, named like vfs.17826129911195593952.s3. The file is a copy of the downloaded file, and is managed by S3TempFile.

I would expect this file to be deleted as soon as it's not needed anymore, but it stays indefinitely. Since I transfer a lot of files over time, this eventually causes the disk to fill up.

I immediately thought this was caused by me not closing the file object, but doing that didn't change anything. After experimenting a bit, I found that any read from the InputStream of an S3FileObject leaves this temp file behind.

Even running the bundled samples/s3-copy script leaves this file. This is seen on version 4.3.1. Adjusting the @Grab in that script, I found I could reproduce it as far back as version 4.1.0; earlier versions fail to load with "Error grabbing Grapes", so I haven't gone further back.

So reproducing it is as simple as running something like samples/s3-copy s3://access:[email protected]/s3-tests-2/source.txt file:///tmp/dest.txt

I've seen this behaviour on Ubuntu 20.04 using OpenJDK 11.

FileTypeSelectorTest fails in commons-vfs

It seems the modifications done to the Apache Commons VFS code - specifically in AbstractFileObject::detach break the unit test FileTypeSelectorTest.

Don't attach file to delete

Can S3FileObject was been deleted directly, without calling doAttach method in S3FileObject class.
It will improve performance on deleting files, because Amazon client won't return exception or error response
Deletes the specified object in the specified bucket. Once deleted, the object can only be restored if versioning was enabled when the object was deleted. If attempting to delete an object that does not exist, Amazon S3 will return a success message instead of an error message.
Is it acceptable for S3FileObject?

Use ObjectMetadata where possible in S3FileObject, to avoid keeping unused http connections open

On the awssdk branch, to avoid ConnectionPoolTimeoutException errors that prevent
most of the tests in S3ProviderTest from completing, use ObjectMetadata instead of S3Object where possible in S3FileObject; this will avoid keeping unused http
connections open, as mentioned in the Javadoc for AmazonS3.getObject(..)

CachingFileProvider.findFileSystem never finds previously created file system if URI contains credentials

CachingFileProvider.findFileSystem never finds previously created file system if URI contains credentials, because doCreateFileSystem adds additional option to FileSystemOptions which is used as key in search.
doCreateFileSystem is called before adding FileSystem to cache.

    protected FileObject findFile(FileName name, FileSystemOptions fileSystemOptions) throws FileSystemException {
        FileSystem fs = findFileSystem(rootName, fileSystemOptions);
        if (fs == null) {
                    fs = doCreateFileSystem(rootName, fileSystemOptions);
                    addFileSystem(rootName, fs);
        }
    }

    protected FileSystem doCreateFileSystem( FileName fileName, FileSystemOptions fileSystemOptions ) throws FileSystemException {
        final S3FileName root = (S3FileName) fileName;
        final S3FileSystemOptions options = new S3FileSystemOptions(fileSystemOptions);

        if (root.hasCredentials()) {
            options.setCredentialsProvider(
                    new AWSStaticCredentialsProvider(new BasicAWSCredentials(root.getAccessKey(), root.getSecretKey()))
            );
        }

As the result new file system is added to map with different key than we was searching for

Is there a way to set auth info other than use environment variables?

Not all classes from aws-java-sdk-s3 are shadowed

Some classes in the com.amazonaws package are not "shaded". This means that when I also import aws-java-sdk-s3 to use it in my own code, the loading of these classes is non-deterministic and runtime exceptions occur because of incompatible classes.

Either all classes from aws-java-sdk-s3 should be "shaded" or it should be used as a dependency (of course, that leads to issues like #64).

P.S. I'm talking about vfs-s3 version 4.2.0.

Is it necessary to generate temporary files?

Temporary files were generated here: com.github.vfss3.S3TempFile#S3TempFile
Will undoing this make uploading faster?

Random-access content is not supported

content.getRandomAccessContent(RandomAccessMode.READWRITE)

This throws an exception that random-access content is not supported. Is this a limitation of S3 or it's just not implemented in vfs-s3?

Thread leak issue with singleton usage

We have experienced thread leaks when using the library in what I understand to be a standard way. By that, I mean we are creating a singleton of the manager at start up and continue to use it until the application shuts down at which point we close it. Also, we are closing the files once they've been used.

To get around the issue we are now closing the file system manager after every usage. This works but seems inefficient and runs counter to the usage as outlined on the Apache VFS site (http://commons.apache.org/proper/commons-vfs/api.html).

S3FileSystemConfigBuilder is not recognized by DelegatingFileSystemOptionsBuilder

S3FileSystemConfigBuilder is not recognized by DelegatingFileSystemOptionsBuilder. The exception is " File provider for URL scheme "s3" does not provide a configuration builder."

Full stack trace:

org.apache.commons.vfs2.FileSystemException: File provider for URL scheme "s3" does not provide a configuration builder.
at org.apache.commons.vfs2.util.DelegatingFileSystemOptionsBuilder.createSchemeMethods(DelegatingFileSystemOptionsBuilder.java:438)
at org.apache.commons.vfs2.util.DelegatingFileSystemOptionsBuilder.getSchemeMethods(DelegatingFileSystemOptionsBuilder.java:423)
at org.apache.commons.vfs2.util.DelegatingFileSystemOptionsBuilder.fillConfigSetters(DelegatingFileSystemOptionsBuilder.java:404)
at org.apache.commons.vfs2.util.DelegatingFileSystemOptionsBuilder.setValues(DelegatingFileSystemOptionsBuilder.java:194)
at org.apache.commons.vfs2.util.DelegatingFileSystemOptionsBuilder.setConfigStrings(DelegatingFileSystemOptionsBuilder.java:139)
at org.apache.commons.vfs2.util.DelegatingFileSystemOptionsBuilder.setConfigString(DelegatingFileSystemOptionsBuilder.java:122)
at test.VfsS3Test.testCreate(VfsS3Test.java:19)

The attached sample demonstrates the issue.
vfss3test.zip

FileObject.copyFrom(..) doesn't create needed parent folders correctly

On the awssdk branch, the current implementation of FileObject.copyFrom(FileObject, FileSelector) doesn't ensure that any needed parent 'folders' are created correctly when copying to a sub-folder that doesn't yet exist; this can be confirmed by checking FileObject.exists(), and FileObject.getType() for any created sub-folders; although the sub-folders 'exist' in S3 according to the Amazon web ui, the above checks will not give the expected result which causes further problems when attempting to delete these folders.

listObjects method doesn't work on very large buckets!

The method listObjects on https://github.com/abashev/vfs-s3/blob/master/src/main/java/com/intridea/io/vfs/provider/s3/S3FileObject.java#L245 doesn't work on VERY LARGE buckets, it eventually times out. Works fine on buckets that don't have potentially millions of documents in them.

I hit the jets3t wiki, and found some links about a listObjectsChunked method that lets you not return all the matches, but just some. And this sample code:

https://bitbucket.org/jmurty/jets3t/src/2573ec7fb362e60ce68585cbd8b349833be9142c/src/org/jets3t/samples/ThreadedObjectListing.java?at=default

That appears to show using the listObjectsChunked in a threaded manner.

I'm still new to this, so any thoughts would be great. Seems like vfs-s3 should use the listObjectsChunked method instead of listObjects.

Upgrade to org.apache.commons:commons-vfs2:2.1

As per the last comment in this issue:

https://issues.apache.org/jira/browse/VFS-574

commons-vfs2 2.1 is out and has more than 171 bugfixes. Please use the latest version from the maven central repository.

Exception with latest aws-sdk

Hello,

i'm using the vfs-s3 with version 1.10.73 of the aws-sdk and it all works fine. However if I upgrade to version 1.11.5 I get this exception:

2016-06-01 13:12:38,166 org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/storefront].[repositoryRestDispatcherServlet] [http-nio-127.0.0.1-8112-exec-3] ERROR: Servlet.service() for servlet [repositoryRestDispatcherServlet] in context with path [/storefront] threw exception
org.apache.commons.vfs2.FileSystemException: Unknown message with code "Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: DE52EFC460A49E56)".
        at com.intridea.io.vfs.provider.s3.S3FileSystem.<init>(S3FileSystem.java:58)
        at com.intridea.io.vfs.provider.s3.S3FileProvider.doCreateFileSystem(S3FileProvider.java:97)
        at org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.getFileSystem(AbstractOriginatingFileProvider.java:117)
        at org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:85)
        at org.apache.commons.vfs2.provider.AbstractOriginatingFileProvider.findFile(AbstractOriginatingFileProvider.java:69)
        at org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:790)
        at org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveFile(DefaultFileSystemManager.java:712)
        at com.nemesis.platform.core.service.fs.impl.AbstractFileStorageService.resolveFile(AbstractFileStorageService.java:54)
        at com.nemesis.platform.core.service.fs.impl.AbstractFileStorageService.storeFile(AbstractFileStorageService.java:120)
        at com.nemesis.platform.core.service.fs.impl.AbstractFileStorageService.storeMedia(AbstractFileStorageService.java:80)
        at com.nemesis.platform.module.restservices.storefront.controller.RestMediaUploadController.upload(RestMediaUploadController.java:49)
        at com.nemesis.platform.module.restservices.storefront.controller.RestMediaUploadController$$FastClassBySpringCGLIB$$b29f6739.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
        at com.nemesis.platform.module.restservices.storefront.controller.RestMediaUploadController$$EnhancerBySpringCGLIB$$9a69e803.upload(<generated>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:832)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:743)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:961)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:895)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:967)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:869)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:843)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.springframework.boot.actuate.autoconfigure.EndpointWebMvcAutoConfiguration$ApplicationContextHeaderFilter.doFilterInternal(EndpointWebMvcAutoConfiguration.java:261)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at com.samplestore.storefront.filter.RequestLoggerFilter.doFilterInternal(RequestLoggerFilter.java:81)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:92)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:115)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:316)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:122)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:48)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:158)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at com.nemesis.platform.core.filter.RestAuthenticationFilter.doFilterInternal(RestAuthenticationFilter.java:70)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:120)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:91)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:53)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:213)
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:176)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:87)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:77)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at com.nemesis.platform.module.cms.storefront.filter.PathLocaleFilter.doFilterInternal(PathLocaleFilter.java:87)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:126)
        at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:65)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:92)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:121)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:103)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
        at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:676)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:522)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1095)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:672)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1502)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1458)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Unknown Source)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: DE52EFC460A49E56), S3 Extended Request ID: OP8OltSpz4gSLWBFCz23dt9UeO9Ufn64OyzAZfTla49wzUzL4Vj6WObV2hBUD3UQbvofoiPp2d8=
        at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1305)
        at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:852)
        at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:630)
        at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:405)
        at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:367)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:318)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3787)
        at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1063)
        at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1021)
        at com.intridea.io.vfs.provider.s3.S3FileSystem.<init>(S3FileSystem.java:45)
        ... 139 more

Getting the last modified timestamp returns different results

When using DigitalOcean Spaces, requesting the last modified time in these two ways yields two different results:

def fileSystemOptions = new FileSystemOptions()
S3FileSystemConfigBuilder.getInstance().setCredentialsProvider(fileSystemOptions,
	new AWSStaticCredentialsProvider(new BasicAWSCredentials('username', 'password')))

def file = VFS.getManager().resolveFile('s3://ams3.digitaloceanspaces.com/folder', fileSystemOptions)
println file.getChildren()[0].getContent().getLastModifiedTime()

file = VFS.getManager().resolveFile('s3://ams3.digitaloceanspaces.com/folder/file.txt', fileSystemOptions)
println file.getContent().getLastModifiedTime()

In my case I get:

1575469137773
1575469137000

That is, when getting the file directly, the timestamp is truncated. This is not a bug in vfs-s3 but rather in DigitalOcean. However, a workaround in vfs-s3 would be nice.

getChildren needs ending slash but won't accept one

AWS CLI will not list the contents of a directory unless the path ends in a slash:

$ aws s3 ls s3://mslinntest/testDir
                           PRE testDir/
$ aws s3 ls s3://mslinntest/testDir/
                           PRE /
2014-09-13 15:18:19          0 
2014-09-13 15:35:52          0 testFile.txt

Seems the Java library works the same way, but vfs-s3 removes the trailing slash so getChildren() does not return the expected value:

import com.intridea.io.vfs.provider.s3.S3FileProvider;
import org.apache.commons.vfs2.FileObject;
import org.apache.commons.vfs2.FileSystemOptions;
import org.apache.commons.vfs2.VFS;
import org.apache.commons.vfs2.auth.StaticUserAuthenticator;
import org.apache.commons.vfs2.impl.DefaultFileSystemConfigBuilder;

import java.io.FileInputStream;
import java.util.Properties;

public class Test {
    public static void main(String[] args) throws Exception {
        Properties config = new Properties();
        config.load(new FileInputStream(System.getProperty("user.home") + "/.aws/config")); // same authentication file used by aws-cli
        StaticUserAuthenticator auth = new StaticUserAuthenticator(null, config.getProperty("aws_access_key_id"), config.getProperty("aws_secret_access_key"));
        FileSystemOptions opts = S3FileProvider.getDefaultFileSystemOptions();
        DefaultFileSystemConfigBuilder.getInstance().setUserAuthenticator(opts, auth);

        FileObject[] files = VFS.getManager().resolveFile("s3://mslinntest/testDir").getChildren();
        for (FileObject file : files)
          System.out.println(file);
    }
}

Output is:

Sep 13, 2014 7:00:57 PM com.intridea.io.vfs.provider.s3.S3FileProvider doCreateFileSystem
INFO: Initialize Amazon S3 service client ...
Sep 13, 2014 7:00:57 PM com.intridea.io.vfs.provider.s3.S3FileProvider doCreateFileSystem
INFO: ... Ok
Sep 13, 2014 7:00:57 PM com.intridea.io.vfs.provider.s3.S3FileSystem <init>
INFO: Created new S3 FileSystem mslinntest
Sep 13, 2014 7:00:59 PM com.intridea.io.vfs.provider.s3.S3FileObject doAttach
INFO: Attach folder to S3 Object: S3Object [key=testDir/, bucket=mslinntest, lastModified=Sat Sep 13 15:18:19 PDT 2014, dataInputStream=null, Metadata={ETag="d41d8cd98f00b204e9800998ecf8427e", Date=Sat Sep 13 19:01:00 PDT 2014, Content-Length=0, id-2=n7/EaF3YOkx4/oFnw0lQQ6NMjXfn2MbTQxUAhrqsHJdhzG+gx2Z6YY9rlIYnTUnE, request-id=A1047A519D88CDC3, Last-Modified=Sat Sep 13 15:18:19 PDT 2014, Content-Type=application/octet-stream}]
Exception in thread "main" org.apache.commons.vfs2.FileSystemException: Invalid descendent file name "/".
    at org.apache.commons.vfs2.impl.DefaultFileSystemManager.resolveName(DefaultFileSystemManager.java:791)
    at org.apache.commons.vfs2.provider.AbstractFileObject.getChildren(AbstractFileObject.java:710)
    at Test.main(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)

Should I not use getChildren() because there is a better way of listing an S3 directory?

Here is my project with both Scala and Java code that demonstrates this problem.

Any know if this is hosted on a public maven repo?

Don't want to import this entire project just to compile it.

S3FileObject#doSetLastModifiedTime doesn't set last modified time in uploaded S3 object

We're using this library to mirror a remote FTP site to S3, and would like to compare the last modified timestamp of the files in FTP to the last modified timestamp in S3, to ensure that the timestamps are the same.

However, setting last modified time on S3FileObject doesn't change the timestamp on the S3 server.
Per documentation (see also discussion), it's impossible. A workaround is to set a custom metadata field (e.g. x-amz-meta-last-modified).

Would you be open to a pull request that sets this metadata on upload and uses it as the last modified date if set?

Rename/Move makes a copy but doesn't delete the original.

S3FileObject.doRename() calls copyObject but doesn't bother deleting the original.

copyFrom returns 200 but file is not created

Hi Abashev,

we're experiencing a weird behavior using the copyFrom method. We're copying a sample text file from local to S3, everything looks fine, the log says the PUT request gets a 200, but no file is available in S3 (from the console nor from the getChildren method).

Have you experienced something similar in the past? Are we missing something?

Thanks,

Setting permissions of uploaded pictures

This could be a question, or a bug - I don't really know. I am able to upload pictures to S3, and I also try to set read permissions with this:

file.setReadable(true, false);

According to the javadoc of VFS this method will Sets the owner's (or everybody's) read permission.. However my pictures are uploaded, but only with owner read permissions. I have to manually select Add more permissions -> Grantee: everyone -> Open/download from the s3 console.

How can I set permissions of the uploaded file? Is this even implemented?

Append-output stream is not supported

content.getOutputStream(true)

Calling this leads to an exception that an append-output stream is not supported. Is this a limitation of S3 or just not implemented in vfs-s3? This is more important for us than random-access contents.

Support Amazon IAM roles for authentication

Amazon recommends accessing S3 through IAM roles instead of specifying AWS access key and secret id. The blog post below has a good summary of how to do this.

http://blogs.aws.amazon.com/security/post/Tx1XG3FX6VMU6O5/A-safer-way-to-distribute-AWS-credentials-to-EC2

For backwards comparability I suppose if credentials are provided they would be used as-is today otherwise this constructor could be used instead if no credentials are specified:

http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Client.html#AmazonS3Client(com.amazonaws.ClientConfiguration)

S3FileSystemConfigBuilder.getInstance().getAWSCredentials(fsOptions) currently fails if access key or secret key are empty. As well S3FileProvider.doCreateFileSystem(...) would need to call the appropriate AmazonS3Client constructor depending on if AWSCredentials are specified by the user or not.

Assets Missing From BinTray?

I'm trying to test this library out, but when I include it as a dependency, grandle isn't able to find it.

I assumed this should be available via jcentral, but I don't see it here:
http://jcenter.bintray.com/com/github/ -- It would be under vfs-s3, but that directory doesn't exist.

Is there another repository we need to add to pull this dependency?

What are the concurrency issues that were fixed?

It says in the README:

Please be aware that vfs-s3 is using custom build of commons-vfs2. It works in the same way as original commons-vfs2 but it was patched to fix some concurrency issues.

Which were those concurrency issues and why don't you create a pull-request in the main repo so that they can be fixed upstream? That way all users of VFS could benefit from them. :)