Coder Social home page Coder Social logo

sftp-fs's Introduction

sftp-fs

Maven Central Build Status Quality Gate Status Coverage Known Vulnerabilities

The sftp-fs library provides support for SFTP NIO.2 file systems, allowing SFTP servers to be accessed in a similar way to local file systems.

Creating file systems

If the SFTP file system library is available on the class path, it will register a FileSystemProvider for scheme sftp. This allows you to create SFTP file systems using the newFileSystem methods of class FileSystems. You can use class SFTPEnvironment to help create the environment maps for those methods:

SFTPEnvironment env = new SFTPEnvironment()
        .withUsername(username)
        .withUserInfo(userInfo);
FileSystem fs = FileSystems.newFileSystem(URI.create("sftp://example.org"), env);

Providing credentials

Credentials can be provided either through the URI or through the environment, as shown above. For security reasons the latter is preferred.

There are a few ways to provide credentials through the environment. The easiest way is to call withUserInfo on an SFTPEnvironment instance with a UserInfo object to provide the password, and if necessary the passphrase. The sftp-fs library provides class SimpleUserInfo to provide a fixed password and passphrase. Note that any prompt will be answered with "no". That means that you will most likely also need to set a HostKeyRepository by calling withHostKeyRepository on an SFTPEnvironment instance to verify the host.

If the UserInfo object will only provide a fixed password and do nothing else, you can also call withPassword on the SFTPEnvironment instance instead. This too most likely requires a HostKeyRepository to be set.

Instead of using a password, it's also possible to use other authentication methods using an IdentityRepository, which can be set by calling withIdentityRepository on an SFTPEnvironment instance. The following code snippet shows how you can use pageant to authenticate:

SFTPEnvironment env = new SFTPEnvironment()
        .withUsername(username)
        .withIdentityRepository(new AgentIdentityRepository(new PageantConnector()));
FileSystem fs = FileSystems.newFileSystem(URI.create("sftp://example.org"), env);

Default directory

The default directory can be provided through the URI or trough the environment using withDefaultDirectory, as follows:

URI path No default directory in the environment Default directory in the environment
None The default directory is defined by the SFTP server The default directory is defined by the environment
/ The default directory is / The default directory is defined by the environment
Other The default directory is equal to the URI path Not allowed

Creating paths

After a file system has been created, Paths can be created through the file system itself using its getPath method. As long as the file system is not closed, it's also possible to use Paths.get. Note that if the file system was created with credentials, the username must be part of the URL. For instance:

// created without credentials
Path path1 = Paths.get(URI.create("sftp://example.org"));
// created with credentials
Path path2 = Paths.get(URI.create("sftp://[email protected]"));

If the username in the URI does not match the username used to create the file system, or if no file system has been created for the URI, a new file system will be created. This works like Creating file systems. Since no environment can be provided this way, settings can still be provided through SFTPEnvironment.setDefault and query parameters; see usages of QueryParam and QueryParams for the possible query parameters. If creating a new file system fails, a FileSystemNotFoundException will be thrown.

Attributes

File attributes

SFTP file systems fully support read-access to the following attributes:

It's not possible to set the last access time or creation time, either through one of the file attribute views or through a file system. Attempting to do so will result in an UnsupportedOperationException. All other attributes are supported, although when setting the owner or group the name must be the UID/GID.

File store attributes

When calling getAttribute on a file store, the following attributes are supported:

These values are only supported by SFTP servers that support the [email protected] extension. If this extension is not supported, these methods will all return Long.MAX_VALUE.

There is no support for FileStoreAttributeView. Calling getFileStoreAttributeView on a file store will simply return null.

Error handling

Not all SFTP servers support the same set of error codes. Because of this, most methods may not throw the correct exception (NoSuchFileException, AccessDeniedException, etc).

To allow this behaviour to be modified, class SFTPEnvironment has method withFileSystemExceptionFactory that allows you to specify a custom FileSystemExceptionFactory implementation which will be used to create exceptions based on the reply code and string of the command that triggered the error. By default, an instance of class DefaultFileSystemExceptionFactory is used.

Thread safety

The SFTP protocol is fundamentally not thread safe. To overcome this limitation, SFTP file systems maintain multiple connections to SFTP servers. The number of connections determines the number of concurrent operations that can be executed. If all connections are busy, a new operation will block until a connection becomes available. Class SFTPEnvironment has method withPoolConfig that allows you to configure the connection pool:

  • The initial pool size - the number of connections that are created when an SFTP file system is created. The default is 1.
  • The maximum pool size - the maximum number of concurrent operations. The default is 5.
  • The maximum wait time - this determines how long to wait until a connection is available. The default is to wait indefinitely.
  • The maximum time that connections can be idle. The default is indefinitely.

When a stream or channel is opened for reading or writing, the connection will block because it will wait for the download or upload to finish. This will not occur until the stream or channel is closed. It is therefore advised to close streams and channels as soon as possible.

Connection management

Because SFTP file systems use multiple connections to an SFTP server, it's possible that one or more of these connections become stale. Class SFTPFileSystemProvider has static method keepAlive that, if given an instance of an SFTP file system, will send a keep-alive signal over each of its idle connections. You should ensure that this method is called on a regular interval. An alternative is to set a maximum idle time (see Thread safety).

Limitations

SFTP file systems knows the following limitations:

  • All paths use / as separator. / is not allowed inside file or directory names.
  • File attributes cannot be set when creating files or directories.
  • Symbolic links can be read and traversed, but not created.
  • There is no support for hard links.
  • Files can be marked as executable if the SFTP server indicates it is. That does not mean the file can be executed in the local JVM.
  • SeekableByteChannel is supported because it's used by Files.createFile. However, these channels do not support seeking specific positions or truncating.
  • FileSystem.getFileStores() will only return a FileStore for the root path, even if the SFTP server actually has several mount points.
  • Although FileSystemProvider.getFileStore(Path) will return a FileStore for the actual Path, its name will always be /, even if the file or directory is located on a different mount point.
  • There is no support for UserPrincipalLookupService.
  • There is no support for WatchService.

sftp-fs's People

Contributors

robtimus avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

sftp-fs's Issues

java.nio.file.NotDirectoryException if server is not returning . and ..

Files.list( path ) is throwing a java.nio.file.NotDirectoryException if the server is not returning the entries . and ...
This is because of the check in SFTPFileSystem.newDirectoryStream(...).

Is there a standard (?) way of forcing the server to return the required entries?
Could this check be removed or added a config switch to deactivate it?

Example:

try (FileSystem fs = FileSystems.newFileSystem( URI.create( "sftp://myhost" ), sftpEnv ))
{
    final Path dir = fs.getPath( "Test1" );
    System.out.println( Files.isRegularFile( dir ) );
    System.out.println( Files.isDirectory( dir ) );
    System.out.println( Files.isReadable( dir ) );
    System.out.println( Files.isWritable( dir ) );
    System.out.println( Files.exists( dir ) );

    Files.list( dir );
}

isRegularFile is returning false, next four true, but Files.list( dir ) is throwing NotDirectoryException.

Thanks

java.nio.file.FileSystemAlreadyExistsException

When create a new filesystem why can I not create 2 to the same host and username?

I have a batch program that will open and close the connections.
If I have 2 processes running and one closes it I'll get an exception.
If 1 is already running and another tries to open a filesystem I get the java.nio.file.FileSystemAlreadyExistsException: exception.

The logic does not make sense for me.

Files.createDirectories does not work as expected with SFTP FS

The implementation of Files.createDirectory for the SFTP file system does not throw a FileAlreadyExistsException if a folder already was created. This breaks the implementation of Files.createDirectories to be thrown. This is visible from the stack:

java.nio.file.FileSystemException: /some/folder: Failure
at com.github.robtimus.filesystems.sftp.DefaultFileSystemExceptionFactory.asFileSystemException(DefaultFileSystemExceptionFactory.java:132)
at com.github.robtimus.filesystems.sftp.DefaultFileSystemExceptionFactory.createCreateDirectoryException(DefaultFileSystemExceptionFactory.java:74)
at com.github.robtimus.filesystems.sftp.SSHChannelPool$Channel.mkdir(SSHChannelPool.java:516)
at com.github.robtimus.filesystems.sftp.SFTPFileSystem.createDirectory(SFTPFileSystem.java:381)
at com.github.robtimus.filesystems.sftp.SFTPPath.createDirectory(SFTPPath.java:175)
at com.github.robtimus.filesystems.sftp.SFTPFileSystemProvider.createDirectory(SFTPFileSystemProvider.java:262)
at java.nio.file.Files.createDirectory(Files.java:674)
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
at java.nio.file.Files.createDirectories(Files.java:767)
...
Caused by: com.jcraft.jsch.SftpException: Failure
at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2873)
at com.jcraft.jsch.ChannelSftp.mkdir(ChannelSftp.java:2182)
at com.github.robtimus.filesystems.sftp.SSHChannelPool$Channel.mkdir(SSHChannelPool.java:514)
... 17 common frames omitted

The problem is that this check is never triggered due to the wrong exception (FileSystemException instead of FileAlreadyExistsException) being thrown:

try {
  createDirectory(dir, attrs);
} catch (FileAlreadyExistsException x) {
  if (!isDirectory(dir, LinkOption.NOFOLLOW_LINKS))
    throw x;
}

It'd be nice if the connection pool was grown in parallel and/or grown on demand

At the moment connecting to a server is quite slow by default because each connection in the pool is added serially, and it defaults to five connections. So that's like logging in 5 times before you can do anything.

Is it possible to make the pool expand on demand so the need to manually configure the pool size and wait for it to be built can be fixed?

Paths.getPath() not working

To reproduce:

Paths.get(URI.create("sftp://somewhere.com));

Expected:
As per NIO FileSystem behaviour get() should create a file system and return a path

Actual:

java.nio.file.FileSystemNotFoundException: sftp://example.org
	at com.github.robtimus.filesystems.FileSystemMap.get(FileSystemMap.java:165)
	at com.github.robtimus.filesystems.sftp.SFTPFileSystemProvider.getExistingFileSystem(SFTPFileSystemProvider.java:135)
	at com.github.robtimus.filesystems.sftp.SFTPFileSystemProvider.getPath(SFTPFileSystemProvider.java:129)

Reading the readme I understand that the FS should be created before calling Paths.get(), but this does not make much sense IMHO for the following reasons:

  1. if we have already a FS then we can use it to get a path and Paths would be useless
  2. it forces to create a direct dependency between an app and stfp-fs, which is exactly what NIO FS (and Paths) tries to avoid
  3. an app is forced to have specific set up for this library even if never used.

Algorithm negotiation fail ssh-rsa

Hi,

trying to use your library and ran into issue upon initial connection:
"Algorithm negotiation fail: algorithmName="server_host_key" jschProposal="rsa-sha2-512,rsa-sha2-256,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521" serverProposal="ssh-rsa""

I'm using:

    .withUsername("...")
    .withUserInfo(SimpleUserInfo("...").toCharArray())
    .withKnownHosts(file)

Would appreciate some guidance on how to make this work.

Here is a full stack:

Exception in thread "main" java.nio.file.FileSystemException: Algorithm negotiation fail: algorithmName="server_host_key" jschProposal="rsa-sha2-512,rsa-sha2-256,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521" serverProposal="ssh-rsa"
	at com.github.robtimus.filesystems.sftp.SFTPEnvironment.asFileSystemException(SFTPEnvironment.java:730)
	at com.github.robtimus.filesystems.sftp.SFTPEnvironment.connect(SFTPEnvironment.java:655)
	at com.github.robtimus.filesystems.sftp.SFTPEnvironment.openChannel(SFTPEnvironment.java:526)
	at com.github.robtimus.filesystems.sftp.SSHChannelPool$Channel.<init>(SSHChannelPool.java:111)
	at com.github.robtimus.filesystems.sftp.SSHChannelPool$Channel.<init>(SSHChannelPool.java:106)
	at com.github.robtimus.filesystems.sftp.SSHChannelPool.lambda$new$0(SSHChannelPool.java:75)
	at com.github.robtimus.pool.Pool.createObject(Pool.java:382)
	at com.github.robtimus.pool.Pool.fillPool(Pool.java:100)
	at com.github.robtimus.pool.Pool.<init>(Pool.java:83)
	at com.github.robtimus.filesystems.sftp.SSHChannelPool.<init>(SSHChannelPool.java:75)
	at com.github.robtimus.filesystems.sftp.SFTPFileSystem.<init>(SFTPFileSystem.java:107)
	at com.github.robtimus.filesystems.sftp.SFTPFileSystemProvider.createFileSystem(SFTPFileSystemProvider.java:76)
	at com.github.robtimus.filesystems.FileSystemMap.createFileSystem(FileSystemMap.java:127)
	at com.github.robtimus.filesystems.FileSystemMap.add(FileSystemMap.java:108)
	at com.github.robtimus.filesystems.sftp.SFTPFileSystemProvider.newFileSystem(SFTPFileSystemProvider.java:99)
	at java.base/java.nio.file.FileSystems.newFileSystem(FileSystems.java:339)
	at java.base/java.nio.file.FileSystems.newFileSystem(FileSystems.java:288)
	at ...
Caused by: com.jcraft.jsch.JSchAlgoNegoFailException: Algorithm negotiation fail: algorithmName="server_host_key" jschProposal="rsa-sha2-512,rsa-sha2-256,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521" serverProposal="ssh-rsa"
	at com.jcraft.jsch.KeyExchange.guess(KeyExchange.java:147)
	at com.jcraft.jsch.Session.receive_kexinit(Session.java:602)
	at com.jcraft.jsch.Session.connect(Session.java:334)
	at com.jcraft.jsch.Session.connect(Session.java:194)
	at com.github.robtimus.filesystems.sftp.SFTPEnvironment.connect(SFTPEnvironment.java:650)
	... 17 more

java.nio.file.FileSystemException: java.net.ConnectException: Connection timed out: connect

I am getting a connection timed out error with the following code. I've included similar code using Jsch that works without issue. Not sure if this is an issue with my usage of sftp-fs or something with the library

String sftpHost = "sftp://ftp.server.com"; String user = "hello"; String pwd = "world"; char[] cPwd = pwd.toCharArray(); SFTPEnvironment env = new SFTPEnvironment() .withUsername(user).withPassword(cPwd).withConfig("StrictHostKeyChecking", "no"); FileSystem fs = FileSystems.newFileSystem(URI.create(sftpHost), env); Path p = fs.getPath("images"); System.out.println(p.getNameCount());

Note: I've tried appending port '22' to the end of the URI but that also fails with timed out error

The following code using straight Jsch (but not creating NIO file system) succeeds:

String SFTPHOST = "ftp.server.com"; String SFTPPORT = 22; String SFTPUSER = "hello"; String SFTPPASS = "world"; String SFTPBASEDIR = "images"; try { JSch jsch = new JSch(); session = jsch.getSession(SFTPUSER, SFTPHOST, SFTPPORT); session.setPassword(SFTPPASS); java.util.Properties config = new java.util.Properties(); config.put("StrictHostKeyChecking", "no"); session.setConfig(config); session.connect(); channel = session.openChannel("sftp"); channel.connect(); channelSftp = (ChannelSftp) channel; } catch (Exception ex) { _log.error("error connecting in init " + ex.getMessage(), ex); throw ex; }

Channel not being returned pool if underlying stream is already closed

Hello, I am experiencing an issue where our SSHChannelPool will slowly "leak" channels until the pool gets exhausted. I have been debugging this issue and it seems like a channel taken from the pool will not be returned to the pool if an SFTP input/output stream gets unexpectedly closed before we close it ourselves. (See [2]) This eventually leads to pool exhaustion.

I have been trying to understand why this happens and my hypothesis is that when we call SFTPOutputStream::close this will attempt to close the underlying output stream. This will cause flush to be called which in our case throws because the stream is already closed. (Why it is getting closed unexpectedly I have not managed to determine)

When this happens SFTPOutputStream::finalizeStream is never called which in turn causes the channel to never be returned to the pool. (See [1])

My immediate idea for a fix would be to make sure SFTPOutputStream::finalizeStream (and SFTPInputStream) always gets called even if an exception is thrown inside the method.

I can make the relevant changes for this in a PR, but I would like to know if my hypothesis sounds correct before doing the actual work. Any feedback would be appreciated. ๐Ÿ™‚

[1] Relevant code where a channel might not be returned to the pool if close throws:

public void close() throws IOException {
if (open) {
out.close();
open = false;
finalizeStream();
if (deleteOnClose) {
delete(path, false);
}
closedOutputStream(LOGGER, channelId, path);
}
}

[2] Stack-trace I believe causes a connection to be lost in the pool

java.io.IOException: Pipe closed at java.base/java.io.PipedInputStream.read(PipedInputStream.java:307) at java.base/java.io.PipedInputStream.read(PipedInputStream.java:377) at com.jcraft.jsch.ChannelSftp.fill(ChannelSftp.java:2909) at com.jcraft.jsch.ChannelSftp.header(ChannelSftp.java:2935) at com.jcraft.jsch.ChannelSftp.checkStatus(ChannelSftp.java:2473) at com.jcraft.jsch.ChannelSftp.access$300(ChannelSftp.java:36) at com.jcraft.jsch.ChannelSftp$1.flush(ChannelSftp.java:851) at com.jcraft.jsch.ChannelSftp$1.close(ChannelSftp.java:867) at com.github.robtimus.filesystems.sftp.SSHChannelPool$Channel$SFTPOutputStream.close(SSHChannelPool.java:460) at java.base/java.io.FilterOutputStream.close(FilterOutputStream.java:191) at no.finntech.adimport.ImportFtpService.closeOutFiles(ImportFtpService.java:82) at no.finntech.adimport.ImportFtpService.readXML(ImportFtpService.java:129) at no.finntech.adimport.ImportClient.processZipFile(ImportClient.java:177) at no.finntech.adimport.ImportClient.lambda$run$2(ImportClient.java:119) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at no.finntech.adimport.ImportClient.run(ImportClient.java:115) at no.finntech.adimport.ImportClient$$FastClassBySpringCGLIB$$75ceb0e.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) at java.base/java.lang.Thread.run(Thread.java:832) Suppressed: java.io.IOException: Pipe closed at java.base/java.io.PipedInputStream.read(PipedInputStream.java:307) at java.base/java.io.PipedInputStream.read(PipedInputStream.java:377) at com.jcraft.jsch.ChannelSftp.fill(ChannelSftp.java:2909) at com.jcraft.jsch.ChannelSftp.header(ChannelSftp.java:2935) at com.jcraft.jsch.ChannelSftp.checkStatus(ChannelSftp.java:2473) at com.jcraft.jsch.ChannelSftp.access$300(ChannelSftp.java:36) at com.jcraft.jsch.ChannelSftp$1.flush(ChannelSftp.java:851) at com.github.robtimus.filesystems.sftp.SSHChannelPool$Channel$SFTPOutputStream.flush(SSHChannelPool.java:454) at java.base/java.io.BufferedOutputStream.flush(BufferedOutputStream.java:143) at java.base/java.io.FilterOutputStream.close(FilterOutputStream.java:182) ... 16 more

Creating Identity from byte array instead of file

I am trying to create an Identity from a byte array rather than a file, as our security requirements are that the identities are fetched from Azure Key Vault and not local filesystems.

There does not appear to be a way to construct an Identity without using files, which seems to rule out the use of that class, leaving implementing the JSch IdentityRepository interface as the only option. However, the implementations of that interface rely on access to the JSch instance, which is not available when the SFTP environment is being created, and doesn't seem to be injected.

Is there a way around this?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.