net-ssh / net-sftp Goto Github PK

Push of files to a remote via ruby net-SFTP causes "Connection reset by peer - recvfrom(2)" error . Pulling of remote of files is fine.

Found out our routers intrusion detection and prevention was causing this error trying to stop this type of venerability: SSH_EVENT_RESPOVERFLOW.

This does effect Linux command line scp pushes too. scp pulls are also fine.

It seems to occur on larger file sizes.

I thought I would report this as an issue to help others debug "Connection reset by peer - recvfrom(2)".

Net::SFTP::Operations::Download fires the on_close event to indicate that a file has finished transfer before it closes the file handles - see L329

if response.eof?
    update_progress(:close, entry)
    entry.sink.close
    request = sftp.close(entry.handle, &method(:on_close))

On Windows at least, this makes it difficult to use the on_close event to set file modification time, as the operating system gives an access denied error if the local file is open. There may be other similar problems for other kinds of local file access during the download.

It would be great if the sequence of this code could change so that handles are closed before the event. However, I realise that might cause problems for existing third-party event handlers that expect the file to be open.

An alternative might be to allow the event handler to close the file handle if required - currently if you call entry.sink.close in the event handler there is then an IO error once control returns to Net:SFTP and it tries to close it again.

I'd be happy to submit a patch and tests if you indicate which approach you'd prefer.

Error message from bundler:

The gem net-ssh-2.9.1 can't be installed because the security policy didn't allow it, with the message: trusted root certificate
/CN=delano/DC=solutious/DC=com checksum does not match signing root certificate checksum

When using sftp.upload to upload a directory recursively, an exception is thrown if the destination directory(s) already exist.

Patched (only in my private copy) sftp/operations/upload.rb lines 162 and 302 to append:
|| response.code == 4 # where 4 appears to be the response code for: target directory already exists

This takes care of the problem, but I don't know the code base well enough to know if it should be done differently.

I.m using net-sftp gem v 2.0.5 on Windows. The gem was a fresh install yesterday.

Should this be handled differently? Or can someone appropriate please add these changes to the gem?

Thanks
Harrison

Using version 2.0.5 with net-ssh 2.6.3. OpenSSH_5.9p1, OpenSSL 0.9.8r 8 Feb 2011 (rMBP, ML).

I'm trying to list a directory with a funny UTF8 character such as 牌九 or Frosty ☃.

I connect:

@path = "/Users/me/牌九"
@handle = Net::SFTP.start(@server.host, @server.username, opts)
@entries = @handle.dir.entries(@path)

But the code never makes it past @handle.dir.entries. When I turn on :verbose => :debug for my Net::SFTP::start options, I see the following:

...
negotiating sftp protocol version, mine is 6
server reports sftp version 3
negotiated version is 3
sending opendir packet (0)
queueing packet nr 11 type 94 len 60
sent 84 bytes
read 140 bytes
received packet nr 12 type 96 len 12
channel_eof: 0
received packet nr 13 type 98 len 44
channel_request: 0 exit-status false
received packet nr 14 type 97 len 12
channel_close: 0
queueing packet nr 12 type 97 len 28
sftp channel closed
sent 52 bytes

And it stops here and stays here forever.

C:\Rails4Projects>gem cert --add gem-public_cert.pem
Added '/CN=delano/DC=solutious/DC=com'

C:\Rails4Projects>gem install net-sftp -P HighSecurity
Fetching: net-sftp-2.1.2.gem (100%)
ERROR: While executing gem ... (Gem::Security::Exception)
certificate /CN=delano/DC=solutious/DC=com not valid after 2014-02-06 11:57:45 UTC

Operations::File assumes that all operations start with the position at 0, when in append mode the assumption is that the position is EOF.

I'm trying to use :permissions with #mkdir but the results
are affected by umask and I can't see a way to set that via the API.

Client:

• ubuntu 10.10
• ruby 1.8.7 (stock ubuntu)
• net/ssh 2.1.4
• net/sftp 2.0.5

(net/sftp, net/ssh installed by rubygems, ubuntu stock libnet-ssh-ruby
and libnet-sftp-ruby packages not installed)

Server:

• local or remote ubuntu or solaris, stock sshd
• umask 027

Minimal test:

Net::SFTP.start( 'localhost', ENV['USER'] ) do |sftp|
  sftp.mkdir!( '/tmp/mydir', :permissions => 0560 )
end

Results:

0560 as above, gives:
dr-xr----- 2 martin martin 4096 2011-05-05 06:07 /tmp/mydir
(0540)

0666 gives:
drw-r----- 2 martin martin 4096 2011-05-05 06:09 /tmp/mydir
(0640)

0777 gives:
drwxr-x--- 2 martin martin 4096 2011-05-05 06:10 /tmp/mydir
(0750)

Hi,

The Dir#glob method has some wildcards that Net::SFTP::Operations::Dir#glob doesn't understand, there is plan to implement that features?

I wrote code that was working in pure ruby, but copied my code over to rails and now I get:

Encoding::UndefinedConversionError: "\xA8" from ASCII-8BIT to UTF-8

Ruby 2 and Rails 4

What can I change in my code to get this working?

  def self.get_recent_file(ftp_file, local_file)
    Net::SFTP.start(Config::A_FTP[:domain], Config::A_FTP[:username], :password => Config::A_FTP[:password]) do |sftp|
      sftp.download!(ftp_file, local_file)
    end
  end

Log

Encoding::UndefinedConversionError: "\xA8" from ASCII-8BIT to UTF-8
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-sftp-2.1.2/lib/net/sftp/operations/download.rb:339:in `write'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-sftp-2.1.2/lib/net/sftp/operations/download.rb:339:in `write'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-sftp-2.1.2/lib/net/sftp/operations/download.rb:339:in `on_read'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-sftp-2.1.2/lib/net/sftp/request.rb:87:in `call'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-sftp-2.1.2/lib/net/sftp/request.rb:87:in `respond_to'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-sftp-2.1.2/lib/net/sftp/session.rb:948:in `dispatch_request'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-sftp-2.1.2/lib/net/sftp/session.rb:911:in `when_channel_polled'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/channel.rb:311:in `call'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/channel.rb:311:in `process'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:222:in `block in preprocess'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:222:in `each'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:222:in `preprocess'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:205:in `process'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:169:in `block in loop'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:169:in `loop'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:169:in `loop'
... 13 levels...
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:222:in `preprocess'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:205:in `process'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:169:in `block in loop'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:169:in `loop'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:169:in `loop'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-sftp-2.1.2/lib/net/sftp/session.rb:802:in `loop'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-sftp-2.1.2/lib/net/sftp/session.rb:787:in `connect!'
    from /usr/local/opt/rbenv/versions/2.1.0-rc1/lib/ruby/gems/2.1.0/gems/net-sftp-2.1.2/lib/net/sftp.rb:32:in `start'

When trying to run the test suite in net-sftp-2.0.4 I get ~95 errors about readpartial being undefined. Here is an example output:

93. Error:
    test_upload_io_should_send_io_as_file(UploadTest):
    NoMethodError: undefined method readpartial' for #<Net::SSH::Test::Socket:0x7f770ee6f438> /usr/lib64/ruby/gems/1.8/gems/net-ssh-2.0.17/lib/net/ssh/transport/server_version.rb:48:innegotiate!'
    /usr/lib64/ruby/gems/1.8/gems/net-ssh-2.0.17/lib/net/ssh/transport/server_version.rb:45:in loop' /usr/lib64/ruby/gems/1.8/gems/net-ssh-2.0.17/lib/net/ssh/transport/server_version.rb:45:innegotiate!'
    /usr/lib64/ruby/gems/1.8/gems/net-ssh-2.0.17/lib/net/ssh/transport/server_version.rb:43:in loop' /usr/lib64/ruby/gems/1.8/gems/net-ssh-2.0.17/lib/net/ssh/transport/server_version.rb:43:innegotiate!'
    /usr/lib64/ruby/gems/1.8/gems/net-ssh-2.0.17/lib/net/ssh/transport/server_version.rb:32:in initialize' /usr/lib64/ruby/gems/1.8/gems/net-ssh-2.0.17/lib/net/ssh/transport/session.rb:75:innew'
    /usr/lib64/ruby/gems/1.8/gems/net-ssh-2.0.17/lib/net/ssh/transport/session.rb:75:in initialize' /usr/lib64/ruby/gems/1.8/gems/net-ssh-2.0.17/lib/net/ssh/test.rb:74:innew'
    /usr/lib64/ruby/gems/1.8/gems/net-ssh-2.0.17/lib/net/ssh/test.rb:74:in transport' /usr/lib64/ruby/gems/1.8/gems/net-ssh-2.0.17/lib/net/ssh/test.rb:67:inconnection'
    ./common.rb:39:in sftp' ./common.rb:54:inassert_scripted_command'
    /usr/lib64/ruby/gems/1.8/gems/net-ssh-2.0.17/lib/net/ssh/test.rb:84:in assert_scripted' ./common.rb:53:inassert_scripted_command'
    ./test_upload.rb:140:in test_upload_io_should_send_io_as_file' /usr/lib64/ruby/gems/1.8/gems/mocha-0.9.8/lib/mocha/integration/test_unit/ruby_version_186_and_above.rb:19:insend'
    /usr/lib64/ruby/gems/1.8/gems/mocha-0.9.8/lib/mocha/integration/test_unit/ruby_version_186_and_above.rb:19:in `run'

I noticed that there was a change regarding readpartial in net-ssh-2.0.16, perhaps this is related?

Hello, I cannot find a method to validate that the server which I open a connection is a known server.
Is it possible to add a method to get the fingerprint of the connected server (something like http://php.net/manual/en/function.ssh2-fingerprint.php ) ?

I've attached some sample code. I'm trying to get the exit status of commands, and also use the same connection to upload a file. You can see in the example that the command runs fine until I've used .sftp from the ssh connection. Also, if I do this after the sftp call it works:

connection.sftp.close_channel()
connection.instance_variable_set('@sftp', nil)

I'm thinking there's a way to keep both channels open (for the sftp and the command) and still have it work.

Thanks for the great library. Let me know if I can help track this one down.

require 'net/ssh'
require 'net/sftp'


class TestSSH
  def run_command(connection, command)
    exit_code = nil

    connection.open_channel do |channel|
      channel.exec(command) do |ch, success|
        unless success
          abort "FAILED: couldn't execute command (connection.channel.exec)"
        end

        channel.on_request("exit-status") do |ch,data|
          exit_code = data.read_long
        end
      end
    end
    puts "Loop on connection"
    connection.loop
    puts "Finished Loop"

    return exit_code
  end

  def test
    connection = Net::SSH.start('44dates.com', 'ubuntu')

    puts "Run ls"
    run_command(connection, "ls")

    puts "Upload File"
    io = StringIO.new("some text")
    connection.sftp.upload!(io, "somefile.txt")

    puts "Run ls Again"
    run_command(connection, "ls")
    puts "Finished Running ls again"

    connection.close
  end
end

TestSSH.new.test

Hello,
When calling abort! on an in-progress download(er), pending operations will crash the SFTP library.
example crash:

/usr/lib/ruby/gems/1.9.1/gems/net-sftp-2.0.5/lib/net/sftp/operations/download.rb:337:in `on_read': read /somefile.txt: invalid handle (9) (RuntimeError)
        from /usr/lib/ruby/gems/1.9.1/gems/net-sftp-2.0.5/lib/net/sftp/request.rb:87:in `call'
        from /usr/lib/ruby/gems/1.9.1/gems/net-sftp-2.0.5/lib/net/sftp/request.rb:87:in `respond_to'
        from /usr/lib/ruby/gems/1.9.1/gems/net-sftp-2.0.5/lib/net/sftp/session.rb:948:in `dispatch_request'
        from /usr/lib/ruby/gems/1.9.1/gems/net-sftp-2.0.5/lib/net/sftp/session.rb:911:in `when_channel_polled'
        from /usr/lib/ruby/gems/1.9.1/gems/net-ssh-2.0.23/lib/net/ssh/connection/channel.rb:311:in `call'
        from /usr/lib/ruby/gems/1.9.1/gems/net-ssh-2.0.23/lib/net/ssh/connection/channel.rb:311:in `process'
        from /usr/lib/ruby/gems/1.9.1/gems/net-ssh-2.0.23/lib/net/ssh/connection/session.rb:214:in `block in preprocess'
        from /usr/lib/ruby/gems/1.9.1/gems/net-ssh-2.0.23/lib/net/ssh/connection/session.rb:214:in `each'
        from /usr/lib/ruby/gems/1.9.1/gems/net-ssh-2.0.23/lib/net/ssh/connection/session.rb:214:in `preprocess'
        from /usr/lib/ruby/gems/1.9.1/gems/net-ssh-2.0.23/lib/net/ssh/connection/session.rb:197:in `process'
        from /usr/lib/ruby/gems/1.9.1/gems/net-ssh-2.0.23/lib/net/ssh/connection/session.rb:161:in `block in loop'
        from /usr/lib/ruby/gems/1.9.1/gems/net-ssh-2.0.23/lib/net/ssh/connection/session.rb:161:in `loop'
        from /usr/lib/ruby/gems/1.9.1/gems/net-ssh-2.0.23/lib/net/ssh/connection/session.rb:161:in `loop'
        from /usr/lib/ruby/gems/1.9.1/gems/net-sftp-2.0.5/lib/net/sftp/session.rb:802:in `loop'
        from /usr/lib/ruby/gems/1.9.1/gems/net-sftp-2.0.5/lib/net/sftp/request.rb:72:in `wait'
        from /usr/lib/ruby/gems/1.9.1/gems/net-sftp-2.0.5/lib/net/sftp/session.rb:842:in `wait_for'
        from /usr/lib/ruby/gems/1.9.1/gems/net-sftp-2.0.5/lib/net/sftp/session.rb:191:in `open!'
        from /usr/lib/ruby/gems/1.9.1/gems/net-sftp-2.0.5/lib/net/sftp/operations/file_factory.rb:40:in `open'
        from ./FTP_Mirror.rb:95:in `copy'
        from ./FTP_Mirror.rb:72:in `block in recurse'
        from /usr/lib/ruby/gems/1.9.1/gems/net-sftp-2.0.5/lib/net/sftp/operations/dir.rb:33:in `block in foreach'
        from /usr/lib/ruby/gems/1.9.1/gems/net-sftp-2.0.5/lib/net/sftp/operations/dir.rb:33:in `each'
        from /usr/lib/ruby/gems/1.9.1/gems/net-sftp-2.0.5/lib/net/sftp/operations/dir.rb:33:in `foreach'
        from ./FTP_Mirror.rb:59:in `recurse'
        from ./FTP_Mirror.rb:67:in `block in recurse'
        from /usr/lib/ruby/gems/1.9.1/gems/net-sftp-2.0.5/lib/net/sftp/operations/dir.rb:33:in `block in foreach'
        from /usr/lib/ruby/gems/1.9.1/gems/net-sftp-2.0.5/lib/net/sftp/operations/dir.rb:33:in `each'
        from /usr/lib/ruby/gems/1.9.1/gems/net-sftp-2.0.5/lib/net/sftp/operations/dir.rb:33:in `foreach'
        from ./FTP_Mirror.rb:59:in `recurse'
        from ./FTP_Mirror.rb:67:in `block in recurse'
        from /usr/lib/ruby/gems/1.9.1/gems/net-sftp-2.0.5/lib/net/sftp/operations/dir.rb:33:in `block in foreach'
        from /usr/lib/ruby/gems/1.9.1/gems/net-sftp-2.0.5/lib/net/sftp/operations/dir.rb:33:in `each'
        from /usr/lib/ruby/gems/1.9.1/gems/net-sftp-2.0.5/lib/net/sftp/operations/dir.rb:33:in `foreach'
        from ./FTP_Mirror.rb:59:in `recurse'
        from ./FTP_Mirror.rb:67:in `block in recurse'
        from /usr/lib/ruby/gems/1.9.1/gems/net-sftp-2.0.5/lib/net/sftp/operations/dir.rb:33:in `block in foreach'
        from /usr/lib/ruby/gems/1.9.1/gems/net-sftp-2.0.5/lib/net/sftp/operations/dir.rb:33:in `each'
        from /usr/lib/ruby/gems/1.9.1/gems/net-sftp-2.0.5/lib/net/sftp/operations/dir.rb:33:in `foreach'
        from ./FTP_Mirror.rb:59:in `recurse'
        from ./FTP_Mirror.rb:67:in `block in recurse'
        from /usr/lib/ruby/gems/1.9.1/gems/net-sftp-2.0.5/lib/net/sftp/operations/dir.rb:33:in `block in foreach'
        from /usr/lib/ruby/gems/1.9.1/gems/net-sftp-2.0.5/lib/net/sftp/operations/dir.rb:33:in `each'
        from /usr/lib/ruby/gems/1.9.1/gems/net-sftp-2.0.5/lib/net/sftp/operations/dir.rb:33:in `foreach'
        from ./FTP_Mirror.rb:59:in `recurse'
        from ./FTP_Mirror.rb:67:in `block in recurse'
        from /usr/lib/ruby/gems/1.9.1/gems/net-sftp-2.0.5/lib/net/sftp/operations/dir.rb:33:in `block in foreach'
        from /usr/lib/ruby/gems/1.9.1/gems/net-sftp-2.0.5/lib/net/sftp/operations/dir.rb:33:in `each'
        from /usr/lib/ruby/gems/1.9.1/gems/net-sftp-2.0.5/lib/net/sftp/operations/dir.rb:33:in `foreach'
        from ./FTP_Mirror.rb:59:in `recurse'
        from ./FTP_Mirror.rb:268:in `block (2 levels) in '
        from ./FTP_Mirror.rb:267:in `each'
        from ./FTP_Mirror.rb:267:in `block in '
        from ./FTP_Mirror.rb:263:in `each'
        from ./FTP_Mirror.rb:263:in `'

This was provoked thusly:

dl [email protected](src,dst) do |bla|
some block
end
...
dl.abort!
...

This small change seems to fix it:

-- /usr/lib/ruby/gems/1.9.1/gems/net-sftp-2.0.5/lib/net/sftp/operations/download.rb.orig       2010-10-01 15:59:15.237631526 -0700
+++ /usr/lib/ruby/gems/1.9.1/gems/net-sftp-2.0.5/lib/net/sftp/operations/download.rb    2010-10-01 18:20:19.946939769 -0700
@@ -325,6 +325,7 @@
       # and returned data, this will call #download_next_chunk to read the
       # next bit from the file. Otherwise the file will be closed.
       def on_read(response)
+        return unless active?
         entry = response.request[:entry]
 
         if response.eof?

on_read is called but the response.request for the aborted transfer has already been cleared by the call to abort! (I think)

One of our clients is experiencing a production bug where net-sftp is looping indefinitely. We believe this is because the server to which they're connecting stops receiving data, but doesn't close the connection.

I'd like to submit a PR with a configurable time-out, but I note that net-sftp is in maintenance mode. @delano, would you in principle be interested in such a PR?

Hello Im trying to upload a file with a forwardtick on the filename, the upload proccess works but on the remote server the forwardtick is changed by an interrogation symbol

Example
Local file name: The Posesión.pdf
Remote Result: The Posesi?n.pdf

http://stackoverflow.com/questions/28049446/ruby-net-sftp-changes-the-forwardtick-for-an-interrogation-symbol-on-the-remote

I am trying to list directory, but it displayed as junk characters. However the same site working without any issues with perl, php
Sample code:
Net::SFTP.start(host, username, :password => password, :port => 22) do |sftp|
sftp.dir.foreach(folder) do |entry|
next if (entry.name == '..' || entry.name == '.')
puts entry.name
end
end
I am getting like this:
�epubtesepubtest��Y��dY���Y����..���epubtesepubtestA�Y��Y�\�Y�I`�

If I code entry.longname
I get exception like this:
/home/dev/.rvm/gems/ruby-2.2.1/gems/net-sftp-2.1.2/lib/net/sftp/protocol/04/name.rb:59:in %': can't convert nil into Integer (TypeError) from /home/dev/.rvm/gems/ruby-2.2.1/gems/net-sftp-2.1.2/lib/net/sftp/protocol/04/name.rb:59:in longname'
from sftp_test1.rb:34:in block (2 levels) in <main>' from /home/dev/.rvm/gems/ruby-2.2.1/gems/net-sftp-2.1.2/lib/net/sftp/operations/dir.rb:33:in block in foreach'
from /home/dev/.rvm/gems/ruby-2.2.1/gems/net-sftp-2.1.2/lib/net/sftp/operations/dir.rb:33:in each' from /home/dev/.rvm/gems/ruby-2.2.1/gems/net-sftp-2.1.2/lib/net/sftp/operations/dir.rb:33:in foreach'
from sftp_test1.rb:31:in block in <main>' from /home/dev/.rvm/gems/ruby-2.2.1/gems/net-sftp-2.1.2/lib/net/sftp/session.rb:939:in call'
from /home/dev/.rvm/gems/ruby-2.2.1/gems/net-sftp-2.1.2/lib/net/sftp/session.rb:939:in block in do_version' from /home/dev/.rvm/gems/ruby-2.2.1/gems/net-sftp-2.1.2/lib/net/sftp/session.rb:939:in each'
from /home/dev/.rvm/gems/ruby-2.2.1/gems/net-sftp-2.1.2/lib/net/sftp/session.rb:939:in do_version' from /home/dev/.rvm/gems/ruby-2.2.1/gems/net-sftp-2.1.2/lib/net/sftp/session.rb:909:in when_channel_polled'
from /home/dev/.rvm/gems/ruby-2.2.1/gems/net-ssh-4.1.0/lib/net/ssh/connection/channel.rb:320:in call' from /home/dev/.rvm/gems/ruby-2.2.1/gems/net-ssh-4.1.0/lib/net/ssh/connection/channel.rb:320:in process'
from /home/dev/.rvm/gems/ruby-2.2.1/gems/net-ssh-4.1.0/lib/net/ssh/connection/session.rb:238:in block in ev_preprocess' from /home/dev/.rvm/gems/ruby-2.2.1/gems/net-ssh-4.1.0/lib/net/ssh/connection/session.rb:529:in each'
from /home/dev/.rvm/gems/ruby-2.2.1/gems/net-ssh-4.1.0/lib/net/ssh/connection/session.rb:529:in each_channel' from /home/dev/.rvm/gems/ruby-2.2.1/gems/net-ssh-4.1.0/lib/net/ssh/connection/session.rb:238:in ev_preprocess'
from /home/dev/.rvm/gems/ruby-2.2.1/gems/net-ssh-4.1.0/lib/net/ssh/connection/event_loop.rb:99:in each' from /home/dev/.rvm/gems/ruby-2.2.1/gems/net-ssh-4.1.0/lib/net/ssh/connection/event_loop.rb:99:in ev_preprocess'
from /home/dev/.rvm/gems/ruby-2.2.1/gems/net-ssh-4.1.0/lib/net/ssh/connection/event_loop.rb:27:in process' from /home/dev/.rvm/gems/ruby-2.2.1/gems/net-ssh-4.1.0/lib/net/ssh/connection/session.rb:216:in process'
from /home/dev/.rvm/gems/ruby-2.2.1/gems/net-ssh-4.1.0/lib/net/ssh/connection/session.rb:178:in block in loop' from /home/dev/.rvm/gems/ruby-2.2.1/gems/net-ssh-4.1.0/lib/net/ssh/connection/session.rb:178:in loop'
from /home/dev/.rvm/gems/ruby-2.2.1/gems/net-ssh-4.1.0/lib/net/ssh/connection/session.rb:178:in loop' from /home/dev/.rvm/gems/ruby-2.2.1/gems/net-sftp-2.1.2/lib/net/sftp/session.rb:802:in loop'
from /home/dev/.rvm/gems/ruby-2.2.1/gems/net-sftp-2.1.2/lib/net/sftp/session.rb:787:in connect!' from /home/dev/.rvm/gems/ruby-2.2.1/gems/net-sftp-2.1.2/lib/net/sftp.rb:32:in start'
from sftp_test1.rb:29:in `

Installing v2.1.3-rc1 using Bundler results in the following error

$ bundle
Fetching gem metadata from https://rubygems.org/.........
Fetching https://github.com/net-ssh/net-sftp.git

[!] There was an error while loading `net-sftp.gemspec`: cannot load such file -- net/ssh/version
Does it try to require a relative path? That's been removed in Ruby 1.9. Bundler cannot continue.

 #  from /Users/tomlubitz/.rvm/gems/ruby-2.5.1@transmogrifier/bundler/gems/net-sftp-13cba0e0d801/net-sftp.gemspec:1
 #  -------------------------------------------
 >  require_relative 'lib/net/sftp/version'
 #  
 #  -------------------------------------------

Digging into the code, it looks like net/sftp/version is requiring net/ssh/version, which may not be part of the load path yet. It's only guaranteed to be in the load path after the gem's dependencies are installed, and the gemspec needs to be loaded in order to determine which dependencies to install.

I think the solution is to:

1. use $LOAD_PATH.unshift File.expand_path("../lib", __FILE__) instead of require_relative, though this may be a red herring. this is how it works in e.g.rspec.gemspec.
2. don't require net/ssh/version inside net/sftp/version. This means Net::SFTP::Version can't be a subclass of Net::SSH::Version ¯_(ツ)_/¯

I'm not familiar enough with the history of this repo to open a PR without more information, but I may be able to make the above changes if they sound good.

tagging @mfazekas since it looks like the issue was introduced in 4ffe058

I'd like to stream a file from SFTP to AWS S3. Currently, this is not supported because Net::SFTP::Operations::File is not a real IO.

AWS checks for IO-like objects for S3 operations (source) like so:

def io_like?(value)
  value.respond_to?(:read) &&
  value.respond_to?(:rewind) &&
  value.respond_to?(:size)
end

Would there be any objection and/or issues with me submitting a PR to add read and size methods to Net::SFTP::Operations::File?

This would allow users to stream a file directly to S3 with something like:

s3_client = Aws::S3::Client.new

Net::SFTP.start('host', 'username', :password => 'password') do |sftp|
  sftp.file.open("/path/to/remote", "r") do |f|
    s3_client.put_object(
      bucket: config[:bucket],
      key: 'file.txt',
      body: f
    )
  end
end

In my opinion, this would be a useful feature to have.

For some reason, the SSH session initialized in the sftp.start method raises an exception when it's closed at line 36. Everything else I do in the session (downloading files, etc) appears to work fine. It just blows up on the session.close.

I'm running this against a WingFTP server, on Ubuntu 12.10 w/ OpenSSL version OpenSSL '1.0.1c 10 May 2012' and the SSH protocol version that's being picked up is V1.

Here's a simple script that blows up:

ruby
require 'net/sftp'

Net::SFTP.start("localhost", "user", :password=>"password") do |sftp|
sftp.dir.foreach(".") do |entry|
puts entry.name
end
end

$ openssl x509 -text -in gem-public_cert.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 0 (0x0)
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: CN=delano, DC=solutious, DC=com
        Validity
            Not Before: Feb  6 11:57:45 2013 GMT
            Not After : Feb  6 11:57:45 2014 GMT
        Subject: CN=delano, DC=solutious, DC=com
... <snip> ...

Certificate expired about two years ago, it seems. Consider updating it to allow secure installs of the net-sftp gem?

Hi there,

I wondered how I can connect to an SFTP Server and specify my keys (custom location). I first thought it isn't possible but then discovered that you can piggy-back on a net-ssh connection. net-ssh allows to define a location for the keys so I ended up with:

Net::SSH.start('server', 'seekda', {keys: ['mykeyfile'], passphrase: 'pass'}) do |ssh|
ssh.sftp.connect do |sftp|
sftp.dir.foreach(".") do |entry|
puts entry.longname
end
end
end

Is there a way to do this with SFTP directly?
I think it would make sense to add it to the documentation otherwise as many people are looking for it.

Thanks,
Christoph

I'm having a problem connecting to an SFTP server using Net::SFTP.

The code is fairly simple:

Net::SFTP.start(ftp_host, user_name, :keys => [], :verbose => :debug, :password => password) do |sftp|
  sftp.upload!(file, remotefile)
end

In the debug output, I get the following:

establishing connection to sftp.server.com:22
connection established
negotiating protocol version
remote is SSH-2.0-OpenSSH_4.2-chrootssh' local isSSH-2.0-Ruby/Net::SSH_2.5.2 x86_64-darwin11.4.1'
read 704 bytes
received packet nr 0 type 20 len 700
got KEXINIT from server
sending KEXINIT
queueing packet nr 0 type 20 len 1620
sent 1624 bytes
negotiating algorithms
negotiated:
kex: diffie-hellman-group-exchange-sha1
host_key: ssh-rsa
encryption_server: aes128-cbc
encryption_client: aes128-cbc
hmac_client: hmac-sha1
hmac_server: hmac-sha1
compression_client: none
compression_server: none
language_client:
language_server:
exchanging keys
queueing packet nr 1 type 34 len 20
sent 24 bytes
read 152 bytes
received packet nr 1 type 31 len 148
queueing packet nr 2 type 32 len 140
sent 144 bytes
read 720 bytes
received packet nr 2 type 33 len 700
queueing packet nr 3 type 21 len 20
sent 24 bytes
received packet nr 3 type 21 len 12
beginning authentication of `sftpusername'
queueing packet nr 4 type 5 len 28
sent 52 bytes
read 52 bytes
received packet nr 4 type 6 len 28
trying publickey
connecting to ssh-agent
sending agent request 1 len 51
received agent packet 2 len 5
sending agent request 11 len 0
received agent packet 12 len 5
trying hostbased
sending agent request 11 len 0
received agent packet 12 len 5
trying password
queueing packet nr 5 type 50 len 92
sent 116 bytes
read 36 bytes
received packet nr 5 type 52 len 12
password succeeded
queueing packet nr 6 type 90 len 44
sent 68 bytes
read 52 bytes
received packet nr 6 type 91 len 28
channel_open_confirmation: 0 0 0 32768
requesting sftp subsystem
sending channel request "subsystem"
queueing packet nr 7 type 98 len 44
sent 68 bytes
read 88 bytes
received packet nr 7 type 93 len 28
channel_window_adjust: 0 +131072
received packet nr 8 type 99 len 12
channel_success: 0
sftp subsystem successfully started
queueing packet nr 8 type 94 len 28
sent 52 bytes
read 224 bytes
received packet nr 9 type 94 len 60
channel_data: 0 41b
received packet nr 10 type 98 len 44
channel_request: 0 exit-status false
received packet nr 11 type 96 len 12
channel_eof: 0
received packet nr 12 type 97 len 12
channel_close: 0
queueing packet nr 9 type 97 len 28
sftp channel closed
forcing connection closed

I believe the issue is in negotiating the SFTP Protocol version, after "sftp subsystem successfully started". If anyone can point me to how I can go about debugging further to get enough information to solve this problem, I'd be happy to submit a patch.

Hey!

I just created a sftp server docker image with alpine for my integration tests:
https://github.com/CheckoutFinland/docker-dummy-sftp-server

I then tried to use glob file matching like this:

def fetch_sftp
  Net::SFTP.start(uri.host, uri.user, port: uri.port, key_data: [private_key], keys: [], keys_only: true) do |sftp|
      sftp.dir.glob(uri.path, '*.csv') do |remote_file|
           puts remote_file
      end
  end
end

Which raises weird looking:

Traceback (most recent call last):
        2: from (irb):5
        1: from test.rb:45:in `block in fetch_sftp'
Net::SFTP::StatusException (Net::SFTP::StatusException)

I found out that this probably happens because alpine linux uses musl-c library which doesn't support glob.

I'm not sure if we can deduce if glob is supported from the sftp connection or not but this error message could be better. At least I posted this issue here so that others can find it with google.

Could you please navigate me how to debug this issue. Happens in production not on local server

When trying the tests from current master with ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-linux] I'm getting a number of test failures. With ruby 1.8 I'm also seeing failures but the test suite appears to hang altogether. I'm seeing similar failures with the released 2.0.5.

  1) Failure:
test_from_buffer_should_correctly_parse_buffer_and_return_attribute_object(Protocol::V04::TestAttributes) [/home/graaff/playground/github-clones/net-sftp/test/protocol/04/test_attributes.rb:15]:
<9> expected but was
<5>.

  2) Failure:
test_from_buffer_should_correctly_parse_buffer_with_attribute_subset_and_return_attribute_object(Protocol::V04::TestAttributes) [/home/graaff/playground/github-clones/net-sftp/test/protocol/04/test_attributes.rb:47]:
<1> expected but was
<5>.

  3) Failure:
test_parse_name_packet_should_use_correct_name_class(Protocol::V04::TestBase) [/home/graaff/playground/github-clones/net-sftp/test/protocol/04/test_base.rb:25]:
<493> expected but was
<16777217>.

  4) Failure:
test_parse_name_packet_should_use_correct_name_class(Protocol::V05::TestBase) [/home/graaff/playground/github-clones/net-sftp/test/protocol/04/test_base.rb:25]:
<493> expected but was
<16777217>.

  5) Failure:
test_from_buffer_should_correctly_parse_buffer_and_return_attribute_object(Protocol::V06::TestAttributes) [/home/graaff/playground/github-clones/net-sftp/test/protocol/06/test_attributes.rb:9]:
<9> expected but was
<5>.

  6) Failure:
test_from_buffer_should_correctly_parse_buffer_with_attribute_subset_and_return_attribute_object(Protocol::V06::TestAttributes) [/home/graaff/playground/github-clones/net-sftp/test/protocol/06/test_attributes.rb:51]:
<1> expected but was
<5>.

  7) Failure:
test_parse_name_packet_should_use_correct_name_class(Protocol::V06::TestBase) [/home/graaff/playground/github-clones/net-sftp/test/protocol/04/test_base.rb:25]:
<493> expected but was
<16777217>.

  8) Error:
test_upload_directory_should_mirror_directory_structure_remotely(UploadTest):
RuntimeError: expected string "\x00\x00\x00\x1C\x0E\x00\x00\x00\x00\x00\x00\x00\x0F/path/to/remote\x00\x00\x00\x00" but got "\x00\x00\x00&\x03\x00\x00\x00\x00\x00\x00\x00\x15/path/to/remote/file1\x00\x00\x00\x1A\x00\x00\x00\x00"
    /usr/lib64/ruby/gems/1.9.1/gems/net-ssh-2.4.0/lib/net/ssh/test/local_packet.rb:46:in `block in process'
    /usr/lib64/ruby/gems/1.9.1/gems/net-ssh-2.4.0/lib/net/ssh/test/local_packet.rb:36:in `each'
    /usr/lib64/ruby/gems/1.9.1/gems/net-ssh-2.4.0/lib/net/ssh/test/local_packet.rb:36:in `process'
    /usr/lib64/ruby/gems/1.9.1/gems/net-ssh-2.4.0/lib/net/ssh/test/script.rb:153:in `process'
    /usr/lib64/ruby/gems/1.9.1/gems/net-ssh-2.4.0/lib/net/ssh/test/extensions.rb:79:in `test_enqueue_packet'
    /usr/lib64/ruby/gems/1.9.1/gems/net-ssh-2.4.0/lib/net/ssh/transport/session.rb:225:in `enqueue_message'
    /usr/lib64/ruby/gems/1.9.1/gems/net-ssh-2.4.0/lib/net/ssh/connection/session.rb:368:in `send_message'
    /usr/lib64/ruby/gems/1.9.1/gems/net-ssh-2.4.0/lib/net/ssh/connection/channel.rb:493:in `enqueue_pending_output'
    /usr/lib64/ruby/gems/1.9.1/gems/net-ssh-2.4.0/lib/net/ssh/test/extensions.rb:105:in `send_data_for_test'
    /home/graaff/playground/github-clones/net-sftp/lib/net/sftp/session.rb:814:in `send_packet'
    /home/graaff/playground/github-clones/net-sftp/lib/net/sftp/protocol/base.rb:45:in `send_request'
    /home/graaff/playground/github-clones/net-sftp/lib/net/sftp/protocol/01/base.rb:90:in `open'
    /home/graaff/playground/github-clones/net-sftp/lib/net/sftp/session.rb:830:in `request'
    /home/graaff/playground/github-clones/net-sftp/lib/net/sftp/session.rb:182:in `open'
    /home/graaff/playground/github-clones/net-sftp/lib/net/sftp/operations/upload.rb:300:in `open_file'
    /home/graaff/playground/github-clones/net-sftp/lib/net/sftp/operations/upload.rb:271:in `process_next_entry'
    /home/graaff/playground/github-clones/net-sftp/lib/net/sftp/operations/upload.rb:173:in `initialize'
    /home/graaff/playground/github-clones/net-sftp/lib/net/sftp/session.rb:98:in `new'
    /home/graaff/playground/github-clones/net-sftp/lib/net/sftp/session.rb:98:in `upload'
    /home/graaff/playground/github-clones/net-sftp/test/test_upload.rb:113:in `block in test_upload_directory_should_mirror_directory_structure_remotely'
    /home/graaff/playground/github-clones/net-sftp/test/common.rb:56:in `block in assert_scripted_command'
    /usr/lib64/ruby/gems/1.9.1/gems/net-ssh-2.4.0/lib/net/ssh/test.rb:84:in `assert_scripted'
    /home/graaff/playground/github-clones/net-sftp/test/common.rb:54:in `assert_scripted_command'
    /home/graaff/playground/github-clones/net-sftp/test/test_upload.rb:112:in `test_upload_directory_should_mirror_directory_structure_remotely'
    /usr/lib64/ruby/gems/1.9.1/gems/mocha-0.12.3/lib/mocha/integration/mini_test/version_230_to_2101.rb:28:in `run'

  9) Error:
test_upload_directory_with_handler_should_report_progress(UploadTest):
RuntimeError: expected string "\x00\x00\x00\x1C\x0E\x00\x00\x00\x00\x00\x00\x00\x0F/path/to/remote\x00\x00\x00\x00" but got "\x00\x00\x00&\x03\x00\x00\x00\x00\x00\x00\x00\x15/path/to/remote/file1\x00\x00\x00\x1A\x00\x00\x00\x00"
    /usr/lib64/ruby/gems/1.9.1/gems/net-ssh-2.4.0/lib/net/ssh/test/local_packet.rb:46:in `block in process'
    /usr/lib64/ruby/gems/1.9.1/gems/net-ssh-2.4.0/lib/net/ssh/test/local_packet.rb:36:in `each'
    /usr/lib64/ruby/gems/1.9.1/gems/net-ssh-2.4.0/lib/net/ssh/test/local_packet.rb:36:in `process'
    /usr/lib64/ruby/gems/1.9.1/gems/net-ssh-2.4.0/lib/net/ssh/test/script.rb:153:in `process'
    /usr/lib64/ruby/gems/1.9.1/gems/net-ssh-2.4.0/lib/net/ssh/test/extensions.rb:79:in `test_enqueue_packet'
    /usr/lib64/ruby/gems/1.9.1/gems/net-ssh-2.4.0/lib/net/ssh/transport/session.rb:225:in `enqueue_message'
    /usr/lib64/ruby/gems/1.9.1/gems/net-ssh-2.4.0/lib/net/ssh/connection/session.rb:368:in `send_message'
    /usr/lib64/ruby/gems/1.9.1/gems/net-ssh-2.4.0/lib/net/ssh/connection/channel.rb:493:in `enqueue_pending_output'
    /usr/lib64/ruby/gems/1.9.1/gems/net-ssh-2.4.0/lib/net/ssh/test/extensions.rb:105:in `send_data_for_test'
    /home/graaff/playground/github-clones/net-sftp/lib/net/sftp/session.rb:814:in `send_packet'
    /home/graaff/playground/github-clones/net-sftp/lib/net/sftp/protocol/base.rb:45:in `send_request'
    /home/graaff/playground/github-clones/net-sftp/lib/net/sftp/protocol/01/base.rb:90:in `open'
    /home/graaff/playground/github-clones/net-sftp/lib/net/sftp/session.rb:830:in `request'
    /home/graaff/playground/github-clones/net-sftp/lib/net/sftp/session.rb:182:in `open'
    /home/graaff/playground/github-clones/net-sftp/lib/net/sftp/operations/upload.rb:300:in `open_file'
    /home/graaff/playground/github-clones/net-sftp/lib/net/sftp/operations/upload.rb:271:in `process_next_entry'
    /home/graaff/playground/github-clones/net-sftp/lib/net/sftp/operations/upload.rb:173:in `initialize'
    /home/graaff/playground/github-clones/net-sftp/lib/net/sftp/session.rb:98:in `new'
    /home/graaff/playground/github-clones/net-sftp/lib/net/sftp/session.rb:98:in `upload'
    /home/graaff/playground/github-clones/net-sftp/test/test_upload.rb:121:in `block in test_upload_directory_with_handler_should_report_progress'
    /home/graaff/playground/github-clones/net-sftp/test/common.rb:56:in `block in assert_scripted_command'
    /usr/lib64/ruby/gems/1.9.1/gems/net-ssh-2.4.0/lib/net/ssh/test.rb:84:in `assert_scripted'
    /home/graaff/playground/github-clones/net-sftp/test/common.rb:54:in `assert_scripted_command'
    /home/graaff/playground/github-clones/net-sftp/test/test_upload.rb:120:in `test_upload_directory_with_handler_should_report_progress'
    /usr/lib64/ruby/gems/1.9.1/gems/mocha-0.12.3/lib/mocha/integration/mini_test/version_230_to_2101.rb:28:in `run'

419 tests, 1026 assertions, 7 failures, 2 errors, 0 skips

When doing:
sftp.setstat!("/remote/path", :owner => "user", :group => "group")

the translation of "user" to uid and of "group" to gid is done using the local /etc/passwd.
This causes errors if a user/group exists on the local machine, but not on the remote one.
(probably having a user with the same name, but different uid is even worse)

Uploading (recursively) a directory tree containing files and subdirectories fails when an empty subdirectory is encountered. In the error below, I'm uploading a "bare clone" of a new git repo. The bare clone contains files and folders, one of which, "branches" is empty.

Net::SFTP::StatusException (Net::SFTP::StatusException mkdir /sftp_repo_testing/test.git/branches (2, "no such file"))
/Users/a028541/.rvm/gems/ree-1.8.7-2012.02/gems/net-sftp-2.1.2/lib/net/sftp/operations/upload.rb:310:in on_mkdir' /Users/a028541/.rvm/gems/ree-1.8.7-2012.02/gems/net-sftp-2.1.2/lib/net/sftp/operations/upload.rb:307:into_proc'
/Users/a028541/.rvm/gems/ree-1.8.7-2012.02/gems/net-sftp-2.1.2/lib/net/sftp/request.rb:87:in call' /Users/a028541/.rvm/gems/ree-1.8.7-2012.02/gems/net-sftp-2.1.2/lib/net/sftp/request.rb:87:inrespond_to'
/Users/a028541/.rvm/gems/ree-1.8.7-2012.02/gems/net-sftp-2.1.2/lib/net/sftp/session.rb:948:in dispatch_request' /Users/a028541/.rvm/gems/ree-1.8.7-2012.02/gems/net-sftp-2.1.2/lib/net/sftp/session.rb:911:inwhen_channel_polled'
/Users/a028541/.rvm/gems/ree-1.8.7-2012.02/gems/net-sftp-2.1.2/lib/net/sftp/session.rb:893:in to_proc' /Users/a028541/.rvm/gems/ree-1.8.7-2012.02/gems/net-ssh-2.9.1/lib/net/ssh/connection/channel.rb:311:incall'
/Users/a028541/.rvm/gems/ree-1.8.7-2012.02/gems/net-ssh-2.9.1/lib/net/ssh/connection/channel.rb:311:in process' /Users/a028541/.rvm/gems/ree-1.8.7-2012.02/gems/net-ssh-2.9.1/lib/net/ssh/connection/session.rb:222:inpreprocess'
/Users/a028541/.rvm/gems/ree-1.8.7-2012.02/gems/net-ssh-2.9.1/lib/net/ssh/connection/session.rb:222:in each' /Users/a028541/.rvm/gems/ree-1.8.7-2012.02/gems/net-ssh-2.9.1/lib/net/ssh/connection/session.rb:222:inpreprocess'
/Users/a028541/.rvm/gems/ree-1.8.7-2012.02/gems/net-ssh-2.9.1/lib/net/ssh/connection/session.rb:205:in process' /Users/a028541/.rvm/gems/ree-1.8.7-2012.02/gems/net-ssh-2.9.1/lib/net/ssh/connection/session.rb:169:inloop'
/Users/a028541/.rvm/gems/ree-1.8.7-2012.02/gems/net-ssh-2.9.1/lib/net/ssh/connection/session.rb:169:in loop_forever' /Users/a028541/.rvm/gems/ree-1.8.7-2012.02/gems/net-ssh-2.9.1/lib/net/ssh/connection/session.rb:169:inloop'
/Users/a028541/.rvm/gems/ree-1.8.7-2012.02/gems/net-sftp-2.1.2/lib/net/sftp/session.rb:802:in loop' /Users/a028541/.rvm/gems/ree-1.8.7-2012.02/gems/net-sftp-2.1.2/lib/net/sftp/operations/upload.rb:203:inwait'
/Users/a028541/.rvm/gems/ree-1.8.7-2012.02/gems/net-sftp-2.1.2/lib/net/sftp/session.rb:103:in upload!' app/models/course.rb:243:indo_git_operations'
app/controllers/courses_controller.rb:225:in generate_html' /Users/a028541/.rvm/rubies/ree-1.8.7-2012.02/lib/ruby/1.8/webrick/httpserver.rb:104:inservice'
/Users/a028541/.rvm/rubies/ree-1.8.7-2012.02/lib/ruby/1.8/webrick/httpserver.rb:65:in run' /Users/a028541/.rvm/rubies/ree-1.8.7-2012.02/lib/ruby/1.8/webrick/server.rb:173:instart_thread'
/Users/a028541/.rvm/rubies/ree-1.8.7-2012.02/lib/ruby/1.8/webrick/server.rb:162:in `start'
/Users/a028541/.rvm/rubies/ree-1.8.7-2012.02/lib/ruby/1.8/webrick/server.rb:162

from: http://github.com/jamis/net-sftp/issues/closed/#issue/1
by: appoxy

I ran into this problem with net-sftp:

I catch the Exception from net-sftp and when I try to call ex.message, I get a can't modify frozen string exception.

Code:

begin
    ... something with net-sftp 
rescue Exception => ex
    puts 'CAUGHT:' + ex.message 
end

Which in turn throws another exception due to the ex.message:

Exception Message:
can't modify frozen string

Backtrace:

/usr/local/lib/ruby/gems/1.9.1/gems/net-sftp-2.0.2/lib/net/sftp/errors.rb:35:in message' run.rb:168:inrescue in block in run'
run.rb:184:in `block in run'

Confirmed that this fixes it:

def message
  m = super.dup
  m << " #{text}" if text
  m << " (#{code}, #{description.inspect})"
end

$ gem install net-sftp -P HighSecurity

Fetching: net-ssh-3.2.0.gem (100%)
ERROR: While executing gem ... (Gem::Security::Exception)
root cert /CN=net-ssh/DC=solutious/DC=com is not trusted

NOTE: I ended up with corrupted data because of this issue...

When I use Net::SFTP::Operations::Download with defaults my downloads take dramatically longer than if I use the command line Ubuntu sftp transfer.

I figured if I upped the :read_size it would go faster. The doc says:

:read_size - the maximum number of bytes to read at a time from the source. Increasing this value might improve throughput. It defaults to 32,000 bytes.

Well I set it to some big numbers (like 104857600) and it did indeed go MUCH faster. But it also skipped much of the data without any warning. I lost some significant data as I didn't realize that the transfered data was corrupted until later and deleted the source files.

It looks like the software updates the offset by the :read_size value but the data returned in the my SftpDownloadProgressHandler#on_get where all I have is:

    log.debug "writing #{data.length} bytes to #{file.local} starting at #{offset}"

Here is a sample of the output

D, [2013-03-06 03:51:10#16632] DEBUG -- : writing 65536 bytes to puchases.csv.gz starting at 0
D, [2013-03-06 03:51:10#16632] DEBUG -- : writing 65536 bytes to puchases.csv.gz starting at 104857600
D, [2013-03-06 03:51:10#16632] DEBUG -- : writing 65536 bytes to puchases.csv.gz starting at 209715200
D, [2013-03-06 03:51:10#16632] DEBUG -- : writing 65536 bytes to puchases.csv.gz starting at 314572800
...

The downloaded file was obviously corrupted.

I presume that somewhere, maybe on the server side, there is an enforced max read size of 65536 or the value is an unsigned int16 or something lame and ancient...

So unless I'm doing something wrong, at a minimum the Net::SFTP::Operations::Download and probably Net::SFTP::Operations::Upload documentation for :read_size should have a warning.

Better yet would be if somewhere in the code it could notice that if its not eof and the response data.length is less than the :read_size, it threw an exception or adapted its :read_size or somehow let the user know....

Do let me know if I am misunderstanding something or otherwise crazy!

Thanks!

HI!
Version:

ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-linux]
net-sftp (2.0.5)
net-ssh (2.6.1)

1.9.3-p194 :031 > ne.sftp.download!("/home/msgplus/p_bak", "/home/msgplus/pp") {|event, dw, *args| 
  case event 
  when :get  then 
    p args[0]; 
  end;
} 
# <struct Net::SFTP::Operations::Download::Entry remote="/home/msgplus/p_bak",

local="/home/msgplus/pp", directory=nil, size=nil, handle="\x00\x00\x00\x03", offset=32000, sink=#
File:/home/msgplus/pp>

1.9.3-p194 :032 > ne.sftp.upload!("/home/msgplus/p_bak", "/home/msgplus/pp") {|event, up, *args|
  case event 
  when :put then 
    p args[0]; 
  end;
} 

# <struct Net::SFTP::Operations::Upload::LiveFile local="/home/msgplus/p_bak",

remote="/home/msgplus/pp", io=#File:/home/msgplus/p_bak, size=7971, 
handle="\x00\x00\x00\x03">

io = sftp.file.open claims to produce an IO-like object. Unfortunately, it doesn't seem to be quite io-like enough for ruby's built-in CSV library. If I do CSV.new(io), I get an error:

ArgumentError: wrong number of arguments (given 2, expected 0..1)
from /Users/kevin/.rvm/gems/ruby-2.3.1/gems/net-sftp-2.1.2/lib/net/sftp/operations/file.rb:85:in `gets'

Tracing this through, it looks like CSV is ultimately trying to call io.get(separator, limit) (which is supported in the full io spec, but sftp's io-like object only allows one argument.

It's pretty simple to reproduce, but here's a test case anyway:

require 'net/sftp'
require 'csv'

Net::SFTP.start(
  '0.0.0.0',
  '...',
  password: '...',
  port: 2222
) do |sftp|
  filepath = './upload/test_file.csv'
  stream = sftp.file.open(filepath)
  csv = CSV.new(stream) # Errors here: wrong number of arguments (given 2, expected 0..1)
end

This isn't an urgent bug for me— I'll just ditch streams entirely and read the whole (small) file into memory before parsing it :). Just thought I'd log this in case someone else runs into the same issue (or in case I'm missing something and it's not a bug).

Hi,

I am using this code:
Net::SFTP.start(backup.hostname, backup.username, :password => password, :port => backup.port) do |sftp|
sftp.upload!(filename, backup.path + "/" + File.basename(filename))
end

to upload a file to an external server.

The file is 33 MB in size

The uploading uses 32 GB !!!! and often 100% CPU time (it finished after a while, but ruins my server)

Any idea why ?

net-sftp (2.0.5)
net-ssh (2.6.1)

Linux Debian 64 Bit

I added some debugging in the code

sftp.upload!(filename, backup.path + "/" + File.basename(filename)) do |event, uploader, *args|

      case event
        when :open then
          # args[0] : file metadata
          logger.error "starting upload: #{args[0].local} -> #{args[0].remote} (#{args[0].size} bytes}"
          logger.error("memory= #{MemoryBenchmark.memory}")
        when :put then
          # args[0] : file metadata
          # args[1] : byte offset in remote file
          # args[2] : data being written (as string)
          logger.error "writing #{args[2].length} bytes to #{args[0].remote} starting at #{args[1]}"
          logger.error("memory= #{MemoryBenchmark.memory}")
        when :close then
          # args[0] : file metadata
          logger.error "finished with #{args[0].remote}"
          logger.error("memory= #{MemoryBenchmark.memory}")
        when :mkdir then
          # args[0] : remote path name
          logger.error "creating directory #{args[0]}"
          logger.error("memory= #{MemoryBenchmark.memory}")
        when :finish then
          logger.error "all done!"
          logger.error("memory= #{MemoryBenchmark.memory}")
      end

    end

and MemoryBenchmark does this:

def self.memory
return ps -o rss= -p #{Process.pid}.to_i
end

which results in this:
also the upload is getting slower and slower

The memory is only the RSS the virtual is nearly twice as high and I stopped on 17MB upload
So nearly 3GB
And its exponential growing!

memory= 240928
writing 32000 bytes to /test.zip starting at 0
memory= 240932
writing 32000 bytes to /test.zip starting at 32000
memory= 240932
writing 32000 bytes to /test.zip starting at 64000
memory= 240932
writing 32000 bytes to /test.zip starting at 96000
memory= 240932
writing 32000 bytes to /test.zip starting at 128000
memory= 240932
writing 32000 bytes to /test.zip starting at 160000
memory= 240932
writing 32000 bytes to /test.zip starting at 192000
memory= 240932
writing 32000 bytes to /test.zip starting at 224000
memory= 240932
writing 32000 bytes to /test.zip starting at 256000
memory= 240932
writing 32000 bytes to /test.zip starting at 288000
memory= 240932
writing 32000 bytes to /test.zip starting at 320000
memory= 240932
writing 32000 bytes to /test.zip starting at 352000
memory= 240932
writing 32000 bytes to /test.zip starting at 384000
memory= 240932
writing 32000 bytes to /test.zip starting at 416000
memory= 240932
writing 32000 bytes to /test.zip starting at 448000
memory= 240932
writing 32000 bytes to /test.zip starting at 480000
memory= 240932
writing 32000 bytes to /test.zip starting at 512000
memory= 240932
writing 32000 bytes to /test.zip starting at 544000
memory= 240932
writing 32000 bytes to /test.zip starting at 576000
memory= 240932
writing 32000 bytes to /test.zip starting at 608000
memory= 240932
writing 32000 bytes to /test.zip starting at 640000
memory= 240932
writing 32000 bytes to /test.zip starting at 672000
memory= 240932
writing 32000 bytes to /test.zip starting at 704000
memory= 240932
writing 32000 bytes to /test.zip starting at 736000
memory= 240932
writing 32000 bytes to /test.zip starting at 768000
memory= 240932
writing 32000 bytes to /test.zip starting at 800000
memory= 240932
writing 32000 bytes to /test.zip starting at 832000
memory= 240932
writing 32000 bytes to /test.zip starting at 864000
memory= 240932
writing 32000 bytes to /test.zip starting at 896000
memory= 240932
writing 32000 bytes to /test.zip starting at 928000
memory= 240932
writing 32000 bytes to /test.zip starting at 960000
memory= 240932
writing 32000 bytes to /test.zip starting at 992000
memory= 240932
writing 32000 bytes to /test.zip starting at 1024000
memory= 240932
writing 32000 bytes to /test.zip starting at 1056000
memory= 240932
writing 32000 bytes to /test.zip starting at 1088000
memory= 240932
writing 32000 bytes to /test.zip starting at 1120000
memory= 240932
writing 32000 bytes to /test.zip starting at 1152000
memory= 240932
writing 32000 bytes to /test.zip starting at 1184000
memory= 240932
writing 32000 bytes to /test.zip starting at 1216000
memory= 240932
writing 32000 bytes to /test.zip starting at 1248000
memory= 240932
writing 32000 bytes to /test.zip starting at 1280000
memory= 240932
writing 32000 bytes to /test.zip starting at 1312000
memory= 240932
writing 32000 bytes to /test.zip starting at 1344000
memory= 240932
writing 32000 bytes to /test.zip starting at 1376000
memory= 240932
writing 32000 bytes to /test.zip starting at 1408000
memory= 240956
writing 32000 bytes to /test.zip starting at 1440000
memory= 240956
writing 32000 bytes to /test.zip starting at 1472000
memory= 240956
writing 32000 bytes to /test.zip starting at 1504000
memory= 240956
writing 32000 bytes to /test.zip starting at 1536000
memory= 240956
writing 32000 bytes to /test.zip starting at 1568000
memory= 240956
writing 32000 bytes to /test.zip starting at 1600000
memory= 240956
writing 32000 bytes to /test.zip starting at 1632000
memory= 240956
writing 32000 bytes to /test.zip starting at 1664000
memory= 240956
writing 32000 bytes to /test.zip starting at 1696000
memory= 240956
writing 32000 bytes to /test.zip starting at 1728000
memory= 240956
writing 32000 bytes to /test.zip starting at 1760000
memory= 240956
writing 32000 bytes to /test.zip starting at 1792000
memory= 240956
writing 32000 bytes to /test.zip starting at 1824000
memory= 240956
writing 32000 bytes to /test.zip starting at 1856000
memory= 240956
writing 32000 bytes to /test.zip starting at 1888000
memory= 240956
writing 32000 bytes to /test.zip starting at 1920000
memory= 240956
writing 32000 bytes to /test.zip starting at 1952000
memory= 240956
writing 32000 bytes to /test.zip starting at 1984000
memory= 240956
writing 32000 bytes to /test.zip starting at 2016000
memory= 240956
writing 32000 bytes to /test.zip starting at 2048000
memory= 240956
writing 32000 bytes to /test.zip starting at 2080000
memory= 242964
writing 32000 bytes to /test.zip starting at 2112000
memory= 242964
writing 32000 bytes to /test.zip starting at 2144000
memory= 242964
writing 32000 bytes to /test.zip starting at 2176000
memory= 242964
writing 32000 bytes to /test.zip starting at 2208000
memory= 243152
writing 32000 bytes to /test.zip starting at 2240000
memory= 245316
writing 32000 bytes to /test.zip starting at 2272000
memory= 245316
writing 32000 bytes to /test.zip starting at 2304000
memory= 245316
writing 32000 bytes to /test.zip starting at 2336000
memory= 245452
writing 32000 bytes to /test.zip starting at 2368000
memory= 245452
writing 32000 bytes to /test.zip starting at 2400000
memory= 245452
writing 32000 bytes to /test.zip starting at 2432000
memory= 245628
writing 32000 bytes to /test.zip starting at 2464000
memory= 245748
writing 32000 bytes to /test.zip starting at 2496000
memory= 245748
writing 32000 bytes to /test.zip starting at 2528000
memory= 248192
writing 32000 bytes to /test.zip starting at 2560000
memory= 248192
writing 32000 bytes to /test.zip starting at 2592000
memory= 248192
writing 32000 bytes to /test.zip starting at 2624000
memory= 248536
writing 32000 bytes to /test.zip starting at 2656000
memory= 248536
writing 32000 bytes to /test.zip starting at 2688000
memory= 248536
writing 32000 bytes to /test.zip starting at 2720000
memory= 248536
writing 32000 bytes to /test.zip starting at 2752000
memory= 248536
writing 32000 bytes to /test.zip starting at 2784000
memory= 248536
writing 32000 bytes to /test.zip starting at 2816000
memory= 248536
writing 32000 bytes to /test.zip starting at 2848000
memory= 248536
writing 32000 bytes to /test.zip starting at 2880000
memory= 248536
writing 32000 bytes to /test.zip starting at 2912000
memory= 248536
writing 32000 bytes to /test.zip starting at 2944000
memory= 248536
writing 32000 bytes to /test.zip starting at 2976000
memory= 248536
writing 32000 bytes to /test.zip starting at 3008000
memory= 248536
writing 32000 bytes to /test.zip starting at 3040000
memory= 248536
writing 32000 bytes to /test.zip starting at 3072000
memory= 248536
writing 32000 bytes to /test.zip starting at 3104000
memory= 248536
writing 32000 bytes to /test.zip starting at 3136000
memory= 248536
writing 32000 bytes to /test.zip starting at 3168000
memory= 248536
writing 32000 bytes to /test.zip starting at 3200000
memory= 248536
writing 32000 bytes to /test.zip starting at 3232000
memory= 249256
writing 32000 bytes to /test.zip starting at 3264000
memory= 249288
writing 32000 bytes to /test.zip starting at 3296000
memory= 249288
writing 32000 bytes to /test.zip starting at 3328000
memory= 249348
writing 32000 bytes to /test.zip starting at 3360000
memory= 249368
writing 32000 bytes to /test.zip starting at 3392000
memory= 249368
writing 32000 bytes to /test.zip starting at 3424000
memory= 249444
writing 32000 bytes to /test.zip starting at 3456000
memory= 249476
writing 32000 bytes to /test.zip starting at 3488000
memory= 249476
writing 32000 bytes to /test.zip starting at 3520000
memory= 249528
writing 32000 bytes to /test.zip starting at 3552000
memory= 249568
writing 32000 bytes to /test.zip starting at 3584000
memory= 249600
writing 32000 bytes to /test.zip starting at 3616000
memory= 249600
writing 32000 bytes to /test.zip starting at 3648000
memory= 249664
writing 32000 bytes to /test.zip starting at 3680000
memory= 249696
writing 32000 bytes to /test.zip starting at 3712000
memory= 249696
writing 32000 bytes to /test.zip starting at 3744000
memory= 249756
writing 32000 bytes to /test.zip starting at 3776000
memory= 249760
writing 32000 bytes to /test.zip starting at 3808000
memory= 249760
writing 32000 bytes to /test.zip starting at 3840000
memory= 249852
writing 32000 bytes to /test.zip starting at 3872000
memory= 249884
writing 32000 bytes to /test.zip starting at 3904000
memory= 249884
writing 32000 bytes to /test.zip starting at 3936000
memory= 249920
writing 32000 bytes to /test.zip starting at 3968000
memory= 249976
writing 32000 bytes to /test.zip starting at 4000000
memory= 250008
writing 32000 bytes to /test.zip starting at 4032000
memory= 250008
writing 32000 bytes to /test.zip starting at 4064000
memory= 250068
writing 32000 bytes to /test.zip starting at 4096000
memory= 250080
writing 32000 bytes to /test.zip starting at 4128000
memory= 250080
writing 32000 bytes to /test.zip starting at 4160000
memory= 250164
writing 32000 bytes to /test.zip starting at 4192000
memory= 250196
writing 32000 bytes to /test.zip starting at 4224000
memory= 250196
writing 32000 bytes to /test.zip starting at 4256000
memory= 250240
writing 32000 bytes to /test.zip starting at 4288000
memory= 250288
writing 32000 bytes to /test.zip starting at 4320000
memory= 250320
writing 32000 bytes to /test.zip starting at 4352000
memory= 250320
writing 32000 bytes to /test.zip starting at 4384000
memory= 250384
writing 32000 bytes to /test.zip starting at 4416000
memory= 250400
writing 32000 bytes to /test.zip starting at 4448000
memory= 250400
writing 32000 bytes to /test.zip starting at 4480000
memory= 250464
writing 32000 bytes to /test.zip starting at 4512000
memory= 250464
writing 32000 bytes to /test.zip starting at 4544000
memory= 250464
writing 32000 bytes to /test.zip starting at 4576000
memory= 250464
writing 32000 bytes to /test.zip starting at 4608000
memory= 250464
writing 32000 bytes to /test.zip starting at 4640000
memory= 250464
writing 32000 bytes to /test.zip starting at 4672000
memory= 250464
writing 32000 bytes to /test.zip starting at 4704000
memory= 250464
writing 32000 bytes to /test.zip starting at 4736000
memory= 250464
writing 32000 bytes to /test.zip starting at 4768000
memory= 250464
writing 32000 bytes to /test.zip starting at 4800000
memory= 250784
writing 32000 bytes to /test.zip starting at 4832000
memory= 250788
writing 32000 bytes to /test.zip starting at 4864000
memory= 250788
writing 32000 bytes to /test.zip starting at 4896000
memory= 250852
writing 32000 bytes to /test.zip starting at 4928000
memory= 250912
writing 32000 bytes to /test.zip starting at 4960000
memory= 250916
writing 32000 bytes to /test.zip starting at 4992000
memory= 250948
writing 32000 bytes to /test.zip starting at 5024000
memory= 250948
writing 32000 bytes to /test.zip starting at 5056000
memory= 251008
writing 32000 bytes to /test.zip starting at 5088000
memory= 251040
writing 32000 bytes to /test.zip starting at 5120000
memory= 251040
writing 32000 bytes to /test.zip starting at 5152000
memory= 251104
writing 32000 bytes to /test.zip starting at 5184000
memory= 251136
writing 32000 bytes to /test.zip starting at 5216000
memory= 251164
writing 32000 bytes to /test.zip starting at 5248000
memory= 251164
writing 32000 bytes to /test.zip starting at 5280000
memory= 251292
writing 32000 bytes to /test.zip starting at 5312000
memory= 251328
writing 32000 bytes to /test.zip starting at 5344000
memory= 251328
writing 32000 bytes to /test.zip starting at 5376000
memory= 251484
writing 32000 bytes to /test.zip starting at 5408000
memory= 251540
writing 32000 bytes to /test.zip starting at 5440000
memory= 251604
writing 32000 bytes to /test.zip starting at 5472000
memory= 251604
writing 32000 bytes to /test.zip starting at 5504000
memory= 251732
writing 32000 bytes to /test.zip starting at 5536000
memory= 251792
writing 32000 bytes to /test.zip starting at 5568000
memory= 251824
writing 32000 bytes to /test.zip starting at 5600000
memory= 251824
writing 32000 bytes to /test.zip starting at 5632000
memory= 251984
writing 32000 bytes to /test.zip starting at 5664000
memory= 252044
writing 32000 bytes to /test.zip starting at 5696000
memory= 252044
writing 32000 bytes to /test.zip starting at 5728000
memory= 252144
writing 32000 bytes to /test.zip starting at 5760000
memory= 252236
writing 32000 bytes to /test.zip starting at 5792000
memory= 252292
writing 32000 bytes to /test.zip starting at 5824000
memory= 252292
writing 32000 bytes to /test.zip starting at 5856000
memory= 252420
writing 32000 bytes to /test.zip starting at 5888000
memory= 252460
writing 32000 bytes to /test.zip starting at 5920000
memory= 252460
writing 32000 bytes to /test.zip starting at 5952000
memory= 252616
writing 32000 bytes to /test.zip starting at 5984000
memory= 252668
writing 32000 bytes to /test.zip starting at 6016000
memory= 252668
writing 32000 bytes to /test.zip starting at 6048000
memory= 258668
writing 32000 bytes to /test.zip starting at 6080000
memory= 270584
writing 32000 bytes to /test.zip starting at 6112000
memory= 276732
writing 32000 bytes to /test.zip starting at 6144000
memory= 276764
writing 32000 bytes to /test.zip starting at 6176000
memory= 276764
writing 32000 bytes to /test.zip starting at 6208000
memory= 277064
writing 32000 bytes to /test.zip starting at 6240000
memory= 277336
writing 32000 bytes to /test.zip starting at 6272000
memory= 277552
writing 32000 bytes to /test.zip starting at 6304000
memory= 277552
writing 32000 bytes to /test.zip starting at 6336000
memory= 277796
writing 32000 bytes to /test.zip starting at 6368000
memory= 277880
writing 32000 bytes to /test.zip starting at 6400000
memory= 277880
writing 32000 bytes to /test.zip starting at 6432000
memory= 278124
writing 32000 bytes to /test.zip starting at 6464000
memory= 278172
writing 32000 bytes to /test.zip starting at 6496000
memory= 278172
writing 32000 bytes to /test.zip starting at 6528000
memory= 278352
writing 32000 bytes to /test.zip starting at 6560000
memory= 278504
writing 32000 bytes to /test.zip starting at 6592000
memory= 278548
writing 32000 bytes to /test.zip starting at 6624000
memory= 278548
writing 32000 bytes to /test.zip starting at 6656000
memory= 278736
writing 32000 bytes to /test.zip starting at 6688000
memory= 278824
writing 32000 bytes to /test.zip starting at 6720000
memory= 278824
writing 32000 bytes to /test.zip starting at 6752000
memory= 279076
writing 32000 bytes to /test.zip starting at 6784000
memory= 279112
writing 32000 bytes to /test.zip starting at 6816000
memory= 279112
writing 32000 bytes to /test.zip starting at 6848000
memory= 279300
writing 32000 bytes to /test.zip starting at 6880000
memory= 279392
writing 32000 bytes to /test.zip starting at 6912000
memory= 279392
writing 32000 bytes to /test.zip starting at 6944000
memory= 279544
writing 32000 bytes to /test.zip starting at 6976000
memory= 279624
writing 32000 bytes to /test.zip starting at 7008000
memory= 279600
writing 32000 bytes to /test.zip starting at 7040000
memory= 279600
writing 32000 bytes to /test.zip starting at 7072000
memory= 280496
writing 32000 bytes to /test.zip starting at 7104000
memory= 280492
writing 32000 bytes to /test.zip starting at 7136000
memory= 280492
writing 32000 bytes to /test.zip starting at 7168000
memory= 280480
writing 32000 bytes to /test.zip starting at 7200000
memory= 280504
writing 32000 bytes to /test.zip starting at 7232000
memory= 280504
writing 32000 bytes to /test.zip starting at 7264000
memory= 280516
writing 32000 bytes to /test.zip starting at 7296000
memory= 280504
writing 32000 bytes to /test.zip starting at 7328000
memory= 280464
writing 32000 bytes to /test.zip starting at 7360000
memory= 280464
writing 32000 bytes to /test.zip starting at 7392000
memory= 280464
writing 32000 bytes to /test.zip starting at 7424000
memory= 280492
writing 32000 bytes to /test.zip starting at 7456000
memory= 280492
writing 32000 bytes to /test.zip starting at 7488000
memory= 280504
writing 32000 bytes to /test.zip starting at 7520000
memory= 280448
writing 32000 bytes to /test.zip starting at 7552000
memory= 280448
writing 32000 bytes to /test.zip starting at 7584000
memory= 280436
writing 32000 bytes to /test.zip starting at 7616000
memory= 280532
writing 32000 bytes to /test.zip starting at 7648000
memory= 280464
writing 32000 bytes to /test.zip starting at 7680000
memory= 280464
writing 32000 bytes to /test.zip starting at 7712000
memory= 280528
writing 32000 bytes to /test.zip starting at 7744000
memory= 280476
writing 32000 bytes to /test.zip starting at 7776000
memory= 280476
writing 32000 bytes to /test.zip starting at 7808000
memory= 280472
writing 32000 bytes to /test.zip starting at 7840000
memory= 280480
writing 32000 bytes to /test.zip starting at 7872000
memory= 280480
writing 32000 bytes to /test.zip starting at 7904000
memory= 280448
writing 32000 bytes to /test.zip starting at 7936000
memory= 280504
writing 32000 bytes to /test.zip starting at 7968000
memory= 280404
writing 32000 bytes to /test.zip starting at 8000000
memory= 280404
writing 32000 bytes to /test.zip starting at 8032000
memory= 280468
writing 32000 bytes to /test.zip starting at 8064000
memory= 280500
writing 32000 bytes to /test.zip starting at 8096000
memory= 280500
writing 32000 bytes to /test.zip starting at 8128000
memory= 280484
writing 32000 bytes to /test.zip starting at 8160000
memory= 280372
writing 32000 bytes to /test.zip starting at 8192000
memory= 280372
writing 32000 bytes to /test.zip starting at 8224000
memory= 280436
writing 32000 bytes to /test.zip starting at 8256000
memory= 280468
writing 32000 bytes to /test.zip starting at 8288000
memory= 280468
writing 32000 bytes to /test.zip starting at 8320000
memory= 280416
writing 32000 bytes to /test.zip starting at 8352000
memory= 280448
writing 32000 bytes to /test.zip starting at 8384000
memory= 280356
writing 32000 bytes to /test.zip starting at 8416000
memory= 280356
writing 32000 bytes to /test.zip starting at 8448000
memory= 280420
writing 32000 bytes to /test.zip starting at 8480000
memory= 280452
writing 32000 bytes to /test.zip starting at 8512000
memory= 280452
writing 32000 bytes to /test.zip starting at 8544000
memory= 280484
writing 32000 bytes to /test.zip starting at 8576000
memory= 280508
writing 32000 bytes to /test.zip starting at 8608000
memory= 280508
writing 32000 bytes to /test.zip starting at 8640000
memory= 280444
writing 32000 bytes to /test.zip starting at 8672000
memory= 280484
writing 32000 bytes to /test.zip starting at 8704000
memory= 280372
writing 32000 bytes to /test.zip starting at 8736000
memory= 280372
writing 32000 bytes to /test.zip starting at 8768000
memory= 280436
writing 32000 bytes to /test.zip starting at 8800000
memory= 280468
writing 32000 bytes to /test.zip starting at 8832000
memory= 280468
writing 32000 bytes to /test.zip starting at 8864000
memory= 280404
writing 32000 bytes to /test.zip starting at 8896000
memory= 280420
writing 32000 bytes to /test.zip starting at 8928000
memory= 280420
writing 32000 bytes to /test.zip starting at 8960000
memory= 280340
writing 32000 bytes to /test.zip starting at 8992000
memory= 280388
writing 32000 bytes to /test.zip starting at 9024000
memory= 280248
writing 32000 bytes to /test.zip starting at 9056000
memory= 280276
writing 32000 bytes to /test.zip starting at 9088000
memory= 280276
writing 32000 bytes to /test.zip starting at 9120000
memory= 280340
writing 32000 bytes to /test.zip starting at 9152000
memory= 280200
writing 32000 bytes to /test.zip starting at 9184000
memory= 280212
writing 32000 bytes to /test.zip starting at 9216000
memory= 280212
writing 32000 bytes to /test.zip starting at 9248000
memory= 280276
writing 32000 bytes to /test.zip starting at 9280000
memory= 280304
writing 32000 bytes to /test.zip starting at 9312000
memory= 280304
writing 32000 bytes to /test.zip starting at 9344000
memory= 279928
writing 32000 bytes to /test.zip starting at 9376000
memory= 279936
writing 32000 bytes to /test.zip starting at 9408000
memory= 279936
writing 32000 bytes to /test.zip starting at 9440000
memory= 280000
writing 32000 bytes to /test.zip starting at 9472000
memory= 280056
writing 32000 bytes to /test.zip starting at 9504000
memory= 280060
writing 32000 bytes to /test.zip starting at 9536000
memory= 280060
writing 32000 bytes to /test.zip starting at 9568000
memory= 280124
writing 32000 bytes to /test.zip starting at 9600000
memory= 280156
writing 32000 bytes to /test.zip starting at 9632000
memory= 280156
writing 32000 bytes to /test.zip starting at 9664000
memory= 278476
writing 32000 bytes to /test.zip starting at 9696000
memory= 278508
writing 32000 bytes to /test.zip starting at 9728000
memory= 278508
writing 32000 bytes to /test.zip starting at 9760000
memory= 278576
writing 32000 bytes to /test.zip starting at 9792000
memory= 278600
writing 32000 bytes to /test.zip starting at 9824000
memory= 278600
writing 32000 bytes to /test.zip starting at 9856000
memory= 278664
writing 32000 bytes to /test.zip starting at 9888000
memory= 278704
writing 32000 bytes to /test.zip starting at 9920000
memory= 278728
writing 32000 bytes to /test.zip starting at 9952000
memory= 278728
writing 32000 bytes to /test.zip starting at 9984000
memory= 278792
writing 32000 bytes to /test.zip starting at 10016000
memory= 278820
writing 32000 bytes to /test.zip starting at 10048000
memory= 278820
writing 32000 bytes to /test.zip starting at 10080000
memory= 278896
writing 32000 bytes to /test.zip starting at 10112000
memory= 278916
writing 32000 bytes to /test.zip starting at 10144000
memory= 278916
writing 32000 bytes to /test.zip starting at 10176000
memory= 278980
writing 32000 bytes to /test.zip starting at 10208000
memory= 279024
writing 32000 bytes to /test.zip starting at 10240000
memory= 279040
writing 32000 bytes to /test.zip starting at 10272000
memory= 279040
writing 32000 bytes to /test.zip starting at 10304000
memory= 279104
writing 32000 bytes to /test.zip starting at 10336000
memory= 279156
writing 32000 bytes to /test.zip starting at 10368000
memory= 279156
writing 32000 bytes to /test.zip starting at 10400000
memory= 279236
writing 32000 bytes to /test.zip starting at 10432000
memory= 279248
writing 32000 bytes to /test.zip starting at 10464000
memory= 279248
writing 32000 bytes to /test.zip starting at 10496000
memory= 283460
writing 32000 bytes to /test.zip starting at 10528000
memory= 283596
writing 32000 bytes to /test.zip starting at 10560000
memory= 283636
writing 32000 bytes to /test.zip starting at 10592000
memory= 283636
writing 32000 bytes to /test.zip starting at 10624000
memory= 283764
writing 32000 bytes to /test.zip starting at 10656000
memory= 283820
writing 32000 bytes to /test.zip starting at 10688000
memory= 283820
writing 32000 bytes to /test.zip starting at 10720000
memory= 283976
writing 32000 bytes to /test.zip starting at 10752000
memory= 284012
writing 32000 bytes to /test.zip starting at 10784000
memory= 284012
writing 32000 bytes to /test.zip starting at 10816000
memory= 284132
writing 32000 bytes to /test.zip starting at 10848000
memory= 284172
writing 32000 bytes to /test.zip starting at 10880000
memory= 284180
writing 32000 bytes to /test.zip starting at 10912000
memory= 284180
writing 32000 bytes to /test.zip starting at 10944000
memory= 284244
writing 32000 bytes to /test.zip starting at 10976000
memory= 284272
writing 32000 bytes to /test.zip starting at 11008000
memory= 284272
writing 32000 bytes to /test.zip starting at 11040000
memory= 284336
writing 32000 bytes to /test.zip starting at 11072000
memory= 284340
writing 32000 bytes to /test.zip starting at 11104000
memory= 284340
writing 32000 bytes to /test.zip starting at 11136000
memory= 284380
writing 32000 bytes to /test.zip starting at 11168000
memory= 284376
writing 32000 bytes to /test.zip starting at 11200000
memory= 284376
writing 32000 bytes to /test.zip starting at 11232000
memory= 284408
writing 32000 bytes to /test.zip starting at 11264000
memory= 284428
writing 32000 bytes to /test.zip starting at 11296000
memory= 284484
writing 32000 bytes to /test.zip starting at 11328000
memory= 284484
writing 32000 bytes to /test.zip starting at 11360000
memory= 284536
writing 32000 bytes to /test.zip starting at 11392000
memory= 284536
writing 32000 bytes to /test.zip starting at 11424000
memory= 284536
writing 32000 bytes to /test.zip starting at 11456000
memory= 284692
writing 32000 bytes to /test.zip starting at 11488000
memory= 284652
writing 32000 bytes to /test.zip starting at 11520000
memory= 284652
writing 32000 bytes to /test.zip starting at 11552000
memory= 284716
writing 32000 bytes to /test.zip starting at 11584000
memory= 284708
writing 32000 bytes to /test.zip starting at 11616000
memory= 284776
writing 32000 bytes to /test.zip starting at 11648000
memory= 284776
writing 32000 bytes to /test.zip starting at 11680000
memory= 284792
writing 32000 bytes to /test.zip starting at 11712000
memory= 284844
writing 32000 bytes to /test.zip starting at 11744000
memory= 284844
writing 32000 bytes to /test.zip starting at 11776000
memory= 284892
writing 32000 bytes to /test.zip starting at 11808000
memory= 284816
writing 32000 bytes to /test.zip starting at 11840000
memory= 284816
writing 32000 bytes to /test.zip starting at 11872000
memory= 284932
writing 32000 bytes to /test.zip starting at 11904000
memory= 284956
writing 32000 bytes to /test.zip starting at 11936000
memory= 284860
writing 32000 bytes to /test.zip starting at 11968000
memory= 284860
writing 32000 bytes to /test.zip starting at 12000000
memory= 285012
writing 32000 bytes to /test.zip starting at 12032000
memory= 284928
writing 32000 bytes to /test.zip starting at 12064000
memory= 284928
writing 32000 bytes to /test.zip starting at 12096000
memory= 296836
writing 32000 bytes to /test.zip starting at 12128000
memory= 296872
writing 32000 bytes to /test.zip starting at 12160000
memory= 296872
writing 32000 bytes to /test.zip starting at 12192000
memory= 296996
writing 32000 bytes to /test.zip starting at 12224000
memory= 292516
writing 32000 bytes to /test.zip starting at 12256000
memory= 292548
writing 32000 bytes to /test.zip starting at 12288000
memory= 292548
writing 32000 bytes to /test.zip starting at 12320000
memory= 316784
writing 32000 bytes to /test.zip starting at 12352000
memory= 328964
writing 32000 bytes to /test.zip starting at 12384000
memory= 328964
writing 32000 bytes to /test.zip starting at 12416000
memory= 353384
writing 32000 bytes to /test.zip starting at 12448000
memory= 365600
writing 32000 bytes to /test.zip starting at 12480000
memory= 365600
writing 32000 bytes to /test.zip starting at 12512000
memory= 402584
writing 32000 bytes to /test.zip starting at 12544000
memory= 414916
writing 32000 bytes to /test.zip starting at 12576000
memory= 414916
writing 32000 bytes to /test.zip starting at 12608000
memory= 439716
writing 32000 bytes to /test.zip starting at 12640000
memory= 464564
writing 32000 bytes to /test.zip starting at 12672000
memory= 477100
writing 32000 bytes to /test.zip starting at 12704000
memory= 477100
writing 32000 bytes to /test.zip starting at 12736000
memory= 502152
writing 32000 bytes to /test.zip starting at 12768000
memory= 514684
writing 32000 bytes to /test.zip starting at 12800000
memory= 514684
writing 32000 bytes to /test.zip starting at 12832000
memory= 552600
writing 32000 bytes to /test.zip starting at 12864000
memory= 565272
writing 32000 bytes to /test.zip starting at 12896000
memory= 565272
writing 32000 bytes to /test.zip starting at 12928000
memory= 590684
writing 32000 bytes to /test.zip starting at 12960000
memory= 616168
writing 32000 bytes to /test.zip starting at 12992000
memory= 628932
writing 32000 bytes to /test.zip starting at 13024000
memory= 628932
writing 32000 bytes to /test.zip starting at 13056000
memory= 654612
writing 32000 bytes to /test.zip starting at 13088000
memory= 667500
writing 32000 bytes to /test.zip starting at 13120000
memory= 667548
writing 32000 bytes to /test.zip starting at 13152000
memory= 706336
writing 32000 bytes to /test.zip starting at 13184000
memory= 719280
writing 32000 bytes to /test.zip starting at 13216000
memory= 719280
writing 32000 bytes to /test.zip starting at 13248000
memory= 745324
writing 32000 bytes to /test.zip starting at 13280000
memory= 771480
writing 32000 bytes to /test.zip starting at 13312000
memory= 784544
writing 32000 bytes to /test.zip starting at 13344000
memory= 784544
writing 32000 bytes to /test.zip starting at 13376000
memory= 810848
writing 32000 bytes to /test.zip starting at 13408000
memory= 824056
writing 32000 bytes to /test.zip starting at 13440000
memory= 824056
writing 32000 bytes to /test.zip starting at 13472000
memory= 863808
writing 32000 bytes to /test.zip starting at 13504000
memory= 877056
writing 32000 bytes to /test.zip starting at 13536000
memory= 877056
writing 32000 bytes to /test.zip starting at 13568000
memory= 903764
writing 32000 bytes to /test.zip starting at 13600000
memory= 930532
writing 32000 bytes to /test.zip starting at 13632000
memory= 943908
writing 32000 bytes to /test.zip starting at 13664000
memory= 943908
writing 32000 bytes to /test.zip starting at 13696000
memory= 970864
writing 32000 bytes to /test.zip starting at 13728000
memory= 984320
writing 32000 bytes to /test.zip starting at 13760000
memory= 984320
writing 32000 bytes to /test.zip starting at 13792000
memory= 1011432
writing 32000 bytes to /test.zip starting at 13824000
memory= 1024936
writing 32000 bytes to /test.zip starting at 13856000
memory= 1024936
writing 32000 bytes to /test.zip starting at 13888000
memory= 1065732
writing 32000 bytes to /test.zip starting at 13920000
memory= 1079376
writing 32000 bytes to /test.zip starting at 13952000
memory= 1079376
writing 32000 bytes to /test.zip starting at 13984000
memory= 1106700
writing 32000 bytes to /test.zip starting at 14016000
memory= 1134144
writing 32000 bytes to /test.zip starting at 14048000
memory= 1147848
writing 32000 bytes to /test.zip starting at 14080000
memory= 1147848
writing 32000 bytes to /test.zip starting at 14112000
memory= 1175520
writing 32000 bytes to /test.zip starting at 14144000
memory= 1189296
writing 32000 bytes to /test.zip starting at 14176000
memory= 1189348
writing 32000 bytes to /test.zip starting at 14208000
memory= 1231024
writing 32000 bytes to /test.zip starting at 14240000
memory= 1245036
writing 32000 bytes to /test.zip starting at 14272000
memory= 1245036
writing 32000 bytes to /test.zip starting at 14304000
memory= 1272960
writing 32000 bytes to /test.zip starting at 14336000
memory= 1301080
writing 32000 bytes to /test.zip starting at 14368000
memory= 1315216
writing 32000 bytes to /test.zip starting at 14400000
memory= 1315216
writing 32000 bytes to /test.zip starting at 14432000
memory= 1343496
writing 32000 bytes to /test.zip starting at 14464000
memory= 1351596
writing 32000 bytes to /test.zip starting at 14496000
memory= 1351596
writing 32000 bytes to /test.zip starting at 14528000
memory= 1394508
writing 32000 bytes to /test.zip starting at 14560000
memory= 1408828
writing 32000 bytes to /test.zip starting at 14592000
memory= 1408828
writing 32000 bytes to /test.zip starting at 14624000
memory= 1437560
writing 32000 bytes to /test.zip starting at 14656000
memory= 1466464
writing 32000 bytes to /test.zip starting at 14688000
memory= 1480908
writing 32000 bytes to /test.zip starting at 14720000
memory= 1480908
writing 32000 bytes to /test.zip starting at 14752000
memory= 1509904
writing 32000 bytes to /test.zip starting at 14784000
memory= 1524456
writing 32000 bytes to /test.zip starting at 14816000
memory= 1524456
writing 32000 bytes to /test.zip starting at 14848000
memory= 1568324
writing 32000 bytes to /test.zip starting at 14880000
memory= 1582952
writing 32000 bytes to /test.zip starting at 14912000
memory= 1582952
writing 32000 bytes to /test.zip starting at 14944000
memory= 1612352
writing 32000 bytes to /test.zip starting at 14976000
memory= 1641844
writing 32000 bytes to /test.zip starting at 15008000
memory= 1656592
writing 32000 bytes to /test.zip starting at 15040000
memory= 1656592
writing 32000 bytes to /test.zip starting at 15072000
memory= 1686216
writing 32000 bytes to /test.zip starting at 15104000
memory= 1701092
writing 32000 bytes to /test.zip starting at 15136000
memory= 1701092
writing 32000 bytes to /test.zip starting at 15168000
memory= 1730928
writing 32000 bytes to /test.zip starting at 15200000
memory= 1745868
writing 32000 bytes to /test.zip starting at 15232000
memory= 1745928
writing 32000 bytes to /test.zip starting at 15264000
memory= 1791016
writing 32000 bytes to /test.zip starting at 15296000
memory= 1806072
writing 32000 bytes to /test.zip starting at 15328000
memory= 1806072
writing 32000 bytes to /test.zip starting at 15360000
memory= 1836260
writing 32000 bytes to /test.zip starting at 15392000
memory= 1866492
writing 32000 bytes to /test.zip starting at 15424000
memory= 1881672
writing 32000 bytes to /test.zip starting at 15456000
memory= 1881672
writing 32000 bytes to /test.zip starting at 15488000
memory= 1881672
writing 32000 bytes to /test.zip starting at 15520000
memory= 1942744
writing 32000 bytes to /test.zip starting at 15552000
memory= 1942744
writing 32000 bytes to /test.zip starting at 15584000
memory= 1942744
writing 32000 bytes to /test.zip starting at 15616000
memory= 2004068
writing 32000 bytes to /test.zip starting at 15648000
memory= 2004068
writing 32000 bytes to /test.zip starting at 15680000
memory= 2004068
writing 32000 bytes to /test.zip starting at 15712000
memory= 2065704
writing 32000 bytes to /test.zip starting at 15744000
memory= 2065704
writing 32000 bytes to /test.zip starting at 15776000
memory= 2065704
writing 32000 bytes to /test.zip starting at 15808000
memory= 2112060
writing 32000 bytes to /test.zip starting at 15840000
memory= 2112060
writing 32000 bytes to /test.zip starting at 15872000
memory= 2158664
writing 32000 bytes to /test.zip starting at 15904000
memory= 2174244
writing 32000 bytes to /test.zip starting at 15936000
memory= 2174244
writing 32000 bytes to /test.zip starting at 15968000
memory= 2205540
writing 32000 bytes to /test.zip starting at 16000000
memory= 2236788
writing 32000 bytes to /test.zip starting at 16032000
memory= 2252520
writing 32000 bytes to /test.zip starting at 16064000
memory= 2252520
writing 32000 bytes to /test.zip starting at 16096000
memory= 2284192
writing 32000 bytes to /test.zip starting at 16128000
memory= 2299956
writing 32000 bytes to /test.zip starting at 16160000
memory= 2300008
writing 32000 bytes to /test.zip starting at 16192000
memory= 2347660
writing 32000 bytes to /test.zip starting at 16224000
memory= 2363532
writing 32000 bytes to /test.zip starting at 16256000
memory= 2363532
writing 32000 bytes to /test.zip starting at 16288000
memory= 2383840
writing 32000 bytes to /test.zip starting at 16320000
memory= 2415992
writing 32000 bytes to /test.zip starting at 16352000
memory= 2432072
writing 32000 bytes to /test.zip starting at 16384000
memory= 2432072
writing 32000 bytes to /test.zip starting at 16416000
memory= 2464324
writing 32000 bytes to /test.zip starting at 16448000
memory= 2480496
writing 32000 bytes to /test.zip starting at 16480000
memory= 2480496
writing 32000 bytes to /test.zip starting at 16512000
memory= 2529260
writing 32000 bytes to /test.zip starting at 16544000
memory= 2545524
writing 32000 bytes to /test.zip starting at 16576000
memory= 2545524
writing 32000 bytes to /test.zip starting at 16608000
memory= 2578176
writing 32000 bytes to /test.zip starting at 16640000
memory= 2610952
writing 32000 bytes to /test.zip starting at 16672000
memory= 2627336
writing 32000 bytes to /test.zip starting at 16704000
memory= 2627336
writing 32000 bytes to /test.zip starting at 16736000
memory= 2660252
writing 32000 bytes to /test.zip starting at 16768000
memory= 2676748
writing 32000 bytes to /test.zip starting at 16800000
memory= 2676748
writing 32000 bytes to /test.zip starting at 16832000
memory= 2726464
writing 32000 bytes to /test.zip starting at 16864000
memory= 2743032
writing 32000 bytes to /test.zip starting at 16896000
memory= 2743032
writing 32000 bytes to /test.zip starting at 16928000
memory= 2776316
writing 32000 bytes to /test.zip starting at 16960000
memory= 2809688
writing 32000 bytes to /test.zip starting at 16992000
memory= 2826376
writing 32000 bytes to /test.zip starting at 17024000
memory= 2826376
writing 32000 bytes to /test.zip starting at 17056000
memory= 2859912
writing 32000 bytes to /test.zip starting at 17088000
memory= 2876728
writing 32000 bytes to /test.zip starting at 17120000
memory= 2876728
writing 32000 bytes to /test.zip starting at 17152000
memory= 2910488
writing 32000 bytes to /test.zip starting at 17184000
memory= 2927364
writing 32000 bytes to /test.zip starting at 17216000
memory= 2927424
writing 32000 bytes to /test.zip starting at 17248000
memory= 2978348
writing 32000 bytes to /test.zip starting at 17280000
memory= 2995348
writing 32000 bytes to /test.zip starting at 17312000
memory= 2995348
writing 32000 bytes to /test.zip starting at 17344000
memory= 3029424
writing 32000 bytes to /test.zip starting at 17376000
memory= 3063564

Any help appreciated....

Regards
Till

I've got a simple script for enumerating files on an SFTP server and downloading a subset of the files there. Sometimes the script works. Other times, it hangs. I've attached the output running w/ :verbose => :debug. I print the names of the files found with puts " found: #{obj.name}" -- notice that the first file name is fine, but the 2nd is blank (!), and the 3rd is garbage. Further down, the script hangs after "sent 52 bytes".

The relevant part of my script is:

sftp.dir.foreach('.') do |obj|
  puts "  found: #{obj.name}" if obj.name
  next unless obj.file?
  next unless obj.name.present?
  next unless obj.name.match(/^trf\-/)  # only files matching "trf-*"
  ...save the file name...
end

Gem version:
net-sftp (2.0.5)
net-ssh (>= 2.0.9)

I'm on a Mac running OS X Lion. The SFTP server is Debian etch running
OpenSSH 4.3p2.

negotiating protocol version
remote is `SSH-2.0-OpenSSH_4.3p2 Debian-9etch3'
local is `SSH-2.0-Ruby/Net::SSH_2.2.1 x86_64-linux'
read 704 bytes
received packet nr 0 type 20 len 700
got KEXINIT from server
sending KEXINIT
queueing packet nr 0 type 20 len 556
sent 560 bytes
negotiating algorithms
negotiated:
* kex: diffie-hellman-group-exchange-sha1
* host_key: ssh-rsa
* encryption_server: aes128-cbc
* encryption_client: aes128-cbc
* hmac_client: hmac-sha1
* hmac_server: hmac-sha1
* compression_client: none
* compression_server: none
* language_client:
* language_server:
exchanging keys
queueing packet nr 1 type 34 len 20
sent 24 bytes
read 152 bytes
received packet nr 1 type 31 len 148
queueing packet nr 2 type 32 len 140
sent 144 bytes
read 720 bytes
received packet nr 2 type 33 len 700
queueing packet nr 3 type 21 len 20
sent 24 bytes
received packet nr 3 type 21 len 12
beginning authentication of `health123'
queueing packet nr 4 type 5 len 28
sent 52 bytes
read 52 bytes
received packet nr 4 type 6 len 28
trying publickey
connecting to ssh-agent
could not connect to ssh-agent
trying publickey (d3:be:2f:eb:bd:61:15:0d:62:8c:65:d1:53:9c:28:a8)
queueing packet nr 5 type 50 len 348
sent 372 bytes
read 324 bytes
received packet nr 5 type 60 len 300
queueing packet nr 6 type 50 len 620
sent 644 bytes
read 36 bytes
received packet nr 6 type 52 len 12
publickey succeeded (d3:be:2f:eb:bd:61:15:0d:62:8c:65:d1:53:9c:28:a8)
queueing packet nr 7 type 90 len 44
sent 68 bytes
read 52 bytes
received packet nr 7 type 91 len 28
channel_open_confirmation: 0 0 0 32768
requesting sftp subsystem
sending channel request "subsystem"
queueing packet nr 8 type 98 len 44
sent 68 bytes
read 88 bytes
received packet nr 8 type 93 len 28
channel_window_adjust: 0 +131072
received packet nr 9 type 99 len 12
channel_success: 0
sftp subsystem successfully started
queueing packet nr 9 type 94 len 28
sent 52 bytes
read 148 bytes
received packet nr 10 type 94 len 124
channel_data: 0 95b
received sftp packet 2 len 91
negotiating sftp protocol version, mine is 6
server reports sftp version 5
negotiated version is 5
sending opendir packet (0)
queueing packet nr 10 type 94 len 28
sent 52 bytes
read 52 bytes
received packet nr 11 type 94 len 28
channel_data: 0 14b
received sftp packet 102 len 10
sending readdir packet (1)
queueing packet nr 11 type 94 len 28
sent 52 bytes
read 964 bytes
received packet nr 12 type 94 len 940
channel_data: 0 918b
received sftp packet 104 len 914
  found: h123-20111117-044053.csv
  found:
  found: csv??  health123chadm??N?#?N???N???h123-20111202-023549.csv??
health123chadm??N?9?N?9?N?9?h123-20111201-200033.csv??
health123chadm??N?$N???N???h123-20111202-030147.csv??
health123chadm??N???N???N???..?
health123chadmE?N?CqN???N???h123-20111202-025247.csv??
health123chadm??N?=N?=N?=.?
health123chadmE?N?CqN???N???h123-20111202-025952.csv??
health123chadm??N??*N??*N??*h123-20111202-030227.csv??
health123chadm??N???N???N???
sending readdir packet (2)
queueing packet nr 12 type 94 len 28
sent 52 bytes
read 84 bytes
received packet nr 13 type 94 len 60
channel_data: 0 34b
received sftp packet 101 len 30
sending close packet (3)
queueing packet nr 13 type 94 len 28
sent 52 bytes
read 68 bytes
received packet nr 14 type 94 len 44
channel_data: 0 30b
received sftp packet 101 len 26
closing remaining channels (1 open)
queueing packet nr 14 type 97 len 28
sent 52 bytes

Overview

@testflyjets over at SD Ruby came across a puzzling issue when downloading files larger than 4 GB using Net::SFTP#download!. In summary, the download runs (with 64K read_size) up to 4294049792 bytes (≈3.99915 GB), at which point the process hangs. When given an interrupt (ctrl+c) after the hang, the top stack trace entry points to /net-ssh-2.6.7/lib/net/ssh/ruby_compat.rb:30. I know this is within Net::SSH, but I thought this might be a good place to start, since this use case is pretty specific to SFTP.

The Gist linked below contains a script that I've been using to troubleshoot. It's a pretty basic Net::SFTP implementation that downloads a single file. I'm using a large tar file that is 4404802560 bytes (≈4.1 GB).

https://gist.github.com/bradland/5568217

To reproduce, alter the parameters of the script to work for your local environment (you'll have to create your own large file; see Util section near end), then execute. You'll see the progress output stop at around 4294049792 bytes, at which point you can issue ctrl+c to see a stack trace.

Investigating the stack trace

The original author was using SFTP#download!, but the stack trace when using SFTP#download is a lot easier to parse. You can see the output in 001-net_sftp_download.txt. Below is a linked list of the stack locations for easy reference:

• /net-ssh-2.6.7/lib/net/ssh/ruby_compat.rb:30:in `select': Interrupt
• /net-ssh-2.6.7/lib/net/ssh/ruby_compat.rb:30:in `io_select'
• /net-ssh-2.6.7/lib/net/ssh/connection/session.rb:204:in `process'
• /net-ssh-2.6.7/lib/net/ssh/connection/session.rb:164:in `block in loop'
• /net-ssh-2.6.7/lib/net/ssh/connection/session.rb:164:in `loop'
• /net-ssh-2.6.7/lib/net/ssh/connection/session.rb:164:in `loop'
• /net-sftp-2.1.2/lib/net/sftp/session.rb:802:in `loop'
• /net-sftp-2.1.2/lib/net/sftp.rb:35:in `start'
• ./sftp.rb:15:in `'

Execution environment:

• uname -a output: Darwin macbookair.local 12.3.0 Darwin Kernel Version 12.3.0: Sun Jan 6 22:37:10 PST 2013; root:xnu-2050.22.13~1/RELEASE_X86_64 x86_64
• When running, the Ruby process is listed as "Intel (64 bit)" in activity monitor
• sshd is also listed as "Intel (64 bit)"
• sshd version: OpenSSH_5.9p1, OpenSSL 0.9.8r 8 Feb 2011
• Ruby version: 1.9.3 p392
• Gems (clean rvm gemset)
  • bundler (1.3.5)
  • columnize (0.3.6)
  • debugger (1.5.0)
  • debugger-linecache (1.2.0)
  • debugger-ruby_core_source (1.2.0)
  • net-sftp (2.1.2)
  • net-ssh (2.6.7)
• Tested using localhost for SFTP#start host argument, as well as another MacBook Pro running Mountain Lion on the same LAN
• Original author's execution environment was CentOS 64-bit (no information yet on version)

Introspection (rdebug, gdb)

The hang appears to occur at the IO#select call within Net::SSH, ruby_compat.rb. I don't know how type safe IO#select is, but given the proximity to 4^32, and the fact that this is the point that we jump in to a system call (from my amateur perspective), it seems a likely place for a buffer overflow. I'm not terribly proficient at this low level. Because ruby_compat.rb is just a compatibility wrapper, we can look at Net::SSH::Connection#process for the setup:

def process(wait=nil, &block)
  return false unless preprocess(&block)

  r = listeners.keys
  w = r.select { |w2| w2.respond_to?(:pending_write?) && w2.pending_write? }
  readers, writers, = Net::SSH::Compat.io_select(r, w, nil, wait)

  postprocess(readers, writers)
end

This is the point where I think it makes sense to start inspecting, but I'm kind of at a loss for how to do that.

I invoked the script with the Ruby debugger gem active, but once the process hangs, the rdb console becomes unresponsive. I fired up gdb -p <process_id> in order to get the output of where listed in 003-gdb-where.txt (also in the Gist above).

This takes a long time to reproduce, so I've left a hung process on a secondary computer with a gdb session attached. I can issue any commands you'd like, or I'm available for a screen sharing session (I have a MeetingBurner account we could use) if a maintainer would like to see what's going on under the hood.

Update: 4/13 session with @erikh

erikh suggested that the hang may not be a problem with select at all, but rather sshd has sent something back to Net::SSH that is not being handled properly by Net::SSH, and the processing loop just jumps right back in to asking for more data. At that point sshd just remains silent, and because the SSH loop is using an infinite wait period, it hangs.

Util

To generate a large file quickly. WARNING! This will create a large file (≈4.1 GB; ≈4.4 GB according to Finder... which is wrong) on your hard drive. Make sure you have the space.

mkfile 4404802560 test-data.txt

This uses the same bytesize as my test file, which is kind of arbitrary. I haven't tested for the lower boundary of the bug.

Edits

• Updated large file creator to use mkfile because dd was tossing an invalid argument error when given /dev/zero on OS X
• Updated Gist to use test-data.txt instead of tar
• Updated Gist to use Net::SFTP#download instead of Net::SFTP#download! for cleaner stack trace
• Eliminated include Net to keep the namespace as clean as possible (was vestigial from a code example)

I'm trying to connect to a sftp server and I'm getting a timeout when I try to .dir.foreach('/absolute/path'), but it works if I use .dir.foreach('./relative/path').

When using open or open! I get the content of a given file instead of a nice file handle to read chunks from.

I am using sftp.download!(remote_dir, local_dir, recurssive => true) to download and entire directory (anywhere from 1 to 15 .zip files). I am getting an error #<IOError: closed stream>.

I have been told:

net-sftp tries to keep 16 simultaneous sftp requests active (multiplexed over the same channel)

I tried forcing it to D/L one file at a time using (:requests => 1) option but it still crashed after downloading 15.9MB

Anyone else run into a problem like this?

#!/usr/bin/env ruby
require 'net/sftp'
require 'securerandom'

@connection = Net::SFTP.start("test-host",
                              ENV['USER'],
                              port: 22)

buffer_size = 1024 * 255
loop do
  data = StringIO.new(SecureRandom.random_bytes((1024**2) * 128))
  path = File.join("/tmp/sftptest-#{SecureRandom.hex}",
                   SecureRandom.hex,
                   SecureRandom.hex,
                   SecureRandom.hex)
  puts "Writing to #{path} with #{buffer_size} bytes buffer size"
  @connection.session.exec!("mkdir -p '#{File.dirname(path)}'")
  read = 0
  start = Time.now
  @connection.file.open(path, "wb") do |f|
    while buffer = data.read(buffer_size)
      time_passed = Time.now - start
      read += buffer.bytesize
      print "\r #{read / (data.string.bytesize / 100)} % | #{(read / time_passed)/1024} kB/s \t"
      f.write buffer
    end
  end
  puts
end

This test upload random data to "test-host". I can get 800-1000 Kb/s on a 1 Gbit ethernet connection. CPU is almost at 100%. Is there a way to get that any faster or is it a limitation of net-sftp ?

I am getting NoMethodError: undefined methodany?' for nil:NilClassinnet-sftp-2.1.2/lib/net/sftp/session.rb:801`.

What fills pending_requests? Is it desired to check it for nil?

I workarounded it by:
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/net-sftp-2.1.2/lib/net/sftp/session.rb at line 801 by adding pending_requests = pending_requests ? pending_requests : {}:

# Runs the SSH event loop while the given block returns true. This lets
      # you set up a state machine and then "fire it off". If you do not specify
      # a block, the event loop will run for as long as there are any pending
      # SFTP requests. This makes it easy to do thing like this:
      #
      #   sftp.remove("/path/to/file")
      #   sftp.loop
      def loop(&block)
        pending_requests = pending_requests ? pending_requests : {}
        block ||= Proc.new { pending_requests.any? }
        session.loop(&block)
      end

Original log from Backup gem:
Please see - backup/backup#544

[2014/05/14 09:04:25][error] --- Wrapped Exception ---
[2014/05/14 09:04:25][error] NoMethodError: undefined method `any?' for nil:NilClass
[2014/05/14 09:04:25][error] 
[2014/05/14 09:04:25][error] Backtrace:
[2014/05/14 09:04:25][error]   /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/net-sftp-2.1.2/lib/net/sftp/session.rb:801:in `block in loop'
[2014/05/14 09:04:25][error]   /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/net-ssh-2.7.0/lib/net/ssh/connection/session.rb:220:in `preprocess'
[2014/05/14 09:04:25][error]   /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/net-ssh-2.7.0/lib/net/ssh/connection/session.rb:205:in `process'
[2014/05/14 09:04:25][error]   /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/net-ssh-2.7.0/lib/net/ssh/connection/session.rb:169:in `block in loop'
[2014/05/14 09:04:25][error]   /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/net-ssh-2.7.0/lib/net/ssh/connection/session.rb:169:in `loop'
[2014/05/14 09:04:25][error]   /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/net-ssh-2.7.0/lib/net/ssh/connection/session.rb:169:in `loop'
[2014/05/14 09:04:25][error]   /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/net-sftp-2.1.2/lib/net/sftp/session.rb:802:in `loop'
[2014/05/14 09:04:25][error]   /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/net-sftp-2.1.2/lib/net/sftp.rb:35:in `start'
[2014/05/14 09:04:25][error]   /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/backup-4.0.1/lib/backup/storage/sftp.rb:29:in `connection'
[2014/05/14 09:04:25][error]   /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/backup-4.0.1/lib/backup/storage/sftp.rb:35:in `transfer!'
[2014/05/14 09:04:25][error]   /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/backup-4.0.1/lib/backup/storage/base.rb:35:in `perform!'
[2014/05/14 09:04:25][error]   /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/backup-4.0.1/lib/backup/model.rb:269:in `each'
[2014/05/14 09:04:25][error]   /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/backup-4.0.1/lib/backup/model.rb:269:in `block in perform!'
[2014/05/14 09:04:25][error]   /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/backup-4.0.1/lib/backup/model.rb:268:in `each'
[2014/05/14 09:04:25][error]   /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/backup-4.0.1/lib/backup/model.rb:268:in `perform!'
[2014/05/14 09:04:25][error]   /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/backup-4.0.1/lib/backup/cli.rb:153:in `perform'
[2014/05/14 09:04:25][error]   /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thor-0.18.1/lib/thor/command.rb:27:in `run'
[2014/05/14 09:04:25][error]   /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thor-0.18.1/lib/thor/invocation.rb:120:in `invoke_command'
[2014/05/14 09:04:25][error]   /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thor-0.18.1/lib/thor.rb:363:in `dispatch'
[2014/05/14 09:04:25][error]   /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/thor-0.18.1/lib/thor/base.rb:439:in `start'
[2014/05/14 09:04:25][error]   /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/backup-4.0.1/bin/backup:5:in `<top (required)>'
[2014/05/14 09:04:25][error]   /opt/chef/embedded/bin/backup:23:in `load'
[2014/05/14 09:04:25][error]   /opt/chef/embedded/bin/backup:23:in `<main>'

While trying to make distkeys work with Ruby 1.9 in order to get it included in Debian, I tried to work around bug

ruby-net-ssh: can't add a new key into hash during iteration during ssh.exec
net-ssh/net-ssh#110

by using sftp.rename!, however openssh 6.2p1 does not yet support sftp protocol version 5 for flags 0x0001 for overwrite and 0x0002 for atomic rewrite yet:

Bug 2123 - does not seem to support sftp protocol version 5 rename flag for overwriting files
https://bugzilla.mindrot.org/show_bug.cgi?id=2123

According to Darren it does however support a vendor extension to protocol version 3 ("[email protected]")¹. Please support this especially as he says its unlikely that OpenSSH will support SFTP protocol 5 soon.

Git repo for distkeys is

git://oss.teamix.org/distkeys.git

reproducer tag is openssh-bug-2123.

[1] https://bugzilla.mindrot.org/show_bug.cgi?id=2123#c2

Thanks,
Martin

I found it highly confusing that it is not possible to upload a directory to overwrite existing files in existing directories. Since the upload operation tries to create every directory it encounters, it raises an exception and aborts as soon as one of the directories exists. I had to resort to monkey-patching to work around this issue but I think this should be easy enough to add to the library itself with a new option for the upload operation.

I don't know if this library is still actively maintained, but if there is actually somebody in charge, let me know - I'll gladly invest the time to implement this feature and create a pull request.

What is the reason for directory? to not return false if a directory doesn't exist? Instead you receive a "File not found" and are required to handle the exception.

Hello,

I'm using net-ssh 2.0.5 on ruby 1.9.3p194
I use the method sftp.upload! with the do |event, uploader, *args| in order to save in my database the progress of the upload. (We are dealing with large uploads) and to keep the number of Bytes sent to compare them with my file.

This works quite well but sometimes the process just hangs. I guess the connection has reset. No exceptions are raised and the program stays on upload!

How do you deal with upload timeout ? Any way I could detect it ?

Thanks !

I have a small problem connecting to one of our servers. I have this little test code:

require 'net/sftp'

Net::SFTP.start('SERVER', 'USER', {:password => 'PASSWORT'}) do |sftp| 
  puts "connected"

  sftp.dir.foreach(".") do |entry|
    puts entry.longname
  end
end

I get this exception:

net-ssh-2.1.3/lib/net/ssh/buffer.rb:294:in `pack': can't convert nil into Integer (TypeError)
from /path/net-ssh-2.1.3/lib/net/ssh/buffer.rb:294:in `write_long'
from /path/net-sftp-2.0.5/lib/net/sftp/protocol/04/attributes.rb:144:in `block in encode_acl'
from /path/net-sftp-2.0.5/lib/net/sftp/protocol/04/attributes.rb:143:in `each'
from /path/net-sftp-2.0.5/lib/net/sftp/protocol/04/attributes.rb:143:in `encode_acl'
from /path/net-sftp-2.0.5/lib/net/sftp/protocol/01/attributes.rb:286:in `block in to_s'
from /path/net-sftp-2.0.5/lib/net/sftp/protocol/01/attributes.rb:283:in `each'
from /path/net-sftp-2.0.5/lib/net/sftp/protocol/01/attributes.rb:283:in `to_s'
from /path/net-sftp-2.0.5/lib/net/sftp/protocol/04/name.rb:59:in `inspect'
from /path/net-sftp-2.0.5/lib/net/sftp/protocol/04/name.rb:59:in `longname'
from test.rb:7:in `block (2 levels) in <main>'
from /path/net-sftp-2.0.5/lib/net/sftp/operations/dir.rb:33:in `block in foreach'
from /path/net-sftp-2.0.5/lib/net/sftp/operations/dir.rb:33:in `each'
from /path/net-sftp-2.0.5/lib/net/sftp/operations/dir.rb:33:in `foreach'
from test.rb:6:in `block in <main>'
from /path/net-sftp-2.0.5/lib/net/sftp/session.rb:939:in `call'
from /path/net-sftp-2.0.5/lib/net/sftp/session.rb:939:in `block in do_version'
from /path/net-sftp-2.0.5/lib/net/sftp/session.rb:939:in `each'
from /path/net-sftp-2.0.5/lib/net/sftp/session.rb:939:in `do_version'
from /path/net-sftp-2.0.5/lib/net/sftp/session.rb:909:in `when_channel_polled'
from /path/net-ssh-2.1.3/lib/net/ssh/connection/channel.rb:311:in `call'
from /path/net-ssh-2.1.3/lib/net/ssh/connection/channel.rb:311:in `process'
from /path/net-ssh-2.1.3/lib/net/ssh/connection/session.rb:214:in `block in preprocess'
from /path/net-ssh-2.1.3/lib/net/ssh/connection/session.rb:214:in `each'
from /path/net-ssh-2.1.3/lib/net/ssh/connection/session.rb:214:in `preprocess'
from /path/net-ssh-2.1.3/lib/net/ssh/connection/session.rb:197:in `process'
from /path/net-ssh-2.1.3/lib/net/ssh/connection/session.rb:161:in `block in loop'
from /path/net-ssh-2.1.3/lib/net/ssh/connection/session.rb:161:in `loop'
from /path/net-ssh-2.1.3/lib/net/ssh/connection/session.rb:161:in `loop'
from /path/net-sftp-2.0.5/lib/net/sftp/session.rb:802:in `loop'
from /path/net-sftp-2.0.5/lib/net/sftp/session.rb:787:in `connect!'
from /path/net-sftp-2.0.5/lib/net/sftp.rb:32:in `start'
from test.rb:3:in `<main>'

I have added a

 puts attributes.inspect

into the name.rb to check it's contents and it was this:

?bodobodoA?M{??M{?Mx??

can this mean that there was an encoding issue and the code couldn't parse the answer from the server?

When using a sftp command line utility or java, I can connect without any problem. The server seems to be working perfectly.

When convert a private key ppk file to Open SSH using PuttyGen, then this exception appear "Permission denied (publickey)" on client.Connect() method