Make SCP a bit smarter or replace it with rsync about beaker

[electrical]

For my tests it hangs on the scp step for quite a while.
Checked if there are any symlinks but there are none.
Some runs take about 10 minutes now, would be nice to improve that :-)

[igalic]

I'm wondering now, how many core files you have there in your repo that it takes 10 minutes to scp…

[electrical]

Apparently none at all.
But... i do have a .vendor dir in there for the specific gems for each build. i most likely could drop that. ( 84MB )
Will give that a shot.

[electrical]

Seems the .vendor dir gets scp'ed as well. Scp runs are much faster now.
Any improvements we can think of ?

[daenney]

There's a bunch of dirs that are nasty, .vendor, .bundle, .vagrant, .gem, .git that we could exclude by default I think.

[daenney]

Honestly I would prefer a solution the other way around. Exclude everything by default unless it's in include_paths, and we can probably add manifests, lib, templates and files by default to include_paths.

[electrical]

Im gonna play with rsync ( http://rubygems.org/gems/rsync ) and see if we could replace it.
Might take a couple days :-)

[anodelman]

@electrical any progress with rsync?

[electrical]

@anodelman no luck so far :-(

[igalic]

s/days/months/ ;)

[igalic]

I'm running into a related problem lately, where the scp job takes 10 minutes:

ubuntu-server-12044-x64 18:03:03$  apt-get install -y puppet                                                                                                              
Reading package lists...
Building dependency tree...
Reading state information...
puppet is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

ubuntu-server-12044-x64 executed in 0.59 seconds                                                                                                                          
localhost $ scp /home/igalic/src/acme/puppet/environment/production_redux/modules/acme-pusher ubuntu-server-12044-x64:/etc/puppet/modules/pusher {}                       

ubuntu-server-12044-x64 18:13:20$ env PATH="/usr/bin:/opt/puppet-git-repos/hiera/bin:${PATH}" RUBYLIB="/opt/puppet-git-repos/hiera/lib:/opt/puppet-git-repos/hiera-puppet/lib:${RUBYLIB}" puppet module  install puppetlabs-stdlib                                                                                                                  
Notice: Preparing to install into /etc/puppet/modules ...
Notice: Downloading from https://forge.puppetlabs.com ...
Notice: Installing -- do not interrupt ...
/etc/puppet/modules
└── puppetlabs-stdlib (v4.1.0)

ubuntu-server-12044-x64 executed in 3.43 seconds                                                                                                                          

[igalic]

So… I was wondering, why does it take soooooooooooooooooooooooooooooooooooooooo looooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooong…

I ran a strace against the process,

stat("/etc/puppet/modules/pusher/vendor/bundle/ruby/1.9.1/gems/serverspec-1.0.0/lib/serverspec/matchers/be_installed.rb", 0x7fffa8dc0290) = -1 ENOENT (No such file or directory)
open("/etc/puppet/modules/pusher/vendor/bundle/ruby/1.9.1/gems/serverspec-1.0.0/lib/serverspec/matchers/be_installed.rb", O_WRONLY|O_CREAT, 0644) = 3
write(1, "\0", 1)                       = 1
fstat(3, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
fcntl(0, F_GETFL)                       = 0 (flags O_RDONLY)
fcntl(0, F_SETFL, O_RDONLY|O_NONBLOCK)  = 0
read(0, "RSpec::Matchers.define :be_installed do\n  match do |name|\n  "..., 233) = 233
fcntl(0, F_GETFL)                       = 0x800 (flags O_RDONLY|O_NONBLOCK)
fcntl(0, F_SETFL, O_RDONLY)             = 0
write(3, "RSpec::Matchers.define :be_installed do\n  match do |name|\n  "..., 233) = 233
ftruncate(3, 233)                       = 0
close(3)                                = 0
read(0, "\0", 1)                        = 1
write(1, "\0", 1)                       = 1
read(0, "C", 1)                         = 1
read(0, "0", 1)                         = 1
read(0, "6", 1)                         = 1
read(0, "4", 1)                         = 1
read(0, "4", 1)                         = 1
read(0, " ", 1)                         = 1
read(0, "2", 1)                         = 1
read(0, "5", 1)                         = 1
read(0, "9", 1)                         = 1
read(0, " ", 1)                         = 1
read(0, "b", 1)                         = 1
read(0, "e", 1)                         = 1
read(0, "_", 1)                         = 1
read(0, "e", 1)                         = 1
read(0, "n", 1)                         = 1
read(0, "a", 1)                         = 1
read(0, "b", 1)                         = 1
read(0, "l", 1)                         = 1
read(0, "e", 1)                         = 1
read(0, "d", 1)                         = 1
read(0, ".", 1)                         = 1
read(0, "r", 1)                         = 1
read(0, "b", 1)                         = 1
read(0, "\n", 1)                        = 1
stat("/etc/puppet/modules/pusher/vendor/bundle/ruby/1.9.1/gems/serverspec-1.0.0/lib/serverspec/matchers/be_enabled.rb", 0x7fffa8dc0290) = -1 ENOENT (No such file or directory)
open("/etc/puppet/modules/pusher/vendor/bundle/ruby/1.9.1/gems/serverspec-1.0.0/lib/serverspec/matchers/be_enabled.rb", O_WRONLY|O_CREAT, 0644) = 3
write(1, "\0", 1)                       = 1
fstat(3, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
fcntl(0, F_GETFL)                       = 0 (flags O_RDONLY)
fcntl(0, F_SETFL, O_RDONLY|O_NONBLOCK)  = 0
read(0, 0x7f1e872949f0, 259)            = -1 EAGAIN (Resource temporarily unavailable)
poll([{fd=0, events=POLLIN}], 1, -1)    = 1 ([{fd=0, revents=POLLIN}])
read(0, "RSpec::Matchers.define :be_enabled do\n  match do |subject|\n "..., 259) = 259
fcntl(0, F_GETFL)                       = 0x800 (flags O_RDONLY|O_NONBLOCK)
fcntl(0, F_SETFL, O_RDONLY)             = 0
write(3, "RSpec::Matchers.define :be_enabled do\n  match do |subject|\n "..., 259) = 259
ftruncate(3, 259)                       = 0
close(3)                                = 0
read(0, "\0", 1)                        = 1
write(1, "\0", 1)                       = 1
read(0, "C", 1)                         = 1
read(0, "0", 1)                         = 1
read(0, "6", 1)                         = 1
read(0, "4", 1)                         = 1
read(0, "4", 1)                         = 1
read(0, " ", 1)                         = 1
read(0, "1", 1)                         = 1
read(0, "1", 1)                         = 1
read(0, "8", 1)                         = 1
read(0, " ", 1)                         = 1
read(0, "r", 1)                         = 1
read(0, "e", 1)                         = 1
read(0, "t", 1)                         = 1
read(0, "u", 1)                         = 1
read(0, "r", 1)                         = 1
read(0, "n", 1)                         = 1
read(0, "_", 1)                         = 1
read(0, "s", 1)                         = 1
read(0, "t", 1)                         = 1
read(0, "d", 1)                         = 1
read(0, "e", 1)                         = 1
read(0, "r", 1)                         = 1
read(0, "r", 1)                         = 1
read(0, ".", 1)                         = 1
read(0, "r", 1)                         = 1
read(0, "b", 1)                         = 1
read(0, "\n", 1)                        = 1
stat("/etc/puppet/modules/pusher/vendor/bundle/ruby/1.9.1/gems/serverspec-1.0.0/lib/serverspec/matchers/return_stderr.rb", 0x7fffa8dc0290) = -1 ENOENT (No such file or directory)
open("/etc/puppet/modules/pusher/vendor/bundle/ruby/1.9.1/gems/serverspec-1.0.0/lib/serverspec/matchers/return_stderr.rb", O_WRONLY|O_CREAT, 0644) = 3
write(1, "\0", 1)                       = 1
fstat(3, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
fcntl(0, F_GETFL)                       = 0 (flags O_RDONLY)
fcntl(0, F_SETFL, O_RDONLY|O_NONBLOCK)  = 0
read(0, 0x7f1e872949f0, 118)            = -1 EAGAIN (Resource temporarily unavailable)
poll([{fd=0, events=POLLIN}], 1, -1)    = 1 ([{fd=0, revents=POLLIN}])
read(0, "RSpec::Matchers.define :return_stderr do |content|\n  match d"..., 118) = 118
fcntl(0, F_GETFL)                       = 0x800 (flags O_RDONLY|O_NONBLOCK)
fcntl(0, F_SETFL, O_RDONLY)             = 0
write(3, "RSpec::Matchers.define :return_stderr do |content|\n  match d"..., 118) = 118
ftruncate(3, 118)                       = 0
close(3)                                = 0
read(0, ^C <unfinished ...>
Process 1657 detached

turns out it's reading 1 byte at a time.

[daenney]

That's a pretty spiffy way to do it 😛.

[jantman]

Yeah, I'm getting the ~10 minute scp when running the acceptance tests for puppetlabs-rabbitmq... quite painful given that this is the first time I've done work with beaker and there's lots of iteration involved...

[anodelman]

There is a pending PR for including an :ignore option to scp commands in beaker - but that doesn't explain the 1-byte-at-a-time that you are seeing.

[anodelman]

I'm going to attempt to get this into the next Beaker sprint. I'd like to re-create and diagnose.

[anodelman]

Potential fix in #385 - at least it will get us more info.

[igalic]

\o/

[jantman]

@anodelman do you have a link to the PR that includes the :ignore option? (I couldn't find mention of any such option supported upstream in Net::SCP)

Having bundler install to somewhere outside of my module directory helped a lot of this problem, and I was able to work around the rest of it by cleaning my spec directory (fixtures) before running the acceptance tests. Getting modules that aren't on the forge (or module dependencies that are direct from git) are a bit of a problem, but my awful hack around that was to clean before acceptance, and then system("bundle exec rake spec_prep") at the right place in my acceptance helper.

[anodelman]

Here's the 'scp ignore' PR #330, it will land in the next gem.

[igalic]

closing, seems to have been addressed (and released)