generic/fedora29 (v1.8.40, libvirt) stuck on .device systemd jobs about robox

[ladar]

@gorshunovr thank you for the report. I think I noticed this problem as well, but didn't investigate because the system did eventually boot.

What I'm not understanding is your solution. Based on what you wrote, it sounds like the device mapper UUID is changing for the swap partition, and that's causing the hang? If so, the question is, what is causing the UUID to change? Is it because we're forcing the clone to generate a new machine-id? Or is it something else?

I don't have time to look now, but I'm guessing Fedora is using LVM by default. I could force a standard partition scheme instead of LVM, but prefer not to go that route. The relevant kickstart config lines:

autopart --type=plain --nohome
clearpart --all --initlabel

Do you think rebuilding the kernel image (aka running dracut) before I shutdown the template machine will fix this? I haven't noticed a delay in reboots during the build process, which is why I suspect the issue is triggered by the cloning, but I could be wrong?

Thank you for all your boxes. Your project is probably the only one which builds boxes for parallels, hyperv, virtualbox, libvirt, and vmware_desktop - 5 platforms at the same time, allowing to use consistent environment across different virtualization engines

Your welcome. You nailed the reason I started uploading the generic boxes, (and not just the magma or lineage specific builds.

This project allowed me to write nifty scripts to automate spawning all 27 distros, across 1 or more hypervisors, then running shell commands in each... logging the output in a text file, an cleaning everything up. It's been great for seeing what X does across distros, but is probably also the reason I keep wearing out my SSDs.

I'd like to add more distros/providers, but I'm bumping up a resource wall. It currently takes ~2 to ~3 days to build all of the boxes with my army of robots (aka surplus dev/testing notebooks). And another ~1 day to test all of images, to ensure they boot properly, and that vagrant can connect via ssh. (Disclaimer, I don't test the hyperv/parallels images this way because singular Windows and singular Mac build bots are too slow.

Instead I'm trying to add more providers that I can build on Linux, since I have more of those machines. I've already started adding docker boxes. And I'm hoping to add lxc and possibly even aws images in the future. We'll see if the sponsor I'm talking to comes through... which should also allow me to bump the update cycle from ~2 weeks, to hopefully ~1 week.

[ladar]

I tested the 1.8.48 version of Fedora 29 on libvirt, and it seemed to work just fine. Closing this ticket.

[gorshunovr]

Hello Ladar. I think I still have the issue. Moreover, with 2 kernels installed simultaneously in 1.8.48 box, I have to remove old kernel and specify explicitly for the dracut which initrd for which kernel version to build.
Also eth0 loses IP address from time to time, but that might be my local issue with DHCP, I'm not yet sure. Will open a separate issue if I would be able to confirm something more meaningful.

[ladar]

@gorshunovr I just ran another test using the v1.8.48 version (make sure you have the latest image), and I was able to download and create a Fedora 29 box running on libvirt in 1 min, 6 secs, and most of that minute was spent downloading the box file. I'll paste the output here, in case it helps you diagnose the problem:

# time ./simple.sh generic fedora29 libvirt
+ [[ ./simple.sh != ./simple.sh ]]
+ export CMD=./simple.sh
+ CMD=./simple.sh
++ dirname ./simple.sh
+ pushd .
++ pwd -P
+ BASE=/home/ladar/Data/vagrant
+ popd
+ cd /home/ladar/Data/vagrant
+ export VAGRANT_HOME=/home/ladar/Data/vagrant/vagrant.d/
+ VAGRANT_HOME=/home/ladar/Data/vagrant/vagrant.d/
+ '[' libvirt = libvirt ']'
+ plugin-libvirt
+ '[' '!' -d fedora29-libvirt ']'
+ mkdir fedora29-libvirt
+ vagrant plugin list
vagrant-libvirt (0.0.45, global)
+ cp --force fedora29.tpl fedora29-libvirt/Vagrantfile
+ cd fedora29-libvirt
+ '[' libvirt == vmware ']'
+ vagrant box add --clean --force --provider libvirt generic/fedora29
==> box: Loading metadata for box 'generic/fedora29'
    box: URL: https://vagrantcloud.com/generic/fedora29
==> box: Adding box 'generic/fedora29' (v1.8.48) for provider: libvirt
    box: Downloading: https://vagrantcloud.com/generic/boxes/fedora29/versions/1.8.48/providers/libvirt.box
    box: Download redirected to host: vagrantcloud-files-production.s3.amazonaws.com
==> box: Successfully added box 'generic/fedora29' (v1.8.48) for 'libvirt'!
+ vagrant up --provider libvirt
Bringing machine 'default' up with 'libvirt' provider...
==> default: Checking if box 'generic/fedora29' is up to date...
==> default: Uploading base box image as volume into libvirt storage...
==> default: Creating image (snapshot of base box volume).
==> default: Creating domain with the following settings...
==> default:  -- Name:              fedora29-libvirt_default
==> default:  -- Domain type:       kvm
==> default:  -- Cpus:              2
==> default:  -- Feature:           acpi
==> default:  -- Feature:           apic
==> default:  -- Feature:           pae
==> default:  -- Memory:            2048M
==> default:  -- Management MAC:    
==> default:  -- Loader:            
==> default:  -- Nvram:             
==> default:  -- Base box:          generic/fedora29
==> default:  -- Storage pool:      default
==> default:  -- Image:             /var/lib/libvirt/images/fedora29-libvirt_default.img (32G)
==> default:  -- Volume Cache:      default
==> default:  -- Kernel:            
==> default:  -- Initrd:            
==> default:  -- Graphics Type:     vnc
==> default:  -- Graphics Port:     -1
==> default:  -- Graphics IP:       127.0.0.1
==> default:  -- Graphics Password: Not defined
==> default:  -- Video Type:        cirrus
==> default:  -- Video VRAM:        256
==> default:  -- Sound Type:	
==> default:  -- Keymap:            en-us
==> default:  -- TPM Path:          
==> default:  -- INPUT:             type=mouse, bus=ps2
==> default: Creating shared folders metadata...
==> default: Starting domain.
==> default: Waiting for domain to get an IP address...
==> default: Waiting for SSH to become available...
    default: 
    default: Vagrant insecure key detected. Vagrant will automatically replace
    default: this with a newly generated keypair for better security.
    default: 
    default: Inserting generated public key within guest...
    default: Removing insecure key from the guest if it's present...
    default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Configuring and enabling network interfaces...
    default: SSH address: 192.168.121.121:22
    default: SSH username: vagrant
    default: SSH auth method: private key

real	1m6.833s
user	0m9.898s
sys	0m3.821s

The simple script is one of several helpers I use to help automate the process of setting up the environment, testing the images, and cleaning up, etc. I had echo enabled, so you can see the actual commands used were pretty simple. I'm using the following Vagrantfile template, let me know if it fixes the problem. I've been meaning to automate the process of testing the boxes with just the vagrant init template, but haven't had the time (nor do I have enough robots to test both at the moment),

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure(2) do |config|

  config.vm.box = "generic/fedora29"

  config.vm.box_check_update = true
  config.vm.box_download_checksum = true
  config.vm.box_download_checksum_type = "sha256"

  config.vm.provider :libvirt do |v, override|
    v.driver = "kvm"
    v.video_vram = 256
    v.memory = 2048
    v.cpus = 2
  end

  config.vm.provider :hyperv do |v, override|
    v.maxmemory = 2048
    v.memory = 2048
    v.cpus = 2
  end

  config.vm.provider :virtualbox do |v, override|
    v.gui = false
    v.customize ["modifyvm", :id, "--memory", 2048]
    v.customize ["modifyvm", :id, "--cpus", 2]
  end

  ["vmware_fusion", "vmware_workstation", "vmware_desktop"].each do |provider|
    config.vm.provider provider do |v, override|
      v.ssh_info_public = true
      v.gui = false
      v.vmx["memsize"] = "2048"
      v.vmx["numvcpus"] = "2"
      v.vmx["cpuid.coresPerSocket"] = "1"
    end
  end

end

[ladar]

Moreover, with 2 kernels installed simultaneously in 1.8.48 box, I have to remove old kernel and specify explicitly for the dracut which initrd for which kernel version to build.

@gorshunovr It's funny you bring this up. I actually have a module whose job is cleaning up old kernels after going through an update/reboot cycle... but the Fedora version appears to be commented out because package-cleanup was removed and I never got around to investigating.

So thanks for the reminder. I tracked down the new command, and I'll update the module so the next time I run a full build (trying to do it every 2-3 weeks), the Fedora new boxes should only have one kernel installed (the most recent version available when the box is built).

[ladar]

The updated scripts should resolve this issue. If the next round of images > 1.8.48 still have multiple kernels installed, please let me know.