Failed to mount Resource Disk about walinuxagent

[yuezh]

@phstudy. The log you attached shows resource disk was mount correctly. If the resource disk is large, it will take some time to mount it.

[phstudy]

@yuezh Is it normal to take 5 minutes+ to mount the resource disk? you can see the log console.TXT.

[thomas1206]

5 minutes+ seems too long. Normally it wouldn't take that long. Does it always happen for your VM? Azure storage has some throttling configuration for each subscription. So if you've too many storage IOs (like formatting resource disk) in a short time, you might hit it.

[szarkos]

I'm assuming the waagent.txt is from your serial boot log, gauging from some of the missing characters and timestamps are a bit off. Can you provide the full log from /var/log/waagent.log?

The first log from console.txt suggests that you resolved the issue by manually restarting the agent service. When this mount issue occurs again can you confirm that waagent is running, and if not can you query journald to ask why and paste the output here?

[phstudy]

@thomas1206 This VM is created for testing this issue only, because I found it happened in two of my VMs. The only thing I do before testing is sudo yum update.

@szarkos Manually restarting waagent does not always work. I have attached the full log.
20160127_console.txt
20160127_waagent.txt

[LiliDeng]

@phstudy Hi, could you repro this issue stably? Would you mind telling me the steps for repro?
1 Setup VM against gallery image (CentOS7.1), which size? which location? Is it under ASM (classic) mode?
2 After machine can be accessed, do yum update
3 then check /dev/sdb1 is mounted? reboot many times to double check?
Thanks!

[phstudy]

@LiliDeng Yes, I can repro it stably. The steps were in 20160127_console.txt.

The VM settings:

• Deploy Tool: Azure quickstart template
• Mode: ARM
• Size: Standard_A1
• Location: Southeast Asia

Azure quickstart template settings:

• centos71_single_template.json.txt
• vm-waagent-test.parameters.json.txt

VM spec:

[LiliDeng]

@phstudy thanks, I will try to repro from my side based on your provide info.

[lizzha]

@phstudy we're able to repro this issue, created a CentOS 7.1 VM, do "yum update", and then reboot.
In waagent.log the disk is mounted:
2016/01/27 09:37:15 Resource disk (/dev/sdb1) is mounted at /mnt/resource with fstype ext4

Also found following logs in /var/log/messages
Jan 27 09:37:15 lilicentos71 kernel: sdb: sdb1
Jan 27 09:37:15 lilicentos71 kernel: SGI XFS with ACLs, security attributes, no debug enabled
Jan 27 09:37:15 lilicentos71 kernel: EXT4-fs (sdb1): mounted filesystem with ordered data mode. Opts: (null)
Jan 27 09:37:15 lilicentos71 systemd: Unmounting /mnt/resource...
Jan 27 09:37:15 lilicentos71 systemd: Requested transaction contradicts existing jobs: Resource deadlock avoided
Jan 27 09:37:16 lilicentos71 systemd: Unmounted /mnt/resource.

We could see that the resource disk was mounted at first, but then unmounted by systemd. We need to look into why systemd did that.

[lizzha]

@phstudy Found some discussion of the issue, seems it's issue in systemd-219, and has been resovled in github.
https://bugs.freedesktop.org/show_bug.cgi?id=89383
https://bugs.gentoo.org/show_bug.cgi?id=541402
https://forums.gentoo.org/viewtopic-t-998456-highlight-.html

For now, as a workaround, probably you could try to revert the systemd to an older version.

[lizzha]

Also, we will file a bug to Redhat Bugzilla for them to adopt the fix.

[phstudy]

@lizzha I will try old version systemd, thanks for your information 😄

[phstudy]

@lizzha I found there is no easy to downgrade systemd...

[ahmetb]

@lizzha is this ok to close now?

[lizzha]

As it's not a waagent issue, close it.