readdirplus: entry with attributes_follow = 0 about libnfs

[sahlberg]

Hi,

I have seen servers that does not support READDIRPLUS at all and there I
fallback to old READDIR (that contains no attributes)
and a GETATTR loop to fetch the attributes them self. This is currently
triggered only for the case where the server responds with an RPC FUNCRION
NOT SUPPORTED error.

Only time I have seen a server return all zero attributes for READDIRPLUS
is when there is another subvolume mounted ontop of it.

It should not be too horrible to solve this.
Can you collect a network trace to share with me so I can see exactly how
the reply from the server looks like.
The easiest solution here might be if I can detect this situation from the
reply so I can trigger the fallback for "bad" servers that I already have
for broken READDIRPLUS.

I am curious. What is the exact type of NFS server you use? As I said I
have not seen this before so I am a very curious :-)

On Mon, Apr 13, 2015 at 9:43 AM, doktorstick [email protected]
wrote:

If you have a server that returns the filenames in a readdirplus call, but
does not return the attributes, then libnfs presents the file's attributes
as all 0's. Instead, it should to do a followup GETATTR on the offending
entry(-ies).

—
Reply to this email directly or view it on GitHub
#115.

[doktorstick]

I'll see what I can do. The NetApp filer contains confidential information and is not under my control (i.e., I may not be able to setup a "safe" reproducible environment.)

I will say that examining various directories entries in the pcap, many have different fattr3.fsid, but this may be an artifact of how NetApp manages things.

[sahlberg]

It is really just the readdirplus reply I am interested in.

Maybe you can print it as ASCII using tshark and then edit and remove the
file-names themself
and thus remove any sensitive data.

On Tue, Apr 14, 2015 at 9:18 AM, doktorstick [email protected]
wrote:

I'll see what I can do. The NetApp filer contains confidential information
and is not under my control (i.e., I may not be able to setup a "safe"
reproducible environment.)

I will say that examining various directories entries in the pcap, many
have different fattr3.fsid, but this may be an artifact of how NetApp
manages things.

—
Reply to this email directly or view it on GitHub
#115 (comment).

[doktorstick]

$ tshark -r ffstat-rewrite.pcap -V -O rpc,nfs | obfus.py > ffstat-rewrite-obfus.txt

This has the mounting, RD+s, and LOOKUPs for one directory. I rewrote network info and substituted the filenames with a consistent mapping (i.e., file0005 is the same file everywhere file0005 is seen). You can grab it here: https://dl.dropboxusercontent.com/u/12952897/Bug/ffstat-rewrite-obfus.txt

[sahlberg]

Can you test this branch : readdirplus-weird-server-fix
I started working on triggering a fallback to READDIRPLUS based on whether
the reply contains entries without any attributes.

I think the approach for this patch is not workable for a solution in
"master" since I start realizing that there are several problems with the
approach.
However, it is still valuable as a test to check that using LOOKUP on these
"files" will work, which is still uncertain.

Is there something unusual about these files? such are they some kind of
snapshots or similar?

The only times I have seen an NFS server return null attributes for a
directory entry is in the case where the entry is a mountpoint on the
server on which a different filesystem is mounted.
For example if a server exports both /export/data and
/export/data/other-filesystem and these are two different filesystems.
For that case when you would try a READDIRPLUS of data or if you would
try a LOOKUP of /export/data/other-filesystem, then the server would
normally
return an entry without any attributes.
(Reason for this is because the protocol does not allow an NFS operation to
start in one filesystem and return a filehandle for a file in a different
filesystem.
This is also the reason why for these cases a normal nfs client would have
to mount BOTH filesystems from the server).

So, please try this branch and see if you can see the file attributes. If
that works then I can try to come up with a proper solution to drive the
failback to a LOOKUP loop.

On Mon, Apr 27, 2015 at 8:55 AM, doktorstick [email protected]
wrote:

$ tshark -r ffstat-rewrite.pcap -V -O rpc,nfs | obfus.py >
ffstat-rewrite-obfus.txt

This has the mounting, RD+s, and LOOKUPs for one directory. I rewrote
network info and substituted the filenames with a consistent mapping (i.e.,
file0005 is the same file everywhere file0005 is seen). You can grab it
here:
https://dl.dropboxusercontent.com/u/12952897/Bug/ffstat-rewrite-obfus.txt

—
Reply to this email directly or view it on GitHub
#115 (comment).

[doktorstick]

I'm being told that it is not an overlapping mount; I also issued a showmount -e $serverip and it returns one mountpoint: / (everyone). I'm hoping to get to tryout your branch today, tomorrow at the outside.

[sahlberg]

Ok, Also if you could do a 'stat ' on each of the files
without attributes from a command line prompt when mounting it with the
kernel nfs client.
And the same 'stat ' from a few files that does have attributes as
well as for the parent directory.

On Mon, May 4, 2015 at 9:05 AM, doktorstick [email protected]
wrote:

I'm being told that it is not an overlapping mount; I also issued a showmount
-e $serverip and it returns one mountpoint: / (everyone). I'm hoping to
get to tryout your branch today, tomorrow at the outside.

—
Reply to this email directly or view it on GitHub
#115 (comment).

[doktorstick]

I believe your instinct is correct. After reading about server exports in the NFS Illustrated book, checking the filesystem in question, I notice that it does change fsid. In this case:

fsid=AA /
fsid=BB /foo/
fsid=CC /bar/

I made a small program to readdir on a specified directory. Grep'ing the tshark output--for reference, there are 2147 objects inside of /:

root# grep attributes_follow rd.tshark | sort | uniq -c
     39             attributes_follow: no value (0)
      2         attributes_follow: no value (0)
      2             attributes_follow: value follows (1)
   4348         attributes_follow: value follows (1)

Keep in mind, this is using your test branch you pointed me at. I didn't validate the data I'm getting back is correct with your test code, however.

The simple code isn't much more complicated than:

  nfs_opendir (nfs, "/foo", &nfsdir);
  // snip readdir loop
  nfs_closedir (nfs, nfsdir);

Okay, so I went back to the libnfs code base I'm using, which is a few version back and tried the simple test code:

root# grep attributes_follow rd-old.tshark | sort | uniq -c
   2145             attributes_follow: no value (0)
      2             attributes_follow: value follows (1)
     53         attributes_follow: value follows (1)

Sure enough, there's what I expect.

When looking at what the linux client is doing it does:

$ mount -t nfs ...
$ stat ~/mnt/foo
Network File System, FSINFO Call DH: 0xb8c81e4f
Network File System, FSINFO Reply
Network File System, PATHCONF Call DH: 0xb8c81e4f
Network File System, PATHCONF Reply
Network File System, FSINFO Call DH: 0xb8c81e4f
Network File System, FSINFO Reply
Network File System, ACCESS Call, FH: 0xb8c81e4f, [Check: RD LU MD XT DL]
Network File System, ACCESS Reply, [Access Denied: MD XT DL], [Allowed: RD LU]
Network File System, LOOKUP Call DH: 0xb8c81e4f/foo
# Including more information since you'll observe that it returns the attributes even
# though it is on a different filesystem.
Network File System, LOOKUP Reply FH: 0x26b49599
    [Program Version: 3]
    [V3 Procedure: LOOKUP (3)]
    Status: NFS3_OK (0)
    object
        length: 44
        [hash (CRC-32): 0x26b49599]
        decode type as: unknown
        filehandle: 00010000676e00800000000040000000883d25290a040080...
    obj_attributes  Directory mode: 0777 uid: 625 gid: 3000
        attributes_follow: value follows (1)
        attributes  Directory mode: 0777 uid: 625 gid: 3000
            Type: Directory (2)
            Mode: 0777, S_IRUSR, S_IWUSR, S_IXUSR, S_IRGRP, S_IWGRP, S_IXGRP, S_IROTH, S_IWOTH, S_IXOTH
                .... .... .... .... .... 0... .... .... = S_ISUID: No
                .... .... .... .... .... .0.. .... .... = S_ISGID: No
                .... .... .... .... .... ..0. .... .... = S_ISVTX: No
                .... .... .... .... .... ...1 .... .... = S_IRUSR: Yes
                .... .... .... .... .... .... 1... .... = S_IWUSR: Yes
                .... .... .... .... .... .... .1.. .... = S_IXUSR: Yes
                .... .... .... .... .... .... ..1. .... = S_IRGRP: Yes
                .... .... .... .... .... .... ...1 .... = S_IWGRP: Yes
                .... .... .... .... .... .... .... 1... = S_IXGRP: Yes
                .... .... .... .... .... .... .... .1.. = S_IROTH: Yes
                .... .... .... .... .... .... .... ..1. = S_IWOTH: Yes
                .... .... .... .... .... .... .... ...1 = S_IXOTH: Yes
            nlink: 81
            uid: 625
            gid: 3000
            size: 40960
            used: 45056
            rdev: 0,0
                specdata1: 0
                specdata2: 0
            fsid: 0x0000000080006e67 (2147511911)
            fileid: 1852244032
            atime: May  5, 2015 20:53:03.958862000 UTC
                seconds: 1430859183
                nano seconds: 958862000
            mtime: May  5, 2015 21:29:57.018064000 UTC
                seconds: 1430861397
                nano seconds: 18064000
            ctime: May  5, 2015 21:29:57.018064000 UTC
                seconds: 1430861397
                nano seconds: 18064000
    dir_attributes  Directory mode: 0755 uid: 0 gid: 0
        attributes_follow: value follows (1)
        attributes  Directory mode: 0755 uid: 0 gid: 0
            Type: Directory (2)
            Mode: 0755, S_IRUSR, S_IWUSR, S_IXUSR, S_IRGRP, S_IXGRP, S_IROTH, S_IXOTH
                .... .... .... .... .... 0... .... .... = S_ISUID: No
                .... .... .... .... .... .0.. .... .... = S_ISGID: No
                .... .... .... .... .... ..0. .... .... = S_ISVTX: No
                .... .... .... .... .... ...1 .... .... = S_IRUSR: Yes
                .... .... .... .... .... .... 1... .... = S_IWUSR: Yes
                .... .... .... .... .... .... .1.. .... = S_IXUSR: Yes
                .... .... .... .... .... .... ..1. .... = S_IRGRP: Yes
                .... .... .... .... .... .... ...0 .... = S_IWGRP: No
                .... .... .... .... .... .... .... 1... = S_IXGRP: Yes
                .... .... .... .... .... .... .... .1.. = S_IROTH: Yes
                .... .... .... .... .... .... .... ..0. = S_IWOTH: No
                .... .... .... .... .... .... .... ...1 = S_IXOTH: Yes
            nlink: 2146
            uid: 0
            gid: 0
            size: 356352
            used: 360448
            rdev: 0,0
                specdata1: 0
                specdata2: 0
            fsid: 0x000000008000040a (2147484682)
            fileid: 67764288
            atime: May  5, 2015 21:00:14.726911000 UTC
                seconds: 1430859614
                nano seconds: 726911000
            mtime: Apr 23, 2015 16:16:59.608679000 UTC
                seconds: 1429805819
                nano seconds: 608679000
            ctime: Apr 23, 2015 16:16:59.608679000 UTC
                seconds: 1429805819
                nano seconds: 608679000

And then on a file inside of foo:

$ stat ~/mnt/foo/.
Network File System, LOOKUP Call DH: 0xb8c81e4f/foo
Network File System, LOOKUP Reply FH: 0x26b49599
Network File System, FSINFO Call DH: 0x26b49599
Network File System, FSINFO Reply
Network File System, PATHCONF Call DH: 0x26b49599
Network File System, PATHCONF Reply
Network File System, FSINFO Call DH: 0x26b49599
Network File System, FSINFO Reply
Network File System, ACCESS Call, FH: 0x26b49599, [Check: RD LU MD XT DL]
Network File System, ACCESS Reply, [Allowed: RD LU MD XT DL]

To me this tells me that the client noticed that the fsid was different, so it had to do a subsequent FSINFO to track that filesystem.

I thought you might be interested when I do a fresh mount and an ls /:

root# grep 'Network File System, READDIRPLUS Call' ls.tshark | sort | uniq -c
    303 Network File System, READDIRPLUS Call FH: 0xb8c81e4f
root# grep 'Network File System, LOOKUP Call' ls.tshark | wc -l
2144
root# grep "attributes_follow" ls.tshark | sort | uniq -c
   2266             attributes_follow: no value (0)
      1         attributes_follow: no value (0)
      2             attributes_follow: value follows (1)
   4593         attributes_follow: value follows (1)

[doktorstick]

You can also cause the "no attributes" without needing other fsid'd filesystems. Not that this is the same problem and can be solved in the same way (since it's a permission problem). But, it may be useful for your testing.

notroot@filer:/exports/test2/a/b$ ls -la .
total 12
drwxrwxr-x 3 notroot notroot 4096 May 11 22:51 .
drwxrwxr-x 3 notroot notroot 4096 May 11 22:51 ..
dr-------- 3 notroot notroot 4096 May 12 00:22 c
notroot@filer:/exports/test2/a/b$ ls -la c
ls: cannot access c/d: Permission denied
ls: cannot access c/c.1: Permission denied
ls: cannot access c/c.2: Permission denied
ls: cannot access c/.: Permission denied
ls: cannot access c/..: Permission denied
total 0
d????????? ? ? ? ?            ? .
d????????? ? ? ? ?            ? ..
-????????? ? ? ? ?            ? c.1
-????????? ? ? ? ?            ? c.2
d????????? ? ? ? ?            ? d
notroot@filer:/exports/test2/a/b$

[sahlberg]

Sorry but I completely forgot about this.

I have reworked it completely so that I now conditionaly use LOOKUPs for
those entrues where we do not get any attributes.

Please test this branch :
readdirplos-lookup

Once it is good I will merge it to master and release a new version.

Best regards
ronnie sahlberg

On Tue, May 12, 2015 at 1:31 PM, doktorstick [email protected]
wrote:

You can also cause the "no attributes" without needing other fsid'd
filesystems.

notroot@filer:/exports/test2/a/b$ ls -la .
total 12
drwxrwxr-x 3 notroot notroot 4096 May 11 22:51 .
drwxrwxr-x 3 notroot notroot 4096 May 11 22:51 ..
dr-------- 3 notroot notroot 4096 May 12 00:22 c
notroot@filer:/exports/test2/a/b$ ls -la c
ls: cannot access c/d: Permission denied
ls: cannot access c/c.1: Permission denied
ls: cannot access c/c.2: Permission denied
ls: cannot access c/.: Permission denied
ls: cannot access c/..: Permission denied
total 0
d????????? ? ? ? ? ? .
d????????? ? ? ? ? ? ..
-????????? ? ? ? ? ? c.1
-????????? ? ? ? ? ? c.2
d????????? ? ? ? ? ? d
notroot@filer:/exports/test2/a/b$

—
Reply to this email directly or view it on GitHub
#115 (comment).

[doktorstick]

It appears to be working properly. My observation was that it performed the READDIRPLUS as normal, and then sent a bunch of LOOKUPs out back-to-back and the collected the results.

I had only one entry (of 2043) that the LOOKUP did not fill in, and that was because of a NFS3ERR_ACCESS on the LOOKUP.

I call it success. Once it is in the main branch, I'll end up running it through its paces across other filers/mounts, too.

[sahlberg]

Thanks for verifying it.

I merged it into master so it will be part of the next release.

On Wed, Jun 3, 2015 at 1:59 PM, doktorstick [email protected]
wrote:

It appears to be working properly. My observation was that it performed
the READDIRPLUS as normal, and then sent a bunch of LOOKUPs out
back-to-back and the collected the results.

I had only one entry (of 2043) that the LOOKUP did not fill in, and that
was because of a NFS3ERR_ACCESS on the LOOKUP.

I call it success. Once it is in the main branch, I'll end up running it
through its paces across other filers/mounts, too.

—
Reply to this email directly or view it on GitHub
#115 (comment).