Comments (12)
Hi,
I have seen servers that does not support READDIRPLUS at all and there I
fallback to old READDIR (that contains no attributes)
and a GETATTR loop to fetch the attributes them self. This is currently
triggered only for the case where the server responds with an RPC FUNCRION
NOT SUPPORTED error.
Only time I have seen a server return all zero attributes for READDIRPLUS
is when there is another subvolume mounted ontop of it.
It should not be too horrible to solve this.
Can you collect a network trace to share with me so I can see exactly how
the reply from the server looks like.
The easiest solution here might be if I can detect this situation from the
reply so I can trigger the fallback for "bad" servers that I already have
for broken READDIRPLUS.
I am curious. What is the exact type of NFS server you use? As I said I
have not seen this before so I am a very curious :-)
On Mon, Apr 13, 2015 at 9:43 AM, doktorstick [email protected]
wrote:
If you have a server that returns the filenames in a readdirplus call, but
does not return the attributes, then libnfs presents the file's attributes
as all 0's. Instead, it should to do a followup GETATTR on the offending
entry(-ies).—
Reply to this email directly or view it on GitHub
#115.
from libnfs.
I'll see what I can do. The NetApp filer contains confidential information and is not under my control (i.e., I may not be able to setup a "safe" reproducible environment.)
I will say that examining various directories entries in the pcap, many have different fattr3.fsid
, but this may be an artifact of how NetApp manages things.
from libnfs.
It is really just the readdirplus reply I am interested in.
Maybe you can print it as ASCII using tshark and then edit and remove the
file-names themself
and thus remove any sensitive data.
On Tue, Apr 14, 2015 at 9:18 AM, doktorstick [email protected]
wrote:
I'll see what I can do. The NetApp filer contains confidential information
and is not under my control (i.e., I may not be able to setup a "safe"
reproducible environment.)I will say that examining various directories entries in the pcap, many
have different fattr3.fsid, but this may be an artifact of how NetApp
manages things.—
Reply to this email directly or view it on GitHub
#115 (comment).
from libnfs.
$ tshark -r ffstat-rewrite.pcap -V -O rpc,nfs | obfus.py > ffstat-rewrite-obfus.txt
This has the mounting, RD+s, and LOOKUPs for one directory. I rewrote network info and substituted the filenames with a consistent mapping (i.e., file0005
is the same file everywhere file0005
is seen). You can grab it here: https://dl.dropboxusercontent.com/u/12952897/Bug/ffstat-rewrite-obfus.txt
from libnfs.
Can you test this branch : readdirplus-weird-server-fix
I started working on triggering a fallback to READDIRPLUS based on whether
the reply contains entries without any attributes.
I think the approach for this patch is not workable for a solution in
"master" since I start realizing that there are several problems with the
approach.
However, it is still valuable as a test to check that using LOOKUP on these
"files" will work, which is still uncertain.
Is there something unusual about these files? such are they some kind of
snapshots or similar?
The only times I have seen an NFS server return null attributes for a
directory entry is in the case where the entry is a mountpoint on the
server on which a different filesystem is mounted.
For example if a server exports both /export/data and
/export/data/other-filesystem and these are two different filesystems.
For that case when you would try a READDIRPLUS of data or if you would
try a LOOKUP of /export/data/other-filesystem, then the server would
normally
return an entry without any attributes.
(Reason for this is because the protocol does not allow an NFS operation to
start in one filesystem and return a filehandle for a file in a different
filesystem.
This is also the reason why for these cases a normal nfs client would have
to mount BOTH filesystems from the server).
So, please try this branch and see if you can see the file attributes. If
that works then I can try to come up with a proper solution to drive the
failback to a LOOKUP loop.
On Mon, Apr 27, 2015 at 8:55 AM, doktorstick [email protected]
wrote:
$ tshark -r ffstat-rewrite.pcap -V -O rpc,nfs | obfus.py >
ffstat-rewrite-obfus.txtThis has the mounting, RD+s, and LOOKUPs for one directory. I rewrote
network info and substituted the filenames with a consistent mapping (i.e.,
file0005 is the same file everywhere file0005 is seen). You can grab it
here:
https://dl.dropboxusercontent.com/u/12952897/Bug/ffstat-rewrite-obfus.txt—
Reply to this email directly or view it on GitHub
#115 (comment).
from libnfs.
I'm being told that it is not an overlapping mount; I also issued a showmount -e $serverip
and it returns one mountpoint: / (everyone)
. I'm hoping to get to tryout your branch today, tomorrow at the outside.
from libnfs.
Ok, Also if you could do a 'stat ' on each of the files
without attributes from a command line prompt when mounting it with the
kernel nfs client.
And the same 'stat ' from a few files that does have attributes as
well as for the parent directory.
On Mon, May 4, 2015 at 9:05 AM, doktorstick [email protected]
wrote:
I'm being told that it is not an overlapping mount; I also issued a showmount
-e $serverip and it returns one mountpoint: / (everyone). I'm hoping to
get to tryout your branch today, tomorrow at the outside.—
Reply to this email directly or view it on GitHub
#115 (comment).
from libnfs.
I believe your instinct is correct. After reading about server exports in the NFS Illustrated book, checking the filesystem in question, I notice that it does change fsid
. In this case:
fsid=AA /
fsid=BB /foo/
fsid=CC /bar/
I made a small program to readdir
on a specified directory. Grep'ing the tshark
output--for reference, there are 2147 objects inside of /
:
root# grep attributes_follow rd.tshark | sort | uniq -c
39 attributes_follow: no value (0)
2 attributes_follow: no value (0)
2 attributes_follow: value follows (1)
4348 attributes_follow: value follows (1)
Keep in mind, this is using your test branch you pointed me at. I didn't validate the data I'm getting back is correct with your test code, however.
The simple code isn't much more complicated than:
nfs_opendir (nfs, "/foo", &nfsdir);
// snip readdir loop
nfs_closedir (nfs, nfsdir);
Okay, so I went back to the libnfs
code base I'm using, which is a few version back and tried the simple test code:
root# grep attributes_follow rd-old.tshark | sort | uniq -c
2145 attributes_follow: no value (0)
2 attributes_follow: value follows (1)
53 attributes_follow: value follows (1)
Sure enough, there's what I expect.
When looking at what the linux client is doing it does:
$ mount -t nfs ...
$ stat ~/mnt/foo
Network File System, FSINFO Call DH: 0xb8c81e4f
Network File System, FSINFO Reply
Network File System, PATHCONF Call DH: 0xb8c81e4f
Network File System, PATHCONF Reply
Network File System, FSINFO Call DH: 0xb8c81e4f
Network File System, FSINFO Reply
Network File System, ACCESS Call, FH: 0xb8c81e4f, [Check: RD LU MD XT DL]
Network File System, ACCESS Reply, [Access Denied: MD XT DL], [Allowed: RD LU]
Network File System, LOOKUP Call DH: 0xb8c81e4f/foo
# Including more information since you'll observe that it returns the attributes even
# though it is on a different filesystem.
Network File System, LOOKUP Reply FH: 0x26b49599
[Program Version: 3]
[V3 Procedure: LOOKUP (3)]
Status: NFS3_OK (0)
object
length: 44
[hash (CRC-32): 0x26b49599]
decode type as: unknown
filehandle: 00010000676e00800000000040000000883d25290a040080...
obj_attributes Directory mode: 0777 uid: 625 gid: 3000
attributes_follow: value follows (1)
attributes Directory mode: 0777 uid: 625 gid: 3000
Type: Directory (2)
Mode: 0777, S_IRUSR, S_IWUSR, S_IXUSR, S_IRGRP, S_IWGRP, S_IXGRP, S_IROTH, S_IWOTH, S_IXOTH
.... .... .... .... .... 0... .... .... = S_ISUID: No
.... .... .... .... .... .0.. .... .... = S_ISGID: No
.... .... .... .... .... ..0. .... .... = S_ISVTX: No
.... .... .... .... .... ...1 .... .... = S_IRUSR: Yes
.... .... .... .... .... .... 1... .... = S_IWUSR: Yes
.... .... .... .... .... .... .1.. .... = S_IXUSR: Yes
.... .... .... .... .... .... ..1. .... = S_IRGRP: Yes
.... .... .... .... .... .... ...1 .... = S_IWGRP: Yes
.... .... .... .... .... .... .... 1... = S_IXGRP: Yes
.... .... .... .... .... .... .... .1.. = S_IROTH: Yes
.... .... .... .... .... .... .... ..1. = S_IWOTH: Yes
.... .... .... .... .... .... .... ...1 = S_IXOTH: Yes
nlink: 81
uid: 625
gid: 3000
size: 40960
used: 45056
rdev: 0,0
specdata1: 0
specdata2: 0
fsid: 0x0000000080006e67 (2147511911)
fileid: 1852244032
atime: May 5, 2015 20:53:03.958862000 UTC
seconds: 1430859183
nano seconds: 958862000
mtime: May 5, 2015 21:29:57.018064000 UTC
seconds: 1430861397
nano seconds: 18064000
ctime: May 5, 2015 21:29:57.018064000 UTC
seconds: 1430861397
nano seconds: 18064000
dir_attributes Directory mode: 0755 uid: 0 gid: 0
attributes_follow: value follows (1)
attributes Directory mode: 0755 uid: 0 gid: 0
Type: Directory (2)
Mode: 0755, S_IRUSR, S_IWUSR, S_IXUSR, S_IRGRP, S_IXGRP, S_IROTH, S_IXOTH
.... .... .... .... .... 0... .... .... = S_ISUID: No
.... .... .... .... .... .0.. .... .... = S_ISGID: No
.... .... .... .... .... ..0. .... .... = S_ISVTX: No
.... .... .... .... .... ...1 .... .... = S_IRUSR: Yes
.... .... .... .... .... .... 1... .... = S_IWUSR: Yes
.... .... .... .... .... .... .1.. .... = S_IXUSR: Yes
.... .... .... .... .... .... ..1. .... = S_IRGRP: Yes
.... .... .... .... .... .... ...0 .... = S_IWGRP: No
.... .... .... .... .... .... .... 1... = S_IXGRP: Yes
.... .... .... .... .... .... .... .1.. = S_IROTH: Yes
.... .... .... .... .... .... .... ..0. = S_IWOTH: No
.... .... .... .... .... .... .... ...1 = S_IXOTH: Yes
nlink: 2146
uid: 0
gid: 0
size: 356352
used: 360448
rdev: 0,0
specdata1: 0
specdata2: 0
fsid: 0x000000008000040a (2147484682)
fileid: 67764288
atime: May 5, 2015 21:00:14.726911000 UTC
seconds: 1430859614
nano seconds: 726911000
mtime: Apr 23, 2015 16:16:59.608679000 UTC
seconds: 1429805819
nano seconds: 608679000
ctime: Apr 23, 2015 16:16:59.608679000 UTC
seconds: 1429805819
nano seconds: 608679000
And then on a file inside of foo
:
$ stat ~/mnt/foo/.
Network File System, LOOKUP Call DH: 0xb8c81e4f/foo
Network File System, LOOKUP Reply FH: 0x26b49599
Network File System, FSINFO Call DH: 0x26b49599
Network File System, FSINFO Reply
Network File System, PATHCONF Call DH: 0x26b49599
Network File System, PATHCONF Reply
Network File System, FSINFO Call DH: 0x26b49599
Network File System, FSINFO Reply
Network File System, ACCESS Call, FH: 0x26b49599, [Check: RD LU MD XT DL]
Network File System, ACCESS Reply, [Allowed: RD LU MD XT DL]
To me this tells me that the client noticed that the fsid
was different, so it had to do a subsequent FSINFO
to track that filesystem.
I thought you might be interested when I do a fresh mount and an ls /
:
root# grep 'Network File System, READDIRPLUS Call' ls.tshark | sort | uniq -c
303 Network File System, READDIRPLUS Call FH: 0xb8c81e4f
root# grep 'Network File System, LOOKUP Call' ls.tshark | wc -l
2144
root# grep "attributes_follow" ls.tshark | sort | uniq -c
2266 attributes_follow: no value (0)
1 attributes_follow: no value (0)
2 attributes_follow: value follows (1)
4593 attributes_follow: value follows (1)
from libnfs.
You can also cause the "no attributes" without needing other fsid
'd filesystems. Not that this is the same problem and can be solved in the same way (since it's a permission problem). But, it may be useful for your testing.
notroot@filer:/exports/test2/a/b$ ls -la .
total 12
drwxrwxr-x 3 notroot notroot 4096 May 11 22:51 .
drwxrwxr-x 3 notroot notroot 4096 May 11 22:51 ..
dr-------- 3 notroot notroot 4096 May 12 00:22 c
notroot@filer:/exports/test2/a/b$ ls -la c
ls: cannot access c/d: Permission denied
ls: cannot access c/c.1: Permission denied
ls: cannot access c/c.2: Permission denied
ls: cannot access c/.: Permission denied
ls: cannot access c/..: Permission denied
total 0
d????????? ? ? ? ? ? .
d????????? ? ? ? ? ? ..
-????????? ? ? ? ? ? c.1
-????????? ? ? ? ? ? c.2
d????????? ? ? ? ? ? d
notroot@filer:/exports/test2/a/b$
from libnfs.
Sorry but I completely forgot about this.
I have reworked it completely so that I now conditionaly use LOOKUPs for
those entrues where we do not get any attributes.
Please test this branch :
readdirplos-lookup
Once it is good I will merge it to master and release a new version.
Best regards
ronnie sahlberg
On Tue, May 12, 2015 at 1:31 PM, doktorstick [email protected]
wrote:
You can also cause the "no attributes" without needing other fsid'd
filesystems.notroot@filer:/exports/test2/a/b$ ls -la .
total 12
drwxrwxr-x 3 notroot notroot 4096 May 11 22:51 .
drwxrwxr-x 3 notroot notroot 4096 May 11 22:51 ..
dr-------- 3 notroot notroot 4096 May 12 00:22 c
notroot@filer:/exports/test2/a/b$ ls -la c
ls: cannot access c/d: Permission denied
ls: cannot access c/c.1: Permission denied
ls: cannot access c/c.2: Permission denied
ls: cannot access c/.: Permission denied
ls: cannot access c/..: Permission denied
total 0
d????????? ? ? ? ? ? .
d????????? ? ? ? ? ? ..
-????????? ? ? ? ? ? c.1
-????????? ? ? ? ? ? c.2
d????????? ? ? ? ? ? d
notroot@filer:/exports/test2/a/b$—
Reply to this email directly or view it on GitHub
#115 (comment).
from libnfs.
It appears to be working properly. My observation was that it performed the READDIRPLUS as normal, and then sent a bunch of LOOKUPs out back-to-back and the collected the results.
I had only one entry (of 2043) that the LOOKUP did not fill in, and that was because of a NFS3ERR_ACCESS on the LOOKUP.
I call it success. Once it is in the main branch, I'll end up running it through its paces across other filers/mounts, too.
from libnfs.
Thanks for verifying it.
I merged it into master so it will be part of the next release.
On Wed, Jun 3, 2015 at 1:59 PM, doktorstick [email protected]
wrote:
It appears to be working properly. My observation was that it performed
the READDIRPLUS as normal, and then sent a bunch of LOOKUPs out
back-to-back and the collected the results.I had only one entry (of 2043) that the LOOKUP did not fill in, and that
was because of a NFS3ERR_ACCESS on the LOOKUP.I call it success. Once it is in the main branch, I'll end up running it
through its paces across other filers/mounts, too.—
Reply to this email directly or view it on GitHub
#115 (comment).
from libnfs.
Related Issues (20)
- NFS v3 CREATE issues unnessesary LOOKUP request
- nfs4 open with share_deny modes? HOT 2
- question about the locks semantic HOT 1
- NFSv4 Basic chown test fails in Ubuntu 23.10 VM HOT 1
- malloc() can return NULL in OOM conditions, resulting in a NULL dereference whilst trying to create an error string HOT 6
- nfs_mount_async failed. Failed to start connection. Failed to start connection. Trying to connect while already connected HOT 3
- readme.md references README.win32.txt, but there is no such file HOT 2
- Is it permissible to initialize the NFS channel through the extended main program's socket channel? HOT 1
- RFE: is it possible to start making github releases?🤔 HOT 2
- nfsv3 create file tries creating files with junk characters HOT 2
- Does libnfs support RPC over TLS? HOT 1
- nfsv3 async write sometimes send malformed packets HOT 16
- nfs-ls does not work with NFSv4-only server HOT 4
- Specifying custom (non-2049) port number with RFC 1738 <hostport> URL syntax does not work HOT 1
- NFSv3 UDP connection segmentation fault. HOT 2
- Build error on macOS: `error: 'IFNAMSIZ' undeclared here (not in a function)` etc. HOT 2
- Re: rpc_set_error: set error to "out of memory" if the malloc failed HOT 4
- libnfs-5.0.3 fails to build with GCC14 with [-Wimplicit-function-declaration] HOT 1
- Filename with special symbols in it not coming properly HOT 1
- Have we tested TCP_NODELAY option with RPC connections? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from libnfs.