kahing / goofys Goto Github PK

View Code? Open in Web Editor NEW

5.0K 117.0 519.0 4.8 MB

a high-performance, POSIX-ish Amazon S3 file system written in Go

License: Apache License 2.0

Go 93.34% Makefile 0.07% Shell 5.28% Python 0.49% Gnuplot 0.55% Dockerfile 0.28%

s3-bucket s3 posix filesystem aws-s3 cloud-storage golang fuse gcs google-cloud-storage

goofys's Introduction

Goofys is a high-performance, POSIX-ish Amazon S3 file system written in Go

Overview

Goofys allows you to mount an S3 bucket as a filey system.

It's a Filey System instead of a File System because goofys strives for performance first and POSIX second. Particularly things that are difficult to support on S3 or would translate into more than one round-trip would either fail (random writes) or faked (no per-file permission). Goofys does not have an on disk data cache (checkout catfs), and consistency model is close-to-open.

Installation

On Linux, install via pre-built binaries. You may also need to install fuse too if you want to mount it on startup.
On macOS, install via Homebrew:

$ brew cask install osxfuse
$ brew install goofys

Or build from source with Go 1.10 or later:

$ export GOPATH=$HOME/work
$ go get github.com/kahing/goofys
$ go install github.com/kahing/goofys

Usage

$ cat ~/.aws/credentials
[default]
aws_access_key_id = AKID1234567890
aws_secret_access_key = MY-SECRET-KEY
$ $GOPATH/bin/goofys <bucket> <mountpoint>
$ $GOPATH/bin/goofys <bucket:prefix> <mountpoint> # if you only want to mount objects under a prefix

Users can also configure credentials via the AWS CLI or the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.

To mount an S3 bucket on startup, make sure the credential is configured for root, and can add this to /etc/fstab:

goofys#bucket   /mnt/mountpoint        fuse     _netdev,allow_other,--file-mode=0666,--dir-mode=0777    0       0

Got more questions? Check out questions other people asked

Benchmark

Using --stat-cache-ttl 1s --type-cache-ttl 1s for goofys -ostat_cache_expire=1 for s3fs to simulate cold runs. Detail for the benchmark can be found in bench.sh. Raw data is available as well. The test was run on an EC2 m5.4xlarge in us-west-2a connected to a bucket in us-west-2. Units are seconds.

To run the benchmark, configure EC2's instance role to be able to write to $TESTBUCKET, and then do:

$ sudo docker run -e BUCKET=$TESTBUCKET -e CACHE=false --rm --privileged --net=host -v /tmp/cache:/tmp/cache kahing/goofys-bench
# result will be written to $TESTBUCKET

License

Licensed under the Apache License, Version 2.0

Current Status

goofys has been tested under Linux and macOS.

List of non-POSIX behaviors/limitations:

only sequential writes supported
does not store file mode/owner/group
- use --(dir|file)-mode or --(uid|gid) options
does not support symlink or hardlink
ctime, atime is always the same as mtime
cannot rename directories with more than 1000 children
unlink returns success even if file is not present
fsync is ignored, files are only flushed on close

Compatibility with non-AWS S3

goofys has been tested with the following non-AWS S3 providers:

Amplidata / WD ActiveScale
Ceph (ex: Digital Ocean Spaces, DreamObjects, gridscale)
EdgeFS
EMC Atmos
Google Cloud Storage
Minio (limited)
OpenStack Swift
S3Proxy
Scaleway
Wasabi

Additionally, goofys also works with the following non-S3 object stores:

Azure Blob Storage
Azure Data Lake Gen1
Azure Data Lake Gen2

References

Data is stored on Amazon S3
Amazon SDK for Go
Other related fuse filesystems
- catfs: caching layer that can be used with goofys
- s3fs: another popular filesystem for S3
- gcsfuse: filesystem for Google Cloud Storage. Goofys borrowed some skeleton code from this project.
S3Proxy is used for go test
fuse binding, also used by gcsfuse

goofys's People

Contributors

Stargazers

Watchers

Forkers

gaul waynehuang13 ketqi scott-he louisyoo caidongyun jango2015 lrowe choleraehyq imjerrybao mantyr x5u joonhochoi reo7sp mdabbagh88 cloudxtreme alex1528 henrypfhu vadirajks codeskyblue grace-hung651 pomero cm-moriwaki blampe fremanzhang nunb crazed robbkistler xiaolushare ieee0824 monken andyliuliming absolutarin sspans di-stars mfowlewebs hirsaeki-mki perxhub jlertle freelyfly84 ahmedaljunied cread jamvideosolutions sungjunyoung lalyos sanwan unclaimedpants enricomariafusi andypern prai14 vinayasathyanarayana albundy83 archerslaw declan94 pawanrana dummy-ai 40a gophersgang masahide guangminglion manni901 jtwang83 robmadole masterxavierfox khoan zeyingli hit1943 inevity pranaysinghdev dxhome cleverlzc dreamingbinary c4pt0r nekojarashi javilumbrales sqleejan golib queniao dingjs moonchaser diogouchoas tbuenger-zz any-fuse akitoueno-fr guersam doomoolmori alageshan-m etsangsplk telunsu avonshire longshanksmo taichikageyama zhezhel igiroux akhilesh2018git bob01 myena jakedsouza sagansystems cloudferro

goofys's Issues

Build failure

I am trying to install goofys 0.0.2 on a SUSE 13.1 system, but getting the following messages/errors (the preview shows some big bold text. If it posts as that, I apologize in advance, I'm not doing that on purpose):

goofys-0.0.2> make
wget https://oss.sonatype.org/content/repositories/snapshots/org/gaul/s3proxy/1.5.0-SNAPSHOT/s3proxy-1.5.0-20151012.215145-8-jar-with-dependencies.jar -O s3proxy.jar
--2015-11-07 02:56:19-- https://oss.sonatype.org/content/repositories/snapshots/org/gaul/s3proxy/1.5.0-SNAPSHOT/s3proxy-1.5.0-20151012.215145-8-jar-with-dependencies.jar
Resolving oss.sonatype.org (oss.sonatype.org)... 107.23.94.232, 52.21.253.142, 52.21.106.20
Connecting to oss.sonatype.org (oss.sonatype.org)|107.23.94.232|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 7743535 (7.4M) [application/java-archive]
Saving to: ‘s3proxy.jar’

s3proxy.jar 100%[================================================>] 7.38M 3.49MB/s in 2.1s

2015-11-07 02:56:21 (3.49 MB/s) - ‘s3proxy.jar’ saved [7743535/7743535]

./test/run-tests.sh
warning: building out-of-date packages:
github.com/aws/aws-sdk-go/aws
github.com/aws/aws-sdk-go/aws/request
github.com/aws/aws-sdk-go/aws/client
github.com/aws/aws-sdk-go/aws/corehandlers
github.com/aws/aws-sdk-go/aws/ec2metadata
github.com/aws/aws-sdk-go/aws/credentials/ec2rolecreds
github.com/aws/aws-sdk-go/aws/defaults
github.com/aws/aws-sdk-go/aws/session
github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil
github.com/aws/aws-sdk-go/private/protocol/query
github.com/aws/aws-sdk-go/private/protocol/rest
github.com/aws/aws-sdk-go/private/protocol/restxml
github.com/aws/aws-sdk-go/private/signer/v4
github.com/aws/aws-sdk-go/service/s3
github.com/jacobsa/fuse/internal/buffer
github.com/jacobsa/fuse
github.com/jacobsa/fuse/fuseutil
installing these packages with 'go test -i ./...' will speed future tests.

? _/home/john/goofys/test2/goofys-0.0.2 [no test files]

github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil

../../../gocode/src/github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil/xml_to_struct.go:79: e.EncodeToken undefined (type *xml.Encoder has no field or method EncodeToken)
../../../gocode/src/github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil/xml_to_struct.go:82: e.EncodeToken undefined (type *xml.Encoder has no field or method EncodeToken)
../../../gocode/src/github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil/xml_to_struct.go:103: e.EncodeToken undefined (type *xml.Encoder has no field or method EncodeToken)

github.com/aws/aws-sdk-go/aws

../../../gocode/src/github.com/aws/aws-sdk-go/aws/types.go:87: syntax error: unexpected :, expecting ]

github.com/jacobsa/fuse/internal/buffer

../../../gocode/src/github.com/jacobsa/fuse/internal/buffer/runtime.s:24 6a: No such file or directory: textflag.h
FAIL _/home/john/goofys/test2/goofys-0.0.2/internal [build failed]
make: *** [run-test] Error 2

Error building with latest aws-sdk-go

I'm unable to build goofyfs using the latest aws-sdk-go as there are breaking changes in v0.10.0

$ go get github.com/kahing/goofys
# github.com/kahing/goofys/internal
src/github.com/kahing/goofys/internal/goofys.go:99: cannot use awsConfig (type *aws.Config) as type client.ConfigProvider in argument to s3.New:
    *aws.Config does not implement client.ConfigProvider (missing ClientConfig method)
src/github.com/kahing/goofys/internal/goofys.go:125: cannot use awsConfig (type *aws.Config) as type client.ConfigProvider in argument to s3.New:
    *aws.Config does not implement client.ConfigProvider (missing ClientConfig method)

I had to checkout aws-sdk-go at v0.9.17 for it to build.

Does not unmount on SIGTERM

I'm not entirely sure what the expected behaviour should be here... When the goofys process is killed by SIGTERM no attempt is made to unmount, so a subsequent start will fail:

$ goofys -f ...
2015/11/12 01:52:34.132755 main.INFO File system has been successfully mounted.

    # kill -INT <pid> from another terminal.

2015/11/12 01:52:49.936056 main.INFO Received SIGINT, attempting to unmount...
2015/11/12 01:52:49.946200 main.INFO Successfully exiting.

$ goofys -f ...
2015/11/12 01:54:11.416722 main.INFO File system has been successfully mounted.

    # kill -TERM <pid> from another terminal.

$ goofys -f ...
2015/11/12 01:56:20.199572 main.FATAL Mounting file system: Mount: Statting mount point: stat /s3/encode-files: transport endpoint is not connected

At this point one must forcefully umount the mountpoint sudo umount /mnt.

(I discovered this writing a systemd configuration for my goofys mount. Setting KillSignal=SIGINT seems to make systemctl restart work ok.)

High CPU utilization on idle mount.

I'm seeing 2 cores maxed out by the goofys daemon.

Mount opts are:

123.bucketname.xxx.com on /content-goofys type fuse (rw,nosuid,nodev,default_permissions,allow_other)

top shows 2 threads maxed out:

top - 17:13:33 up 1 day, 18:38,  8 users,  load average: 4.08, 4.33, 5.07
Tasks: 190 total,   1 running, 188 sleeping,   0 stopped,   1 zombie
%Cpu(s): 50.5 us,  0.2 sy,  0.0 ni, 49.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:  16433132 total, 13873464 used,  2559668 free,   245852 buffers
KiB Swap:        0 total,        0 used,        0 free.  8638932 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
10801 root      20   0 7016036 2.715g   4276 S 200.1 17.3   1556:48 goofys

version is:

goofys version 0.0.4

terminating & restarting the mount resolved the CPU issue. Not sure what caused it in the 1st place.

more tests

we need more fuse level tests and use the new travis support for fuse to get that under CI

support creating files bigger than 50GB

Right now each MPU part is fixed at 5MB, and since S3 has a 10000 maximum part limit we cannot create files bigger than 50GB. We can automatically adjust part size (ie: first 100 parts 5MB, then 50MB, etc) to support bigger files

--uid/gid not working

Hi,

I have a node.js app and thought to share a mount between nodes in my cluster (EC2 instance) using an S3 mount (for users upload, which will be served through a static s3 site).

I am trying to use goofys for mounting the S3 bucket.

I followed the installation instruction and I am able to mount my bucket and see the files as the root user (centos). I am not able to access it as the web user (which supposed to run the node.js application):

[centos@ip-172-31-16-222` ~]$ id web
uid=995(web) gid=992(web) groups=992(web)
[centos@ip-172-31-16-222 ~]$ $GOPATH/bin/goofys --uid "995" --gid "992" -o allow_other my-app-static /mymount
[centos@ip-172-31-16-222 ~]$ cd /mymount/share/
files/      index.html  store/      uploads/    users/
[centos@ip-172-31-16-222 ~]$ sudo su web
[web@ip-172-31-16-222 centos]$ cd /mymount/share
bash: cd: /mymount/share: Permission denied
[web@ip-172-31-16-222 centos]$

Am I doing something wrong?

PS - the installed version I have is goofys version 0.0.4... shouldn't I get the latest version following the install directions in the main readme page?

when mounted by fstab files created are invisible

$touch /mnt/goofysmount/test
$rm /mnt/goofysmount/test
$touch /mnt/goofysmount/test
$ls -la

nothing created the second time

Append question

Is append to file implemented and if so how(hopefully not via overwrite the whole file)?
I would like to use goofys to store append only file for redis

Question: What is "filey system"?

Hello @kahing , thank you for your great work.
README tells me that Goofys is a "filey" system instead of file system, which I have never heard before. And I cannot google out any valuable information about it. Could you please give me some references about it? Thanks a lot.

Add _netdev to fstab options in example

Without this, ec2 nodes hang on boot and never come up.

ie:

goofys#bucket   /mnt/mountpoint        fuse    _netdev,allow_other,--file-mode=0666    0       0

rather than:

goofys#bucket   /mnt/mountpoint        fuse    allow_other,--file-mode=0666    0       0

Are there any other options that should be used for AWS ec2 nodes?

Question: Is HTTP or HTTPS used to communicate with S3?

Some S3 libraries use HTTP S3 by default. Does Goofys ensure that HTTPS is used?

Travis integration

To prevent regressions and ensure a consistent build configuration.

fuse-utils dependency gotcha

I know it might be super obvious, but just a fore-warning that fuse-utils needs to be installed before you can install goofys itself. Took me awhile, but I finally figured it out while I was messing around with it in a docker container here.

Might be worth adding to documentation/readme?

Small file optimization

Presently goofys uses multi-part upload for all writes, even those less than 5 GB. For a small enough files, e.g., 1 MB, it should instead use a single-part upload.

Option to increase ulimit -n?

Opening many files concurrently can cause goofys to run out of file descriptors. This gets reported in /var/log/kern.log:

Mar  1 23:12:58 ip-172-31-20-148 /usr/local/bin/goofys[1100]: s3.ERROR code=RequestError msg=send request failed, err=Get https://s3-us-west-2.amazonaws.com/bucket/filename: dial tcp: lookup s3-us-west-2.amazonaws.com on 172.31.0.2:53: dial udp 172.31.0.2:53: socket: too many open files

To avoid this I can increase the limit in /etc/security/limits.conf, e.g:

root soft nofile 10240

Ideally we'd only increase this for the goofys process rather than all root processes. I'm not sure how to run goofys as a different user from fstab. Perhaps goofys should grow an option to request an increase to its ulimit at startup, see: http://stackoverflow.com/questions/17817204/how-to-set-ulimit-n-from-a-golang-program

Encryption key support

As pointed out here s3 allow you to bring your own key

https://aws.amazon.com/blogs/aws/s3-encryption-with-your-keys/

IAM Instance Profile support?

Does goofys support instance profile-based access? I can't see any mention or reference to it.

rsync -a causes input/output error

ubuntu@ip-10-0-0-27:/mnt$ rsync -a /mnt/filestore.ebs/ /mnt/cpirepsarchive.s3/
rsync: failed to set times on "/mnt/cpirepsarchive.s3/.": Operation not permitted (1)
And that same error for every file.

Then the mount looks like this:
d????????? ? ? ? ? ? cpirepsarchive.s3/

Multi-delete optimization

When deleting many files, e.g., rm -rf of a directory, goofys could issue one multi-object delete instead of many single-object deletes. POSIX does not express this naturally but goofys could heuristically buffer unlinks for 1 second and then issue the backend delete. Since most S3 implementations provide eventual consistency for read-after-delete, this optimization just stretches the semantics a little further.

code request signature does not match

Hey, I've used aws-cli configure to setup my aws credentials but still get the following messages when I attempt to mount:

2015/11/04 23:02:53.910241 s3.ERROR code=The request signature we calculated does not match the signature you provided. Check your key and signing method. msg=403 request=

2015/11/04 23:02:53.910435 s3.ERROR code=The request signature we calculated does not match the signature you provided. Check your key and signing method. msg=403 request=

2015/11/04 23:02:53.910538 s3.INFO Unable to detect bucket region, staying at 'us-west-2'
2015/11/04 23:02:53.914630 main.INFO File system has been successfully mounted.

Can a goofys mount be added to fstab?

Filesystem cache corruption when interrupting cp

I seem to be able to reliably trigger corruption of the FS cache in the following scenario:

# Terminal 1 
$ sudo goofys -o allow_other --dir-mode 0555 --file-mode 0444 --stat-cache-ttl 0 --type-cache-ttl 0 <bucket> /mnt

# Terminal 2 - correct md5sum for 1.6G file.
$ md5sum /mnt/TEST.fastq.gz
458add84ee7ecfb38d399d8002c2f3aa  /mnt/TEST.fastq.gz
$ ls -l /mnt/TEST.fastq.gz
-r--r--r-- 1 root root 1627202877 Oct  1 06:23 /mnt/TEST.fastq.gz

# Terminal 1 - ^C to interrupt and remount:
$ sudo goofys -o allow_other --dir-mode 0555 --file-mode 0444 --stat-cache-ttl 0 --type-cache-ttl 0 <bucket> /mnt

# Terminal 2, interrupt after a few seconds
$ cp /mnt/TEST.fastq.gz TEST.fastq.gz
^C
$ rm TEST.fastq.gz 
rm: remove write-protected regular file ‘TEST.fastq.gz’? y

# Terminal 2, allow to completion
$ cp /mnt/TEST.fastq.gz TEST.fastq.gz
$ md5sum TEST.fastq.gz 
6ac211f707d9cc937ac10f119a459086  TEST.fastq.gz
$ md5sum /mnt/TEST.fastq.gz 
6ac211f707d9cc937ac10f119a459086  /mnt/TEST.fastq.gz
$ ls -l TEST.fastq.gz 
-r--r--r-- 1 ubuntu ubuntu 1627202877 Oct 22 22:52 TEST.fastq.gz

goofyfs rev: ca3fc3c
Ubuntu 14.04 (ami-1c1eff2f)
golang from ppa:ubuntu-lxc/lxd-stable

ls only list file names below 1000

hello, I use ls command, but I found it can just list file names below 1000.

auto-tune buffer sizes according to amount of memory available

Flush() doesn't block until its done, and other IO ops can execute in between

Hi Team,

looks like i have many errors and not able to figure out further.

[root@img01 bin]# grep "no such file or directory" /var/log/messages | wc -l
12844

Mar 15 10:34:33 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df703 Mar 15 10:34:33 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df704 Mar 15 10:34:33 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df704 Mar 15 10:34:33 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df705 Mar 15 10:34:33 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df705 Mar 15 10:34:33 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df706 Mar 15 10:34:33 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df706 Mar 15 10:34:33 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df707 Mar 15 10:34:33 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df707 Mar 15 10:34:33 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df708 Mar 15 10:34:33 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df708 Mar 15 10:34:33 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df709 Mar 15 10:34:33 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df709 Mar 15 10:34:33 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df70a Mar 15 10:34:34 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df70a Mar 15 10:34:34 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df70b Mar 15 10:34:34 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df70b Mar 15 10:34:34 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df70c Mar 15 10:34:34 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df70c Mar 15 10:34:34 img01 rsyslogd-2177: imuxsock begins to drop Mar 15 10:34:40 img01 rsyslogd-2177: imuxsock lost 322 messages Mar 15 10:34:40 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df750 Mar 15 10:34:40 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df750 Mar 15 10:34:40 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df751 Mar 15 10:34:40 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df751 Mar 15 10:34:40 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df752 Mar 15 10:34:41 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df752 Mar 15 10:34:41 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df753 Mar 15 10:34:41 img01 /bin/goofys[26093]: fuse.DEBUG Op 0x003df753 connection.go:476] -> Error: "invalid argument"
connection.go:395] <- GetInodeAttributes (inode 72157)
connection.go:474] -> OK
connection.go:395] <- OpenFile (inode 72157)
connection.go:474] -> OK
connection.go:395] <- ReadFile (inode 72157, handle 92893, offset 0, 4096 bytes)
connection.go:476] -> Error: "no such file or directory"
connection.go:395] <- FlushFile (inode 72157)
connection.go:474] -> OK
connection.go:395] <- ReadFile (inode 72157, handle 92893, offset 0, 4096 bytes)
connection.go:476] -> Error: "no such file or directory"
connection.go:395] <- ReleaseFileHandle
connection.go:474] -> OK
connection.go:395] <- LookUpInode (parent 936, name "6713_sku_1280x720Padded_9caf092d7a7dd2466a5c8c215d006561.jpg")
connection.go:476] -> Error: "no such file or directory"
connection.go:395] <- FlushFile (inode 72157)
connection.go:476] -> Error: "invalid argument"
connection.go:395] <- GetInodeAttributes (inode 72157)
connection.go:474] -> OK
messages from pid 26093 due to rate-limiting
from pid 26093 due to rate-limiting
connection.go:395] <- LookUpInode (parent 16, name "2")
connection.go:474] -> OK
connection.go:395] <- LookUpInode (parent 275, name "3")
connection.go:474] -> OK
connection.go:395] <- LookUpInode (parent 278, name "c")
connection.go:474] -> OK
connection.go:395] <- LookUpInode (parent 43060, name "6713_sku_ATOM_SIZE_MACRO_23c6d55b8d500f3bac05c71835198cea.jpg")
connection.go:476] -> Error: "no such file or directory"

[root@img01 bin]# ./goofys --version
goofys version 0.0.5

/etc/fstab:
goofys#size-img01 /img01 fuse _netdev,allow_other,--region='ap-southeast-1',--file-mode=0666,--debug_fuse,--debug_s3,--storage-class="REDUCED_REDUNDANCY" 0 0

Please help me to see if anything i need to make changes.

Thanks,
Vadiraj

rclone (for inspiration)

I just wanted to share another tool written in Go that interacts between the local filesystem and S3. I am only sharing because I came across it awhile ago and maybe it might get the innovation juices flowing...

Here's the link to the source code for that: https://github.com/ncw/rclone

Multi-part upload for rename

Currently goofys can only rename files less than 5 GB. Multi-part upload allows much bigger files.

Unexpected out of order reads streaming with block size 32768

Trying to dig into why gunzip was slow, I enabled the debug logging while reading a file:

sudo goofys -o allow_other --dir-mode 0555 --file-mode 0444 --stat-cache-ttl 0 --debug_fuse --debug_s3 <bucket> /mnt

When reading with a block size of 131072 (the same as cat) I see 311 responses logged with status 206 Partial Content.

$ time dd if=/mnt/TEST.fastq.gz of=/dev/null bs=131072
12414+1 records in
12414+1 records out
1627202877 bytes (1.6 GB) copied, 61.5945 s, 26.4 MB/s

real    1m1.812s
user    0m0.024s
sys 0m0.644s

When reading with a block size of 32768 (the same as gunzip --stdout) I see 621 responses logged with status 206 Partial Content.

$ time dd if=/mnt/TEST.fastq.gz of=/dev/null bs=32768
49658+1 records in
49658+1 records out
1627202877 bytes (1.6 GB) copied, 75.1237 s, 21.7 MB/s

real    1m15.252s
user    0m0.008s
sys 0m0.804s

The difference in requests seems to be accounted for by 310 log lines reporting:

out of order read: 6 [TEST.fastq.gz] [5242880 5308416]

goofys gets stuck when free memory used by filesystem cache

Running md5sum across a number of large files I noticed that goofys seems to hang and my md5sum process gets stuck waiting on a read.

I'm able to reproduce this consistently with 0.0.4 release and a build from current master on a c4.large (3.75GB RAM) ec2 instance running Ubuntu 15.10 (ami-4b37d42b - ubuntu/images/hvm-ssd/ubuntu-wily-15.10-amd64-server-20160217.1). I've made a publicly readable bucket in the Oregon region so you can reproduce easily by running goofys like so:

$ sudo goofys -o allow_other --file-mode=0444 --dir-mode=0555 --stat-cache-ttl=0  -f --debug_fuse --debug_s3 lrowe-goofys-test /mnt

From another terminal I launch my md5suming process:

$ strace md5sum /mnt/*

In top, I see goofys running at ~90% cpu until the free memory is exhausted and cached mem reaching almost the amount of the machine. At this point the md5sum process has worked through 16 files totalling 3GB and is part way through the 17th.

top - 02:40:20 up 59 min,  4 users,  load average: 0.02, 0.24, 0.22
Tasks:  93 total,   1 running,  92 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   3853432 total,  3808888 used,    44544 free,    33356 buffers
KiB Swap:        0 total,        0 used,        0 free.  3260580 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                                    
 7867 root      20   0  591088 372696   9840 S   0.0  9.7   0:50.90 goofys

At that point /proc/meminfo show:

$ cat /proc/meminfo 
MemTotal:        3853432 kB
MemFree:          339792 kB
MemAvailable:    3636940 kB
Buffers:           33444 kB
Cached:          3235316 kB
SwapCached:            0 kB
Active:           498520 kB
Inactive:        2901268 kB
Active(anon):     131472 kB
Inactive(anon):     5020 kB
Active(file):     367048 kB
Inactive(file):  2896248 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:        131024 kB
Mapped:            29476 kB
Shmem:              5468 kB
Slab:              80340 kB
SReclaimable:      63012 kB
SUnreclaim:        17328 kB
KernelStack:        1968 kB
PageTables:         4280 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     1926716 kB
Committed_AS:     585624 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       15312 kB
VmallocChunk:   34359720776 kB
HardwareCorrupted:     0 kB
AnonHugePages:     86016 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       53248 kB
DirectMap2M:     3878912 kB

And goofys logs show:

2016/02/20 02:36:39.743499 fuse.DEBUG Op 0x00005f66        connection.go:395] <- ReadFile (inode 18, handle 18, offset 10027008, 131072 bytes)
2016/02/20 02:36:39.743906 fuse.DEBUG ReadFile 18 test17 [10027008 131072]
2016/02/20 02:36:39.745070 fuse.DEBUG < readFromStream 18 test17 [131072]
2016/02/20 02:36:39.745098 fuse.DEBUG < ReadFile 18 test17 [131072]
2016/02/20 02:36:39.745156 fuse.DEBUG Op 0x00005f66        connection.go:474] -> OK
2016/02/20 02:36:39.747050 fuse.DEBUG Op 0x00005f67        connection.go:395] <- ReadFile (inode 18, handle 18, offset 10158080, 131072 bytes)
2016/02/20 02:36:39.747324 fuse.DEBUG ReadFile 18 test17 [10158080 131072]
2016/02/20 02:36:39.748424 fuse.DEBUG < readFromStream 18 test17 [131072]
2016/02/20 02:36:39.748450 fuse.DEBUG < ReadFile 18 test17 [131072]
2016/02/20 02:36:39.748483 fuse.DEBUG Op 0x00005f67        connection.go:474] -> OK
2016/02/20 02:36:39.748607 fuse.DEBUG Op 0x00005f68        connection.go:395] <- ReadFile (inode 18, handle 18, offset 10289152, 131072 bytes)
2016/02/20 02:36:39.748664 fuse.DEBUG ReadFile 18 test17 [10289152 131072]
2016/02/20 02:36:39.749794 fuse.DEBUG < readFromStream 18 test17 [131072]
2016/02/20 02:36:39.749818 fuse.DEBUG < ReadFile 18 test17 [131072]
2016/02/20 02:36:39.749850 fuse.DEBUG Op 0x00005f68        connection.go:474] -> OK
2016/02/20 02:36:39.751648 fuse.DEBUG Op 0x00005f69        connection.go:395] <- ReadFile (inode 18, handle 18, offset 10420224, 131072 bytes)
2016/02/20 02:36:39.752018 fuse.DEBUG ReadFile 18 test17 [10420224 131072]
2016/02/20 02:36:39.753127 fuse.DEBUG < readFromStream 18 test17 [131072]
2016/02/20 02:36:39.753151 fuse.DEBUG < ReadFile 18 test17 [131072]
2016/02/20 02:36:39.753183 fuse.DEBUG Op 0x00005f69        connection.go:474] -> OK
2016/02/20 02:36:39.753314 fuse.DEBUG Op 0x00005f6a        connection.go:395] <- ReadFile (inode 18, handle 18, offset 10551296, 131072 bytes)
2016/02/20 02:36:39.753399 fuse.DEBUG ReadFile 18 test17 [10551296 131072]
2016/02/20 02:36:39.753421 fuse.DEBUG cutover to the parallel algorithm 18 test17 []
2016/02/20 02:36:39.753441 fuse.DEBUG readahead 18 test17 [10551296 20971520 0]
2016/02/20 02:36:39.756124 fuse.DEBUG readahead 18 test17 [31522816 20971520 20971520]
2016/02/20 02:36:39.758779 s3.DEBUG DEBUG: Request s3/GetObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET http://s3-us-west-2.amazonaws.com/lrowe-goofys-test/test17 HTTP/1.1
Host: s3-us-west-2.amazonaws.com
User-Agent: aws-sdk-go/1.1.4 (go1.5.1; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=ASIAJHVZIVDQJVRGQYBA/20160220/us-west-2/s3/aws4_request, SignedHeaders=host;range;x-amz-date;x-amz-security-token, Signature=680a998a042153a35336c4ddd86f74816b4fea741c0c521f3896ff76fbea5073
Range: bytes=10551296-31522815
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20160220T023639Z
X-Amz-Security-Token: AQoDYXdzEOP//////////wEa4ANpBx7GQTRYD+hoxbd+iXXEDF5dQa7+liIg9BVvYd/EgiUYLV+NmahOiSx7pinUKp8wKihm8QRTfIibmW7Eoaryr+fdAIFopWDDfm3oIk6bTyqs+69LatTKGjWjv/iSuVhkDI/gvZrUPITEAZ/MSVHflFG8ghyp0HJuVlJ0ohSis+m9OUt3Mq/Ac73E17yYIzRxZCClswaRyo7WKcETNCO9XToo+zMwR2fNCSIxtmIwTDPCXSHn4ygy/w+dz6J65Ji6yuTS9WBBjFFDBGupd7Qn0frwqFXdx/hHfePQ1MAS2eOslmNsHdy45Bg0vtLWXCqBNIgJ8PUoYhaW12h8FXQuUsPtMFK9lWtaM8ngjb9vC7HIw5gYlnMqg5wWmYlCZpMtJ17WElQHLEOnGvKz7JRJ5dcOkKgO4DNkm1IOQaEGmRC0pgdLZys/gAX4noSptGCJKDwbnGg3uTMjFTI6aWW9S9d9PkFiIU6Grtb6Dadq0aQ5sAy2icmLWPNuaQOIlfdVjDkSqUT8QZkSBbU4PsVHJzqz16bHF2sGmD7CGeXfRO3BBQZc/UR8bAnoeJFcjqBbJBkyxsRTTuKPQuhACcOPy/tpsFCLGHWqeHHhDJbLxUs2iYAeNgM8EG+5AiktV14g25SftgU=


-----------------------------------------------------
2016/02/20 02:36:39.762025 fuse.DEBUG readahead 18 test17 [52494336 20971520 41943040]
2016/02/20 02:36:39.764627 s3.DEBUG DEBUG: Request s3/GetObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET http://s3-us-west-2.amazonaws.com/lrowe-goofys-test/test17 HTTP/1.1
Host: s3-us-west-2.amazonaws.com
User-Agent: aws-sdk-go/1.1.4 (go1.5.1; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=ASIAJHVZIVDQJVRGQYBA/20160220/us-west-2/s3/aws4_request, SignedHeaders=host;range;x-amz-date;x-amz-security-token, Signature=91a0a6e1e8d176cd07f59619830fd1a80e8b022434282035902893ec873997e2
Range: bytes=31522816-52494335
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20160220T023639Z
X-Amz-Security-Token: AQoDYXdzEOP//////////wEa4ANpBx7GQTRYD+hoxbd+iXXEDF5dQa7+liIg9BVvYd/EgiUYLV+NmahOiSx7pinUKp8wKihm8QRTfIibmW7Eoaryr+fdAIFopWDDfm3oIk6bTyqs+69LatTKGjWjv/iSuVhkDI/gvZrUPITEAZ/MSVHflFG8ghyp0HJuVlJ0ohSis+m9OUt3Mq/Ac73E17yYIzRxZCClswaRyo7WKcETNCO9XToo+zMwR2fNCSIxtmIwTDPCXSHn4ygy/w+dz6J65Ji6yuTS9WBBjFFDBGupd7Qn0frwqFXdx/hHfePQ1MAS2eOslmNsHdy45Bg0vtLWXCqBNIgJ8PUoYhaW12h8FXQuUsPtMFK9lWtaM8ngjb9vC7HIw5gYlnMqg5wWmYlCZpMtJ17WElQHLEOnGvKz7JRJ5dcOkKgO4DNkm1IOQaEGmRC0pgdLZys/gAX4noSptGCJKDwbnGg3uTMjFTI6aWW9S9d9PkFiIU6Grtb6Dadq0aQ5sAy2icmLWPNuaQOIlfdVjDkSqUT8QZkSBbU4PsVHJzqz16bHF2sGmD7CGeXfRO3BBQZc/UR8bAnoeJFcjqBbJBkyxsRTTuKPQuhACcOPy/tpsFCLGHWqeHHhDJbLxUs2iYAeNgM8EG+5AiktV14g25SftgU=


-----------------------------------------------------
2016/02/20 02:36:39.765950 fuse.DEBUG readahead 18 test17 [73465856 20971520 62914560]
2016/02/20 02:36:39.770443 s3.DEBUG DEBUG: Request s3/GetObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET http://s3-us-west-2.amazonaws.com/lrowe-goofys-test/test17 HTTP/1.1
Host: s3-us-west-2.amazonaws.com
User-Agent: aws-sdk-go/1.1.4 (go1.5.1; linux; amd64)
Authorization: AWS4-HMAC-SHA256 Credential=ASIAJHVZIVDQJVRGQYBA/20160220/us-west-2/s3/aws4_request, SignedHeaders=host;range;x-amz-date;x-amz-security-token, Signature=bf7c178f19630b76fd07f1a30699a976a49a0be138ca9084713ac93998b8485e
Range: bytes=52494336-73465855
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20160220T023639Z
X-Amz-Security-Token: AQoDYXdzEOP//////////wEa4ANpBx7GQTRYD+hoxbd+iXXEDF5dQa7+liIg9BVvYd/EgiUYLV+NmahOiSx7pinUKp8wKihm8QRTfIibmW7Eoaryr+fdAIFopWDDfm3oIk6bTyqs+69LatTKGjWjv/iSuVhkDI/gvZrUPITEAZ/MSVHflFG8ghyp0HJuVlJ0ohSis+m9OUt3Mq/Ac73E17yYIzRxZCClswaRyo7WKcETNCO9XToo+zMwR2fNCSIxtmIwTDPCXSHn4ygy/w+dz6J65Ji6yuTS9WBBjFFDBGupd7Qn0frwqFXdx/hHfePQ1MAS2eOslmNsHdy45Bg0vtLWXCqBNIgJ8PUoYhaW12h8FXQuUsPtMFK9lWtaM8ngjb9vC7HIw5gYlnMqg5wWmYlCZpMtJ17WElQHLEOnGvKz7JRJ5dcOkKgO4DNkm1IOQaEGmRC0pgdLZys/gAX4noSptGCJKDwbnGg3uTMjFTI6aWW9S9d9PkFiIU6Grtb6Dadq0aQ5sAy2icmLWPNuaQOIlfdVjDkSqUT8QZkSBbU4PsVHJzqz16bHF2sGmD7CGeXfRO3BBQZc/UR8bAnoeJFcjqBbJBkyxsRTTuKPQuhACcOPy/tpsFCLGHWqeHHhDJbLxUs2iYAeNgM8EG+5AiktV14g25SftgU=


-----------------------------------------------------
2016/02/20 02:36:39.790891 s3.DEBUG DEBUG: Response s3/GetObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 206 Partial Content
Content-Length: 20971520
Accept-Ranges: bytes
Content-Range: bytes 10551296-31522815/108381107
Content-Type: binary/octet-stream
Date: Sat, 20 Feb 2016 02:36:40 GMT
Etag: "5d01d76c1480703834a17b377369cbe0-13"
Last-Modified: Sat, 20 Feb 2016 00:49:44 GMT
Server: AmazonS3
X-Amz-Id-2: X+RCw3FcBN7K/mwW+ZnOTDnUrIGytvwpHp1G8Wp+1VEsiIoRzUmUV001dpKljirX7kzz/9HXtUU=
X-Amz-Request-Id: B304483B97939AA4


-----------------------------------------------------
2016/02/20 02:36:39.872454 s3.DEBUG DEBUG: Response s3/GetObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 206 Partial Content
Content-Length: 20971520
Accept-Ranges: bytes
Content-Range: bytes 52494336-73465855/108381107
Content-Type: binary/octet-stream
Date: Sat, 20 Feb 2016 02:36:40 GMT
Etag: "5d01d76c1480703834a17b377369cbe0-13"
Last-Modified: Sat, 20 Feb 2016 00:49:44 GMT
Server: AmazonS3
X-Amz-Id-2: YDwn15h627h8n0t0ZrskkVRnFQCiyh8M6nNCC2+uJ3yEi8MihyZ4UeIXDvmCbpUeL1hCwEG4EhE=
X-Amz-Request-Id: 98E75541098D2050


-----------------------------------------------------
2016/02/20 02:36:39.887170 s3.DEBUG DEBUG: Response s3/GetObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 206 Partial Content
Content-Length: 20971520
Accept-Ranges: bytes
Content-Range: bytes 31522816-52494335/108381107
Content-Type: binary/octet-stream
Date: Sat, 20 Feb 2016 02:36:40 GMT
Etag: "5d01d76c1480703834a17b377369cbe0-13"
Last-Modified: Sat, 20 Feb 2016 00:49:44 GMT
Server: AmazonS3
X-Amz-Id-2: Ksw24t58PQqfzYqH6G6pNSWzbQIyR65IX46LETXDCacit1Cn1g2bXbbOcE+DwoWMsbmBYqs/B+c=
X-Amz-Request-Id: 751AEBA8DAA5427E


-----------------------------------------------------

While the strace md5sum shows:

read(3, "\366\236\251s\33\346(\0\31M\30\235\263\310\r\20\27\3018\261\16\272\351\344\"$@\344Y\274\366\3"..., 32768) = 32768
read(3,

If I attache with the delve debugger I see these threads and goroutines:

$ GOPATH=~/goofys sudo bin/dlv attach 7867
Type 'help' for list of commands.
(dlv) threads
* Thread 7867 at 0x45fcc3 /usr/lib/go/src/runtime/sys_linux_amd64.s:289 runtime.futex
  Thread 7868 at 0x45fcc3 /usr/lib/go/src/runtime/sys_linux_amd64.s:289 runtime.futex
  Thread 7869 at 0x45fcc3 /usr/lib/go/src/runtime/sys_linux_amd64.s:289 runtime.futex
  Thread 7870 at 0x45fcc3 /usr/lib/go/src/runtime/sys_linux_amd64.s:289 runtime.futex
  Thread 7871 at 0x45fcc3 /usr/lib/go/src/runtime/sys_linux_amd64.s:289 runtime.futex
  Thread 7872 at 0x45fcc3 /usr/lib/go/src/runtime/sys_linux_amd64.s:289 runtime.futex
  Thread 7874 at 0x4b31c4 /usr/lib/go/src/syscall/asm_linux_amd64.s:27 syscall.Syscall
  Thread 7876 at 0x45fcc3 /usr/lib/go/src/runtime/sys_linux_amd64.s:289 runtime.futex
  Thread 7877 at 0x45fe49 /usr/lib/go/src/runtime/sys_linux_amd64.s:420 runtime.epollwait
  Thread 7883 at 0x45fcc3 /usr/lib/go/src/runtime/sys_linux_amd64.s:289 runtime.futex
(dlv) goroutines
[16 goroutines]
  Goroutine 1 - User: ./src/github.com/jacobsa/fuse/mounted_file_system.go:42 github.com/jacobsa/fuse.(*MountedFileSystem).Join (0x4dc7e6)
  Goroutine 2 - User: /usr/lib/go/src/runtime/proc.go:186 runtime.gopark (0x42ed83)
  Goroutine 3 - User: /usr/lib/go/src/runtime/proc.go:186 runtime.gopark (0x42ed83)
  Goroutine 4 - User: /usr/lib/go/src/runtime/proc.go:186 runtime.gopark (0x42ed83)
  Goroutine 5 - User: /usr/lib/go/src/os/signal/signal_unix.go:22 os/signal.loop (0x49dbd3)
  Goroutine 10 - User: /usr/lib/go/src/runtime/proc.go:186 runtime.gopark (0x42ed83)
  Goroutine 17 - User: /usr/lib/go/src/runtime/asm_amd64.s:1697 runtime.goexit (0x45e9e1)
  Goroutine 29 - User: /usr/lib/go/src/runtime/sema.go:237 sync.runtime_Syncsemacquire (0x43ed99)
  Goroutine 30 - User: /usr/lib/go/src/runtime/sema.go:237 sync.runtime_Syncsemacquire (0x43ed99)
  Goroutine 33 - User: ./src/github.com/kahing/goofys/main.go:54 main.registerSIGINTHandler.func1 (0x40235b)
  Goroutine 41 - User: /usr/lib/go/src/syscall/asm_linux_amd64.s:27 syscall.Syscall (0x4b31c4)
  Goroutine 42 - User: /usr/lib/go/src/runtime/proc.go:186 runtime.gopark (0x42ed83)
  Goroutine 51 - User: /usr/lib/go/src/runtime/proc.go:186 runtime.gopark (0x42ed83)
  Goroutine 52 - User: /usr/lib/go/src/runtime/proc.go:186 runtime.gopark (0x42ed83)
  Goroutine 53 - User: /usr/lib/go/src/runtime/proc.go:186 runtime.gopark (0x42ed83)
  Goroutine 25954 - User: /usr/lib/go/src/runtime/sema.go:237 sync.runtime_Syncsemacquire (0x43ed99)

investigate doing multi-part prefetch for read

currently read doesn't saturate 1Gbps, possibly because we only ever have one GET request outstanding per-file handle.

cannot mount from /etc/fstab

I have tried the following but get the message: mount: can't find /mnt/mountpoint/ in /etc/fstab

goofys#bucketname /mnt/mountpoint fuse rw -o allow-other --dir-mode="0777",--file-mode="0777",--uid="33",--gid="33",--storage-class="STANDARD_IA" 0 0

Limit concurrent IOs

Writing a large file overloads small uplinks with too many concurrent IOs. Can goofys add an option to limit this number?

Support AWS2 auth type?

I would like to connect with goofys to an S3 compatible storage which uses AWS2 auth type, but it looks goofys supports AWS4 auth only.

AWS2 auth type sample:
Authorization: AWS user1:+4G7GkgyTtws8L0mFbPnxPyOh6Z=

AWS4 auth type sample:
Authorization: AWS4-HMAC-SHA256 Credential=user1/20160311/us-west-2/s3/aws4_request, SignedHeaders=host;x-amz-date, Signature=a69111c636a160e34e31a394040f3c0f33db57c53b8bb74ecdf4e349fd554e7x

Is there a way I can connect with goofy to an S3 compatible service which supports only AWS2 auth type?

Thanks!

Chris

Goofys consumes all memory with extended concurrent transfers

I'm using goofys to expose test data on s3 to an ec2 instance.

Out load test script runs continuous jobs with the s3 transfer rate being the limiting factor.
There are 5 concurrent transfers gong at a time.
Data file size is often 100-400MB.

One test run completed successfully, then a subsequent one failed as the s3 data transfers quit working.

From syslog: (trimmed for previty)

Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.289878] slurmctld invoked oom-killer: gfp_mask=0x3000d0, order=1, oom_score_adj=0
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290013] [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290064] [11362] 0 11362 3653986 3494764 6877 0 0 goofys

Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.290106] Out of memory: Kill process 11362 (goofys) score 826 or sacrifice child
Mar 7 09:52:12 ip-10-33-48-7 kernel: [229416.295024] Killed process 11362 (goofys) total-vm:14615944kB, anon-rss:13979056kB, file-rss:0kB

Is there any log/debug flag I can set to gather data from goofys?

Full log here:
https://gist.github.com/6b9ee07c1b8c0b5ebeca

The current status shows that it's still mounted, but it's not working:

root@ip-10-33-48-7:/var/log# ls /content
ls: cannot access /content: Transport endpoint is not connected
root@ip-10-33-48-7:/var/log# mount | grep content
s3fs on /content-s3fs type fuse.s3fs (rw,nosuid,nodev,allow_other)
v4test.demo.nextissue.com on /content-goofys type fuse (rw,nosuid,nodev,default_permissions,allow_other)

Remounting the bucket resolved the hangup issue.
From logs after umount/mount:

Mar 7 22:50:39 ip-10-33-48-7 /usr/local/bin/goofys[1467]: s3.INFO Switching from region 'us-west-2' to 'us-east-1'
Mar 7 22:50:39 ip-10-33-48-7 /usr/local/bin/goofys[1467]: main.INFO File system has been successfully mounted.

fstab ignores allow_other

Here is my fstab entry:

`goofys#cpirepsarchive   /mnt/cpirepsarchive.s3 fuse    allow_other,--file-mode=0777,--dir-mode=0777,--uid="33,--gid="33",--storage-class="REDUCED_REDUNDANCY"    0       0`

Here is the strace:

`ubuntu@ip-10-0-0-27:~$ sudo strace -f mount /mnt/mountpoint >& xout
^C
ubuntu@ip-10-0-0-27:~$ grep exec xout
execve("/bin/mount", ["mount", "/mnt/mountpoint"], [/* 15 vars */]) = 0
[pid  6240] execve("/sbin/mount.fuse", ["/sbin/mount.fuse", "goofys#bucketname", "/mnt/mountpoint", "-o", "rw,allow_other,--file-mode=0777,"...], [/* 11 vars */]) = 0
[pid  6240] execve("/bin/sh", ["/bin/sh", "-c", "'goofys' 'bucketname' '/mnt/"...], [/* 12 vars */]) = 0
[pid  6241] execve("/usr/bin/goofys", ["goofys", "bucketname", "/mnt/mountpoint", "-o", "rw,allow_other,--file-mode=0777,"...], [/* 13 vars */]) = 0
[pid  6244] execve("/home/ubuntu/.gvm/pkgsets/go1.5.1/global/bin/goofys", ["/home/ubuntu/.gvm/pkgsets/go1.5."..., "bucketname", "/mnt/mountpoint", "-o", "rw,allow_other,--file-mode=0777,"...], [/* 14 vars */]) = 0
[pid  6250] execve("/bin/fusermount", ["fusermount", "-o", "default_permissions,fsname=cpire"..., "--", "/mnt/mountpoint"], [/* 16 vars */] <unfinished ...>
[pid  6250] <... execve resumed> )      = 0
[pid  6253] execve("/bin/mount", ["/bin/mount", "--no-canonicalize", "-i", "-f", "-t", "fuse", "-o", "rw,default_permissions,allow_oth"..., "bucketname", "/mnt/mountpoint"], [/* 0 vars */]) = 0`

Directory listings missing `.` and `..` directory entries

I'm running goofyfs 6c5cf92 on Ubuntu 14.04 on EC2.

When I list the fuse mount point or subdirectory I do not see the . or .. directory entries.

$ sudo ls -la /mnt
total 0
dr-xr-xr-x 2 root root 4096 Oct 21 21:54 subdirectory
...

I do see these when the bucket is mounted with riofs.

fallback to v2 signing for non AWS endpoints

It's would be an awesome feature if you support to customize endpoint or host (s3.mystorage.com), I need to connect to my ceph-s3 system

Unable to read read-only public bucket

example:

[root@ip ec2-user]# export GOPATH=$HOME/gopath
[root@ip ec2-user]# $GOPATH/bin/goofys -o ro --dir-mode 0555 --file-mode 444   lidar_fema /femadata/
[root@ip ec2-user]# ls /femadata/
ls: reading directory /femadata/: Input/output error

Mount from fstab runs in foreground

used goofys#bucketname /mnt/mountpoint fuse allow_other,--file-mode=0777,--dir-mode=0777,--uid="33,--gid="33",--storage-class="REDUCED_REDUNDANCY" in /etc/fstab.

sudo mount /mnt/mountpoint mounted the volume with the following messages left open in foreground:

2015/11/07 21:34:10.777609 s3.INFO Switching from region 'us-west-2' to 'us-east-1'
2015/11/07 21:34:10.816434 main.INFO File system has been successfully mounted.

CTRL-C unmounted the volume.

Out of memory: Kill process 3155 (goofys)

this error :Feb 11 12:50:11 localhost kernel: Out of memory: Kill process 3155 (goofys) score 202 or sacrifice child
is about buffer memory?
How I can fixit manually?
Thanks

fuse allow-other invalid argument

Tried mounting with "goofys -o allow-other --dir-mode "0777" --file-mode "0777" --uid "33" --gid "33" --storage-class "STANDARD_IA" bucket name /mnt/mountpoint &"

Got message:

"2015/11/05 17:39:10 mount helper error: fusermount: mount failed: Invalid argument
2015/11/05 17:39:10.689705 main.FATAL Mounting file system: Mount: mount: fusermount: exit status 1"

log to syslog if goofys is running in the background

related to #32

goofys does not free buffer pool during idle

On my system goofys always uses 200 MB of memory even though it is idle most of the time. Instead it should free this memory.

Use /etc/mime.types to set Content-Type accordingly

My server created bunch of css files automatically and they got written to S3 with Content-Type:binary/octet-stream, which is the default.

It would be nice if goofys could guess MIME type and set the header accordingly (same as s3fs does it).

Random bogus part numbers

I see errors of the form:

2015/10/08 21:29:23.665162 code=Part number must be an integer between 1 and 10000, inclusive msg=400 request=

randomly when writing small files to AWS S3.

Unable to access mount point as other user with `-o allow_other`

I'm running goofyfs 6c5cf92 on Ubuntu 14.04 on EC2, run with:

sudo goofys -o allow_other --dir-mode 0555 --file-mode 0444 <bucket> /mnt

I'm not able to access the directory as another user:

$ sudo ls -dl /mnt
dr-xr-xr-x 2 root root 4096 Oct 21 21:54 /mnt
$ ls -dl /mnt
ls: cannot access /mnt: Permission denied
$ ls -l /
...
d?????????   ? ?    ?        ?            ? mnt
...

Perhaps this is because the fuse option is not set:

$ mount
...
<bucket> on /mnt type fuse (rw,nosuid,nodev,default_permissions)

With riofs this works and I see:

$ mount
...
/dev/fuse on /mnt type fuse (rw,nosuid,nodev,default_permissions,allow_other)

Occur errors when move a huge file to goofys mount point

Hi Kahing,

I successfully write files to goofys mount point but if I write a huge file to this mount point, it's error:

my file is about 70GB or more

panic: invalid part number: 10001

goroutine 1659259 [running]:
github.com/kahing/goofys/internal.(*FileHandle).mpuPartNoSpawn(0xc8202d01a0, 0xc8200ae400, 0xc823ba2000, 0x500000, 0x500000, 0x2711, 0x0, 0x0)
        /opt/go/src/github.com/kahing/goofys/internal/handles.go:383 +0xa68
github.com/kahing/goofys/internal.(*FileHandle).mpuPart(0xc8202d01a0, 0xc8200ae400, 0xc823ba2000, 0x500000, 0x500000, 0x2711)
        /opt/go/src/github.com/kahing/goofys/internal/handles.go:424 +0xca
created by github.com/kahing/goofys/internal.(*FileHandle).WriteFile
        /opt/go/src/github.com/kahing/goofys/internal/handles.go:492 +0x919

goroutine 1 [select, 19 minutes]:
github.com/jacobsa/fuse.(*MountedFileSystem).Join(0xc820146630, 0x7fbbc5e40b08, 0xc8200103e0, 0x0, 0x0)
        /opt/go/src/github.com/jacobsa/fuse/mounted_file_system.go:42 +0x167
main.main.func1(0xc8200cc6c0)
        /opt/go/src/github.com/kahing/goofys/main.go:200 +0x8b5
github.com/codegangsta/cli.(*App).Run(0xc8200cc5a0, 0xc82000a090, 0x9, 0x9, 0x0, 0x0)
        /opt/go/src/github.com/codegangsta/cli/app.go:180 +0x1051
main.main()
        /opt/go/src/github.com/kahing/goofys/main.go:209 +0xf6

goroutine 17 [syscall, 19 minutes, locked to thread]:
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1696 +0x1

goroutine 5 [syscall, 19 minutes]:
os/signal.loop()
        /usr/lib/golang/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
        /usr/lib/golang/src/os/signal/signal_unix.go:28 +0x37

goroutine 21 [semacquire]:
sync.runtime_Syncsemacquire(0xc820170280)
        /usr/lib/golang/src/runtime/sema.go:237 +0x201
sync.(*Cond).Wait(0xc820170270)
        /usr/lib/golang/src/sync/cond.go:62 +0x9b
io.(*pipe).read(0xc820170240, 0xc820240558, 0xaa8, 0xaa8, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/io/pipe.go:52 +0x2d2
io.(*PipeReader).Read(0xc820148028, 0xc820240558, 0xaa8, 0xaa8, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/io/pipe.go:134 +0x50
bufio.(*Scanner).Scan(0xc826133f10, 0xc826133eb8)
        /usr/lib/golang/src/bufio/scan.go:180 +0x877
github.com/Sirupsen/logrus.(*Logger).writerScanner(0xc820014820, 0xc820148028)
        /opt/go/src/github.com/Sirupsen/logrus/writer.go:20 +0x129
created by github.com/Sirupsen/logrus.(*Logger).Writer
        /opt/go/src/github.com/Sirupsen/logrus/writer.go:12 +0x17b

goroutine 20 [semacquire, 19 minutes]:
sync.runtime_Syncsemacquire(0xc8201701c0)
        /usr/lib/golang/src/runtime/sema.go:237 +0x201
sync.(*Cond).Wait(0xc8201701b0)
        /usr/lib/golang/src/sync/cond.go:62 +0x9b
io.(*pipe).read(0xc820170180, 0xc82023e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/io/pipe.go:52 +0x2d2
io.(*PipeReader).Read(0xc820148018, 0xc82023e000, 0x1000, 0x1000, 0x43c654, 0x0, 0x0)
        /usr/lib/golang/src/io/pipe.go:134 +0x50
bufio.(*Scanner).Scan(0xc820036730, 0x1000)
        /usr/lib/golang/src/bufio/scan.go:180 +0x877
github.com/Sirupsen/logrus.(*Logger).writerScanner(0xc820172190, 0xc820148018)
        /opt/go/src/github.com/Sirupsen/logrus/writer.go:20 +0x129
created by github.com/Sirupsen/logrus.(*Logger).Writer
        /opt/go/src/github.com/Sirupsen/logrus/writer.go:12 +0x17b

goroutine 66 [syscall]:
syscall.Syscall(0x0, 0x8, 0xc8202a4018, 0x21000, 0xc81fdb9961, 0x0, 0x411766)
        /usr/lib/golang/src/syscall/asm_linux_amd64.s:18 +0x5
syscall.read(0x8, 0xc8202a4018, 0x21000, 0x21000, 0xc820153680, 0x0, 0x0)
        /usr/lib/golang/src/syscall/zsyscall_linux_amd64.go:783 +0x5f
syscall.Read(0x8, 0xc8202a4018, 0x21000, 0x21000, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/syscall/syscall_unix.go:160 +0x4d
os.(*File).read(0xc82024c008, 0xc8202a4018, 0x21000, 0x21000, 0xa6eb30, 0x0, 0x0)
        /usr/lib/golang/src/os/file_unix.go:211 +0x53
os.(*File).Read(0xc82024c008, 0xc8202a4018, 0x21000, 0x21000, 0x411766, 0x0, 0x0)
        /usr/lib/golang/src/os/file.go:95 +0x8a
github.com/jacobsa/fuse/internal/buffer.(*InMessage).Init(0xc8202a4000, 0x7fbbc5e40270, 0xc82024c008, 0x0, 0x0)
        /opt/go/src/github.com/jacobsa/fuse/internal/buffer/in_message.go:53 +0xa2
github.com/jacobsa/fuse.(*Connection).readMessage(0xc820252000, 0xc8202a4000, 0x0, 0x0)
        /opt/go/src/github.com/jacobsa/fuse/connection.go:317 +0x7a
github.com/jacobsa/fuse.(*Connection).ReadOp(0xc820252000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /opt/go/src/github.com/jacobsa/fuse/connection.go:379 +0x81
github.com/jacobsa/fuse/fuseutil.(*fileSystemServer).ServeOps(0xc820144420, 0xc820252000)
        /opt/go/src/github.com/jacobsa/fuse/fuseutil/file_system.go:98 +0x53
github.com/jacobsa/fuse.Mount.func1(0x7fbbc2e16390, 0xc820144420, 0xc820252000, 0xc820148038)
        /opt/go/src/github.com/jacobsa/fuse/mount.go:89 +0x35
created by github.com/jacobsa/fuse.Mount
        /opt/go/src/github.com/jacobsa/fuse/mount.go:92 +0x827

goroutine 67 [select, 19 minutes, locked to thread]:
runtime.gopark(0xa721d8, 0xc82003c728, 0x987558, 0x6, 0x42e718, 0x2)
        /usr/lib/golang/src/runtime/proc.go:185 +0x163
runtime.selectgoImpl(0xc82003c728, 0x0, 0x18)
        /usr/lib/golang/src/runtime/select.go:392 +0xa64
runtime.selectgo(0xc82003c728)
        /usr/lib/golang/src/runtime/select.go:212 +0x12
runtime.ensureSigM.func1()
        /usr/lib/golang/src/runtime/signal1_unix.go:227 +0x353
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1696 +0x1

goroutine 68 [chan receive, 19 minutes]:
main.registerSIGINTHandler.func1(0xc820192300, 0x7fffca03389b, 0x9)
        /opt/go/src/github.com/kahing/goofys/main.go:53 +0x87
created by main.registerSIGINTHandler
        /opt/go/src/github.com/kahing/goofys/main.go:64 +0x17f

goroutine 1659054 [IO wait]:
net.runtime_pollWait(0x7fbbc5e41c98, 0x72, 0xc8200101e0)
        /usr/lib/golang/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc8248ef410, 0x72, 0x0, 0x0)
        /usr/lib/golang/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8248ef410, 0x0, 0x0)
        /usr/lib/golang/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8248ef3b0, 0xc820128000, 0x1000, 0x1000, 0x0, 0x7fbbc5e3c050, 0xc8200101e0)
        /usr/lib/golang/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc82003e000, 0xc820128000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7fbbc2e16000, 0xc82003e000, 0xc82022a108, 0xc820128000, 0x1000, 0x1000, 0xc8248c9200, 0x0, 0x0)
        /usr/lib/golang/src/net/http/transport.go:1370 +0x67
net/http.(*noteEOFReader).Read(0xc8202f49a0, 0xc820128000, 0x1000, 0x1000, 0xc82609d93e, 0x0, 0x0)
        <autogenerated>:126 +0xd0
bufio.(*Reader).fill(0xc8248c90e0)
        /usr/lib/golang/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc8248c90e0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc82022a0b0)
        /usr/lib/golang/src/net/http/transport.go:876 +0xf7
created by net/http.(*Transport).dialConn
        /usr/lib/golang/src/net/http/transport.go:685 +0xc78

goroutine 1659185 [select]:
net/http.(*persistConn).roundTrip(0xc82022a0b0, 0xc82609d7e0, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/net/http/transport.go:1164 +0xb0d
net/http.(*Transport).RoundTrip(0xc82000a240, 0xc82016e1c0, 0xba, 0x0, 0x0)
        /usr/lib/golang/src/net/http/transport.go:235 +0x530
net/http.send(0xc82016e1c0, 0x7fbbc5e404d8, 0xc82000a240, 0xc8201acc40, 0x0, 0x0)
        /usr/lib/golang/src/net/http/client.go:220 +0x52c
net/http.(*Client).send(0xc24540, 0xc82016e1c0, 0x62, 0x0, 0x0)
        /usr/lib/golang/src/net/http/client.go:143 +0x15a
net/http.(*Client).doFollowingRedirects(0xc24540, 0xc82016e1c0, 0xa71dc8, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/net/http/client.go:380 +0xbc0
net/http.(*Client).Do(0xc24540, 0xc82016e1c0, 0x1e3, 0x0, 0x0)
        /usr/lib/golang/src/net/http/client.go:178 +0x188
github.com/aws/aws-sdk-go/aws/corehandlers.glob.func2(0xc821672580)
        /opt/go/src/github.com/aws/aws-sdk-go/aws/corehandlers/handlers.go:65 +0x55
github.com/aws/aws-sdk-go/aws/request.(*HandlerList).Run(0xc8216726b0, 0xc821672580)
        /opt/go/src/github.com/aws/aws-sdk-go/aws/request/handlers.go:115 +0x9f
github.com/aws/aws-sdk-go/aws/request.(*Request).Send(0xc821672580, 0x0, 0x0)
        /opt/go/src/github.com/aws/aws-sdk-go/aws/request/request.go:225 +0x57c
github.com/aws/aws-sdk-go/service/s3.(*S3).UploadPart(0xc8201d8028, 0xc82483b3e0, 0x1, 0x0, 0x0)
        /opt/go/src/github.com/aws/aws-sdk-go/service/s3/api.go:1586 +0x4f
github.com/kahing/goofys/internal.(*FileHandle).mpuPartNoSpawn(0xc8202d01a0, 0xc8200ae400, 0xc820900000, 0x500000, 0x500000, 0x270f, 0x0, 0x0)
        /opt/go/src/github.com/kahing/goofys/internal/handles.go:396 +0x4c3
github.com/kahing/goofys/internal.(*FileHandle).mpuPart(0xc8202d01a0, 0xc8200ae400, 0xc820900000, 0x500000, 0x500000, 0x270f)
        /opt/go/src/github.com/kahing/goofys/internal/handles.go:424 +0xca
created by github.com/kahing/goofys/internal.(*FileHandle).WriteFile
        /opt/go/src/github.com/kahing/goofys/internal/handles.go:492 +0x919

goroutine 1658917 [select]:
net/http.(*persistConn).roundTrip(0xc82022a000, 0xc8203c9ee0, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/net/http/transport.go:1164 +0xb0d
net/http.(*Transport).RoundTrip(0xc82000a240, 0xc82016e0e0, 0xba, 0x0, 0x0)
        /usr/lib/golang/src/net/http/transport.go:235 +0x530
net/http.send(0xc82016e0e0, 0x7fbbc5e404d8, 0xc82000a240, 0xc82023c5b0, 0x0, 0x0)
        /usr/lib/golang/src/net/http/client.go:220 +0x52c
net/http.(*Client).send(0xc24540, 0xc82016e0e0, 0x62, 0x0, 0x0)
        /usr/lib/golang/src/net/http/client.go:143 +0x15a
net/http.(*Client).doFollowingRedirects(0xc24540, 0xc82016e0e0, 0xa71dc8, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/net/http/client.go:380 +0xbc0
net/http.(*Client).Do(0xc24540, 0xc82016e0e0, 0x1e3, 0x0, 0x0)
        /usr/lib/golang/src/net/http/client.go:178 +0x188
github.com/aws/aws-sdk-go/aws/corehandlers.glob.func2(0xc821672000)
        /opt/go/src/github.com/aws/aws-sdk-go/aws/corehandlers/handlers.go:65 +0x55
github.com/aws/aws-sdk-go/aws/request.(*HandlerList).Run(0xc821672130, 0xc821672000)
        /opt/go/src/github.com/aws/aws-sdk-go/aws/request/handlers.go:115 +0x9f
github.com/aws/aws-sdk-go/aws/request.(*Request).Send(0xc821672000, 0x0, 0x0)
        /opt/go/src/github.com/aws/aws-sdk-go/aws/request/request.go:225 +0x57c
github.com/aws/aws-sdk-go/service/s3.(*S3).UploadPart(0xc8201d8028, 0xc820298240, 0x1, 0x0, 0x0)
        /opt/go/src/github.com/aws/aws-sdk-go/service/s3/api.go:1586 +0x4f
github.com/kahing/goofys/internal.(*FileHandle).mpuPartNoSpawn(0xc8202d01a0, 0xc8200ae400, 0xc8210d4000, 0x500000, 0x500000, 0x270d, 0x0, 0x0)
        /opt/go/src/github.com/kahing/goofys/internal/handles.go:396 +0x4c3
github.com/kahing/goofys/internal.(*FileHandle).mpuPart(0xc8202d01a0, 0xc8200ae400, 0xc8210d4000, 0x500000, 0x500000, 0x270d)
        /opt/go/src/github.com/kahing/goofys/internal/handles.go:424 +0xca
created by github.com/kahing/goofys/internal.(*FileHandle).WriteFile
        /opt/go/src/github.com/kahing/goofys/internal/handles.go:492 +0x919

goroutine 1659270 [select]:
net/http.(*persistConn).roundTrip(0xc82012c210, 0xc826148cb0, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/net/http/transport.go:1164 +0xb0d
net/http.(*Transport).RoundTrip(0xc82000a240, 0xc8200b02a0, 0xbb, 0x0, 0x0)
        /usr/lib/golang/src/net/http/transport.go:235 +0x530
net/http.send(0xc8200b02a0, 0x7fbbc5e404d8, 0xc82000a240, 0xc8248ee380, 0x0, 0x0)
        /usr/lib/golang/src/net/http/client.go:220 +0x52c
net/http.(*Client).send(0xc24540, 0xc8200b02a0, 0x63, 0x0, 0x0)
        /usr/lib/golang/src/net/http/client.go:143 +0x15a
net/http.(*Client).doFollowingRedirects(0xc24540, 0xc8200b02a0, 0xa71dc8, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/net/http/client.go:380 +0xbc0
net/http.(*Client).Do(0xc24540, 0xc8200b02a0, 0x1e4, 0x0, 0x0)
        /usr/lib/golang/src/net/http/client.go:178 +0x188
github.com/aws/aws-sdk-go/aws/corehandlers.glob.func2(0xc82486e000)
        /opt/go/src/github.com/aws/aws-sdk-go/aws/corehandlers/handlers.go:65 +0x55
github.com/aws/aws-sdk-go/aws/request.(*HandlerList).Run(0xc82486e130, 0xc82486e000)
        /opt/go/src/github.com/aws/aws-sdk-go/aws/request/handlers.go:115 +0x9f
github.com/aws/aws-sdk-go/aws/request.(*Request).Send(0xc82486e000, 0x0, 0x0)
        /opt/go/src/github.com/aws/aws-sdk-go/aws/request/request.go:225 +0x57c
github.com/aws/aws-sdk-go/service/s3.(*S3).UploadPart(0xc8201d8028, 0xc8216d80c0, 0x1, 0x0, 0x0)
        /opt/go/src/github.com/aws/aws-sdk-go/service/s3/api.go:1586 +0x4f
github.com/kahing/goofys/internal.(*FileHandle).mpuPartNoSpawn(0xc8202d01a0, 0xc8200ae400, 0xc8203de000, 0x500000, 0x500000, 0x2710, 0x0, 0x0)
        /opt/go/src/github.com/kahing/goofys/internal/handles.go:396 +0x4c3
github.com/kahing/goofys/internal.(*FileHandle).mpuPart(0xc8202d01a0, 0xc8200ae400, 0xc8203de000, 0x500000, 0x500000, 0x2710)
        /opt/go/src/github.com/kahing/goofys/internal/handles.go:424 +0xca
created by github.com/kahing/goofys/internal.(*FileHandle).WriteFile
        /opt/go/src/github.com/kahing/goofys/internal/handles.go:492 +0x919

goroutine 1657418 [IO wait]:
net.runtime_pollWait(0x7fbbc5e41ed8, 0x72, 0xc8200101e0)
        /usr/lib/golang/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc8203a9560, 0x72, 0x0, 0x0)
        /usr/lib/golang/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8203a9560, 0x0, 0x0)
        /usr/lib/golang/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8203a9500, 0xc823b68000, 0x1000, 0x1000, 0x0, 0x7fbbc5e3c050, 0xc8200101e0)
        /usr/lib/golang/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc82024c028, 0xc823b68000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7fbbc2e16000, 0xc82024c028, 0xc82022a058, 0xc823b68000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/net/http/transport.go:1370 +0x67
net/http.(*noteEOFReader).Read(0xc8202e30a0, 0xc823b68000, 0x1000, 0x1000, 0xc8200bc760, 0x0, 0x0)
        <autogenerated>:126 +0xd0
bufio.(*Reader).fill(0xc82608aae0)
        /usr/lib/golang/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc82608aae0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc82022a000)
        /usr/lib/golang/src/net/http/transport.go:876 +0xf7
created by net/http.(*Transport).dialConn
        /usr/lib/golang/src/net/http/transport.go:685 +0xc78

goroutine 1654922 [select]:
net/http.(*persistConn).writeLoop(0xc82012c210)
        /usr/lib/golang/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
        /usr/lib/golang/src/net/http/transport.go:686 +0xc9d

goroutine 1657419 [select]:
net/http.(*persistConn).writeLoop(0xc82022a000)
        /usr/lib/golang/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
        /usr/lib/golang/src/net/http/transport.go:686 +0xc9d

goroutine 1659055 [select]:
net/http.(*persistConn).writeLoop(0xc82022a0b0)
        /usr/lib/golang/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
        /usr/lib/golang/src/net/http/transport.go:686 +0xc9d

goroutine 1659557 [runnable]:
fmt.(*fmt).padString(0xc8200741f8, 0xc0377f, 0xd)
        /usr/lib/golang/src/fmt/format.go:130 +0x406
fmt.(*fmt).fmt_s(0xc8200741f8, 0xc0377f, 0xd)
        /usr/lib/golang/src/fmt/format.go:322 +0x61
fmt.(*pp).fmtString(0xc8200741a0, 0xc0377f, 0xd, 0xc800000076)
        /usr/lib/golang/src/fmt/print.go:518 +0x144
fmt.(*pp).printArg(0xc8200741a0, 0x7ffee0, 0xc821640450, 0x76, 0x0, 0x0)
        /usr/lib/golang/src/fmt/print.go:797 +0xd91
fmt.(*pp).doPrintf(0xc8200741a0, 0x982140, 0x5, 0xc8208f3d10, 0x2, 0x2)
        /usr/lib/golang/src/fmt/print.go:1219 +0x1dd8
fmt.Sprintf(0x982140, 0x5, 0xc8208f3d10, 0x2, 0x2, 0x0, 0x0)
        /usr/lib/golang/src/fmt/print.go:203 +0x6f
github.com/jacobsa/fuse.(*Connection).debugLog(0xc820252000, 0x186aa8, 0x1, 0x9823d0, 0x5, 0x0, 0x0, 0x0)
        /opt/go/src/github.com/jacobsa/fuse/connection.go:196 +0x20f
github.com/jacobsa/fuse.(*Connection).Reply(0xc820252000, 0x7fbbc0f93128, 0xc824849380, 0x0, 0x0)
        /opt/go/src/github.com/jacobsa/fuse/connection.go:474 +0x3d0
github.com/jacobsa/fuse/fuseutil.(*fileSystemServer).handleOp(0xc820144420, 0xc820252000, 0x7fbbc0f93128, 0xc824849380, 0x7dde40, 0xc824849200)
        /opt/go/src/github.com/jacobsa/fuse/fuseutil/file_system.go:191 +0x183
created by github.com/jacobsa/fuse/fuseutil.(*fileSystemServer).ServeOps
        /opt/go/src/github.com/jacobsa/fuse/fuseutil/file_system.go:108 +0x18f

goroutine 1654921 [IO wait]:
net.runtime_pollWait(0x7fbbc5e41f98, 0x72, 0xc8200101e0)
        /usr/lib/golang/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc8200de290, 0x72, 0x0, 0x0)
        /usr/lib/golang/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8200de290, 0x0, 0x0)
        /usr/lib/golang/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8200de230, 0xc82023f000, 0x1000, 0x1000, 0x0, 0x7fbbc5e3c050, 0xc8200101e0)
        /usr/lib/golang/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc82003e040, 0xc82023f000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7fbbc2e16000, 0xc82003e040, 0xc82012c268, 0xc82023f000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/net/http/transport.go:1370 +0x67
net/http.(*noteEOFReader).Read(0xc820244520, 0xc82023f000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        <autogenerated>:126 +0xd0
bufio.(*Reader).fill(0xc8200be720)
        /usr/lib/golang/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc8200be720, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc82012c210)
        /usr/lib/golang/src/net/http/transport.go:876 +0xf7
created by net/http.(*Transport).dialConn
        /usr/lib/golang/src/net/http/transport.go:685 +0xc78

Can't disable Caching

Hi,

I.ve experimented with --stat-cache-ttl=0,--type-cache-ttl=0 in fstab, but I can't disable caching.
Example:

perl -e 'print "found\n" if (-f "/mnt/07ooBP_g87NjF2oX6mGSzHnWodE/abcd.pdf")'
=> found
Unlink the file abcd.pdf via Web-UI or from another host mounting the same Bucket
perl -e 'print "found\n" if (-f "/mnt/07ooBP_g87NjF2oX6mGSzHnWodE/abcd.pdf")'
=> found
perl -e 'print "found\n" if (-f "/mnt/07ooBP_g87NjF2oX6mGSzHnWodE/abcd.pdf")'
from another host mounting the same bucket
=> not found

I need to disable caching because I want to check for existance of a file in a bucket. And need to have the files with a prefix like above to make them downloadable via S3.

My fstab-entry:
goofys#mybucket /mnt fuse _netdev,allow_other,--uid=300,--stat-cache-ttl=0,--type-cache-ttl=0 0 0

Any hints?

I can flush the inodes cache with 'echo 2 > /proc/sys/vm/drop_caches', what helps, but this is an performance killer...

crash caused by suspected race after last file in a directory is deleted

Running 'mount' shows that the OS still considers 'files' mounted. If I then unmount and then mount files it's once again working fine.

Strangely - I have the same issue with S3FS.

Do you have any ideas? The fstab entry is as follows:

<FULL PATH TO>/goofys#<BUCKET> <MOUNTPOINT> _netdev,noexec,allow_other,--uid=1001,--gid=1002,--dir-mode=0750,--file-mode=0640,--region=eu-west-1,--profile=<AWS CONFIG PROFILE> 0 0

Make codebase go fmt compliant

Most editors reformat Go code to the canonical style and Goofys uses its own style which makes it hard to contribute to. We should also flag the build on style errors via Travis. References #5.