utrechtuniversity / davrods Goto Github PK
View Code? Open in Web Editor NEWAn Apache WebDAV interface to iRODS
License: GNU Lesser General Public License v3.0
An Apache WebDAV interface to iRODS
License: GNU Lesser General Public License v3.0
According to RFC 4918, when a request is made of a collection with a URL missing a trailing forward slash ("/"), the response should contain a Content-Location
header indicating the URL with the trailing forward slash. Please see https://tools.ietf.org/html/rfc4918#page-15.
When a PROPFIND
request is made on a collection with the URL https://host.domain/collection/
, the paths to the members of /collection/
in the DAV:multistatus
response document all have an extra forward slash in the URL immediately after the /collection/
prefix. For example, if object
is a member of collection
, its URL path in the response will be /collection//object
. Here's a real example.
tedgin@dev ~? curl -u tedgin -H 'Depth: 1' -X PROPFIND https://data-dev.cyverse.rocks/dav/cyverse.dev/home/tedgin/IticketTest/
Enter host password for user 'tedgin':
<?xml version="1.0" encoding="utf-8"?>
<D:multistatus xmlns:D="DAV:">
<D:response xmlns:lp1="DAV:" xmlns:lp2="http://apache.org/dav/props/">
<D:href>/dav/cyverse.dev/home/tedgin/IticketTest/</D:href>
<D:propstat>
<D:prop>
<D:creationdate>Wed, 30 Oct 2019 18:58:08 GMT</D:creationdate>
<D:getetag>"01572461888"</D:getetag>
<D:getlastmodified>Wed, 30 Oct 2019 18:58:08 GMT</D:getlastmodified>
<lp1:resourcetype><D:collection/></lp1:resourcetype>
<D:supportedlock>
<D:lockentry>
<D:lockscope><D:exclusive/></D:lockscope>
<D:locktype><D:write/></D:locktype>
</D:lockentry>
<D:lockentry>
<D:lockscope><D:shared/></D:lockscope>
<D:locktype><D:write/></D:locktype>
</D:lockentry>
</D:supportedlock>
<D:lockdiscovery/>
</D:prop>
<D:status>HTTP/1.1 200 OK</D:status>
</D:propstat>
</D:response>
<D:response xmlns:lp1="DAV:" xmlns:lp2="http://apache.org/dav/props/">
<D:href>/dav/cyverse.dev/home/tedgin/IticketTest//LICENSE.txt</D:href>
<D:propstat>
<D:prop>
<D:creationdate>Wed, 30 Oct 2019 18:58:24 GMT</D:creationdate>
<D:getcontentlength>3491</D:getcontentlength>
<D:getetag>"da3-01572461904"</D:getetag>
<D:getlastmodified>Wed, 30 Oct 2019 18:58:24 GMT</D:getlastmodified>
<lp1:resourcetype/>
<D:supportedlock>
<D:lockentry>
<D:lockscope><D:exclusive/></D:lockscope>
<D:locktype><D:write/></D:locktype>
</D:lockentry>
<D:lockentry>
<D:lockscope><D:shared/></D:lockscope>
<D:locktype><D:write/></D:locktype>
</D:lockentry>
</D:supportedlock>
<D:lockdiscovery/>
</D:prop>
<D:status>HTTP/1.1 200 OK</D:status>
</D:propstat>
</D:response>
</D:multistatus>
Why does davrods require irods_server_control_plane_key
? It appears that NFSRODS provides a similar level of access but does not require access to this key.
In general when you are working with federated iRODS instances, your users would need read access to the root (/), zone (/ and other zones) and home (//home) collections to list all possible federated zones they might have access to. This does not work with Davrods. When changing the view to
DavRodsExposedRoot /
users will just see an empty folder as entry point in their webdav client.
Dear Davrods Developers,
is there a way for Davrods to interact with an SSO integration of iRODS, e.g. SAML/OpenID Connect?
I am not certain how that would work precisely but I could imagine that there are some challenges (and also I'm not certain how well iRODS plays here). I'm aware of this:
This looks like there are some roadblocks...
Have I answered my own question with "not yet"?
Is there any further insight that you could provide?
Best wishes,
Manuel
Hello!
Thanks for your development of the davrods application! We've been using it for quite a while now in the Maastricht University and are very happy with it.
Today we got a request from a user to upload large quantities of data through webdav. He cannot do it, because the mounted share in Linux shows that there is only 509G available, when the storage actually still has 14T available.
After a little digging, we found out that the value that Linux shows is just static, and doesn't change based on the value of the underlying resource and unixfilesystem
Is there a way for us to have Linux show the actual size of the unixfilesystem
resource, or will it always return the values below?
http://upload.local.dh.unimaas.nl/ 1333333332 800000000 533333332 61% /tmp/test
A Mac will show the following values and will just allow the user to upload as much as he wants:
http://upload.local.dh.unimaas.nl/ 0Bi 0Bi 0Bi 100% 0 0 100% /Volumes/upload.local.dh.unimaas.nl
Thanks in advance!
Hi,
Thanks for the brilliant WebDAV!
I might have discovered a minor bug, might be configuration too but you might want to check.
If I open http URL:s in a browser like http://ridgehead.pdc.kth.se/home/ilarik
without the trailing slash, the directory listing has invalid links like http://ridgehead.pdc.kth.se:8080/home/firstdir/
which should be http://ridgehead.pdc.kth.se:8080/home/ilarik/firstdir/
.
If I add the trailing slash to the URL like http://ridgehead.pdc.kth.se:8080/home/ilarik/
everything works peachy in the links in the generated index pages.
Hi, I'd be very interested in a Debian/Ubuntu package.
I could compile and make install
davrods on Ubuntu 16.04 using the following commands (using the Ubuntu 14.04 iRODS packages, though).
I could not make the package building work for some reason, though.
Would you be interested in also providing packages for .deb
-based systems?
# Register irods packages
if [[ ! -f /etc/apt/sources.list.d/renci-irods.list ]]; then
release=trusty # no good irods packages for xenial yet
wget -qO - https://packages.irods.org/irods-signing-key.asc | sudo apt-key add -
echo "deb [arch=amd64] https://packages.irods.org/apt/ $release main" | sudo tee /etc/apt/sources.list.d/renci-irods.list
sudo apt-get update
fi
# Install dependencies
apt install cmake make gcc apache2-dev irods-dev irods-dev irods-runtime
# Download davrods
mkdir -p ~/davrods
cd ~/davrods
test -f 4.2.1_1.1.1.tar.gz || wget https://github.com/UtrechtUniversity/davrods/archive/4.2.1_1.1.1.tar.gz
tar xzf 4.2.1_1.1.1.tar.gz
cd davrods-4.2.1_1.1.1/
patch -p0 <davrods.patch
mkdir build
cd build
cmake ..
make
make install
Below is the patch I used
--- CMakeLists.txt 2017-08-14 18:18:39.972615385 +0200
+++ CMakeLists.txt 2017-08-14 19:49:48.549653272 +0200
@@ -28,11 +28,22 @@
if(IS_DIRECTORY /etc/httpd/conf.modules.d)
# This looks like CentOS7's httpd, we know where to put our files on install.
- set(INSTALLABLE_ON_THIS_SYSTEM TRUE)
+ set(SYSTEM_LOOKS_LIKE "CentOS7")
+ set(SO_PATH "modules")
+elseif(IS_DIRECTORY /etc/apache2/mods-available)
+ # This looks like Debian/Ubuntu, we know where to put our files on install.
+ set(SYSTEM_LOOKS_LIKE "Debian")
+ set(SO_PATH "/usr/lib/apache2/modules/")
+
+# Debian notes, postinstall:
+#
+# a2enmod davrods
+# a2endmod dav
+# apache2ctl restart
else()
- set(INSTALLABLE_ON_THIS_SYSTEM FALSE)
+ set(SYSTEM_LOOKS_LIKE "--Unknown--")
message(WARNING "
-Davrods' build system currently only supports the cmake 'install' target on CentOS7-like systems.
+Davrods' build system currently only supports the cmake 'install' target on CentOS7 and Debian-like systems.
If you are running CentOS or similar, make sure httpd is installed before running cmake: This build system requires certain HTTPD directories to be in place.
If you are running a different Linux distribution or if your HTTPD configuration layout differs otherwise, you can install Davrods manually after building. See the instructions in README.md.")
endif()
@@ -76,14 +87,17 @@
add_library(mod_davrods SHARED ${SOURCES})
+configure_file(davrods.conf.in davrods.conf @ONLY)
+
# Remove "lib" prefix from module SO file.
set_property(TARGET mod_davrods PROPERTY PREFIX "")
-if(INSTALLABLE_ON_THIS_SYSTEM)
+# Enable OS-dependent installation targets
+if(SYSTEM_LOOKS_LIKE STREQUAL "CentOS7")
install(TARGETS mod_davrods
DESTINATION ${HTTPD_BUILDSYS_MODULE_DIR})
- install(FILES davrods.conf
+ install(FILES ${CMAKE_BINARY_DIR}/davrods.conf
DESTINATION /etc/httpd/conf.modules.d
RENAME 10-davrods.conf)
@@ -98,9 +112,28 @@
install(DIRECTORY
DESTINATION /var/lib/davrods)
+elseif(SYSTEM_LOOKS_LIKE STREQUAL "Debian")
+ install(TARGETS mod_davrods
+ DESTINATION ${HTTPD_BUILDSYS_MODULE_DIR})
+
+ install(FILES ${CMAKE_BINARY_DIR}/davrods.conf
+ DESTINATION /etc/apache2/mods-available
+ RENAME davrods.load)
+
+ install(FILES davrods-vhost.conf
+ DESTINATION /etc/apache2/conf-available/)
+
+ install(FILES irods_environment.json
+ DESTINATION /etc/apache2/irods/)
+
+ install(FILES README.md COPYING COPYING.LESSER
+ DESTINATION /usr/share/doc/davrods-${DAVRODS_VERSION}/)
+
+ install(DIRECTORY
+ DESTINATION /var/lib/davrods)
endif()
-if(INSTALLABLE_ON_THIS_SYSTEM)
+if(SYSTEM_LOOKS_LIKE EQUAL "CentOS7")
set(CPACK_MONOLITHIC_INSTALL 1)
set(CPACK_CMAKE_GENERATOR "Unix Makefiles")
set(CPACK_GENERATOR "RPM")
@@ -137,5 +170,43 @@
set(CPACK_PACKAGE_FILE_NAME "${CPACK_PACKAGE_NAME}-${CPACK_PACKAGE_VERSION}-${CPACK_RPM_PACKAGE_RELEASE}")
+ include(CPack)
+elseif(SYSTEM_LOOKS_LIKE EQUAL "Debian")
+ set(CPACK_MONOLITHIC_INSTALL 1)
+ set(CPACK_CMAKE_GENERATOR "Unix Makefiles")
+ set(CPACK_GENERATOR "DEB")
+ set(CPACK_PACKAGE_NAME "davrods")
+ set(CPACK_PACKAGE_VENDOR "Utrecht University <[email protected]>")
+ set(CPACK_PACKAGE_CONTACT "Utrecht University <[email protected]>")
+ set(CPACK_PACKAGE_VERSION "${DAVRODS_VERSION}")
+ #set(CPACK_PACKAGE_VERSION_MAJOR "4")
+ #set(CPACK_PACKAGE_VERSION_MINOR "2")
+ #set(CPACK_PACKAGE_VERSION_PATCH "1")
+ set(CPACK_PACKAGE_DESCRIPTION_FILE "${CMAKE_CURRENT_SOURCE_DIR}/package/description.txt")
+ set(CPACK_PACKAGE_DESCRIPTION_SUMMARY "A DAV level 2 compliant Apache interface to iRODS")
+
+ set(CPACK_RESOURCE_FILE_LICENSE "${CMAKE_CURRENT_SOURCE_DIR}/COPYING.LESSER")
+ set(CPACK_RESOURCE_FILE_README "${CMAKE_CURRENT_SOURCE_DIR}/README.md")
+
+ set(CPACK_RPM_PACKAGE_RELEASE "1")
+ set(CPACK_RPM_PACKAGE_LICENSE "LGPLv3+")
+ set(CPACK_RPM_PACKAGE_REQUIRES "apache2-server >= 2.4, irods-runtime = ${IRODS_VERSION}")
+ set(CPACK_RPM_PACKAGE_URL "https://github.com/UtrechtUniversity/davrods")
+ set(CPACK_RPM_CHANGELOG_FILE "${CMAKE_CURRENT_SOURCE_DIR}/package/changelog.txt")
+ set(CPACK_RPM_PACKAGE_AUTOREQ 0)
+ set(CPACK_RPM_PACKAGE_AUTOPROV 0)
+ set(CPACK_RPM_POST_INSTALL_SCRIPT_FILE "${CMAKE_CURRENT_SOURCE_DIR}/package/postinst.sh")
+
+# set(CPACK_RPM_USER_FILELIST
+# "%doc /usr/share/doc/davrods-${DAVRODS_VERSION}/README.md"
+# "%doc /usr/share/doc/davrods-${DAVRODS_VERSION}/COPYING"
+# "%doc /usr/share/doc/davrods-${DAVRODS_VERSION}/COPYING.LESSER"
+# "%config(noreplace) /etc/apache2/mods-available/10-davrods.conf"
+# "%config(noreplace) /etc/apache2/conf-available/davrods-vhost.conf"
+# "%config(noreplace) /etc/apache2/irods/irods_environment.json"
+# "%attr(700,apache,apache) /var/lib/davrods")
+
+ set(CPACK_PACKAGE_FILE_NAME "${CPACK_PACKAGE_NAME}-${CPACK_PACKAGE_VERSION}-${CPACK_RPM_PACKAGE_RELEASE}")
+
include(CPack)
endif()
Hi,
I recently noticed that when putting files via Davrods to a replicated resource in iRODS 4.1.10 or iRODS 4.1.11, the replication fails and the new data object lands only in the first resource server it hits.
The really strange thing is that the failure only occurs when connecting via the OS X built-in WebDAV. Linux webdav clients my colleagues tested (GNOME Nautilus or davfs I suppose) worked perfectly. Also CyberDuck works.
For example:
-bash-4.2$ ils -l
/tempZone/home/rods:
rods 0 replResc;rodsresc2Resource 5606 2017-12-12.16:23 & bash-history-orig.txt
rods 1 replResc;rodsresc1Resource 5606 2017-12-12.16:23 & bash-history-orig.txt
rods 0 replResc;rodsresc2Resource 4096 2017-12-12.16:18 & ._SC17 Final Program.pdf
rods 1 replResc;rodsresc1Resource 4096 2017-12-12.16:18 & ._SC17 Final Program.pdf
rods 0 replResc;rodsresc2Resource 5500794 2017-12-12.16:18 & SC17 Final Program.pdf
rods 0 replResc;rodsresc2Resource 2593877 2017-12-12.16:16 & updates.img
-bash-4.2$
Here the file updates.img
was uploaded via OS X native client, as well as SC17 Final Program.pdf
(which had extended attributes). The file bash-history.orig.txt
was uploaded from OS X via CyberDuck. To add insult to injury, iRODS replicated the xattr file but not the actual file, as you can see.
My personal guess is that the OS X WebDAV client does something stupid, which Davrods doesn't expect and causes a misbehavior in the iRODS protocol, which breaks replication.
Apache error log gave something possibly related:
[Tue Dec 12 16:18:41.282197 2017] [dav:error] [pid 12128] [client 192.168.56.1:61710] Unable to deliver content. [500, #0]
[Tue Dec 12 16:18:41.282231 2017] [dav:error] [pid 12128] (32)Broken pipe: [client 192.168.56.1:61710] Could not write contents to filter. [500, #0]
The iRODS endpoint server (in this case the iCAT) logs were clean as well as the rodsLog.*
files on both of the resource servers.
The server environment was a cleanly built virtual cluster via irods-provisioner
on CentOS 7 hosts, davrods-1.3.0 and iRODS 4.1.1.11.
Due to a restructuring of the client libraries in iRODS 4.2, Davrods currently fails to build against this iRODS version.
davrods is an incredibly useful software package, again: thanks for it.
It would be great if davrods would compute checksums for uploaded files. Is this possible?
Hi, I have problems with accessing BAM files through HTTPS+Davrods in IGV.
Does Davrods support HTTP range queries?
I'm getting the following access/error logs
==> /var/log/apache2/error.log <==
[Thu Aug 24 14:34:28.938949 2017] [dav:error] [pid 2451:tid 139841358481152] [client 172.16.128.8:57958] Unable to deliver content. [500, #0]
[Thu Aug 24 14:34:28.939049 2017] [dav:error] [pid 2451:tid 139841358481152] (104)Connection reset by peer: [client 172.16.128.8:57958] Could not write contents to filter. [500, #0]
==> /var/log/apache2/other_vhosts_access.log <==
cubi-webdav-omics-prototype.bihealth.org:443 172.16.128.8 - holtgrem [24/Aug/2017:14:32:27 +0200] "GET /omicsZone/projects/project8/bio_samples/33900/33900-N1/33900-N1-DNA1/33900-N1-DNA1-WES1/2017-08-23/GRCh37/ngs_mapping/out/bwa.33900-N1-DNA1-WES1.bam.bai HTTP/1.1" 500 519832 "-" "IGV Version 2.3.97 (157)07/10/2017 10:52 PM"
cubi-webdav-omics-prototype.bihealth.org:443 172.16.128.8 - holtgrem [24/Aug/2017:14:34:35 +0200] "HEAD /omicsZone/projects/project8/bio_samples/33900/33900-N1/33900-N1-DNA1/33900-N1-DNA1-WES1/2017-08-23/GRCh37/ngs_mapping/out/bwa.33900-N1-DNA1-WES1.bam.bai HTTP/1.1" 200 2352 "-" "IGV Version 2.3.97 (157)07/10/2017 10:52 PM"
cubi-webdav-omics-prototype.bihealth.org:443 172.16.128.8 - holtgrem [24/Aug/2017:14:34:35 +0200] "HEAD /omicsZone/projects/project8/bio_samples/33900/33900-N1/33900-N1-DNA1/33900-N1-DNA1-WES1/2017-08-23/GRCh37/ngs_mapping/out/bwa.33900-N1-DNA1-WES1.bam HTTP/1.1" 200 277 "-" "IGV Version 2.3.97 (157)07/10/2017 10:52 PM"
IGV is giving me
Error loading https://cubi-webdav-omics-prototype.bihealth.org/omicsZone/projects/project8/bio_samples/33900/33900-N1/33900-N1-DNA1/33900-N1-DNA1-WES1/2017-08-23/GRCh37/ngs_mapping/out/bwa.33900-N1-DNA1-WES1.bam: Premature EOF. Expected 4 but only received 0; BinaryCodec in readmode; streamed file (filename not available)
I think I've found a bug in davrods. I have apache configured with mod_cache/mod_cache_disk and mod_dav/davrods. The cache has a copy of an iRODS data object. The copy is valid, meaning the data object in iRODS still exists and hasn't been modified since the copy was cached. The copy is expired though. According to RFC 2616, when a GET
request is made for this data object, the apache should check to see if the copy is still valid in iRODS. Since it is, apache should update the expiration time and return the cached copy as the body of a 200
(OK) response. Instead, when I make the GET
request, apache returns a 304
(NOT MODIFIED) response with an empty body.
I wasn't certain if this was an issue with davrods or something else. To test if this was a bug in davrods, I added mod_dav_fs based WebDAV repository on the apache server's local filesystem. I performed the same test. When I made the GET
request for the file in local WebDAV repository, the cached copy was returned as the body of a 200
response. This implies that the bug is likely in davrods.
I'm using iRODS 4.2.8. For the WebDAV server running on CentOS 7, I'm using apache 2.4.6 and davrods 4.2.8_1.5.0. Here's the virtual host configuration.
<VirtualHost *:80>
ServerName 128.196.65.41
### MOD_CACHE CONFIGURATION
CacheDetailHeader On
CacheEnable disk /
CacheRoot /var/cache/httpd/proxy
# Have cached files expire quickly
CacheMaxExpire 1
###
### MOD_DAV_FS CONFIGURATION
DavLockDB /var/www/DavLock
Alias /dav_fs /var/www/webdav
<Location /dav_fs/>
AuthType None
Require all granted
Dav On
</Location>
###
### DAVRODS CONFIGURATION
<Location /davrods/>
AuthType None
Require all granted
Dav davrods-locallock
DavRodsEnvFile /etc/httpd/irods/irods_environment.json
DavRodsServer 128.196.65.131 1247
DavRodsZone cyverse.k8s
DavRodsAnonymousMode On
DavRodsAnonymousLogin "anonymous" ""
DavRodsExposedRoot /cyverse.k8s/home/shared
DavRodsLockDB /var/lib/davrods/lockdb_locallock
DirectoryIndex disabled
</Location>
###
</VirtualHost>
Here's a curl based example of how mod_dav_fs responds to caching. Notice that when retrieving a file from the cache when the cached copy is expired but still valid, it refreshes the cached copy and returns as the body of a 200
response.
prompt> curl -v 128.196.65.41/dav_fs/MOTD
* Trying 128.196.65.41...
* TCP_NODELAY set
* Connected to 128.196.65.41 (128.196.65.41) port 80 (#0)
> GET /dav_fs/MOTD HTTP/1.1
> Host: 128.196.65.41
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Tue, 25 Aug 2020 18:58:31 GMT
< Server: Apache/2.4.6 (CentOS)
< Last-Modified: Tue, 25 Aug 2020 18:21:52 GMT
< Content-Length: 20
< ETag: "14-5adb7c6fc1c8e"
< Accept-Ranges: bytes
< X-Cache-Detail: "conditional cache hit: entity refreshed" from 128.196.65.41
<
Hi from mod_dav_fs!
* Connection #0 to host 128.196.65.41 left intact
Here's one for how davrods responds to caching. Notice that when retrieving a file from the cache when the cached copy is expired but still valid, it returns a 304
response.
prompt> curl -v 128.196.65.41/davrods/MOTD
* Trying 128.196.65.41...
* TCP_NODELAY set
* Connected to 128.196.65.41 (128.196.65.41) port 80 (#0)
> GET /davrods/MOTD HTTP/1.1
> Host: 128.196.65.41
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 304 Not Modified
< Date: Tue, 25 Aug 2020 18:58:46 GMT
< Server: Apache/2.4.6 (CentOS)
< ETag: "11-01598381793"
<
* Connection #0 to host 128.196.65.41 left intact
Hi,
Could you please bump your package so that is supports iRODS latest version (4.2.6) ?
Cheers, Samuel from CINES (France)
It is often the case that a web application needs to access a user's data managed by another service. One solution, is to have the web service connect to the data service as the user, using the user's account name and password for authentication. This isn't always viable. For example, if an authenticated user schedules an HPC job that processes some of the user's data, the job may not have access to the user's password for the data service. iRODS supports this case by allowing the web application to impersonate the user. The web application would authenticate with iRODS using a rodsadmin type account and provide the user's iRODS account name as the client user.
I'm requesting that davrods be extended to support iRODS style user impersonation. The web application could connect to a davrods based data service with basic authentication, providing its iRODS account name and password. The client name could be provided in a custom HTTP header, e.g., X-Davrods-Impersonate-User: CLIENT-USERNAME
. The client's account name could be passed to iRODS through the iRODS C API similar to how the iCommands pass it in.
Hello,
I am using Davrods to drop files into my irods instance. Every thing is working fine. I can drop images and check through ils -L icommand. I also was working with irules. So I created an irule to check if the file is an image and extract all metadata and insert into irods. My irule works fine when I use iput command, but when I test to drop the images through Davrods I got an error on the httpd server.
tail /etc/httpd/logs/error_log
[Wed Oct 26 10:41:29.282775 2016] [dav:error] [pid 11405] [client 145.117.144.70:59273] Unable to deliver content. [500, #0]
[Wed Oct 26 10:41:29.282820 2016] [dav:error] [pid 11405] (32)Broken pipe: [client 145.117.144.70:59273] Could not write contents to filter. [500, #0]
ERROR: [-] iRODS/lib/core/src/procApiRequest.cpp:266:readAndProcApiReply : status [SYS_HEADER_READ_LEN_ERR] errno [] -- message []
[-] iRODS/lib/core/src/sockComm.cpp:199:readMsgHeader : status [SYS_HEADER_READ_LEN_ERR] errno [] -- message [failed to call 'read header']
[-] libssl.cpp:577:ssl_read_msg_header : status [SYS_HEADER_READ_LEN_ERR] errno [] -- message [read 0 expected 4]
[Wed Oct 26 10:41:36.192629 2016] [davrods:error] [pid 11389] [client 145.117.144.70:59275] rcDataObjClose failed: -4000 = SYS_HEADER_READ_LEN_ERR
[Wed Oct 26 10:41:36.192643 2016] [dav:error] [pid 11389] [client 145.117.144.70:59275] Could not close the uploaded resource [500, #0]
ERROR: [-] iRODS/lib/core/src/rcConnect.cpp:269:rcDisconnect : status [SYS_HEADER_WRITE_LEN_ERR] errno [Broken pipe] -- message []
[-] iRODS/lib/core/src/sockComm.cpp:1394:sendRodsMsg : status [SYS_HEADER_WRITE_LEN_ERR] errno [Broken pipe] -- message [failed to call 'write body']
[-] libssl.cpp:1058:ssl_send_rods_msg : status [SYS_HEADER_WRITE_LEN_ERR] errno [Broken pipe] -- message [Write message header failed.]
[-] iRODS/lib/core/src/sockComm.cpp:473:writeMsgHeader : status [SYS_HEADER_WRITE_LEN_ERR] errno [Broken pipe] -- message []
[-] libssl.cpp:1001:ssl_write_msg_header : status [SYS_HEADER_WRITE_LEN_ERR] errno [Broken pipe] -- message [Wrote -1 expected 140.]
ERROR: [-] iRODS/lib/core/src/rcConnect.cpp:281:rcDisconnect : status [SSL_SHUTDOWN_ERROR] errno [] -- message []
[-] iRODS/lib/core/src/sockComm.cpp:100:sockClientStop : status [SSL_SHUTDOWN_ERROR] errno [] -- message [failed to call 'client stop']
[-] libssl.cpp:648:ssl_client_stop : status [SSL_SHUTDOWN_ERROR] errno [] -- message [error shutting down the SSL connection]
tail /var/lib/irods/iRODS/server/log/rodsLog.2016.10.26
Authenticated
Oct 26 10:40:30 pid:11611 NOTICE: writeLine: inString = Hello world! A image file was uploaded [/var/lib/irods/iRODS/Vault/home/felipe/nave.jpg].
terminate called after throwing an instance of 'Magick::ErrorCorruptImage'
what(): irodsAgent: Empty input file `/var/lib/irods/iRODS/Vault/home/felipe/nave.jpg' @ error/jpeg.c/JPEGErrorHandler/316
Oct 26 10:40:30 pid:11315 NOTICE: Agent process 11611 exited with status 6
cat /etc/irods/training_acPostProcForPut.re
acPostProcForPut {
if ($filePath like "*.jpg" || $filePath like "*.jpeg" || $filePath like "*.bmp" || $filePath like "*.tif" || $filePath like "*.tiff" || $filePath like "*.rif" || $filePath like "*.gif" || $filePath like "*.png" || $filePath like "*.svg" || $filePath like "*.xpm") {
writeLine("serverLog", "Hello world! A image file was uploaded [$filePath].");
msiget_image_meta($filePath, *meta);
writeLine("serverLog", "image file 1111");
msiString2KeyValPair(*meta, *meta_kvp);
writeLine("serverLog", "image file 2222");
msiAssociateKeyValuePairsToObj(*meta_kvp, $objPath, "-d");
writeLine("serverLog", "image file 3333");
} # if
} # acPostProcForPut
Hello,
Is there a way to make the davrods package give read-only access to iRODS? I would assume some configuration somewhere (apache?) could do that but pointers to where would be greatly appreciated.
Thanks,
Burnce
I upgraded my irods servers to 4.3.1 which are configured with PAM authentication. And then could no more authenticate with webdav.
I upgraded my docker-davrods container to 4.3.0_1.5.0 but no success.
When trying to authenticate, the log of the container reports:
[Wed Mar 27 08:20:22.455920 2024] [davrods:warn] [pid 17] [client 172.17.0.1:45460] rcPamAuthRequest failed: -994000 = PAM_AUTH_PASSWORD_INVALID_TTL
[Wed Mar 27 08:20:22.463480 2024] [auth_basic:error] [pid 17] [client 172.17.0.1:45460] AH01617: user bzizou: authentication failure for "/davrods": Password Mismatch
172.17.0.1 - bzizou [27/Mar/2024:08:20:22 +0000] "GET /davrods HTTP/1.1" 401 381 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"
I tried to change PAM
to pam_password
into the /etc/httpd/irods/irods_environment.json
(as it is a change of 4.2.x -> 4.3.x) with no more success
Build fails on RHEL6. I think I have all required packages installed.
Where are AUTHN_PROVIDER_VERSION and AP_AUTH_INTERNAL_PER_CONF supposed to come from?
[root@icat build]# make
[ 12%] Building C object CMakeFiles/mod_davrods.dir/auth.c.o
/root/davrods-4.2.1_1.3.0/auth.c: In function ‘davrods_auth_register’:
/root/davrods-4.2.1_1.3.0/auth.c:442: warning: implicit declaration of function ‘ap_register_auth_provider’
/root/davrods-4.2.1_1.3.0/auth.c:445: error: ‘AUTHN_PROVIDER_VERSION’ undeclared (first use in this function)
/root/davrods-4.2.1_1.3.0/auth.c:445: error: (Each undeclared identifier is reported only once
/root/davrods-4.2.1_1.3.0/auth.c:445: error: for each function it appears in.)
/root/davrods-4.2.1_1.3.0/auth.c:447: error: ‘AP_AUTH_INTERNAL_PER_CONF’ undeclared (first use in this function)
make[2]: *** [CMakeFiles/mod_davrods.dir/auth.c.o] Error 1
make[1]: *** [CMakeFiles/mod_davrods.dir/all] Error 2
make: *** [all] Error 2
Using the iCommand imcoll
, a tar data object can be mounted as a collection, allowing other iCommands to transparently access the contents of the tar data object. For example,
prompt> ls set
file1 file2
prompt> tar -c -C set -f set.tar .
prompt> iput -D tar set.tar /zone/home/user
prompt> imkdir /zone/home/user/set
prompt> imcoll -m tar /zone/home/user/set.tar /zone/home/user/set
prompt> ils set
file1
file2
It would be nice if davrods could provide WebDAV support for mounted tar data objects. At the moment, davrods allows navigation of the directories in the mounted tar data object, but it doesn't allow any files to be downloaded. When a download is attempted, it fails and logs a rcDataObjLseek failed: -19000 = SYS_FILE_DESC_OUT_OF_RANGE
error.
Hi Davrods folks,
We've got davrods deployed but, while we can download large (eg 10GB) files, we cannot upload them.
The upload fails with:
413 Request Entity Too Large
Do you have instructions or advice for increasing the upload file size limit? In the long run, we may need 100GB -> 1TB.
Thanks for this project!
Best,
Blake
Hey,
I'm currently evaluating davrods and I face the issue, that davrods cannot connect to the irods server. My setup consists of two container (one for irods, one for davrods) in the same network.
When I run icommands inside davrods I can connect to the irods server without any problem. When I use the same settings/credentials with davrods and the provided UI at port 80 (localhost:8888 is mapped to port 80 inside davrods), I receive an internal server error.
Based on the log messages I think the issue starts here. I'm not very familiar with C but as far as I understood the whole (optional) SSL handling happens after the call to rcConnect.
Setup:
Do you have any hints on how to fix this issue? Thanks!
Logs:
[Wed Feb 16 10:23:53.674751 2022] [davrods:debug] [pid 568:tid 140737085699840] auth.c(406): [client 172.19.0.1:44882] Authenticating iRODS username 'rods' using Native auth scheme., referer: http://localhost:8888/
[Wed Feb 16 10:23:53.674761 2022] [davrods:debug] [pid 568:tid 140737085699840] auth.c(128): [client 172.19.0.1:44882] Connecting to iRODS using address <irods:1247>, username <rods> and zone <zone_1>, referer: http://localhost:8888/
[Wed Feb 16 10:23:53.674774 2022] [davrods:debug] [pid 568:tid 140737085699840] auth.c(137): [client 172.19.0.1:44882] Using iRODS env file at </etc/apache2/irods/irods_environment.json>, referer: http://localhost:8888/
remote addresses: 172.19.0.1 ERROR: [-] /irods/lib/core/src/sockComm.cpp:811:int connectToRhost(rcComm_t *, int, int) : status [SSL_HANDSHAKE_ERROR] errno [] -- message [failed to call 'client start']
[-] /irods/lib/core/src/sockComm.cpp:64:irods::error sockClientStart(irods::network_object_ptr, rodsEnv *) : status [SSL_HANDSHAKE_ERROR] errno [] -- message [failed to call 'client start']
[-] /irods/plugins/network/ssl/libssl.cpp:691:irods::error ssl_client_start(irods::plugin_context &, rodsEnv *) : status [SSL_HANDSHAKE_ERROR] errno [] -- message [error in SSL_connect]
remote addresses: 172.19.0.1 ERROR: _rcConnect: connectToRhost error, server on irods:1247 is probably down status = -2103000 SSL_HANDSHAKE_ERROR
[Wed Feb 16 10:23:53.777908 2022] [davrods:error] [pid 568:tid 140737085699840] [client 172.19.0.1:44882] Could not connect to iRODS using address <irods:1247>, username <rods> and zone <zone_1>. iRODS says: '_rcConnect: connectToRhost failed\n', referer: http://localhost:8888/
Update :
I by passed this problem with a link to the dir /usr/lib64 instead of /usr.lib
end.
Using the 4.2_1.1.1 rpm , at the start of httpd, an error came up:
Jul 17 10:57:02 foo.foo.fr httpd[8255]: httpd: Syntax error on line 56 of /etc/httpd/conf/httpd.conf: Syntax error on line 1 of /etc/httpd/conf.modules.d/10-davrods.conf: Cannot load modules/mod_davrods.so into server: libirods_client.so**.4.2.0**: cannot open shared object file: No such file or directory
Jul 17 10:57:02 foo.foo.fr systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
This is because the lib version installed are : 4.2.1 NOT 4.2.0
Can you fix that / How can i fix that ?
Thx
Cheers
S.
Hi,
I'm not sure whether this would be a request for enhancement or a bug report, but in any case I decided to report the following. Due to certain workarounds we had to establish in our environment after iRODS 4.2 upgrade, we had to start using the so called dynamic policy enforcement points, i.e pep_api_data_obj_put_post
instead of acPostProcForPut
.
For some reason Davrods seems to trigger only acPostProcForPut
.
Test scenario using irods-provisioner
and spinning up a test cluster.
$ cd /tmp
$ git clone https://github.com/KTH-PDC/irods-provisioner.git
$ cd irods-provisioner
$ vagrant up
$ ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -b -i hosts-test irods-cluster.yml
$ vagrant ssh rodsmaster1
[vagrant@rodsmaster1 ~]$ sudo su - irods
-bash-4.2$ cat >>/etc/irods/core-override.re
acPostProcForPut
{
writeLine("serverLog", "acPostProcForPut: objPath='$objPath'");
}
pep_api_data_obj_put_post(*INSTANCE_NAME, *COMM, *DATAOBJINP, *BUFFER, *PORTAL_OPR_OUT)
{
*objPath = *DATAOBJINP.obj_path;
writeLine("serverLog", "pep_api_data_obj_put_post: objPath = '*objPath'");
}
^D
At the host one can mount the WebDAV endpoint (macOS commands displayed):
$ mkdir -p /tmp/webdav
$ mount_webdav -i http://rodsfront1:8443/home/rods/ /tmp/webdav
Username: rods
Password:
$ echo blaa > blaa.txt
$ cp blaa.txt /tmp/webdav
And afterwards we can observe in rodsLog
:
Dec 10 10:05:16 pid:12570 NOTICE: writeLine: inString = acPostProcForPut: objPath='/tempZone/home/rods/blaa.txt'
Dec 10 10:05:16 pid:12570 NOTICE: writeLine: inString = acPostProcForPut: objPath='/tempZone/home/rods/blaa.txt'
Hi everyone, I have problems viewing or downloading files using webdav interface after I've deleted the associated files on the cache folder.
Is there any way con configure davrods to access directly to the files store instead of the cache folder?
I don't know if this is a bug or a implemented way of use of the WebDAV interface, but in my case, I need that the files showed in the WebDAV interface will be pulled directly from the archive repository instead of the cache repo. Our archive repository is growing faster and the cache disk is too small to have a copy of ALL the data stored.
PS: I'm using a compound resource in which the cache is stored in a local disk and the archive folder is stored in an Amazon S3 bucket... (the iRODS resource server disk is a virtual machine too, so the disk used in the machine is a virtual disk.) ...double size double cost....
Thank you for your help
Alejandro Lebrero
Is it possible/how to run davrods on a iRODS 4.2.9 setup?
Hello All and thank you for the DavRODS package!
I see 4.2.9_1.5.0 is in a "pre-release" state. But I also see there is an .rpm available already.
I'm doing an Ubuntu 18 install and was planning on using your .deb. Will that be available soon or should I work-around?
Thanks,
Blake
It would be very useful to allow access via iRODS tickets in one way or another. The ticket could be part of the login link (problematic for direct login as modern browsers warn if you get a http://user:password@host
link as this is commonly used for phishing).
Another way would require another sort of authentication, which probably implies using a different (sub) domain as WebDAV has to be at the root URL for many client implementation. The ticket could also just be a query parameter as is the (infamous) session ID in early PHP/Java/... session implementations.
I have the following use case in mind.
E.g., I would call my custom script "publish-ucsc-track.sh local/path". The script uploads things below a predefined location on my iRODS server. It then creates a ticket and dumps a URL to the new collection in DavRods, e.g., https://my-davrods.example.com/track-hubs/random-id?ticket=ticket-value
.
I tried to be terse here but I don't know if this request makes sense already. Let me know where this needs more work to be more clear (even if you don't consider this for immediate implementation).
Hello,
I'm using irods for physical storage and I have an irods montage (davrods) on my ubuntu 18.04 machine of about 3To size.
The aim is to run python scripts for satellite image processing.
I successfully downloaded and stored my images (~50Go) however when i runned my python code for image processing (stack of some images) the output file never exceeded 32 MB. It increases to 32MB and then empties and refills. It is as if the maximum size was limited to 32 MB...
In the irods_environment.json file, i modified the irods_maximum_size_for_single_buffer_in_megabytes from 32 to 32000 but it didn't solve the problem.
here is a trace of log in irods (v 4.2.5)
Oct 29 17:22:44 pid:8380 remote addresses: 127.0.0.1, 134.158.150.48,
193.50.94.68 ERROR: [-]
/tmp/tmppTB_kL/server/core/src/rsApiHandler.cpp:540:int
readAndProcClientMsg(rsComm_t *, int) : status
[SYS_HEADER_READ_LEN_ERR] errno [] -- message [only read [0] of [4]]
and in davrods (V 4.2.6_1.4.2-1).
[Tue Oct 29 18:00:22.212903 2019] [dav:error] [pid 164681] [client
193.50.94.68:45502] Unable to deliver content. [500, #0]
[Tue Oct 29 18:00:22.212939 2019] [dav:error] [pid 164681]
(104)Connection reset by peer: [client 193.50.94.68:45502] Could not wri
te contents to filter. [500, #0]
Thank's a lot for your advice
P.M.!
Because compiler remove all 'assert' from release build, assert(set_exposed_root(conf, exposed_root) >= 0);
in davrods_merge_dir_config
never evaluated and exposed_root
always set to "User".
The solution is to move out set_exposed_root
from assert.
--- a/src/config.c
+++ b/src/config.c
@@ -123,7 +123,8 @@ void *davrods_merge_dir_config(apr_pool_t *p, void *_parent, void *_child) {
DAVRODS_PROP_MERGE(anonymous_auth_username);
DAVRODS_PROP_MERGE(anonymous_auth_password);
- assert(set_exposed_root(conf, exposed_root) >= 0);
+ int rc = set_exposed_root(conf, exposed_root);
+ assert(rc >= 0);
DAVRODS_PROP_MERGE(html_head);
DAVRODS_PROP_MERGE(html_header);
When I install all the RPMs listed in the readme, I get the following errors
/usr/bin/ld: cannot find -lstdc++
/usr/bin/ld: cannot find -lboost_system
/usr/bin/ld: cannot find -lboost_filesystem
/usr/bin/ld: cannot find -lboost_regex
/usr/bin/ld: cannot find -lboost_thread
/usr/bin/ld: cannot find -lboost_chrono
/usr/bin/ld: cannot find -ljansson
collect2: error: ld returned 1 exit status
I have been able to build after adding the following packages:
I used the provided davrods-vhost.conf
and received a SegFault when trying to connect. Uncommenting the line
DavrodsExposedRoot User
fixed the issue. Havn't tried it for other config values.
Cheers
(gdb) bt full
#0 __strcasecmp_l_avx () at ../sysdeps/x86_64/multiarch/strcmp-sse42.S:199
No locals.
#1 0x00007ffff522b59e in set_exposed_root (exposed_root=0x0, conf=<optimized out>, conf=<optimized out>) at /home/vagrant/davrods/src/config.c:31
No locals.
#2 0x00007ffff522b7f0 in davrods_merge_dir_config (p=<optimized out>, _parent=0x7ffff7fa1ea8, _child=0x7ffff7f3c4d0) at /home/vagrant/davrods/src/config.c:140
ret = <optimized out>
parent = 0x7ffff7fa1ea8
child = 0x7ffff7f3c4d0
conf = 0x7ffff7eb9a58
exposed_root = <optimized out>
__PRETTY_FUNCTION__ = "davrods_merge_dir_config"
#3 0x00005555555ae70f in ap_merge_per_dir_configs ()
No symbol table info available.
#4 0x00005555555a6862 in ap_location_walk ()
No symbol table info available.
#5 0x00005555555a6c28 in ap_process_request_internal ()
No symbol table info available.
#6 0x00005555555c6228 in ap_process_async_request ()
No symbol table info available.
#7 0x00005555555c28e0 in ?? ()
No symbol table info available.
#8 0x00005555555b8090 in ap_run_process_connection ()
No symbol table info available.
#9 0x00007ffff1665ecf in ?? () from /usr/lib/apache2/modules/mod_mpm_event.so
No symbol table info available.
#10 0x00007ffff16668e5 in ?? () from /usr/lib/apache2/modules/mod_mpm_event.so
No symbol table info available.
#11 0x00007ffff74e96db in start_thread (arg=0x7fffefe01700) at pthread_create.c:463
pd = 0x7fffefe01700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140737217828608, 6623695043794037053, 140737217826688, 0, 140737353479696, 140737353475416, -6623730396236617411, -6623676774370776771}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
#12 0x00007ffff721271f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
No locals.
I'm currently trying to deploy davrods on a debian 12 environment, as a standalone server (irods 4.3.1 is deployed on an external server).
Currently renci only ships irods-runtime 4.3.0 and 4.3.1 for Debian, and unfortunately building from source is not a reliable option for me.
Do you plan to package a DEB for debian 12 in the future ?
Thanks,
technics
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.