cern-fts / davix

High-performance file management over WebDAV / HTTP

Davix

Davix aims to make the task of managing files over HTTP-based protocols simple. It is being developed by IT-ST at CERN, and while the project's purpose is its use on the CERN grid, the functionality offered is generic.

Documentation

Visit https://davix.web.cern.ch to view the latest documentation.

HTTP File Management

HTTP is gaining popularity for file management tasks, beyond its traditional use for serving web pages. It is versatile enough to be fit for this purpose; PUT, MOVE and DELETE requests can be used for basic file manipulation, for example. (uploading, moving, and deleting a file, respectively)

Some common file-management operations are not possible to do with plain HTTP, however, which is why the WebDAV extensions were developed, which davix supports.

Davix also supports a plethora of authentication methods:

• x509 user certificate
• VOMS proxy
• RFC proxy with VOMS extensions support
• username / password
• AWS S3 compatible services
• Microsoft Azure compatible services

Usage

Davix provides a shared library as well as a few command line tools. The library offers two sets of APIs, a file-oriented and a POSIX-like interface.

Here are some example invocations of the command-line tools.

# upload a file using a VOMS proxy
davix-put myfile https://someserver/dir/myfile -E /tmp/x509up_u1000
# download a file from an Amazon S3 bucket
davix-get https://mybucket.s3.amazonaws.com/somefile --s3accesskey [..] --s3secretkey [..]
# do an ls on a WebDAV-enabled server
davix-ls https://someserver/dir

Compiling

1. Install the necessary dependencies:

   • cmake
   • libxml2-devel
   • openssl-devel

2. Compile:

git clone https://github.com/cern-fts/davix.git
cd davix
git submodule update --recursive --init
mkdir build && cd build
cmake ..
make

You can now try running an example command:

./src/tools/davix-get https://www.kernel.org/pub/linux/kernel/v4.x/ChangeLog-4.0.1

Release tarballs

To generate a blessed release tarball, run packaging/make-dist.sh from the root of this git repository. The tarball will appear under ./build, and will correspond to whichever git commit or tag you're currently on.

Development

The official repository is the one on GitHub. It's automatically mirrored on CERN Gitlab for CI purposes. This means:

• Use GitHub for new commits, issues, or pull requests.
• Please don't commit directly on GitLab.
• After a commit, GitLab will mirror the changes automatically, and run CI. Treat Gitlab as if it were ie a Jenkins CI instance.

The project uses a simplified GitFlow branching approach. The main branch is the devel branch, with master being reserved only for tagged released.

Feature branches are developed separately and merged into the devel branch. When preparing a release, either devel is merged directly into master or a release branch is created. Hotfix branches start from master, have a very targeted objective before being merged back into master and should be employed only in case of necessity. Changes done on release and hotfix branches must be merged back into devel.

Contact

Suggestions and patches are more than welcome. You can send an email to the davix-devel CERN mailing list, or contact directly the current maintainer.

davix's Issues

[regression] set_target_properties called with incorrect number of arguments

Version 0.6.9 fails:

-- Looking for uuid_generate_random in uuid - not found
-- Found uuid: /usr/local/include  
CMake Error at src/CMakeLists.txt:40 (set_target_properties):
  set_target_properties called with incorrect number of arguments.


-- generate pkgconfig file for libdavix under /usr/ports/www/davix/work/davix-R_0_6_9/src/davix.pc
-- Configuring incomplete, errors occurred!

HTTP performance markers use blocking reads

In DAVIX, it appears that a blocking read is utilized when waiting on performance markers:

https://github.com/cern-it-sdc-id/davix/blob/devel/src/modules/copy/copy.cpp#L296

The request->readLine method will only return when 1024 bytes of data are available (for the xrootd-tpc implementation of the HTTP COPY verb, this is once every 20 markers or so).

However, the underlying NEON interface can return data once every encoded chunk (in xrootd-tpc, this is once per performance marker). This means we could provide the performance markers once they come in, instead of buffering into 1KB batches.

Encrypted key files do not parse correctly

Hi, I'm trying to upload files with an encrypted PEM key, but I get the following:

tlater ~/projects/webdav-test $ davix-put --key tlater.key.pem --cert tlater.cert.pem test dav://server/test
(Davix::OpenSSL) Error: parse PEM private key failed tlater.key.pem : error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt
(Davix::HttpRequest) Error: (Neon): SSL handshake failed, client certificate was requested: sslv3 alert handshake failure

Unencrypted key files do work, and curl will happily use this one, so I think there's something wrong here.

Behaviour in forked processes

It looks like davix cannot be used in child processes if already used in the parent forking the child.
Spurious buffers maybe present in new connections open by children. In a test ROOT application the problem is systematically occurring when more than 3 child processes are created, rare with 1 or 2 children.

It looks like something does not get re-initialized on subsequent open calls.

Any hint would be welcome.
Is there a way to completely reset Davix in a new process?

A ROOT reproducer can be provided if needed.

davix version 0.8.0 cannot authenticate using username and password

Hi,
I faced with an issue with davix-* commands when trying to authenticate by username and password. It was working before but suddenly stopped and constantly returns: (Davix::HttpRequest) Error: HTTP 401 : Authentification Error

So after investigation I found out the davix client version in the machine was upgraded to version 0.8.0 so I tested it with older version successfully.

We connect to a server with valid certificate over https, but to reproduce the issue here; I use a docker image as a webdav server:

1. Create a docker-compose.yml file with content:

version: '3'
services:
  webdav:
    image: bytemark/webdav
    restart: always
    ports:
      - "80:80"
    environment:
      AUTH_TYPE: Digest
      USERNAME: alice
      PASSWORD: secret1234
    volumes:
      - ./dav:/var/lib/dav

2. Run docker compose up -d to start the webdav server
3. Add some data files in ./dav/data path to have some thing to show
4. I have two versions of davix client installed on my machine 0.7.6 and 0.8.0 let's first try the older version that works:

/usr/local/Cellar/davix/0.7.6/bin/davix-ls -l --userlogin alice --userpass secret1234 http://localhost
-rwxrwxrwx 0     10         2021-10-28 19:17:29 data1.txt
-rwxrwxrwx 0     16         2021-10-28 19:44:54 data2.txt

5. Now try with new version that fails:

/usr/local/Cellar/davix/0.8.0/bin/davix-ls -l --userlogin alice --userpass secret1234 http://localhost
(Davix::HttpRequest) Error: HTTP 401 : Authentification Error

6. For more details we can run it of course with debug version:

/usr/local/Cellar/davix/0.8.0/bin/davix-ls -l --debug --userlogin alice --userpass secret1234 http://localhost
DAVIX(socket): ssl: Initialized OpenSSL thread-safety callbacks for 1 locks.

DAVIX(core): HTTP/SSL Session caching ENABLED
DAVIX(core): Redirection Session caching ENABLED
DAVIX(core): libdavix path /usr/local/Cellar/davix/0.8.0/lib/libdavix.0.8.0.dylib, version: 0.8.0
DAVIX(posix):  -> opendirpp
DAVIX(http): Create HttpRequest for http://localhost
DAVIX(http):  -> negotiateRequest
DAVIX(http): NEON start internal request
DAVIX(http): no cached ne_session, create a new one
HTTP session to http://localhost:80 begins.
DAVIX(http): configure session...
DAVIX(http): define connection timeout to 30
DAVIX(http): enable login/password authentication
DAVIX(http): enable client cert authentication by callback
DAVIX(ssl): ah_create, for WWW-Authenticate
DAVIX(socket): Running pre_send hooks
> PROPFIND / HTTP/1.1
> User-Agent: libdavix/0.8.0 neon/0.0.29
> Keep-Alive:
> Connection: Keep-Alive
> TE: trailers
> Host: localhost
> Depth: 1
> Content-Length: 303
>

DAVIX(socket): Sending request-line and headers:
DAVIX(socket): Doing DNS lookup on localhost...
DAVIX(socket): Sending request body:
DAVIX(socket): Request body sent successfully
DAVIX(socket): Request sent; retry is 0.
< HTTP/1.1 401 Unauthorized
< Date: Thu, 28 Oct 2021 18:01:19 GMT
< Server: Apache/2.4.37 (Unix)
< WWW-Authenticate: Digest realm="WebDAV", nonce="kv9pf23PBQA=87dcf2978519b77f6a33ecf1023df29d2c0c5cd6", algorithm=MD5, qop="auth"
< Content-Length: 381
< Keep-Alive: timeout=5, max=100
< Connection: Keep-Alive
< Content-Type: text/html; charset=iso-8859-1
<
DAVIX(socket): End of headers.
DAVIX(socket): Running post_headers hooks
DAVIX(ssl): Disable Session recycling
DAVIX(http):  <- negotiateRequest
DAVIX(core): Destroy HttpRequest
DAVIX(socket): sess: Closing connection.
DAVIX(socket): sess: Connection closed.
DAVIX(ssl): Disable Session recycling
DAVIX(socket): Running destroy hooks.
DAVIX(socket): Request ends.
DAVIX(socket): sess: Destroying session.
DAVIX(posix):  <- opendirpp
(Davix::HttpRequest) Error: HTTP 401 : Authentification Error

7. This scenario happens also for other commands like davix-get, etc.

Thanks a lot,
Shahram.

[0.7.6.200.3c48eab] davix-get failing auth with DAVIX_USE_LIBCURL

Hi,

there seems to be an auth problem when running davix-get with DAVIX_USE_LIBCURL

[root@d1b4129fd75a davix]# davix-get  --capath /etc/grid-security/certificates/ --cert $X509_USER_PROXY https://lhcbwebdav-kit.gridka.de:2880/pnfs/gridka.de/lhcb/LHCb_USER/lhcb/user/c/chaen/zozo.xml zozo.xml
[root@d1b4129fd75a davix]# export DAVIX_USE_LIBCURL=1
[root@d1b4129fd75a davix]# davix-get  --capath /etc/grid-security/certificates/ --cert $X509_USER_PROXY https://lhcbwebdav-kit.gridka.de:2880/pnfs/gridka.de/lhcb/LHCb_USER/lhcb/user/c/chaen/zozo.xml zozo.xml
(Davix::HttpRequest) Error: Result HTTP 401 : Authentification Error  after 3 attempts

This is done on a cern/cc7-base docker container with the lcg-CA and the following davix rpms

https://storage-ci.web.cern.ch/storage-ci/davix/devel/el7/x86_64/davix-0.7.6.200.3c48eab-1.el7.cern.x86_64.rpm
https://storage-ci.web.cern.ch/storage-ci/davix/devel/el7/x86_64/davix-devel-0.7.6.200.3c48eab-1.el7.cern.x86_64.rpm
https://storage-ci.web.cern.ch/storage-ci/davix/devel/el7/x86_64/davix-libs-0.7.6.200.3c48eab-1.el7.cern.x86_64.rpm
https://storage-ci.web.cern.ch/storage-ci/davix/devel/el7/x86_64/davix-debuginfo-0.7.6.200.3c48eab-1.el7.cern.x86_64.rpm
https://storage-ci.web.cern.ch/storage-ci/davix/devel/el7/x86_64/davix-doc-0.7.6.200.3c48eab-1.el7.cern.noarch.rpm

Here is an extract of the debug logs that keeps repeating

DAVIX(socket): ssl: Initialized OpenSSL thread-safety callbacks for 41 locks.

DAVIX(core): HTTP/SSL Session caching ENABLED
DAVIX(core): Redirection Session caching ENABLED
DAVIX(core): libdavix path /lib64/libdavix.so.0, version: 0.7.6.200.3c48eab
DAVIX(chain):  -> readToFd
DAVIX(chain): request size 0
DAVIX(http): Create HttpRequest for https://lhcbwebdav-kit.gridka.de:2880/pnfs/gridka.de/lhcb/LHCb_USER/lhcb/user/c/chaen/zozo.xml
DAVIX(http):  -> negotiateRequest
DAVIX(http): NEON start internal request
DAVIX(http): add CA PATH /etc/grid-security/certificates/

< GET /pnfs/gridka.de/lhcb/LHCb_USER/lhcb/user/c/chaen/zozo.xml HTTP/1.1
< Host: lhcbwebdav-kit.gridka.de:2880
< Accept: */*
< User-Agent: libdavix/0.7.6.200.3c48eab libcurl/7.69.0-DEV

> HTTP/1.1 401 login failed
> Date: Fri, 06 Aug 2021 09:06:53 GMT
> Server: dCache/6.2.24
> WWW-Authenticate: Basic realm=""
> Cache-Control: must-revalidate,no-cache,no-store
> Content-Type: text/html;charset=iso-8859-1
> Content-Length: 379
DAVIX(http):  ->  Error when using reycling of session/redirect : cancel and try again
DAVIX(http):  -> negotiateRequest

davix-put fails on files with spaces in their names

Version: built from latest github source

davix-put with the following options:

-r10 -k --no-100-continue

Getting errors that look like this:

Davix::DavixOp::PutOp Failed to PUT //2D_seismic_data/kevin_2D/SEG Y Export Job.LOG
(Davix::HttpRequest) Error: davs://dav.box.com/dav//2D_seismic_data/kevin_2D/SEG Y Export Job.LOG is not not a valid HTTP or Webdav URL

This seems to be happening with every file that has a space in the name. I think the URL is not being quoted (or special characters are not being URL encoded).

Unable to compile using tarball

I am unable to build using the tarball at
https://github.com/cern-fts/davix/releases/download/R_0_7_3/davix-0.7.3.tar.gz
due to the following error:

Traceback (most recent call last):
  File "/home/conda/feedstock_root/build_artifacts/davix_1557317461225/work/genversion.py", line 253, in <module>
    main()
  File "/home/conda/feedstock_root/build_artifacts/davix_1557317461225/work/genversion.py", line 224, in main
    gitDescribe = GitDescribe(sh("git describe --dirty").strip())
  File "/home/conda/feedstock_root/build_artifacts/davix_1557317461225/work/genversion.py", line 164, in sh
    raise Exception("Command {0} exited with code {1}".format(cmd, retcode))
Exception: Command git describe --dirty exited with code 128
Configuring cmake for davix version: 0.7.3

This occurs both when configuring with CMake and running make. The script used for the build and can be found here and works when checking out the tag directly.

You can see a full build log here (line 465 onwards).

davix leaks file descriptors

The davix library leaks one file descriptor for write-opened file.

lrwx------. 1 root   root   64 Feb 11 13:25 66 -> /tmp/.davix_tmp_file_XXXXXXXXXXXXXXXXXXyzgG0q (deleted)
lrwx------. 1 root   root   64 Feb 11 13:25 65 -> /tmp/.davix_tmp_file_XXXXXXXXXXXXXXXXXXZ6E10r (deleted)
lrwx------. 1 root   root   64 Feb 11 13:25 63 -> /tmp/.davix_tmp_file_XXXXXXXXXXXXXXXXXXItaC8s (deleted)

The error is in the desctructor here, it unlinks the temporary file, but does not close _fd.

struct IOBufferLocalFile{
    IOBufferLocalFile(int fd, const std::string & filepath): _fd(fd), _filepath(filepath){}
    virtual ~IOBufferLocalFile(){
        DAVIX_SLOG(DAVIX_LOG_TRACE, DAVIX_LOG_CHAIN, "Delete tmp file {}", _filepath);
        unlink(_filepath.c_str());
    }
    int _fd;
    std::string _filepath;
};

It would be also very nice, if the stage location could be modified because /tmp/ is not an optimal place for staging possibly large files.

Finduuid.cmake is not suitable for MacOS

On MacOS the kernel provides uuid functions (e.g. uuid_generate_random()) without the need to link to libuuid.so: https://developer.apple.com/documentation/kernel/1470612-uuid_generate_random?language=objc.

However the Finduuid.cmake module checks for the library and fails to configure if it's not found.
Even on Linux the functions come from the kernel (on my system libuuid.so exists, but has no symbols).

Please consider updating the module to not require the library (or I could create a pull request, just let me know).

Please add instructions how to run tests to the README

Update CMake FindPackage Python to allow Python3 on macOS

Current macOS does not come with python 2 anymore. CMake should be updated to use the new FindPackage Python, similar to what's done here: https://github.com/dpiparo/vdt/pull/12/files - that will happily find Python3.

This blocks building ROOT on current macOS systems, version 12.3.

davix-put doesn't allow to upload local directory to http server

With davix-0.7.3-1, we do not manage to upload a local directory to http server:

$ davix-put -H "Authorization: Bearer ${AT}" test_folder/ https://example.org/dir1
(Davix::Tools::davix-put) Error: test_folder/is not a valid regular file

This should be possibile, according to man:
EXAMPLES
[...]
Upload local directory to a http server
davix-put mydir http://example.org/dir1

[RFE] ~/.netrc support

For authentication, would it be possible to add ~/.netrc support for providing credentials? This feature is used a lot by FTP command line tools.
PS: thanks so much davix (I use it for WebDAV access)!

Davix bundles an old version of libneon

Could Davix please offer the possibility to build against an external libneon package? I intend to package Davix in Gentoo Linux for use as a dependency by ROOT, but in Gentoo packages must not bundle any dependencies. When trying to use an external libneon, I noticed that neon-0.30 (available in Gentoo) is incompatible with neon-0.29 bundled in Davix.

CVE-2022-32221 in bundled curl library

https://curl.se/docs/CVE-2022-32221.html

Affected versions: libcurl 7.7 to and including 7.85.0
Not affected versions: libcurl < 7.7 and >= 7.86.0

davix bundles 7.69.0

The bundled library is used in the EPEL 7 and EPEL 8 builds, because the system version is too old.
EPEL 9 and Fedora uses the system version.

I have backported the commit fixing the CVE (a one line patch) to the packages in EPEL 7 and EPEL 8.

Infinite loop in davix slow unit tests when using external libcurl

The test seems to hang because of a version mismatch between strings in the test and the actual library.
I had to apply the following change for the tests to pass for me:

--- a/test/slow-unit/test-utils.hpp
+++ b/test/slow-unit/test-utils.hpp
@@ -33,7 +33,7 @@
 #define SSTR(message) static_cast<std::ostringstream&>(std::ostringstream().flush() << message).str()
 
 inline std::string getCurlUserAgent() {
-  return SSTR("User-Agent: " << Davix::RequestParams().getUserAgent() << " libcurl/7.69.0-DEV\r\n");
+  return SSTR("User-Agent: " << Davix::RequestParams().getUserAgent() << " libcurl/7.85.0\r\n");
 }
 
 inline std::string getDefaultUserAgent() {

I was compiling against curl-7.85.0. This string may have to be generated for the right library version for the test to always work.

test_config_parser.cpp:(.text+<snip>): undefined reference to Davix::davix_config_tokenize(...)

Davix has been recently packaged in Gentoo, but it is failing to build with testing turned on, as shown below. For more information, please see https://bugs.gentoo.org/651098.

CMakeFiles/g_test_unit_davix_exe.dir/parser/test_config_parser.cpp.o: In function `ConfigParser_Tokenizer_Test::TestBody()':
test_config_parser.cpp:(.text+0x22f): undefined reference to `Davix::davix_config_tokenize(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)'
test_config_parser.cpp:(.text+0x576): undefined reference to `Davix::davix_config_tokenize(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)'
test_config_parser.cpp:(.text+0x804): undefined reference to `Davix::davix_config_tokenize(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)'
test_config_parser.cpp:(.text+0xc02): undefined reference to `Davix::davix_config_tokenize(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&)'

  -------------------------------------------------------------------

  This is an unstable amd64 chroot image at a tinderbox (==build bot)
  name: 17.0-developer_20180319-191208

  -------------------------------------------------------------------

gcc-config -l:
 [1] x86_64-pc-linux-gnu-7.3.0 *

Available Python interpreters, in order of preference:
  [1]   python3.5
  [2]   python2.7 (fallback)

davix version 0.8.5 cannot authenticate using username and password with DAVIX_USE_LIBCURL

[root@txhk2 ~]# export DAVIX_USE_LIBCURL=1
[root@txhk2 ~]# davix-ls --userlogin xxx --userpass xxx https://toi.teracloud.jp/dav/xxx-dat/ --debug
DAVIX(socket): ssl: Initialized OpenSSL thread-safety callbacks for 1 locks.

DAVIX(core): HTTP/SSL Session caching ENABLED
DAVIX(core): Redirection Session caching ENABLED
DAVIX(core): libdavix path /lib64/libdavix.so.0, version: 0.8.5
DAVIX(posix):  -> opendirpp
DAVIX(http): Create HttpRequest for https://toi.teracloud.jp/dav/xxx-dat/
DAVIX(http):  -> negotiateRequest
DAVIX(http): NEON start internal request
DAVIX(http): Failed to load certificate via callback: No valid client credential provided
DAVIX(http): Connection problem, retry
DAVIX(http):  -> negotiateRequest
DAVIX(http): NEON start internal request
DAVIX(http): Failed to load certificate via callback: No valid client credential provided
DAVIX(http): Connection problem, retry
DAVIX(http):  -> negotiateRequest
DAVIX(http): NEON start internal request
DAVIX(http): Failed to load certificate via callback: No valid client credential provided
DAVIX(http): Connection problem, retry
DAVIX(http):  -> negotiateRequest
DAVIX(http): NEON start internal request
DAVIX(http): Failed to load certificate via callback: No valid client credential provided
DAVIX(http): Connection problem: eradicate session
DAVIX(http):  <- negotiateRequest
DAVIX(http):  <- negotiateRequest
DAVIX(http):  <- negotiateRequest
DAVIX(http):  <- negotiateRequest
DAVIX(core): Destroy HttpRequest
DAVIX(posix):  <- opendirpp
(Davix::HttpRequest) Error: Failed to load certificate
[root@txhk2 ~]#

anybody can help ?

R_0_7_2 fails: set_target_properties called with incorrect number of arguments

-- Found uuid: /usr/local/include  
CMake Error at src/CMakeLists.txt:40 (set_target_properties):
  set_target_properties called with incorrect number of arguments.

FreeBSD 11.2

davix-get writes empty file on failure

Consider the following case:

1. Use davix-get to retrieve a file from a remote server.

2. The retrieval is successful and the file is written. :)

3. Use davix-get again to retrieve an updated version of the same file.

4. The retrieval fails due to some error (e.g. name resolution failure), and the old file is overwritten with an empty file. :(

For some context, I use davix to synchronize SSL certificates between multiple remote computers. A temporary failure caused davix to overwrite the certificates with empty files, resulting in errors from the programs using those certificates.

unit tests fail

Test project /usr/ports/www/davix/work/.build
    Start 1: g_test_unit_davix
1/1 Test #1: g_test_unit_davix ................***Failed    0.06 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   0.07 sec

The following tests FAILED:
	  1 - g_test_unit_davix (Failed)
Errors while running CTest
FAILED: CMakeFiles/test.util 

OS: FreeBSD 11.2

Bad regression in 0.4.1

Davix 0.4.1 in epel seems affected by two regression bugs:

• it prints CRs on the screen at every attempt
• davix-get is rejected by S3 when redirected to it (HEAD gets 401)

This is affecting the Data Bridge operation.

Davix 0.4.1-dev works perfectly instead. I suspect a bad error in the last push to epel, or an unfortunate push to the git head.

Show error messages returned by the WebDAV server

Hi!

Davix hides error messages coming from the server, making it difficult to debug issues or use Davix as a client in automated tests.

Example:

$ davix-ls -H "Authorization: Bearer ${BT}" https://storm.example:8443/wlcg
(Davix::HttpRequest) Error: Authentification failed on server

There's a typo in the error message, it should read "Authentication failed on server", but the main problem
is that Davix hides error information coming from the server that could help the user understand what's
going on.

11:27 $ davix-ls --debug -H "Authorization: Bearer ${BT}" https://storm.example:8443/wlcg
DAVIX(socket): ssl: Initialized OpenSSL thread-safety callbacks for 1 locks.

DAVIX(core): HTTP/SSL Session caching ENABLED
DAVIX(core): Redirection Session caching ENABLED
...
DAVIX(socket): Running pre_send hooks
> PROPFIND /wlcg HTTP/1.1
> User-Agent: libdavix/0.7.6 neon/0.0.29
> Keep-Alive:
> Connection: Keep-Alive
> TE: trailers
> Host: storm.example:8443
> Depth: 1
> Authorization: Bearer eyJra....a3re25tgugeWlETlAcay1pxNwdGQCQSfpg
> Content-Length: 303
>
...
DAVIX(socket): Sending request body:
DAVIX(socket): Request body sent successfully
DAVIX(socket): Request sent; retry is 0.
< HTTP/1.1 401 Unauthorized
< WWW-Authenticate: Bearer error="invalid_token", error_description="Unknown token issuer: https://tf.cloud.cnaf.infn.it", error_uri="https://tools.ietf.org/html/rfc6750#section-3.1"
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< Strict-Transport-Security: max-age=31536000 ; includeSubDomains
< X-Frame-Options: DENY
< Content-Length: 0
<
DAVIX(socket): End of headers.
DAVIX(socket): Running post_headers hooks
DAVIX(socket): Running post_send hooks
DAVIX(ssl): ah_post_send (#0), code is 401 (want 401), WWW-Authenticate is Bearer error="invalid_token", error_description="Unknown token issuer: https://tf.cloud.cnaf.infn.it", error_uri="https://tools.ietf.org/html/rfc6750#section-3.1"

DAVIX(ssl): auth: Got challenge (code 401).
DAVIX(ssl): auth: Ignored parameter: error = "invalid_token
DAVIX(ssl): auth: Ignored parameter: error_description = "Unknown token issuer: https://tf.cloud.cnaf.infn.it
DAVIX(ssl): auth: Ignored parameter: error_uri = "https://tools.ietf.org/html/rfc6750#section-3.1

Davix version (on MacOS Mojave):

11:35 $ davix-ls --version
Version: 0.7.6

Thanks!
A.

FTBFS with openssl 1.1.0

davix does not compile with openssl 1.1.0, as reported to Debian BTS:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=828279

Cache login credentials

When I do something like:

davix-put -r 4 a-directory https://somehost.com/owncloud/remote.php/webdav/remotedir/

it works but it ask for login credentials for every file it wants to upload. Could this please be cached?

Return value of `DavFile::get` is -1 but no error is thrown

Hello,

I would like to use DavFile::get to download some files. For the few cases I have tried it seems that DavFile::get returns -1 instead of the number of bytes read, but as far as I can tell, the read succeeds and no DavixError is thrown. I wanted to ask if I am indeed making an error here.

e.g.

#include <davix.hpp>
#include <iostream>
#include <vector>

int main() {
   Davix::Context c;
   Davix::RequestParams p;
   Davix::DavFile file(c, p, Davix::Uri("https://root.cern/files/LHCb002_data_v1.root"));

   std::vector<char> buf;
   auto ret = file.get(nullptr, buf);
   // file size is 11461 B
   if (buf.size() != 11461) {
      std::cout << "expected 11461 bytes read, got " << buf.size() << "\n";
   }
   std::cout << "got ret: " << ret << "\n";
   return 0;
}

prints

got ret: -1

Thank you very much for your time.

Setup

• davix built from source, devel branch
• Ubuntu 18.04

CVE-2023-38545 in bundled curl library

CVE-2023-38545

Affected versions: libcurl 7.69.0 to and including 8.3.0
Not affected versions: libcurl < 7.69.0 and >= 8.4.0

Davix bundles libcurl 7.69.0 on EPEL7 and EPEL8, which is affected.
On EPEL9 and recent Fedora versions, Davix uses the system curl.

A patch must be applied in the build process for the bundled libcurl.

1 test fails, and several tests hang

The failing test:

      Start  2: slow-unit-tests
 2/10 Test  #2: slow-unit-tests .............................................................................***Failed    3.02 sec
Running main() from gtest_main.cc
[==========] Running 10 tests from 3 test cases.
[----------] Global test environment set-up.
[----------] 2 tests from DrunkServer
[ RUN      ] DrunkServer.NoClients
[       OK ] DrunkServer.NoClients (1000 ms)
[ RUN      ] DrunkServer.ClientInteraction
/usr/ports/www/davix/work/davix-R_0_8_5/test/slow-unit/drunk-server.cpp:37: Failure
Value of: connSrv
  Actual: false
Expected: true
[  FAILED  ] DrunkServer.ClientInteraction (1001 ms)
[----------] 2 tests from DrunkServer (2001 ms total)

[----------] 4 tests from Standalone_Neon_Request
[ RUN      ] Standalone_Neon_Request.BasicSanity
/usr/ports/www/davix/work/davix-R_0_8_5/test/slow-unit/standalone-request.cpp:66: Failure
Value of: request->startRequest().ok()
  Actual: false
Expected: true
[  FAILED  ] Standalone_Neon_Request.BasicSanity (2 ms)
[ RUN      ] Standalone_Neon_Request.Redirect
/usr/ports/www/davix/work/davix-R_0_8_5/test/slow-unit/standalone-request.cpp:173: Failure
Value of: request->startRequest().ok()
  Actual: false
Expected: true
[  FAILED  ] Standalone_Neon_Request.Redirect (0 ms)
[ RUN      ] Standalone_Neon_Request.NetworkError
/usr/ports/www/davix/work/davix-R_0_8_5/test/slow-unit/standalone-request.cpp:257: Failure
Value of: "(Neon): Could not read status line: connection was closed by server"
Expected: st.getErrorMessage()
Which is: "Could not connect to server"
[  FAILED  ] Standalone_Neon_Request.NetworkError (1 ms)
[ RUN      ] Standalone_Neon_Request.StopNoStart
[       OK ] Standalone_Neon_Request.StopNoStart (0 ms)
[----------] 4 tests from Standalone_Neon_Request (3 ms total)

[----------] 4 tests from Standalone_Curl_Request
[ RUN      ] Standalone_Curl_Request.BasicSanity
CONSUMED: GET /chickens HTTP/1.1

CONSUMED: Host: localhost:22222

CONSUMED: Accept: */*

CONSUMED: I like: Turtles

WRITING RESPONSE
Response written successfully
[       OK ] Standalone_Curl_Request.BasicSanity (1005 ms)
[ RUN      ] Standalone_Curl_Request.Redirect
CONSUMED: GET /test HTTP/1.1

CONSUMED: Host: localhost:22222

CONSUMED: Accept: */*

CONSUMED: User-Agent: libdavix/0.8.5 libcurl/8.2.1

WRITING RESPONSE
Response written successfully
[       OK ] Standalone_Curl_Request.Redirect (1 ms)
[ RUN      ] Standalone_Curl_Request.NetworkError
[       OK ] Standalone_Curl_Request.NetworkError (0 ms)
[ RUN      ] Standalone_Curl_Request.StopNoStart
[       OK ] Standalone_Curl_Request.StopNoStart (0 ms)
[----------] 4 tests from Standalone_Curl_Request (1006 ms total)

[----------] Global test environment tear-down
[==========] 10 tests from 3 test cases ran. (3010 ms total)
[  PASSED  ] 6 tests.
[  FAILED  ] 4 tests, listed below:
[  FAILED  ] DrunkServer.ClientInteraction
[  FAILED  ] Standalone_Neon_Request.BasicSanity
[  FAILED  ] Standalone_Neon_Request.Redirect
[  FAILED  ] Standalone_Neon_Request.NetworkError

 4 FAILED TESTS

The several next tests hang.

Version: 0_8_5
FreeBSD 13.2

Davix-get/cp does not allow to process more than a one file at a time

with Davix 0.7.5, it doesn't seem possible to process more than one file at a time with a single davix-get/davix-cp command.

For instance, I've tried this syntax to get content from https source to locale:
davix-get -P grid https://example.org:PORT/folder/test1 https://example.org:PORT/folder/test2 > txtfile
which results into: (Davix::Tools::davix-get) Error: File not found for destination file https://example.org/folder/test2

Moreover, if I try to get from the same two files into a local folder:
davix-get -P grid https://example.org:PORT/folder/test1 https://example.org:PORT/folder/test2 ./localfolder/
I get: davix-get, Error: Wrong argument format

Finally, if I try:
davix-cp -P grid https://example.org:PORT/folder/test1 https://example.org:PORT/folder/test2 ./localfolder/

The result is: (Davix::HttpThirdPartyCopy) Error: Transfer failed: failure: SSLException while pushing https://example.org/folder/test2 Broken pipe (Write failed).

All the aforementioned commands have been tried also with the -r 1 option instead of -P, which however did not change the result.

Is it possible to have that functionality?

Thanks a lot for your attention and effort

the MacPorts port

I highly appreciate your effort in providing open source software and supporting legacy equipment like older macs the same.

Naturally, Homebrew is a valid option. However, one may prefer the option to use the MacPorts package manager and there is a davix port available too.

---

http://dmc.web.cern.ch/projects/davix/home - Installation - Easy Install

Please change from:

OSX: brew install davix

to

OSX: brew install davix with Homebrew or sudo port install davix with MacPorts

---

Ref to https://ports.macports.org/port/davix/summary

To my own knowledge the davix port installs nicely on Mac OS X / macOS 10.12.6 Sierra as:

• davix @0.7.5_1

A clean install in a fresh environment on the a.m. OS X would result in:

$ port installed
The following ports are currently installed:
  autoconf @2.69_5 (active)
  automake @1.16.1_1 (active)
  bzip2 @1.0.8_0 (active)
  cmake @3.15.4_0 (active)
  curl @7.66.0_0+ssl (active)
  curl-ca-bundle @7.66.0_1 (active)
  davix @0.7.5_1 (active)
  db48 @4.8.30_4 (active)
  expat @2.2.9_0 (active)
  gdbm @1.18.1_1 (active)
  gettext @0.19.8.1_2 (active)
  glib2 @2.58.3_0+x11 (active)
  gperf @3.1_0 (active)
  help2man @1.47.11_0 (active)
  icu @65.1_0 (active)
  libarchive @3.4.0_1 (active)
  libcxx @5.0.1_4 (active)
  libedit @20190324-3.1_0 (active)
  libffi @3.2.1_0 (active)
  libiconv @1.16_0 (active)
  libidn2 @2.2.0_0 (active)
  libpsl @0.21.0-20190419_0 (active)
  libtool @2.4.6_6 (active)
  libunistring @0.9.10_0 (active)
  libuv @1.33.1_0 (active)
  libxml2 @2.9.9_3 (active)
  lz4 @1.9.2_0 (active)
  lzo2 @2.10_0 (active)
  ncurses @6.1_0 (active)
  openssl @1.1.1d_0 (active)
  p5.28-locale-gettext @1.70.0_0 (active)
  pcre @8.43_0 (active)
  perl5 @5.26.1_0+perl5_28 (active)
  perl5.28 @5.28.2_0 (active)
  pkgconfig @0.29.2_0 (active)
  python3_select @0.0_1 (active)
  python37 @3.7.5_1 (active)
  python_select @0.3_8 (active)
  readline @8.0.000_0 (active)
  sqlite3 @3.30.1_0 (active)
  texinfo @6.7_0 (active)
  xz @5.2.4_0 (active)
  zlib @1.2.11_0 (active)
  zstd @1.4.3_0 (active)

Hope this helps.

---

Happy hacking. 👍

[FreeBSD] Davix build fails on FreeBSD

building on FreeBSD fails because BSD sed -i requires a prefix argument (it is optional in GNU sed)
PR follows

Unit tests fail when run by more than one user on the same machine

Davix's unit tests create a file in /tmp/davix-tests-tmp-file. However, the test doesn't remove this file, so if you run the tests as another user on the same machine (i.e. as root to install davix system-wide), the file will already be there and creation of the file will fail for the second user (with permission denied, as it already exists), causing the test to get a segfault when dereferencing the FILE pointer here, as fopen returns NULL. Debugged this last Friday with @joaopblopes.

Davix 0.7.4 conflicts with system gtest 1.8.1

When trying to compile davix 0.7.4 I ran into a compiler error because a function definition changed in the bundled version of googletest since the latest release 1.8.1

In file included from davix/deps/googletest/googletest/src/gtest-all.cc:45:
davix/deps/googletest/googletest/src/gtest-port.cc:1211:13: error: ambiguating new declaration of 'const char* testing::internal::StringFromGTestEnv(const char*, const char*)'
 const char* StringFromGTestEnv(const char* flag, const char* default_value) {
             ^~~~~~~~~~~~~~~~~~
In file included from common/include/gtest/internal/gtest-internal.h:40,
                 from /common/include/gtest/gtest.h:58,
                 from /davix/deps/googletest/googletest/src/gtest-all.cc:39:
include/gtest/internal/gtest-port.h:2549:13: note: old declaration 'std::__cxx11::string testing::internal::StringFromGTestEnv(const char*, const char*)'
 std::string StringFromGTestEnv(const char* flag, const char* default_val);
             ^~~~~~~~~~~~~~~~~~

googletest 1.8.1 is installed in a common include directory with libuuid so I cannot easily hide it from davix.

As a simple solution, would it be possible to include an option to not build the tests at all?

Introspect redirections for HTTP TPC xfers

To better debug failing HTTP-TPC transfers, it would be nice to expose to users of libdavix the hostname actually responsible for the transfer after redirection, if any.

I originally thought that adding a getCurrentUri analogous to getOriginalUri to expose the protected _current variable from BackendRequest would be sufficient to get things going

davix/src/backend/BackendRequest.hpp

Line 178 in 23a591d

std::shared_ptr<Uri> getOriginalUri() const;

... but this is the wrong place to solve it. In DavixCopyInternal::copy, there is logic to work through the redirections and then find the "correct" host for one side of the transfer:

davix/src/modules/copy/copy.cpp

Lines 229 to 298 in 23a591d

	HttpRequest* request = NULL;
	do {
	nextSrc = _full_url(prevSrc, nextSrc);
	prevSrc = nextSrc;
	if (request) {
	request->discardBody(&internalError);
	if (!internalError)
	request->endRequest(&internalError);
	if (internalError) {
	DavixError::propagatePrefixedError(error, internalError, __func__);
	break;
	}
	}
	delete request;
	DAVIX_SLOG(DAVIX_LOG_VERBOSE, DAVIX_LOG_GRID, "Hop: {}",
	nextSrc);
	request = context.createRequest(nextSrc, &internalError);
	if (internalError) {
	DavixError::propagatePrefixedError(error, internalError, __func__);
	break;
	}
	request->setRequestMethod("COPY");
	if(parameters->getCopyMode() == CopyMode::Push){
	request->addHeaderField("Destination", destination);
	}else if(parameters->getCopyMode() == CopyMode::Pull){
	request->addHeaderField("Source", destination);
	}
	request->addHeaderField("X-Number-Of-Streams", nstreamsStr);
	// for lcgdm-dav, ask for secure redirection in all cases for COPY
	request->addHeaderField("Secure-Redirection", "1");
	// for lcgdm-dav -> S3, add NoHead flag to suppress final head-to-close request
	if(suppressFinalHead)
	request->addHeaderField("Copy-Flags", "NoHead");
	request->setParameters(requestParams);
	request->beginRequest(&internalError);
	if (internalError) {
	DavixError::propagatePrefixedError(error, internalError, __func__);
	break;
	}
	// If we get a X-Delegate-To, before continuing, delegate
	if (request->getAnswerHeader("X-Delegate-To", delegationEndpoint)) {
	std::vector<std::string> delegationEndpoints = parseXDelegateTo(nextSrc, delegationEndpoint, &internalError);
	if (internalError) {
	DavixError::propagatePrefixedError(error, internalError, __func__);
	break;
	}
	DAVIX_SLOG(DAVIX_LOG_VERBOSE, DAVIX_LOG_GRID, "Got delegation endpoint: {}",
	delegationEndpoint.c_str());
	std::string dlg_id = DavixDelegation::delegate(context, delegationEndpoints,
	parameters, &internalError);
	if (internalError) {
	DavixError::propagatePrefixedError(error, internalError, __func__);
	break;
	}
	DAVIX_SLOG(DAVIX_LOG_VERBOSE, DAVIX_LOG_GRID, "Got delegation ID {}",
	dlg_id.c_str());
	dlg_id.clear();
	}
	} while (!shouldCancel() && request->getAnswerHeader("Location", nextSrc) && request->getRequestCode() >= 300 && request->getRequestCode() < 400);

There isn't the analogous logic for the 2nd side of the transfer (which is stored in the destination variable regardless of push/pull mode), the Uri is passed as-is as a Source/Destination header, then the resolution of the redirects happens on the machine we're connected to: (the top-level redirector for each site is xrootd-vanderbilt.sites.opensciencegrid.org and xrootd.cmsaf.mit.edu, note that the Source side is the redirector and not the "resolved" version)

DEBUG Davix: > COPY ///store/user/meloam/TESTMV.txt?authz=BearerXXX HTTP/1.1
> User-Agent: gfal2-util/1.7.1 gfal2/2.20.5 neon/0.0.29
> Keep-Alive: 
> Connection: Keep-Alive
> TE: trailers
> Host: xrootd-se2-vanderbilt.sites.opensciencegrid.org:1094
> Source: https://xrootd.cmsaf.mit.edu:1094//store/temp/user/meloam/hosts.txt
> X-Number-Of-Streams: 0
> Secure-Redirection: 1
> TransferHeaderAuthorization: xx
> Credential: none
> Authorization: xx
> RequireChecksumVerification: false

Since libdavix doesn't resolve the COPY redirects for the destination, there's no way to report back the actual destination host (in this case). But, we do resolve the PROPFIND verb for the destination:

DEBUG Davix: > PROPFIND //store/temp/user/meloam/hosts.txt HTTP/1.1
> User-Agent: gfal2-util/1.7.1 gfal2/2.20.5 neon/0.0.29
> Keep-Alive: 
> Connection: Keep-Alive
> TE: trailers
> Host: xrootd.cmsaf.mit.edu:1094
> Depth: 0
> Authorization: xxx
> 

DEBUG Davix: Sending request-line and headers:
DEBUG Davix: Doing DNS lookup on xrootd.cmsaf.mit.edu...
DEBUG Davix: clicert callback 
DEBUG Davix: call client cert callback 
DEBUG Davix: Request sent; retry is 0.
DEBUG Davix: < HTTP/1.1 302 Redirect
DEBUG Davix: < Connection: Keep-Alive
DEBUG Davix: < Content-Length: 0
DEBUG Davix: < Location: https://xrootd6.cmsaf.mit.edu:1094/store/temp/user/meloam/hosts.txt?authz=BearerXXX
DEBUG Davix: < 
DEBUG Davix: End of headers.
DEBUG Davix: Running post_headers hooks
DEBUG Davix: Add cached redirection <PROPFIND davs://xrootd.cmsaf.mit.edu:1094//store/temp/user/meloam/hosts.txt https://xrootd6.cmsaf.mit.edu:1094/store/temp/user/meloam/hosts.txt?authz=BearerXXX>

Is this on purpose? If instead DavixCopyInternal::copy() resolved redirects and stashed them into Davix::DavixCopyInternal, then accessors could be added to DavixCopy for current source/dest Uris. The changes seem relatively straightforward to implement, but it is a behavioral change, so I don't want to implement if there would be no desire to add that functionality.

Cannot build against system libcurl

The reason is that the library is called libcurl.so and needs to be linked as -lcurl not -llibcurl. However, the build system has no code to support the case in which the embedded library is not used. See the error below.

FAILED: src/libdavix.so.0.8.0 
: && /usr/bin/x86_64-pc-linux-gnu-g++ -fPIC -march=native -O2 -pipe -std=c++11  -Wl,-O1 -Wl,--as-needed -shared -Wl,-soname,libdavix.so.0 -o src/libdavix.so.0.8.0 deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_207.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_acl3744.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_alloc.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_auth.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_basic.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_compress.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_dates.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_i18n.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_locks.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_md5.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_ntlm.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_oldacl.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_pkcs11.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_props.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_redirect.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_request.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_session.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_socket.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_socks.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_sspi.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_string.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_uri.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_utils.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_xml.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_xmlreq.c.o deps/libneon/CMakeFiles/LibNeonObjects.dir/src/ne_openssl.c.o src/CMakeFiles/libdavix.dir/auth/davix_openssl.cpp.o src/CMakeFiles/libdavix.dir/auth/davixauth.cpp.o src/CMakeFiles/libdavix.dir/auth/davixx509cred.cpp.o src/CMakeFiles/libdavix.dir/backend/BackendRequest.cpp.o src/CMakeFiles/libdavix.dir/backend/SessionFactory.cpp.o src/CMakeFiles/libdavix.dir/backend/StandaloneNeonRequest.cpp.o src/CMakeFiles/libdavix.dir/core/ContentProvider.cpp.o src/CMakeFiles/libdavix.dir/core/RedirectionResolver.cpp.o src/CMakeFiles/libdavix.dir/curl/CurlSession.cpp.o src/CMakeFiles/libdavix.dir/curl/CurlSessionFactory.cpp.o src/CMakeFiles/libdavix.dir/curl/HeaderlineParser.cpp.o src/CMakeFiles/libdavix.dir/curl/ResponseBuffer.cpp.o src/CMakeFiles/libdavix.dir/curl/StandaloneCurlRequest.cpp.o src/CMakeFiles/libdavix.dir/deprecated/httpcachetoken.cpp.o src/CMakeFiles/libdavix.dir/file/davfile.cpp.o src/CMakeFiles/libdavix.dir/file/davposix.cpp.o src/CMakeFiles/libdavix.dir/fileops/AzureIO.cpp.o src/CMakeFiles/libdavix.dir/fileops/chain_factory.cpp.o src/CMakeFiles/libdavix.dir/fileops/davix_reliability_ops.cpp.o src/CMakeFiles/libdavix.dir/fileops/davmeta.cpp.o src/CMakeFiles/libdavix.dir/fileops/fileutils.cpp.o src/CMakeFiles/libdavix.dir/fileops/httpiochain.cpp.o src/CMakeFiles/libdavix.dir/fileops/httpiovec.cpp.o src/CMakeFiles/libdavix.dir/fileops/iobuffmap.cpp.o src/CMakeFiles/libdavix.dir/fileops/S3IO.cpp.o src/CMakeFiles/libdavix.dir/fileops/SwiftIO.cpp.o src/CMakeFiles/libdavix.dir/hooks/davix_hooks.cpp.o src/CMakeFiles/libdavix.dir/libs/alibxx/chrono/timepoint.cpp.o src/CMakeFiles/libdavix.dir/libs/alibxx/crypto/base64.cpp.o src/CMakeFiles/libdavix.dir/libs/alibxx/crypto/hmacsha.cpp.o src/CMakeFiles/libdavix.dir/libs/alibxx/str/format.cpp.o src/CMakeFiles/libdavix.dir/libs/datetime/datetime_utils.cpp.o src/CMakeFiles/libdavix.dir/modules/modules_profiles.cpp.o src/CMakeFiles/libdavix.dir/neon/neonrequest.cpp.o src/CMakeFiles/libdavix.dir/neon/neonsession.cpp.o src/CMakeFiles/libdavix.dir/neon/neonsessionfactory.cpp.o src/CMakeFiles/libdavix.dir/params/davixrequestparams.cpp.o src/CMakeFiles/libdavix.dir/request/httprequest.cpp.o src/CMakeFiles/libdavix.dir/status/DavixStatus.cpp.o src/CMakeFiles/libdavix.dir/status/davixstatusrequest.cpp.o src/CMakeFiles/libdavix.dir/utils/checksum_extractor.cpp.o src/CMakeFiles/libdavix.dir/utils/CompatibilityHacks.cpp.o src/CMakeFiles/libdavix.dir/utils/davix_azure_utils.cpp.o src/CMakeFiles/libdavix.dir/utils/davix_gcloud_utils.cpp.o src/CMakeFiles/libdavix.dir/utils/davix_logger.cpp.o src/CMakeFiles/libdavix.dir/utils/davix_misc_utils.cpp.o src/CMakeFiles/libdavix.dir/utils/davix_cs3_utils.cpp.o src/CMakeFiles/libdavix.dir/utils/davix_s3_utils.cpp.o src/CMakeFiles/libdavix.dir/utils/davix_swift_utils.cpp.o src/CMakeFiles/libdavix.dir/utils/simple_get_pass.cpp.o src/CMakeFiles/libdavix.dir/utils/stringutils.cpp.o src/CMakeFiles/libdavix.dir/utils/davixuri.cpp.o src/CMakeFiles/libdavix.dir/xml/azurepropparser.cpp.o src/CMakeFiles/libdavix.dir/xml/davdeletexmlparser.cpp.o src/CMakeFiles/libdavix.dir/xml/davix_ptree.cpp.o src/CMakeFiles/libdavix.dir/xml/davpropxmlparser.cpp.o src/CMakeFiles/libdavix.dir/xml/davxmlparser.cpp.o src/CMakeFiles/libdavix.dir/xml/metalinkparser.cpp.o src/CMakeFiles/libdavix.dir/xml/s3deleteparser.cpp.o src/CMakeFiles/libdavix.dir/xml/S3MultiPartInitiationParser.cpp.o src/CMakeFiles/libdavix.dir/xml/s3propparser.cpp.o src/CMakeFiles/libdavix.dir/xml/swiftpropparser.cpp.o src/CMakeFiles/libdavix.dir/davixcontext.cpp.o  -llibcurl  -lssl  -lcrypto  -lxml2  -lpthread  -luuid  -lz && :
/usr/lib/gcc/x86_64-pc-linux-gnu/10.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: cannot find -llibcurl
collect2: error: ld returned 1 exit status

For now, in order to package Davix, I am using the following patch:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index c65ed46b..35950a9a 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -117,6 +117,8 @@ set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${CXX11_FLAG_ENABLE}")
 if(EMBEDDED_LIBCURL)
   include(buildCurl REQUIRED)
   buildCurl()
+else()
+  find_package(CURL REQUIRED)
 endif()
 
 include_directories(${CMAKE_CURRENT_SOURCE_DIR}/deps/libneon/src/ ${UUID_INCLUDE_DIRS})
diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index 79a8858b..53455abb 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -108,7 +108,7 @@ add_library(libdavix  SHARED ${DAVIX_SOURCES} )
 
 target_include_directories(libdavix PRIVATE ${DAVIX_INTERNAL_INCLUDES} )
 target_link_libraries(libdavix
-  PRIVATE libcurl
+  PRIVATE CURL::libcurl
   PUBLIC ${LIBSSL_PKG_LIBRARIES}
          ${LIBXML2_LIBRARIES}
          ${CMAKE_THREAD_LIBS_INIT}
@@ -146,7 +146,7 @@ set_target_properties(libdavix_static PROPERTIES
 target_include_directories(libdavix_static PRIVATE ${DAVIX_INTERNAL_INCLUDES} )
 
 target_link_libraries(libdavix_static
-  PRIVATE libcurl
+  PRIVATE CURL::libcurl
   PUBLIC  ${OPENSSL_LIBRARIES}
           ${CMAKE_THREAD_LIBS_INIT}
           dl ${UUID_LIBRARIES}

However, that is not a full solution, as that breaks the case with embedded libcurl. It would be nice to have davix support building against all external libraries rather than only the embedded ones. Thanks!

The role of the different branches is unclear

Hi,

In most projects I've seen, new features are committed to the master branch. Although this repo has a master branch, it seems that work is mostly committed to the devel branch.

I think it would help if there was some documentation (perhaps the README.md file) that included a brief description on how the different branches are used.

Cheers,
Paul.

Copyrighted song lyrics in codebase

In

davix/test/unit/response-buffer.cpp

Line 136 in 454950a

std::string contents = "We're no strangers to love. You know the rules and so do I. A full commitment's what I'm thinking of. You wouldn't get this from any other guy. I just wanna tell you how I'm feeling. Gotta make you understand. Never gonna give you up. Never gonna let you down. Never gonna run around and desert you. Never gonna make you cry. Never gonna say goodbye. Never gonna tell a lie and hurt you. We've known each other for so long. Your heart's been aching but you're too shy to say it. Inside we both know what's been going on. We know the game and we're gonna play it. And if you ask me how I'm feeling. Don't tell me you're too blind to see. Never gonna give you up. Never gonna let you down. Never gonna run around and desert you. Never gonna make you cry. Never gonna say goodbye. Never gonna tell a lie and hurt you. Never gonna give you up. Never gonna let you down. Never gonna run around and desert you. Never gonna make you cry. Never gonna say goodbye. Never gonna tell a lie and hurt you. Never gonna give, never gonna give. (Give you up). (Ooh) Never gonna give, never gonna give. (Give you up). We've known each other for so long. Your heart's been aching but you're too shy to say it. Inside we both know what's been going on. We know the game and we're gonna play it. I just wanna tell you how I'm feeling. Gotta make you understand. Never gonna give you up. Never gonna let you down. Never gonna run around and desert you. Never gonna make you cry. Never gonna say goodbye. Never gonna tell a lie and hurt you. Never gonna give you up. Never gonna let you down. Never gonna run around and desert you. Never gonna make you cry. Never gonna say goodbye. Never gonna tell a lie and hurt you. Never gonna give you up. Never gonna let you down. Never gonna run around and desert you. Never gonna make you cry.";

the lyrics of Rick Astley's hit song Never Gonna Give You Up are used. As far as I know, the song lyrics have not been released under a free license. The file copyright statement also claims that the whole file is © Cern which is at odds with the inclusion of the lyrics.

Related issue in the Debian bug tracker: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1012467

Missing libneon.a in 0.8.1

Hi,

Are we (ROOT) configuring / building davix incorrectly? We get

15:18:07  [7702/10606] Linking CXX shared library lib/libRDAVIX.so.6.27.01
15:18:07  FAILED: : && /usr/bin/c++ -fPIC -fdiagnostics-color=always -std=c++14 -pipe  -Wshadow -Wall -W -Woverloaded-virtual -fsigned-char -pthread -O3  -Wl,--no-undefined -Wl,--hash-style="both" -shared -Wl,-soname,libRDAVIX.so.6.27 -o lib/libRDAVIX.so.6.27.01 net/davix/CMakeFiles/G__RDAVIX.dir/G__RDAVIX.cxx.o net/davix/CMakeFiles/RDAVIX.dir/src/RRawFileDavix.cxx.o net/davix/CMakeFiles/RDAVIX.dir/src/TDavixFile.cxx.o net/davix/CMakeFiles/RDAVIX.dir/src/TDavixSystem.cxx.o  -Wl,-rpath,/mnt/build/workspace/root-pullrequests-build/build/lib:  lib/libNet.so.6.27.01  lib/libRIO.so.6.27.01  builtins/davix/DAVIX-prefix/lib/libdavix.a  builtins/davix/DAVIX-prefix/lib/libneon.a  -luuid  /usr/lib/x86_64-linux-gnu/libssl.so  /usr/lib/x86_64-linux-gnu/libcrypto.so  /usr/lib/x86_64-linux-gnu/libxml2.so  -ldl  lib/libThread.so.6.27.01  lib/libCore.so.6.27.01  -lpthread && :
15:18:07  c++: error: builtins/davix/DAVIX-prefix/lib/libneon.a: No such file or directory

That's coming from https://github.com/root-project/root/blob/master/builtins/davix/CMakeLists.txt#L19-L22

foreach(lib davix neon)
  set(libname ${CMAKE_STATIC_LIBRARY_PREFIX}${lib}${CMAKE_STATIC_LIBRARY_SUFFIX})
  list(APPEND DAVIX_LIBRARIES ${DAVIX_PREFIX}/lib/${libname})
endforeach()

Does Davix export CMake targets (Davix::Davix or similar) these days? I didn't find those.

IIUC, https://github.com/cern-fts/davix/blob/devel/src/CMakeLists.txt#L150 is relevant here and it "includes" curl, so we don't even need to link against neon anymore?

Thanks for any insights :-)

Axel

Uploading large files (streaming) to Google S3 fail (issue with multi-part)

Davix-libs 0.8.0-1, gfal2 2.20.0-1

Even though Google claims it's compatible with multi-part uploads via their S3 compatibility endpoints, it is not working:

$ gfal-copy -v -v -v testfile.4000M s3://burt-test-1.storage.googleapis.com/big5 yields a 403 error after uploading the first chunk with the following output:

xfer.log

It also fails with davix-libs 0.7.6-1 quickly with a different error (413).

I tried recompiling davix 0.7.6-1 and forcing https://github.com/cern-fts/davix/blob/devel/src/fileops/S3IO.cpp#L39 to return false; then the transfer succeeds.

CTest tests removed

Maybe they were removed by accident in commit 011b247?
Notice that include(CTest) line was removed from CMakeLists.txt in that commit.

davix-cp and davix-mkdir commands exit with 0 in case of errors

Hi guys,

there's a bug in error handling for these two utilities:

$ davix-cp -P grid https://a.example/a https://b.example/b
(Davix::HttpRequest) Error: copy Domain name resolution failed
$ echo $?
0
$ davix-mkdir -P grid https://a.example/a
(Davix::HttpRequest) Error: Domain name resolution failed
$ echo $?
0
$ davix-cp --version
Version: 0.6.9-

Thanks!
A.

Possibly wrong library path/pkg_arch on debian/ubuntu 64bit

Hi,

when compiling DAVIX on debian/ubuntu 64bit the logic to determine the library path suffix is not correct: It checks for the existence of /usr/lib64 but this directory does (usually) not exist on debian/ubuntu. As such PKG_ARCH is usually set to i386 but that doesn't seem to be a problem. However we do have some issues wen using davix with ROOT as the ROOT compilation cannot find the library in some cases.

The official GNUInstallDirs module from cmake determines the path differently by checking if /etc/debian_version exists and doing a different logic in that case. Would it be possible to adapt the same logic for davix? Or just use GNUInstallDirs?

Cheers,

Martin

Recursively deleting directories with davix-rm not working

Hi,
I'm trying to recursively delete a directory with davix-rm (as reported here https://www.systutorials.com/docs/linux/man/1-davix-rm/) but it does not seem to work.
This is the directory:

davix-ls -l --capath /etc/grid-security/certificates/ -H "Authorization: Bearer ${BEARER_TOKEN}" davs://xfer-archive.cr.cnaf.infn.it:8443/jlab12/fornariclas12
-rwxrwxrwx 0 9442 2021-05-19 09:24:14 certs.p12
-rwxrwxrwx 0 494 2021-05-19 09:24:37 oidc-agent-stat.sh

And this is what I see when I try to delete it:

davix-rm --capath /etc/grid-security/certificates/ -H "Authorization: Bearer ${BEARER_TOKEN}" davs://xfer-archive.cr.cnaf.infn.it:8443/jlab12/fornariclas12/
(Davix::rm) Error: HTTP 412 : Server Error with url davs://xfer-archive.cr.cnaf.infn.it:8443/jlab12/fornariclas12/

Anyways, gfal-rm -r seems to work on the same directory:

gfal-rm -r davs://xfer-archive.cr.cnaf.infn.it:8443/jlab12/fornariclas12 2>/dev/null
davs://xfer-archive.cr.cnaf.infn.it:8443/jlab12/fornariclas12/certs.p12 DELETED
davs://xfer-archive.cr.cnaf.infn.it:8443/jlab12/fornariclas12/oidc-agent-stat.sh DELETED
davs://xfer-archive.cr.cnaf.infn.it:8443/jlab12/fornariclas12 RMDIR

davix-ls -l --capath /etc/grid-security/certificates/ -H "Authorization: Bearer $AT" davs://xfer-archive.cr.cnaf.infn.it:8443/jlab12/fornariclas12
(Davix::HttpRequest) Error: HTTP 404 : File not found

Is it possible to make davix-rm behave the same way as gfal-rm -r with respect to directories?
For your information, these are the versions involved:

gfal-rm --version
gfal2-util version 1.5.3 (gfal2 2.18.1)
dcap-2.18.1
file-2.18.1
gridftp-2.18.1
http-2.18.1
lfc-2.18.1
rfio-2.18.1
sftp-2.18.1
srm-2.18.1
xrootd-2.18.1

davix-rm --version
Version: 0.7.5

Thanks for your attention.

Wrong format for downloaded files for release 0.8.4

The download link on the release page does not result in a file of the correct format.
If I download:

$ wget https://github.com/cern-fts/davix/releases/download/R_0_8_4/davix-0.8.4.tar.gz

The retrieved file is a zip file, not a gzipped tar archive.

$ file davix-0.8.4.tar.gz
davix-0.8.4.tar.gz: Zip archive data, at least v2.0 to extract, compression method=deflate

If I inflate this file

$ mkdir z
$ cd z
$ unzip ../davix-0.8.4.tar.gz
Archive: ../davix-0.8.4.tar.gz
inflating: davix-0.8.4.tar.gz
$ file davix-0.8.4.tar.gz
davix-0.8.4.tar.gz: gzip compressed data, last modified: Thu Feb 2 11:17:28 2023, from Unix, original size modulo 2^32 36597760

I get the correct file. So the link give a zipped gzipped tar archive instead of a gzipped archive.

The corresponding link for the previous release gives the right format.

$ wget https://github.com/cern-fts/davix/releases/download/R_0_8_3/davix-0.8.3.tar.gz

$ file davix-0.8.3.tar.gz
davix-0.8.3.tar.gz: gzip compressed data, last modified: Tue Sep 13 10:19:00 2022, from Unix, original size modulo 2^32 36587520

(The davix-0.8.4-el7.x86_64.tar.gz link also gives the wrong format.)

The tarball of the 0.7.4 release has suddenly changed

fetch: http://grid-deployment.web.cern.ch/grid-deployment/dms/lcgutil/tar/davix/0.7.4/davix-0.7.4.tar.gz: size mismatch: expected 4343614, actual 4340677

See the full log here: http://beefy5.nyi.freebsd.org/data/120i386-default/506094/logs/davix-0.7.4.log

What happened? Did somebody hack your server?

Davix static library crash on CentOS7

Environment:

• Davix version 0.3.6 with embedded externals and statically build.
• ROOT master build with -Dbuiltin_davix=ON
• CentOS7 with native compiler (gcc 4.8)

The traceback

#10 0x00007fd5daf680d3 in std::string::size() const () from /lib64/libstdc++.so.6
#11 0x00007fd5db51f3b7 in std::operator+<char, std::char_traits<char>, std::allocator<char> > (__lhs=0x7fd5cd0cdc3f " davix/", __rhs=<error reading variable: Cannot access memory at address 0xffffffffffffffe8>) at /usr/include/c++/4.8.2/bits/basic_string.tcc:700
#12 0x00007fd5ccf09ba8 in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535) at /mnt/build/jenkins/workspace/root-package/BUILDTYPE/Release/COMPILER/native/LABEL/cc7/VERSION/master/root/net/davix/src/TDavixFile.cxx:60
#13 0x00007fd5ccf09c78 in _GLOBAL__sub_I_TDavixFile.cxx(void) () at /mnt/build/jenkins/workspace/root-package/BUILDTYPE/Release/COMPILER/native/LABEL/cc7/VERSION/master/root/net/davix/src/TDavixFile.cxx:721
#14 0x00007fd5dbb6f4e3 in _dl_init_internal () from /lib64/ld-linux-x86-64.so.2
#15 0x00007fd5dbb73b04 in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
#16 0x00007fd5dbb6f2f4 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#17 0x00007fd5dbb7321b in _dl_open () from /lib64/ld-linux-x86-64.so.2
#18 0x00007fd5d9f9d02b in dlopen_doit () from /lib64/libdl.so.2
#19 0x00007fd5dbb6f2f4 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#20 0x00007fd5d9f9d62d in _dlerror_run () from /lib64/libdl.so.2
#21 0x00007fd5d9f9d0c1 in dlopen

The problem is that TDavixFile.cxx uses Davix::version() in an static initialization and is probably not initialized yet.

[RFE] command line tools --quiet flag

Another suggestion is to add a --quiet flag for displaying only error messages, and muting all progress messages.

configure fails: set_target_properties called with incorrect number of arguments.

CMake Error at src/CMakeLists.txt:40 (set_target_properties):
  set_target_properties called with incorrect number of arguments.

This is the offending instruction:

set_target_properties(libdavix PROPERTIES
                            VERSION ${VERSION_MAJOR}.${VERSION_MINOR}.${VERSION_PATCH}
                            SOVERSION ${VERSION_MAJOR}
                            CLEAN_DIRECT_OUTPUT 1
                            OUTPUT_NAME ${OUTPUT_NAME_DAVIX})

Regression in version 0.7.1
FreeBSD 11.2 amd64
cmake-3.12.3

how to use in wndows?

how to use in wndows?