irods / irods_client_icommands Goto Github PK

I'm trying to use the icommands in an HPC environment in which I have no admin privileges, which means I can't use .deb and .rpm distributions. Additionally, building the icommands requires a lot of build tools which I also can't install for the same reasons. Could you provide a binary distribution of the icommands where you just provide the binaries and any dynamic libraries that are necessary, in addition to the .deb and .rpm packages?

Dear all,

I am currently in the process of building the irods-icommands package on our i386 machines, we currently have around 640 i386 machine in our web archive cluster. I was wondering where I can find the irods-dev and irods-runtime source code to build the packages on our i386 systems. I have checked the irods repository and found the i386 packages empty.

Thank you in advance for your help.

This is related to issue #21.

I created a directory (test) with files:

file1.txt, file2.txt, file3.txt, file4.txt

In addition to these, I created a symbolic link dne.txt that point to a non-existent file.

When I run irsync -rv --link /home/jjames/test i:test, I get a USER_INPUT_PATH_ERR due to the invalid symbolic link.

This is unexpected since the "--link" option is supposed to ignore soft links. Other than error itself, this is causing other files to not be synced (see issue #21).

Note: If the symbolic link is good, the symbolic link file is not added to iRODS so that part works.

Could the iRODS icommands package be configured to support the relocatable option in the RPM package?

This describes the option:
http://www.rpm.org/max-rpm/ch-rpm-reloc.html

Dear all,
when I try to remove a file in a nested directory with a very long tree name, I obtain the error

ERROR: main: parseCmdLinePath error. status = -346000 USER_PATH_EXCEEDS_MAX

I'm using icommand version 4.2.0-1
How can I solve?

Best regards. Giusy

Hi, glad to meet you again.
I followed (this) [https://docs.irods.org/4.2.3/getting_started/installation/#non-package-install-run-in-place] documentation, in 'Non-Package Install (Run In Place)' section.

user@hostname:~ /irods/ $ export IRODS_INSTALL_DIR=/path/to/the/non-package-root
user@hostname:~ /irods/ $ mkdir build
user@hostname:~ /irods/build $ cmake -DCMAKE_INSTALL_PREFIX=$IRODS_INSTALL_DIR -DIRODS_EXTERNALS_PACKAGE_ROOT=$IRODS_EXTERNALS ../
user@hostname:~ /irods/build $ make non-package-install-postgres

I've done it up to here. But There is no /usr in the $IRODS_INSTALL_DIR.

user@hostname:~ /irods_client_icommands/build $ cmake -DCMAKE_INSTALL_PREFIX=$IRODS_INSTALL_DIR -DIRODS_DIR=$IRODS_INSTALL_DIR/usr/lib/irods/cmake ../
user@hostname:~ /irods_client_icommands/build $ make install
user@hostname:~ /irods_client_icommands/build $ export LD_LIBRARY_PATH=$IRODS_INSTALL_DIR/usr/lib
user@hostname:~ /irods_client_icommands/build $ export PATH=$IRODS_INSTALL_DIR/usr/bin:$IRODS_INSTALL_DIR/usr/sbin:$PATH

So, /usr was removed and the execution was completed.

user@hostname:/var/lib/irods $ python ./scripts/setup_irods.py

However, an error occurs here.
Here is my error message.

syslab@syslab:~/workspace_chc/install_irods/var/lib/irods$ sudo python ./scripts/setup_irods.py
[sudo] password for syslab:
Warning: Hostname `syslab` should be a fully qualified domain name.
Error encountered running setup_irods:
Traceback (most recent call last):
  File "./scripts/setup_irods.py", line 447, in main
    test_mode=options.test_mode)
  File "./scripts/setup_irods.py", line 68, in setup_server
    check_hostname()
  File "./scripts/setup_irods.py", line 168, in check_hostname
    if not irods.lib.hostname_resolves_to_local_address(hostname):
  File "/home/syslab/workspace_chc/install_irods/var/lib/irods/scripts/irods/lib.py", line 386, in hostname_resolves_to_local_address
    hostname])
  File "/home/syslab/workspace_chc/install_irods/var/lib/irods/scripts/irods/execute.py", line 89, in execute_command_permissive
    p = execute_command_nonblocking(args, stdout=stdout, stderr=stderr, **kwargs)
  File "/home/syslab/workspace_chc/install_irods/var/lib/irods/scripts/irods/execute.py", line 64, in execute_command_nonblocking
    sys.exc_info()[2])
  File "/home/syslab/workspace_chc/install_irods/var/lib/irods/scripts/irods/execute.py", line 54, in execute_command_nonblocking
    return subprocess.Popen(args, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
IrodsError: Call to open process with ['/home/syslab/workspace_chc/install_irods/usr/sbin/hostname_resolves_to_local_address', 'syslab'] failed:
  Could not find the requested executable '/home/syslab/workspace_chc/install_irods/usr/sbin/hostname_resolves_to_local_address'; please ensure '/home/syslab/workspace_chc/install_irods/usr/sbin/hostname_resolves_to_local_address' is installed and in the path.

Thank you for reading it. I look forward to hearing from you.

Can I use multi threads when I run iput?

I want to change the number of threads during iput execution, what should I do?

iput -N 32 data

Is this the right way to use it?

I wonder if you currently support multi-threads.

iinit's help text needs to be updated to warn users that the password will be in clear-text.

As irods 4.2.0 has been released, could you also release irods_client_icommands 4.2.0 ?

In many places imeta detects an error due to missing arguments but does not return immediately which leads to uncaught exceptions. The code should return with SYS_INVALID_INPUT in these cases.

Examples:

imeta ls -d
imeta ls -C
imeta mod -d
imeta mod -d file.dat
imeta rm -d
etc.

This is related to #18 but is meant to be more general to cover all of these cases.

hi @korydraughn
after finished in irods server
I met another problem,
error message showed as below, is that matter with something update? or it just not finished yet.
Thank you

error message:
[ 73%] Building CXX object CMakeFiles/imeta.dir/src/imeta.cpp.o
/root/irods_client_icommands/src/imeta.cpp:1000:38: error: no member named 'condInput' in 'modAVUMetadataInp_t'
addKeyVal(&modAVUMetadataInp.condInput, ADMIN_KW, "");
~~~~~~~~~~~~~~~~~ ^
/root/irods_client_icommands/src/imeta.cpp:1093:38: error: no member named 'condInput' in 'modAVUMetadataInp_t'
addKeyVal(&modAVUMetadataInp.condInput, ADMIN_KW, "");
~~~~~~~~~~~~~~~~~ ^
2 errors generated.
make[2]: *** [CMakeFiles/imeta.dir/build.make:76: CMakeFiles/imeta.dir/src/imeta.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:2191: CMakeFiles/imeta.dir/all] Error 2
make: *** [Makefile:156: all] Error 2

We are facing a similar problem as reported in the closed issue #22.
When we use irsync with the option --link, soft links are ignored except when they are broken.

What did you try to do?

A user wants to upload a directory, which contains (broken) symlinks.
I wanted to check whether we could exclude symlinks with the --link option

What did you expect to happen?

When you upload a symlink with irsync --link i:, it is ignored, no matter whether the symlink works or not.

The main reason why some users were interested in this option in the past is because they have potentially broken symlinks, and would like to avoid errors when syncing large collections.

Note: I'm a bit confused whether this is the intended behaviour or not.
Based on #22, a broken symlink should be ignored, not throwing an error.
However, the thread which it refers to as solution (irods/irods#3988) seems to imply the error is intended behaviour.

What was the actual result?

Working symlinks are ignored, but broken symlinks throw the following error:

remote addresses: 172.23.253.137 ERROR: resolveRodsTarget: source brokenlink does not exist
remote addresses: 172.23.253.137 ERROR: rsyncUtil: resolveRodsTarget status = -317000 USER_INPUT_PATH_ERR

If you upload a directory with a broken symlink in it, the error is as follows:

Running recursive pre-scan... remote addresses: 172.23.253.137 ERROR: USER_INPUT_PATH_ERR: No such file or directory
Path = test_symlinks/brokenlink

pre-scan complete... errors found.
Aborting data transfer.

This causes other files in the directory to not be synced.

Minimal reproducable example

# create two files
echo "hello" > reference1
echo "hello" > reference2

# create soft links
ln -s reference1 workinglink
ln -s reference2 brokenlink

# remove the second file, creating a broken link
rm reference2

# execute irsync --link on good link
# nothing happens, no error occurs
irsync --link workinglink i:workinglink

# execute irsync --link on good link
# will throw a USER_INPUT_PATH_ERR error
irsync --link brokenlink i:brokenlink

Environment details

I tested this on the following setups:

• iRODS 4.3.1 with iCommands 4.3.1 and a Mysql database (production environment)
• iRODS 4.3.1 with iCommands 4.3.1 and a Mysql database (test environment)
• iRODS 4.3.1 with iCommands 4.3.1 and postgres database (test environment)

To be clear, I'm not intending on syncing anything in the holiday period :)
But I wanted to report it before losing track of it.
If I have anything wrong, just let me know.

Best regards and happy holidays,
Jef

imeta mod throws error when modifying metadata (icommands v 4.2.1 or 4.2.2):

imeta mod -d test.txt KEY value v:myValue
remote addresses: irods-server ERROR: rcModAVUMetadata failed with error -816000 CAT_INVALID_ARGUMENT

while with icommands v 4.1.10 and higher the command works as expected.

Due to this bug irods-contrib/metalnx-web#234 I got some AVU entries in ICAT which I was not able to delete with imeta rm ..., iRODS complained about empty string being invalid, I was able to delete it with "imeta rmi...", but I do not see an obvious way to obtain the metadata ID with the icommands (I got that info with a little python-irodsclient script).

I suggest to print the metadata ID with imeta ls -V...

By the way, imeta ls -d file, imeta ls -d -v file and imeta ls -d -V file print the exact same output in my test environment which leaves me somewhat confused about the point of the options -v and -V.

$ iput VERSION.json
$ imeta rm -d VERSION.json a

Error: Not enough arguments provided to rm

terminating with uncaught exception of type boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::bad_any_cast> >: boost::bad_any_cast: failed conversion using boost::any_cast
Aborted (core dumped)

The existence of the "value" argument is checked for here, but it is then unconditionally used here.

After brief inspection, should probably return early with something like SYS_INVALID_INPUT_PARAM.

This error is repeated throughout the file.

Currently, iadmin moduser makes use of the challenge string provided by the server and the ObfEncodeByKeyV2 function to encode passwords. This challenge string is generated on a per-server basis, so when the password request change is initiated forwarded from a resource server to a catalog server to actually decode, scramble, and store the new password in the database, the decode fails.

This would be an acceptable limitation if the interface required it. However, the generalAdmin interface passes what it's sent on to the database plugin in the exact same way that the interface ipasswd uses (userAdmin) does, and ipasswd does not make use of or require a challenge, and, indeed, works when connected to a catalog consumer server.

This is because the database plugin's mod user function will decode passwords WITH OR WITHOUT a challenge encoded in them, based on whether or not the challenge was encoded. Therefore, we can just encode passwords for iadmin moduser in the same way we do for ipasswd. Because they are still encoded using a known password, they are still secure to the same degree any API call made by ipasswd is.

I'm currently trying to create a NIX package of the irods-icommands to provide an easy way to deploy them to hundreds of heterogeneous computing nodes.
As I failed with irods 4.1.8, I was wondering if the new build system of irods 4.2, which seems to be based on cmake, can help me.
I'm facing a first issue:

bzizou@killeen:~/git/irods_client_icommands/build$` cmake ..                 
CMake Error at CMakeLists.txt:4 (find_package):                             
  By not providing "FindIRODS.cmake" in CMAKE_MODULE_PATH this project has  
  asked CMake to find a package configuration file provided by "IRODS", but 
  CMake did not find one.                                                   

  Could not find a package configuration file provided by "IRODS" (requested
  version 4.2.0) with any of the following names:                           

    IRODSConfig.cmake                                                       
    irods-config.cmake                                                      

  Add the installation prefix of "IRODS" to CMAKE_PREFIX_PATH or set        
  "IRODS_DIR" to a directory containing one of the above files.  If "IRODS" 
  provides a separate development package or SDK, be sure it has been       
  installed.                                                                


-- Configuring incomplete, errors occurred!                                 

Is there any developer's documentation? I would be very glad to contribute, I think that having Nix packages for the irods clients could be a great thing to bring irods to any kind of unix client hosts.

Which commit on the 4-2-stable branch is the correct one to build iRODS client compatible with iRODS 4.2.7? The HEAD doesn't appear to be compatible any more. There are a number of commits that work, but it's not clear which is preferred, or correct.

Please consider tagging 4-2-stable with the corresponding release number. It would be very helpful when reproducing a build.

Thanks.

I created a directory (test) with files:

file1.txt, file2.txt, file3.txt, file4.txt

In addition to these, I created a symbolic link dne.txt that point to a non-existent file.

When I run irsync -rv /home/jjames/test i:test, I get a USER_INPUT_PATH_ERR due to the invalid symbolic link. That is good. However, the other files within /home/jjames/test do not get synced. Sometimes some of the files will be synced but not all. In this case none of them did.

Note: If this is recursive and there are other higher level directories being synced, the irsync will continue on those. If it is to continue elsewhere it should continue with all files in the current directory as well.

Hello, i've installed icommands on multiple machines and version 4.3.0 fails to run with pam module.

Error occurred while authenticating user [dvo0012] [PLUGIN_ERROR_MISSING_SHARED_OBJECT: [-]     /usr/include/irods/irods_load_plugin.hpp:157:irods::error irods::load_plugin(PluginType *&, const std::string &, const std::string &, const std::string &, const Ts &...) [PluginType = irods::experimental::auth::authentication_base, Ts = <char [14]>] :  status [PLUGIN_ERROR_MISSING_SHARED_OBJECT]  errno [] -- message [shared library does not exist [/usr/lib/irods/plugins/auth/libirods_auth_plugin-pam_client.so]]



] [ec=-1827000] failed with error -1827000 PLUGIN_ERROR_MISSING_SHARED_OBJECT

icommands 4.2.7 works correctly. Please can you look at it?

We would like to test 4.3.1. And for this, we have a test environment running irods 4.3.1, iCommands 4.3.1(main) on almalinux8.

I wanted to test sub-commands/features done through the commits available for irods/irods#7274. Also I am seeing some documents https://docs.irods.org/4.3.1/system_overview/configuration/#configuring-authentication-in-r_grid_configuration and here https://github.com/irods/irods_client_icommands/blob/main/src/iadmin.cpp#L2672-L2692. However, I dont see how to use these subcommands in iadmin -h, they seem not exist.

But if I run these sub commands, I am getting:

[irods@dev-u0137480 irods]$ iadmin get_grid_configuration authentication password_max_time
Failed to get grid configuration for namespace [authentication] and option [password_max_time] [ec=-808000]

Also R_GRID_CONFIGURATION table (alanking/irods_docs@21fbfd0#diff-36178197e1823e07e5432ebea993d7e94af368f2b7160359ee4c61b3f8cddcc0R890) doesnt have more than:

MySQL [irods]> select * from R_GRID_CONFIGURATION;
+--------------+----------------+---------------------------------------+
| namespace    | option_name    | option_value                          |
+--------------+----------------+---------------------------------------+
| database     | schema_version | 11                                    |
| delay_server | leader         | dev-u0137480.irods.t.icts.kuleuven.be |
| delay_server | successor      |                                       |
+--------------+----------------+---------------------------------------+
3 rows in set (0.001 sec)

This repo doesnt have 4.3.1 release for iCommands yet. Am I missing something?

document this in the imeta help text

crib logic from https://github.com/irods/irods/blob/4-1-stable/packaging/build.sh#L1555

In version 4.2.8, of iCommands, ihelp doesn't mention the new iCommand iunreg.

When using the pam(_password) method for authentication in irods, it is expected that .irods/irods_environment.json contains a value for irods_authentication_scheme that indicates pam is needed for logging in.

Up to version 4.2.11, the value that should be specified, was "PAM". In 4.3.0 this apparently has changed to "pam_password" (see #314), leading us to include a line

iinit -h | grep Version | grep -v -q 4.3.0 || sed -i 's/"irods_authentication_scheme": "PAM"/"irods_authentication_scheme": "pam_password"/' ~/.irods/irods_environment.json

in the snippet we distribute to our users for authentication.

However, as it turns out, the deb release of 4.3.0 for focal https://packages.irods.org/apt/, still expects "PAM", while the rpm release of 4.3.0 on https://packages.irods.org/yum/ now expects "pam_password". I.e. the above version check is not enough.

Functionality would be the same as Unix rmdir.

Requested in irods/irods#3117.

I have iRODS 4.3.0 installed and found the bash completion script here. After sourcing it, it kind of works, but it seems to always think I'm in my home directory. For example:

$ ipwd <enter>
/iplant/home/chandles
$ ils <enter>
/iplant/home/chandles:
  C- /iplant/home/chandles/analyses
$ ils <tab>(autocompletes to:)
$ ils analyses/<tab><tab>(lists possible options:)
analyses/Cloud_Shell_analysis1-2022-03-11-17-53-14.3/  analyses/Datahog_analysis1-2022-09-06-18-51-07.8/      analyses/Datahog_analysis1-2022-09-14-21-51-41.6/

$ ils analyses/ <enter>
/iplant/home/chandles/analyses:
  C- /iplant/home/chandles/analyses/Cloud_Shell_analysis1-2022-03-11-17-53-14.3
  C- /iplant/home/chandles/analyses/Datahog_analysis1-2022-09-06-18-51-07.8
  C- /iplant/home/chandles/analyses/Datahog_analysis1-2022-09-14-21-51-41.6
$ 

So it worked ok there, but if i icd into analyses directory it still think I'm in my home dir:

$ icd <enter>
$ icd <tab>(autocompletes to:)
$ icd ./analyses/ <enter>
$ icd <tab>(autocompletes to:)
$ icd analyses/ <enter>
No such directory (collection): /iplant/home/chandles/analyses/analyses
$ 

If I try to see above, it doesn't do anything:

$ ils ../<tab><tab>(nothing happens)

Actually it gets hung here and I have to ^C. Further, if I icd into our shared community data, it still thinks i'm in my home directory:

$ ils ../^C^C
$ ^C
$ ^C
$ ipwd
/iplant/home/chandles/analyses
$ icd ../../shared/agi_data <enter>
$ icd <tab>(autocompletes to:)
$ icd analyses/ <enter>
No such directory (collection): /iplant/home/shared/agi_data/analyses
$ ils <enter>
/iplant/home/shared/agi_data:
  C- /iplant/home/shared/agi_data/Asian_rice_pangenome_annotations
  C- /iplant/home/shared/agi_data/Dr_Lorieux_Data
  C- /iplant/home/shared/agi_data/HiFI_DATA
  C- /iplant/home/shared/agi_data/KAUST
$ ils H<tab>(nothing happens)

Hi Foks,

I noticed that when this command runs and completes successfully, it gives an exit code that is greater than 1 and I was wondering what the significance was.

$ ifsck -K -r /irods-seq-i23-bc/
$ echo $?
192

• master
• 4-2-stable

Since I upgrated to 4.2.0, I am facing following error:

• The input line in the .r rule file specifies arguments as following
  INPUT *filePath=$, *fileSize=$, *mainResource=$

• and the rule is called according to irule's help page:
  irule -s -F customPut.r *filePath=/my/path *fileSize=3125 *mainResource=myResc

Still that, supplied values are not available during execution

status = -1211000 RE_UNABLE_TO_READ_LOCAL_VAR
Level 0: DEBUG: error: unable to read local variable *filePath
line 2, col 20

E.g. https://github.com/irods/irods_client_icommands/blob/4.2.1/src/imeta.cpp#L1446

This error is repeated throughout the file.

I was able to install version 4.3.0 of iCommands on a CentOS 8 machine by following the instructions on https://packages.irods.org and doing yum install irods-icommands. Unfortunately, the dependency irods-runtime wasn't automatically installed, and I had to install it with a second yum call, yum install irods-runtime.