Genders is a static cluster configuration database used for cluster configuration management. It is used by a variety of tools and scripts for management of large clusters.
License: GNU General Public License v2.0
Sometimes it might be useful to have hostnames with domain parts in the genders file (say, a naming convention like nodeN.rackM.someDomain). But currently nodeattr -k at least complains about that.
Would it be possible to allow that?
This is an enhancement request.
It would be helpful to have an option to provide a union of all genders across a list of hosts. For instance whereas nodeattr -l <host> lists genders from one host, you could enhance the option to print a list of unique genders across the all hosts in the list.
In version 1-27-1
/bin/bash ../../libtool --tag=CC --mode=link gcc -I../../config -Os -pipe -march=native -mfpmath=sse -fopenmp -pthread -fno-unroll-all-loops -fno-unroll-loops -fno-peel-loops -Wformat -Wformat-security -Warray-bounds -Wclobbered -Wvolatile-register-var -Wdate-time -Os -pipe -march=native -mfpmath=sse -fopenmp -pthread -fno-unroll-all-loops -fno-unroll-loops -fno-peel-loops -Wformat -Wformat-security -Warray-bounds -Wclobbered -Wvolatile-register-var -Wdate-time -floop-interchange -ftree-loop-distribute-patterns -fpredictive-commoning -ftree-loop-distribution -fsplit-paths -ftree-vectorize -ftree-partial-pre -fgcse-after-reload -fivopts -Wimplicit-function-declaration -Wno-error=implicit-function-declaration -Wl,-O1 -Wl,--as-needed -Wl,-fuse-ld=gold -lpthread -lgomp -Wl,-O1 -Wl,--as-needed -Wl,-O1 -Wl,--as-needed -Wl,-fuse-ld=gold -lpthread -lgomp -o libcommon.la libcommon_la-hash.lo libcommon_la-hostlist.lo libcommon_la-list.lo libcommon_la-thread.lo
libtool: link: ar cru .libs/libcommon.a .libs/libcommon_la-hash.o .libs/libcommon_la-hostlist.o .libs/libcommon_la-list.o .libs/libcommon_la-thread.o
libtool: link: ranlib .libs/libcommon.a
libtool: link: ( cd ".libs" && rm -f "libcommon.la" && ln -s "../libcommon.la" "libcommon.la" )
make[2]: Leaving directory '/var/tmp/portage/sys-cluster/genders-1.27.1/work/genders-genders-1-27-1/src/libcommon'
Making all in libgenders
make[2]: Entering directory '/var/tmp/portage/sys-cluster/genders-1.27.1/work/genders-genders-1-27-1/src/libgenders'
bison -y -d -b genders_query ./genders_query.y
make[2]: *** No rule to make target 'genders_query.tab.c', needed by 'all'. Stop.
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory '/var/tmp/portage/sys-cluster/genders-1.27.1/work/genders-genders-1-27-1/src/libgenders'
make[1]: *** [Makefile:380: all-recursive] Error 1
make[1]: Leaving directory '/var/tmp/portage/sys-cluster/genders-1.27.1/work/genders-genders-1-27-1/src'
make: *** [Makefile:439: all-recursive] Error 1
configured with --with-java-extensions
Making all in java
make[3]: Entering directory '/var/tmp/portage/sys-cluster/genders-9999/work/genders-9999/src/extensions/java'
/bin/bash ../../../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I../../../config -Os -pipe -march=native -mfpmath=sse -fopenmp -pthread -fno-unroll-all-loops -fno-unroll-loops -fno-peel-loops -Wformat -Wformat-security -Warray-bounds -Wclobbered -Wvolatile-register-var -Wdate-time -Os -pipe -march=native -mfpmath=sse -fopenmp -pthread -fno-unroll-all-loops -fno-unroll-loops -fno-peel-loops -Wformat -Wformat-security -Warray-bounds -Wclobbered -Wvolatile-register-var -Wdate-time -floop-interchange -ftree-loop-distribute-patterns -fpredictive-commoning -ftree-loop-distribution -fsplit-paths -ftree-vectorize -ftree-partial-pre -fgcse-after-reload -fivopts -Wimplicit-function-declaration -Wno-error=implicit-function-declaration -Wl,-O1 -Wl,--as-needed -Wl,-fuse-ld=gold -lpthread -lgomp -I/usr/lib64/icedtea8/include -I/usr/lib64/icedtea8/include/linux -c -o Gendersjni.lo `test -f 'src/Gendersjni.c' || echo './'`src/Gendersjni.c
libtool: compile: gcc -DHAVE_CONFIG_H -I. -I../../../config -Os -pipe -march=native -mfpmath=sse -fopenmp -pthread -fno-unroll-all-loops -fno-unroll-loops -fno-peel-loops -Wformat -Wformat-security -Warray-bounds -Wclobbered -Wvolatile-register-var -Wdate-time -Os -pipe -march=native -mfpmath=sse -fopenmp -pthread -fno-unroll-all-loops -fno-unroll-loops -fno-peel-loops -Wformat -Wformat-security -Warray-bounds -Wclobbered -Wvolatile-register-var -Wdate-time -floop-interchange -ftree-loop-distribute-patterns -fpredictive-commoning -ftree-loop-distribution -fsplit-paths -ftree-vectorize -ftree-partial-pre -fgcse-after-reload -fivopts -Wimplicit-function-declaration -Wno-error=implicit-function-declaration -Wl,-O1 -Wl,--as-needed -Wl,-fuse-ld=gold -lpthread -lgomp -I/usr/lib64/icedtea8/include -I/usr/lib64/icedtea8/include/linux -c src/Gendersjni.c -fPIC -DPIC -o .libs/Gendersjni.o
src/Gendersjni.c:8:10: fatal error: genders.h: No such file or directory
8 | #include <genders.h>
| ^~~~~~~~~~~
compilation terminated.
make[3]: *** [Makefile:497: Gendersjni.lo] Error 1
make[3]: Leaving directory '/var/tmp/portage/sys-cluster/genders-9999/work/genders-9999/src/extensions/java'
make[2]: *** [Makefile:380: all-recursive] Error 1
make[2]: Leaving directory '/var/tmp/portage/sys-cluster/genders-9999/work/genders-9999/src/extensions'
make[1]: *** [Makefile:380: all-recursive] Error 1
make[1]: Leaving directory '/var/tmp/portage/sys-cluster/genders-9999/work/genders-9999/src'
make: *** [Makefile:439: all-recursive] Error 1
configured with --with-java-extensions
make[4]: Leaving directory '/var/tmp/portage/sys-cluster/genders-9999/work/genders-9999/src/libgenders'
/bin/bash ../../../libtool --tag=CC --mode=link gcc -Os -pipe -march=native -mfpmath=sse -fopenmp -pthread -fno-unroll-all-loops -fno-unroll-loops -fno-peel-loops -Wformat -Wformat-security -Warray-bounds -Wclobbered -Wvolatile-regis
ter-var -Wdate-time -Os -pipe -march=native -mfpmath=sse -fopenmp -pthread -fno-unroll-all-loops -fno-unroll-loops -fno-peel-loops -Wformat -Wformat-security -Warray-bounds -Wclobbered -Wvolatile-register-var -Wdate-time -floop-interchan
ge -ftree-loop-distribute-patterns -fpredictive-commoning -ftree-loop-distribution -fsplit-paths -ftree-vectorize -ftree-partial-pre -fgcse-after-reload -fivopts -Wimplicit-function-declaration -Wno-error=implicit-function-declaration -W
l,-O1 -Wl,--as-needed -Wl,-fuse-ld=gold -lpthread -lgomp -I/usr/lib64/icedtea8/include -I/usr/lib64/icedtea8/include/linux -I./../../libgenders/ -version-info 0:0:0 -Wl,-O1 -Wl,--as-needed -Wl,-O1 -Wl,--as-needed -Wl,-fuse-ld=gold -lpth
read -lgomp -o src/libGendersjni.la -rpath /usr/lib64 Gendersjni.lo ../../libgenders/libgenders.la
libtool: link: gcc -shared -fPIC -DPIC .libs/Gendersjni.o -Wl,-rpath -Wl,/var/tmp/portage/sys-cluster/genders-9999/work/genders-9999/src/libgenders/.libs ../../libgenders/.libs/libgenders.so -lpthread -lgomp -Os -march=native -mfpma
th=sse -fopenmp -pthread -Os -march=native -mfpmath=sse -fopenmp -pthread -Wl,-O1 -Wl,--as-needed -Wl,-fuse-ld=gold -Wl,-O1 -Wl,--as-needed -Wl,-O1 -Wl,--as-needed -Wl,-fuse-ld=gold -fopenmp -pthread -Wl,-soname -Wl,libGendersjni.so.0
-o src/.libs/libGendersjni.so.0.0.0
libtool: link: (cd "src/.libs" && rm -f "libGendersjni.so.0" && ln -s "libGendersjni.so.0.0.0" "libGendersjni.so.0")
libtool: link: (cd "src/.libs" && rm -f "libGendersjni.so" && ln -s "libGendersjni.so.0.0.0" "libGendersjni.so")
libtool: link: ( cd "src/.libs" && rm -f "libGendersjni.la" && ln -s "../libGendersjni.la" "libGendersjni.la" )
/usr/lib64/icedtea8/bin/javac ./gov/llnl/lc/chaos/Genders.java ./gov/llnl/lc/chaos/GendersException.java ./gov/llnl/lc/chaos/GendersExceptionOpen.java ./gov/llnl/lc/chaos/GendersExceptionRead.java ./gov/llnl/lc/chaos/GendersExceptionPars
e.java ./gov/llnl/lc/chaos/GendersExceptionParameters.java ./gov/llnl/lc/chaos/GendersExceptionNotfound.java ./gov/llnl/lc/chaos/GendersExceptionSyntax.java ./gov/llnl/lc/chaos/GendersExceptionInternal.java
/usr/lib64/icedtea8/bin/jar cvf ./gov/llnl/lc/chaos/Genders.jar ./gov/llnl/lc/chaos/*.class
added manifest
adding: gov/llnl/lc/chaos/Genders.class(in = 1583) (out= 607)(deflated 61%)
adding: gov/llnl/lc/chaos/GendersException.class(in = 250) (out= 194)(deflated 22%)
adding: gov/llnl/lc/chaos/GendersExceptionInternal.class(in = 281) (out= 203)(deflated 27%)
adding: gov/llnl/lc/chaos/GendersExceptionNotfound.class(in = 281) (out= 203)(deflated 27%)
adding: gov/llnl/lc/chaos/GendersExceptionOpen.class(in = 273) (out= 199)(deflated 27%)
adding: gov/llnl/lc/chaos/GendersExceptionParameters.class(in = 285) (out= 204)(deflated 28%)
adding: gov/llnl/lc/chaos/GendersExceptionParse.class(in = 275) (out= 200)(deflated 27%)
adding: gov/llnl/lc/chaos/GendersExceptionRead.class(in = 273) (out= 199)(deflated 27%)
adding: gov/llnl/lc/chaos/GendersExceptionSyntax.class(in = 277) (out= 201)(deflated 27%)
/usr/lib64/icedtea8/bin/javadoc -d javadoc ./gov/llnl/lc/chaos/Genders.java ./gov/llnl/lc/chaos/GendersException.java ./gov/llnl/lc/chaos/GendersExceptionOpen.java ./gov/llnl/lc/chaos/GendersExceptionRead.java ./gov/llnl/lc/chaos/Genders
ExceptionParse.java ./gov/llnl/lc/chaos/GendersExceptionParameters.java ./gov/llnl/lc/chaos/GendersExceptionNotfound.java ./gov/llnl/lc/chaos/GendersExceptionSyntax.java ./gov/llnl/lc/chaos/GendersExceptionInternal.java
Loading source file ./gov/llnl/lc/chaos/Genders.java...
Loading source file ./gov/llnl/lc/chaos/GendersException.java...
Loading source file ./gov/llnl/lc/chaos/GendersExceptionOpen.java...
Loading source file ./gov/llnl/lc/chaos/GendersExceptionRead.java...
Loading source file ./gov/llnl/lc/chaos/GendersExceptionParse.java...
Loading source file ./gov/llnl/lc/chaos/GendersExceptionParameters.java...
Loading source file ./gov/llnl/lc/chaos/GendersExceptionNotfound.java...
Loading source file ./gov/llnl/lc/chaos/GendersExceptionSyntax.java...
Loading source file ./gov/llnl/lc/chaos/GendersExceptionInternal.java...
Constructing Javadoc information...
Creating destination directory: "javadoc/"
Standard Doclet version 1.8.0_222
Building tree for all the packages and classes...
Generating javadoc/gov/llnl/lc/chaos/Genders.html...
./gov/llnl/lc/chaos/Genders.java:58: warning: no @return
public native int getnumnodes() throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:65: warning: no @return
public native int getnumattrs() throws GendersException;
./gov/llnl/lc/chaos/Genders.java:65: warning: no @return
public native int getnumattrs() throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:73: warning: no @return
public native int getmaxattrs() throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:81: warning: no @return
public native String getnodename() throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:88: warning: no @return
public native String[] getnodes() throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:95: warning: no @param for attr
public native String[] getnodes(String attr) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:95: warning: no @return
public native String[] getnodes(String attr) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:102: warning: no @param for attr
public native String[] getnodes(String attr, String val) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:102: warning: no @param for val
public native String[] getnodes(String attr, String val) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:102: warning: no @return
public native String[] getnodes(String attr, String val) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:109: warning: no @return
public native String[] getattr() throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:116: warning: no @param for node
public native String[] getattr(String node) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:116: warning: no @return
public native String[] getattr(String node) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:123: warning: no @return
public native String[] getattr_all() throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:131: warning: no @param for attr
public native String getattrval(String attr) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:131: warning: no @return
public native String getattrval(String attr) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:140: warning: no @param for node
public native String getattrval(String node, String attr) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:140: warning: no @param for attr
public native String getattrval(String node, String attr) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:140: warning: no @return
public native String getattrval(String node, String attr) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:147: warning: no @param for attr
public native boolean testattr(String attr) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:147: warning: no @return
public native boolean testattr(String attr) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:154: warning: no @param for node
public native boolean testattr(String node, String attr) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:154: warning: no @param for attr
public native boolean testattr(String node, String attr) throws GendersException;
^
public native boolean testattr(String node, String attr) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:154: warning: no @return
public native boolean testattr(String node, String attr) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:161: warning: no @param for attr
public native boolean testattrval(String attr, String val) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:161: warning: no @param for val
public native boolean testattrval(String attr, String val) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:161: warning: no @return
public native boolean testattrval(String attr, String val) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:168: warning: no @param for node
public native boolean testattrval(String node, String attr, String val) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:168: warning: no @param for attr
public native boolean testattrval(String node, String attr, String val) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:168: warning: no @param for val
public native boolean testattrval(String node, String attr, String val) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:168: warning: no @return
public native boolean testattrval(String node, String attr, String val) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:175: warning: no @param for node
public native boolean isnode(String node) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:175: warning: no @return
public native boolean isnode(String node) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:182: warning: no @param for attr
public native boolean isattr(String attr) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:182: warning: no @return
public native boolean isattr(String attr) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:189: warning: no @param for attr
public native boolean isattrval(String attr, String val) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:189: warning: no @param for val
public native boolean isattrval(String attr, String val) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:189: warning: no @return
public native boolean isattrval(String attr, String val) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:193: error: bad HTML entity
* with '||', intersection with '&&', * difference with '--', and
^
./gov/llnl/lc/chaos/Genders.java:193: error: bad HTML entity
* with '||', intersection with '&&', * difference with '--', and
^
./gov/llnl/lc/chaos/Genders.java:200: warning: no @param for query
public native String[] query(String query) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:200: warning: no @return
public native String[] query(String query) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:207: warning: no @param for query
public native boolean testquery(String query) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:207: warning: no @return
public native boolean testquery(String query) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:214: warning: no @param for node
^
./gov/llnl/lc/chaos/Genders.java:214: warning: no @param for node
public native boolean testquery(String node, String query) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:214: warning: no @param for query
public native boolean testquery(String node, String query) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:214: warning: no @return
public native boolean testquery(String node, String query) throws GendersException;
^
./gov/llnl/lc/chaos/Genders.java:40: warning: no @throws for gov.llnl.lc.chaos.GendersException
public Genders() throws GendersException
^
./gov/llnl/lc/chaos/Genders.java:48: warning: no @param for filename
public Genders(String filename) throws GendersException
^
./gov/llnl/lc/chaos/Genders.java:48: warning: no @throws for gov.llnl.lc.chaos.GendersException
public Genders(String filename) throws GendersException
^
Generating javadoc/gov/llnl/lc/chaos/GendersException.html...
Generating javadoc/gov/llnl/lc/chaos/GendersExceptionInternal.html...
Generating javadoc/gov/llnl/lc/chaos/GendersExceptionNotfound.html...
Generating javadoc/gov/llnl/lc/chaos/GendersExceptionOpen.html...
Generating javadoc/gov/llnl/lc/chaos/GendersExceptionParameters.html...
Generating javadoc/gov/llnl/lc/chaos/GendersExceptionParse.html...
Generating javadoc/gov/llnl/lc/chaos/GendersExceptionRead.html...
Generating javadoc/gov/llnl/lc/chaos/GendersExceptionSyntax.html...
Generating javadoc/gov/llnl/lc/chaos/package-frame.html...
Generating javadoc/gov/llnl/lc/chaos/package-summary.html...
Generating javadoc/gov/llnl/lc/chaos/package-tree.html...
Generating javadoc/constant-values.html...
Generating javadoc/serialized-form.html...
Building index for all the packages and classes...
Generating javadoc/overview-tree.html...
Generating javadoc/index-all.html...
Generating javadoc/deprecated-list.html...
Building index for all classes...
Generating javadoc/allclasses-frame.html...
Generating javadoc/allclasses-noframe.html...
Generating javadoc/index.html...
Generating javadoc/help-doc.html...
2 errors
48 warnings
make[3]: *** [Makefile:720: all] Error 1
make[3]: Leaving directory '/var/tmp/portage/sys-cluster/genders-9999/work/genders-9999/src/extensions/java'
make[2]: *** [Makefile:380: all-recursive] Error 1
make[2]: Leaving directory '/var/tmp/portage/sys-cluster/genders-9999/work/genders-9999/src/extensions'
make[1]: *** [Makefile:380: all-recursive] Error 1
make[1]: Leaving directory '/var/tmp/portage/sys-cluster/genders-9999/work/genders-9999/src'
make: *** [Makefile:439: all-recursive] Error 1
While generating the javadoc, make fails with the following two errors:
./gov/llnl/lc/chaos/Genders.java:170: error: bad HTML entity
* with '||', intersection with '&&', * difference with '--', and
^
./gov/llnl/lc/chaos/Genders.java:170: error: bad HTML entity
* with '||', intersection with '&&', * difference with '--', and
^
Full logs: https://gist.github.com/dunn/844d0e975e137b995dec#file-02-make-L491-L496
Would be great to have the ability for nested genders files...
eg. /etc/genders to be
include /etc/genders_c1
include /etc/genders_c2
name1 attr1,attr2=a
This would allow us to have 1 genders file manage multiple clusters.
Currently there are three selection modes - name exists, name=value, or the complement thereof. I propose adding relative comparisons of the following types:
< <= >= >
These should be constrained to either integer or float values. For example:
mem>64
I cannot at the moment think of a float value. I think automatic casting is probably overkill and (personally) would think integer is sufficient.
In the event a non-integer value is supplied, there are a few possibilities:
I prefer the second option, since a non-integer string is an error.
Note that, if the NOT operator previous suggested is implemented, then you could get by with implementing only < and >.
That said, I think <= and >= are a bit more mainstream and definitely easier to read and understand.
configured with --with-java-extensions
my jni_md.h is in /usr/lib64/icedtea8/include/linux
Making all in java
make[3]: Entering directory '/var/tmp/portage/sys-cluster/genders-9999/work/genders-9999/src/extensions/java'
/bin/bash ../../../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I../../../config -Os -pipe -march=native -mfpmath=sse -fopenmp -pthread -fno-unroll-all-loops -fno-unroll-loops -fno-peel-loops -Wformat -Wformat-security -Warray-bounds -Wclobbered -Wvolatile-register-var -Wdate-time -Os -pipe -march=native -mfpmath=sse -fopenmp -pthread -fno-unroll-all-loops -fno-unroll-loops -fno-peel-loops -Wformat -Wformat-security -Warray-bounds -Wclobbered -Wvolatile-register-var -Wdate-time -floop-interchange -ftree-loop-distribute-patterns -fpredictive-commoning -ftree-loop-distribution -fsplit-paths -ftree-vectorize -ftree-partial-pre -fgcse-after-reload -fivopts -Wimplicit-function-declaration -Wno-error=implicit-function-declaration -Wl,-O1 -Wl,--as-needed -Wl,-fuse-ld=gold -lpthread -lgomp -I/usr/lib64/icedtea8/include -c -o Gendersjni.lo `test -f 'src/Gendersjni.c' || echo './'`src/Gendersjni.c
libtool: compile: gcc -DHAVE_CONFIG_H -I. -I../../../config -Os -pipe -march=native -mfpmath=sse -fopenmp -pthread -fno-unroll-all-loops -fno-unroll-loops -fno-peel-loops -Wformat -Wformat-security -Warray-bounds -Wclobbered -Wvolatile-register-var -Wdate-time -Os -pipe -march=native -mfpmath=sse -fopenmp -pthread -fno-unroll-all-loops -fno-unroll-loops -fno-peel-loops -Wformat -Wformat-security -Warray-bounds -Wclobbered -Wvolatile-register-var -Wdate-time -floop-interchange -ftree-loop-distribute-patterns -fpredictive-commoning -ftree-loop-distribution -fsplit-paths -ftree-vectorize -ftree-partial-pre -fgcse-after-reload -fivopts -Wimplicit-function-declaration -Wno-error=implicit-function-declaration -Wl,-O1 -Wl,--as-needed -Wl,-fuse-ld=gold -lpthread -lgomp -I/usr/lib64/icedtea8/include -c src/Gendersjni.c -fPIC -DPIC -o .libs/Gendersjni.o
In file included from src/Gendersjni.c:1:
/usr/lib64/icedtea8/include/jni.h:45:10: fatal error: jni_md.h: No such file or directory
45 | #include "jni_md.h"
| ^~~~~~~~~~
compilation terminated.
make[3]: *** [Makefile:497: Gendersjni.lo] Error 1
make[3]: Leaving directory '/var/tmp/portage/sys-cluster/genders-9999/work/genders-9999/src/extensions/java'
make[2]: *** [Makefile:380: all-recursive] Error 1
make[2]: Leaving directory '/var/tmp/portage/sys-cluster/genders-9999/work/genders-9999/src/extensions'
make[1]: *** [Makefile:380: all-recursive] Error 1
make[1]: Leaving directory '/var/tmp/portage/sys-cluster/genders-9999/work/genders-9999/src'
make: *** [Makefile:439: all-recursive] Error 1
configured with --with-java-extensions
my jni.h is in /usr/lib64/icedtea8/include/
Making all in java
make[3]: Entering directory '/var/tmp/portage/sys-cluster/genders-9999/work/genders-9999/src/extensions/java'
/bin/bash ../../../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I../../../config -Os -pipe -march=native -mfpmath=sse -fopenmp -pthread -fno-unroll-all-loops -fno-unroll-loops -fno-peel-loops -Wformat -Wformat-security -Warray-bounds -Wclobbered -Wvolatile-register-var -Wdate-time -Os -pipe -march=native -mfpmath=sse -fopenmp -pthread -fno-unroll-all-loops -fno-unroll-loops -fno-peel-loops -Wformat -Wformat-security -Warray-bounds -Wclobbered -Wvolatile-register-var -Wdate-time -floop-interchange -ftree-loop-distribute-patterns -fpredictive-commoning -ftree-loop-distribution -fsplit-paths -ftree-vectorize -ftree-partial-pre -fgcse-after-reload -fivopts -Wimplicit-function-declaration -Wno-error=implicit-function-declaration -Wl,-O1 -Wl,--as-needed -Wl,-fuse-ld=gold -lpthread -lgomp -c -o Gendersjni.lo `test -f 'src/Gendersjni.c' || echo './'`src/Gendersjni.c
libtool: compile: gcc -DHAVE_CONFIG_H -I. -I../../../config -Os -pipe -march=native -mfpmath=sse -fopenmp -pthread -fno-unroll-all-loops -fno-unroll-loops -fno-peel-loops -Wformat -Wformat-security -Warray-bounds -Wclobbered -Wvolatile-register-var -Wdate-time -Os -pipe -march=native -mfpmath=sse -fopenmp -pthread -fno-unroll-all-loops -fno-unroll-loops -fno-peel-loops -Wformat -Wformat-security -Warray-bounds -Wclobbered -Wvolatile-register-var -Wdate-time -floop-interchange -ftree-loop-distribute-patterns -fpredictive-commoning -ftree-loop-distribution -fsplit-paths -ftree-vectorize -ftree-partial-pre -fgcse-after-reload -fivopts -Wimplicit-function-declaration -Wno-error=implicit-function-declaration -Wl,-O1 -Wl,--as-needed -Wl,-fuse-ld=gold -lpthread -lgomp -c src/Gendersjni.c -fPIC -DPIC -o .libs/Gendersjni.o
src/Gendersjni.c:1:10: fatal error: jni.h: No such file or directory
1 | #include <jni.h>
| ^~~~~~~
compilation terminated.
make[3]: *** [Makefile:497: Gendersjni.lo] Error 1
make[3]: Leaving directory '/var/tmp/portage/sys-cluster/genders-9999/work/genders-9999/src/extensions/java'
make[2]: *** [Makefile:380: all-recursive] Error 1
make[2]: Leaving directory '/var/tmp/portage/sys-cluster/genders-9999/work/genders-9999/src/extensions'
make[1]: *** [Makefile:380: all-recursive] Error 1
make[1]: Leaving directory '/var/tmp/portage/sys-cluster/genders-9999/work/genders-9999/src'
make: *** [Makefile:439: all-recursive] Error 1
./configure --prefix=/usr --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc --localstatedir=/var/lib --disable-dependency-tracking --disable-silent-rules --docdir=/usr/share/doc/genders-9999 --htmldir=/usr/share/doc/genders-9999/html --with-sysroot=/ --libdir=/usr/lib64 --disable-static --with-non-shortened-hostnames --with-cplusplus-extensions --with-java-extensions --with-perl-extensions --with-python-extensions
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking target system type... x86_64-pc-linux-gnu
checking for a BSD-compatible install... /usr/lib/portage/pypy/ebuild-helpers/xattr/install -c
checking whether build environment is sane... yes
/var/tmp/portage/sys-cluster/genders-9999/work/genders-9999/config/missing: Unknown `--is-lightweight' option
Try `/var/tmp/portage/sys-cluster/genders-9999/work/genders-9999/config/missing --help' for more information
configure: WARNING: 'missing' script is too old or missing
checking for a thread-safe mkdir -p... /bin/mkdir -p
Hi,
we recently stumbled over the fact that nodeattr does not allow queries involving UTF-8 encoded attribute values.
We have actually quite a lot of them but apparently never needed to use them in queries. Also, I haven't seen any
documentation that says attributes must be ASCII.
The problem seems to be the lex tokenizer that matches only ASCII characters.
While it's not really straightforward to teach flex about UTF-8, a relatively simple patch seems to do most of the work:
--- genders-1.28.1/src/libgenders/genders_query_parse.l 2021-02-28 11:13:08.580111309 +0100
+++ genders-1.28.1.utf8/src/libgenders/genders_query_parse.l 2021-02-28 11:13:45.383330719 +0100
@@ -41,8 +41,19 @@
%}
+ASC [a-zA-Z0-9]
+ASCC [a-zA-Z0-9_\.\=:%\\\/\+]
+
+U [\x80-\xbf]
+U2 [\xc2-\xdf]
+U3 [\xe0-\xef]
+U4 [\xf0-\xf4]
+
+UT {ASC}|{U2}{U}|{U3}{U}{U}|{U4}{U}{U}{U}
+UTC {ASCC}|{U2}{U}|{U3}{U}{U}|{U4}{U}{U}{U}
+
%%
-[a-zA-Z0-9][a-zA-Z0-9_\.\=:%\\\/\+]*([\-\|&]?[a-zA-Z0-9_\.\=:%\\\/\+]+)* yylval.attr = strdup(yytext); return ATTRTOK;
+{UT}{UTC}*([\-\|&]?{UTC}+)* yylval.attr = strdup(yytext); return ATTRTOK;
\( return LPARENTOK;
\) return RPARENTOK;
\|\| return UNIONTOK;
diff -r -u genders-1.28.1/src/libgenders/Makefile.am genders-1.28.1.utf8/src/libgenders/Makefile.am
--- genders-1.28.1/src/libgenders/Makefile.am 2020-05-15 21:52:08.000000000 +0200
+++ genders-1.28.1.utf8/src/libgenders/Makefile.am 2021-02-28 11:18:00.873911772 +0100
@@ -31,7 +31,7 @@
# achu: -o option in lex/flex is not portable, use -t and write to stdout
genders_query_parse.c: genders_query.c $(srcdir)/genders_query_parse.l
- $(LEX) -t $(srcdir)/genders_query_parse.l > $(srcdir)/genders_query_parse.c
+ $(LEX) -8 -t $(srcdir)/genders_query_parse.l > $(srcdir)/genders_query_parse.c
# achu: -o option in yacc/bison is not portable, use -b instead
genders_query.c: $(srcdir)/genders_query.y
Any chance to see something like that in the next releases ?
I get this when building the latest code
genders_query.tab.c: In function 'yyparse':
genders_query.tab.c:1908:3: error: expected expression at end of input
1908 | if (yychar <= YYEOF)
| ^~
genders_query.tab.c:1908:3: error: expected declaration or statement at end of input
genders_query.tab.c:1897:5: error: label 'yydefault' used but not defined
1897 | goto yydefault;
| ^~~~
genders_query.tab.c:1882:5: error: label 'yyacceptlab' used but not defined
1882 | YYACCEPT;
| ^~~~~~~~
genders_query.tab.c:1877:9: error: label 'yyabortlab' used but not defined
1877 | YYABORT;
| ^~~~~~~
genders_query.tab.c:1859:11: error: label 'yyexhaustedlab' used but not defined
1859 | goto yyexhaustedlab;
| ^~~~
I would like to be able to override the default path with an environment variable, like GENDERS_FILE (or maybe just GENDERS?).
I use genders on my laptop and desktop and maintain my genders file in a git repo. Unlike our servers, my laptop and desktop are not under configuration management, so it's messy to keep /etc/genders updated -- I have to symlink to my checkout in my home directory, manually override the path whenever I run nodeattr or manually copy the updated file into place.
I can also imagine scenarios where an individual user might want to use genders/pdsh and, while she might be able to convince her sysadmins to install the packages, having them maintain a shared database is more problematic. Being able to use a personal copy would be a good first step to bringing the tool into a new environment.
I can also imagine potential security implications with allowing users to override the location, if genders/nodeattr/pdsh are used in scripts run with elevated privileges, so it would probably be necessary to include a compile-time flag to disable this feature for sites that might have such requirements.
It would be helpful if genders supported delimited lists (a sort of sub-fielding) in values. I envision it this way:
# in the genders file
hostx mount=fs1:fs2:fs3
Then to find any host that mounts fs2:
nodeattr -d: -q mount?fs2
The -d: option sets the list delimiter to the colon, which I would recommend as the default.
The ? is a "list contains" operator. It is interpreted as "using : as a delimiter, is the item fs2 in the mount list?"
I would not suggest wildcards in the ? argument. I'm sure some would love it, but I personally have never seen the need.
By the way, I implement this capability now, but it requires some conventions to be imposed in the genders data, uses an external script or function, and is a multi-step process. Having the capability in-built would be a blessing!
In version 1-26-1
make -j4 -l8
Making all in src
make[1]: Entering directory '/var/tmp/portage/sys-cluster/genders-1.26.1/work/genders-genders-1-26-1/src'
Making all in libcommon
make[2]: Entering directory '/var/tmp/portage/sys-cluster/genders-1.26.1/work/genders-genders-1-26-1/src/libcommon'
make[2]: *** No rule to make target 'fd.c', needed by 'libcommon_la-fd.lo'. Stop.
make[2]: Leaving directory '/var/tmp/portage/sys-cluster/genders-1.26.1/work/genders-genders-1-26-1/src/libcommon'
make[1]: *** [Makefile:294: all-recursive] Error 1
make[1]: Leaving directory '/var/tmp/portage/sys-cluster/genders-1.26.1/work/genders-genders-1-26-1/src'
make: *** [Makefile:344: all-recursive] Error 1
Several internal data structures in libcommon (list.c, hash.c minimally) are not compiled with the WITH_PTHREADS macro, thus are not thread safe. If some genders functions are called by multiple threads, badness can occur.
For example, genders_isattrval() is one example which creates list iterators. If multiple threads were to call genders_isattrval() on the same attr, multiple iterators could be created on the same list, leading to possible collisions in the internals of list.c.
If it is desired to make libgenders thread safe, code will probably need to be auditted to see what else may not be thread safe.
In version 1-26-1
***** autoconf *****
***** PWD: /var/tmp/portage/sys-cluster/genders-1.26.1/work/genders-genders-1-26-1
***** autoconf --force
configure.ac:11: error: possibly undefined macro: AC_META
If this token and others are legitimate, please use m4_pattern_allow.
See the Autoconf documentation.
configure.ac:89: error: possibly undefined macro: AC_DEBUG
configure.ac:155: error: possibly undefined macro: AC_GENDERS_FILE
configure.ac:160: error: possibly undefined macro: AC_EXTENSION_DESTDIR
configure.ac:165: error: possibly undefined macro: AC_PERL_EXTENSIONS
configure.ac:177: error: possibly undefined macro: AC_PERL_SITE_ARCH
configure.ac:178: error: possibly undefined macro: AC_PERL_VENDOR_ARCH
configure.ac:194: error: possibly undefined macro: AC_PYTHON_EXTENSIONS
configure.ac:200: error: possibly undefined macro: AC_CPLUSPLUS_EXTENSIONS
configure.ac:206: error: possibly undefined macro: AC_JAVA_EXTENSIONS
Just installed 1.22 packaged by Fedora, released May 20 2015.
Prefix[x-y] doesn't work while Prefix[x-y]Suffix does
me $ echo "db[1-3] example" >/etc/genders
me $ nodeattr --parse-check
nodeattr: 0 parse errors discovered
me $ nodeattr -A
db[1-3]
me $ echo "db[1-3]-t test" >.genders
me $ nodeattr -A
db1-t,db2-t,db3-t
Same for Prefix[a,b,c,...]. Doesn't work without a suffix.
How is it possible no one has noticed this error before?
While the complement "~(expression)" feature is good, it is syntactically ugly - especially in more complex queries. I would like to see a NOT operator as it would make expressions simpler to write and read. I think the best choice is to use the bang (!) to negate the following expression:
!name
name!=value
This would allow expressions such as:
!compute && type!=admin && mem=64
rather than:
~(compute || type=admin) && mem=64
As I said earlier, I believe the benefit becomes more apparent as the complexity of the query increases.
there are a couple of perl files that aren't getting packaged:
ls usr/lib/genders/
gendlib.pl hostlist.pl
The old www.llnl.gov/linux url that's all over the place should be updated to the new github address.
As mentioned in my request for the ? "list contains" operator, I have implemented the capability external to genders (specifically, nodeattr as the sites I work with use bash predominantly for admin scripts.) Let me explain what I currently do, before I say what I'd like to see.
Background
Consider the case where fs1, fs2, fs3 are NFS mount paths. Those can be rather long strings (abbreviated here for clarity). If you have systems that make a half dozen NFS mounts, having to list those in every set of hosts can make the genders record rather long and the file large. Add to that the maintenance headache, say for example if the host for fs2 changes and fs2 is referenced by a few dozen or hundred records. That's a lot of editing and a lot of opportunity for errors!
I currently handle this using a second genders database as a sort of meta database:
# main genders file
hostx mount=@datamover
# meta database file
mount_datamover fs1,fs2,fs3
This allows me to fetch the mounts for hostx with the query:
nodeattr -v hostx mount
When I see the value begins with the @, then I perform a secondary query by combining the name (mount) and the value sans @ (datamover) with an underscore (_):
nodeattr -f metadata -l mount_datamover
I can also reverse the lookup (similar to the ? "list contains" feature requested separately) in two steps. In psuedo-code:
foreach name,value in split('_', nodeattr -q fs2)
nodeattr -q name=@value
The Request
What I'd like to see is genders support a sort of macro/variable substitution. For example:
# genders file
@datamover_mounts fs1:fs2:fs3
hostx mount=@datamover_mounts
To find the actual mounts for hostx:
nodeattr -v hostx mount
fs1:fs2:fs3
To find what mounts fs2:
nodeattr -d: -q mount?fs2
hostx
In either case, genders would have to scan for @macros and perform the substitution, before it could apply the query and return results.
For the use case I cited in my example, this would permit a single file to be used, would simplify database maintenance, and help reduce the size of the genders file.
Note: I know nothing of the internals of how the genders file is handled, whether it is read into memory and expanded and then queried, if it is examined and expanded a line at a time, etc. In other words, I admit to having no idea how simple or complex such a proposal would be.
I have a suggestion if the %r mentioned in the FAQ is still open for discussion.
I have implemented on the application end a generalization of the idea of %r.
Let attribute values contain additional nodename format strings of the form %NT.
N is an integer value 0-9. At present, even a cray does not use more than 10 segments in a name.
T is a segment type
%NT is replaced by the Nth segment of type T appearing in the node name.
In my initial implementation the supported values of T are 'd' (integer substrings), 'm' (substrings separated by '-'), and 'u' (substrings separated by '_'). Note that multiple separators of the - or _ types are collapsed to one, so '--' is equivalent to '-' in this scheme.
E.g.
node-gw23 fastnet=node-gw%0d-ib
node-gw23 otherthing=ot-%1m
yields
node-gw23 fastnet=node-gw23-ib
node-gw23 otherthing=ot-gw23
This notation allows collapsing definitions in a manner similar to the FAQ %r idea, but with more control.
The exact semantics are given by a bash script below. Equivalent c/c++ are not hard to cook up (I have a c++ version).
gender_substitute () {
ghost=$1
val=$2
if test -z $2; then
echo ""
return
fi
ilist=$(echo $ghost | sed -e 's/[-_a-zA-Z]*/ /g')
wlist=$(echo $ghost | sed -e 's/-/ /g')
ulist=$(echo $ghost | sed -e 's/_/ /g')
vnew=$val
n=0
for i in $ilist; do
sub="%${n}d"
vnew=$(echo ${vnew//$sub/$i})
((n++))
done
n=0
for w in $wlist; do
sub="%${n}m"
vnew=$(echo ${vnew//$sub/$w})
((n++))
done
n=0
for u in $ulist; do
sub="%${n}u"
vnew=$(echo ${vnew//$sub/$u})
((n++))
done
echo $vnew
}You may or may not want this... but its a patch we run to make genders MUCH faster at parsing the database.
As some background, we are using genders as the source of truth for lots of information about each node... so every node has its own entry and we have LOTS of atributes (mac addresses, ip, serial numbers etc). The file is over 10MB in size. Reading it 1 character a time was very very slow. This patch makes it at least 10x faster.
I haven't fully dug into this, but in my SLES build I'm seeing the following error:
[ 124s] calling /usr/lib/rpm/brp-suse.d/brp-35-rpath
[ 124s] ERROR: RPATH "/home/abuild/rpmbuild/BUILD/genders-1.22/src/libgenders/.libs" on /home/abuild/rpmbuild/BUILDROOT/genders-ohpc-1.22-18.1.x86_64/usr/lib/perl5/vendor_perl/5.18.2/x86_64-linux-thread-multi/auto/Libgenders/Libgenders.so is not allowed
Probably not ideal to have RPATH pointing back to the build directory.
In commit d3638a5 the number of "valid" lines in https://github.com/chaos/genders/blob/master/src/testsuite/libgenders/testdatabases/genders.parse_error_hostrange_unbalanced was changed.
Assuming an older version of hostlist considered all unbalanced brackets invalid hostnames, but current version of hostlist appears not to. Perhaps newer version of hostlist would work, or newer version of hostlist has bug in it.
How about we add support for a genders.d directory of genders files, similar to the many other /etc/*.d directories?
This would be great for some ansible use cases, where we have individual roles that each want to generate genders attributes. They could each own their own file under /etc/genders.d.
This might serve as an alternative to issue #23 as well.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.