dcache / oncrpc4j Goto Github PK

currently a FIXME.
makes errors harder to track down

A call through GenericPortmapClient to client.dump agains a fresh OncRpcbindServer instance returns
the following :
prog: 100000, vers: 2, netid: tcp, addr: 0.0.0.0.0.111, owner: unspecified
prog: 100000, vers: 2, netid: udp, addr: 0.0.0.0.0.111, owner: unspecified

here owner is "unspecified" while it is initialized to "superuser" in OncRpcbindServer constructor

I was monitoring my rpc service via jmx and noticed the io-strategy was set to same-thread. That seemed strange since I requested leader-follower in my code. I started examining oncrpc4j source code and noticed that the nio transport builders are hard-coding same-thread strategy rather than using the one configured by the oncrpcsrv builder:

        IoStrategy ioStrategy = builder.getIoStrategy();
        String serviceName = builder.getServiceName();
        ThreadPoolConfig selectorPoolConfig = getSelectorPoolCfg(ioStrategy,
                serviceName,
                builder.getSelectorThreadPoolSize());

        if ((protocol & IpProtocolType.TCP) != 0) {
            final TCPNIOTransport tcpTransport = TCPNIOTransportBuilder
                    .newInstance()
                    .setReuseAddress(true)
                    .setIOStrategy(SameThreadIOStrategy.getInstance())
                    .setSelectorThreadPoolConfig(selectorPoolConfig)
                    .setSelectorRunnersCount(selectorPoolConfig.getMaxPoolSize())
                    .build();
            _transports.add(tcpTransport);
        }

        if ((protocol & IpProtocolType.UDP) != 0) {
            final UDPNIOTransport udpTransport = UDPNIOTransportBuilder
                    .newInstance()
                    .setReuseAddress(true)
                    .setIOStrategy(SameThreadIOStrategy.getInstance())
                    .setSelectorThreadPoolConfig(selectorPoolConfig)
                    .setSelectorRunnersCount(selectorPoolConfig.getMaxPoolSize())
                    .build();
            _transports.add(udpTransport);
        }

Is this intentional?

It seems the specified io-strategy is only used to set the size of the thread pool.

For convince and readability can we generate java enums for x file enumerations

Hi,
I mentioned the char [] array in .x file
Using rpcgen it is parsing as char array
But jrpcgen generating byte [] array in Java class . Is there any issue ? .
I have a c++ server
Java as client side which uses oncrpc .
When I converted the hexa string to byte array from Java and send to c++ .it is printing all zeros .

When worker thread strategy is used, then OncRpcSvc starts a fixed number of threads to handle requests. However, if some threads die due to errors, they get not respawn. Worse, when all
thread die, then service becomes responsive.

oncrpc4j contains org.dcache.utils packages wich exist in other dcache software modules as well. As java9 module sybsystem requires, that package exist only in a single jar org.dcache.utils must be removed/renames on oncrpc4j

Hi kofemann,
I have a c++ rpc server .I want to generate different version numbers for that c++ rpc server from java ..can I use rpcsvc class .is it possible from your code?
Example :
Program number : 10007
But multiple versions(1, 2,3,4 up to 32 versions ) on different machine address( 4 IP address)
For the same c++ rpc server programme.
100007 1 tcp 38903 c++rpc
100007 2 tcp 60872 c++rpc
100007. 3 tcp 60987 c++rpc
Please give me your suggestion regarding this .
svc = new OncRpcSvcBuilder()
.withoutAutoPublish()
.withTCP()
.withMinPort(1)
.withMaxport(60000)
.withWorkerThreadIoStrategy()
.withBindAddress("127.0.0.1")
.withSelectorThreadPoolSize(1)
.withWorkerThreadPoolSize(1)
.withRpcService(new OncRpcProgram(PROGNUM, PROGVER), echo)
.withServiceName("c++rpc")
.build();
svc.start()
Here Program version(1to 32),IP address(1 to 4) I need to pass it differently??. And I want to do the same the process on different address like ip address1, ip address2 ,ip address3, ip address 4.. is it possible . I tried with above code and changed the bind address.what is the use of withbindaddress()? but it is throwing bind exception ..am run it on Windows but I pass the parameters in withbindaddress(Linux machine ip adderess1). I checked etc/hosts in Linux .I gave the localhost address from that file .still it is throwing bind exception. But when I gave the local address of Windows .it is registering the portmapper in Windows .please give me your suggestions regarding this

Hi,
I am running multiple rpc process on the server . Multiple threads are sending request to server .each thread need to identify server process .To establish the connection with rpc . In the code you mentioned one constructor .it will take the parameters port number , version, tcp ,program number . Port number I can able to get using portmapperclient object . How can I get the program version number .because each thread need to access the separate server process .
Thanks

Hi,

Cisco system recommends that client must not open more than 3 connection at a time at RPC server.

Is there any way we can achieve these?

Regards

this is useful for developers working on OSs where there's no native portmap service to be found (windows) or the existing rpcbind server is dysfunctional (mac osx).

given a "fat jar" it would be relatively easy to run a portmap server by issuing:

java -cp fat.jar org.dcache.xdr.portmap.OncRpcbindServer

which is also easier to use in service definition files.

Hello. In my project (Electronic Access Control) we are looking to replace an old ONCRPC implementation in the server with something new. But we still need ONCRPC to be compatible with existing door controller hardware. Our old implementation has the feature to embed the ONCRPC server in an existing server by creating the ONCRPC server with an input- and an output stream. Our solution is built that way because we have several protocols talking on the same server port.

So my question is can ONCRPC4J be embedded some way? Prefferably using asynchronous channels.

Regards, Oscar Lantz

oncrpc4j uses dCache alternatives of com.sun.security.auth.UnixNumericUserPrincipal to make code independent from SUN/Oracle specific code. However, UnixNumericUserPrincipal is a part of official JDK and can be safely used.

Additionally, dCache internal implementation released under a different license and make use of oncrpc4j by external projects with incompatible license impossible (#80 ).

Despite oncrpc4j API looks perfectly asynchronous I've failed utterly trying to send responses in a separate thread. Unfortunately I don't have the stack trace at hand, but just out of curiosity: is it a known problem?

Hi,
I have a multiple threads are calling the rpc server .in the code you gave the constructor .when I pass the parameters like port number , version , tcp .it is establishing connection with rpc server . But threads are making multiple call to rpc server .I don't want to open multiple connections .how can I hold the connection . Two threads need to open the one connection with
rpc server and post the data to server.is there any methods I need to call from your code ?? .thanks in advance

Hi ,
I am not able to set the owner name for rpc service ..I already set the value for owner field in portmapper client .but when I type the rpcinfo .
It is showing the service owner as unspecified.do you have any suggestions. I changed to portmapper v4.but there is no use

the timeout arguments are used for Future.get().
however, the CompletionHandler is not removed from the underlying transport's ReplyQueue

Hi,

***This is a query and not an issue. ***
I am trying to call RPC in UNIX server, but before actual call in need to look for the port of the program from portmap using program id and version number.

I have tested string Length program, and rpcinfo is giving below details :

program version netid address service owner
100000 4 tcp6 ::.0.111 portmapper superuser
100000 3 tcp6 ::.0.111 portmapper superuser
100000 4 udp6 ::.0.111 portmapper superuser
100000 3 udp6 ::.0.111 portmapper superuser
100000 4 tcp 0.0.0.0.0.111 portmapper superuser
100000 3 tcp 0.0.0.0.0.111 portmapper superuser
100000 2 tcp 0.0.0.0.0.111 portmapper superuser
100000 4 udp 0.0.0.0.0.111 portmapper superuser
100000 3 udp 0.0.0.0.0.111 portmapper superuser
100000 2 udp 0.0.0.0.0.111 portmapper superuser
100000 4 local /run/rpcbind.sock portmapper superuser
100000 3 local /run/rpcbind.sock portmapper superuser
117 1 tcp 0:0:0:0:0:0:0:0.7.127 - unknown
117 1 tcp6 0:0:0:0:0:0:0:0.7.127 - unknown

The program is registered with ID 177 and Version 1, But how can i determine the port grammatically.

I have gone through the code and found a class GenericPortmapClient and tried to get the dump.Still i am not getting the port of the program. I have used port 1919 while running server application.

Regards
Taslim Alam

Java9 module subsystem expects that a package exists only in a single jar file. This is in general true for oncrpc4j. However a better package names will makes no sense:

org.dcache.oncrpc4j
org.dcache.oncrpc4j.rpc
org.dcache.oncrpc4j.xdr
org.dcache.oncrpc4j.net

When checking portmapper from Linux I see something strange on service list

# rpcinfo -p 10.10.10.20
   program vers proto   port  service
    100004    1   udp  51014  ypserv
    100004    24294967295  64916  ypserv
    100004    2   udp  51014  ypserv
    100000    2   tcp    111  portmapper
    100007    2   tcp  64833  ypbind
    100004    14294967295  64916  ypserv
    100004    2   tcp  64916  ypserv
    100004    1   tcp  64916  ypserv
    100000    2   udp    111  portmapper
    100007    2   udp  51010  ypbind
    100007    24294967295  64833  ypbind

Also command fails to connect to actual services

# rpcinfo -u 10.10.10.20 ypserv 1
rpcinfo: RPC: Remote system error
program 100004 version 1 is not available

Code is just simple stub classes running on Windows NetBeans

public class YpService {
    private final static int RPC_YPPROG = 100004;
    private final static int RPC_YPBIND = 100007;

    public static void main(String[] args) {
        new OncRpcEmbeddedPortmap(); 
        YpBindProg ypBind = new YpBindProg();
        YpProg ypProg = new YpProg();
        OncRpcSvc ypBindService = new OncRpcSvcBuilder()
                .withUDP()
                .withTCP()
                .withAutoPublish()
                .withSameThreadIoStrategy()
                .withRpcService(new OncRpcProgram(RPC_YPBIND, 2),ypBind)
                .build();
        OncRpcSvc ypService = new OncRpcSvcBuilder()
                .withUDP()
                .withTCP()
                .withAutoPublish()
                .withSameThreadIoStrategy()
                .withRpcService(new OncRpcProgram(RPC_YPPROG, 1),ypProg)
                .withRpcService(new OncRpcProgram(RPC_YPPROG, 2),ypProg)
                .build();       
        try {
            ypBindService.start();
            ypService.start();
            System.in.read();
        } catch (IOException ex) {
            Logger.getLogger(YpService.class.getName()).log(Level.SEVERE, null, ex);
        }
    }
}

Libs are
core-2.5.3
guava-19.0
grizzly-framework-2.3.25
(slf4+auth deps)

rpcbind protocol v2 doesn't supports only IPv4 and makes embedded portmap not useful in IPv6 enabled environments.

current rpc client implementation (2.4.x) doesn't supports RPCSEC_GSS, which makes it impossible to use with kerberized servers.

Xdr#xdrEncodeByteBuffer calls Buffer#flip on the input parameter before using it. This is rather surprising since Buffer based APIs normally use the position and limit of a Buffer to communicate which range of data should be used. Calling flip on the buffer makes it impossible to use any other position than 0.
Is this intentional or a bug?

Hello,

I am trying write a ONCRPC program on IBM AIX 5.1. First, I tested it on my windows 7 desktop and worked, but when I run it on IBM AIX 5.1, the program can't be registered to portmap service. The log listed below:

09:43:33.828 [WrapperSimpleAppMain] INFO com.csc.dl.rpc.server.Runner - Starting program...
09:43:33.860 [WrapperSimpleAppMain] INFO com.csc.dl.rpc.server.Runner - Start RPC Server @ Program: 9999
09:44:51.460 [WrapperSimpleAppMain] INFO com.csc.dl.rpc.server.Runner - Starting program...
09:44:51.498 [WrapperSimpleAppMain] INFO com.csc.dl.rpc.server.Runner - Start RPC Server @ Program: 9999
09:44:52.077 [WrapperSimpleAppMain] DEBUG c.c.d.rpc.helpers.EnvironmentHelper - Get env property by IP: test
09:44:54.542 [WrapperSimpleAppMain] INFO com.csc.dl.rpc.server.Limiter - Load threshold data
09:45:00.221 [WrapperSimpleAppMain] INFO org.dcache.xdr.OncRpcSvc - Registering new program [9999:1] : com.csc.dl.rpc.server.Applic
ation@a8bf6e6e
09:45:00.553 [WrapperSimpleAppMain] DEBUG o.d.xdr.portmap.RpcbindV4Client - portmap pina
0mpmppcbindV4Cli59 [WrapperSimpleAppMain] DEBUG org.dcache.xdr.ReplyQueue - Registering key 1
09:45:00.582 [WrapperSimpleAppMain] DEBUG org.dcache.xdr.ReplyQueue - query key 1 with timeout
09:45:00.593 [Grizzly-kernel(1) SelectorRunner] DEBUG org.dcache.xdr.ReplyQueue - updating key 1
09:45:00.596 [WrapperSimpleAppMain] DEBUG o.d.xdr.portmap.RpcbindV4Client - portmap unset port: prog: 9999 vers: 1, owner: root
09:45:00.601 [WrapperSimpleAppMain] DEBUG org.dcache.xdr.ReplyQueue - Registering key 2
09:45:00.603 [WrapperSimpleAppMain] DEBUG org.dcache.xdr.ReplyQueue - query key 2 with timeout
09:45:00.605 [Grizzly-kernel(1) SelectorRunner] DEBUG org.dcache.xdr.ReplyQueue - updating key 2
09:45:00.628 [WrapperSimpleAppMain] DEBUG o.d.xdr.portmap.RpcbindV4Client - portmap ping
09:45:00.629 [WrapperSimpleAppMain] DEBUG org.dcache.xdr.ReplyQueue - Registering key 3
09:45:00.631 [WrapperSimpleAppMain] DEBUG org.dcache.xdr.ReplyQueue - query key 3 with timeout
09:45:00.633 [Grizzly-kernel(1) SelectorRunner] DEBUG org.dcache.xdr.ReplyQueue - updating key 3
09:45:00.636 [WrapperSimpleAppMain] DEBUG o.d.xdr.portmap.RpcbindV4Client - portmap set port: prog: 9999 vers: 1, netid: udp addr: 0
:0:0:0:0:0:0:0.117.48, owner: root
09:45:00.638 [WrapperSimpleAppMain] DEBUG org.dcache.xdr.ReplyQueue - Registering key 4
09:45:00.639 [WrapperSimpleAppMain] DEBUG org.dcache.xdr.ReplyQueue - query key 4 with timeout
09:45:00.641 [Grizzly-kernel(1) SelectorRunner] DEBUG org.dcache.xdr.ReplyQueue - updating key 4
09:45:00.642 [WrapperSimpleAppMain] INFO com.csc.dl.rpc.server.Application - RPC Server started.

Can you provide any suggestion?
Thank you.

OncRpcSvc shares a single reply queue to all clients. As a result, the isConnect field is shared as well. When a client disconnects, the connection state changes to false and any attempt to use callbacks throws EOF exception.

It is possible for server to send rpc requests to the client by the connection established by the client. Nevertheless, client code does not provide a way how to register a callback handler.

rpc calls allocate ~0.5MB of buffer:

new Xdr(Xdr.MAX_XDR_SIZE)

this, combined with a large number of pending async IO resulting in a lot of heap space being used.

As a single SSLFilter is shared by all connections, once STARTTLS is seen, the TLS is enforced for all connections.

		assertTrue( portmapClient.getPort(OncRpcPortmap.PORTMAP_PROGRAMM, OncRpcPortmap.PORTMAP_V2, "tcp").equals("127.0.0.1.0.111") ); // check port

What is the port number here 0.111
Please let me know what is the format it is giving

In some setups is required to force client to use a privilege port number as well as to tell server to check that privilege source port was used by client.

RPC protocol support various security flavors that provide data integrity and protection. However TLS based security is widely used outside of RPC. There is a low priority activity in IETF to adopt TLS as RPC transport. Oncrpc4j based on grizzly framework, thus adding TLS support should be easy.

I know that NFSv4 has standardized on UTF-8 for encoding/decoding xdr strings. However, NFSv3 does not have that standard. (See RFC1813 section 3.2.5.)

I used jrpcgen to generate a mount3 client and am using that client to read the list of exported paths and mount them. (Then I use a nfs3 client to do some other unrelated stuff.) The problem I'm having is that some of the export paths on a customer server (a Windows Server) are encoded with ISO-8859-1. (I don't know what the coding is in advance, but I have since discovered the encoding.) When I get the exports using the jrpcgen-generated mount3 client, all string values (like path) are decoded with UTF-8, so if that isn't the correct encoding I get a string that can have one or more "replacement characters" for the characters that couldn't be decoded as UTF-8. When that happens no exception is thrown, so I don't know that there is anything wrong until I try to use that path string to mount the export path. At that point, the string with the replacement char in it is encoded back to a byte array by the XdrEncoder but does not match the exported path on the server so I get an error.

Even if an exception were thrown when one or more characters can't be decoded using UTF-8, there appears to be no mechanism for me to attempt to decode a string with a different Charset.

One workaround I've found is to modify the xdr file that I generate the client from and change the dirpath typedef from string to opaque. That allows me to get and set any dirpath values at a byte array rather than a string and puts the responsibility for decoding to String and encoding back to byte array in my program code.

That's not ideal though, because various components of our product (some of them not even Java) all need to do RPC things and prefer to share the same xdr definitions.

I don't know enough about the internal workings of OncRpc4J to know what an ideal solution would look like, but maybe this can be the start of a discussion about it?

My workaround code is based on some Python code used elsewhere in our product that first tries to decode strings with 'utf-8' and if a UnicodeDecodeError is raised it will then try to decode with 'latin-1'. That function then returns a both the decoded string and the char encoding that successfully decoded it as a tuple. But as I said above, in order to even be able to do that I need to first change the typedef from string to opaque, and that's not ideal.

Have others run into this issue? How have they handled it? Thanks.

.x file ..when I create the function without arguments ..
It is throwing syntax error ..
But when I create the function with argument.
It is working properly .
Is there any solution .??
Because I have c++ code which is running on sever .I want to make the call to that functions ..c++ code contain the functions without parameters .

I have a need to supply a threadPoolSize to the OncRpcSvcBuilder's withSelectorThreadPoolSize method for generated clients.

Is there already a way? If not, I'm happy to provide a PR. I see that in the past that's been done by adding additional constructors to OncRpcClient.java. Maybe the most recent time was for PR 53?

Before I give you a PR, I wanted to ask, would it be useful to have a OncRpcClient constructor that takes an instance of OncRpcSvcBuilder so that we wouldn't need additional constructors every time we wanted to pass in a parameter that isn't already exposed? For example, I only need to limit the size of the selectorThreadPool but maybe someone else would need to limit the size of the workerThreadPool next month and would need to go through this process again. If one of the constructors took a OncRpcSvcBuilder then it could also make sure that everything that is needed for a client is set, like .withClientMode() and .withPort(localPort), but otherwise allow the end user to construct the client service with everything else allowed by the builder.

I'm thinking in my PR I'd leave all of the existing constructors alone (so it would be backward compatible with existing code) but change the one that all of the other's call, making it look like this:

public OncRpcClient(InetSocketAddress socketAddress, int protocol, int localPort, IoStrategy ioStrategy, String serviceName) {
    this(socketAddress, localPort, new OncRpcSvcBuilder()
            .withIpProtocolType(protocol)
            .withIoStrategy(ioStrategy)
            .withServiceName(serviceName));
}

and then I'd add this one (called by the one above, but could also be used directly by end users who need it like me):

public OncRpcClient(InetSocketAddress socketAddress, int localPort, OncRpcSvcBuilder builder) {
    _socketAddress = socketAddress;
    _rpcsvc = (builder == null ? new OncRpcSvcBuilder() : builder)
            .withClientMode()
            .withPort(localPort)
            .build();
}

I wanted to vet that idea with you before I do the work of committing a PR. It seems like it would prevent additional future churn just to expose additional things that the builder already provides.

Thanks for your help and input.

if you leave an opaque value null, you get something like this:

java.lang.NullPointerException
    at org.dcache.xdr.Xdr.xdrEncodeDynamicOpaque(Xdr.java:554)
    at org.dcache.oncrpc4j.rpcgen.Value.xdrEncode(Value.java:33)
    at org.dcache.xdr.RpcCall.acceptedReply(RpcCall.java:238)
    at org.dcache.xdr.RpcCall.reply(RpcCall.java:225)

im not sure if anything can be done, since the specs (rfcs 1014, 1832 and 4056) dont say anything about nulls. and variable-length fields are encoded starting with the length as an unsigned number ...

im opening this just to verify that this should be treated as user error.

Hi, I see that oncrpc4j says it is subject to the LGPL license, but it has a dependency on org.dcache.common:dcache-auth which contains source (such as DesiredRole.java) that is under AGPL. I believe that means oncrpc4j must also be licensed under AGPL.

Is there a way to break this dependency, so oncrpc4j can be cleanly under LGPL?

The methods getLong and getInt in class Bytes do not check that the given array is sufficiently long, leading to occasional ArrayIndexOutOfBoundsExceptions.

AUTH_SYS is limited to 16 groups, however many deployments require more that 16 groups support. This is typically handled by discovering users membership information based only on provided uid and querying an external service for missing information.

Failed to read artifact descriptor for org.dcache:oncrpc4j-core:jar:3.0.1: Could not transfer artifact org.dcache:oncrpc4j-core:pom:3.0.1 from/to dcache releases (https://download.dcache.org/nexus/content/repositories/releases): Failed to transfer file: https://download.dcache.org/nexus/content/repositories/releases/org/dcache/oncrpc4j-core/3.0.1/oncrpc4j-core-3.0.1.pom. Return code is: 500 , ReasonPhrase:javax.servlet.ServletException: com.orientechnologies.orient.core.exception.OStorageException: Error in data flush background thread, please restart database and send full stack trace inside of bug report

try {
   program.dispatchOncRpcCall(call);
} catch (RpcException e) {
   call.reject(e.getStatus(), e.getRpcReply());
   _log.warn("Failed to process RPC request: {}", e.getMessage());
} catch (OncRpcException e) {
   call.failRpcGarbage();
   _log.warn("Failed to process RPC request: {}", e.getMessage());
} catch (IOException e) {
   call.failRpcGarbage();
   _log.warn("Failed to process RPC request: {}", e.getMessage());
}

any user exceptions (like NPEs) thrown out of dispatchOncRpcCall() are not caught, and not logged.
this makes tracking down bugs in user code more difficult

The following xdr definition triggers a NumberFormatException in jrpcgen.translateConstant :

enum colors { RED, ORANGE, GREEN };

Call resolves to new BigInteger("0+1+1",8) where the string "0+1+1" is not a valid number representation.

Remark : When the definition has only two elements (such as enum colors { RED, ORANGE}), there is no error as the call resolves to new BigInteger("0+1",8) where "0+1" is a valid octal representation...

Hi, I'm very new to RPC. I am trying to convert some old Sun RPC code to Java in order to run on Windows. I was wondering if the Windows Portmapper service (RPC endpoint mapper) is supported by oncrpc4j or do I need to run your Java portmapdaemon?

Sorry for creating an issue for this question, as there is no way for me to message you via GitHub.

generated code already has this:

public void shutdown() throws IOException {
   rpcClient.close();
}

just rename to close, make class implement Closeable, and now its try-with-resources compatible