implement netty support about barchart-udt

yes, we planned netty support for some time;

your can commit your proposed solutions here:

for netty 3
https://github.com/barchart/barchart-udt/tree/master/barchart-udt-netty3

for netty 4
https://github.com/barchart/barchart-udt/tree/master/barchart-udt-netty4

[He-Pin]

okay,i will try,as my current progress,i was going to write and udt agent to send or receive the file~
i am writing an p2p software

note that netty project
https://github.com/netty/netty/

suggested that we merge barchart-udt as part of standard netty distribution
netty/netty#844

the plan would be as follows:

1. design & test integration in a sandbox here:
   https://github.com/barchart/barchart-udt/tree/master/barchart-udt-netty3
   https://github.com/barchart/barchart-udt/tree/master/barchart-udt-netty4

2. when ready, propagate to netty proper via the following fork
   https://github.com/barchart/netty-udt

@hepin1989 : please take a look

https://github.com/barchart/barchart-udt/tree/master/barchart-udt-netty4/src/test/java/example/echo/stream

[He-Pin]

i have implement my own usage of of UDT with netty,for now,i have send file to remote peer,and send the live video stream with the remote peer,the UDT works so pretty.
for now,i implement my own frame decoder encoder,and protobuf encoder decoder too,so i may join this project when i am free.

[normanmaurer]

@hepin1989 I would be interested how you implement it... is it opensource?

[He-Pin]

it's open source,but i need more test ~~

[He-Pin]

i really want your guys implement it ~~it's not that complicated,just need some work,but may be the connection state indication need some work.

[srnm]

I wanted to try things out, so after cloning barchart-udt:

cd barchart-udt/barchart-udt-netty4
mvn clean install

[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] ../barchart-udt/barchart-udt-netty4/src/main/java/io/netty/channel/socket/UdtChannelConfig.java:[23,16] cannot find symbol
symbol : class StandardSocketOptions
location: package java.net
[INFO] 1 error

The project version is declared (and enforced) as JDK1.6, but StandardSocketOptions is JDK1.7
Is this project intended to be compiled standalone?

Thanks for contributing the code!

Steven:

this is how netty project itself is dealing with mixed java 6 / java 7 support:

1. declare 1.6 java source and target in maven pom.xml
2. yet build with actual 1.7 java JDK

please take a look at:

1. eclipse maven invoker
2. example maven rc

same rules apply to IDE; for example in eclipse:

1. select 1.6 SE project compliance
2. yet map 1.6 SE to 1.7 JDK installation

if you find nicer way of doing this, please report here:
https://github.com/netty/netty/issues?state=open

if you feel these instruction are confusing, please write more clear wiki entry here:
https://github.com/barchart/barchart-udt/wiki/_pages

cheers,

Andrei.

@srnm
since netty udt integration does not need java 7, I removed the dependency on java 7.
previous instructions are no longer needed. please confirm it works for you?

[srnm]

Thanks for helping this newbie along.

I should say that I did a mvn install at the top-level of the project first, and then attempted mvn install on barchart-udt-netty4 standalone after the top-level build failed. Is barchart-udt-netty4 intended to be built as a standalone project?

The barchart-udt-netty4 compile now completes successfully after the latest repo pull.
However, I am getting an error during mvn test/install.

I didn't realise that the netty4 implementation relied on JNI -- I thought it was a reimplementation in Java. I will spend some more time reading the wiki and browsing the code.

thanks again!

---

T E S T S

Running io.netty.channel.socket.nio.TestNioUdtProvider
13.649 [ main] INFO c.b.u.SocketUDT 102 - library location : ./lib/bin
13.655 [ main] INFO c.b.u.SocketUDT 106 - library loader : com.barchart.udt.lib.LibraryLoaderDefaultUDT
13.658 [ main] INFO c.b.udt.lib.AOL 31 - OS_NAME=Mac OS X
13.658 [ main] INFO c.b.udt.lib.AOL 32 - OS_ARCH=x86_64
13.663 [ main] INFO b.u.l.LibraryUDT loadCore 151 - library source: NAR production
13.665 [ main] DEBUG c.b.udt.lib.RES extractResource 79 - sourceURL=jar:file:.m2/repository/com/barchart/udt/barchart-udt-bundle/2.2.0-SNAPSHOT/barchart-udt-bundle-2.2.0-SNAPSHOT.jar!/lib/x86_64-MacOSX-gpp/jni/libbarchart-udt-core-2.2.0-SNAPSHOT.jnilib
13.665 [ main] DEBUG c.b.udt.lib.RES extractResource 90 - targetFile=barchart-udt/barchart-udt-netty4/./lib/bin/x86_64-MacOSX-gpp/libbarchart-udt-core-2.2.0-SNAPSHOT.jnilib
13.665 [ main] DEBUG c.b.udt.lib.RES extractResource 93 - targetFolder=barchart-udt/barchart-udt-netty4/./lib/bin/x86_64-MacOSX-gpp
13.666 [ main] INFO c.b.udt.lib.RES sureTargetFolder 142 - found folder=barchart-udt/barchart-udt-netty4/./lib/bin/x86_64-MacOSX-gpp
13.673 [ main] INFO c.b.udt.lib.RES extractResource 100 -
already extracted;
sourcePath=/lib/x86_64-MacOSX-gpp/jni/libbarchart-udt-core-2.2.0-SNAPSHOT.jnilib
targetPath=./lib/bin/x86_64-MacOSX-gpp/libbarchart-udt-core-2.2.0-SNAPSHOT.jnilib
13.779 [ main] ERROR c.b.u.SocketUDT 129 - java/native SIGNATURE inconsistent; terminating

Results :

Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 17.553s
[INFO] Finished at: Tue Jan 01 13:12:13 EST 2013
[INFO] Final Memory: 21M/81M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test (default-test) on project barchart-udt-netty4: Execution default-test of goal org.apache.maven.plugins:maven-surefire-plugin:2.12.4:test failed: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ? -> [Help 1]

[srnm]

cd barchart-udt
mvn -U clean

fails with

[ERROR] Failed to execute goal on project barchart-udt-it-02-bundle-test: Could not resolve dependencies for project com.barchart.udt:barchart-udt-it-02-bundle-test:jar:0.0.0: Could not find artifact com.barchart.udt:barchart-udt-it-01-bundle:jar:0.0.0 in sonatype-nexus-archon (https://oss.sonatype.org/content/groups/public) -> [Help 1]

thanks again...

re: "Is barchart-udt-netty4 intended to be built as a standalone project?" yes, except:

bad timing for doing clone of master and building from scratch :-)

the commit above solved your original problem alright,
but made master incompatible with current snapshots :-)
(I was in the middle of netty cleanup).

also, if you want to build native stuff, you need full tool chain setup for you mac
https://github.com/barchart/barchart-udt/wiki/Build-System

I will post back when things stabilize.

[sequoiar]

any interest to try udt with node.js? node.js is a pretty net/web server.

https://github.com/InstantWebP2P/node-httpp/wiki/An-introduction-to-node-httpp

@srnm:

I finished/committed netty cleanup.

I also added barchart-udt-netty4 job to jenkins, so you need not build anything;

just add to your project sonatype repository similar to this:
https://github.com/barchart/barchart-udt/blob/master/barchart-udt-demo/pom.xml

if in doubt, check here:
https://oss.sonatype.org/content/groups/public/com/barchart/udt/barchart-udt-netty4/

@sequoiar: thank you for stopping by.

according to this benchmark
http://blog.evanweaver.com/2012/02/29/hello-heroku-world/

we should be happy with current netty-udt integration - performance wise.

in terms of compatibility, if you submit a pull request here with java test case
which connects netty-udt to node.js udt, we will incorporate it into our build.

this can be done with help of separate maven profile invoking
https://github.com/sirrapa/nodejs-maven-plugin

finally, do you know which udt version is used in node.js
and which udt arch/os combinations are available in node.js?

cheers.

[sequoiar]

@carrot-garden
for node.js with udt, udt version is v4.10. it works well on linux(x86, arm), windows. Mac should be working as well.

@sequoiar
we also build against udt version is v4.10.
do you know how is it built for arm?
this is what we are trying to do with arm now: #2

[sequoiar]

@carrot-garden

to compile node-httpp for arm, you need arm toolchain, then cross compile it.

@sequoiar
in other words, you do not run any unit tests in native environment?
btw, which arm os you run it on? android? rpi?

[srnm]

I am testing barchart-udt-netty4 as a naive user of the library on Linux, Mac and Windows.

I created a new maven project with a simple unit test based on TestNioUdtProvider.java
The pom must explicitly include barchart-udt-core, barchart-udt-bundle, netty-transport, and netty-handler because they are scoped as 'provided' via barchart-udt-netty4. Is the scope going to change to 'compile' in the release version?

The unit tests run on Linux and Windows.
There is a failure on Macintosh. I don't know much about JNI but happy to keep testing as instructed...

09:27:21.627 [main] INFO com.barchart.udt.SocketUDT - library location : ./lib/bin
09:27:21.627 [main] INFO com.barchart.udt.SocketUDT - library loader : com.barchart.udt.lib.LibraryLoaderDefaultUDT
09:27:21.628 [main] INFO com.barchart.udt.lib.AOL - OS_NAME=Mac OS X
09:27:21.628 [main] INFO com.barchart.udt.lib.AOL - OS_ARCH=x86_64
09:27:21.630 [main] INFO com.barchart.udt.lib.LibraryUDT - library source: NAR production
09:27:21.631 [main] WARN com.barchart.udt.lib.RES -
classpath resource not found: /lib/x86_64-MacOSX-gpp/jni/libbarchart-udt-core-2.2.0-SNAPSHOT.dylib
09:27:21.631 [main] WARN com.barchart.udt.lib.LibraryUDT -
IllegalArgumentException resource not found
09:27:21.631 [main] INFO com.barchart.udt.lib.LibraryUDT - library source: CDT testing
09:27:21.631 [main] WARN com.barchart.udt.lib.RES -
classpath resource not found: /libbarchart-udt-core-x86_64-MacOSX-gpp.dylib
09:27:21.631 [main] WARN com.barchart.udt.lib.LibraryUDT -
IllegalArgumentException resource not found
09:27:21.631 [main] INFO com.barchart.udt.lib.LibraryUDT - library source: NAR testing
09:27:21.632 [main] WARN com.barchart.udt.lib.RES -
classpath resource not found: /barchart-udt-core-2.2.0-SNAPSHOT-x86_64-MacOSX-gpp-jni//lib/x86_64-MacOSX-gpp/jni/libbarchart-udt-core-2.2.0-SNAPSHOT.dylib
09:27:21.632 [main] WARN com.barchart.udt.lib.LibraryUDT -
IllegalArgumentException resource not found
09:27:21.638 [main] ERROR com.barchart.udt.SocketUDT - failed to LOAD native library; terminating
java.lang.Exception: core library load failed
at com.barchart.udt.lib.LibraryUDT.loadCore(LibraryUDT.java:183) ~[barchart-udt-core-2.2.0-SNAPSHOT.jar:na]
at com.barchart.udt.lib.LibraryUDT.load(LibraryUDT.java:104) ~[barchart-udt-core-2.2.0-SNAPSHOT.jar:na]
at com.barchart.udt.lib.LibraryLoaderDefaultUDT.load(LibraryLoaderDefaultUDT.java:18) ~[barchart-udt-core-2.2.0-SNAPSHOT.jar:na]
at com.barchart.udt.SocketUDT.(SocketUDT.java:144) ~[barchart-udt-core-2.2.0-SNAPSHOT.jar:na]
at com.barchart.udt.nio.SelectorProviderUDT.openServerSocketChannel(SelectorProviderUDT.java:117) [barchart-udt-core-2.2.0-SNAPSHOT.jar:na]
at io.netty.channel.socket.nio.NioUdtProvider.newAcceptorChannelUDT(NioUdtProvider.java:134) [barchart-udt-netty4-1.0.0-SNAPSHOT.jar:na]
at io.netty.channel.socket.nio.NioUdtBaseAcceptorChannel.(NioUdtBaseAcceptorChannel.java:70) [barchart-udt-netty4-1.0.0-SNAPSHOT.jar:na]
at io.netty.channel.socket.nio.NioUdtByteAcceptorChannel.(NioUdtByteAcceptorChannel.java:29) [barchart-udt-netty4-1.0.0-SNAPSHOT.jar:na]
at io.netty.channel.socket.nio.NioUdtProvider.newChannel(NioUdtProvider.java:208) [barchart-udt-netty4-1.0.0-SNAPSHOT.jar:na]
at io.netty.channel.socket.nio.NioUdtProvider.newChannel(NioUdtProvider.java:42) [barchart-udt-netty4-1.0.0-SNAPSHOT.jar:na]

@srnm

1. re: "Is the scope going to change to 'compile' in the release version?""
   barchart-udt-netty4 is just temporary/staging project which helps to submit this pull request:
   netty/netty#867

after netty project accepts pull request, and udt becomes part of netty distribution
barchart-udt-netty4 is pretty much discarded. it should never be released.

2. re: "java.lang.Exception: core library load failed"
   a) do not include barchart-udt-core as dependency
   b) instead include barchart-udt-bundle

3. can you please look inside the barchart-udt-bundle JAR in your local maven repo:

${user.home}/.m2/repository/com/barchart/udt/barchart-udt-bundle/2.2.0-SNAPSHOT/barchart-udt-bundle-2.2.0-SNAPSHOT.jar

and verify that there is in fact dylib is inside the jar at this path:

/barchart-udt-core-2.2.0-SNAPSHOT-x86_64-MacOSX-gpp-jni//lib/x86_64-MacOSX-gpp/jni/libbarchart-udt-core-2.2.0-SNAPSHOT.dylib

@srnm
correction: path inside jar should be:

/lib/x86_64-MacOSX-gpp/jni/libbarchart-udt-core-2.2.0-SNAPSHOT.dylib

@srnm
make sure you can load bundle on mac similar to this project
https://github.com/barchart/barchart-udt/blob/master/barchart-udt-demo/pom.xml
w/o netty

[srnm]

The test project pom includes the sonatype snapshot repo as per barchart-udt-demo/pom.xml.

I removed the explicit dependency on barchart-udt-core, (leaving barchart-udt-bundle, netty-transport, and netty-handler ) and rebuilt on the Mac with "mvn -U clean test". It still fails.

So finally checked for the existence of /lib/x86_64-MacOSX-gpp/jni/libbarchart-udt-core-2.2.0-SNAPSHOT.dylib in ${user.home}/.m2/repository/com/barchart/udt/barchart-udt-bundle/2.2.0-SNAPSHOT/barchart-udt-bundle-2.2.0-SNAPSHOT.jar. It does not exist. Instead I found libbarchart-udt-core-2.2.0-SNAPSHOT.jnilib .

thanks!

[srnm]

I confirmed that

https://oss.sonatype.org/content/groups/public/com/barchart/udt/barchart-udt-bundle/2.2.0-SNAPSHOT/barchart-udt-bundle-2.2.0-20130101.212113-15.jar

does not contain .dylib.

ok, got it, thank you. little mix up jnilib vs dylib. will post back when ready.

@srnm please try again. now should use only *.jnilib throughout.

[srnm]

My test project now works across Linux, Mac and Windows using the latest snapshot.
Thanks!

great. do you mind sharing what is your use case for udt?

[sequoiar]

@carrot-garden

cross compiled node.js with udt on rpi

[srnm]

@carrot-garden my use case is accelerated file transfer.

I appreciate the effort you all have put in to patching the UDT sources, building the JNI layer, and providing netty support! Thanks again!

@sequoiar got it; thanks.

@srnm re: "accelerated file transfer" - cool.
in this case you may want to play with just implemented SocketUDT.sendFile()/receiveFile().
it does not work :-) so you may try to figure out why.
see test case ; report issues here: #5

[He-Pin]

i use this to exchange data betweet the peers faster~

netty accepted barchart udt as officially supported transport
https://github.com/barchart/barchart-udt/wiki/Version-History

[He-Pin]

wow~~!congratulation!thanks for your hard work,thanks all of you!