Coder Social home page Coder Social logo

containerised jvm's about jattach HOT 16 CLOSED

jattach avatar jattach commented on August 19, 2024
containerised jvm's

from jattach.

Comments (16)

apangin avatar apangin commented on August 19, 2024

There was a number of improvements in jattach specially for container support.

The tool automatcially switches to the mount namespace of the target process and automatically changes uid/gid to match the target process. Are you running jattach by root?

Please check if the container has /tmp/.java_pidXXX socket file.

One of the common problems with dynamic attach is scheduled tasks that clean /tmp and thus remove JVM's attach socket.

from jattach.

mrsiano avatar mrsiano commented on August 19, 2024

no it does not have the socket file at all.
and yes I'm running jattach with root user, the container runs root as well (privileged container)

I run the same java app (just a sleep app) on the docker host, and it create the socket file, but inside my container I can't see it at all (after wide search).

I've tried few things, and it's terms out like running java app inside a container will not create a socket file.

from jattach.

apangin avatar apangin commented on August 19, 2024

What if you start Java (in the container) with -XX:+StartAttachListener option? Will jattach work then?

from jattach.

apangin avatar apangin commented on August 19, 2024

Can you share the output of strace jattach ... ?

from jattach.

mrsiano avatar mrsiano commented on August 19, 2024
~/jattach/build # strace ./jattach 1235 threaddump
execve("./jattach", ["./jattach", "1235", "threaddump"], [/* 28 vars */]) = 0
brk(NULL)                               = 0x25ce000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa37bd3e000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=37529, ...}) = 0
mmap(NULL, 37529, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fa37bd34000
close(3)                                = 0
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340$\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=2151832, ...}) = 0
mmap(NULL, 3981792, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fa37b751000
mprotect(0x7fa37b914000, 2093056, PROT_NONE) = 0
mmap(0x7fa37bb13000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c2000) = 0x7fa37bb13000
mmap(0x7fa37bb19000, 16864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fa37bb19000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa37bd33000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa37bd31000
arch_prctl(ARCH_SET_FS, 0x7fa37bd31740) = 0
mprotect(0x7fa37bb13000, 16384, PROT_READ) = 0
mprotect(0x601000, 4096, PROT_READ)     = 0
mprotect(0x7fa37bd3f000, 4096, PROT_READ) = 0
munmap(0x7fa37bd34000, 37529)           = 0
geteuid()                               = 0
getegid()                               = 0
brk(NULL)                               = 0x25ce000
brk(0x25ef000)                          = 0x25ef000
brk(NULL)                               = 0x25ef000
open("/proc/1235/status", O_RDONLY)     = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa37bd3d000
read(3, "Name:\tjava\nUmask:\t0022\nState:\tS "..., 1024) = 1024
read(3, "00000,00000000,00000000,00000000"..., 1024) = 120
read(3, "", 1024)                       = 0
close(3)                                = 0
munmap(0x7fa37bd3d000, 4096)            = 0
open("/proc/self/ns/mnt", O_RDONLY)     = 3
open("/proc/1235/ns/mnt", O_RDONLY)     = 4
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
fstat(4, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
setns(4, 0)                             = 0
rt_sigaction(SIGPIPE, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7fa37b787280}, {SIG_DFL, [], 0}, 8) = 0
stat("/tmp/.java_pid1235", 0x7fffef52bae0) = -1 ENOENT (No such file or directory)
creat("/proc/1235/cwd/.attach_pid1235", 0660) = -1 ENOENT (No such file or directory)
creat("/tmp/.attach_pid1235", 0660)     = 5
close(5)                                = 0
kill(1235, SIGQUIT)                     = 0
nanosleep({0, 100000000}, NULL)         = 0
stat("/tmp/.java_pid1235", 0x7fffef52b620) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL)         = 0
stat("/tmp/.java_pid1235", 0x7fffef52b620) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL)         = 0
stat("/tmp/.java_pid1235", 0x7fffef52b620) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL)         = 0
stat("/tmp/.java_pid1235", 0x7fffef52b620) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL)         = 0
stat("/tmp/.java_pid1235", 0x7fffef52b620) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL)         = 0
stat("/tmp/.java_pid1235", 0x7fffef52b620) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL)         = 0
stat("/tmp/.java_pid1235", 0x7fffef52b620) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL)         = 0
stat("/tmp/.java_pid1235", 0x7fffef52b620) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL)         = 0
stat("/tmp/.java_pid1235", 0x7fffef52b620) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL)         = 0
stat("/tmp/.java_pid1235", 0x7fffef52b620) = -1 ENOENT (No such file or directory)
unlink("/tmp/.attach_pid1235")          = 0
dup(2)                                  = 5
fcntl(5, F_GETFL)                       = 0x8002 (flags O_RDWR|O_LARGEFILE)
fstat(5, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 7), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa37bd3d000
write(5, "Could not start attach mechanism"..., 60Could not start attach mechanism: No such file or directory
) = 60
close(5)                                = 0
munmap(0x7fa37bd3d000, 4096)            = 0
exit_group(1)                           = ?
+++ exited with 1 +++

from jattach.

apangin avatar apangin commented on August 19, 2024

Thanks. It gets a bit more clear. Either jattach does not obtain correct process info, or dynamic attach is disabled in the target jvm.

  1. Please share /proc/PID/status if possible, where PID is the host process id of the target jvm.
  2. Check if jattach works if the jvm runs with -XX:+StartAttachListener command line option.

from jattach.

mrsiano avatar mrsiano commented on August 19, 2024

Sure,
-XX:+StartAttachListener did not work.

see the pid status

~/jattach/build # cat /proc/16998/status
Name:	java
Umask:	0022
State:	S (sleeping)
Tgid:	16998
Ngid:	0
Pid:	16998
PPid:	1
TracerPid:	0
Uid:	0	0	0	0
Gid:	0	0	0	0
FDSize:	256
Groups:
VmPeak:	 6401680 kB
VmSize:	 6401680 kB
VmLck:	       0 kB
VmPin:	       0 kB
VmHWM:	   18736 kB
VmRSS:	   18736 kB
RssAnon:	    8288 kB
RssFile:	   10448 kB
RssShmem:	       0 kB
VmData:	 6348988 kB
VmStk:	     132 kB
VmExe:	       4 kB
VmLib:	   17612 kB
VmPTE:	     272 kB
VmSwap:	       0 kB
Threads:	15
SigQ:	0/61513
SigPnd:	0000000000000000
ShdPnd:	0000000000000000
SigBlk:	0000000000000000
SigIgn:	0000000000000001
SigCgt:	0000000181005cce
CapInh:	0000001fffffffff
CapPrm:	0000001fffffffff
CapEff:	0000001fffffffff
CapBnd:	0000001fffffffff
CapAmb:	0000000000000000
Seccomp:	0
Cpus_allowed:	f
Cpus_allowed_list:	0-3
Mems_allowed:	00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list:	0
voluntary_ctxt_switches:	2
nonvoluntary_ctxt_switches:	4

and strace

~/jattach/build # strace ./jattach 16998 threaddump
execve("./jattach", ["./jattach", "16998", "threaddump"], [/* 28 vars */]) = 0
brk(NULL)                               = 0x17cd000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f19fb0e2000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=37529, ...}) = 0
mmap(NULL, 37529, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f19fb0d8000
close(3)                                = 0
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340$\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=2151832, ...}) = 0
mmap(NULL, 3981792, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f19faaf5000
mprotect(0x7f19facb8000, 2093056, PROT_NONE) = 0
mmap(0x7f19faeb7000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c2000) = 0x7f19faeb7000
mmap(0x7f19faebd000, 16864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f19faebd000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f19fb0d7000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f19fb0d5000
arch_prctl(ARCH_SET_FS, 0x7f19fb0d5740) = 0
mprotect(0x7f19faeb7000, 16384, PROT_READ) = 0
mprotect(0x601000, 4096, PROT_READ)     = 0
mprotect(0x7f19fb0e3000, 4096, PROT_READ) = 0
munmap(0x7f19fb0d8000, 37529)           = 0
geteuid()                               = 0
getegid()                               = 0
brk(NULL)                               = 0x17cd000
brk(0x17ee000)                          = 0x17ee000
brk(NULL)                               = 0x17ee000
open("/proc/16998/status", O_RDONLY)    = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f19fb0e1000
read(3, "Name:\tjava\nUmask:\t0022\nState:\tS "..., 1024) = 1024
read(3, "00000,00000000,00000000,00000000"..., 1024) = 120
read(3, "", 1024)                       = 0
close(3)                                = 0
munmap(0x7f19fb0e1000, 4096)            = 0
open("/proc/self/ns/mnt", O_RDONLY)     = 3
open("/proc/16998/ns/mnt", O_RDONLY)    = 4
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
fstat(4, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
setns(4, 0)                             = 0
rt_sigaction(SIGPIPE, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f19fab2b280}, {SIG_DFL, [], 0}, 8) = 0
stat("/tmp/.java_pid16998", 0x7ffd494442d0) = -1 ENOENT (No such file or directory)
creat("/proc/16998/cwd/.attach_pid16998", 0660) = -1 ENOENT (No such file or directory)
creat("/tmp/.attach_pid16998", 0660)    = 5
close(5)                                = 0
kill(16998, SIGQUIT)                    = 0
nanosleep({0, 100000000}, NULL)         = 0
stat("/tmp/.java_pid16998", 0x7ffd49443e10) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL)         = 0
stat("/tmp/.java_pid16998", 0x7ffd49443e10) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL)         = 0
stat("/tmp/.java_pid16998", 0x7ffd49443e10) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL)         = 0
stat("/tmp/.java_pid16998", 0x7ffd49443e10) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL)         = 0
stat("/tmp/.java_pid16998", 0x7ffd49443e10) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL)         = 0
stat("/tmp/.java_pid16998", 0x7ffd49443e10) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL)         = 0
stat("/tmp/.java_pid16998", 0x7ffd49443e10) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL)         = 0
stat("/tmp/.java_pid16998", 0x7ffd49443e10) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL)         = 0
stat("/tmp/.java_pid16998", 0x7ffd49443e10) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL)         = 0
stat("/tmp/.java_pid16998", 0x7ffd49443e10) = -1 ENOENT (No such file or directory)
unlink("/tmp/.attach_pid16998")         = 0
dup(2)                                  = 5
fcntl(5, F_GETFL)                       = 0x8002 (flags O_RDWR|O_LARGEFILE)
fstat(5, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f19fb0e1000
write(5, "Could not start attach mechanism"..., 60Could not start attach mechanism: No such file or directory
) = 60
close(5)                                = 0
munmap(0x7f19fb0e1000, 4096)            = 0
exit_group(1)                           = ?
+++ exited with 1 +++

from jattach.

apangin avatar apangin commented on August 19, 2024

strace and /proc/pid/status look sane. The problem is that target JVM does not create attach socket.

In case JVM is started with -XX:+StartAttachListener option, it should automatically create attach socket at /tmp/.java_pid<PID>. If this does not happen, the problem seems to be on JVM or OS side. I am afraid jattach can do nothing with this.

Make sure you use OpenJDK or Oracle JDK, attach mechanism is not disabled (e.g. by -XX:+DisableAttachMechanism option), and /tmp allows creation of UNIX domain sockets.

from jattach.

johnpbatty avatar johnpbatty commented on August 19, 2024

Thanks for the great tool!

I just hit the same problem, and think I have identified the underlying cause.

I am running CentOS7.4.1708, kernel version:

$ uname -r
3.10.0-693.17.1.el7.x86_64

The host VM pid of my JVM process is 27158, and within my target container the JVM pid is 11.

I added some debugging printfs to jattach, and noticed that the pid used by jattach for the filename to trigger the JVM attach was the host VM pid (27518), not the container pid (11). Obviously the target JVM will be looking for the container pid.

Looking at the strace above, you can see that the same is happening there...

// Host VM pid is 16998
execve("./jattach", ["./jattach", "16998", "threaddump"], [/* 28 vars */]) = 0
...
// Same pid used in the .attach_pid<pid> filename.  This should be the container pid.
creat("/proc/16998/cwd/.attach_pid16998", 0660) = -1 ENOENT (No such file or directory)

Looking at the jattach code, I can see it is trying to do the right thing - it uses the variable that should contain the container pid: nspid.

The fundamental problem is that it is failing to obtain the correct value for nspid.

The jattach code for obtaining nspid is in get_process_info, where it reads /proc//status and looks for the field NStgid:

        } else if (strncmp(line, "NStgid:", 7) == 0) {
            // PID namespaces can be nested; the last one is the innermost one
            *nspid = atoi(strrchr(line, '\t'));
        }

Looking in my /proc//status file, I see that it does not contain this field! :-(
The main() function initializes nspid to be the value of pid, so if the field is not found it uses the (incorrect) host VM pid.

$ cat /proc/27158/status
Name:	java
Umask:	0022
State:	S (sleeping)
Tgid:	27158
Ngid:	0
Pid:	27158
PPid:	27135
TracerPid:	0
Uid:	0	0	0	0
Gid:	0	0	0	0
FDSize:	256
Groups:	0 1 2 3 4 6 10 11 20 26 27 
VmPeak:	10147776 kB
VmSize:	10139924 kB
VmLck:	       0 kB
VmPin:	       0 kB
VmHWM:	  380108 kB
VmRSS:	  275252 kB
RssAnon:	  253412 kB
RssFile:	   21840 kB
RssShmem:	       0 kB
VmData:	10057160 kB
VmStk:	     136 kB
VmExe:	       4 kB
VmLib:	   72160 kB
VmPTE:	    1112 kB
VmSwap:	       0 kB
Threads:	74
SigQ:	0/127960
SigPnd:	0000000000000000
ShdPnd:	0000000000000000
SigBlk:	0000000000000000
SigIgn:	0000000200001000
SigCgt:	2000000001004ccf
CapInh:	00000000a80425fb
CapPrm:	00000000a80425fb
CapEff:	00000000a80425fb
CapBnd:	00000000a80425fb
CapAmb:	0000000000000000
Seccomp:	0
Cpus_allowed:	ff
Cpus_allowed_list:	0-7
Mems_allowed:	00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list:	0
voluntary_ctxt_switches:	25
nonvoluntary_ctxt_switches:	16

It therefore looks like a different approach is needed to obtain the nspid on this kernel version. I will do some investigation to see if I can work out how to get the required value.

@apangin What OS/kernel version have you tested with?
@mrsiano What OS/kernel version were you running?

To confirm that this is the issue, I hacked my jattach code to hard-code the value of nspid to the correct value, and this did indeed get the tool working!

from jattach.

johnpbatty avatar johnpbatty commented on August 19, 2024

From http://man7.org/linux/man-pages/man5/proc.5.html

              * NStgid : Thread group ID (i.e., PID) in each of the PID
                namespaces of which [pid] is a member.  The leftmost entry
                shows the value with respect to the PID namespace of the
                reading process, followed by the value in successively
                nested inner namespaces.  (Since Linux 4.1.)

Which explains why it does not work with my 3.10 kernel...

from jattach.

apangin avatar apangin commented on August 19, 2024

@johnpbatty Thank you for a great analysis!
I also noticed that NStgid is missing in /proc/pid/status, but incorrectly thought that it means container pid == host pid.
Indeed kernel version is the clue. I tested on Linux 4.4 and newer, and it worked fine.

I think the workaround will be to call setns to change PID namespace in the same way jattach does it for mount namespace. Will try this shortly.

from jattach.

johnpbatty avatar johnpbatty commented on August 19, 2024

It does look like a tricky problem to solve - I haven't found an easy / reliable way to do it.

In this thread, the final comment observes that the host VM PID is accessible within the container within /proc/<nspid>/sched. Indeed it is (although his grep command did not work for me), but to access this you need to know the container id, and it is pretty ugly.

I found another thread which seems to have a better solution - getting the kernel to translate the PID by sending it via credentials between the two namespaces. Although sounds like a fair bit of effort to implement.

I did come up with a method that works for me, using nsenter. But this does rely on having nsenter installed and may not work in all cases.

// 27158 is my host pid, 11 is the nspid
$ sudo nsenter --target 27158 --mount ps | grep java | awk '{print $1}'
11

A short term workaround might be to implement an optional --nspid flag to jattach to allow the user to specify the nspid directly. This would work for me, as I could either use the command above to obtain it, or in the majority of cases I know that my java app has PID 1 within the container.

from jattach.

apangin avatar apangin commented on August 19, 2024

I did some research on the issue, and discovered that setns + sending credentials does not work on Linux 3.10 - the host pid is not translated.

Fortunately, /proc/pid/sched trick works, so the solution would be to browse all pids in the namespace of the target process, trying to find which /proc/pid/sched points back to the host pid.

A good news is that setns() is not even needed in this case, since /proc/hostpid/root/proc allows to browse processes in the target namespace without switching into it.

I've implemented this idea in container branch. Please check if it works for you.

from jattach.

mrsiano avatar mrsiano commented on August 19, 2024

@apangin yep thats the solution,
I using nsenter since few days ago to grab the statistics.
https://github.com/mrsiano/pbench/blob/ea6900727eefd0926b80602d4d2d8d271bb1e508/agent/tool-scripts/jmap#L51-L61

thanks from your time and effort I'll try you work later on

from jattach.

johnpbatty avatar johnpbatty commented on August 19, 2024

@apangin I tried out your container branch and can confirm that it works for me. Awesome :-)

from jattach.

apangin avatar apangin commented on August 19, 2024

Thanks! I've merged the fix into master.

from jattach.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.