Comments (16)
There was a number of improvements in jattach specially for container support.
The tool automatcially switches to the mount namespace of the target process and automatically changes uid/gid to match the target process. Are you running jattach by root?
Please check if the container has /tmp/.java_pidXXX
socket file.
One of the common problems with dynamic attach is scheduled tasks that clean /tmp and thus remove JVM's attach socket.
from jattach.
no it does not have the socket file at all.
and yes I'm running jattach with root user, the container runs root as well (privileged container)
I run the same java app (just a sleep app) on the docker host, and it create the socket file, but inside my container I can't see it at all (after wide search).
I've tried few things, and it's terms out like running java app inside a container will not create a socket file.
from jattach.
What if you start Java (in the container) with -XX:+StartAttachListener
option? Will jattach work then?
from jattach.
Can you share the output of strace jattach ...
?
from jattach.
~/jattach/build # strace ./jattach 1235 threaddump
execve("./jattach", ["./jattach", "1235", "threaddump"], [/* 28 vars */]) = 0
brk(NULL) = 0x25ce000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa37bd3e000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=37529, ...}) = 0
mmap(NULL, 37529, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fa37bd34000
close(3) = 0
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340$\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=2151832, ...}) = 0
mmap(NULL, 3981792, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fa37b751000
mprotect(0x7fa37b914000, 2093056, PROT_NONE) = 0
mmap(0x7fa37bb13000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c2000) = 0x7fa37bb13000
mmap(0x7fa37bb19000, 16864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fa37bb19000
close(3) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa37bd33000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa37bd31000
arch_prctl(ARCH_SET_FS, 0x7fa37bd31740) = 0
mprotect(0x7fa37bb13000, 16384, PROT_READ) = 0
mprotect(0x601000, 4096, PROT_READ) = 0
mprotect(0x7fa37bd3f000, 4096, PROT_READ) = 0
munmap(0x7fa37bd34000, 37529) = 0
geteuid() = 0
getegid() = 0
brk(NULL) = 0x25ce000
brk(0x25ef000) = 0x25ef000
brk(NULL) = 0x25ef000
open("/proc/1235/status", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa37bd3d000
read(3, "Name:\tjava\nUmask:\t0022\nState:\tS "..., 1024) = 1024
read(3, "00000,00000000,00000000,00000000"..., 1024) = 120
read(3, "", 1024) = 0
close(3) = 0
munmap(0x7fa37bd3d000, 4096) = 0
open("/proc/self/ns/mnt", O_RDONLY) = 3
open("/proc/1235/ns/mnt", O_RDONLY) = 4
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
fstat(4, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
setns(4, 0) = 0
rt_sigaction(SIGPIPE, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7fa37b787280}, {SIG_DFL, [], 0}, 8) = 0
stat("/tmp/.java_pid1235", 0x7fffef52bae0) = -1 ENOENT (No such file or directory)
creat("/proc/1235/cwd/.attach_pid1235", 0660) = -1 ENOENT (No such file or directory)
creat("/tmp/.attach_pid1235", 0660) = 5
close(5) = 0
kill(1235, SIGQUIT) = 0
nanosleep({0, 100000000}, NULL) = 0
stat("/tmp/.java_pid1235", 0x7fffef52b620) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL) = 0
stat("/tmp/.java_pid1235", 0x7fffef52b620) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL) = 0
stat("/tmp/.java_pid1235", 0x7fffef52b620) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL) = 0
stat("/tmp/.java_pid1235", 0x7fffef52b620) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL) = 0
stat("/tmp/.java_pid1235", 0x7fffef52b620) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL) = 0
stat("/tmp/.java_pid1235", 0x7fffef52b620) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL) = 0
stat("/tmp/.java_pid1235", 0x7fffef52b620) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL) = 0
stat("/tmp/.java_pid1235", 0x7fffef52b620) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL) = 0
stat("/tmp/.java_pid1235", 0x7fffef52b620) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL) = 0
stat("/tmp/.java_pid1235", 0x7fffef52b620) = -1 ENOENT (No such file or directory)
unlink("/tmp/.attach_pid1235") = 0
dup(2) = 5
fcntl(5, F_GETFL) = 0x8002 (flags O_RDWR|O_LARGEFILE)
fstat(5, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 7), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fa37bd3d000
write(5, "Could not start attach mechanism"..., 60Could not start attach mechanism: No such file or directory
) = 60
close(5) = 0
munmap(0x7fa37bd3d000, 4096) = 0
exit_group(1) = ?
+++ exited with 1 +++
from jattach.
Thanks. It gets a bit more clear. Either jattach does not obtain correct process info, or dynamic attach is disabled in the target jvm.
- Please share /proc/PID/status if possible, where PID is the host process id of the target jvm.
- Check if jattach works if the jvm runs with
-XX:+StartAttachListener
command line option.
from jattach.
Sure,
-XX:+StartAttachListener
did not work.
see the pid status
~/jattach/build # cat /proc/16998/status
Name: java
Umask: 0022
State: S (sleeping)
Tgid: 16998
Ngid: 0
Pid: 16998
PPid: 1
TracerPid: 0
Uid: 0 0 0 0
Gid: 0 0 0 0
FDSize: 256
Groups:
VmPeak: 6401680 kB
VmSize: 6401680 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 18736 kB
VmRSS: 18736 kB
RssAnon: 8288 kB
RssFile: 10448 kB
RssShmem: 0 kB
VmData: 6348988 kB
VmStk: 132 kB
VmExe: 4 kB
VmLib: 17612 kB
VmPTE: 272 kB
VmSwap: 0 kB
Threads: 15
SigQ: 0/61513
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000000001
SigCgt: 0000000181005cce
CapInh: 0000001fffffffff
CapPrm: 0000001fffffffff
CapEff: 0000001fffffffff
CapBnd: 0000001fffffffff
CapAmb: 0000000000000000
Seccomp: 0
Cpus_allowed: f
Cpus_allowed_list: 0-3
Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 2
nonvoluntary_ctxt_switches: 4
and strace
~/jattach/build # strace ./jattach 16998 threaddump
execve("./jattach", ["./jattach", "16998", "threaddump"], [/* 28 vars */]) = 0
brk(NULL) = 0x17cd000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f19fb0e2000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=37529, ...}) = 0
mmap(NULL, 37529, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f19fb0d8000
close(3) = 0
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340$\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=2151832, ...}) = 0
mmap(NULL, 3981792, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f19faaf5000
mprotect(0x7f19facb8000, 2093056, PROT_NONE) = 0
mmap(0x7f19faeb7000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c2000) = 0x7f19faeb7000
mmap(0x7f19faebd000, 16864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f19faebd000
close(3) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f19fb0d7000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f19fb0d5000
arch_prctl(ARCH_SET_FS, 0x7f19fb0d5740) = 0
mprotect(0x7f19faeb7000, 16384, PROT_READ) = 0
mprotect(0x601000, 4096, PROT_READ) = 0
mprotect(0x7f19fb0e3000, 4096, PROT_READ) = 0
munmap(0x7f19fb0d8000, 37529) = 0
geteuid() = 0
getegid() = 0
brk(NULL) = 0x17cd000
brk(0x17ee000) = 0x17ee000
brk(NULL) = 0x17ee000
open("/proc/16998/status", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f19fb0e1000
read(3, "Name:\tjava\nUmask:\t0022\nState:\tS "..., 1024) = 1024
read(3, "00000,00000000,00000000,00000000"..., 1024) = 120
read(3, "", 1024) = 0
close(3) = 0
munmap(0x7f19fb0e1000, 4096) = 0
open("/proc/self/ns/mnt", O_RDONLY) = 3
open("/proc/16998/ns/mnt", O_RDONLY) = 4
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
fstat(4, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
setns(4, 0) = 0
rt_sigaction(SIGPIPE, {SIG_IGN, [PIPE], SA_RESTORER|SA_RESTART, 0x7f19fab2b280}, {SIG_DFL, [], 0}, 8) = 0
stat("/tmp/.java_pid16998", 0x7ffd494442d0) = -1 ENOENT (No such file or directory)
creat("/proc/16998/cwd/.attach_pid16998", 0660) = -1 ENOENT (No such file or directory)
creat("/tmp/.attach_pid16998", 0660) = 5
close(5) = 0
kill(16998, SIGQUIT) = 0
nanosleep({0, 100000000}, NULL) = 0
stat("/tmp/.java_pid16998", 0x7ffd49443e10) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL) = 0
stat("/tmp/.java_pid16998", 0x7ffd49443e10) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL) = 0
stat("/tmp/.java_pid16998", 0x7ffd49443e10) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL) = 0
stat("/tmp/.java_pid16998", 0x7ffd49443e10) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL) = 0
stat("/tmp/.java_pid16998", 0x7ffd49443e10) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL) = 0
stat("/tmp/.java_pid16998", 0x7ffd49443e10) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL) = 0
stat("/tmp/.java_pid16998", 0x7ffd49443e10) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL) = 0
stat("/tmp/.java_pid16998", 0x7ffd49443e10) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL) = 0
stat("/tmp/.java_pid16998", 0x7ffd49443e10) = -1 ENOENT (No such file or directory)
nanosleep({0, 100000000}, NULL) = 0
stat("/tmp/.java_pid16998", 0x7ffd49443e10) = -1 ENOENT (No such file or directory)
unlink("/tmp/.attach_pid16998") = 0
dup(2) = 5
fcntl(5, F_GETFL) = 0x8002 (flags O_RDWR|O_LARGEFILE)
fstat(5, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f19fb0e1000
write(5, "Could not start attach mechanism"..., 60Could not start attach mechanism: No such file or directory
) = 60
close(5) = 0
munmap(0x7f19fb0e1000, 4096) = 0
exit_group(1) = ?
+++ exited with 1 +++
from jattach.
strace and /proc/pid/status look sane. The problem is that target JVM does not create attach socket.
In case JVM is started with -XX:+StartAttachListener
option, it should automatically create attach socket at /tmp/.java_pid<PID>
. If this does not happen, the problem seems to be on JVM or OS side. I am afraid jattach can do nothing with this.
Make sure you use OpenJDK or Oracle JDK, attach mechanism is not disabled (e.g. by -XX:+DisableAttachMechanism
option), and /tmp
allows creation of UNIX domain sockets.
from jattach.
Thanks for the great tool!
I just hit the same problem, and think I have identified the underlying cause.
I am running CentOS7.4.1708, kernel version:
$ uname -r
3.10.0-693.17.1.el7.x86_64
The host VM pid of my JVM process is 27158, and within my target container the JVM pid is 11.
I added some debugging printfs to jattach, and noticed that the pid used by jattach for the filename to trigger the JVM attach was the host VM pid (27518), not the container pid (11). Obviously the target JVM will be looking for the container pid.
Looking at the strace above, you can see that the same is happening there...
// Host VM pid is 16998
execve("./jattach", ["./jattach", "16998", "threaddump"], [/* 28 vars */]) = 0
...
// Same pid used in the .attach_pid<pid> filename. This should be the container pid.
creat("/proc/16998/cwd/.attach_pid16998", 0660) = -1 ENOENT (No such file or directory)
Looking at the jattach code, I can see it is trying to do the right thing - it uses the variable that should contain the container pid: nspid
.
The fundamental problem is that it is failing to obtain the correct value for nspid
.
The jattach code for obtaining nspid is in get_process_info
, where it reads /proc//status and looks for the field NStgid
:
} else if (strncmp(line, "NStgid:", 7) == 0) {
// PID namespaces can be nested; the last one is the innermost one
*nspid = atoi(strrchr(line, '\t'));
}
Looking in my /proc//status file, I see that it does not contain this field! :-(
The main() function initializes nspid to be the value of pid, so if the field is not found it uses the (incorrect) host VM pid.
$ cat /proc/27158/status
Name: java
Umask: 0022
State: S (sleeping)
Tgid: 27158
Ngid: 0
Pid: 27158
PPid: 27135
TracerPid: 0
Uid: 0 0 0 0
Gid: 0 0 0 0
FDSize: 256
Groups: 0 1 2 3 4 6 10 11 20 26 27
VmPeak: 10147776 kB
VmSize: 10139924 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 380108 kB
VmRSS: 275252 kB
RssAnon: 253412 kB
RssFile: 21840 kB
RssShmem: 0 kB
VmData: 10057160 kB
VmStk: 136 kB
VmExe: 4 kB
VmLib: 72160 kB
VmPTE: 1112 kB
VmSwap: 0 kB
Threads: 74
SigQ: 0/127960
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000200001000
SigCgt: 2000000001004ccf
CapInh: 00000000a80425fb
CapPrm: 00000000a80425fb
CapEff: 00000000a80425fb
CapBnd: 00000000a80425fb
CapAmb: 0000000000000000
Seccomp: 0
Cpus_allowed: ff
Cpus_allowed_list: 0-7
Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 25
nonvoluntary_ctxt_switches: 16
It therefore looks like a different approach is needed to obtain the nspid on this kernel version. I will do some investigation to see if I can work out how to get the required value.
@apangin What OS/kernel version have you tested with?
@mrsiano What OS/kernel version were you running?
To confirm that this is the issue, I hacked my jattach code to hard-code the value of nspid to the correct value, and this did indeed get the tool working!
from jattach.
From http://man7.org/linux/man-pages/man5/proc.5.html
* NStgid : Thread group ID (i.e., PID) in each of the PID
namespaces of which [pid] is a member. The leftmost entry
shows the value with respect to the PID namespace of the
reading process, followed by the value in successively
nested inner namespaces. (Since Linux 4.1.)
Which explains why it does not work with my 3.10 kernel...
from jattach.
@johnpbatty Thank you for a great analysis!
I also noticed that NStgid
is missing in /proc/pid/status, but incorrectly thought that it means container pid == host pid.
Indeed kernel version is the clue. I tested on Linux 4.4 and newer, and it worked fine.
I think the workaround will be to call setns
to change PID namespace in the same way jattach does it for mount namespace. Will try this shortly.
from jattach.
It does look like a tricky problem to solve - I haven't found an easy / reliable way to do it.
In this thread, the final comment observes that the host VM PID is accessible within the container within /proc/<nspid>/sched. Indeed it is (although his grep command did not work for me), but to access this you need to know the container id, and it is pretty ugly.
I found another thread which seems to have a better solution - getting the kernel to translate the PID by sending it via credentials between the two namespaces. Although sounds like a fair bit of effort to implement.
I did come up with a method that works for me, using nsenter. But this does rely on having nsenter installed and may not work in all cases.
// 27158 is my host pid, 11 is the nspid
$ sudo nsenter --target 27158 --mount ps | grep java | awk '{print $1}'
11
A short term workaround might be to implement an optional --nspid
flag to jattach
to allow the user to specify the nspid
directly. This would work for me, as I could either use the command above to obtain it, or in the majority of cases I know that my java app has PID 1 within the container.
from jattach.
I did some research on the issue, and discovered that setns + sending credentials does not work on Linux 3.10 - the host pid is not translated.
Fortunately, /proc/pid/sched trick works, so the solution would be to browse all pids in the namespace of the target process, trying to find which /proc/pid/sched points back to the host pid.
A good news is that setns() is not even needed in this case, since /proc/hostpid/root/proc
allows to browse processes in the target namespace without switching into it.
I've implemented this idea in container branch. Please check if it works for you.
from jattach.
@apangin yep thats the solution,
I using nsenter since few days ago to grab the statistics.
https://github.com/mrsiano/pbench/blob/ea6900727eefd0926b80602d4d2d8d271bb1e508/agent/tool-scripts/jmap#L51-L61
thanks from your time and effort I'll try you work later on
from jattach.
@apangin I tried out your container branch and can confirm that it works for me. Awesome :-)
from jattach.
Thanks! I've merged the fix into master.
from jattach.
Related Issues (20)
- Example to Dump Heap on Remote Machine HOT 1
- Error with IBM J9 HOT 1
- Whether to support perf-map-agent in different containers? HOT 6
- JVM did not respond: Resource temporarily unavailable HOT 7
- jcmd in IBM java 8 OpenJ9 HOT 3
- ARM unable to use jattach. HOT 3
- Attach is not supported by the target process (error code = 100) HOT 5
- Hang when attaching on Winapi ConnectNamedPipe HOT 1
- Winapi CreateNamedPipe return value should be INVALID_HANDLE_VALUE not NULL HOT 1
- Process's guid changed after attach operation using jattach as library in linux HOT 1
- Support ARM64 HOT 5
- Does jattach work with a JRE (not a JDK)? HOT 2
- Android aarch64 not working HOT 2
- Could jattach enable remote debug dynamically? HOT 3
- Attach to OpenJ9 VM cause Assertion failed on Windows HOT 1
- Could you please clarify if jattach has been tested to work with a JRE on Windows ? HOT 1
- JVM did not respond: Resource temporarily unavailable HOT 2
- why load command need double read in jattach_hotspot.c? HOT 2
- Attaching to a hotspot VM in container may fail in `start_attach_mechanism` HOT 5
- Support for jmap -live HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from jattach.