glXChooseVisual always returns NULL about glsl-debugger

[napcode]

Do you happen to run on an optimus configuration?

[singron]

No, it's not optimus. It's a desktop. GeForce GTX 680 and Intel 3770K. I was able to reproduce it today with updated packages.

[sosswald]

I had the same issue and found two reasons:

1. The NVIDIA driver uses libglvnd since nvidia-361 for dispatching GL, GLX, and EGL calls to different vendor-specific libraries depending on what library it deems most suitable for the given display device. NVIDIA's libGLX.so is a wrapper library that first calls __glx_Main(...) on all known vendor libraries (libGLX_nvidia.so, libGLX_indirect.so, ...) to get the library's function table and version information. Afterwards, libGLX.so dispatches all other function calls to one of the vendor libraries. The GLSL-Debugger interferes with this mechanism by intercepting the call to __glx_Main(...) and bouncing it back to the dispatcher libGLX.so, where the function is undefined and a dynamically created stub method answers NULL. As the dispatcher library does not recognize any vendor library, all subsequent GLX calls fail. The same happens for libEGL.so and __egl_Main(...). One possibility to patch this issue would be to pass calls to these methods to the original dlsym():

Index: GLSL-Debugger/glsldb/DebugLib/libglsldebug.c
===================================================================
--- GLSL-Debugger/glsldb/DebugLib/libglsldebug.c	2017-09-18 14:38:42.000000000 +0200
+++ GLSL-Debugger/glsldb/DebugLib/libglsldebug.c	2017-09-22 16:37:33.063630891 +0200
@@ -1321,6 +1321,11 @@
 		unsetenv("GLSL_DEBUGGER_DLSYM");
 	}
 
+	// Call original dlsym for internal methods of NVIDIA's libglvnd dispatcher
+	if (strcmp(symbol, "__glx_Main") == 0 || strncmp(symbol, "__egl_Main", 10) == 0) {
+		return g.origdlsym(handle, symbol);
+	}
+
 	if (g.initialized) {
 		sym = (void*)glXGetProcAddressHook((GLubyte *)symbol);
 		if (sym) {

2. The name of libGL.so is hardcoded in glsldb/DebugLib/libglsldebug.c. On my Ubuntu 14.04 system, this is a problem as for some reason ld-linux.so resolves libGL.so to the Mesa library (from libgl1-mesa-dev), but libGL.so.1 is resolved to NVIDIA's GLVND dispatcher:

$ ldconfig -p | grep x86-64.*libGL.so
	libGL.so.1 (libc6,x86-64) => /usr/lib/nvidia-375/libGL.so.1 -> libGL.so.1.0.0
	libGL.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libGL.so -> mesa/libGL.so.1.2.0
	libGL.so (libc6,x86-64) => /usr/lib/nvidia-375/libGL.so -> libGL.so.1.0.0

When I run glxgears outside the debugger, libGL.so.1 resolves to NVIDIA's dispatcher and everything is fine. When I run glxgears inside the debugger, the hard-coded libGL.so resolves to Mesa while libGLX.so.0 still resolves to NVIDIA's dispatcher, which obviously causes trouble. Disabling USE_DLSYM_HARDCODED_LIB works for me, but there's probably a reason for hard-coding libGL.so and disabling the flag might have side effects on other systems.

[golightlyb]

I was having a similar problem

glXQueryExtensionsString -> returns NULL
glXChooseFBConfig -> returns NULL or count 0

I also use the NVIDIA driver on Linux.

I can confirm the above post solved my issue - rebuilding after patching the code as indicated and also commenting out at the top of the file.

//#define USE_DLSYM_HARDCODED_LIB

[10110111]

I think the best solution would be to avoid dlsymming private symbols at all. Otherwise we can get failures to e.g. load DRI drivers, which I got on Ubuntu 18.04 with Intel GPU (this version of Ubuntu uses libglvnd even when Mesa is the only driver installed). Here's how I fixed this (commit):

diff --git a/glsldb/DebugLib/libglsldebug.c b/glsldb/DebugLib/libglsldebug.c
index 32c293f..f871e13 100644
--- a/glsldb/DebugLib/libglsldebug.c
+++ b/glsldb/DebugLib/libglsldebug.c
@@ -1321,7 +1321,7 @@ void *dlsym(void *handle, const char *symbol)
 		unsetenv("GLSL_DEBUGGER_DLSYM");
 	}
 
-	if (g.initialized) {
+	if (g.initialized && !(symbol[0]=='_' && symbol[1]=='_')) {
 		sym = (void*)glXGetProcAddressHook((GLubyte *)symbol);
 		if (sym) {
 			return sym;