[arch,kerberos] Freerdp 3 authentication hangs due to broken krb5.conf about freerdp

[akallabeth]

@giox069 @fredizzimo ok, I´ve found a way to fix this for my case here with the following /etc/krb5.conf:

[libdefaults]
rdns = false
dns_lookup_kdc = 0

this effectively disables DNS lookup, failing kerberos immediately.

[fredizzimo]

Ok, I will try that out now, and come back to you with the results.

[fredizzimo]

Ah, I just double checked, all of the activation of that was by myself through systemctl restart, not by the system.

1. I did it before rebooting in the hope that the configuration was not reloaded.
2. And then after it had failed to connect after the reboot. That restart was in the middle of the 10 minute wait according to the timestamp, so I think we can assume that it's unrelated.

[fredizzimo]

No, my response was just about the krb5-kdc.service. So the original problem remains.

I understand that it's hard to know what the problem is with this little information, so I can try to debug it during the weekend, to at least provide more information.

[akallabeth]

@fredizzimo @giox069 can you add a full log of your failed connections with kerberos debugging enabled? (see https://web.mit.edu/kerberos/krb5-1.12/doc/admin/env_variables.html for details for kerberos debugging)
[note] you can PM me in our matrix chat if you don´t want to publish the logs.

[giox069]

@akallabeth my /ectkrb5.conf does not exists. I will able to produce debug trace later this night (CET), not now. If you need, I can open a remote TCP port from a fixed IP address/subnet so you can do tests by yourself. I can setup it this night.

Remember that the error appears when:
/v: contains a numeric IP address
/d: contains an internet domain (with at least a dot). Both resolvable or not in my case.

[giox069]

@giox069 @fredizzimo ok, I´ve found a way to fix this for my case here with the following /etc/krb5.conf:

[libdefaults]
rdns = false
dns_lookup_kdc = 0

this effectively disables DNS lookup, failing kerberos immediately.

This workaround is working! 👍

[akallabeth]

what is written in your /etc/krb5.conf?
we had several reports that arch ships a example configuration as the actual thing (and that leads to unreachable KDC which then needs to time out for each kerberos request)

[fredizzimo]

Yes, it definitely looks like some example

❯ cat /etc/krb5.conf
[libdefaults]
        default_realm = ATHENA.MIT.EDU

[realms]
# use "kdc = ..." if realm admins haven't put SRV records into DNS
        ATHENA.MIT.EDU = {
                admin_server = kerberos.mit.edu
        }
        ANDREW.CMU.EDU = {
                admin_server = kdc-01.andrew.cmu.edu
        }

[domain_realm]
        mit.edu = ATHENA.MIT.EDU
        csail.mit.edu = CSAIL.MIT.EDU
        .ucsc.edu = CATS.UCSC.EDU

[logging]
#       kdc = CONSOLE

Is it enough to comment out everything, or do I need to create a proper configuration, I don't know how to do at the moment, but I can probably figure it out? I'm currently on freerdp 2 again, so it takes a while to switch and test, therefore I'm asking instead of testing it myself.

[akallabeth]

@fredizzimo no, an empty file (or commented) is ok.

[fredizzimo]

Unfortunately, it did not help, not even after restarting the system.

But I see the following in the journal log now

apr 24 17:15:40 fredarch krb5kdc[5592]: Configuration file does not specify default realm - while attempting to retrieve default realm
apr 24 17:15:40 fredarch krb5kdc[5592]: krb5kdc: Configuration file does not specify default realm, attempting to retrieve default realm
apr 24 17:15:40 fredarch systemd[1]: krb5-kdc.service: Main process exited, code=exited, status=1/FAILURE
apr 24 17:15:40 fredarch systemd[1]: krb5-kdc.service: Failed with result 'exit-code'.
apr 24 17:15:40 fredarch systemd[1]: krb5-kdc.service: Scheduled restart job, restart counter is at 1.
apr 24 17:15:40 fredarch systemd[1]: Started Kerberos 5 KDC.

I also tried restarting the service after I joined the VPN, but it gives the same result. So maybe I need to try to configure it?

Also while doing that I noticed that after 10 minutes, the xfreerdp printed this in the log, but was still hanging
[17:20:25:492] [3958:00000f77] [ERROR][com.winpr.sspi.Kerberos] - [kerberos_AcquireCredentialsHandleA]: krb5glue_get_init_creds (Cannot find KDC for realm "DOMAIN.ORG" [-1765328230])

The domain name is the correct, so it got that right at least. Maybe it's misconfiguration on the workplace side? I can try to contact the IT support there.

[akallabeth]

no, that is ok (it should fail fast if you are not using kerberos to authenticate, which is the case most of the time if you are not directly in the same network

[akallabeth]

@fredizzimo also, why is your system trying to start a KDC ? are you hosting a kerberos server instance?

[akallabeth]

@fredizzimo so, does your initial use case work now or is there still something to look at?

[akallabeth]

a debug build with a running debugger to have a backtrace on where the application is hanging would be really helpful here.

[fredizzimo]

I don't have the symbols for the system libraries at the moment, but this is the callstack

libc.so.6!connect (Unknown Source:0)
libc.so.6![Unknown/Just-In-Time compiled code] (Unknown Source:0)
libc.so.6!__res_context_send (Unknown Source:0)
libc.so.6!__res_context_query (Unknown Source:0)
libc.so.6!__res_context_search (Unknown Source:0)
libc.so.6!res_nsearch (Unknown Source:0)
libkrb5.so.3![Unknown/Just-In-Time compiled code] (Unknown Source:0)
libkrb5.so.3!krb5_sendto_kdc (Unknown Source:0)
libkrb5.so.3![Unknown/Just-In-Time compiled code] (Unknown Source:0)
libkrb5.so.3!krb5_init_creds_get (Unknown Source:0)
libwinpr3.so.3!krb5glue_get_init_creds(krb5_context ctx, krb5_principal princ, krb5_ccache ccache, krb5_prompter_fct prompter, char * password, SEC_WINPR_KERBEROS_SETTINGS * krb_settings) (/home/fredizzimo/proj/FreeRDP/winpr/libwinpr/sspi/Kerberos/krb5glue_mit.c:237)
libwinpr3.so.3!kerberos_AcquireCredentialsHandleA(SEC_CHAR * pszPrincipal, SEC_CHAR * pszPackage, ULONG fCredentialUse, void * pvLogonID, void * pAuthData, SEC_GET_KEY_FN pGetKeyFn, void * pvGetKeyArgument, PCredHandle phCredential, PTimeStamp ptsExpiry) (/home/fredizzimo/proj/FreeRDP/winpr/libwinpr/sspi/Kerberos/kerberos.c:341)
libwinpr3.so.3!negotiate_AcquireCredentialsHandleA(SEC_CHAR * pszPrincipal, SEC_CHAR * pszPackage, ULONG fCredentialUse, void * pvLogonID, void * pAuthData, SEC_GET_KEY_FN pGetKeyFn, void * pvGetKeyArgument, PCredHandle phCredential, PTimeStamp ptsExpiry) (/home/fredizzimo/proj/FreeRDP/winpr/libwinpr/sspi/Negotiate/negotiate.c:1457)
libwinpr3.so.3!winpr_AcquireCredentialsHandleA(SEC_CHAR * pszPrincipal, SEC_CHAR * pszPackage, ULONG fCredentialUse, void * pvLogonID, void * pAuthData, SEC_GET_KEY_FN pGetKeyFn, void * pvGetKeyArgument, PCredHandle phCredential, PTimeStamp ptsExpiry) (/home/fredizzimo/proj/FreeRDP/winpr/libwinpr/sspi/sspi_winpr.c:1299)
libfreerdp3.so.3!credssp_auth_setup_client(rdpCredsspAuth * auth, const char * target_service, const char * target_hostname, const SEC_WINNT_AUTH_IDENTITY_W * identity, const char * pkinit) (/home/fredizzimo/proj/FreeRDP/libfreerdp/core/credssp_auth.c:291)
libfreerdp3.so.3!nla_client_init(rdpNla * nla) (/home/fredizzimo/proj/FreeRDP/libfreerdp/core/nla.c:451)
libfreerdp3.so.3!nla_client_begin(rdpNla * nla) (/home/fredizzimo/proj/FreeRDP/libfreerdp/core/nla.c:475)
libfreerdp3.so.3!transport_connect_nla(rdpTransport * transport, BOOL earlyUserAuth) (/home/fredizzimo/proj/FreeRDP/libfreerdp/core/transport.c:381)
libfreerdp3.so.3!nego_try_connect(rdpNego * nego) (/home/fredizzimo/proj/FreeRDP/libfreerdp/core/nego.c:315)
libfreerdp3.so.3!nego_security_connect(rdpNego * nego) (/home/fredizzimo/proj/FreeRDP/libfreerdp/core/nego.c:347)
libfreerdp3.so.3!nego_connect(rdpNego * nego) (/home/fredizzimo/proj/FreeRDP/libfreerdp/core/nego.c:282)
libfreerdp3.so.3!rdp_client_connect(rdpRdp * rdp) (/home/fredizzimo/proj/FreeRDP/libfreerdp/core/connection.c:430)
libfreerdp3.so.3!freerdp_connect_begin(freerdp * instance) (/home/fredizzimo/proj/FreeRDP/libfreerdp/core/freerdp.c:156)
libfreerdp3.so.3!freerdp_connect(freerdp * instance) (/home/fredizzimo/proj/FreeRDP/libfreerdp/core/freerdp.c:174)
xf_client_thread(LPVOID param) (/home/fredizzimo/proj/FreeRDP/client/X11/xf_client.c:1501)
libwinpr3.so.3!thread_launcher(void * arg) (/home/fredizzimo/proj/FreeRDP/winpr/libwinpr/thread/thread.c:528)
libc.so.6![Unknown/Just-In-Time compiled code] (Unknown Source:0)

So it looks like it's the Kerberos connection that fails to connect. I will try to check if freerdp 2 does something different, since that works.

[fredizzimo]

I bisected it down to this commit c9e61ff (cmake] simplify krb5 detection)

NOTE: I had to do the bisecting with a clean build directory each time, if I just tried incremental builds then event the latest version would work, which also indicate that there might be something wrong with the cmake configuration.

Probably detecting the wrong type of Kerberos implementation

[fredizzimo]

Ah, now I see, before that commit it defaulted to OFF, and now it defaults to ON. And indeed if I set -DWITH_KRB5=OFF it works on master.

And it also works with /auth-pkg-list:!kerberos. But I don't see how to pass that option when, using remmina right now.

If it can be disabled, I guess it "fixes" my problem, not sure if it's worth trying to dig further into this and found out the cause for the hang though.

[giox069]

Very similar problem here. I have just switched to (K)Ubuntu 24.04 on a couple of PC, and I can no longer connect by numeric IP address to AD domain member machines with Remmina.

xfreerdp3 /v:192.168.98.1 /d:mydom.dom /u:xxxxx

The error is:

[09:15:07:823] [1348:00000545] [ERROR][com.winpr.sspi.Kerberos] - [kerberos_InitializeSecurityContextA]: krb5_get_credentials (Configuration file does not specify default realm [-1765328160])

The problem disappear by using a FQDN hostname instead of a numeric IP address, or by adding /auth-pkg-list:!kerberos to the xfreerdp3 command line.

Same problem when manually compiling master branch of FreeRDP3 on a Ubuntu 22.04 or Ubuntu 20.04

[fredizzimo]

I opened a Remmina feature request for being able to disable Kerberos. https://gitlab.com/Remmina/Remmina/-/issues/3104

[fredizzimo]

Some more information, by just looking at my callstack and the krb5 code, it's this nameserver lookup called from this that fail
https://github.com/krb5/krb5/blob/0a3acc20564e82ba33741248cf25ca4d085d777f/src/lib/krb5/os/locate_kdc.c#L823

My guess is that some parts of the company internal network are not reachable through the VPN, and therefore fail.

[giox069]

@fredizzimo can you try with the short netbios domain /d:mydom (no DNS domain name, no dots) ?

In my case it works as workaround. But I'm still having a customer that has NETBIOS doman identical to DNS domain (with dot inside). So I cannot use this workaround.

[akallabeth]

@giox069 @fredizzimo the krb5 stuff must fail in your cases, but there should be a NTLM fallback in place.
for some reason it does not trigger for you and that would be interesting why.

the issue I mentioned before (krb5.conf being some default) leads to incredibly high timeouts for the fallback to trigger, but if it does not trigger in your case then something else is off.

@giox069 you run a build with ntlm fallback enabled, right?

[giox069]

I'm using two xfreerdp 3: the stock version of Ubuntu 24.04, and my own compiled version from master branch on Ubuntu 22.04. In CMakeCache.txt of the compiled version can find WITH_KRB5_NO_NTLM_FALLBACK:BOOL=OFF
Other ways to check if ntlm fallback is enabled?

[akallabeth]

@giox069 seems active.
would be interesting where the error exit is coming from, to identify the branch that does not fall back to NTLM

[giox069]

I did some bisecting, the commit that introduced the problem is c9e61ff
I will try to understand where is the problem, but that commit is quite large.

[giox069]

... and it's the same commit bisected by @fredizzimo ;)

[akallabeth]

@giox069 and as @fredizzimo already found out the commit that enabled krb5 support.
no surprise there, but the interesting part is why your connectin attempt does not fall back to NTLM ...

[akallabeth]

@giox069 also, do you have some stuff in your krb5.conf? did a test on my debian machine to a domain member and that instantly connects.

only message is [13:15:55:50] [97473:00017cc2] [ERROR][com.winpr.sspi.Kerberos] - [kerberos_AcquireCredentialsHandleA]: krb5glue_get_init_creds (Cannot find KDC for realm "RDTEST" [-1765328230]) which is expected (aka no kerberos available and fall back to NTLM) while your message suggests that this stepp succeeded and only later on aborts in the following calls.

[akallabeth]

@fredizzimo ok, did manage to get a slowdown (DNS lookup delay) but no hang.
what did I do:

1. connect with xfreerdp /v:ip /u:user /d:domain.local
2. the .local is not resolvable from local environment

[13:29:57:889] [99619:00018525] [WARN][com.freerdp.crypto] - [verify_cb]: Certificate verification failure 'self-signed certificate (18)' at stack position 0
[13:29:57:889] [99619:00018525] [WARN][com.freerdp.crypto] - [verify_cb]: CN = RD2.rdtest.local
[99619] 1714994997.396297: Matching [email protected] in collection with result: -1765328243/Can't find client principal [email protected] in cache collection
[99619] 1714994997.396298: Resolving unique ccache of type MEMORY
[99619] 1714994997.396299: Initializing MEMORY:wLqbmfR with default princ [email protected]
[99619] 1714994997.396300: Getting initial credentials for [email protected]
[99619] 1714994997.396301: Retrieving [email protected] -> krb5_ccache_conf_data/pa_type/krbtgt\/RDTEST.LOCAL\@RDTEST.LOCAL@X-CACHECONF: from MEMORY:wLqbmfR with result: -1765328243/Matching credential not found
[99619] 1714994997.396303: Retrieving [email protected] -> krb5_ccache_conf_data/pa_config_data/krbtgt\/RDTEST.LOCAL\@RDTEST.LOCAL@X-CACHECONF: from MEMORY:wLqbmfR with result: -1765328243/Matching credential not found
[99619] 1714994997.396304: Sending unauthenticated request
[99619] 1714994997.396305: Sending request (185 bytes) to RDTEST.LOCAL
[99619] 1714994997.396306: Sending DNS URI query for _kerberos.RDTEST.LOCAL.
[99619] 1714995007.408895: No URI records found
[99619] 1714995007.408896: Sending DNS SRV query for _kerberos._udp.RDTEST.LOCAL.
[99619] 1714995017.419469: Sending DNS SRV query for _kerberos._tcp.RDTEST.LOCAL.
[99619] 1714995027.429896: No SRV records found
[13:30:27:927] [99619:00018525] [ERROR][com.winpr.sspi.Kerberos] - [kerberos_AcquireCredentialsHandleA]: krb5glue_get_init_creds (Cannot find KDC for realm "RDTEST.LOCAL" [-1765328230])
[99619] 1714995027.429897: Destroying ccache MEMORY:wLqbmfR
[99619] 1714995027.429898: Matching [email protected] in collection with result: -1765328243/Can't find client principal [email protected] in cache collection
[99619] 1714995027.429899: Resolving unique ccache of type MEMORY
[99619] 1714995027.429900: Initializing MEMORY:dM7nvS9 with default princ [email protected]
[99619] 1714995027.429901: Getting initial credentials for [email protected]
[99619] 1714995027.429902: Retrieving [email protected] -> krb5_ccache_conf_data/pa_type/krbtgt\/RDTEST.LOCAL\@RDTEST.LOCAL@X-CACHECONF: from MEMORY:dM7nvS9 with result: -1765328243/Matching credential not found
[99619] 1714995027.429904: Retrieving [email protected] -> krb5_ccache_conf_data/pa_config_data/krbtgt\/RDTEST.LOCAL\@RDTEST.LOCAL@X-CACHECONF: from MEMORY:dM7nvS9 with result: -1765328243/Matching credential not found
[99619] 1714995027.429905: Sending unauthenticated request
[99619] 1714995027.429906: Sending request (185 bytes) to RDTEST.LOCAL
[99619] 1714995027.429907: Sending DNS URI query for _kerberos.RDTEST.LOCAL.
[99619] 1714995037.440758: No URI records found
[99619] 1714995037.440759: Sending DNS SRV query for _kerberos._udp.RDTEST.LOCAL.
[99619] 1714995047.449855: Sending DNS SRV query for _kerberos._tcp.RDTEST.LOCAL.
[99619] 1714995057.456781: No SRV records found
[13:30:57:954] [99619:00018525] [ERROR][com.winpr.sspi.Kerberos] - [kerberos_AcquireCredentialsHandleA]: krb5glue_get_init_creds (Cannot find KDC for realm "RDTEST.LOCAL" [-1765328230])
[99619] 1714995057.456782: Destroying ccache MEMORY:dM7nvS9
[13:30:57:226] [99619:00018525] [WARN][com.freerdp.core.license] - [license_read_binary_blob_data]: license binary blob::type BB_ERROR_BLOB, length=0, skipping.

[akallabeth]

@giox069 ok, I´ll wait.
the sample above was exactly such a setup, /v:192.168.xx.yy /u:user /d:domain.local