Hi @mark-robustelli

I’ve been investigating the issue over the last few weeks.

Having taken a JVM dump while the issue was occurring, here is the stack trace for the thread causing the deadlock

{ "tid": "8820", "name": "HTTP client [\/[0:0:0:0:0:0:0:1]:50154]", "stack": [ "java.base\/jdk.internal.misc.Unsafe.park(Native Method)", "java.base\/java.lang.VirtualThread.parkOnCarrierThread(VirtualThread.java:677)", "java.base\/java.lang.VirtualThread.parkNanos(VirtualThread.java:648)", "java.base\/java.lang.System$2.parkVirtualThread(System.java:2652)", "java.base\/jdk.internal.misc.VirtualThreads.park(VirtualThreads.java:67)", "java.base\/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:408)", "java.base\/sun.nio.ch.Poller.pollIndirect(Poller.java:137)", "java.base\/sun.nio.ch.Poller.poll(Poller.java:102)", "java.base\/sun.nio.ch.Poller.poll(Poller.java:87)", "java.base\/sun.nio.ch.NioSocketImpl.park(NioSocketImpl.java:175)", "java.base\/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:280)", "java.base\/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:304)", "java.base\/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:346)", "java.base\/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:796)", "java.base\/java.net.Socket$SocketInputStream.read(Socket.java:1099)", "java.base\/java.io.BufferedInputStream.fill(BufferedInputStream.java:291)", "java.base\/java.io.BufferedInputStream.read1(BufferedInputStream.java:347)", "java.base\/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)", "java.base\/java.io.BufferedInputStream.read(BufferedInputStream.java:399)", "java.base\/sun.net.www.http.ChunkedInputStream.fastRead(ChunkedInputStream.java:244)", "java.base\/sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:698)", "java.base\/java.io.FilterInputStream.read(FilterInputStream.java:119)", "java.base\/sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3677)", "java.base\/java.io.BufferedInputStream.read1(BufferedInputStream.java:345)", "java.base\/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)", "java.base\/java.io.BufferedInputStream.read(BufferedInputStream.java:405)", "com.inversoft.rest.JSONResponseHandler$BetterBufferedInputStream.read(JSONResponseHandler.java:127)", "com.fasterxml.jackson.core.json.UTF8StreamJsonParser._loadMore(UTF8StreamJsonParser.java:220)", "com.fasterxml.jackson.core.json.UTF8StreamJsonParser._loadMoreGuaranteed(UTF8StreamJsonParser.java:2457)", "com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2540)", "com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString(UTF8StreamJsonParser.java:2520)", "com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:294)", "com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:42)", "com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:11)", "com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:137)", "com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:302)", "com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:169)", "com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:137)", "com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:302)", "com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:169)", "com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeNoNullChecks(CollectionDeserializer.java:501)", "com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:358)", "com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:245)", "com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:29)", "com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:137)", "com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:302)", "com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:169)", "com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:342)", "com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4971)", "com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3924)", "com.inversoft.rest.JSONResponseHandler.apply(JSONResponseHandler.java:68)", "com.inversoft.rest.RESTClient.go(RESTClient.java:430)", "io.fusionauth.client.FusionAuthClient.searchThemes(FusionAuthClient.java:5372)", "io.fusionauth.app.action.admin.theme.IndexAction.lambda$search$0(IndexAction.java:52)", "io.fusionauth.client.LambdaDelegate.execute(LambdaDelegate.java:58)", "io.fusionauth.app.action.admin.theme.IndexAction.search(IndexAction.java:52)", "io.fusionauth.app.action.admin.BaseSearchAction.execute(BaseSearchAction.java:77)", "java.base\/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)", "java.base\/java.lang.reflect.Method.invoke(Method.java:580)", "org.primeframework.mvc.util.ReflectionUtils.invoke(ReflectionUtils.java:443)", "org.primeframework.mvc.action.DefaultActionInvocationWorkflow.execute(DefaultActionInvocationWorkflow.java:77)", "org.primeframework.mvc.action.DefaultActionInvocationWorkflow.perform(DefaultActionInvocationWorkflow.java:60)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.message.DefaultMessageWorkflow.perform(DefaultMessageWorkflow.java:50)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.validation.DefaultValidationWorkflow.perform(DefaultValidationWorkflow.java:45)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.security.DefaultSecurityWorkflow.perform(DefaultSecurityWorkflow.java:79)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.parameter.DefaultPostParameterWorkflow.perform(DefaultPostParameterWorkflow.java:49)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.content.DefaultContentWorkflow.perform(DefaultContentWorkflow.java:74)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.parameter.DefaultParameterWorkflow.perform(DefaultParameterWorkflow.java:58)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.parameter.DefaultURIParameterWorkflow.perform(DefaultURIParameterWorkflow.java:92)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.scope.DefaultScopeRetrievalWorkflow.perform(DefaultScopeRetrievalWorkflow.java:50)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.action.DefaultActionMappingWorkflow.perform(DefaultActionMappingWorkflow.java:130)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.security.DefaultSavedRequestWorkflow.perform(DefaultSavedRequestWorkflow.java:65)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.cors.CORSFilter.doFilter(CORSFilter.java:188)", "org.primeframework.mvc.cors.CORSRequestWorkflow.perform(CORSRequestWorkflow.java:66)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.workflow.DefaultMVCWorkflow.perform(DefaultMVCWorkflow.java:109)", "org.primeframework.mvc.PrimeMVCRequestHandler.handle(PrimeMVCRequestHandler.java:76)", "io.fusionauth.http.server.internal.HTTPWorker.run(HTTPWorker.java:183)", "java.base\/java.lang.VirtualThread.run(VirtualThread.java:329)" ] }

I enabled trace logging in io.fusionauth.http.server.internal to analyse the process in HTTPWorker

2026-03-12 03:53:18.758 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Running HTTP worker. Block while we wait to read the preamble 2026-03-12 03:53:18.758 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Set state [Process]. Call the request handler. 2026-03-12 03:53:18.773 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Handler completed successfully 2026-03-12 03:53:18.773 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Enter Keep-Alive state [KeepAlive] Reset socket timeout [60000]. 2026-03-12 03:53:18.773 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Running HTTP worker. Block while we wait to read the preamble 2026-03-12 03:53:18.775 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Set state [Process]. Call the request handler. 2026-03-12 03:53:18.776 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Handler completed successfully 2026-03-12 03:53:18.776 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Enter Keep-Alive state [KeepAlive] Reset socket timeout [60000]. 2026-03-12 03:53:18.776 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Running HTTP worker. Block while we wait to read the preamble 2026-03-12 03:53:18.777 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Set state [Process]. Call the request handler. 2026-03-12 03:53:19.977 PM TRACE io.fusionauth.http.server.internal.HTTPServerThread - [993] Check worker in state [Write] 2026-03-12 03:53:21.978 PM TRACE io.fusionauth.http.server.internal.HTTPServerThread - [993] Check worker in state [Write] 2026-03-12 03:53:23.979 PM TRACE io.fusionauth.http.server.internal.HTTPServerThread - [993] Check worker in state [Write] ... ... 2026-03-12 03:55:54.019 PM TRACE io.fusionauth.http.server.internal.HTTPServerThread - [993] Check worker in state [Write] 2026-03-12 03:55:56.019 PM DEBUG io.fusionauth.http.server.internal.HTTPServerThread - [993] Check worker in state [Write] writingSlow=[true] writeThroughput=[16271] minimumWriteThroughput=[16384] 2026-03-12 03:55:56.019 PM DEBUG io.fusionauth.http.server.internal.HTTPServerThread - [993] Closing connection readingSlow=[false] writingSlow=[true] timedOut=[false] Min write throughput [16384], actual throughput [16271]. 2026-03-12 03:55:56.019 PM DEBUG io.fusionauth.http.server.internal.HTTPServerThread - [993] Closing client connection [/127.0.0.1:36278] due to inactivity 2026-03-12 03:55:56.021 PM TRACE io.fusionauth.http.server.internal.HTTPServerThread - Thread dump from server side. 2026-03-12 03:55:56.024 PM DEBUG io.fusionauth.http.server.internal.HTTPWorker - [993] Closing socket. The socket was closed by a client, proxy or otherwise.

The HTTPWorker begins the process of reading and writing the response for the request, and continues for 2.6 minutes until the HTTPServerThread terminates it because the minimumWriteThroughput (16KB/sec) threshold has not been met.

HTTPServerCleanerThread kills the server-side connection mid-response due to a write throughput check that measures average bytes/sec since the first socket write.
The client is waiting for data that the server is generating very slowly.

Analyzing why the server is writing so slowly (16271 bytes/sec):
The call is http://localhost:9012 — FusionAuth calls itself. Client and server virtual threads share the same JVM's carrier thread pool.
With Kubernetes CPU limit = 1000m, JDK 21 uses UseContainerSupport (default), so availableProcessors() = 1 → only 1 carrier thread.

com.inversoft.rest.JSONResponseHandler$BetterBufferedInputStream.read(byte[], int, int) is synchronized — causes virtual thread carrier thread pinning in JDK 21.

Client virtual thread enters synchronized read() → calls ChunkedInputStream.fastRead() → calls socket.read() which blocks → PINS the carrier thread (can't unmount from synchronized block) Server virtual thread needs to write more data but cannot get a carrier thread (carrier thread pinned by client) Neither makes progress → throughput decays over time as numberOfBytesWritten is fixed but elapsed keeps growing

BetterBufferedInputStream.read() pins that single carrier thread while blocking on ChunkedInputStream.fastRead(), leaving no carrier thread for the server to write more data. The resulting ping-pong limits throughput to ~16 KB/sec on a 2.5MB response.

HTTPServerCleanerThread computes average throughput since the very first write (not a recent window). With 2.5MB written at 16271 bytes/sec average, that's ~156 seconds (2.6 min) before the average decays below the 16384 bytes/sec threshold.

With 9 themes, the response is small enough to fit entirely within the kernel’s socket buffer (~128KB). The server writes the entire content in one go without blocking on socket.write() (the buffer does not fill up), transitions to the KeepAlive state, and the cleaner can no longer terminate the connection due to write throughput.

As a workaround, I resolved the issue increasing the Kubernetes CPU limit from 1000m to 1050m.
Kubernetes translates the CPU limit into the container’s cgroups, setting 2 carrier threads instead of 1.

With that CPU limit, all themes are recovered properly, also with 35 themes (~6.2 MB response).

Can you reproduce it setting Kubernetes CPU limit to 1000m (1 core) in your environment?

@hvfa said in The request origin could not be verified. Unable to complete this login request with same-instance cross-tenant IdP federation:

@mark-robustelli thanks Mark.

All domains will be sub-domains of the main domain name, but for this staging system, we are using 2026.domain.com and 2026-tenant1.domain.com, 2026-tenant2 etc.

Logins work successfully without the idphint on a per tenant basis but we want all tenants to use the common master tenant (via SSO) as the login, because many users will be members across multiple tenants and their data can be common across tenants.

The auth server uses auth.domain.com

We have the master tenant and sub tenants in FusionAuth. We have the master application and sub-tenant applications for each tenant.
A tenant application has an authorized redirect like 2026-tenant1.domain.com/callback and currently no request origin (I have tried adding this for sub-tenant and master tenant together). I've tried adding the sub-tenants as authorized origins on the master tenant.

There's nothing extra configured on the FusionAuth tenants.

Under settings, Identity Providers, we have an OIDC provider pointing back to the master app in the master tenant. We do see the button to login with master tenant on a sub-tenant (if no idphint is set).

All sub-tenant applications are enabled for this provider (with create reg), I also have the sub-tenants added in this IP and have tried without. Managed domains is blank. Not using the POST method.

Auth endpoints have been manually set (because FusionAuth couldn't self-discover?!) like so:
https://auth.domain.com/oauth2/authorize,
https://auth.domain.com/oauth2/token,
https://auth.domain.com/oauth2/userinfo

There's no groups configured.

Hosting wise, FusionAuth is a Docker container on the same server with the main app and sub-tenants behind a Traefik 2 proxy which is also behind Cloudflare and each site has its own LetsEncrypt SSL cert via Traefik.

Direct login to the master tenant is successful but not via a sub-tenant.

Grok suggests it's a CORS issue. My filter was not enabled. I also tried enabled (current) and allowed all methods with (and without) wildcard origins.

As mentioned, the login is recorded on FusionAuth, it just seems to fail on the callback process via master..

The master login URL when called via a sub-tenant shows the master tenant as the callback URL. I've tried adding the sub-tenant callback as authorised in the master tenant.

I think that's everything..

Any thoughts about this? Please, thanks!

I think the way I'd approach this is:

import all users into FusionAuth

At cutover time:

look at local database to see which password hashes had changed pull the user data from FusionAuth for each of these users delete the user re-import the user with the new password hash and the FusionAuth data, maintaining the same userId (if you provide the UUID, we'll use that)

I get that is an additional complexity, but hopefully that helps.

This is totally possible.

You want to start by understanding FusionAuth passkey setup and the normal flow.

Then, in your application, probably using one of the client libraries, you want to do the following for a user:

see if a user has a passkey set up, using the "retrieve a passkey" API. If this returns 0 passkeys, show the prompt. for the prompt, you have two options: use the API/client library to start the passkey registration process from within your application directly send them to the user management page to add a passkey (requires a paid license)

The right way to do the latter depends on your application needs (are you okay with a redirect) and whether or not you have at least a starter license.

For reporting on the number of users that have set up passkeys, unfortunately you have to query all your users and then pull the passkey data individually. There's no way to use the elasticsearch syntax to do the query as of yet. There's an open github issue to add that functionality.

@ralph Thanks for following up and sharing!

@james-hudson You may want to check out this blog post. Hopefully that can help.

@scottw Hopefully, I can get a little time over the next couple of days and see if I can duplicate it. I will let you know if I find anything. Anyone else seeing his behavior?

@rgros Do you have Debug enabled?

Screenshot 2026-02-18 at 10.43.14 AM.png

Then you should check your Event Log.

Screenshot 2026-02-18 at 10.44.20 AM.png

Let us know what you find.

@tim-clark Can you please point to the community discussion where this comes up? I could not find it in the issues.

@hemanth18pages raising the support ticket is the way to go for sure. It will be good to know if anyone else is experiencing this too.

@bernardo-munz Did you play with the SameSite setting?

@dan said in Error getting list application due to sql error (mysql):

@traperwaze said in Error getting list application due to sql error (mysql):

MariaDB server

I believe the issues is that we don't support MariaDB.

This is a known issue: https://github.com/FusionAuth/fusionauth-issues/issues/367 poki

Do you see the same issue with a supported version of MySQL or PostgreSQL?

Thanks for confirming. That lines up with what I’m seeing as well. We are indeed running MariaDB, and it looks like the JSON operator (->>) used in that query isn’t supported the same way in MariaDB, which would explain the syntax error.

We haven’t tested this yet on a supported database, but based on the linked issue and your comment, it does appear to be a MariaDB compatibility problem rather than a misconfiguration on our side. We’ll plan to test against a supported version of MySQL and/or PostgreSQL to confirm.

Appreciate you pointing us to the GitHub issue — that helps clarify things a lot.

@dalamenona We had the same error with Prometheus.
The following opened FusionAuth issue contains information about the topic

https://github.com/FusionAuth/fusionauth-issues/issues/3082

Best regards.

@simon-chrzanowski can you please share the code you using? (please be sure to hide anything sensitive like your API Key)

@dan said in Claims to check when using google as an idp for google workspace:

You should start by checking the relevant google documentation.

As of writing, this is what their doc says:

Using the email, email_verified and hd fields, you can determine if Google hosts and is authoritative for an email address. In the cases where Google is authoritative, the user is known to be the legitimate account owner, and you may skip password or other challenge methods.

Cases where Google is authoritative:

email has a @gmail.com suffix, this is a Gmail account. email_verified is true and hd is set, this is a Google Workspace account.

Users may register for Google Accounts without using Gmail or Google Workspace. When email does not contain a @gmail.com suffix and hd is absent, Google is not authoritative and password or other challenge methods are recommended to verify the user. email_verified can also be true as Google initially verified the user when the Google account was created, however ownership of the third party email account may have since changed.

So in this case, you want to check that hd is set as well as that email_verified is true.

With FusionAuth, you can check this using a reconcile lambda and looking at the id_token:

https://fusionauth.io/docs/extend/code/lambdas/google-reconcile https://fusionauth.io/docs/extend/code/lambdas/openid-connect-response-reconcile

Thank you from bringing this to light.

@alexandros-nafas , were you able to figure it out?

@johnmiller It looks like the issue is that the 'retrieve a user via JWT' functionality was removed in 1.60.0. (It appears the User API JWT authentication method was deprecated in version 1.50.0. An issue has been filed to remove it from the client libraries.

Thank you from bringing this to light.

@mark-robustelli Thanks, I think that's resolved it!

@Ruka , this seems like something that should be reported as an issue.

@patrick_ag Is this just when you are loading the page or are you trying to take some action?