@hvfa said in The request origin could not be verified. Unable to complete this login request with same-instance cross-tenant IdP federation:

@mark-robustelli thanks Mark.

All domains will be sub-domains of the main domain name, but for this staging system, we are using 2026.domain.com and 2026-tenant1.domain.com, 2026-tenant2 etc.

Logins work successfully without the idphint on a per tenant basis but we want all tenants to use the common master tenant (via SSO) as the login, because many users will be members across multiple tenants and their data can be common across tenants.

The auth server uses auth.domain.com

We have the master tenant and sub tenants in FusionAuth. We have the master application and sub-tenant applications for each tenant.
A tenant application has an authorized redirect like 2026-tenant1.domain.com/callback and currently no request origin (I have tried adding this for sub-tenant and master tenant together). I've tried adding the sub-tenants as authorized origins on the master tenant.

There's nothing extra configured on the FusionAuth tenants.

Under settings, Identity Providers, we have an OIDC provider pointing back to the master app in the master tenant. We do see the button to login with master tenant on a sub-tenant (if no idphint is set).

All sub-tenant applications are enabled for this provider (with create reg), I also have the sub-tenants added in this IP and have tried without. Managed domains is blank. Not using the POST method.

Auth endpoints have been manually set (because FusionAuth couldn't self-discover?!) like so:
https://auth.domain.com/oauth2/authorize,
https://auth.domain.com/oauth2/token,
https://auth.domain.com/oauth2/userinfo

There's no groups configured.

Hosting wise, FusionAuth is a Docker container on the same server with the main app and sub-tenants behind a Traefik 2 proxy which is also behind Cloudflare and each site has its own LetsEncrypt SSL cert via Traefik.

Direct login to the master tenant is successful but not via a sub-tenant.

Grok suggests it's a CORS issue. My filter was not enabled. I also tried enabled (current) and allowed all methods with (and without) wildcard origins.

As mentioned, the login is recorded on FusionAuth, it just seems to fail on the callback process via master..

The master login URL when called via a sub-tenant shows the master tenant as the callback URL. I've tried adding the sub-tenant callback as authorised in the master tenant.

I think that's everything..

Any thoughts about this? Please, thanks!

@ralph Thanks for following up and sharing!

I think the way I'd approach this is:

import all users into FusionAuth

At cutover time:

look at local database to see which password hashes had changed pull the user data from FusionAuth for each of these users delete the user re-import the user with the new password hash and the FusionAuth data, maintaining the same userId (if you provide the UUID, we'll use that)

I get that is an additional complexity, but hopefully that helps.

This is totally possible.

You want to start by understanding FusionAuth passkey setup and the normal flow.

Then, in your application, probably using one of the client libraries, you want to do the following for a user:

see if a user has a passkey set up, using the "retrieve a passkey" API. If this returns 0 passkeys, show the prompt. for the prompt, you have two options: use the API/client library to start the passkey registration process from within your application directly send them to the user management page to add a passkey (requires a paid license)

The right way to do the latter depends on your application needs (are you okay with a redirect) and whether or not you have at least a starter license.

For reporting on the number of users that have set up passkeys, unfortunately you have to query all your users and then pull the passkey data individually. There's no way to use the elasticsearch syntax to do the query as of yet. There's an open github issue to add that functionality.

@james-hudson You may want to check out this blog post. Hopefully that can help.

@scottw Hopefully, I can get a little time over the next couple of days and see if I can duplicate it. I will let you know if I find anything. Anyone else seeing his behavior?

@rgros Do you have Debug enabled?

Screenshot 2026-02-18 at 10.43.14 AM.png

Then you should check your Event Log.

Screenshot 2026-02-18 at 10.44.20 AM.png

Let us know what you find.

Hi @mark-robustelli

I’ve been investigating the issue over the last few weeks.

Having taken a JVM dump while the issue was occurring, here is the stack trace for the thread causing the deadlock

{ "tid": "8820", "name": "HTTP client [\/[0:0:0:0:0:0:0:1]:50154]", "stack": [ "java.base\/jdk.internal.misc.Unsafe.park(Native Method)", "java.base\/java.lang.VirtualThread.parkOnCarrierThread(VirtualThread.java:677)", "java.base\/java.lang.VirtualThread.parkNanos(VirtualThread.java:648)", "java.base\/java.lang.System$2.parkVirtualThread(System.java:2652)", "java.base\/jdk.internal.misc.VirtualThreads.park(VirtualThreads.java:67)", "java.base\/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:408)", "java.base\/sun.nio.ch.Poller.pollIndirect(Poller.java:137)", "java.base\/sun.nio.ch.Poller.poll(Poller.java:102)", "java.base\/sun.nio.ch.Poller.poll(Poller.java:87)", "java.base\/sun.nio.ch.NioSocketImpl.park(NioSocketImpl.java:175)", "java.base\/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:280)", "java.base\/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:304)", "java.base\/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:346)", "java.base\/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:796)", "java.base\/java.net.Socket$SocketInputStream.read(Socket.java:1099)", "java.base\/java.io.BufferedInputStream.fill(BufferedInputStream.java:291)", "java.base\/java.io.BufferedInputStream.read1(BufferedInputStream.java:347)", "java.base\/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)", "java.base\/java.io.BufferedInputStream.read(BufferedInputStream.java:399)", "java.base\/sun.net.www.http.ChunkedInputStream.fastRead(ChunkedInputStream.java:244)", "java.base\/sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:698)", "java.base\/java.io.FilterInputStream.read(FilterInputStream.java:119)", "java.base\/sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3677)", "java.base\/java.io.BufferedInputStream.read1(BufferedInputStream.java:345)", "java.base\/java.io.BufferedInputStream.implRead(BufferedInputStream.java:420)", "java.base\/java.io.BufferedInputStream.read(BufferedInputStream.java:405)", "com.inversoft.rest.JSONResponseHandler$BetterBufferedInputStream.read(JSONResponseHandler.java:127)", "com.fasterxml.jackson.core.json.UTF8StreamJsonParser._loadMore(UTF8StreamJsonParser.java:220)", "com.fasterxml.jackson.core.json.UTF8StreamJsonParser._loadMoreGuaranteed(UTF8StreamJsonParser.java:2457)", "com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2540)", "com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString(UTF8StreamJsonParser.java:2520)", "com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:294)", "com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:42)", "com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:11)", "com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:137)", "com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:302)", "com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:169)", "com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:137)", "com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:302)", "com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:169)", "com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeNoNullChecks(CollectionDeserializer.java:501)", "com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:358)", "com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:245)", "com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:29)", "com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:137)", "com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:302)", "com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:169)", "com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:342)", "com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4971)", "com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3924)", "com.inversoft.rest.JSONResponseHandler.apply(JSONResponseHandler.java:68)", "com.inversoft.rest.RESTClient.go(RESTClient.java:430)", "io.fusionauth.client.FusionAuthClient.searchThemes(FusionAuthClient.java:5372)", "io.fusionauth.app.action.admin.theme.IndexAction.lambda$search$0(IndexAction.java:52)", "io.fusionauth.client.LambdaDelegate.execute(LambdaDelegate.java:58)", "io.fusionauth.app.action.admin.theme.IndexAction.search(IndexAction.java:52)", "io.fusionauth.app.action.admin.BaseSearchAction.execute(BaseSearchAction.java:77)", "java.base\/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)", "java.base\/java.lang.reflect.Method.invoke(Method.java:580)", "org.primeframework.mvc.util.ReflectionUtils.invoke(ReflectionUtils.java:443)", "org.primeframework.mvc.action.DefaultActionInvocationWorkflow.execute(DefaultActionInvocationWorkflow.java:77)", "org.primeframework.mvc.action.DefaultActionInvocationWorkflow.perform(DefaultActionInvocationWorkflow.java:60)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.message.DefaultMessageWorkflow.perform(DefaultMessageWorkflow.java:50)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.validation.DefaultValidationWorkflow.perform(DefaultValidationWorkflow.java:45)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.security.DefaultSecurityWorkflow.perform(DefaultSecurityWorkflow.java:79)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.parameter.DefaultPostParameterWorkflow.perform(DefaultPostParameterWorkflow.java:49)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.content.DefaultContentWorkflow.perform(DefaultContentWorkflow.java:74)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.parameter.DefaultParameterWorkflow.perform(DefaultParameterWorkflow.java:58)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.parameter.DefaultURIParameterWorkflow.perform(DefaultURIParameterWorkflow.java:92)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.scope.DefaultScopeRetrievalWorkflow.perform(DefaultScopeRetrievalWorkflow.java:50)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.action.DefaultActionMappingWorkflow.perform(DefaultActionMappingWorkflow.java:130)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.security.DefaultSavedRequestWorkflow.perform(DefaultSavedRequestWorkflow.java:65)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.cors.CORSFilter.doFilter(CORSFilter.java:188)", "org.primeframework.mvc.cors.CORSRequestWorkflow.perform(CORSRequestWorkflow.java:66)", "org.primeframework.mvc.workflow.SubWorkflowChain.continueWorkflow(SubWorkflowChain.java:50)", "org.primeframework.mvc.workflow.DefaultMVCWorkflow.perform(DefaultMVCWorkflow.java:109)", "org.primeframework.mvc.PrimeMVCRequestHandler.handle(PrimeMVCRequestHandler.java:76)", "io.fusionauth.http.server.internal.HTTPWorker.run(HTTPWorker.java:183)", "java.base\/java.lang.VirtualThread.run(VirtualThread.java:329)" ] }

I enabled trace logging in io.fusionauth.http.server.internal to analyse the process in HTTPWorker

2026-03-12 03:53:18.758 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Running HTTP worker. Block while we wait to read the preamble 2026-03-12 03:53:18.758 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Set state [Process]. Call the request handler. 2026-03-12 03:53:18.773 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Handler completed successfully 2026-03-12 03:53:18.773 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Enter Keep-Alive state [KeepAlive] Reset socket timeout [60000]. 2026-03-12 03:53:18.773 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Running HTTP worker. Block while we wait to read the preamble 2026-03-12 03:53:18.775 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Set state [Process]. Call the request handler. 2026-03-12 03:53:18.776 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Handler completed successfully 2026-03-12 03:53:18.776 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Enter Keep-Alive state [KeepAlive] Reset socket timeout [60000]. 2026-03-12 03:53:18.776 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Running HTTP worker. Block while we wait to read the preamble 2026-03-12 03:53:18.777 PM TRACE io.fusionauth.http.server.internal.HTTPWorker - [993] Set state [Process]. Call the request handler. 2026-03-12 03:53:19.977 PM TRACE io.fusionauth.http.server.internal.HTTPServerThread - [993] Check worker in state [Write] 2026-03-12 03:53:21.978 PM TRACE io.fusionauth.http.server.internal.HTTPServerThread - [993] Check worker in state [Write] 2026-03-12 03:53:23.979 PM TRACE io.fusionauth.http.server.internal.HTTPServerThread - [993] Check worker in state [Write] ... ... 2026-03-12 03:55:54.019 PM TRACE io.fusionauth.http.server.internal.HTTPServerThread - [993] Check worker in state [Write] 2026-03-12 03:55:56.019 PM DEBUG io.fusionauth.http.server.internal.HTTPServerThread - [993] Check worker in state [Write] writingSlow=[true] writeThroughput=[16271] minimumWriteThroughput=[16384] 2026-03-12 03:55:56.019 PM DEBUG io.fusionauth.http.server.internal.HTTPServerThread - [993] Closing connection readingSlow=[false] writingSlow=[true] timedOut=[false] Min write throughput [16384], actual throughput [16271]. 2026-03-12 03:55:56.019 PM DEBUG io.fusionauth.http.server.internal.HTTPServerThread - [993] Closing client connection [/127.0.0.1:36278] due to inactivity 2026-03-12 03:55:56.021 PM TRACE io.fusionauth.http.server.internal.HTTPServerThread - Thread dump from server side. 2026-03-12 03:55:56.024 PM DEBUG io.fusionauth.http.server.internal.HTTPWorker - [993] Closing socket. The socket was closed by a client, proxy or otherwise.

The HTTPWorker begins the process of reading and writing the response for the request, and continues for 2.6 minutes until the HTTPServerThread terminates it because the minimumWriteThroughput (16KB/sec) threshold has not been met.

HTTPServerCleanerThread kills the server-side connection mid-response due to a write throughput check that measures average bytes/sec since the first socket write.
The client is waiting for data that the server is generating very slowly.

Analyzing why the server is writing so slowly (16271 bytes/sec):
The call is http://localhost:9012 — FusionAuth calls itself. Client and server virtual threads share the same JVM's carrier thread pool.
With Kubernetes CPU limit = 1000m, JDK 21 uses UseContainerSupport (default), so availableProcessors() = 1 → only 1 carrier thread.

com.inversoft.rest.JSONResponseHandler$BetterBufferedInputStream.read(byte[], int, int) is synchronized — causes virtual thread carrier thread pinning in JDK 21.

Client virtual thread enters synchronized read() → calls ChunkedInputStream.fastRead() → calls socket.read() which blocks → PINS the carrier thread (can't unmount from synchronized block) Server virtual thread needs to write more data but cannot get a carrier thread (carrier thread pinned by client) Neither makes progress → throughput decays over time as numberOfBytesWritten is fixed but elapsed keeps growing

BetterBufferedInputStream.read() pins that single carrier thread while blocking on ChunkedInputStream.fastRead(), leaving no carrier thread for the server to write more data. The resulting ping-pong limits throughput to ~16 KB/sec on a 2.5MB response.

HTTPServerCleanerThread computes average throughput since the very first write (not a recent window). With 2.5MB written at 16271 bytes/sec average, that's ~156 seconds (2.6 min) before the average decays below the 16384 bytes/sec threshold.

With 9 themes, the response is small enough to fit entirely within the kernel’s socket buffer (~128KB). The server writes the entire content in one go without blocking on socket.write() (the buffer does not fill up), transitions to the KeepAlive state, and the cleaner can no longer terminate the connection due to write throughput.

As a workaround, I resolved the issue increasing the Kubernetes CPU limit from 1000m to 1050m.
Kubernetes translates the CPU limit into the container’s cgroups, setting 2 carrier threads instead of 1.

With that CPU limit, all themes are recovered properly, also with 35 themes (~6.2 MB response).

Can you reproduce it setting Kubernetes CPU limit to 1000m (1 core) in your environment?