Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UNDERTOW-2424: Prevent connection access-after-free allowed through ChunkedStreamSinkConduit #1648

Merged
merged 1 commit into from
Aug 22, 2024

Conversation

carterkozak
Copy link
Contributor

@carterkozak carterkozak commented Aug 13, 2024

Jira: https://issues.redhat.com/browse/UNDERTOW-2424
2.2.x PR: #1652
2.3.x PR: #1653

ChunkedStreamSinkConduit releases underlying connections for reuse via a conduit listener, however flush calls may still be passed through after writes are terminated and successfully flushed. This would cause requests to be clobbered when the connection had already been reused to send another request, breaking the threading model.

This resolves the issue from #1495

For posterity: My first attempt at a fix was #1645 which resolves the issue, but I think this PR is cleaner and more precise. As always, I'm happy to throw the code out if you prefer a different approach, the important piece is the analysis of the root cause of the failures, which I believe I understand at this point :-)

…hunkedStreamSinkConduit

ChunkedStreamSinkConduit releases underlying connections for reuse via a
conduit listener, however flush calls may still be passed through
after writes are terminated and successfully flushed. This would cause
requests to be clobbered when the connection had already been reused
to send another request, breaking the threading model.
@carterkozak
Copy link
Contributor Author

Specifics of the scenario in #1495:

(note that line numbers may be slightly off due to debug statements I added in testing, method names/paths should be equivalent)

The exchange is completed here:

    at io.undertow.server.protocol.http.HttpServerConnection.exchangeComplete(HttpServerConnection.java:244)
    at io.undertow.server.HttpServerExchange.invokeExchangeCompleteListeners(HttpServerExchange.java:1373)
    at io.undertow.server.HttpServerExchange.terminateResponse(HttpServerExchange.java:1658)
    at io.undertow.server.Connectors.terminateResponse(Connectors.java:182)
    at io.undertow.server.protocol.http.HttpTransferEncoding$3.handleEvent(HttpTransferEncoding.java:199)
    at io.undertow.server.protocol.http.HttpTransferEncoding$3.handleEvent(HttpTransferEncoding.java:197)
    at io.undertow.conduits.ChunkedStreamSinkConduit.invokeFinishListener(ChunkedStreamSinkConduit.java:295)
    at io.undertow.conduits.ChunkedStreamSinkConduit.flush(ChunkedStreamSinkConduit.java:280)
    at io.undertow.conduits.ChunkedStreamSinkConduit.terminateWrites(ChunkedStreamSinkConduit.java:324)
    at io.undertow.conduits.DeflatingStreamSinkConduit.flush(DeflatingStreamSinkConduit.java:376)
    at org.xnio.conduits.ConduitStreamSinkChannel.flush(ConduitStreamSinkChannel.java:162)
    at io.undertow.channels.DetachableStreamSinkChannel.flush(DetachableStreamSinkChannel.java:125)
    at org.xnio.channels.Channels.flushBlocking(Channels.java:63)
    at io.undertow.servlet.spec.ServletOutputStreamImpl.close(ServletOutputStreamImpl.java:627)
    at io.undertow.servlet.spec.HttpServletResponseImpl.closeStreamAndWriter(HttpServletResponseImpl.java:513)
    at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:609)
    at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:336)
    at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:136)
    at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133)
    at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
    at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
    at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:257)
    at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:102)
    at io.undertow.server.Connectors.executeRootHandler(Connectors.java:393)
    at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:867)
    at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
    at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1512)
    at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
    at java.base/java.lang.Thread.run(Thread.java:833)

However DeflatingStreamSinkConduit continues from next.terminateWrites() (which internally calls flush on the ChunkedStreamSinkConduit and completes the exchange) to next.flush().

next.terminateWrites();
return next.flush();

with roughly this stack trace:

    at io.undertow.server.protocol.http.HttpResponseConduit.processWrite(HttpResponseConduit.java:152)
    at io.undertow.server.protocol.http.HttpResponseConduit.flush(HttpResponseConduit.java:801)
    at io.undertow.conduits.ChunkedStreamSinkConduit.flush(ChunkedStreamSinkConduit.java:265)
    at io.undertow.conduits.DeflatingStreamSinkConduit.flush(DeflatingStreamSinkConduit.java:377)
    at org.xnio.conduits.ConduitStreamSinkChannel.flush(ConduitStreamSinkChannel.java:162)
    at io.undertow.channels.DetachableStreamSinkChannel.flush(DetachableStreamSinkChannel.java:125)
    at org.xnio.channels.Channels.flushBlocking(Channels.java:63)
    at io.undertow.servlet.spec.ServletOutputStreamImpl.close(ServletOutputStreamImpl.java:627)
    at io.undertow.servlet.spec.HttpServletResponseImpl.closeStreamAndWriter(HttpServletResponseImpl.java:513)
    at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:609)
    at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:335)
    at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
    at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:132)
    at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
    at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
    at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:256)
    at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:101)
    at io.undertow.server.Connectors.executeRootHandler(Connectors.java:393)
    at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:867)
    at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
    at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
    at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
    at java.base/java.lang.Thread.run(Thread.java:833)

The problem is, once the exchange ended, another request has already been parsed, so a flush invocation will potentially re-write headers to an unexpected location, from a thread not allowed by the threading model, or potentially commit the next exchanges response prematurely. This can also throw the state out of sync because the underlying response conduit state is not volatile (nor should it be), however the illegal access/mutation is likely to result in inconsistent views.
Let me know if this explanation makes sense, happy to chat in greater detail if that's helpful :-)

@@ -200,6 +200,9 @@ int doWrite(final ByteBuffer src) throws IOException {

@Override
public void truncateWrites() throws IOException {
if (anyAreSet(state, FLAG_FINISHED)) {
return;
}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we have another subtle bug in this method. The invokeFinishListener() is called prior to the finally, so the connection may already be used by another thread by the time we call truncateWrites. I'll push an update shortly.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On second thought, this shouldn't matter in practice because terminateWrites will close the underlying persistent connection, so we don't need to worry about reuse.

@baranowb baranowb added bug fix Contains bug fix(es) under verification Currently being verified (running tests, reviewing) before posting a review to contributor waiting peer review PRs that edit core classes might require an extra review labels Aug 22, 2024
@fl4via fl4via added next release This PR will be merged before next release or has already been merged (for payload double check) and removed under verification Currently being verified (running tests, reviewing) before posting a review to contributor waiting peer review PRs that edit core classes might require an extra review labels Aug 22, 2024
@fl4via fl4via merged commit db09a8b into undertow-io:main Aug 22, 2024
34 checks passed
@fl4via fl4via removed the next release This PR will be merged before next release or has already been merged (for payload double check) label Oct 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug fix Contains bug fix(es)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants