[GRADLE-2508] Bounded message queues can deadlock Created: 03/Oct/12  Updated: 10/Feb/17  Resolved: 10/Feb/17

Status: Resolved
Project: Gradle
Affects Version/s: 1.2
Fix Version/s: None

Type: Bug
Reporter: Ronald Blaschke Assignee: Unassigned
Resolution: Won't Fix Votes: 0


 Description   

AsyncDispatch uses a bounded queue (MAX_QUEUE_SIZE = 200). Messaging uses a few of these queues, dispatching Runnables, and the way they interact may cause a deadlock.

The issue is a bit involved, but seems to go like this: Consider a full "message router" queue, filled with events of a TestResultProcessor, trying to dispatch to the socket connection "dispatch" queue.
At the same time, the socket connection "dispatch" queue is full as well, but tries to dispatch a RemoveMethodInvocation(DefaultTestClassRunInfo) to the "message router." Since both queues are full, neither makes progress.

I have attached the stack traces of such an incident below. It is not terribly useful though, because you can't see the queues involved.

The situation happens very often on my machine with parallel test execution. It is easy to spot, because one Test worker process will wait forever.

To work around this I am using a locally patched Gradle with a huge MAX_QUEUE_SIZE.

"Outgoing Connection [3bcde430-6549-42f1-829a-3f7ca27a757d port:3792, addresses:[/127.0.0.1, /0:0:0:0:0:0:0:1]]:5 message router" prio=6 tid=0x18fb9400 nid=0x1c14 waiting on condition [0x18b2f000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x09ed90a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
	at org.gradle.messaging.dispatch.AsyncDispatch.dispatch(AsyncDispatch.java:141)
	at org.gradle.messaging.remote.internal.ProtocolStack$TopConnection.dispatch(ProtocolStack.java:311)
	at org.gradle.messaging.remote.internal.AsyncConnectionAdapter.dispatch(AsyncConnectionAdapter.java:58)
	at org.gradle.messaging.remote.internal.MethodInvocationMarshallingDispatch.dispatch(MethodInvocationMarshallingDispatch.java:58)
	at org.gradle.messaging.remote.internal.MethodInvocationMarshallingDispatch.dispatch(MethodInvocationMarshallingDispatch.java:28)
	at org.gradle.messaging.dispatch.QueuingDispatch.dispatch(QueuingDispatch.java:59)
	at org.gradle.messaging.remote.internal.Router$Endpoint.dispatchIncoming(Router.java:124)
	at org.gradle.messaging.remote.internal.Router$Group.send(Router.java:164)
	at org.gradle.messaging.remote.internal.Router$Endpoint$2.run(Router.java:104)
	at org.gradle.messaging.remote.internal.Router$1.dispatch(Router.java:50)
	at org.gradle.messaging.remote.internal.Router$1.dispatch(Router.java:48)
	at org.gradle.messaging.dispatch.AsyncDispatch.dispatchMessages(AsyncDispatch.java:132)
	at org.gradle.messaging.dispatch.AsyncDispatch.access$000(AsyncDispatch.java:33)
	at org.gradle.messaging.dispatch.AsyncDispatch$1.run(AsyncDispatch.java:72)
	at org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecutorImpl$1.run(DefaultExecutorFactory.java:66)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)

"socket connection at /127.0.0.1:3835 with /127.0.0.1:3792 dispatch" prio=6 tid=0x18fc8000 nid=0x1534 waiting on condition [0x19c5f000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x09ed9278> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
	at org.gradle.messaging.dispatch.AsyncDispatch.dispatch(AsyncDispatch.java:141)
	at org.gradle.messaging.remote.internal.Router$Endpoint.dispatch(Router.java:88)
	at org.gradle.messaging.remote.internal.Router$Endpoint.dispatch(Router.java:67)
	at org.gradle.messaging.remote.internal.MethodInvocationUnmarshallingDispatch.dispatch(MethodInvocationUnmarshallingDispatch.java:55)
	at org.gradle.messaging.remote.internal.MethodInvocationUnmarshallingDispatch.dispatch(MethodInvocationUnmarshallingDispatch.java:28)
	at org.gradle.messaging.dispatch.FailureHandlingDispatch.dispatch(FailureHandlingDispatch.java:29)
	at org.gradle.messaging.dispatch.QueuingDispatch.dispatch(QueuingDispatch.java:59)
	at org.gradle.messaging.remote.internal.ProtocolStack$TopStage.handleIncoming(ProtocolStack.java:260)
	at org.gradle.messaging.remote.internal.ProtocolStack$ProtocolStage$1.run(ProtocolStack.java:185)
	at org.gradle.messaging.remote.internal.ProtocolStack$ExecuteRunnable.dispatch(ProtocolStack.java:120)
	at org.gradle.messaging.remote.internal.ProtocolStack$ExecuteRunnable.dispatch(ProtocolStack.java:116)
	at org.gradle.messaging.dispatch.AsyncDispatch.dispatchMessages(AsyncDispatch.java:132)
	at org.gradle.messaging.dispatch.AsyncDispatch.access$000(AsyncDispatch.java:33)
	at org.gradle.messaging.dispatch.AsyncDispatch$1.run(AsyncDispatch.java:72)
	at org.gradle.internal.concurrent.DefaultExecutorFactory$StoppableExecutorImpl$1.run(DefaultExecutorFactory.java:66)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)


 Comments   
Comment by Szczepan Faber [ 05/Oct/12 ]

Thank you very much for thorough investigation. We'll have a look!

Comment by Adam Murdoch [ 30/Apr/13 ]

This should be fixed in 1.4, at least for test execution. Can you verify that this is the case and let us know?

Comment by Ronald Blaschke [ 03/May/13 ]

Yes, this works nicely now. Thank you very much!

Comment by Benjamin Muschko [ 15/Nov/16 ]

As announced on the Gradle blog we are planning to completely migrate issues from JIRA to GitHub.

We intend to prioritize issues that are actionable and impactful while working more closely with the community. Many of our JIRA issues are inactionable or irrelevant. We would like to request your help to ensure we can appropriately prioritize JIRA issues you’ve contributed to.

Please confirm that you still advocate for your JIRA issue before December 10th, 2016 by:

  • Checking that your issues contain requisite context, impact, behaviors, and examples as described in our published guidelines.
  • Leave a comment on the JIRA issue or open a new GitHub issue confirming that the above is complete.

We look forward to collaborating with you more closely on GitHub. Thank you for your contribution to Gradle!

Comment by Benjamin Muschko [ 10/Feb/17 ]

Thanks again for reporting this issue. We haven't heard back from you after our inquiry from November 15th. We are closing this issue now. Please create an issue on GitHub if you still feel passionate about getting it resolved.

Generated at Wed Jun 30 12:23:52 CDT 2021 using Jira 8.4.2#804003-sha1:d21414fc212e3af190e92c2d2ac41299b89402cf.