Please provide the exact error message and stack trace. Which OS? How are you leveraging the daemon/tooling API?

Currently I cannot give you a stacktrace because regretably I did not save it. Next time it occurs, I will. Note however, that this problem does not occur very frequently but once it does, the problem will persist until the daemon folder is deleted (not even system restart helps).

I have seen this issue on Windows systems (7, Vista, Xp) and not tried with others.

I use the tooling API to provide a plugin for NetBeans: [1]https://github.com/kelemen/netbeans-g... All code relevant to this issue can be found here: [2]https://github.com/kelemen/netbeans-g...
----------------------------------------------------------------------------------------
[1] https://github.com/kelemen/netbeans-gradle-project
[2] https://github.com/kelemen/netbeans-gradle-project/blob/master/src/org/netbeans/gradle/project/model/GradleModelLoader.java

Luckily, I found the stack trace in the log of NetBeans:

org.gradle.launcher.daemon.client.NoUsableDaemonFoundException: Unable to find a usable idle daemon. I have connected to 100 different daemons but I could not use any of them to run build: Build

.
at org.gradle.launcher.daemon.client.DaemonClient.execute(DaemonClient.java:124)
at org.gradle.launcher.daemon.client.DaemonClient.execute(DaemonClient.java:57)
at org.gradle.tooling.internal.provider.DaemonGradleLauncherActionExecuter.execute(DaemonGradleLauncherActionExecuter.java:43)
at org.gradle.tooling.internal.provider.DaemonGradleLauncherActionExecuter.execute(DaemonGradleLauncherActionExecuter.java:30)
at org.gradle.tooling.internal.provider.LoggingBridgingGradleLauncherActionExecuter.execute(LoggingBridgingGradleLauncherActionExecuter.java:53)
at org.gradle.tooling.internal.provider.LoggingBridgingGradleLauncherActionExecuter.execute(LoggingBridgingGradleLauncherActionExecuter.java:30)
at org.gradle.tooling.internal.provider.DefaultConnection.run(DefaultConnection.java:114)
at org.gradle.tooling.internal.provider.DefaultConnection.getTheModel(DefaultConnection.java:108)
at org.gradle.tooling.internal.consumer.connection.AdaptedConnection.getModel(AdaptedConnection.java:53)
at org.gradle.tooling.internal.consumer.ModelProvider.provide(ModelProvider.java:57)
at org.gradle.tooling.internal.consumer.connection.LazyConnection$2.run(LazyConnection.java:111)
at org.gradle.tooling.internal.consumer.connection.LazyConnection.withConnection(LazyConnection.java:119)
at org.gradle.tooling.internal.consumer.connection.LazyConnection.getModel(LazyConnection.java:108)
at org.gradle.tooling.internal.consumer.connection.ProgressLoggingConnection$2.run(ProgressLoggingConnection.java:67)
at org.gradle.tooling.internal.consumer.connection.ProgressLoggingConnection.run(ProgressLoggingConnection.java:81)
at org.gradle.tooling.internal.consumer.connection.ProgressLoggingConnection.getModel(ProgressLoggingConnection.java:65)
at org.gradle.tooling.internal.consumer.connection.LoggingInitializerConnection.getModel(LoggingInitializerConnection.java:53)
at org.gradle.tooling.internal.consumer.async.DefaultAsyncConnection$2.run(DefaultAsyncConnection.java:61)
at org.gradle.tooling.internal.consumer.async.DefaultAsyncConnection$3.run(DefaultAsyncConnection.java:79)
Caused: org.gradle.tooling.GradleConnectionException: Could not fetch model of type 'IdeaProject' using Gradle installation 'C:\Program Files\Gradle\gradle-1.1'.
at org.gradle.tooling.internal.consumer.ResultHandlerAdapter.onFailure(ResultHandlerAdapter.java:55)
at org.gradle.tooling.internal.consumer.async.DefaultAsyncConnection$3.run(DefaultAsyncConnection.java:81)
at org.gradle.messaging.concurrent.DefaultExecutorFactory$StoppableExecutorImpl$1.run(DefaultExecutorFactory.java:66)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
[catch] at java.lang.Thread.run(Thread.java:722)

Attila, do you think it would be possible to reproduce the issue in reasonable time? Can you provide the steps?

When the problem occurs next time, before deleting the daemon folder can you zip it up and attach it to the jira ticket (I'll create the ticket). The daemon folder should contain logs from the daemons.

I have attached the daemon folder when the problem occurred.

I have attached what Gradle wrote to the console when using "-s -debug". I did is by running Gradle from NetBeans however (because for some reason, the problem did not appear when running from the command line, though it usually does).

It simply tries to connect to a bunch of ports on the localhost to which it cannot (obviously, since no process named "java" is running, so there is no daemon).

The error message "A létező kapcsolatot a távoli állomás kényszerítetten bezárta" means that "The existing connection has been forcibly closed by the remote host". I assume that this is WSAECONNRESET (10054).

Thanks a lot for the logs. I'll schedule some time next week to dig this problem further. I'll get back.

Cheers!

Hey Attila,

The gradle console output does not seem to be a debug output. Are you sure you've run with --debug? It would be very useful to see the debug log. If you happen to reproduce the issue with --debug please also attach the matching daemon logs (from the daemon folder).

Thanks a lot for help!

I think I have misread it and used "-debug", sorry. I'll do a different run as soon as I'm able.

I have executed Gradle with "s" and "-debug" argument with the Tooling API but it did not write anything else (beyond what is attached) to the stdout nor to the console (actually nothing is written to the stdout only to the console). I'm sure that the debug log should be displayed if there were any because if I execute a task when the Gradle daemon is working, stdout is flooded with logs. Note that I'm trying to execute Gradle through the Tooling API (rather than from the command line) because I cannot seem to reproduce it from the command line (although I'm sure that it did happen with me with the command line as well). What is the difference from calling the daemon from the command line or from the Tooling API? What might be important to note is that no java process gets spawned when the problem occurs.

What I have tried:

• If I remove the "daemon" directory, the problem disappears immediately.
• If I kill the daemon process and put back the corrupted "daemon" directory, the problem will appear again.
• If I don't kill the daemon and replace the "daemon" directory while a daemon is runnig (except for its log file), the problem will also start to appear.

Thanks a lot for info.

If the problem occurs, can you attach the zip with the corrupted daemon folder?

For logging, when you use the tooling api in the NB plugin, can you temporarily set the verbose logging? You need to cast the GradleConnector to DefaultGradleConnector and use setVerboseLogging() method.

>I have executed Gradle with "s" and "-debug" argument with the Tooling API but it did not write anything else

You can try using the methods on the LongRunningOperation to capture the standard output / standard error of the tooling api execution. Tell me if it helps getting the logs.

>What is the difference from calling the daemon from the command line or from the Tooling API?

The difference is that the client jar is embedded in the host application. However, it shouldn't be problem. Depending on how you configure the daemon via the GradleConnector, they might use different Gradle version, different gradle user home, etc.

Can you tell me what happens if you run builds from command line when the problem starts occurring? If command line Gradle uses the same version / gradle user home as your embedded implementation then you should be able to reuse the same daemon that is used in NB. This should give us a way to reproduce the problem from command line and get a nice --debug log.

Thanks a lot for helping out!

The already attached daemon folder reproduces the problem for me but seeing the debug output, it is very likely that it will only reproduce for me.

These are the additional logs being emitted with verbose logging:

10:49:27.819 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire shared lock on daemon addresses registry.
10:49:27.820 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired.
10:49:27.822 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
10:49:27.824 [DEBUG] [org.gradle.launcher.daemon.client.DefaultDaemonConnector] Found daemon (address: [1a7aa334-1862-4a51-9c18-7d02cb92f724 port:54204, addresses:[/127.0.0.1, /0:0:0:0:0:0:0:1]], idle: true) however it's context does not match the desired criteria.
Java home is different.
Wanted: DefaultDaemonContext[uid=null,javaHome=C:\Program Files\Java\jdk1.7.0_05,daemonRegistryDir=C:\Users\User_2\.gradle\daemon,pid=5668,idleTimeout=null,daemonOpts=-XX:MaxPermSize=256m,-XX:+HeapDumpOnOutOfMemoryError,-Xmx1024m,-Dfile.encoding=windows-1250]
Actual: DefaultDaemonContext[uid=d99a3e4d-418c-453c-8ac0-bfabe6ef274c,javaHome=C:\Program Files\Java\jdk1.7.0_04,daemonRegistryDir=C:\Users\User_2\.gradle\daemon,pid=3228,idleTimeout=10800000,daemonOpts=-XX:MaxPermSize=256m,-XX:+HeapDumpOnOutOfMemoryError,-Xmx1024m,-Dfile.encoding=windows-1250]

Looking for a different daemon...
10:49:27.825 [DEBUG] [org.gradle.messaging.remote.internal.inet.TcpOutgoingConnector] Attempting to connect to [d5b60243-6707-4b71-9c4d-afe682704368 port:60748, addresses:[/127.0.0.1, /0:0:0:0:0:0:0:1]].
10:49:27.825 [DEBUG] [org.gradle.messaging.remote.internal.inet.TcpOutgoingConnector] Trying to connect to address /127.0.0.1.
10:49:27.836 [DEBUG] [org.gradle.messaging.remote.internal.inet.TcpOutgoingConnector] Connected to address /127.0.0.1.

As you can see, I have multiple JDK installed and I use different JDK with NetBeans than the one specified by JAVA_HOME (this is unintentional but I think, this could cause the problem for Gradle).

>Can you tell me what happens if you run builds from command line when the problem starts occurring? If command line Gradle uses the same version / gradle user home as your embedded >implementation then you should be able to reuse the same daemon that is used in NB. This should give us a way to reproduce the problem from command line and get a nice --debug log.

I could not reproduce it from the command line for long time, so I cannot tell you what happens with "--debug" but otherwise it is completely the same. Waits a lot then prints that it has failed after trying a hundred times. I don't think I will be able to reuse the Gradle daemon because when the problem appears, there is no java processes at all, so there is no daemon to reuse.

Different java_homes are not a problem. The thing is that you can connect to the existing idle daemon only if your java home / java args match. If they don't match then new daemon will be forked.

If you make the java home consistent you should be able to reuse the same daemon and maybe reproduce the issue from the command line.

The above log is when the problem occurs? Can you attach a complete one? Above is not conclusive because looking for a different daemon due to a different java is a regular situation and shouldn't be a problem.

I have attached the debug logs written by Gradle and the daemon folder which I currently use to reproduce the issue.

Currently if I run gradle with --daemon from the command line, I cannot reproduce this issue, even though if I run gradle from NetBeans, it does occur. I use the same, installed Gradle distribution for both cases.

There is actually a difference when I'm executing Gradle from the command line: I still use the tooling api from Gradle 1.0 in NetBeans while in the command line, I use 1.1.

I was able to reproduce it from the command line and attached its logs. It does not seem to be much different, except that it has a different call stack.

I have also attached my daemon directory with which I was able to reproduce the problem from the command line. Hopefully with this, you might be able to reproduce it as wel..

Thanks a bunch! I'll take look.

From what I read in the logs here's what happens: no daemon is running but still the client attempts to connect on some address found in the registry. The address is 'connectable' but the daemon on the other side of the wire does not exist (or unlikely: died before writing anything to its log). The problem is reported on windows platform.

I haven't reproduced the issue but I wasn't trying very hard, either.

Here're some options:

1. If we receive an exception (IOException?) when sending build request to the daemon we:
1.1 Remove (forget) this daemon from the registry under the assumption it is 'corrupted'.
1.2 Before that, we can try sending stop request to the daemon. I don't think it will work because the daemon is corrupted and will not successfully execute the 'stop'.
1.3 We could implement more forceful 'stop' that kills the daemon. Not sure if it would help in this scenario because according to user's report no java processes are running anyway.
1.4 Alternatively, before 'forgetting' the daemon, we could try connecting again few times. I don't think this makes sense though - why would the daemon magically start working again?

I'm tempted to go with 1.1 & try harder to reproduce the issue. It would be very useful to find out why the daemon gets corrupted or why does it disappear (no java process) but it is still connectable?

My current guess is that it gets corrupted because when I shutdown Windows and the daemon is still running, possibly Windows does not allow it to write to its file (perhaps because it is a file in the user dir and the daemon gets killed after log out?)

I did a little investigation:

I tried to connect to the port where Gradle believes that the daemon should listen for connections. From a 32 bit app, connection attempt to this port fails. From a 64 bit app, connection succeeds (but reading from it is not possible). I tried "netstat -a -b" and it did not report that anyone is listening on that particular port, however it did print "x: Windows Sockets initialization failed: 5" a few times. What this actually means is a complete mystery to me.

Regardless, if Gradle cannot read from a tcp connection (or cannot parse what it received), it should be safe to assume that no Gradle daemon is listening on that port. I mean, nothing should go wrong on localhost (and I can't see how attaching to a daemon running on a remote host could possibly work, since they do not share the "registry.bin" with the client).

What is far worse, is that I cannot start a server and listen on this port, not even after system restart. This means, that eventually no port will remain. However, this is not a problem with Gradle (i.e., it should not be able to do such thing).

Thanks a lot for the investigation - it is very helpful.

My guess for what's happening here:

1. Daemon starts up, chooses a port and writes this to the registry.
2. Daemon is killed at system shutdown, leaving the entry in the registry.
3. On startup, some other service chooses the port and starts listening on it.
4. Client connects to the port and sends request.
5. Service drops the connection, possibly after some timeout.
6. Repeat 4-5 99 more times.

@Szczepan, we should tidy up a few things here:

1. Daemon should attempt to remove its entry from the registry on JVM exit.
2. All client and daemon reads on the connection should have a timeout.
3. Client removes suspect entries from registry (as discussed).
4. Daemon should exit when its entry is removed from the registry (as discussed).

@Attila, Can you try the latest nightly? (http://gradle.org/nightly) The problem should be resolved now.

I think you can use the older version of Tooling API (say 1.1). The important is that the target Gradle version you connect to is the nightly.

It would be very useful if you tried the nightly and told us if the problem is gone!

Thanks!

I have tried it with the Tooling API I currently use (1.0) and the nightly build set as "useInstallation". I could not reproduce it on the first try, which is a good sign but I will be more confident after a few days.

However, I wanted to replace "registry.bin" with the one attached to try if I can reproduce with it. Attempting to load the model failed with the following exception:

java.io.InvalidClassException: org.gradle.launcher.daemon.registry.DaemonInfo; local class incompatible: stream classdesc serialVersionUID = 1483185088625753413, local class serialVersionUID = 231259346409640868

Although this is unlikely to occur in practice, it would be nice to be fixed as well.

Here is the full stack trace:

java.io.InvalidClassException: org.gradle.launcher.daemon.registry.DaemonInfo; local class incompatible: stream classdesc serialVersionUID = 1483185088625753413, local class serialVersionUID = 231259346409640868
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:604)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1601)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1514)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1750)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at java.util.HashMap.readObject(HashMap.java:1043)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1004)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1866)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1964)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1888)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)
at org.gradle.cache.DefaultSerializer.read(DefaultSerializer.java:44)
at org.gradle.cache.internal.SimpleStateCache.deserialize(SimpleStateCache.java:83)
Caused: org.gradle.api.GradleException: Could not read cache value from 'C:\Users\User_2\.gradle\daemon\1.3-20120902220019+0000\registry.bin'.
at org.gradle.cache.internal.SimpleStateCache.deserialize(SimpleStateCache.java:88)
at org.gradle.cache.internal.SimpleStateCache.access$000(SimpleStateCache.java:26)
at org.gradle.cache.internal.SimpleStateCache$1.create(SimpleStateCache.java:40)
at org.gradle.cache.internal.DefaultFileLockManager$DefaultFileLock.readFile(DefaultFileLockManager.java:141)
at org.gradle.cache.internal.OnDemandFileAccess.readFile(OnDemandFileAccess.java:36)
at org.gradle.cache.internal.SimpleStateCache.get(SimpleStateCache.java:38)
at org.gradle.cache.internal.FileIntegrityViolationSuppressingPersistentStateCacheDecorator.get(FileIntegrityViolationSuppressingPersistentStateCacheDecorator.java:31)
at org.gradle.launcher.daemon.registry.PersistentDaemonRegistry.getAll(PersistentDaemonRegistry.java:64)
at org.gradle.launcher.daemon.registry.PersistentDaemonRegistry.getIdle(PersistentDaemonRegistry.java:79)
at org.gradle.launcher.daemon.client.DefaultDaemonConnector.connect(DefaultDaemonConnector.java:67)
at org.gradle.launcher.daemon.client.DaemonClient.execute(DaemonClient.java:133)
at org.gradle.launcher.daemon.client.DaemonClient.execute(DaemonClient.java:59)
at org.gradle.tooling.internal.provider.DaemonGradleLauncherActionExecuter.execute(DaemonGradleLauncherActionExecuter.java:43)
at org.gradle.tooling.internal.provider.DaemonGradleLauncherActionExecuter.execute(DaemonGradleLauncherActionExecuter.java:30)
at org.gradle.tooling.internal.provider.LoggingBridgingGradleLauncherActionExecuter.execute(LoggingBridgingGradleLauncherActionExecuter.java:53)
at org.gradle.tooling.internal.provider.LoggingBridgingGradleLauncherActionExecuter.execute(LoggingBridgingGradleLauncherActionExecuter.java:30)
at org.gradle.tooling.internal.provider.DefaultConnection.run(DefaultConnection.java:139)
at org.gradle.tooling.internal.provider.DefaultConnection.run(DefaultConnection.java:129)
at org.gradle.tooling.internal.provider.DefaultConnection.getTheModel(DefaultConnection.java:99)
at org.gradle.tooling.internal.consumer.connection.AdaptedConnection.getModel(AdaptedConnection.java:53)
at org.gradle.tooling.internal.consumer.ModelProvider.provide(ModelProvider.java:57)
at org.gradle.tooling.internal.consumer.connection.LazyConnection$2.run(LazyConnection.java:111)
at org.gradle.tooling.internal.consumer.connection.LazyConnection.withConnection(LazyConnection.java:119)
at org.gradle.tooling.internal.consumer.connection.LazyConnection.getModel(LazyConnection.java:108)
at org.gradle.tooling.internal.consumer.connection.ProgressLoggingConnection$2.run(ProgressLoggingConnection.java:67)
at org.gradle.tooling.internal.consumer.connection.ProgressLoggingConnection.run(ProgressLoggingConnection.java:81)
at org.gradle.tooling.internal.consumer.connection.ProgressLoggingConnection.getModel(ProgressLoggingConnection.java:65)
at org.gradle.tooling.internal.consumer.connection.LoggingInitializerConnection.getModel(LoggingInitializerConnection.java:53)
at org.gradle.tooling.internal.consumer.async.DefaultAsyncConnection$2.run(DefaultAsyncConnection.java:61)
at org.gradle.tooling.internal.consumer.async.DefaultAsyncConnection$3.run(DefaultAsyncConnection.java:79)
Caused: org.gradle.tooling.GradleConnectionException: Could not fetch model of type 'IdeaProject' using Gradle installation 'C:\Program Files\Gradle\gradle-1.3-20120902220019+0000'.
at org.gradle.tooling.internal.consumer.ResultHandlerAdapter.onFailure(ResultHandlerAdapter.java:55)
at org.gradle.tooling.internal.consumer.async.DefaultAsyncConnection$3.run(DefaultAsyncConnection.java:81)
at org.gradle.messaging.concurrent.DefaultExecutorFactory$StoppableExecutorImpl$1.run(DefaultExecutorFactory.java:66)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
[catch] at java.lang.Thread.run(Thread.java:722)

Hey Attila,

The error you see is expected. At the moment we don't provide any backwards/forwards compatibility for the daemon registry format (it's using standard java serialization). This is something we want to do in future (it is actually being discussed on the dev mailing list).

Thanks a lot of taking the nightly to spin! Let us know if you run into the problem. Unfortunately, you cannot use the old registry to reproduce the issue.