Due to this bug, it is not possible to run Gradle internally for some component testings using our prototype of replacing the application class loader with a non-URLClassLoader.

Is it possible to implement a simple workaround to check if the application class loader is an instance of URLClassLoader before the cast in Gradle 2.5 release?

It was pointed out that the reason why Gradle appends new paths to the classpath of the application class loader to workaround the windows ocmmand-line length limitation.

@argfile support is added in JDK 9 b80 that can be used to replace the current workaround:
https://bugs.openjdk.java.net/browse/JDK-8027634

JDK9 has added a new feature that could be used to solve this issue.

As I understand it, this feature of using the BootstrapSecurityManager was designed to work around the length of classpath arguments which can be very, very long with a complex process or a deep filesystem.

But JDK9 finally introduced @argfiles to the java command (something available to javac since JDK7).

So....

In ./subprojects/core/src/main/groovy/org/gradle/process/internal/child/ApplicationClassesInSystemClassLoaderWorkerFactory.java

As a togglable option, instead of creating the subprocess with:
-Djava.security.manager=jarjar.org.gradle.process.internal.child.BootstrapSecurityManager
and serializing the classpath and alternate security manager, Write the classpath to a temporary file
in the form of:
-classpath
"path_to_first_element:\
path_to_second_element:\
...
path_to_last_element"
Some notes on this format. Anything in quotes is white space compressed. I use this feature to indent my path elements, but this is not really needed. The \ at the end is a continuation as would be expected.

With the classpath written to a temp file (for example classpath.txt), invocation becomes:
java @classpath.txt [the rest of the arguments]
If an alternate security manager is requested, then that would be pasted on the command line too. Note that the whole command argument to java could be stuffed into the @argfile, so it really becomes a question of style and taste. Sometimes I prefer to make it -classpath @classpath.txt, leaving the text file to be only the classpath arguments - it might be a little bit clearer for debugging instead of just @classpath.txt

Then of course, don't serialize the classpath and security manager.

I think that there would not need to be any changes to GradleWorkerMain

BTW - using @argfile could certainly be toggled automatically if the running Java is >= JDK9, but it would be nice to have a user level toggle to force the issue. My team needs to build with JDK8 for a number of reasons - but would like to be able to use JDK9 for our JUnit testing, and so I want to be able to specify the test executable:
test

Two minor issues holding me up with trying this and giving you a patch. I requested the process be started on getting approval for the Contributors Agreement, but that might take quite a while given the process (and lawyers) in a large corporation. The last issue is understanding how to expose the toggle to user level gradle.