[GRADLE-1946] Gradle memory usage is too high Created: 20/Nov/11  Updated: 10/Feb/17  Resolved: 10/Feb/17

Status: Resolved
Project: Gradle
Affects Version/s: 1.0-milestone-6
Fix Version/s: None

Type: Bug
Reporter: Andrew Schetinin Assignee: Unassigned
Resolution: Won't Fix Votes: 2

Attachments: Text File exception1.txt    

 Description   

Hi,

Today I've got an out of memory exception while running the build.
I've checked the processes memory usage, and found that Gradle seems to accumulate a lot of memory when running the unit tests (at least I suspect so).

There are 2 Java processes running - one that actually runs the build, and another one running the unit tests.

The process that runs the unit tests happily jumps from 200MB to 500-800MB and back - it's okay because it returns the memory back.

But the main Gradle process is problematic - it starts with about 600MB, slowly grows to 800MB while compiling and testing some modules, and then at some point, after executing some 150+ unit tests, it accidentally jumps to 1,150 MB and remains there even on more light-weight modules.

A bit too much for a mere build script, don't you think?

I have a feeling that it happens right after executing the unit tests, when HTML/XML reports are being generated. But that's only suspicion. The trouble is that the memory is not returned after passing that heavy module.

I first had it with the snapshot 5, and after installing the snapshot 6 it did not repeat.

When I increased the memory usage with JAVA_OPTS, the build passed (actually it passes occasionally even without increasing the max. Java memory - the crash seems to be an accidental).

I'm not using the gradle daemon - the build starts from scratch every time.
And yes, it is a clean build, so it is not about incremental build leak (as in the ticket 1443).

Regards,

Andrew



 Comments   
Comment by Andrew Schetinin [ 20/Nov/11 ]

Sorry, I've made a mistake in one critical sentence:

"I first had it with the snapshot 5, and after installing the snapshot 6 it did not repeat."

should be:

"I first had it with the snapshot 5, and after installing the snapshot 6 IT REPEATED."

Comment by Adam Murdoch [ 20/Nov/11 ]

Can you run with -XX:+HeapDumpOnOutOfMemoryError in your GRADLE_OPTS, and send us the resulting heap dump?

Comment by Peter Lynch [ 01/Dec/11 ]

I tried to build the spock framework with gradle and got OOM. Should be easy to repro using current master. Took a heap dump in case it is the similar problem since it fails while running tests.

Heap dumps are too large to attach here ( you have 10MB attach limit).

So get from my dropbox:
http://dl.dropbox.com/u/14591789/java_pid9295.hprof.zip

Comment by Andrew Schetinin [ 01/Dec/11 ]

BTW, I'm constantly running with HeapDumpOnOutOfMemoryError, but meanwhile the problem did not reproduce on my machine. I'll probably try limiting JVM heap size, just for the sake of creating that heap dump.

Comment by Andrew Schetinin [ 12/Dec/11 ]

I've got OOM, but a heap dump file was not created. I only got the exception (attached). I've searched everywhere, but could not find a dump file. Is it possible that the exception was wrapped in another exception and no dump file was created?

Comment by Andrew Schetinin [ 12/Dec/11 ]

OOM error stack trace

Comment by Scott Palmer [ 05/Jun/12 ]

I am also running into this. With the max heap set to 640M .. gradle slows to a crawl spending at least half the time in GC. I need to run on 32-bit Java and sometimes when I up the heap to 1024M there can be problems allocating such a large block of memory.
The thing is, my build is just invoking gradle (from a gradle script) multiple times on a bunch of other projects... each project on it's own doesn't need much memory so after each one is done the heap usage should fall back and stabilize.. but the memory usage grows and seems excessive.

This is with 1.0-rc-3 (has been a problem since I can remember with milestone-3)
Is this really a 'minor' issue? I causes builds to fail and seems to be a real leak of some sort.

Comment by Andrew Schetinin [ 06/Jun/12 ]

BTW, in my case the problem was with the unit tests runner - my unit tests were not optimized and accumulated too much memory during their execution.

I worked around it by forking a separate processes per test class. It can be configured in the master/build.gradle like this (see the first line with forkEvery):

build.gradle
        test {
          forkEvery = 1 // prevent Java OOM error on over 1 GB of mem usage

          excludes = ["**/Abstract*.class"]

          // reports about too long unit tests
          afterTest { descr, result ->
                        def time = result.endTime - result.startTime
                        if( time > 3000 )
                                logger.warn( "Test ${descr.className} ${descr.name} takes too much time: $time msec" )
          }
        }

Forking separate test processes did not increase the total execution time by more than 5-10% (AFAIR), and it significantly reduced the memory usage.

Comment by Benjamin Muschko [ 15/Nov/16 ]

As announced on the Gradle blog we are planning to completely migrate issues from JIRA to GitHub.

We intend to prioritize issues that are actionable and impactful while working more closely with the community. Many of our JIRA issues are inactionable or irrelevant. We would like to request your help to ensure we can appropriately prioritize JIRA issues you’ve contributed to.

Please confirm that you still advocate for your JIRA issue before December 10th, 2016 by:

  • Checking that your issues contain requisite context, impact, behaviors, and examples as described in our published guidelines.
  • Leave a comment on the JIRA issue or open a new GitHub issue confirming that the above is complete.

We look forward to collaborating with you more closely on GitHub. Thank you for your contribution to Gradle!

Comment by Benjamin Muschko [ 10/Feb/17 ]

Thanks again for reporting this issue. We haven't heard back from you after our inquiry from November 15th. We are closing this issue now. Please create an issue on GitHub if you still feel passionate about getting it resolved.

Generated at Wed Jun 30 12:08:37 CDT 2021 using Jira 8.4.2#804003-sha1:d21414fc212e3af190e92c2d2ac41299b89402cf.