[GRADLE-1554] Allow build to continue on test failure, yet still end with a failed status Created: 16/May/11  Updated: 10/Feb/17  Resolved: 10/Feb/17

Status: Resolved
Project: Gradle
Affects Version/s: 1.0-milestone-3
Fix Version/s: None

Type: Improvement
Reporter: Luke Daley Assignee: Unassigned
Resolution: Won't Fix Votes: 13


 Description   

The "ignoreFailures" flag on the test task produces the desired behaviour by having the build continue on test failure, but the build finished with a success status which is confusing.

This feature is needed (at least) for long builds in a CI environment. Typically, developers want the CI build to run all the tests to get the most accurate feedback, but don't want the build to be considered successful if any of the tests failed.



 Comments   
Comment by Steve Saliman [ 24/Jul/12 ]

While it is not as clean as deferring failure for later, There is a workaround for this issue. The details can be found at http://forums.gradle.org/gradle/topics/can_you_set_a_property_to_defer_test_failure

Comment by Per Lambæk [ 06/Jan/14 ]

Any plan on prioritizing this? To me it seems like a pretty basic feature.

Comment by Luke Daley [ 06/Jan/14 ]

Does --continue not do what you want?

Comment by Per Lambæk [ 06/Jan/14 ]

Ah, that looks exactly right, my bad, but maybe being able to configure this, especially on test tasks, would make sense, instead of using a switch that enables it for all tasks. But as a workaround it should work, thanks.

Comment by Luke Daley [ 06/Jan/14 ]

Per Lambæk I agree, there's still room for improvement here.

Comment by Mathias Birkkjær Nielsen [ 09/Jan/14 ]

We have the same problem on our pipeline. We have set ignoreFailures and made jenkins detect failures. This works fine, but then when developers run the tests locally it doesn't fail.

The --continue solution seems like a bit of a hack for a pretty common situation. If all task failures are ignored, then it is harder to detect what is actually wrong (because failures ripples down to the next tasks).

Multiple failed tests are perfectly fine when tests are independent (one test failure doesn't result in another test failure).

Comment by Luke Daley [ 09/Jan/14 ]

Can you elaborate on how --continue is not what you want?

If you want to make this implicitly on always, add this to your build script:

gradle.startParameter.continueOnFailure = true

Comment by Mathias Birkkjær Nielsen [ 10/Jan/14 ]

Well i'm afraid I did not read the docs careful enough.
From --continue:
"If a task fails, any subsequent tasks that were depending on it will not be executed, as it is not safe to do so".
That was what i was missing. So you're right. It does what i want.

Still it could be nice as a test task option continueOnFailure, making it more of a feature specifically for tests, not your entire build.

Comment by Benjamin Muschko [ 15/Nov/16 ]

As announced on the Gradle blog we are planning to completely migrate issues from JIRA to GitHub.

We intend to prioritize issues that are actionable and impactful while working more closely with the community. Many of our JIRA issues are inactionable or irrelevant. We would like to request your help to ensure we can appropriately prioritize JIRA issues you’ve contributed to.

Please confirm that you still advocate for your JIRA issue before December 10th, 2016 by:

  • Checking that your issues contain requisite context, impact, behaviors, and examples as described in our published guidelines.
  • Leave a comment on the JIRA issue or open a new GitHub issue confirming that the above is complete.

We look forward to collaborating with you more closely on GitHub. Thank you for your contribution to Gradle!

Comment by Benjamin Muschko [ 10/Feb/17 ]

Thanks again for reporting this issue. We haven't heard back from you after our inquiry from November 15th. We are closing this issue now. Please create an issue on GitHub if you still feel passionate about getting it resolved.

Generated at Wed Jun 30 11:58:28 CDT 2021 using Jira 8.4.2#804003-sha1:d21414fc212e3af190e92c2d2ac41299b89402cf.