[GRADLE-3345] antlr task fails if tokenVocab references package declaration Created: 02/Oct/15  Updated: 24/Jan/17  Resolved: 24/Jan/17

Status: Resolved
Project: Gradle
Affects Version/s: 2.7
Fix Version/s: None

Type: Bug
Reporter: René Gröschke (Inactive) Assignee: Unassigned
Resolution: Duplicate Votes: 4


reported via forum: https://discuss.gradle.org/t/antlr-4-behavior-change-between-gradle-2-4-and-2-7/11966/5

I recently switched locally to use Gradle 2.7 in Hibernate to resolve the regression in package generation with Antlr 3 introduced in 2.4.

However, we are developing a new query parser that uses Antlr 4 and that same change torpedos that build. With Gradle 2.7 and Antlr 4 how should one (now) structure the sources? In this new development we still keep the grammar source in a "package structure". But running the Antlr plugin/task against this fails to work properly; I get errors.

This is the repo containing the new parser: https://github.com/hibernate/hibernate-semantic-query/tree/34f556b0e465c584b922e3cfcf4e51ff9b9602033

The wrapper here is defined for 2.4. So if you run the build using the wrapper, it works fine. However, when I try to use 2.7 it fails:

error(160): org/hibernate/query/parser/internal/hql/antlr/HqlParser.g4:4:12: cannot find tokens file /home/sebersole/projects/hibernate/hibernate-semantic-query/hibernate-query-interpreter/build/generated-src/antlr/main/org/hibernate/query/parser/internal/hql/antlr/HqlLexer.tokens

The "problem" is that the plugin creates the package dir structure twice. The output is generated into /home/sebersole/projects/hibernate/hibernate-semantic-query/hibernate-query-interpreter/build/generated-src/antlr/main/org/hibernate/query/parser/internal/hql/antlr/org/hibernate/query/parser/internal/hql/antlr/ rather than /home/sebersole/projects/hibernate/hibernate-semantic-query/hibernate-query-interpreter/build/generated-src/antlr/main/org/hibernate/query/parser/internal/hql/antlr.

I figure part of the problem is the override of generateGrammarSource.outputDirectory (which was needed with 2.4). So I removed that, but that leads to a similar error:

error(160): org/hibernate/query/parser/internal/hql/antlr/HqlParser.g4:4:12: cannot find tokens file /home/sebersole/projects/hibernate/hibernate-semantic-query/hibernate-query-interpreter/build/generated-src/antlr/main/HqlLexer.tokens

Now the generated files are put into the proper directory (/home/sebersole/projects/hibernate/hibernate-semantic-query/hibernate-query-interpreter/build/generated-src/antlr/main/org/hibernate/query/parser/internal/hql/antlr). However, Antlr is not able to find things.

So how am I supposed to get Antlr 4 and Gradle 2.7 playing together nicely (using the Gradle AntlrPlugin)? I realize that there are other Antlr plugins for Gradle, is that my best option?

Comment by René Gröschke (Inactive) [ 03/Oct/15 ]

some more digging:

The importVocab statemtn cannot be used if referencing grammar with package declaration. possible solution:

1. the generated source must be generated in an order that respects `importVocab` dependencies in the grammar.
2. gradle should pass the output directory of a grammar file as -lib folder to allow referencing grammar from same package that has been generated before

Comment by Kenzie Togami [ 02/Apr/16 ]

As it currently stands I think Gradle has 0 support for using packages with separated lexer and parser files, since passing in `-lib` and `package` myself does nothing. I'd like to be able to put my grammars in a package, but that seems to be impossible as it currently stands.

Comment by Steve Ebersole [ 21/Jul/16 ]

In the meantime, is there at least any workaround for this? It stinks to have to stick with 2.4 to get working Antlr 4 support.

@Rene, I do not use importVocab (just tokenVocab in the parser) with the Antlr 4 grammars. So the issue is not limited to that

@Kenzie, fwiw I use split lexer and parser design with packages and it "works".. so long as you stick to Gradle 2.4. In my experience I have not been able to move past 2.4 or it breaks.

Comment by Benjamin Muschko [ 15/Nov/16 ]

As announced on the Gradle blog we are planning to completely migrate issues from JIRA to GitHub.

We intend to prioritize issues that are actionable and impactful while working more closely with the community. Many of our JIRA issues are inactionable or irrelevant. We would like to request your help to ensure we can appropriately prioritize JIRA issues you’ve contributed to.

Please confirm that you still advocate for your JIRA issue before December 10th, 2016 by:

  • Checking that your issues contain requisite context, impact, behaviors, and examples as described in our published guidelines.
  • Leave a comment on the JIRA issue or open a new GitHub issue confirming that the above is complete.

We look forward to collaborating with you more closely on GitHub. Thank you for your contribution to Gradle!

Comment by Steve Ebersole [ 15/Nov/16 ]

I still "advocate" for a total revamp of the Antlr plugin. It really does not handle different Antlr versions well (or really at all).

Comment by Eric Wendelin [ 15/Nov/16 ]

@Steve Thank you. Would you be willing to contribute to such a "revamp" if it began?

Comment by Steve Ebersole [ 15/Nov/16 ]

I have already...

  1. https://github.com/hibernate/hibernate-semantic-query/blob/master/build.gradle#L159
  2. https://discuss.gradle.org/t/gradle-3-and-antlr/18514/11
  3. not to mention that I helped with the original Antlr (v2) support in Gradle

My plugin (1st link) has some poor design points in the sake of ease-of-use in Hibernate, which is what I needed this for.

A more generally useful plugin/task for Antlr v4 (and v3 I believe) would perform discovery and actually inspect the grammar files to figure out things like generation-order, package names and corresponding directory structure, etc. Antlr uses itself at runtime to parse these grammar files; you can manually perform all these steps yourself which gives you access to valuable information.

I can help some with that. I might be beneficial to reach out the Antlr team and see if they are willing to take over that plugin. Contribution is how they started supporting the Maven plugin.

Comment by Eric Wendelin [ 15/Nov/16 ]

Of course — I didn't mean that you haven't contributed already

Thank you for the suggestion.

Comment by Steve Ebersole [ 15/Nov/16 ]

I just mean that discussion and that plugin are a good start on specifically Antlr v4 that actually works between Gradle 2 and 3.

I know compat with older versions of Gradle is not the top priority of the Gradle team, but it is an amazing thing when it happens to happen with no extra work

As I said on the forum thread (2nd link) I personally think this needs specific plugins/tasks for each version of Antlr. But honestly that's the same thing I suggested when it was decided to fold Antlr v3 support into the same v2-based plugin. The plugin I wrote for Hibernate would not work at all against versions of Antlr prior to v4.

Comment by Eric Wendelin [ 24/Jan/17 ]

Moved to https://github.com/gradle/gradle/issues/1240 — I expect that we will reach out to the Antlr team for their help soon.

Generated at Wed Jun 30 12:46:22 CDT 2021 using Jira 8.4.2#804003-sha1:d21414fc212e3af190e92c2d2ac41299b89402cf.