Copying of files with unicode in filename fails with "java.lang.UnsupportedOperationException: ENOENT"

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Resolution: Fixed
  • Affects Version/s: 1.0-milestone-9
  • Fix Version/s: 1.1-rc-1

Description

After upgrading from milestone-8 to milestone-9 and running "gradle copyFiles -S" on the example attached to this bug, the copy operations fails with the following output (+ stack trace):

FAILURE: Build failed with an exception.

  • Where:
    Build file '/Users/Rene/gradleware/github/breskeby/diagnosticbuilds/bugs/unicodefiles/copy_bug/build.gradle' line: 2
  • What went wrong:
    Execution failed for task ':copyFiles'.
    > Could not copy file '/Users/Rene/gradleware/github/breskeby/diagnosticbuilds/bugs/unicodefiles/copy_bug/res/??????? ???????? - ?????.lnk' to '/Users/Rene/gradleware/github/breskeby/diagnosticbuilds/bugs/unicodefiles/copy_bug/build/resources/????? ???????? - ???????.lnk'.
  • Try:
    Run with --info or --debug option to get more log output.
  • Exception is:
    org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':copyFiles'.
    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:68)
    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:46)
    at org.gradle.api.internal.tasks.execution.PostExecutionAnalysisTaskExecuter.execute(PostExecutionAnalysisTaskExecuter.java:34)
    at org.gradle.api.internal.changedetection.CacheLockHandlingTaskExecuter$1.run(CacheLockHandlingTaskExecuter.java:34)
    at org.gradle.cache.internal.DefaultCacheAccess$2.create(DefaultCacheAccess.java:200)
    at org.gradle.cache.internal.DefaultCacheAccess.longRunningOperation(DefaultCacheAccess.java:172)
    at org.gradle.cache.internal.DefaultCacheAccess.longRunningOperation(DefaultCacheAccess.java:198)
    at org.gradle.cache.internal.DefaultPersistentDirectoryStore.longRunningOperation(DefaultPersistentDirectoryStore.java:111)
    at org.gradle.api.internal.changedetection.DefaultTaskArtifactStateCacheAccess.longRunningOperation(DefaultTaskArtifactStateCacheAccess.java:83)
    at org.gradle.api.internal.changedetection.CacheLockHandlingTaskExecuter.execute(CacheLockHandlingTaskExecuter.java:32)
    at org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:55)
    at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:57)
    at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:41)
    at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:51)
    at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:52)
    at org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:42)
    at org.gradle.api.internal.AbstractTask.executeWithoutThrowingTaskFailure(AbstractTask.java:247)
    at org.gradle.execution.DefaultTaskGraphExecuter.executeTask(DefaultTaskGraphExecuter.java:192)
    at org.gradle.execution.DefaultTaskGraphExecuter.doExecute(DefaultTaskGraphExecuter.java:177)
    at org.gradle.execution.DefaultTaskGraphExecuter.execute(DefaultTaskGraphExecuter.java:83)
    at org.gradle.execution.SelectedTaskExecutionAction.execute(SelectedTaskExecutionAction.java:36)
    at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:61)
    at org.gradle.execution.DefaultBuildExecuter.access$200(DefaultBuildExecuter.java:23)
    at org.gradle.execution.DefaultBuildExecuter$2.proceed(DefaultBuildExecuter.java:67)
    at org.gradle.api.internal.changedetection.TaskCacheLockHandlingBuildExecuter$1.run(TaskCacheLockHandlingBuildExecuter.java:31)
    at org.gradle.cache.internal.DefaultCacheAccess$1.create(DefaultCacheAccess.java:111)
    at org.gradle.cache.internal.DefaultCacheAccess.useCache(DefaultCacheAccess.java:126)
    at org.gradle.cache.internal.DefaultCacheAccess.useCache(DefaultCacheAccess.java:109)
    at org.gradle.cache.internal.DefaultPersistentDirectoryStore.useCache(DefaultPersistentDirectoryStore.java:103)
    at org.gradle.api.internal.changedetection.DefaultTaskArtifactStateCacheAccess.useCache(DefaultTaskArtifactStateCacheAccess.java:79)
    at org.gradle.api.internal.changedetection.TaskCacheLockHandlingBuildExecuter.execute(TaskCacheLockHandlingBuildExecuter.java:29)
    at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:61)
    at org.gradle.execution.DefaultBuildExecuter.access$200(DefaultBuildExecuter.java:23)
    at org.gradle.execution.DefaultBuildExecuter$2.proceed(DefaultBuildExecuter.java:67)
    at org.gradle.execution.DryRunBuildExecutionAction.execute(DryRunBuildExecutionAction.java:32)
    at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:61)
    at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:54)
    at org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:155)
    at org.gradle.initialization.DefaultGradleLauncher.doBuild(DefaultGradleLauncher.java:110)
    at org.gradle.initialization.DefaultGradleLauncher.run(DefaultGradleLauncher.java:78)
    at org.gradle.launcher.cli.ExecuteBuildAction.run(ExecuteBuildAction.java:45)
    at org.gradle.launcher.daemon.protocol.Build.run(Build.java:67)
    at org.gradle.launcher.daemon.protocol.Build.run(Build.java:63)
    at org.gradle.launcher.daemon.server.exec.ExecuteBuild.doBuild(ExecuteBuild.java:45)
    at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:34)
    at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)
    at org.gradle.launcher.daemon.server.exec.WatchForDisconnection.execute(WatchForDisconnection.java:45)
    at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)
    at org.gradle.launcher.daemon.server.exec.ResetDeprecationLogger.execute(ResetDeprecationLogger.java:24)
    at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)
    at org.gradle.launcher.daemon.server.exec.ReturnResult.execute(ReturnResult.java:34)
    at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)
    at org.gradle.launcher.daemon.server.exec.ForwardClientInput$4.call(ForwardClientInput.java:116)
    at org.gradle.launcher.daemon.server.exec.ForwardClientInput$4.call(ForwardClientInput.java:114)
    at org.gradle.util.Swapper.swap(Swapper.java:38)
    at org.gradle.launcher.daemon.server.exec.ForwardClientInput.execute(ForwardClientInput.java:114)
    at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)
    at org.gradle.launcher.daemon.server.exec.LogToClient.doBuild(LogToClient.java:60)
    at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:34)
    at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)
    at org.gradle.launcher.daemon.server.exec.EstablishBuildEnvironment.doBuild(EstablishBuildEnvironment.java:61)
    at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:34)
    at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)
    at org.gradle.launcher.daemon.server.exec.StartBuildOrRespondWithBusy.doBuild(StartBuildOrRespondWithBusy.java:49)
    at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:34)
    at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)
    at org.gradle.launcher.daemon.server.exec.HandleStop.execute(HandleStop.java:36)
    at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)
    at org.gradle.launcher.daemon.server.exec.CatchAndForwardDaemonFailure.execute(CatchAndForwardDaemonFailure.java:32)
    at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)
    at org.gradle.launcher.daemon.server.exec.HandleClientDisconnectBeforeSendingCommand.execute(HandleClientDisconnectBeforeSendingCommand.java:21)
    at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)
    at org.gradle.launcher.daemon.server.exec.StopConnectionAfterExecution.execute(StopConnectionAfterExecution.java:27)
    at org.gradle.launcher.daemon.server.exec.DaemonCommandExecution.proceed(DaemonCommandExecution.java:122)
    at org.gradle.launcher.daemon.server.exec.DefaultDaemonCommandExecuter.executeCommand(DefaultDaemonCommandExecuter.java:55)
    at org.gradle.launcher.daemon.server.Daemon$1$1.run(Daemon.java:123)
    at org.gradle.messaging.concurrent.DefaultExecutorFactory$StoppableExecutorImpl$1.run(DefaultExecutorFactory.java:66)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:680)
    Caused by: org.gradle.api.GradleException: Could not copy file '/Users/Rene/gradleware/github/breskeby/diagnosticbuilds/bugs/unicodefiles/copy_bug/res/??????? ???????? - ?????.lnk' to '/Users/Rene/gradleware/github/breskeby/diagnosticbuilds/bugs/unicodefiles/copy_bug/build/resources/????? ???????? - ???????.lnk'.
    at org.gradle.api.internal.file.AbstractFileTreeElement.copyTo(AbstractFileTreeElement.java:68)
    at org.gradle.api.internal.file.copy.MappingCopySpecVisitor$FileVisitDetailsImpl.copyTo(MappingCopySpecVisitor.java:120)
    at org.gradle.api.internal.file.copy.FileCopySpecVisitor.copyFile(FileCopySpecVisitor.java:56)
    at org.gradle.api.internal.file.copy.FileCopySpecVisitor.visitFileOrDir(FileCopySpecVisitor.java:52)
    at org.gradle.api.internal.file.copy.FileCopySpecVisitor.visitFile(FileCopySpecVisitor.java:39)
    at org.gradle.api.internal.file.copy.NormalizingCopySpecVisitor.visitFile(NormalizingCopySpecVisitor.java:70)
    at org.gradle.api.internal.file.copy.MappingCopySpecVisitor.visitFile(MappingCopySpecVisitor.java:50)
    at org.gradle.api.internal.file.collections.DirectoryFileTree.walkDir(DirectoryFileTree.java:151)
    at org.gradle.api.internal.file.collections.DirectoryFileTree.visit(DirectoryFileTree.java:119)
    at org.gradle.api.internal.file.collections.FileTreeAdapter.visit(FileTreeAdapter.java:96)
    at org.gradle.api.internal.file.CompositeFileTree.visit(CompositeFileTree.java:54)
    at org.gradle.api.internal.file.copy.CopyActionImpl.execute(CopyActionImpl.java:63)
    at org.gradle.api.internal.file.DefaultFileOperations.copy(DefaultFileOperations.java:133)
    at org.gradle.api.internal.project.AbstractProject.copy(AbstractProject.java:806)
    at org.gradle.api.internal.file.FileOperations$copy.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
    at org.gradle.api.internal.file.FileOperations$copy.call(Unknown Source)
    at org.gradle.groovy.scripts.DefaultScript.copy(DefaultScript.groovy:141)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:361)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:877)
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)
    at build_4m156efrv2k93ngmaldscnceob$_run_closure1.doCall(/Users/Rene/gradleware/github/breskeby/diagnosticbuilds/bugs/unicodefiles/copy_bug/build.gradle:2)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:877)
    at groovy.lang.Closure.call(Closure.java:412)
    at groovy.lang.Closure.call(Closure.java:425)
    at org.gradle.api.internal.AbstractTask$ClosureTaskAction.execute(AbstractTask.java:452)
    at org.gradle.api.internal.AbstractTask$ClosureTaskAction.execute(AbstractTask.java:436)
    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:60)
    ... 79 more
    Caused by: java.lang.UnsupportedOperationException: ENOENT - /Users/Rene/gradleware/github/breskeby/diagnosticbuilds/bugs/unicodefiles/copy_bug/res/??????? ???????? - ???????.lnk
    at org.gradle.internal.nativeplatform.PosixUtil$POSIXHandlerImpl.error(PosixUtil.java:36)
    at org.jruby.ext.posix.BaseNativePOSIX.stat(BaseNativePOSIX.java:184)
    at org.gradle.internal.nativeplatform.GenericFileSystem.getUnixMode(GenericFileSystem.java:57)
    at org.gradle.api.internal.file.DefaultFileTreeElement.getMode(DefaultFileTreeElement.java:66)
    at org.gradle.api.internal.file.AbstractFileTreeElement.copyTo(AbstractFileTreeElement.java:65)
    ... 122 more

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Luke Daley added a comment -

The problem comes in that we are using JNA. When we cross that boundary the Java (encoding agnostic) String is converted to bytes using the platform encoding.

http://jna.java.net/javadoc/overview-summary.html#strings

We cross the boundary calling a libc function. It's not clear to me what encoding libc is expecting, and I guess it may be platform dependent.

I can't see a clear way to resolve this issue. One option would be to have Gradle set 'jna.encoding' to UTF8 globally, but this might be too far reaching.

It should be noted that any user can workaround this problem by ensuring that the JVM file encoding for the build supports the characters in the filename.

Show
Luke Daley added a comment - The problem comes in that we are using JNA. When we cross that boundary the Java (encoding agnostic) String is converted to bytes using the platform encoding. http://jna.java.net/javadoc/overview-summary.html#strings We cross the boundary calling a libc function. It's not clear to me what encoding libc is expecting, and I guess it may be platform dependent. I can't see a clear way to resolve this issue. One option would be to have Gradle set 'jna.encoding' to UTF8 globally, but this might be too far reaching. It should be noted that any user can workaround this problem by ensuring that the JVM file encoding for the build supports the characters in the filename.
Hide
Permalink
Adam Murdoch added a comment -

Generally, I think, libc functions such as stat(), chmod(), etc don't specify how the file names are encoded - it's the caller's problem to use the same encoding as everyone else that wants to do something with the file name.

Often, I think, file names are encoded using wcstombs(). It appears this is what the JVM is using (or openjdk 7 at least). This uses the platform encoding, specified by $LANG (or $LC_CTYPE). This isn't the same as the JVM's file.encoding.

On the mac, filenames need to be encoded with utf-8. The apple java 6 has file.encoding set to MacRoman, whereas openjdk 7 has file.encoding set to utf-8. $LANG is set to use utf-8, so it looks like the apple jvm is not doing the right thing here.

On linux, I'm not sure what the jvms are doing with file.encoding. I don't have my linux machine with me to try it out. I'm hoping that they honour $LANG. If they do, I think we can just special-case for the mac, where we use utf-8 for filename encodings on the mac, and file.encoding everywhere else.

On windows, we don't need to care. We don't use any of the libc functions, and use the windows unicode-aware functions instead.

Show
Adam Murdoch added a comment - Generally, I think, libc functions such as stat(), chmod(), etc don't specify how the file names are encoded - it's the caller's problem to use the same encoding as everyone else that wants to do something with the file name. Often, I think, file names are encoded using wcstombs(). It appears this is what the JVM is using (or openjdk 7 at least). This uses the platform encoding, specified by $LANG (or $LC_CTYPE). This isn't the same as the JVM's file.encoding. On the mac, filenames need to be encoded with utf-8. The apple java 6 has file.encoding set to MacRoman, whereas openjdk 7 has file.encoding set to utf-8. $LANG is set to use utf-8, so it looks like the apple jvm is not doing the right thing here. On linux, I'm not sure what the jvms are doing with file.encoding. I don't have my linux machine with me to try it out. I'm hoping that they honour $LANG. If they do, I think we can just special-case for the mac, where we use utf-8 for filename encodings on the mac, and file.encoding everywhere else. On windows, we don't need to care. We don't use any of the libc functions, and use the windows unicode-aware functions instead.
Hide
Permalink
Adam Murdoch added a comment -

Another workaround is to use java 7.

Show
Adam Murdoch added a comment - Another workaround is to use java 7.
Hide
Permalink
Luke Daley added a comment -

I'm not sure we can use libc and solve this effectively, since there seems to be no way to give the libc functions anything other than String and let it handle the encoding. Having Gradle globally set jna.encoding seems dangerously invasive to me.

I think the best we can do is provide a good error message that suggests people either use Java 7, set jna.encoding or file.encoding to something that works.

Would utf8 always work?

Show
Luke Daley added a comment - I'm not sure we can use libc and solve this effectively, since there seems to be no way to give the libc functions anything other than String and let it handle the encoding. Having Gradle globally set jna.encoding seems dangerously invasive to me. I think the best we can do is provide a good error message that suggests people either use Java 7, set jna.encoding or file.encoding to something that works. Would utf8 always work?
Hide
Permalink
Luke Daley added a comment -

Rene, it's likely that this works on Ubuntu because the default platform encoding is a character set that supports the characters.

Can you test by running with a something for -Dfile.encoding that does not have these characters please.

Show
Luke Daley added a comment - Rene, it's likely that this works on Ubuntu because the default platform encoding is a character set that supports the characters. Can you test by running with a something for -Dfile.encoding that does not have these characters please.
Hide
Permalink
René Gröschke added a comment -

I've tested the behaviour on ubuntu:

1. gradle copyFiles -> works
2. gradle copyFiles -Dfile.encoding=us-ascii -> fails
3. export LANG=us-ascii + gradle copyFiles" -> fails

Show
René Gröschke added a comment - I've tested the behaviour on ubuntu: 1. gradle copyFiles -> works 2. gradle copyFiles -Dfile.encoding=us-ascii -> fails 3. export LANG=us-ascii + gradle copyFiles" -> fails

People

  • Assignee:
    Unassigned
    Reporter:
    René Gröschke
Vote (1)
Watch (2)

Dates

  • Created:
    Updated:
    Resolved: