(EDIT: The question - see also "How you can help" - would be if others can reproduce the bug, if it is known (Link.), what workarounds are recommended.)
Goal: Create a tiny temp file, then delete file with final filename, then rename temp to final. (Standard safesave-approach. You do use it, right.)
Problem: Fails after many test loop iterations. Files.move() throws FileAlreadyExistsException, tests instead using file.renameTo() return false sometimes. The file is indeed not renamed then. My tests make me feel that the failure frequency is independent of approach. I explicitly created a test class - see below. The problem is definitely not my code.
In very rare cases, Files.move() throws a FileSystemException saying "The process cannot access the file because it is being used by another process.", as if the file had not yet been closed after writing, but Files.write is used, so that "can't" be the case. While working with 3rd party libraries, I have sometimes had problems akin to this and used System.gc() to reliably work around them, that's why I added this feature to the test class, too, but this Exception occurred only once during all tests. That was without System.gc, so I don't know if gc would help.
Why it is a Java bug: FileAlreadyExistsException does not apply, because destination.exists() returns false right before the call and also right after the exception is thrown. Even if the cause is outside of Java, it's a Java bug to claim that the file does already exist when Java also claims that it does not exist.
When it does not occur:
if the rename target is different in every iteration.
if the file is just endlessly renamed back and forth.
if the problem occurs, waiting 1 ms and trying again. But in one rare case the counter of this retry-loop reached 24.
To reproduce: [Create a file X, rename it to Y, delete it.] in a loop, never changing Y.
Conjecture: Java's renaming gets confused by caching (at least on this platform), and Java's file.exists() probably uses a different approach.
How you can help: Please try to reproduce the bug and report in, on the below described and other platforms. Also, please inform the Java dev community. I already found https://bugs.openjdk.java.net/browse/JDK-8150700 but it's rather bare-bones. I am also not informed about the process, don't have an account, and have invested a ton of time already myself, have to get back to coding. Please help so we get this bug squashed.
Reproduced so far:
- In the comments, Luis G. says he confirmed the problem on Windows 10 Home (build 1803) using an executable of the minimal test class using javac version "1.8.0_121", java version "1.8.0_171".
Test environment: Standard Oracle Java release download JDK8 144. Java -version:
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)
Windows 10 Pro, Version 1803 (OS Build 17134.228)
Intel(R) Core(TM) i3-2120 CPU @ 3.30 GHz
Test class result with no problems (because target name different each time):
Doing test withOUT forced Garbage Collection.
Completed create-rename-delete attempt 0x0
Completed create-rename-delete attempt 0x100
...
Completed create-rename-delete attempt 0xfe00
Completed create-rename-delete attempt 0xff00
No fatal problems.
Test class result with the described problems, reliably with 0x10000 tests:
Doing test withOUT forced Garbage Collection.
Completed create-rename-delete attempt 0x0
Completed create-rename-delete attempt 0x100
...
Completed create-rename-delete attempt 0x3900
Files.move() attempt 1 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Completed create-rename-delete attempt 0x3a00
...
Completed create-rename-delete attempt 0x4700
Files.move() attempt 1 of 4096 failed: java.nio.file.FileSystemException: renameBugDemoFile.dat -> renameBugDemoFile.ren: The process cannot access the file because it is being used by another process.
Completed create-rename-delete attempt 0x4800
...
Completed create-rename-delete attempt 0x7700
Files.move() attempt 1 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Files.move() attempt 2 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Completed create-rename-delete attempt 0x7800
...
Completed create-rename-delete attempt 0x9900
Files.move() attempt 1 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Files.move() attempt 2 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Completed create-rename-delete attempt 0x9a00
Files.move() attempt 1 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Files.move() attempt 2 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Completed create-rename-delete attempt 0x9b00
Files.move() attempt 1 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Files.move() attempt 2 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Completed create-rename-delete attempt 0x9c00
...
Completed create-rename-delete attempt 0xa300
Files.move() attempt 1 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Files.move() attempt 2 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Files.move() attempt 3 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Files.move() attempt 4 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Files.move() attempt 5 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Files.move() attempt 6 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Files.move() attempt 7 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Files.move() attempt 8 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Files.move() attempt 9 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Files.move() attempt 10 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Files.move() attempt 11 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Files.move() attempt 12 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Files.move() attempt 13 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Files.move() attempt 14 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Files.move() attempt 15 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Files.move() attempt 16 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Files.move() attempt 17 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Files.move() attempt 18 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Files.move() attempt 19 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Files.move() attempt 20 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Files.move() attempt 21 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Completed create-rename-delete attempt 0xa400
Files.move() attempt 1 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Completed create-rename-delete attempt 0xa500
Completed create-rename-delete attempt 0xa600
Completed create-rename-delete attempt 0xa700
Completed create-rename-delete attempt 0xa800
Files.move() attempt 1 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Completed create-rename-delete attempt 0xa900
Completed create-rename-delete attempt 0xaa00
Completed create-rename-delete attempt 0xab00
Completed create-rename-delete attempt 0xac00
Files.move() attempt 1 of 4096 failed: java.nio.file.FileAlreadyExistsException: renameBugDemoFile.dat -> renameBugDemoFile.ren
Completed create-rename-delete attempt 0xad00
...
Completed create-rename-delete attempt 0xfe00
Completed create-rename-delete attempt 0xff00
No fatal problems.
MINIMAL TEST CLASS
The likelihood that you test this on your computer is higher if you don't have to check lots of code, so here's a minimal version:
import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
final public class FileRenameProblemDemoMinimal {
public static void main(final String[] args) {
final File demofile = new File("renameBugDemoFile.dat");
final File demofileAfterRenaming = new File("renameBugDemoFile.ren");
demofile.delete();
demofileAfterRenaming.delete();
for (int i = 0; i < 0x10000; i++) {
createDemoFile(demofile);
// System.gc();
if (!renameUsingFilesMoveAndBruteForce(demofile, demofileAfterRenaming)) {
System.err.println("q.e.d.: FAILURE! (Attempt was 0x" + Integer.toHexString(i) + ")");
System.exit(-1);
} else {
if (!demofileAfterRenaming.delete()) {
throw new Error();
} else {
if (demofileAfterRenaming.exists()) {
throw new Error();
}
}
if (i % 0x100 == 0) {
System.err.println("Completed create-rename-delete attempt 0x" + Integer.toHexString(i));
}
}
}
System.err.println("No fatal problems.");
System.exit(0);
}
private static boolean renameUsingFilesMoveAndBruteForce(final File source, final File destination) {
if (destination.exists()) {
throw new Error("destination already exists: " + destination);
}
final int amountOfRetriesWith1MsPause = 0x1000;
int i = 0;
while (i <= amountOfRetriesWith1MsPause) {
i++;
try {
Files.move(source.toPath(), destination.toPath()); // , StandardCopyOption.REPLACE_EXISTING); // WOULD NOT MAKE A DIFFERENCE!
return true;
} catch (IOException e) {
if (destination.exists()) {
throw new Error();
}
System.err.println("Files.move() attempt " + i + " of " + amountOfRetriesWith1MsPause + " failed: " + e.toString());
}
try {
Thread.sleep(1);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
return false;
}
private static void createDemoFile(final File file) {
final byte[] bytes = new byte[] { 1, 2, 3, 4, 5, 6, 7, 8 };
try {
Files.write(file.toPath(), bytes);
} catch (Exception e) {
e.printStackTrace();
}
}
}
FULL TEST CLASS
The test class has code for all cases. You have to choose via (un)commenting. As is, the code will demonstrate the problem happening but without failing, because it just retries the renaming. System.gc use (which slows the test down) is currently off.
import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
final public class FileRenameProblemDemo {
final private static File DEMOFILE = new File("renameBugDemoFile.dat");
final private static File DEMOFILE_AFTER_RENAMING = new File("renameBugDemoFile.ren");
final private static int AMOUNT_OF_MAIN_ITERATIONS = 0x10000;
final private static int AMOUNT_OF_RETRIES_WITH_1MS_PAUSE = 0x1000;
public static void main(final String[] args) {
deleteFileOptionallyButWithExtremeParanoia(DEMOFILE);
deleteFileOptionallyButWithExtremeParanoia(DEMOFILE_AFTER_RENAMING);
// runTestOnlyWithRenames(); // <-- No problems at all.
runTestWithCreateAndRenames(false); // <-- Reliable problems. Sometimes it only takes a few iterations, often a few thousand.
System.err.println("No fatal problems.");
System.exit(0);
}
private static File createDifferentFileName(final int i) {
return DEMOFILE_AFTER_RENAMING;
// return new File(String.valueOf(i) + "_" + DEMOFILE_AFTER_RENAMING.getName());
}
private static void runTestOnlyWithRenames() {
createDemoFile(DEMOFILE);
for (int i = 0; i < AMOUNT_OF_MAIN_ITERATIONS; i++) {
// if (!renameUsingFilesMove(DEMOFILE, DEMOFILE_AFTER_RENAMING)) {
if (!renameUsingRename(DEMOFILE, DEMOFILE_AFTER_RENAMING)) {
// if (!renameUsingFilesMoveAndBruteForce(DEMOFILE, DEMOFILE_AFTER_RENAMING)) {
// if (!renameUsingRenameAndBruteForce(DEMOFILE, DEMOFILE_AFTER_RENAMING)) {
System.err.println("q.e.d.: FAILURE! (Attempt was 0x" + Integer.toHexString(i) + ")");
System.exit(-1);
} else {
// if (!renameUsingFilesMove(DEMOFILE_AFTER_RENAMING, DEMOFILE)) {
// if (!renameUsingRename(DEMOFILE_AFTER_RENAMING, DEMOFILE)) {
if (!renameUsingFilesMoveAndBruteForce(DEMOFILE_AFTER_RENAMING, DEMOFILE)) {
// if (!renameUsingRenameAndBruteForce(DEMOFILE_AFTER_RENAMING, DEMOFILE)) {
throw new Error(); // We're confident that this approach never fails and want to keep the code short.
}
if (i % 0x100 == 0) {
// System.err.println("Completed back-and-forth renaming attempt " + Integer.toHexString(i));
System.err.println("Completed rename-back&forth attempt 0x" + Integer.toHexString(i));
}
}
}
}
private static void runTestWithCreateAndRenames(final boolean forceGarbageCollection) {
if (forceGarbageCollection) {
System.err.println("Doing test WITH forced Garbage Collection.\n");
} else {
System.err.println("Doing test withOUT forced Garbage Collection.\n");
}
for (int i = 0; i < AMOUNT_OF_MAIN_ITERATIONS; i++) {
createDemoFile(DEMOFILE);
if (forceGarbageCollection) {
System.gc(); // Because in a different project, I had experienced consistent move-problems (Consistent = different than here.) which were consistently solved by running GC after closing after read access.
}
final File demoFileAfterRenaming = createDifferentFileName(i);
// if (!renameUsingFilesMove(DEMOFILE, demoFileAfterRenaming)) {
// if (!renameUsingRename(DEMOFILE, demoFileAfterRenaming)) {
if (!renameUsingFilesMoveAndBruteForce(DEMOFILE, demoFileAfterRenaming)) {
// if (!renameUsingRenameAndBruteForce(DEMOFILE, demoFileAfterRenaming)) {
System.err.println("q.e.d.: FAILURE! (Attempt was 0x" + Integer.toHexString(i) + ")");
System.exit(-1);
} else {
if (!demoFileAfterRenaming.delete()) {
throw new Error();
} else {
// Deletion supposedly successful.
if (demoFileAfterRenaming.exists()) {
throw new Error();
}
}
if (i % 0x100 == 0) {
System.err.println("Completed create-rename-delete attempt 0x" + Integer.toHexString(i));
}
}
}
}
private static boolean renameUsingFilesMove(final File source, final File destination) {
if (destination.exists()) {
throw new Error();
}
try {
Files.move(source.toPath(), destination.toPath());
// Files.move(source.toPath(), destination.toPath(), StandardCopyOption.REPLACE_EXISTING); // DOESN'T MAKE A DIFFERENCE!
} catch (IOException e) {
if (destination.exists()) {
throw new Error();
}
System.err.println("Files.move() attempt one of one failed: " + toStringIntlForThrowables(e));
return false;
}
return true;
}
private static boolean renameUsingRename(final File source, final File destination) {
if (destination.exists()) {
throw new Error();
}
if (source.renameTo(destination)) {
if (source.exists()) {
throw new Error();
}
if (!destination.exists()) {
throw new Error();
}
return true;
} else {
if (!source.exists()) {
throw new Error();
}
if (destination.exists()) {
throw new Error();
}
System.err.println("renameTo() attempt one of one failed.");
return false;
}
}
private static boolean renameUsingFilesMoveAndBruteForce(final File source, final File destination) {
if (destination.exists()) {
throw new Error("destination already exists: " + destination);
}
int i = 0;
while (i <= AMOUNT_OF_RETRIES_WITH_1MS_PAUSE) {
i++;
try {
Files.move(source.toPath(), destination.toPath());
return true;
// Files.move(source.toPath(), destination.toPath(), StandardCopyOption.REPLACE_EXISTING); // DOESN'T MAKE A DIFFERENCE!
} catch (IOException e) {
if (destination.exists()) {
throw new Error();
}
System.err.println("Files.move() attempt " + i + " of " + AMOUNT_OF_RETRIES_WITH_1MS_PAUSE + " failed: " + toStringIntlForThrowables(e));
}
try {
Thread.sleep(1);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
return false;
}
private static boolean renameUsingRenameAndBruteForce(final File source, final File destination) {
if (destination.exists()) {
throw new Error();
}
int i = 0;
while (i <= AMOUNT_OF_RETRIES_WITH_1MS_PAUSE) {
i++;
if (source.renameTo(destination)) {
if (source.exists()) {
throw new Error();
}
if (!destination.exists()) {
throw new Error();
}
return true;
} else {
if (!source.exists()) {
throw new Error();
}
if (destination.exists()) {
throw new Error();
}
System.err.println("renameTo() attempt " + i + " of " + AMOUNT_OF_RETRIES_WITH_1MS_PAUSE + " failed.");
}
try {
Thread.sleep(1);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
return false;
}
private static void deleteFileOptionallyButWithExtremeParanoia(final File file) {
if (file.exists()) {
if (file.isFile()) {
if (!file.delete()) {
throw new Error();
}
if (file.exists()) {
throw new Error();
}
} else {
throw new Error();
}
}
}
private static void createDemoFile(final File file) {
final byte[] bytes = new byte[] { 1, 2, 3, 4, 5, 6, 7, 8 };
try {
Files.write(file.toPath(), bytes);
} catch (Exception e) {
e.printStackTrace();
}
}
/**
* @param t the exception (Null would cause the return of null.)
* @return the same as exception.toString except with getMessage() instead of getLocalizedMessage(), because a log
* file with e.g. Danish error text (CUSTOMER SYSTEM SETTINGS.) doesn't help the developer.
*/
public static String toStringIntlForThrowables(final Throwable t) {
if (t == null) {
return null;
}
final String className = t.getClass().getName();
final String message = t.getMessage();
return (message != null) ? (className + ": " + message) : className;
}
}