Is there any way to pause and resume Files.walkFileTree in Java?

user_4068123 0
Viewed 77

This class walks a path, and for each file that it hits, it throws a thread into a ThreadPoolExecutor which collects the objects being given to it into a HashMap. I have another thread that monitors the HashMap and when that Map has 5,000 elements in it, it gets dumped to a MySQL database and then the records that were recorded are purged from the HashMap and everything continues like that.

However, once the walker gets to where it has hit over 2 million files, the actual purging of the HashMap has lagged to where it's almost a million records behind so I would like to be able to pause the folder walking until the data dumps have caught up, then resume ... rinse repeat ...

Is it possible to pause this class after it starts going? OR, is there any way to slow it down possibly?

public class WalkFilePaths implements Runnable{

    public WalkFilePaths(Path rootPath, ThreadPoolExecutor executor) {
        this.rootPath           = rootPath;
        this.executor           = executor;
    }

    private        final Path               rootPath;
    private static       ThreadPoolExecutor executor;
    private static final FileDataManager    fileDataManager = new FileDataManager();

    @Override public void run() {
        try {
            FolderWalker folderWalker = new FolderWalker();
            Files.walkFileTree(rootPath,folderWalker);
        }
        catch (IOException e) {e.printStackTrace();}
    }

    public static class FolderWalker extends SimpleFileVisitor<Path> {
        @Override
        public FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) {
            return FileVisitResult.CONTINUE;
        }

        @Override
        public FileVisitResult visitFile(Path path, BasicFileAttributes attrs) {
            if(attrs.isRegularFile()) {
                executor.execute(fileDataManager.addFileMap(new FileDataModel(path.toFile(), attrs.creationTime().toInstant(), attrs.lastAccessTime().toInstant())));
            }
            return FileVisitResult.CONTINUE;
        }

        @Override
        public FileVisitResult postVisitDirectory(Path dir, IOException e) {
            return FileVisitResult.CONTINUE;
        }

        @Override
        public FileVisitResult visitFileFailed(Path file, IOException exc) {
            return FileVisitResult.CONTINUE;
        }
    }
}
java-threads threadpool java
1 Answers

You can manage to throttle the process by using a counting Semaphore.

The concept behind this is simple: You start with an initial value of 5_000 permits released on your semaphore; every time your code enqueues a new file, it will need one permit. When the semaphore runs out of permits, your code will wait (blocking call to semaphore.acquire()) until permits are released again. Your existing code will have to release() permits every time it consumes the data accumulated.

A simplistic implementation of the above concept would be:

class FileDataManager {

    private static final int BATCH_SIZE = 5_000;
    private final Semaphore semaphore = new Semaphore(BATCH_SIZE);
    private Map<String, String> data = new HashMap<>(BATCH_SIZE);

    Runnable addFileMap(FileDataModel fileDataModel) {
        try {
            //Try to acquire a permit, or wait (blocking call) until a permit is available
            semaphore.acquire();
            return new Runnable() {
                @Override
                public void run() {
                    //Process file...
                    data.put(fileDataModel.toString(), fileDataModel.toString());
                }
            };
        } catch (InterruptedException ex) {
            Logger.getLogger(FileDataManager.class.getName()).log(Level.SEVERE, null, ex);
            throw new RuntimeException(ex);
        }

    }

    public int accumulatedFileCount() {
        return data.size();
    }

    public void releasePermits() {
        semaphore.release(BATCH_SIZE);
    }

}

The other thread monitoring the map would then:

//...
            if (fileDataManager.accumulatedFileCount() >= 5_000) {
                // store data in RDBMS
                fileDataManager.releasePermits();
            }
//...
user_15051460
Related