I have a lot of XML files (60000+) that I'm putting in my src/main/resources/xmlFiles directory but when I run any of my tests or main class files, the build takes longer than usual to run. On first run after a gradle clean, the processResources task take 1 minute and 10 seconds+ to copy 32 MB worth of 60k+ XML files. Subsequent runs only take 5 seconds to start the program or tests, but it still slow. I assume it is because it is checking the status for each xml file in my resources folder to see if it needs to copy the file into the build folder? These XML files are never changing.
I looked online and I saw some suggestions to zip the thousands of resource files into a jar file, and read the XML files from inside the jar file, but no matter what I try, I cannot get it to work either. I put the xml file in the root folder of the jar file, so getClass().getResourceAsStream("myFile.xml") should apparently work, but it doesn't, I get a null when running this. If I try getClass().getResourceAsStream("xmlFiles.jar") I'm able to load the jar file itself, but I thought you don't need to do this? I thought it unzips/unpacks the jar file when you call getResourceAsStream() and you can get the file directly inside without needing to read the jar file? I also tried getClass().getResourceAsStream("xmlFiles.jar/myFile.xml") but this doesn't work either. Also, in IntelliJ, I tried right clicking on the jar file containing my xml files and adding it as a library, which it seemingly unpacks the jar file and I can see all my files now, but I still cannot access them directly using getResourceAsStream(). The JAR file doesn't contain any class files, just XML files, not sure if it's necessary to have class files in a jar file for it to work?
I'm just wondering what the best approach is for handling a lot of files (or large resource files) in java/gradle/maven? I don't think I'm the first one to encounter this issue but I cannot seem to find any relevant results in Google.