Temp files are not deleted after Writing files to Google Cloud Storage using Java

user_8863925 0
Viewed 321

I am using Hadoop GCS Connector 3-2.2.2 and google-cloud-storage version 1.113.14 to write data into files on Google Cloud Storage.

I have a file class which has a write method which returns an OutputStream as follow using hadoop classes:

protected FileSystem fs;
protected Path path;

public File(String path) {
    this.path = new Path(path);
    Configuration conf = new Configuration();
    conf.set("fs.defaultFS", fsURL);
    conf.setBoolean("dfs.support.append", true);
    conf.setInt("ipc.client.connect.max.retries.on.timeouts", 2);
    this.fs = FileSystem.newInstance(conf);
}

@Override
public OutputStream write(boolean overwriteIfExists) throws IOException {
    return fs.create(path, overwriteIfExists);
}

fs is instance of org.apache.hadoop.fs.FileSystem; path is instance of import org.apache.hadoop.fs.Path;

I tried to even use google cloud storage sdk to implement my write method:

@Override
public OutputStream write(boolean overwriteIfExists, boolean create) throws IOException {
    if (!overwriteIfExists && exists()) {
        throw new IOException("File already exists");
    }

    if (overwriteIfExists && exists()) {
        delete();
    } // If will write it will mean a new file

    BlobId blobId = BlobId.of(bucketName, objectName);
    BlobInfo blobInfo = BlobInfo.newBuilder(blobId).build();
    
    WriteChannel writeChannel = storage.writer(blobInfo);
    writeChannel.setChunkSize(64 * 1024 * 1024);

    return Channels.newOutputStream(writeChannel);
}

Other parts of my application can create an instance of my File class and write to the OutputStream, below is an example of how test/simulate the writing:

public void write() {
try {
  File file = File.newInstance("dir/someFileName");
  try (DataOutputStream os = new DataOutputStream(new BufferedOutputStream(file.write(true), this.bufferSize))) {
    long i;
    for (i = 0L; i < 4096; i++) {
      os.write(String.format("some arbitrary string %d\n", new Object[] { Long.valueOf(i) }).getBytes());
    } 
  } catch (Throwable t) {
    System.out.println("Couldn't write file");
  } 
} catch (IOException e) {
  System.out.println("Couldn't write file");
}

}

After writing when I check the written files using gsutil ls dir/ I expect to find just 1 file the one I just created but I get 2 extra temp files (having the GCS_SYNCABLE_TEMPFILE prefix) that should've been deleted

gs://dir/_GCS_SYNCABLE_TEMPFILE_someFileName.txt.0.24fccb78-8cea-45fd-852d-ed9385e4246b gs://dir/_GCS_SYNCABLE_TEMPFILE_someFileName.txt.0.cf132710-9d71-4939-a5d8-02b0279cdf3c gs://dir/someFileName.txt

Is there some configuration I need to do or why are these temp files not deleted?

Or might this be an incompatibility in any related jar? I am using the following versions of google related dependencies:

  • group: 'org.apache.hadoop', name: 'hadoop-common', version: '3.2.0'
  • group: 'org.apache.hadoop', name: 'hadoop-hdfs', version: '3.2.0'
  • group: 'org.apache.hadoop', name: 'hadoop-hdfs-client', version: '3.2.0'
  • group: ‘com.google.cloud.bigdataoss’, name: ‘gcs-connector’, version: ‘hadoop3-2.2.2’
  • group: ‘com.google.api-client’, name: ‘google-api-client’, version: ‘1.31.3’
  • group: ‘com.google.cloud’, name: ‘google-cloud-core’, version: ‘1.93.4’
  • group: ‘com.google.cloud’, name: ‘google-cloud-core-http’, version: ‘1.93.4’
  • group: ‘com.google.cloud’, name: ‘google-cloud-storage’, version: ‘1.106.0’
  • group: ‘com.google.http-client’, name: ‘google-http-client’, version: ‘1.38.0’
  • group: ‘com.google.http-client’, name: ‘google-http-client-jackson2’, version: ‘1.38.0’
google-cloud-dataproc google-cloud-platform hadoop java
1 Answers
  • The OutputStream is publicly available in the source code provided in this GitHub link
  • The behaviour of the TEMP files is property of the client library not directly GCS.
  • The documentation link attached provides details, but the general overview is :

Prior to the first hsync(), sync() or close() call, this channel will behave the same way as a basic non-syncable channel, writing directly to the destination file. On the first call to hsync()/sync(), the destination file is committed and a new temporary file using a hidden-file prefix (underscore) is created with an additional suffix which differs for each subsequent temporary file in the series; during this time readers can read the data committed to the destination file, but not the bytes written to the temporary file since the last hsync() call. On each subsequent hsync()/sync() call, the temporary file closed(), composed into the destination file, then deleted, and a new temporary file is opened under a new filename for further writes. Caveat: each hsync()/sync() requires many underlying read and mutation requests occurring sequentially, so latency is expected to be fairly high.

If errors occur mid-stream, there may be one or more temporary files failing to be cleaned up, and requires manual intervention to discover and delete any such unused files. Data written prior to the most recent successful hsync() is persistent and safe in such a case.

The possible cause for files not being closed is you are not closing the stream in your code base or there is an error. Debug your code step by step to check if the stream is being closed properly and files are removed as shown here

Note : As the code is asynchronous you need to make sure that the thread that executes the deletion is running.

user_158033650
Related