Google Cloud Resumable download not happening

Viewed 159

My code has three sections

  • CustomDownloadProgressListener
  • downloadToOutputStream() function in which I wrote code to download file from cloud storage
  • RetryHttpInitializerWrapper a wrapper around HttpRequestInitializer as suggested in MediaHttpDownloader class java doc in case response is bad or network breaks.

Same RetryHttpInitializerWrapper I used to achieve resumable upload and its working as expected its uploading in 10-10 MB chunks. Please suggest what I am miss here because for single request it works fine but for resumable download it dint. I simply disconnected my internet while downloading to test resuamble thing but when I connect again it dint resume downloading.

    public void downloadToOutputStream(String bucketName, String objectName, OutputStream data)
            throws IOException, GeneralSecurityException {
        GoogleCredential credential = GoogleCredential.getApplicationDefault();
        if (credential.createScopedRequired()) {
            credential = credential.createScoped(StorageScopes.all());
        }
        HttpTransport httpTransport = GoogleNetHttpTransport.newTrustedTransport();
        // custom HttpRequestInitializer for automatic retry upon failures.
        HttpRequestInitializer httpRequestInitializer = new RetryHttpInitializerWrapper(credential);

        GenericUrl requestUrl = new GenericUrl(
                "https://www.googleapis.com/storage/v1/b/" + bucketName + "/o/" + objectName);
        MediaHttpDownloader downloader = new MediaHttpDownloader(httpTransport, httpRequestInitializer);
        downloader.setProgressListener(new CustomDownloadProgressListener());
        downloader.download(requestUrl, data);
    }

    public class RetryHttpInitializerWrapper implements HttpRequestInitializer {
        private static final Logger LOG = Logger.getLogger(RetryHttpInitializerWrapper.class.getName());
        private final Credential wrappedCredential;
        private final Sleeper sleeper;
        private static final int MILLIS_PER_MINUTE = 60 * 1000;

        /**
         * A constructor using the default Sleeper.
         *
         * @param wrappedCredential
         *            the credential used to authenticate with a Google Cloud
         *            Platform project
         */
        public RetryHttpInitializerWrapper(Credential wrappedCredential) {
            this(wrappedCredential, Sleeper.DEFAULT);
        }

        /**
         * A constructor used only for testing.
         *
         * @param wrappedCredential
         *            the credential used to authenticate with a Google Cloud
         *            Platform project
         * @param sleeper
         *            a user-supplied Sleeper
         */
        RetryHttpInitializerWrapper(Credential wrappedCredential, Sleeper sleeper) {
            this.wrappedCredential = Preconditions.checkNotNull(wrappedCredential);
            this.sleeper = sleeper;
        }

        /**
         * Initialize an HttpRequest.
         *
         * @param request
         *            an HttpRequest that should be initialized
         */
        public void initialize(HttpRequest request) {
            request.setReadTimeout(2 * MILLIS_PER_MINUTE); // 2 minutes read
                                                            // timeout
            final HttpUnsuccessfulResponseHandler backoffHandler = new HttpBackOffUnsuccessfulResponseHandler(
                    new ExponentialBackOff()).setSleeper(sleeper);
            request.setInterceptor(wrappedCredential);
            request.setUnsuccessfulResponseHandler(new HttpUnsuccessfulResponseHandler() {
                public boolean handleResponse(final HttpRequest request, final HttpResponse response,
                        final boolean supportsRetry) throws IOException {
                    if (wrappedCredential.handleResponse(request, response, supportsRetry)) {
                        // If credential decides it can handle it, the return
                        // code or message indicated
                        // something specific to authentication, and no backoff
                        // is desired.
                        return true;
                    } else if (backoffHandler.handleResponse(request, response, supportsRetry)) {
                        // Otherwise, we defer to the judgement of our internal
                        // backoff handler.
                        System.out.println("Retrying " + request.getUrl().toString());
                        return true;
                    } else {
                        return false;
                    }
                }
            });
            request.setIOExceptionHandler(
                    new HttpBackOffIOExceptionHandler(new ExponentialBackOff()).setSleeper(sleeper));
        }
    }

 public class CustomDownloadProgressListener implements MediaHttpDownloaderProgressListener {
    public void progressChanged(MediaHttpDownloader downloader) {
        switch (downloader.getDownloadState()) {
        case MEDIA_IN_PROGRESS:
            System.out.println(downloader.getProgress());
            break;
        case MEDIA_COMPLETE:
            System.out.println("Download is complete!");
        }
    }
}
1 Answers

I'm not posting any code here, but I recently successfully achieved a resumable download from Google Cloud Storage using the Range header.

Suppose I have a 570 byte text file that starts with:

The next morning Hanna distracts Dieter

If I send a Range header with value bytes=0-21 then I get back The next morning Hanna.

If I send a Range header with value bytes=3-32 then I get back next morning Hanna distracts.

(NOTE: The Range header uses an inclusive interval.)

Unless my chunk size is larger then the file size, and barring some network error, I should always expect the response to contain a Content-Range header indicating what range of bytes I just downloaded, as well as the total file size. For example, the response to my first request would contain Content-Range: bytes 0-21/570, so my next request would have its Range start at byte 22, e.g., bytes=22-43.

Related