gsutil rsync 'ascii' codec can't decode byte 0xe2 in position 51: ordinal not in range(128)

user_7883362 0
Viewed 194

I am using gsutil command from my Debian server for rsync files to GCP bucket but it fails by following error:

gsutil -m rsync -r -n files/ gs://my-bucket/files

At source listing 10000...
At source listing 20000...
At source listing 30000...
Caught non-retryable exception while listing file://files/: 'ascii' codec can't decode byte 0xe2 in position 51: ordinal not in range(128)
CommandException: Caught non-retryable exception - aborting rsync

gsutil version: 4.52

Around 1TB data need to be transferred.

gsutil google-cloud-platform rsync
1 Answers

I solved my own problem by following PHP code which will iterates the source directory and move find non-ascii filenames and move them to different location.

One can also remove non-ascii characters from filename and rename with same logic or can delete using unlink method.

<ul>
<?php
// this will help you to debug the code if you get some errors.
ini_set('display_errors', '1');
ini_set('display_startup_errors', '1');
error_reporting(E_ALL);

$source = 'source_files/'; // update this with your source path. 
$iterator = new RecursiveTreeIterator(new RecursiveDirectoryIterator($dir, RecursiveDirectoryIterator::SKIP_DOTS));

$dest = 'destination_path'; // update it with your destination path

foreach($iterator as $filename => $path) {
    if(preg_match('/[^\x20-\x7e]/', $path)) {
        echo "<li>".$filename."</li>";
        // rename($filename, $dest.pathinfo($filename, PATHINFO_BASENAME)); // Uncomment if you want to move it with some other location. 

        // unlink($filename); // Uncomment if you want to delete that file.
    }
}
?>
</ul>

NOTE: Make sure you have set correct permissions to source and destination directory.

Hope it will help someone in the future.

Thanks

user_78833620
Related