I get this error while trying to use the library apache_beam.io.gcp.gcsfilesystem. I want to use the function copy().
I tried a simple example:
from apache_beam.io.gcp.gcsfilesystem import GCSFileSystem
GCSFileSystem(pipeline_options = None).copy(source_file_names = image_urls, destination_file_names = destination_urls)
Is it possible that the problem is due to passing no pipeline options? or what library could be missing, since every else using apache beam, and GCP APIs are working fine.
StorageV1 is from an internal module of Beam Python SDK itself. The existence of this attribute shouldn't matter whether you have GCP dependencies installed or not.
You can check the module by:
>>> from apache_beam.io.gcp.internal.clients import storage
>>> storage.StorageV1
<class 'apache_beam.io.gcp.internal.clients.storage.storage_v1_client.StorageV1'>
>>> vars(storage)
Outputs attributes of this module.
How did you install apache-beam? Is it from source code? Could the source code be corrupted?
You can also try the imports from init.py from the storage package:
from apitools.base.py import *
from apache_beam.io.gcp.internal.clients.storage.storage_v1_client import *
from apache_beam.io.gcp.internal.clients.storage.storage_v1_messages import *
See if any ImportError is thrown that could affect your packaging because the ImportErrors are suppressed here. It's highly possible the google-apitools dependency is not installed and thus silently failed the packaging. In that case, you need to install apache-beam[gcp] to make sure you have the required dependency.
Looking at
it could be because you don't have apitools installed. Did you install apache_beam[grpc]?