How to properly deal with Celery and Azure Storage queues containing a '/' character in the access key?

user_4240413 0
Viewed 773

I am creating a Celery application using the syntax

celery_app = Celery(
    my_config_name,
    backend=my_backend,
    broker=my_broker,
)

For my_broker, I am using the address of an Azure storage account that reads

my_broker = "azurestoragequeues://:jrIEoHgyi7y8L7dJ+0CeYGnR9rHLwzMRsKzrmUle7ZL8OA/EaO5aBxqEYSB2VPXJ2v4C58D==@myazurequeue"

and note that the password bit contains a '/' (that's the corresponding access key that can be read also in the Azure portal).

If I run my code, I get 

ValueError: invalid literal for int() with base 10: 'jrIEoHgyi7y8L7dJ+0CeYGnR9rHLwzMRsKzrmUle7ZL8OA'

which is the same error that I get with 

from kombu.utils.url import parse_url
parse_url(my_broker)

One fix that seems to work is something along the lines of 

from kombu.utils.url import safequote
my_broker_fixed = "azurestoragequeues://:" + safequote(str.replace(my_broker, "azurestoragequeues://:", ""), safe = "=@")

which can encode the '/' character in the access key only (but not in the rest of the address). This also parses fine when firing parse_url(my_broker) as it yields

{'hostname': 'myazurequeue',
 'password': 'jrIEoHgyi7y8L7dJ+0CeYGnR9rHLwzMRsKzrmUle7ZL8OA/EaO5aBxqEYSB2VPXJ2v4C58D==',
 'port': None,
 'transport': 'azurestoragequeues',
 'userid': None,
 'virtual_host': None}

This seems to be OK as a workaround, but is there a better fix?

azure-storage-queues celery urlencode python
1 Answers

The password is the part of the Transport connection that needs to be safely quoted.

Store the username, password, hostname for the connection as environment variables and read those in your application.

from kombu.utils.url import safequote

TRANSPORT_USER = os.getenv('TRANSPORT_USER'),
TRANSPORT_PASS = safequote(os.getenv('TRANSPORT_PASS')),
TRANSPORT_HOST = os.getenv('TRANSPORT_HOST')

my_broker = f'azurestoragequeues://{TRANSPORT_USER}:{TRANSPORT_PASS}@{TRANSPORT_HOST}'
user_51898110
Related