My needs are the following:
I need to scrape the data from the webpage using azure functions, and save the scraping output directly to the azure blob storage as a .csv or .parquet.
Please see the example code:
response = requests.get('example_url')
json_response = response.json()
name = json_response.get('name', 'N/A')
with open('warsaw.txt', 'w') as f:
f.write(name)
How can i write the f directly into the azure blob storage, using Blob Service Client?
I need to scrape the data from the webpage using azure functions, and save the scraping output directly to the azure blob storage as a .csv or .parquet.
I think you can use 'append' method to achieve your requirement:
from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient, BlobType
try:
connect_str = "DefaultEndpointsProtocol=https;AccountName=1123bowman;AccountKey=xxxxxx;EndpointSuffix=core.windows.net"
container_name = "test"
blob_name = "test.txt"
data1 = "\n1,2,3"
data = str.encode(data1)
blob_service_client = BlobServiceClient.from_connection_string(connect_str)
blob_client = blob_service_client.get_blob_client(container=container_name, blob=blob_name)
print("length of data is "+str(len(data)))
blob_client.append_block(data,length=len(data))
except:
connect_str = "DefaultEndpointsProtocol=https;AccountName=1123bowman;AccountKey=xxxxxx;EndpointSuffix=core.windows.net"
container_name = "test"
blob_name = "test.txt"
data1 = "test1,test2,test3"
data = str.encode(data1)
blob_service_client = BlobServiceClient.from_connection_string(connect_str)
blob_client = blob_service_client.get_blob_client(container=container_name, blob=blob_name)
print("length of data is "+str(len(data)))
blob_client.upload_blob(data,blob_type=BlobType.AppendBlob)
Through the above method, you don’t have to store the entire blob every time.
you can use Azure Storage Blobs client library for Python
Example on howto upload a file to Azure Storage Blobs:
from azure.storage.blob import BlobClient
blob = BlobClient.from_connection_string(conn_str="<connection_string>", container_name="my_container", blob_name="my_blob")
with open("./warsaw.txt", "rb") as data:
blob.upload_blob(data)