From local Python function to Google Cloud Function

Question

I created a Python function for an API call so I longer have to do that in Power BI. It creates 5 XML files that are then combined into a single CSV-file. I would like the function to run on Google Cloud (correct me if this is not a good idea).

I don't think it' s possible to create XML files in the function (maybe it's possible to write to a bucket) but ideally I would like to skip the XML file creation and just go straight to creating the CSV.

Please find the code for generating the XML files and combining into CSV below:

offices = ['NL001', 'NL002', 'NL003', 'NL004', 'NL005']
#Voor elke office inloggen, office veranderen en een aparte xml maken
for office in offices:

    xmlfilename = office+'.xml'

    session.service.SelectCompany(office, _soapheaders={'Header': auth_header})
    proces_url = cluster + r'/webservices/processxml.asmx?wsdl'
    proces = Client(proces_url)
    response = proces.service.ProcessXmlString(query.XML_String,  _soapheaders={'Header': auth_header})

    f = open(xmlfilename, 'w')
    f.write(response)
    f.close()

to csv

if os.path.exists('CombinedFinance.csv'):
  os.remove('CombinedFinance.csv')
else:
  print("The file does not exist")

xmlfiles = ['NL001.xml','NL002.xml','NL003.xml','NL004.xml','NL005.xml']
for xmlfile in xmlfiles:
    with open(xmlfile, encoding='windows-1252') as xml_toparse:
        tree = ET.parse(xml_toparse)

    root = tree.getroot()
    columns = [element.attrib['label'] for element in root[0]]
    columns.append('?')

    data = [[field.text for field in row] for row in root[1::]]
    df = pd.DataFrame(data, columns=columns)
    df = df.drop('?', axis=1)

    df.to_csv('CombinedFinance.csv', mode='a', header=not os.path.exists('CombinedFinance.csv'))

Any ideas?

n.b. If i can improve my code please let me know, I'm just learning all of this

EDIT: In response to some comments, code now looks like this. When deploying to cloud I get the following error:

ERROR: (gcloud.functions.deploy) OperationError: code=13, message=Function deployment failed due to a health check failure. This usually indicates that your code was built successfully but failed during a test execution. Examine the logs to determine the cause. Try deploying again in a few minutes if it appears to be transient.

My requirements.txt looks like this:

zeep==3.4.0 pandas

Any ideas?

import pandas as pd
import xml.etree.ElementTree as ET
from zeep import Client
import query
import authentication
import os

sessionlogin = r'https://login.twinfield.com/webservices/session.asmx?wsdl'
login = Client(sessionlogin)

auth = login.service.Logon(authentication.username, authentication.password, authentication.organisation)

auth_header = auth['header']['Header']
cluster = auth['body']['cluster']

#Use cluster to create a session:

url_session = cluster + r'/webservices/session.asmx?wsdl'
session = Client(url_session)

#Select a company for the session:

offices = ['NL001', 'NL002', 'NL003', 'NL004', 'NL005']
#Voor elke office inloggen, office veranderen en een aparte xml maken
for office in offices:

    session.service.SelectCompany(office, _soapheaders={'Header': auth_header})
    proces_url = cluster + r'/webservices/processxml.asmx?wsdl'
    proces = Client(proces_url)
    response = proces.service.ProcessXmlString(query.XML_String,  _soapheaders={'Header': auth_header})


    treetje = ET.ElementTree(ET.fromstring(response))
    root = treetje.getroot()
    columns = [element.attrib['label'] for element in root[0]]
    columns.append('?')

    data = [[field.text for field in row] for row in root[1::]]
    df = pd.DataFrame(data, columns=columns)
    df = df.drop('?', axis=1)

    df.to_csv('/tmp/CombinedFinance.csv', mode='a', header=not os.path.exists('/tmp/CombinedFinance.csv'))

user_328036 · Answer

A few things to consider about turning a regular Python script (what you have here) into a Cloud Function:

Cloud Functions respond to events -- either an HTTP request or some other background trigger. You should think about the question "what is going to trigger my function?"
HTTP functions take in a request that corresponds to the incoming request, and must return some sort of HTTP response
The only available part of the filesystem that you can write to is /tmp. You'll have to write all files there during the execution of your function
The filesystem is ephemeral. You can't expect files to stick around between invocations. Any file you create must either be stored elsewhere (like in a GCS bucket) or returned in the HTTP response (if it's an HTTP function)
A Cloud Function has a very specific signature that you'll need to wrap your existing business logic in:

def my_http_function(request):
    # business logic here
    ...
    return "This is the response", 200

def my_background_function(event, context):
    # business logic here
    ...
    # No return necessary

From local Python function to Google Cloud Function

1 Answers