Terraform External Data Source not working properly

user_7195607 0
Viewed 940

I am using Terraform External Data Source to get a response from the Azure DevOps rest api (objectId of the branch to be used as further input to other scripts/actions).

I'm calling a python script via the external data source resource as following:

data "external" "example" {
  program = ["python", "ex-data-source.py"]

  query = {
    organisation = "https://dev.azure.com/my-org"
    project = "my-proj"
    repositoryId = "my-repo"
    branch = "heads/my-branch"
    new_branch = "heads/newbranch"
    pat = "my-pat"
  }
}

output "branch_id" {
  value = data.external.example.result.branch_id
}

My ex-data-source.py file is:

import base64
import json
import sys
import requests
from azure.cli.core import get_default_cli

# Step#1 - Parse the input as stdin from terraform external data source
input = sys.stdin.read()
input_json = json.loads(input)

# Step#2 - Extract the relevant data based on key passed
pat = input_json.get("pat")
organisation = input_json.get("organisation")
project = input_json.get("project")
repositoryId = input_json.get("repositoryId")
branch = input_json.get("branch")
new_branch = input_json.get("new_branch")

b64userpass = str(base64.b64encode(bytes(':' + pat, 'ascii')), 'ascii')
headers = {
    'Accept': 'application/json',
    'Authorization': 'Basic %s' % b64userpass
}

list_url = f"https://dev.azure.com/{organisation}/{project}/_apis/git/repositories/{repositoryId}/refs?filtercontain={branch}&api-version=6.1-preview.1"

r = requests.get(list_url, headers=headers)
if r.status_code == 200:
    r_json = r.json()
    branch_id = r_json["value"][0]["objectId"]

# Step#3 - Extract the relevant data based on key passed
output = {
    "objectID": branch_id
}

output_json = json.dumps(output)
print(output_json)

Expected Output ideally I should get back the objectId of the branch e.g. a750b8eb8e26a2e4b9910c10f29e6d55d8639d50

Error received

Error: failed to execute "python": Traceback (most recent call last):
│   File "/Users/metro.akaushik/Desktop/Projects/AzDO/ex-data-source.py", line 42, in <module>
│     "objectID": branch_id
│ NameError: name 'branch_id' is not defined
│ 
│ 
│   with data.external.example,
│   on main.tf line 21, in data "external" "example":
│   21: data "external" "example" {

The python code runs just fine when executed outside of terraform. Please help in resolving the issue, as this lays foundation for my future scripts/actions

The unfiltered api response body is:

{'count': 2,
 'value': [{'creator': {'_links': {'avatar': {'href': 'https://dev.azure.com/xxxx/_apis/GraphProfile/MemberAvatars/aad.xxxxx'}},
                        'descriptor': 'aad.xxxxxx',
                        'displayName': 'xxxxx',
                        'id': 'eb6383c7-bf62-6cec-9ae4-57dc29b68deb',
                        'imageUrl': 'https://dev.azure.com/xxxx/_api/_common/identityImage?id=xxxxx',
                        'uniqueName': 'xxxxx',
                        'url': 'https://spsprodweu1.vssps.visualstudio.com/xxxxx/_apis/Identities/xxxx'},
            'name': 'refs/heads/devops',
            'objectId': 'a750b8eb8e26a2e4b9910c10f29e6d55d8639d50',
            'url': 'https://dev.azure.com/xxxx/xxxxx/_apis/git/repositories/xxxxxx/refs?filter=heads%2Fdevops'},
           {'creator': {'_links': {'avatar': {'href': 'https://dev.azure.com/xxxx/_apis/GraphProfile/MemberAvatars/aad.xxxxx'}},
                        'descriptor': 'aad.xxxxx',
                        'displayName': 'xxxx',
                        'id': 'eb6383c7-bf62-6cec-9ae4-57dc29b68deb',
                        'imageUrl': 'https://dev.azure.com/xxxx/_api/_common/identityImage?id=xxxxx',
                        'uniqueName': 'xxxxx',
                        'url': 'https://spsprodweu1.vssps.visualstudio.com/xxxx/_apis/Identities/xxxx'},
            'name': 'refs/heads/main',
            'objectId': 'a2956c20d33541b337c01f7ddfd5315acc796dfe',
            'url': 'https://dev.azure.com/xxxx/xxxx/_apis/git/repositories/xxxx/refs?filter=heads%2Fmain'}]}
external-data-source terraform python
1 Answers

Thanks @Marcin... I actually get a 400 status code and a garbled content from the response body as:

"b'\\xef\\xbb\\xbf{\"$id\":\"1\",\"innerException\":null,\"message\":\"A potentially dangerous Request.Path value was detected from the client (:).\",\"typeName\":\"System.Web.HttpException, System.Web\",\"typeKey\":\"HttpException\",\"errorCode\":0,\"eventId\":0}'"

The requests module doesn't work as expected while running via terraform (while it works like a charm from the terminal)

If I directly try a json response (ex. r = requests.get(list_url, headers=headers).json()), then the I get an error

     r = requests.get(list_url, headers=headers).json()
   File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/models.py", line 900, in json
     return complexjson.loads(self.text, **kwargs)
   File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/__init__.py", line 335, in loads
     raise JSONDecodeError("Unexpected UTF-8 BOM (decode using utf-8-sig)",
 json.decoder.JSONDecodeError: Unexpected UTF-8 BOM (decode using utf-8-sig)

I do try to get past the BOM encoding error, but not successful, as I think the requests module is not giving back the proper response in the first place, before even trying to decode.

r = requests.get(list_url, headers=headers).json()
r_decoded = codecs.decode(r.content, 'utf-8-sig')
branch_id = r_decoded["value"][0]["objectId"]
user_71956070
Related