bash / grep: getting multiple matching elements from a json

user_6082572 0
Viewed 82

In my BitBucket+Bamboo setup, I'm trying to get a list of email addresses of people having access to a particular repository. This is the output from the BitBucket API:

{
    "size": 3,
    "limit": 25,
    "isLastPage": true,
    "values": [
        {
            "user": {
                "name": "name1",
                "emailAddress": "name1.lastname1@domain.com",
                "id": 1,
                "displayName": "Name1 Lastname1",
                "active": true,
                "slug": "name1",
                "type": "NORMAL",
                "links": {
                    "self": [
                        {
                            "href": "https://bitbucket.com/stash/users/name1"
                        }
                    ]
                }
            },
            "permission": "REPO_WRITE"
        },
        {
            "user": {
                "name": "name2",
                "emailAddress": "name2.lastname2@domain.com",
                "id": 2,
                "displayName": "Name2 Lastname2",
                "active": true,
                "slug": "name2",
                "type": "NORMAL",
                "links": {
                    "self": [
                        {
                            "href": "https://bitbucket.com/stash/users/name2"
                        }
                    ]
                }
            },
            "permission": "REPO_WRITE"
        },
        {
            "user": {
                "name": "name3",
                "emailAddress": "name3.lastname3@domain.com",
                "id": 3,
                "displayName": "Name3 Lastname3",
                "active": true,
                "slug": "name3",
                "type": "NORMAL",
                "links": {
                    "self": [
                        {
                            "href": "https://bitbucket.com/stash/users/name3"
                        }
                    ]
                }
            },
            "permission": "REPO_WRITE"
        }
    ],
    "start": 0
}

is there an easy way to, say, put all 3 email addresses into an array or a coma-separated variable within a bash script? I tried using grep and splitting the API output somehow (e.g. by 'permission'), but no luck so far. Let me note that I may be forced to use standard tools like grep, sed or awk, meaning I may not be able to use tools like jq (to process json in bash) since I cannot really temper with available build agents.

Any help would be much appreciated!

sed grep awk
4 Answers

Consider using JQ (or another JSON query tool). It will handle any valid Json, even one that is not pretty-printed or formatted in a specific way. Ca be compined with readarray to build the array in bash.

readarray -t emails <<< "$(jq -r '.values[].user.emailAddress' < file)"

Will produce an array 'emails'

declare -p emails
declare -a emails=([0]=$'name1.lastname1@domain.com' [1]=$'name2.lastname2@domain.com' [2]=$'name3.lastname3@domain.com')

Note 2020-07-22: Added '-t' to strip trailing new lines from result array

user_120984050

Assuming your input is always that regular, this will work using any awk in any shell on every UNIX box:

$ awk -F'"' '$2=="emailAddress"{addrs=addrs sep $4; sep=","} END{print addrs}' file
name1.lastname1@domain.com,name2.lastname2@domain.com,name3.lastname3@domain.com

Save the output in a variable or a file as you see fit, e.g.:

$ var=$(awk -F'"' '$2=="emailAddress"{addrs=addrs sep $4; sep=","} END{print addrs}' file)
$ echo "$var"
name1.lastname1@domain.com,name2.lastname2@domain.com,name3.lastname3@domain.com
user_17450010

Take a look on the python:

You can access directly to your api like this:

import urllib.request
import json
with urllib.request.urlopen('http://your/api') as url:
  data = json.loads(url.read().decode())

or as an example with the local file with the same data as you provided:

import json
with open('./response.json') as f:
  data = json.load(f)

result = {}
for x in data['values']:
  node = x['user']
  result[node['emailAddress']] = x['permission']

result is {'name1.lastname1@domain.com': 'REPO_WRITE', 'name2.lastname2@domain.com': 'REPO_WRITE', 'name3.lastname3@domain.com': 'REPO_WRITE'}

user_42077930 
$ grep -oP '(?<="emailAddress": ).*' file | 
  tr -d '",'                              | 
  paste -sd,

name1.lastname1@domain.com,name2.lastname2@domain.com,name3.lastname3@domain.com

or

$ grep '"emailAddress":' file | 
  cut -d: -f2                 | 
  tr -d '", '                 | 
  paste -sd,
user_14358690
Related