yaml.dump adding unwanted newlines in multiline strings

user_3787051 0
Viewed 15952

I have a multiline string:

>>> import credstash
>>> d = credstash.getSecret('alex_test_key', region='ap-southeast-2')

To see the raw data (first 162 characters):

>>> credstash.getSecret('alex_test_key', region='ap-southeast-2')[0:162]
u'-----BEGIN RSA PRIVATE KEY-----\nMIIEogIBAAKCAQEA6oySC+8/N9VNpk0gJS7Gk8vn9sYN7FhjpAQnoHRqTN/Oaiyx\nxk2AleP2vXpojA/DHldT1JO+o3j56AHD+yfNFFeYvgWKDY35g49HsZZhbyCEAB45\n'

And:

>>> print d[0:162]                                                                                                                                                                                          
-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEA6oySC+8/N9VNpk0gJS7Gk8vn9sYN7FhjpAQnoHRqTN/Oaiyx
xk2AleP2vXpojA/DHldT1JO+o3j56AHD+yfNFFeYvgWKDY35g49HsZZhbyCEAB45

I write to a YAML file:

>>> import yaml
>>> with open('foo.yaml', 'w') as f:                                                                                                                                                                        
...     yaml.dump(d, f, default_flow_style=False, explicit_start=True)
...

Now it looks like this:

$ head -5 foo.yaml 
--- !!python/unicode '-----BEGIN RSA PRIVATE KEY-----

  MIIEogIBAAKCAQEA6oySC+8/N9VNpk0gJS7Gk8vn9sYN7FhjpAQnoHRqTN/Oaiyx

  xk2AleP2vXpojA/DHldT1JO+o3j56AHD+yfNFFeYvgWKDY35g49HsZZhbyCEAB45

i.e. each line has two newlines.

Now if I read it back into a string I see that all is okay in the round-trip:

>>> with open('foo.yaml', 'r') as f:
...     d = yaml.load(f)
... 
>>> print d[0:162]
-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEA6oySC+8/N9VNpk0gJS7Gk8vn9sYN7FhjpAQnoHRqTN/Oaiyx
xk2AleP2vXpojA/DHldT1JO+o3j56AHD+yfNFFeYvgWKDY35g49HsZZhbyCEAB45

(I don't understand why however.)

My real problem is that if humans read this YAML file they will probably assume, as I did, that my program has broken the formatting of the private key file.

Is there a way to use yaml.dump so as to output something without the additional newline characters?

pyyaml serialization python
4 Answers

Better ruamel.yaml option for outputting multiline strings as blocks:

from ruamel.yaml.representer import RoundTripRepresenter
from ruamel.yaml import YAML

multiline_string = """\
-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEA6oySC+8/N9VNpk0gJS7Gk8vn9sYN7FhjpAQnoHRqTN/Oaiyx
xk2AleP2vXpojA/DHldT1JO+o3j56AHD+yfNFFeYvgWKDY35g49HsZZhbyCEAB45
...
"""


def repr_str(dumper: RoundTripRepresenter, data: str):
    if '\n' in data:
        return dumper.represent_scalar('tag:yaml.org,2002:str', data, style='|')
    return dumper.represent_scalar('tag:yaml.org,2002:str', data)


yaml = YAML()
yaml.representer.add_representer(str, repr_str)

with open('file.yaml', 'w') as fp:
    yaml.dump({'a': 1, 'b': 'hello world', 'c': multiline_string}, fp)

Output:

a: 1
b: hello world
c: |
  -----BEGIN RSA PRIVATE KEY-----
  MIIEogIBAAKCAQEA6oySC+8/N9VNpk0gJS7Gk8vn9sYN7FhjpAQnoHRqTN/Oaiyx
  xk2AleP2vXpojA/DHldT1JO+o3j56AHD+yfNFFeYvgWKDY35g49HsZZhbyCEAB45
  ...
user_113542660

Assuming you have a file named "privateKey.pem" with your key you could extract the content into a YAML multiline block with a '|' and 2 spaces indentation:

# Convert To yaml multiline block
cat privateKey.pem | sed -e '1 i PRIVATE_KEY: |' -e 's#^#  #g' > parameters.yaml

cat <<_EOF_ >>parameters.yaml
SSH_HOST: 'host.name.or.ip'
SSH_USER: 'cloud-user'
_EOF_

You get something like:

PRIVATE_KEY: |
  -----BEGIN OPENSSH PRIVATE KEY-----
  b3BlbnNzaC1rZXkt....
  ...
SSH_HOST: 'host.name.or.ip'

Then in your shell script reading from parameters.yaml file

# Read values From Yaml using “PyYAML”
FILE=parameters.yaml

KEY=PRIVATE_KEY
python3 -c "from yaml import load; f = open('$FILE'); y = load(f); print(y['$KEY'])" > priv.key
chmod 600 priv.key

KEY=SSH_HOST
ssh_host=$(python3 -c "from yaml import load; f = open('$FILE'); y = load(f); print(y['$KEY'])")

KEY=SSH_USER
ssh_user=$(python3 -c "from yaml import load; f = open('$FILE'); y = load(f); print(y['$KEY'])")

# test
ssh -i priv.key ${ssh_user}@${ssh_host} "hostname"

More details in https://yaml-multiline.info/

user_107879090
Related