Powershell ConvertFrom-Json Encoding Special Characters Issue

user_1425462 0
Viewed 17229

I have this code in my powershell script and it doesn't do well on the special characters parts. 

 $request = 'http://151.80.109.18:8082/vrageremote/v1/session/players'
 $a = Invoke-WebRequest -ContentType "application/json; charset=utf-8" $request |
 ConvertFrom-Json    |
 Select -expand Data |
 Select -expand players |
 Select displayName, factionTag | Out-file "$scriptPath\getFactionTag.txt"

In my output file I only get '????' for any special characters. Does anyone know how I can get it to show special characters in my output file?

invoke-webrequest character-encoding powershell
3 Answers

Peter Schneider's helpful answer and Nas' helpful answer both address one problem with your approach: You need to:

  • either: access the .Content property on the response object returned by Invoke-WebRequest to get the actual data returned (as a JSON string), which you can then pass to ConvertFrom-Json.

  • or: use Invoke-RestMethod instead, which returns the data directly and parses it into custom objects, so you can work with these objects directly, without the need for ConvertTo-Json; however, with a character-encoding problem such as in this case this is not an option, because explicit re-encoding of the JSON string is needed - see below.

However, you still have a character-encoding problem, because, in the absence of charset information in the response header, PowerShell interprets the UTF-8-encoded JSON string returned as ISO-8859-1-encoded (still applies as of PowerShell 7.0).

There are two possible solutions:

  • Preferably, amend the web service to include charset=utf-8 in the response header's ContenType field.

  • If you can't do that, you must explicitly re-encode the string received to correct the character-encoding misinterpretation.

Here's the implementation of the latter:

$request = 'http://151.80.109.18:8082/vrageremote/v1/session/players'
$a = Invoke-WebRequest -ContentType "application/json; charset=utf-8" $request

# $a.Content now contains the *misinterpreted* JSON string, so we must 
# obtain its byte representation and re-interpret the bytes as UTF-8.
# Encoding 28591 represents the ISO-8859-1 encoding - see https://docs.microsoft.com/en-us/windows/desktop/Intl/code-page-identifiers
$jsonCorrected = [Text.Encoding]::UTF8.GetString(
  [Text.Encoding]::GetEncoding(28591).GetBytes($a.Content)
)

# Now process the reinterpreted string.
$jsonCorrected |
  ConvertFrom-Json    |
  Select -expand Data |
  Select -expand players |
  Select displayName, factionTag | Out-file "$scriptPath\getFactionTag.txt"
user_453750

Use Invoke-RestMethod if you only need the json data without the ParsedHtml, Headers and other objects returned by Invoke-WebRequest

$request = 'http://151.80.109.18:8082/vrageremote/v1/session/players'
$a = Invoke-RestMethod -ContentType "application/json; charset=utf-8" $request |
Select -expand Data |
Select -expand players |
Select displayName, factionTag | Out-file "$scriptPath\getFactionTag.txt"
user_103070350

Try to convert the value of the .Content property to JSON:

$request = 'http://151.80.109.18:8082/vrageremote/v1/session/players'
$a = Invoke-WebRequest -ContentType "application/json; charset=utf-8" $request

($a.Content | convertfrom-json).Data.Players | select DisplayName,FactionTag | Out-file "$scriptPath\getFactionTag.txt" -Encoding Default
user_13548000
Related