Issues with encoding of a downloaded string (UTF-8, ISO-8859-1, FSharp.Data)

user_1942656 0
Viewed 136

I am using the FSharp.Data for downloading and parsing an XML document. The document contains several nordic characters, like: "Lempäälän Keskus". I have a simple code like this:

open FSharp.Data
open System.IO

let xml = Http.RequestString(downloadUrl)
File.WriteAllText("response.xml", xml)

However, when I open the resulting file, the characters are corrupted and I see this: "Lempäälän Keskus". I have tried manually re-saving the file with the ISO-8859-1 encoding and then reopening with UTF-8 (in VS Code). This fixed the displayed text. Then tried to implement a simple encoding conversion function:

open FSharp.Data
open System
open System.IO

let convertEncoding (input: string) =
  let iso = Encoding.GetEncoding("ISO-8859-1")
  let utf8 = Encoding.UTF8
  let isoBytes = iso.GetBytes(input)
  let utfBytes = Encoding.Convert(iso, utf8, isoBytes)
  utf8.GetString(utfBytes)

let xml = Http.RequestString(downloadUrl)
let decoded = convertEncoding xml
File.WriteAllText("response.xml", decoded)

However, when I open the file, I still see "Lempäälän Keskus" instead of "Lempäälän Keskus". What am I doing wrong?

f#-data utf-8 f# xml
2 Answers

Provide encoding when you write text to file

let encoding = Encoding.GetEncoding("ISO-8859-1")
File.WriteAllText("response.xml", xml, encoding)
user_4700050

Actually, this seems to be a weird behavior caused by the Http client from the FSharp.Data library. I am not sure whether this is a bug, or just a functionality that I haven't found the documentation for.

It is enough to replace the FSharp.Data.Http client with a System.Net.Http.HttpClient. Suddenly, all the encoding errors disappear and no conversion is needed whatsoever. Here is a sample solution:

open System
open System.IO
open System.Net.Http

let download (url: string) = 
  async {
    use client = new HttpClient()
    let! xml = client.GetStringAsync(url) 
                |> Async.AwaitTask
    return xml
  }

let result = download url |> Async.RunSynchronously
File.WriteAllText("response.xml", result)
user_19426560
Related