How to find and replace unicode chars in Haskell?

user_4050510 0
Viewed 493

I have a unicode file containing a (Swedish) wikipedia article in MediaText markup. I want to clean it from all markup. In certain cases I want to extract text from the markup tags, such as the link titles from hyperlinks (like a simplified wikiextractor).

My approach is to run a set of regex'es over the file to remove markup. In the link-example, I need to replace [[link]] with link. I manage to fix this well with a regex as long as the text does not contain unicode chars such as ö.

Example of what I've tried:

ghci> :m +Data.Text
ghci> subRegex (mkRegex "\\[\\[([() a-zA-Z]*)\\]\\]") "Se mer om [[Stockholm]]" "\\1"
"Se mer om Stockholm"
ghci> subRegex (mkRegex "\\[\\[([() a-zA-Z]*)\\]\\]") "Se mer om [[Göteborg]]" "\\1"
"Se mer om [[G\246teborg]]"

Why does this not work? How can I make the regex engine realize that ö is indeed a normal letter (at least in Swedish)?

Edit: The issue seems to not really sit in the pattern, but in the engine. If i allow all characters except q in the link text, one could expect ö to be allowed. But not so...

ghci> subRegex (mkRegex "\\[\\[([^q]*)\\]\\]") "[[Goteborg]]" "\\1"
"Goteborg"
ghci> subRegex (mkRegex "\\[\\[([^q]*)\\]\\]") "[[Göteborg]]" "\\1"
"[[G\246teborg]]"
ghci> subRegex (mkRegex "ö") "ö" "q"
"q"
ghci> subRegex (mkRegex "[ö]") "ö" "q"
"\246"

The problem seems to arise specifically when using character classes. It matches öfine on its own.

unicode haskell regex
2 Answers

To find and replace unicode characters in Haskell we can use the streamEdit function, with Megaparsec parsers for the pattern matching (instead of regex). The Megaparsec letterChar parser will match all Swedish letters.

:set -XOverloadedStrings
import Text.Megaparsec
import Text.Megaparsec.Char
import Replace.Megaparsec
import Data.Text as T
import Data.Text.IO as T
import Data.Void

let wikilink :: Parsec Void T.Text [Char]
    wikilink = do
        _ <- chunk "[["
        fst <$> manyTill_ letterChar (chunk "]]")

T.putStr $ streamEdit wikilink T.pack "Se mer om [[Stockholm]]"
T.putStr $ streamEdit wikilink T.pack "Se mer om [[Göteborg]]"

Se mer om Stockholm
Se mer om Göteborg
user_1872230
Related