Memory mapped file is an efficient way for using regex or doing manipulation on large binary files.
In case I have a large text file (~1GB), is it possible to work with an encoding-aware mapped file?
Regex like [\u1234-\u5678] won't work on bytes objects and converting the pattern to unicode will not work either (as "[\u1234-\u5678]".encode("utf-32") for example will not understand the range correctly).
Searching might work if I convert the search pattern from str to bytes using .encode() but it's still somewhat limited and there should be a simpler way instead of decoding and encoding all day.
I have tried wrapping it with io.TextIOWrapper inside an io.BufferedRandom but to no avail:
AttributeError: 'mmap.mmap' object has no attribute 'seekable'
Creating a wrapper (using inheritance) and setting the methods seekable, readable and writable to return True did not work either.
Regarding encoding, a fixed length encoding like utf-32, code-points or the lower BMP of utf-16 (if it's even possible referring just to that part) might be assumed.
Solution is welcome for any python version.