I'm using the new HEIC format to save a masked image to documents directory. When comparing to the PNG version of the same masked image I see that some new articafts have been added.
Here is a screenshot of the effected image:
Here's a link to working repository showcasing this issue (please view the generated image through the new ios 11 Files app). https://github.com/vondiplo/heic_wrong_encode
The mask provided in the git is an 100x98 px, try to use a bigger resolution image as mask that's the reason why there are artifacts, also keep in mind that PNG is uncompressed while hevc/heif is compressed so you can't really compare them.
Edit
Since a better mask won't resolve your problem you'll have to stick with png, artefacts are an known block compression problem as this article describes http://blog.biamp.com/understanding-video-compression-artifacts/
There is a posible solution to that, but i didn't see documentation from apple how it could be implemented, more details in this link http://www.uta.edu/faculty/krrao/dip/Courses/EE5359/Proposal%20HarshalShah-Interim-deblockingfilter.pdf
