How to write Exif to a JPEG with TwelveMonkey's ExifWriter class

Im using the TwelveMonkey's lib to read Exif data from jpeg like:

       try (ImageInputStream stream = ImageIO.createImageInputStream(input)) {
            List<JPEGSegment> exifSegment = JPEGSegmentUtil.readSegments(stream, JPEG.APP1, "Exif");
            InputStream exifData = exifSegment.get(0).data();
            exifData.read(); // Skip 0-pad for Exif in JFIF
            try (ImageInputStream exifStream = ImageIO.createImageInputStream(exifData)) {
                return new EXIFReader().read(exifStream);
            }
       }

therefore I have a CompoundDirectory with a bunch of Entry elements. But how do I use the ExifWriter to a jpeg. Using it to write to the outputstream just corrupts the jpeg (image viewers think it is a broken tiff).

Update: What I like to achieve is reading a jpeg to a BufferedImage, also reading exif data, scaling it and then compressing it to jpeg again retaining the exif data (ie. writing the previously read data to the scaled out jpeg). For this I currently use some verbose version of ImageIO methods. Here is the basic code to do this currently: https://gist.github.com/patrickfav/5a51566f31c472d02884 (exif reader seems to work, writer not of course)