Tika Parser behaviour different in REPL and jar

user_17124550 0
Viewed 176

Description

We use the Tika Parser in a wrapped version behind pantomime with no config (default), which uses the AutoDetectParser. It was essential to upgrade the Tika dependency due to missing functionality. After overwriting tika as a dependency of 1.27 along other dependencies, we observed some unexpected behaviour. While running the service in a lein repl, tika correctly converted the document in question (.doc). Afterwards we packed the clj-code with lein uberjar.

Now the fun part began:
A wild bug hunt into the dep(th)s of tika and type casts followed. The issue was found to be a different parsed data type. While in the REPL the parsed data type was a javax.mail.internet.MimeMessage, with the produced .jar the parsed data type was a javax.mail.util.SharedByteArrayInputStream. A log excerpt:

REPL

[2021-10-08 16:56:30,406] TRACE xxx - Unpacking MimeMessage -1
[2021-10-08 16:56:30,407] DEBUG xxx - |> type javax.mail.internet.MimeMessage
[2021-10-08 16:56:30,415] DEBUG xxx - ||> javax.mail.internet.MimeMultipart@397fca9a
[2021-10-08 16:56:30,415] DEBUG xxx - |-> transforming multipart/mixed to Mime for instance javax.mail.internet.MimeMultipart

jar

[2021-10-08 17:02:55,181] TRACE xxx - Unpacking MimeMessage -1
[2021-10-08 17:02:55,183] DEBUG xxx - |> type javax.mail.internet.MimeMessage
[2021-10-08 17:02:55,191] DEBUG xxx - ||> javax.mail.util.SharedByteArrayInputStream@b83be05
[2021-10-08 17:02:55,193] WARN xxx - Mime unpacking unsuccessful: multipart/mixed - javax.mail.util.SharedByteArrayInputStream

Findings

I suspect that there might be a clash of dependencies, as both the tika-parser and the simplejavamail modules use an implementation of MimeMessage, which is provided by both jakarta.mail and javax.mail. I already resolved the dependency clashes hinted by leiningen, but suspect there might be something going on in how lein repl resolves dependencies, which makes it work in that case.
This behaviour was reproduced with a clean profiles.clj.
Logging and stacktrace hint to this function .getContent, which is responsible for returning the appropriate Object.

project.clj

(defproject my-project :lein-v
  :plugins [[lein-parent "0.3.8"]
            [com.roomkey/lein-v "7.2.0"]]
  :parent-project {:path "../../project.clj"
                   :inherit [:managed-dependencies :repositories :manifest :url :prep-tasks]}
  :dependencies [[org.clojure/clojure]
                 [...]
                 [com.novemberain/pantomime "2.11.0" :exclusions 
                   [org.apache.commons/commons-compress
                    org.apache.pdfbox/fontbox
                    org.apache.tika/tika-parsers]]
                 [org.apache.tika/tika-parsers "1.27"]
                 
                 [com.sun.mail/javax.mail "1.6.2"]
                 [net.htmlparser.jericho/jericho-html "3.4"]
                 [org.simplejavamail/simple-java-mail "6.6.1"]
                 [org.simplejavamail/outlook-module "6.6.1"]]
   :profiles {:uberjar {:global-vars        {*warn-on-reflection* true}
                       :aot                :all
                       :omit-source        true
                       :uberjar-exclusions ["project.clj" #"xxx/.*\.clj$"]
                       :jvm-opts           ["-XX:-OmitStackTraceInFastThrow"]
                       :uberjar-name       "service.jar"}}
  :repl-options {:init-ns xxx.init})

affected code

(ns xxx.convert
  (:import (java.time OffsetDateTime ZoneOffset)
           (java.util Properties UUID Base64 Date)
           (java.io InputStream ByteArrayInputStream)
           (javax.activation MimeType)
           (javax.mail.internet MimeMessage MimeMultipart MimeBodyPart InternetAddress MimeUtility)
           (javax.mail Session Message Message$RecipientType)
           (net.htmlparser.jericho Source Renderer)
           (org.apache.commons.io IOUtils)

(defn- unpack-content [^Message message 
                       {:keys [max-text-length max-attachments] :as params}
                       counter]
  (log/debug "|> type" (type message))
  (when (or (not max-attachments)
            (< @counter max-attachments))
    (let [base-type (-> message .getContentType MimeType. .getBaseType string/lower-case)
          content   (.getContent message)] ;; <<< [!] This is where the conversion happens
      (log/debug "||>" (.toString ^MimeMultipart content))
      {:content-type base-type
       :content      (cond
                       (instance? String content)
                       (do
                         (log/trace "|-> transforming" base-type "to String")
                         (swap! counter inc)
                         (->> (if (html? base-type)
                                (render-html content)
                                content)
                              (truncate max-text-length)))

                       (and (instance? InputStream content)
                            (.getFileName message))
                       (let [filename     (MimeUtility/decodeText (.getFileName message))
                             encoded-file (do
                                            (swap! counter inc)
                                            (base64-file content filename))]
                         (log/trace "|-> transforming" base-type "to InputStream")
                         (.close ^InputStream content)
                         encoded-file)

                       (or (instance? MimeMessage content)
                           (instance? MimeMultipart content)
                           (instance? MimeBodyPart content))
                       (do
                         (log/debug "|-> transforming" base-type "to Mime for instance" (type content))
                         (unpack-mime content params counter))

                       :else
                       (do
                         (log/warn "Mime unpacking unsuccessful:" base-type "-" (type content))
                         {}))})))

Unfortunately I can't share the corresponding .doc file. However it is an official document with a common setup for plain mail with tables, images, etc.

Why does it happen and how do I fix this?

apache-tika leiningen dependency-management jvm clojure
2 Answers

Leiningen has a well-known "feature"[1] regarding dependency order. Lein loads each dependency in order, recursively including all transitive dependencies. Imagine you have lein deps like:

[aaa/a "5.0"]
[bbb/b "5.0"]

However, aaa/a has a transitive dependency on bbb/b version 2.1. Then Lein will bring in version 2.1 of bbb/b and ignore the explicit version you asked for.

The only way to combat this is to use a tiered listing of dependencies in project.clj like:

; priority 1 libs
[bbb/b  "5.0"]
[zzz/z  "10.3"]
...

; priority 2 libs
[org.clojure/clojure "1.10.3"]
[aaa/a "5.0"]
...

Which will force Lein to bring in the libs in the priority order you select.

Please try the above and report back with the results. You may also need to print out the Classpath order from within the code to verify you are getting the order and version numbers you expect.

Side note:

I always recommend you add the lein-ancient plugin to your config like:

  :plugins [[com.jakemccrary/lein-test-refresh "0.24.1"]
            [lein-ancient "0.7.0"]
            ]

You can then define the alias:

lanc () {
    echo ""
    echo ""-----------------------------------------------------------------------------
    echo "project.clj:"
    echo ""
    lein ancient check :all
    echo ""
    echo ""-----------------------------------------------------------------------------
    echo "profiles.clj:"
    echo ""
    lein ancient check-profiles :all
    echo ""-----------------------------------------------------------------------------
    echo ""
}

to identify which dependencies need upgrading. Note that both halves are important!

[1] Ancient computer joke:

  Q:  How do you recognize a computer salesman?
  A:  He says, "That's not a bug, that's a feature!"
user_18223790

The way the leiningen uberjar works, is basically by squashing all deps into a single jar file.

This is no problem for most cases, due to clashes in the file structure are very rare as java packages or clojure namespaces separate things. There is of course the odd naming clash and sometimes libraries copy things from different libraries for local adoptions (a good library maintainer will "shadow" those copies so there is no chance for a clash).

That said, there are files in jars, that are way more likely to clash: things under /META-INF/. In your case both javax.mail and jakarta.mail are battling to be the provider for the mail interfaces via META-INF/services/javax.mail.Provider.

If those files are loaded from the classpath there is the best case scenario, that the libraries explicitly search for all such files and have a resolve strategy in case of more than one result. More likely, libraries just let the class loader decide (most likely the first item in CP, that contains the file, but this is of course an implementation detail and you should not rely on it). In that case you most likely just where lucky while developing in your REPL and the correct one got picked up.

Now if the uberjar squashes all those jars, the last file wins. So neither a resolve-strategy helps nor your "lucky first pick". So what can be done:

  • get rid of one of the two deps. Most likely you want to keep the jakarta ones, as the newer name suggests the newer library; this depends on the other libraries you are using, whether they can work with the new version and of course might need changes in your code too.
  • don't roll an uberjar: I don't know of a plugin, that can give you something like the application plugin in gradle. But you can get all the deps with lein with-profile uberjar cp and the non-uberjar artifact. Then basically start with java -cp libs/* my.main.ns
user_31813920
Related