How to manage additional processed data in MarkLogic

user_5052507 0
Viewed 68

MarkLogic 9.0.8.2

We have around 20M records in MarkLogic.

For one of the business requirement, we need to generate additional data for each xml and then need user will search this data.

As we can't change original document, so need input on what is best way to manage additional data. Following are the few which we have thought of

  1. Create separate collection and store additional data in separate xml with same unique number i.e. same as original xml. So when user search for it, search in this collection and then retrieved original documents and send response back.
  2. Store additional data in original document properties

We also need to create element range index to make sure it works when end user provide data in range operators.

<abc>
        <xyz>
          <quan>qty1</quan>
          <value1>1.01325E+05</value1>
          <unit>Pa</unit>
        </xyz>
        <xyz>
          <quan>qty2</quan>
          <value1>9.73E+02</value1>
          <value2>1.373E+03</value2>
          <unit>K</unit>
        </xyz>
        <xyz>
          <quan>qty3</quan>
          <value1>1.8E+03</value1>
          <unit>s</unit>
        </xyz>
        <xyz>
          <quan>qty4</quan>
          <value1>3.6E+03</value1>
          <unit>s</unit>
        </xyz>
</abc>

We need to process data from value1 element. User will then search for something like

  • qty1 >= minvalue AND qty1<=maxvalue
  • qty2 >= minvalue AND qty2<=maxvalue
  • qty3 >= minvalue AND qty3<=maxvalue

So when user will search for qty1 then it should only get data from element where value is qty1 and so on. So would like to know

  • What is best approach to store data like this
  • What kind of index i should create to implement this
data-processing marklogic indexing
2 Answers

I would recommend wrapping the original data in an envelope, which allows adding extra data in the header. It could also allow creating a canonical view on the relevant pieces of the data, and either store that as instance, and original as 'attachment' (sub-property, not an attached binary), or keep the instance as-is, and put canonical values for indexing in the header.

There is a lengthy blog article about the topic, that discusses pros and cons in high detail: https://www.marklogic.com/blog/envelope-design-pattern/

HTH!

user_9184960

Grtjn's answer would be the recommended solution, as it is more performant to keep all the information inside the document itself, versus having to query across both the document with the properties, but it would require changes to the document.

Option 1 & 2 could both work.

Properties documents already exist, so it doesn't add fragments, but the properties must conform to the schema.

Creating a sidecar document provides more flexibility, because you are creating new documents, it will increase number of fragments.

user_16308930
Related