Design - Dynamic data mapping

user_12515377 0
Viewed 348

I am working on an online tool that serves to a number of merchants(e.g. lets say retail merchants).This application takes data from different merchants and provides some data on their retail shop. The solution that I am trying to incorporate here is that any merchant can signup for the tool, send (may be upload through excel or my application can input a json object) their transaction and inventory data and in turn return the result to merchant.

My application consist of domain that is intrinsic to the application and contain all the datapoints that can be used by merchants, e.g

Product { productId, productName, ... }

But the problem that I am facing is that, each merchant will have their own way of representing data, for e.g. merchant x may call product as prod or merchant y may call product as proddt.

Now I would need to way to convert data represented in merchant format to a way that application understand, i.e each time there is a request from merchant x, application should map prod to product e.t.c e.t.c.

Firstly I was thinking of coding these mappers but then this is not a viable solution as I can't really code these mappings for 1000's of merchants that may join my application.

Another solution I was think was to enable the merchant to map a field from their domain to application domain through UI. And then save this somewhere in DB and on each request from merchant first find the mapping from db and then apply it over any incoming request.(Though I am still confused how this can be done).

Does anyone has faced similar design issue before and know of the better way of solving this problem.

architecture algorithm database
4 Answers

if you can find the order of fields then you can easily map data send by your client and you can return result. for example in Excel you client can mention data in this format:

product | name | quantity | cost

condition: your ALL client should send data in this format.

then it will be easy for you to map these field and access then with correct DTO and later save and process data.

user_103403810

I appreciate this "language" concern, and -in fact- multi-lingual applications do it the way you describe. You need to standardize your terminology at your end, so that each term has only one meaning and only one word/term to describe it. You could even use mnemonics for that, e.g. for "favourite product" you use "Fav_Prod" in your app and in your DB. Then, when you present data to you customer, your app looks-up their preferred term for it in a look-up-table, and uses "favourite product" for customer one, and perhaps the admin, and then "favr prod" for customer two, etc...

Look at SQL and DB design, you'll find that this is a form of normalization.

Are you dealing with legacy systems and/or APIs at the customer end? If so, someone will indeed have to type in the data.

If you have 1000s of customers, but there are only 10..50 terms, it may best to let the customer, not you, set the terms.

You might be lucky, and be able to cluster customers together who use similar or close enough terminology. For new customers you could offer them a menu of terms that they can choose from.

user_52501960

If merchants were required to input their mapping with their data, your tool would not require a DB. In JSON, the input could be like the following:

input = {mapping: {ID: "productId", name: "productName"}, data: {productId: 0, productName: ""}}

Then, you could convert data represented in any merchant's format to your tool's format as follows:

ID = input.data[input.mapping.ID]
name = input.data[input.mapping.name]
user_137441780

To reacp:

  • You have an application
  • You want to load client data (merchants in this case) into your application
  • Your clients “own” and manage this data
  • There are N ways in which such client data can be managed, where N <= the number of possible clients
  • You will run out of money and your business will close before you can build support for all N models
  • Unless you are Microsoft, or Amazon, or Facebook, or otherwise have access to significant resources (time, people, money)

This may seem harsh, but it is pragmatic. You should assume NOTHING about how potential clients will be storing their data. Get anything wrong, your product will process and return bad results, and you will lose that client. Unless your clients are using the same data management tools—and possibly even then—their data structures and formats will differ, and could differ significantly.

Based on my not entirely limited experience, I see three possible ways to handle this.

1) Define your own way of modeling data. Require your customers to provide data in this format. Accept that this will limit your customer base.

2) Identify the most likely ways (models) your potential clients will be storing data (e.g. most common existing software systems they might be using for this.) Build import structures, formats to suppor these models. This, too, will limit your customer base.

3) Start with either of the above. Then, as part of your business model, agree to build out your system to support clients who sign up. If you already support their data model, great! If not, you will have to build it out. Maybe you can work the expense of this into what you charge them, maybe not. Your customer base will be limited by how efficiently you can add new and functional means of loading data to your system.

user_74910
Related