Checking if multiple strings exist in multiple files

Question

I have a translations file which is a JSON object, which may contain more JSON objects and I need to find missing translations. JSON file ex:

export const locale = {
    lang: 'en',
    data: {
        'NAV': {
            'APPLICATIONS': 'Applications',
            'DASHBOARDS'  : 'Dashboards',
            'CALENDAR'    : 'Calendar',
            'ECOMMERCE'   : 'E-Commerce',
            'ACADEMY'     : 'Academy',
            'MAIL'        : {
                'TITLE': 'Mail',
                'BADGE': '25'
            },
            'MAIL_NGRX'        : {
                'TITLE': 'Mail Ngrx',
                'BADGE': '13'
            },
            'CHAT'        : 'Chat',
            'FILE_MANAGER': 'File Manager',
            'CONTACTS'    : 'Contacts',
            'TODO'        : 'To-Do',
            'SCRUMBOARD'  : 'Scrumboard'
        }
    }
};

and I need to get each translation, which is the lowest level item (ex data.NAV.MAIL.BADGE or data.NAV.APPLICATIONS)and see if it exists in the files directory for my project.

Is it inefficient to open a ton of files, or is that difference negligible as compared to going translation by translation and opening each file (for potentially thousands of files)?

I was thinking of flattening the object recursively so I get one flat object with all keys, getting each file, and checking each string if it exists. If it does, I move it to a different object of found strings so it's not checked again. Whatever is left in original object is not used. I will also add whatever is in the git ignore to be ignored here as it is not relevant.

I think recursively going in each file that is in the directory that isn't ignored and then recursively checking if the key is in the file, and then going into the next key if it exists and checking may possibly be better?

Is there anything to make this more efficient that I may be missing?

user_1243641 · Answer

Unless you have so much data that it won't fit into memory (which I really doubt for translation files, but who knows?), then I think you're best off collecting the data from all the files into one array and processing that array once they're all loaded.

Processing it can be fairly easy. Here is an implementation:

const uniq = (xs) => [... new Set (xs)]

const getLeafPaths = (obj) =>
  Object (obj) === obj
    ? Object .entries (obj) .flatMap (
        ([k, v]) => getLeafPaths (v) .map (p => k + (p.length ? '.' + p : ''))
      )
    : ['']

const missing = (
  locales,
  localeKeys = locales .map (locale => [locale .lang, getLeafPaths (locale .data)]),
  allKeys = uniq (localeKeys .flatMap (([, keys]) => keys)) .sort()
) => ({
  allKeys,
  missing: Object .fromEntries (
    localeKeys .map (([lang, keys]) => [lang, allKeys .filter (k => ! keys .includes (k))])
  )
})

const locales = [{lang: "en", data: {NAV: {APPLICATIONS: "Applications", DASHBOARDS: "Dashboards", ECOMMERCE: "E-Commerce", ACADEMY: "Academy", MAIL: {BADGE: "25"}, MAIL_NGRX: {TITLE: "Mail Ngrx"}, CHAT: "Chat", FILE_MANAGER: "File Manager", TODO: "To-Do", SCRUMBOARD: "Scrumboard"}}}, {lang: "es", data: {NAV: {APPLICATIONS: "Aplicaciones", CALENDAR: "Calendario", ECOMMERCE: "Comercio electrónico", MAIL: {TITLE: "Mail", BADGE: "26"}, MAIL_NGRX: {BADGE: "14"}, CHAT: "Chat", FILE_MANAGER: "Administradora de archivos", TODO: "Hacer", SCRUMBOARD: "Tablero de melé"}}}, {lang: "ru", data: {NAV: {DASHBOARDS: "Дашборды", CALENDAR: "Календарь", ACADEMY: "Академия", MAIL: {TITLE: "Почта"}, MAIL_NGRX: {TITLE: "Почта Ngrx", BADGE: "15"}, FILE_MANAGER: "Файловый менеджер", CONTACTS: "Контакты", TODO: "Делать"}}}]

console .log (missing (locales))

.as-console-wrapper {max-height: 100% !important; top: 0}

The helper function uniq simply finds the collection of unique elements in an array.

getLeafPaths is more interesting. It finds the paths, as '.'-separated property names of all the leaf nodes in an object. For the data property of your sample object, it returns

[
  "NAV.APPLICATIONS", "NAV.DASHBOARDS", "NAV.ECOMMERCE", "NAV.ACADEMY", 
  "NAV.MAIL.BADGE", "NAV.MAIL_NGRX.TITLE", "NAV.CHAT", "NAV.FILE_MANAGER", 
  "NAV.TODO", "NAV.SCRUMBOARD"
]

The main function, missing gets the paths for every locale object on the list, combines the keys into a master list, then returns an object with the master list and a listing, per language, of every key from the master list not found for each language. The output looks like this:

{
  allKeys: [
    "NAV.ACADEMY", "NAV.APPLICATIONS", "NAV.CALENDAR", "NAV.CHAT",
    "NAV.CONTACTS", "NAV.DASHBOARDS", "NAV.ECOMMERCE", "NAV.FILE_MANAGER",
    "NAV.MAIL.BADGE", "NAV.MAIL.TITLE", "NAV.MAIL_NGRX.BADGE",
    "NAV.MAIL_NGRX.TITLE", "NAV.SCRUMBOARD","NAV.TODO"
  ],
  missing: {
    en: ["NAV.CALENDAR", "NAV.CONTACTS", "NAV.MAIL.TITLE", "NAV.MAIL_NGRX.BADGE"],
    es: ["NAV.ACADEMY", "NAV.CONTACTS", "NAV.DASHBOARDS", "NAV.MAIL_NGRX.TITLE"],
    ru: ["NAV.APPLICATIONS", "NAV.CHAT", "NAV.ECOMMERCE", "NAV.MAIL.BADGE", "NAV.SCRUMBOARD"]
  }
}

If you already have the master list, this could be simplified to simply pass it into missing rather than generating it.

Checking if multiple strings exist in multiple files

1 Answers