Data flow and Memory problem in Typescript Promise loop

user_12566711 0
Viewed 59

I am working on a project where I have to read around 3000+ JSON files from an Azure DevOps repository and store their contents in a string which I can use later.

Below is the method I am using to fetch the list of items in my repository. I am not allowed to presume any directory hierarchy so I am setting the recursionLevel parameter to full.

Link to the doc:https://docs.microsoft.com/en-us/javascript/api/azure-devops-extension-api/gitrestclient#getitems-string--string--string--versioncontrolrecursiontype--boolean--boolean--boolean--boolean--gitversiondescriptor-

function getItems(repositoryId: string, project?: string, scopePath?: string, recursionLevel?: VersionControlRecursionType, includeContentMetadata?: boolean, latestProcessedChange?: boolean, download?: boolean, includeLinks?: boolean, versionDescriptor?: GitVersionDescriptor)

The above function returns a Promise of Git item list from where I can get the relative path of every file from the root of the repository

After getting a list of items I want to read them one by one and store their contents in a string. For that I am using this method:

function getItemText(repositoryId: string, path: string, project?: string, scopePath?: string, recursionLevel?: VersionControlRecursionType, includeContentMetadata?: boolean, latestProcessedChange?: boolean, download?: boolean, versionDescriptor?: GitVersionDescriptor, includeContent?: boolean, resolveLfs?: boolean)

This function is also available in the same link I shared earlier. You have to scroll down a bit. It returns the content as Promise of string.

My code to read all the files:

AdoClient.getInstance().getItems(this.repoID, this.projectName, this.commonData.inputBranchName).then(data=>{
            data.forEach(element => {
                if(element.gitObjectType==3)
                {
                   AdoClient.getInstance().getItemText(this.repoID,element.path,this.commonData.inputBranchName).then(element=>{ content.concat(element);
});})});

In the above code:

  1. gitObjectType=3 means a file
  2. The functions getItems() and getItemText() that you see are actually my functions. I wrote a definition to just pass the 3 required parameters. In the AdoClient file I have the original call to the functions using GitRestClient object just as you have in the doc.

This is the originial function call in AdoClient file:

public async getItemText(repoId: string, filePath: string, branchName: string) {
        return this.gitClient.then(client => client.getItemText(repoId, filePath, null, null, 'none', false, false, false, {version: branchName, versionOptions: 0, versionType: 0}));
    }

public async getItems(repositoryId: string, project: string,branch: string){
        return this.gitClient.then(client=>client.getItems(repositoryId, project, "/ServiceGroupRoot/", 'full', true, false, false, true, {version: branch, versionOptions: 0, versionType: 0}));
    }

Also please note /ServiceGroupRoot/ is the root directory in my repository.

When I am doing this I am not being able to get all the contents inside the content variable and also Google chrome is showing ERR_INSUFFICIENT_RESOURCES due to 3000+ ajax calls at the same time. This error is not coming in Mozilla tho. But in Mozilla when I am printing the content it shows empty string. What I want is a way to order my execution steps so that I can have all the contents in my content string. Is there a way to run a loop of Promise calls and make them execute one by one? By one by one I mean finish one then start the next one and not load everything at once or is there some other approach I have to follow?

promise azure-devops-rest-api typescript javascript
2 Answers

I guess the easiest way to go is by just using a for loop with await:

const work = i => {
  return new Promise(resolve => {
    setTimeout(() => resolve(i), Math.floor(Math.random() * 100));
  });
};

const array = Array(10).fill(0).map((_, i) => i + 1);

// non sequencial
const test_then = () => {
  array.forEach(i => {
    work(i).then(i => {
      console.log(i);
    });
  });
};

// sequencial
const test_await = async () => {
  for (let i = 0; i < array.length; i++) {
    console.log(await work(i));
  }
};

test_then();
test_await();
user_4514920

There a few critical mistakes in code.

Suggestion to use the traditional for loop so that we can use async await syntax.

I don't know about resources part, but I have pasted a clear code below.

async function main() {
let content = '';
const data = await AdoClient.getInstance().getItems(/* pass params here*/);
for (let i = 0; i < data.length; i++) {
  if(data[i].gitObjectType==3){
   const result = await AdoClient.getInstance().getItemText(/* pass params here*/); 
   content.concat(result);
  }
}

return content;
} // main close


const c = main();
console.log(c);// your output
user_125386180
Related