Efficient way to share node_modules between different Azure Pipeline jobs

user_581814 0
Viewed 1297

My project is a node project using yarn to manage npm dependencies.

In my CI azure pipeline, I have multiple jobs, all of which require running yarn to install npm dependencies. Because the number of dependencies is quite large, it takes almost 4 minutes to finish installing them.

I have tried using pipeline artifact to share the node_modules folder between them, but it took the same amount of time (or even slower) to publish and download the artifact between jobs.

I wonder if there's an efficient way to share large data between pipeline jobs as in this case?

azure-pipelines-yaml azure-pipelines
2 Answers

The easiest way is to run the tasks in one job. Or you can choose to run the jobs in same self-hosted agent, so that you don't need to install the duplicate dependencies in same environment.

If you have specific reason to use multiple jobs, you can also consider pipeline caching. Similar examples: link1,link2.

user_109104500

You didn't specify the type of agents you are using Self-Hosted or MS-Hosted. If you're not against using Self-Hosted agents I would recommend creating a agent with the necessary NPM dependencies already available. You can do this with a traditional VM or Docker. Other than this I believe the only alternatives for you would be to either run all tasks requiring these dependencies in the same job, or run your jobs in parallel to make up for the lost time installing the NPM dependencies. The parallel job option won't be available to you if you are using the free version of MS-Hosted agents only if you are paying for them.

user_40139750
Related