Large data set for web application - use an API for each query or store locally on database?

user_14721167 0
Viewed 301

I'm building an application which uses a publicly available set of data that is rather large. I have two options to query it:

  1. Via an API. For each query, my application would send a request using this dataset's API.

  2. Alternatively, I could download (downloading the CSV files take over 4.0GB) and store the entire dataset locally.

The type of operations and analysis that I'd like to perform on the data for my web application is easily done with either method. However I'm wondering which way is best and why?

The only thing I can think of is that querying a local database would be faster, however using the API would ensure the data is up-to-date ("valid" data in this dataset is said to expire after 10 years according to the organisation's website).

csv rest query-optimization database performance
1 Answers

As you said both options are valid and it depends on your use case which option is better.

Consider the following questions:

How often is the data updated? Is it maybe completely historical data and will never be updated, or only new values will added but existing never change? How much effort would it be to update your locally stored data automatically.

How time critical is the response time and availability? Locally stored data makes you independent against network delay to the API, an outage of the API, a rate limit that the Service provider could implement to throttle the rate of requests, or taking the data offline. How much data is requested on average, what is the response time for the API?

user_15150520
Related