I recently made an online-post-curating service with django. I now upload posts manually everyday but finally realized web-crawler/scraper is highly needed to save my time. In this case, is it better to make a separate django project(for the crawler only)? Or just starting a new app in the existing project is enough?
[Advantage/Ground of making separate project for crawler (I guess)]
- Totally different dependencies needed, so new project
- Mainly asynchronous tasks unlike the existing service
- The function is very universal (not really related to post-curating service)
- Server spec. for crawler must be different, so be prepared in advance with separate project.
[Opposite side - making an app is inough]
- For asynchronous tasks, just adding some library like Celery in your project is enough
- Running multiple projects demands lots and lots of additional effort
- Two projects using one database(saving and reading posts) at the same time can be a threat in some day
Sorry for the tacky question and thank you in advance!