Deploy Scrapy Project with Streamlit

user_13857600 0
Viewed 164

I have a scrapy spider that scrapes products information from amazon based on the product link.

I want to deploy this project with streamlit and take the product link as web input, and product information as output data on the web.

I don't know alot about deployment, so anyone can help me with that.

streamlit scrapy web-scraping python
2 Answers

You can create a public repository on GitHub with streamlit and connect your account with 0auth. Then you can deploy it on the streamlit servers after signing in the streamlit website.

user_128884510

You can run scrapy from a script using scrapy.crawler.CrawlerProcess module

basically, you can run the spider and export the data temporarily and use it in your streamli app -

import scrapy
from scrapy.crawler import CrawlerProcess

class MySpider(scrapy.Spider):
    # Your spider definition
    ...

process = CrawlerProcess(settings={
    "FEEDS": {
        "items.json": {"format": "json"},
    },
})

process.crawl(MySpider)
process.start() # the script will block here until the crawling is finished

Now you can save this script and run using subprocess which will export the data into items.json. Use it in your app.

Here is a helpful streamlit cloud scrapy thread with public streamlit-scrapy project github repo

user_111053560
Related