I'm making a web scraper that I using for drop add at my school but I'm having trouble getting it to run on my debian VPS. It works perfectly on my mac but for some reason pywebcopy just doesn't want to work on the VPS. I'm also willing to use something else to do it because I only need the HTML when from what I have seen pywebcopy is built for getting everything from a website. So recomendations on other ways to go about it or a fix to my issue would be a blessing. Also this class is getting ran every second and is intended to be run until I get all of my classes added. On my mac it is being done with python 3.6 but on the vps it is with python 3.7 so that might be another issue.
def getwebpage(crn):
#18139
kwargs = {'project_name': 'site folder'}
url='https://.edu/bprod/bwckschd.p_disp_listcrse?term_in=202101&subj_in=CSC&crse_in=4780&crn_in='+crn
config.setup_config(
# url pf the website
url,
# folder where the copy will be saved
project_folder='rawhtml/'+crn,
**kwargs
)
wp = WebPage()
wp.get(url)
wp.save_html()
dirs = os.listdir("rawhtml/"+crn+"/site folder/.edu/bprod")
file_path = "rawhtml/"+crn+"/site folder/.edu/bprod/"+dirs[0]
return file_path