I have this API scrapper in phython to extract information automatically from a web based app.It uses intercepted information from the app to work. I haven't been able to understand the code enough to make the change I want. For this change to be done I believe 3 py files need to be examined that follows>
import json
import os
import time
from progress.bar import Bar
from lib.giustizia import get_case_details
from lib.ranges import load_ids_from_json
SCAN_MODE = False
RATE_LIMIT = 0.2
INITIAL_ID = 49751
FINAL_ID = 49950
RANGE_YEAR = 2021
query_range = {RANGE_YEAR: range(INITIAL_ID, FINAL_ID, 1)}
if os.path.exists("json_results.txt") and not SCAN_MODE:
loaded_json = load_ids_from_json("json_results.txt")
query_range = loaded_json if loaded_json else query_range
"""
json_results = open("json_results.txt", "w+")
csv_results = open("csv_results.csv", "w+")
"""
html_results = open("html_results.html", "w+")
for year in query_range:
print("Querying cases from year {}".format(year))
for process_id in Bar('Querying', suffix='%(percent).1f%% - %(eta)ds').iter(query_range[year]):
case = get_case_details(year, process_id)
if case:
print(" - {}".format(case))
"""
json_results.write(json.dumps(case.asdict()))
csv_results.write(str(case))
"""
html_results.write(str(case))
"""
json_results.write("\n")
csv_results.write("\n")
"""
html_results.write("\n")
"""
json_results.flush()
csv_results.flush()
"""
html_results.flush()
else:
print(process_id, "errored")
time.sleep(RATE_LIMIT) # wait a little to prevent DOS
"""
json_results.close()
csv_results.close()
"""
html_results.close()
The part where it says INITIAL_ID and FINAL_ID it allows you to scan and download the information of certain amount of processes (it's a tribunal app) from a certain year. For example the year 2019 you can scan from process ID 1 to 1000 and it will pick up the text only from the ones you want. What I want to change is instead of scanning by each court case ID to scan by each day of a year (the app can also do this, you click in a day in the calendar in the app and it will display all the court cases of that one day) So in INITIAL ID and FINAL_ID range I want to change it for a date range, picking a certain range of days in a month and in a certain year. Something like INITIAL_ID = 2022/08/08 FINAL_ID = 2022/08/29 (so it will pick the court cases in between those days)
The way the API get the information from the APP to be able to use the ID range in this canse and also the DATE range (that I want to change) can be understood with the next two py files. (as you can see at the begining of the code it needs lib.giustizia and lib.ranges to work)
# -*- coding: utf-8 -*-
import re
from datetime import datetime
import requests
from bs4 import BeautifulSoup
from secrets import (
deviceheight, devicename, devicewidth, os_version, token,
user_agent, uuid
)
QUERY_URL = "https://mob1.processotelematico.giustizia.it/proxy/index_mobile.php"
BASE_PAYLOAD = dict(
version="2.0.15",
platform=os_version,
uuid=uuid,
devicename=devicename,
devicewidth=devicewidth,
deviceheight=deviceheight,
token=token,
azione="direttarg_sicid_mobile",
registro="CC",
idufficio="0580910098",
tipoufficio=1
)
HEADERS = {
'User-Agent': user_agent
}
RE_INSCRITO_RUOLO = re.compile("\<li\>iscritto al ruolo il (.+)\<\/li\>")
RE_REMOVE_LAWYER_PREFIX = re.compile("(?<=Avv. ).*")
class Case:
def __init__(self, case_yr, case_no, date_filed, judge_name, date_hearing, case_state, primary_lawyer_initials,
raw_case_content=None, judgement_number=None):
self.year = case_yr
self.number = case_no
self.date_filed = date_filed
self.date_hearing = date_hearing
self.judge_name = judge_name
self.case_state = case_state
self.primary_lawyer_initials = primary_lawyer_initials
self.raw_case_content = raw_case_content
self.judgement_number = judgement_number
def __str__(self):
return ";".join([
'{}/{}'.format(self.number, self.year),
self.raw_case_content
])
def asdict(self):
return {
'raw_case_content': self.raw_case_content,
}
def get_case_details(case_yr, case_no):
payload = BASE_PAYLOAD.copy()
payload.update(dict(
aaproc=str(case_yr),
numproc=str(case_no),
_=int(datetime.now().timestamp())
))
This part of the second file is what might be needed to be changed as well. At the begining you can see the API basically acessess that app web adress and fills it with some phone information (my phone in that case) which I have in another file and are necessary to connect to the webpage where there is the information to be picked by the API.
Now where it says aaproc=str(case_yr), at the last lines of the code, that numproc is how the APP names the ID numbers that the first file uses in the RANGE to scan and retrieve the information. aaproc is how it names the year used in the range. Now "dal" is what the app calls the DATE I want to use in place of the ID RANGE (such information is known by intercepting the data from the app)... so I believe maybe a line of code with "dal" like dal=str(date_filed) could be added in that part of the code to do what I want.
Now there is the ranges.py file>
import json
def load_ids_from_json(filename):
f = open(filename)
json_objs = f.read().split("\n")
f.close()
sets = {}
out = {}
for oj in json_objs:
try:
o = json.loads(oj)
except:
continue
if not sets.get(o['case_yr']):
sets[o['case_yr']] = set()
sets[o['case_yr']].add(int(o['case_no']))
for year in sets:
out[year] = list(sets[year])
out[year].sort()
return out
Here again where it says 'case_no' this is again about the ID numbers used in the ID RANGE as you can see in the giustizia.py file, and 'case_yr'is the year... so again maybe something should be added like 'date_filed' that following the previous file code would be the "dal" equivalent to the DATE that I want to use as a RANGE instead of the IDs at the beginning of the first file code.
These are the parts I grasped that I think might be necessary to understand to do the change, but still I don't know how to go about that so I really apreciate if someone could help! Thank you so much.