I am working with about 1000 XML files. I have written a script where the program loops through the folder containing these XML files and I have achieved the following:
- Created a list with all the paths of the XML files
- Read the files and extract the values I need to work with.
- I have a new dataframe which consists of the only two columns I need to work with.
Here is the full code :
import glob
import pandas as pd
# Empty list to store path of xml files
path_list = []
# Function to iterate folder and store path of xml files.
# Can be modified to take the path as an argument via command line if required
time_sum = []
testcase = []
def calc_time(path):
for path in glob.iglob(f'{path}/*.xml'):
path_list.append(path)
try:
for file in path_list:
xml_df = pd.read_xml(file, xpath=".//testcase")
# Get the classname values from the XML file
testcase_v = xml_df.at[0, 'classname']
testcase.append(testcase_v)
# Get the aggregate time value of all instances of the classname
time_sum_test = xml_df['time'].sum()
time_sum.append(time_sum_test)
new_df = pd.DataFrame({'testcase': testcase, 'time': time_sum})
except Exception as ex:
msg_template = "An exception of type {0} occurred. Arguments:\n{1!r}"
message = msg_template.format(type(ex).__name__, ex.args)
print(message)
calc_time('assignment-1/data')
Now I need to group these values on the following condition.
Equally distribute classname by their time into 5 groups, so that, total time for each group will approximately same.
The new_df looks like this:
'TestMensaSynthesis': 0.49499999999999994,
'SyncVehiclesTest': 0.303,
'CallsPromotionEligibilityTask': 3.722,
'TestSambaSafetyMvrOverCustomer': 8.546,
'TestScheduledRentalPricingEstimateAPI': 1.6360000000000001,
'TestBulkImportWithHWRegistration': 0.7819999999999999,
'calendars.tests.test_intervals.TestTimeInterval': 0.006,
The dataframe has more than 1000 lines containing the classname and time.
I need to add a groupby statement which will make 5 groups of these classes and the total time of these groups will be approximately equal to each other.