Implement pandas groupby method on a dataframe with certain conditions

Viewed 37

I am working with about 1000 XML files. I have written a script where the program loops through the folder containing these XML files and I have achieved the following:

  1. Created a list with all the paths of the XML files
  2. Read the files and extract the values I need to work with.
  3. I have a new dataframe which consists of the only two columns I need to work with.

Here is the full code :

import glob
import pandas as pd

# Empty list to store path of xml files

path_list = []

# Function to iterate folder and store path of xml files.
# Can be modified to take the path as an argument via command line if required
time_sum = []
testcase = []


def calc_time(path):
    for path in glob.iglob(f'{path}/*.xml'):
        path_list.append(path)

    try:
        for file in path_list:
            xml_df = pd.read_xml(file, xpath=".//testcase")
            # Get the classname values from the XML file
            testcase_v = xml_df.at[0, 'classname']
            testcase.append(testcase_v)
            # Get the aggregate time value of all instances of the classname
            time_sum_test = xml_df['time'].sum()
            time_sum.append(time_sum_test)

           
            new_df = pd.DataFrame({'testcase': testcase, 'time': time_sum})

    except Exception as ex:
        msg_template = "An exception of type {0} occurred. Arguments:\n{1!r}"
        message = msg_template.format(type(ex).__name__, ex.args)
        print(message)

calc_time('assignment-1/data')

Now I need to group these values on the following condition.

Equally distribute classname by their time into 5 groups, so that, total time for each group will approximately same.

The new_df looks like this:

  'TestMensaSynthesis': 0.49499999999999994,
  'SyncVehiclesTest': 0.303,
  'CallsPromotionEligibilityTask': 3.722,
  'TestSambaSafetyMvrOverCustomer': 8.546,
  'TestScheduledRentalPricingEstimateAPI': 1.6360000000000001,
  'TestBulkImportWithHWRegistration': 0.7819999999999999,
  'calendars.tests.test_intervals.TestTimeInterval': 0.006,

The dataframe has more than 1000 lines containing the classname and time.

I need to add a groupby statement which will make 5 groups of these classes and the total time of these groups will be approximately equal to each other.

0 Answers
Related