How to split json dataset and save it?

user_15300386 0
Viewed 211

I have taken a json dataset. Dataset name is v2_OpenEnded_mscoco_train2014_questions.json How can I split some of the data from the dataset and save the split data into another json file?

This is the example of my dataset:

{"image_id": 426004, "question": "How many buns are on the plate?", "question_id": 426004002}, {"image_id": 92846, "question": "What is the color of the vase without flowers?", "question_id": 92846000}, {"image_id": 92846, "question": "Is there anything red in this photo?", "question_id": 92846002}, {"image_id": 92846, "question": "What does that vase represent?", "question_id": 92846003}, {"image_id": 262166, "question": "What color is the couch?", "question_id": 262166002}, {"image_id": 262166, "question": "How many seats are available?", "question_id": 262166003}

I have about 443,757 data in my dataset. I want to split the dataset into 400 different dataset each having 100 data. How can I automate this using python? It will be a great help if this can be done with pytorch.

pytorch python json split
1 Answers

Try this:

n_rows = 100
current_data = []
for i, e in enumerate(data):
    if i % n_rows == 0 and i > 0:
        with open(f'dataset_{i - n_rows}-{i}.json', 'w') as f:
            json.dump(current_data, f)
        current_data = []
    
    current_data.append(e)

data is a list with jsons which you have. We iterate over it and every n_rows lines are written in a new file. The last several rows aren't written.

user_67972500
Related