Json data Split in train and test

user_7996402 0
Viewed 1861

I am trying to fit CNN for huffpost news dataset https://www.kaggle.com/rmisra/news-category-dataset. The dataset i am using is in json format. my data format is this

[{"category": "CRIME", "headline": "There Were 2 Mass Shootings In Texas Last Week, But Only 1 On TV", "authors": "Melissa Jeltsen", "link": "https://www.huffingtonpost.com/entry/texas-amanda-painter-mass-shooting_us_5b081ab4e4b0802d69caad89", "short_description": "She left her husband. He killed their children. Just another day in America.", "date": "2018-05-26"} , {"category": "ENTERTAINMENT", "headline": "Will Smith Joins Diplo And Nicky Jam For The 2018 World Cup's Official Song", "authors": "Andy McDonald", "link": "https://www.huffingtonpost.com/entry/will-smith-joins-diplo-and-nicky-jam-for-the-official-2018-world-cup-song_us_5b09726fe4b0fdb2aa541201", "short_description": "Of course it has a song.", "date": "2018-05-26"} ]

And here is the code which i am trying the code source is https://www.kaggle.com/kredy10/simple-lstm-for-text-classification I want to fit LSTM on this data

import pandas as pd
import json
with open('News_Category_Dataset_v2.json', 'r') as f:
    train = json.load(f)

Now i want to spilt train and test data but i dont know how to use arrays their to split data.. can any one help?

X_train,X_test,Y_train,Y_test = train_test_split(X,Y,test_size=0.15)
lstm machine-learning python json
1 Answers

Here is how I did it: I first used train_test_split to get train (70%) and test (30%) set, and then I used the same command on test to get test (50%) and validation (50%) set.

from sklearn.model_selection import train_test_split
   
with open('file_name') as f:
    lines = f.readlines()
    
train, test = train_test_split(lines, test_size=0.3)
val, test = train_test_split(test, test_size=0.5)

Hope this helps!

user_98134640
Related