Dataset splitting as train/test/valid at C#

user_12102649 0
Viewed 116

I have a data as Dictionary<string,List<string>>. Key of dictionary includes class_name, Values of dictionary includes images. After all, I generate this dictionary as dataset in my local pc. However, I must split all data in train(%80)/test(%10)/valid(%10) in C# code. For example, I have selected 90 images and I have 3 class, every class has 30 images. After splitting, Train set class must have 24 image, test and val sets 3 images one by one. How can i do that, anyone help ? I share a png for help to understand.enter image description here

artificial-intelligence dataset split c#
2 Answers

This function will split your dictionary into 3. You can modify it by changing loops condition. It will works if count of List is greater than 1 (because of using Math.Ceiling function)

(Dictionary<string, List<string>> trainData, Dictionary<string, List<string>> testData, Dictionary<string, List<string>> validData) SplitData(Dictionary<string, List<string>> data)
{
    var random = new Random();

    var train = new Dictionary<string, List<string>>();
    var test = new Dictionary<string, List<string>>();
    var valid = new Dictionary<string, List<string>>();

    foreach (KeyValuePair<string, List<string>> dataClass in data)
    {
        train[dataClass.Key] = dataClass.Value.ToList();
        test[dataClass.Key] = new List<string>();
        valid[dataClass.Key] = new List<string>();

        for (int i = 0; i < Math.Ceiling(0.1 * dataClass.Value.Count); i++)
        {
            int idx = random.Next(train[dataClass.Key].Count);
            test[dataClass.Key].Add(train[dataClass.Key][idx]);
            train[dataClass.Key].RemoveAt(idx);
        }

        for (int i = 0; i < Math.Ceiling(0.1 * dataClass.Value.Count); i++)
        {
            int idx = random.Next(train[dataClass.Key].Count);
            valid[dataClass.Key].Add(train[dataClass.Key][idx]);
            train[dataClass.Key].RemoveAt(idx);
        }
    }

    return (train, test, valid);
}
user_131176870

I didn't fully grasp the structure you have, but suppose your Dictionary has 90 entries

var train = new List<KeyValuePair<string,List<string>>>();
var test = new List<KeyValuePair<string,List<string>>>();
var valid = new List<KeyValuePair<string,List<string>>>();

int i = 0;
foreach(var kvp in dictionary90){
  var x = i++ % 10;
  if(x<8)
    train.Add(kvp);
  else if(x<9)
    test.Add(kvp);
  else
    valid.Add(kvp);
}

It'll spread your dictionary entries into 8:1:1 ratio across the lists

user_14106640
Related