Creating Dummy Variable with 200k unique value

Indeed performing this operation takes lots of RAM.

As far as programming solutions I can think of:

• Dimensionality reduction: if somehow there is some relation between your 200K categories and those can be reduced (eg hierarchy levels for those categories so you can group up categories and perform analyses by levels, eg lvl1 = 10 cateogries, lvl2 = 100 etc...). May I ask: what type of data do you have which contains 200K unique category values?
• Splitting up dataset and combining results: I have the below working with numpy. You end up with smaller subsets, each encoded for the 200K categories (even if certain categories aren't present in a subset). Then you need to decide how to further process those subsets.

Somehow import statements were breaking formatting so I have them separate here:

import numpy as np
import random

And the rest of the code:

def np_one_hot_encode(n_categories: int, arr: np.array):
    # Performs one-hot encoding of arr based on n_categories
    # Allows encoding smaller chuncks of a bigger array
    # even if the chunks do not contain 1 occurrence of each category
    # while still producing n_categories columns for each chunks
    result = np.zeros((arr.size, n_categories))
    result[np.arange(arr.size), arr] = 1
    return result


# Testing our encoding function
# even if our input array doesn't contain all categories
# the output does cater for all categories
encoded = np_one_hot_encode(3, np.array([1, 0]))
print('test np_one_hot_encode\n', encoded)
assert np.array_equal(encoded, np.array([[0, 1, 0], [1, 0, 0]]))

# Generating 500K rows with 200K unique categories present at least once
total = int(5e5)
nunique = int(2e5)
uniques = list(range(0, nunique))
random.shuffle(uniques)
values = uniques+(uniques*2)[:total-nunique]
print('Rows count', len(values))
print('Uniques count', len(list(set(values))))

# Produces subsets of the data in (~500K/50 x nuniques) shape:
n_chunks = 50
for i, chunk in enumerate(np.array_split(values, n_chunks)):
    print('chunk', i, 'shape', chunk.shape)
    encoded = np_one_hot_encode(nunique, chunk)
    print('encoded', encoded.shape)

And the output:

test np_one_hot_encode
[[0. 1. 0.]
[1. 0. 0.]]
Rows count 500000
Uniques count 200000
chunk 0 shape (10000,)
encoded (10000, 200000)
chunk 1 shape (10000,)
encoded (10000, 200000)

• Distributed processing with tools like Dask, Spark etc... so you can handle processing of the subsets

• Database: other solution I can think of is normalize your model into a database (either relational or "big" flat data model) where you could leverage indices to filter and process part of the data only (only certain rows and certain categories), thus allowing you to handle a smaller output in memory

But in the end there is no magic, if ultimately you're tyring to load a N-M matrix into memory with N=500K and M=200K, it will take the RAM it needs to take, there is no way around that, thus the most likely gains to be had are dimensionality reduction OR a different approach to data processing altogether (eg distributed computing).