A python file for performing data science loads approximately 200-300 MB into the memory. Then it runs several scikit-learn methods, in particular linear_model.LogisticRegression on it. The sample size is 60,000, the feature dimension is about 100. My PC has 16 GB RAM. I have the latest stable Ubuntu LTS. When executing the program, I monitor the RAM usage via free -m -s 5, so the numbers are to be updated every 5 seconds. When the data have been loaded by the program, the RAM usage is at 2 GB, there are still 14 GB shown to be free. However, the program freezes when executing one of the fit methods.
Freezing means: the fan is getting louder, the program does no more write into .csv files, the mouse does no more react, nor the keyboard - however, the memory monitor in the second command line window still operates and updates every five seconds. I was only able to react via (violent) reboot. What can be the reason, what the remedy? Perhaps linear_model.LogisticRegression cannot handle large sample sizes? In fact I tried a sample size of 1,000, and it worked without problem.
X_test, Yp_test, Yd_test = preprocess_data(sort='test',d=2)
X_train,Yp_train,Yd_train = preprocess_data(sort='train',d=2)
with open('results_d={}.csv'.format(d), 'w', newline='') as csvfile:
writer = csv.writer(csvfile)
writer.writerow('Format: Method; example (left=prediction, right=true value); score for (training) data fit; score for test data fit (generalisation).')
# RIDGE CLASSIFIER
writer.writerow('Ridge regression')
print('Ridge regression')
ridge = linear_model.RidgeClassifier()
print('#')
ridge.fit(X_train,Yd_train)
print('#')
sc_fit = ridge.score(X_train,Yd_train)
print('#')
sc_gen = ridge.score(X_test,Yd_test)
print('#')
writer.writerow('Example (left=prediction, right=true value):')
writer.writerows(np.transpose(np.array([ridge.predict(X_test[:50]),Yd_test[:50]])))
writer.writerow("Mean accuracy = "+str(sc_fit)+str(sc_gen))
# LOGISTIC REGRESSION
writer.writerow('Log. regression')
print('Log. regression')
logreg = linear_model.LogisticRegression()
print('#')
logreg.fit(X_train,Yd_train)
print('#')
sc_fit = logreg.score(X_train,Yd_train)
print('#')
sc_gen = logreg.score(X_test,Yd_test)
print('#')
writer.writerow('Example (left=prediction, right=true value):')
writer.writerows(np.transpose(np.array([logreg.predict(X_test[:50]),Yd_test[:50]])))
writer.writerow("Mean accuracy = "+str(sc_fit)+str(sc_gen))
EDIT: ulimit -a gives
<pre>core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 63526
max locked memory (kbytes, -l) 65536
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 63526
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
</pre>