I have an issue while making predictions on pandas dataframe:
model = joblib.load(f"models/{MODEL}_{FOLD}.pkl")
preds = model.predict(df)
joblib.dump(preds, f"models/predictions_{FOLD}.pkl")
In loop of 5 FOLDS, 2/5 iterations are successful and after start of the 3rd I get this:
Parallel(n_jobs=8): Using backend ThreadingBackend with 8 concurrent workers.
Killed
And program aborts. What could cause it?
EDIT: here is what I've found in /var/log/syslog:
oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/code-server.service,task=python,pid=1133,uid=0
Mar 29 20:26:56 master kernel: [40821.605552] Out of memory: Killed process 1133 (python) total-vm:8036372kB, anon-rss:6906304kB, file-rss:0kB, shmem-rss:4kB
Mar 29 20:26:56 master kernel: [40821.852558] oom_reaper: reaped process 1133 (python), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB
SOLUTION: as Ondrej K. pointed in comments, the issue is due to OOM process call. Closing some of the apps connected to the internet solved the problem.