I have the following problem while trying to use Mask RCNN (not exactly this build(?) but some slightly modified one that allows the use of newer Tensorflow versions) to train a model to detect some objects. For this I modified the aproach shown here. It almost worked by using my CPU but it took 1,5 hours, did not deliver results (after going through the batches it stopped doing anything instead of saving the hopefully generated weights) and was not even the whole dataset. So I tried to get it to work on my GPU, which is not some high-end one but hopefully should still do it faster but I'm currently stuck with a problem. Namely it appears to "crash" without any specific error message shortly after allegedly starting the training process. The last thing one sees is a bunch of warnings like this one:
2022-06-16 09:50:13.484343: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -14 } dim { size: -122 } dim { size: -123 } dim { size: 256 } } } inputs { dtype: DT_FLOAT shape { dim { size: -43 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -43 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } value { dtype: DT_INT32 tensor_shape { dim { size: 2 } } int_val: 14 } } device { type: "GPU" vendor: "NVIDIA" model: "NVIDIA GeForce GTX 860M" frequency: 1019 num_cores: 5 environment { key: "architecture" value: "5.0" } environment { key: "cuda" value: "11020" } environment { key: "cudnn" value: "8100" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 2097152 shared_memory_size_per_multiprocessor: 65536 memory_size: 1428478363 bandwidth: 80160000 } outputs { dtype: DT_FLOAT shape { dim { size: -43 } dim { size: 14 } dim { size: 14 } dim { size: 256 } } }
And the process finishes with this:
Process finished with exit code -1073740791 (0xC0000409)
As I currently am also low on storage on my main drive I also noticed that it suddenly got even scarcer. The reason is that after each time I tried to execute the process it created pretty large (656 MB) crash dumps. Using windbg to open it got me this information:
This dump file has an exception of interest stored in it. The stored exception information can be accessed via .ecxr. (2bb8.29bc): Stack buffer overflow - code c0000409 (first/second chance not available) *** WARNING: Unable to verify checksum for cudnn64_8.dll *** ERROR: Symbol file could not be found. Defaulted to export symbols for cudnn64_8.dll - cudnn64_8!cudnnBackendFinalize+0x3dad: 00007ff9`bb8c420d cd29 int 29h
So it may or may not have something to do with CUDNN. Another thing that may have something to do with it is the GPU, as it has a 100% spike at the crashing moment in the "Copy-section of the task manager.
Anyways I'm not sure how to proceed so maybe someone here can suggest what could resolve this problem. Thanks!
Edit: I noticed that one needs a specific version of CUDA and cdnn for Tensorflow while I used the most current ones. I changed that to 11.2 and 8.1 respectively as intended and there was indeed change ... unfortunately in my opinion to the worse as the "crash" with an hexdec exit code still happens while there is no crash dump anymore.
While executing the code at one point it actually creates a .dmp file with 0 Byte size and it gets deleted when the process exits like a minute later. I never looked before if it was always happening this way but I guess it was while the GPU copy spike was putting what was inside its memory into the file. But now it is not even doing that...