I am using a Basler camera and want to write images that I grabbed at 1000x1000px to disk as fast as possible. Keeping a movie container open and saving as .avi doesn't really help (slows down 2x) and saving single images is even slower. I am currently using openCV cvVideoCreator and imwrite functions. My computer should be fast enough (Intel Xeon 3.6 GHz, 64GB RAM, SSD). Any ideas?
2 Answers
There are multiple factors (steps) you may want to consider in order to reach your goal (saving color images (1K x 1K x 3) with 50 FPS or 150 MB/s).
Image encoding: most of well-known image formats such as png, tif, jpg takes time to encode image (e.g., 5.7 ms for png and 11.5 ms for tif in OpenCV with 3.6 GHz 4-core CPU) to the specific format even before saving the encoded format data to a disk.
File opening: Independent of file size, this may take time (e.g., 1.5 ms on 3.5" WD Black)
File writing: Dependent of file size, this may take time (e.g., 2.0 ms on 3.5" WD Black)
File closing: Dependent of file size, this may take a lot of time (e.g., 14.0 ms on 3.5" WD Black)
This means you have to finish all of the steps in 20 ms per image for your goal, and as I gave examples of timing, you may not be able to achieve your goal with OpenCV imwrite in JPG because the library sequentially does all the steps above in a single thread.
I think you have a couple of options
imwrite into BMP format on SSD as its encoding time is virtually zero (e.g., less than 0.3 ms)
do some of steps above (e.g., encoding or file closing) in a separate thread.
Especially, file closing is a good candidate to be run in a separate thread because it can be asynchronously done with the other steps. I was able to reach 400 MB/s bandwidth of saving with the second option, BMP file format, and a better hard disk.
Hope this helps you and others with similar goals.
The specs you state in your question are related to your ability to process and buffer the data, but not about the speed you can dump to disk.
You're trying to write (some numbers assumed, just replace with your own)
1000*1000 (size) * 4 (data/pixel) * 25 (frame rate) bytes per second.
(or 100M/s)
This is around abouts the limit of a traditional HDD, but if the disk is fragmented or full at all it's unlikely to keep up. As a result you must find a way to either speed up your write time (switch to SSD for example); reduce the data being written (compress, reduction in colour depth / quality / frame rate) or buffer what you want to write while a background thread saves to disk.
The question you must ask is how long do you plan to record for. If it's not long enough to fill up your RAM, then you have all the options available to you. If however you plan to record for extended periods of time then you will have to pick one of the other 2.
Related