Writing 1K images to disk using Task.async_stream in Elixir

user_4952770 0
Viewed 118

I am downloading images through HTTP request with which I am getting a binary image, and writing it to a file such as

File.write(image_with_dir, image, [:binary]) |> File.close

this whole operation of getting HTTP request and then writing it to disk is done in 

|> List.flatten()
|> Enum.sort()
|> Task.async_stream(&(inline_process.(&1, images_directory)), max_concurrency: System.schedulers_online() * 2, timeout: :infinity)
|> Stream.run

When decreasing max_concurrency the process got slow approx 2 minutes, also results of System.schedulers_online() is 8

but with current max_concurrency it faster but with this. Disk IO starts touching the limits

enter image description here

Purpose of writing those files is to send them to Dropbox with a batch of 1000 as dropbox upload session supports 1000 images at a time.

Is there any better way to write images to disk? maybe in memory but I don't know, any help would be wonderful also this operation is being done on Cuda GPU machine but I am not sure how I can use GPU for such purpose.

This process is user defined. user can ask for less/more than 1000 images and those can be one or multiple Task.async_stream's

elixir gpu
1 Answers

A Possible Elixir Approach

I am a newbie in Elixir, but you may try to use the ETS table to write them temporally into memory, but don't forget to wrap it with a GenServer as per the link recommendations. So the tutorial is for key/value cache, that I think it will be easy for you to adapt for your use case.

The Disk Approach

Is there any better way to write images to disk?

Linux OS have ram disks:

RAM disk is also known as RAM drive. It’s a portion of your RAM that are formated with a file system. You can mount it to a directory on your Linux system and use it as a disk partition.

How to Create a RAM Disk in Any Linux Distro

From the linked article I have extracted the main steps for you...

Create a dir:

sudo mkdir /tmp/ramdisk

Mount it with the device name myramdisk and using the tmpfs file system with 10GB of ram disk space:

sudo mount -t tmpfs -o size=10G myramdisk /tmp/ramdisk

NOTE: Adjust the 10G to your needs and available memory on your server.

Check it's mounted:

mount | tail -n 1

You can unmout it at any time to reclaim the the memory:

sudo umount /tmp/ramdisk/

Auto mount it on system boot by adding to /etc/fstab:

myramdisk  /tmp/ramdisk  tmpfs  defaults,size=10G,x-gvfs-show  0  0

Benchmark it

Test write speed with:

sudo dd if=/dev/zero of=/tmp/ramdisk/zero bs=4k count=100000

Test read speed with:

sudo dd if=/tmp/ramdisk/zero of=/dev/null bs=4k count=100000
user_64546220
Related