Convert a list of image URLs to numpy arrays asynchronously

Viewed 28

I've found other ways to do this, but it takes quite a while to run and often has errors.

Say i have a list of image urls in a pandas df:

'https://cdn.inventoryrsc.com/218411526_6321fdae8d1f36399bdc0c50.jpg'
,'https://cdn.inventoryrsc.com/218531088_63220437cb5f291e0a2966d0.jpg'
,'https://storage.googleapis.com/talkinto-production-dealerships-catalog-photos/'
,'https://cdn.inventoryrsc.com/216800732_6321fc3a8dac5d6eccce9c7c.jpg'
,'https://cdn.inventoryrsc.com/217733537_632206e375803c4c2508949a.jpg'

my objective is to read/open each url and convert to a numpy array, then sum each array and throw it back into the df as a column.

The sum piece of code works fine based on the urllib way i was using (shown below):

image_pixels = []
for i in df_test['source_url']:
    images = image_url_to_numpy_array_urllib(url=i)
    images = images.ravel()
    images = images.astype(int)
    
    image_pixel_sum = images.sum()
    image_pixels.append(image_pixel_sum)
    
df_test['sum_pixels'] = image_pixels

Some code to do this using urllib is:

def image_url_to_numpy_array_urllib(url,format=None):
    req = urllib.Request(
        url=url 
        ,headers={
            'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11',
            'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
            'Referer': 'https://cssspritegenerator.com',
            'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
            'Accept-Encoding': 'none',
            'Accept-Language': 'en-US,en;q=0.8',
            'Connection': 'keep-alive'
        }
    )
    
    ## read as HTTPResponse 
    resp = urllib.urlopen(req)
    ## read as 1D bytearray
    resp_byte_array = resp.read()
    ## returns a bytearray object which is a mutable sequence of integers in the range 0 <=x< 256
    mutable_byte_array = bytearray(resp_byte_array)
    ## read as unsigned integer 1D numpy array
    image = np.asarray(mutable_byte_array, dtype="uint8")
    ## To decode the 1D image array into a 2D format with RGB color components we make a call to cv2.imdecode
    image = cv2.imdecode(image, cv2.IMREAD_COLOR)
    if format=='BGR' :
        ## return BGR format array
        return image
    ## cv2.imdecode converted array into BGR format , convert it to RGB format
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    # return the image
    return image

But urllib often throws errors.

Trying to use something like this to read and open each url and convert to a numpy array within the same function:

import asyncio

#NOTEBOOK SPECIFIC 
import nest_asyncio
nest_asyncio.apply()
    
async def download(url):
    filename = url+'.jpg'
    
    async with session.get(url) as response:
        async with aiofiles.open(filename, "wb") as f:
            mutable_byte_array = bytearray(await f.write(await response.read()))
    
            ## read as HTTPResponse 
            # resp = urllib.urlopen(req)
            ## read as 1D bytearray
            # resp_byte_array = resp.read()
            ## returns a bytearray object which is a mutable sequence of integers in the range 0 <=x< 256
            # mutable_byte_array = bytearray(f)
            ## read as unsigned integer 1D numpy array
            image = np.asarray(mutable_byte_array, dtype="uint8")
            ## To decode the 1D image array into a 2D format with RGB color components we make a call to cv2.imdecode
            image = cv2.imdecode(image, cv2.IMREAD_COLOR)
            if format=='BGR' :
                ## return BGR format array
                return image
            ## cv2.imdecode converted array into BGR format , convert it to RGB format
            image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
            # return the image
    return image

But i keep getting these cooroutine error

AttributeError: 'coroutine' object has no attribute 'ravel'

greatly appreciate anyone's help. I know this is a weird one

0 Answers
Related