I've found other ways to do this, but it takes quite a while to run and often has errors.
Say i have a list of image urls in a pandas df:
'https://cdn.inventoryrsc.com/218411526_6321fdae8d1f36399bdc0c50.jpg'
,'https://cdn.inventoryrsc.com/218531088_63220437cb5f291e0a2966d0.jpg'
,'https://storage.googleapis.com/talkinto-production-dealerships-catalog-photos/'
,'https://cdn.inventoryrsc.com/216800732_6321fc3a8dac5d6eccce9c7c.jpg'
,'https://cdn.inventoryrsc.com/217733537_632206e375803c4c2508949a.jpg'
my objective is to read/open each url and convert to a numpy array, then sum each array and throw it back into the df as a column.
The sum piece of code works fine based on the urllib way i was using (shown below):
image_pixels = []
for i in df_test['source_url']:
images = image_url_to_numpy_array_urllib(url=i)
images = images.ravel()
images = images.astype(int)
image_pixel_sum = images.sum()
image_pixels.append(image_pixel_sum)
df_test['sum_pixels'] = image_pixels
Some code to do this using urllib is:
def image_url_to_numpy_array_urllib(url,format=None):
req = urllib.Request(
url=url
,headers={
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Referer': 'https://cssspritegenerator.com',
'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
'Accept-Encoding': 'none',
'Accept-Language': 'en-US,en;q=0.8',
'Connection': 'keep-alive'
}
)
## read as HTTPResponse
resp = urllib.urlopen(req)
## read as 1D bytearray
resp_byte_array = resp.read()
## returns a bytearray object which is a mutable sequence of integers in the range 0 <=x< 256
mutable_byte_array = bytearray(resp_byte_array)
## read as unsigned integer 1D numpy array
image = np.asarray(mutable_byte_array, dtype="uint8")
## To decode the 1D image array into a 2D format with RGB color components we make a call to cv2.imdecode
image = cv2.imdecode(image, cv2.IMREAD_COLOR)
if format=='BGR' :
## return BGR format array
return image
## cv2.imdecode converted array into BGR format , convert it to RGB format
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
# return the image
return image
But urllib often throws errors.
Trying to use something like this to read and open each url and convert to a numpy array within the same function:
import asyncio
#NOTEBOOK SPECIFIC
import nest_asyncio
nest_asyncio.apply()
async def download(url):
filename = url+'.jpg'
async with session.get(url) as response:
async with aiofiles.open(filename, "wb") as f:
mutable_byte_array = bytearray(await f.write(await response.read()))
## read as HTTPResponse
# resp = urllib.urlopen(req)
## read as 1D bytearray
# resp_byte_array = resp.read()
## returns a bytearray object which is a mutable sequence of integers in the range 0 <=x< 256
# mutable_byte_array = bytearray(f)
## read as unsigned integer 1D numpy array
image = np.asarray(mutable_byte_array, dtype="uint8")
## To decode the 1D image array into a 2D format with RGB color components we make a call to cv2.imdecode
image = cv2.imdecode(image, cv2.IMREAD_COLOR)
if format=='BGR' :
## return BGR format array
return image
## cv2.imdecode converted array into BGR format , convert it to RGB format
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
# return the image
return image
But i keep getting these cooroutine error
AttributeError: 'coroutine' object has no attribute 'ravel'
greatly appreciate anyone's help. I know this is a weird one