Python - Decode Base64 string that is longer than the limit

user_10532778 0
Viewed 45

In order to upload large csv files I have implemented a system that splits the file into smaller chunks, encodes those chunks and sends to the backend. On the backend (PHP), I append all the base64 encoded data to a single file as I had planned to decode it all in python using a shell script.

This works great on small files but once I reach files that are over a certain limit (in my case 100mb), I fail to decode the entire file. Here is my code

def getDataFrameFromFile():
    # Read file in - which will be in base64url
    contents = ""
    with open(PATH + FILE_NAME) as f:
        contents = f.read()

    text = b64decode(contents).decode('utf-8')
    print("Length of text: ", len(contents))
    # Create pandas Dataframe, first column is time, second is raw
    DATA = StringIO(text)
    df = pd.read_csv(DATA, sep=',')
    return df

This is the output I got from the print statement: Length of text: 69905200. I believe this is too large for function b64decode to handle in one call.

Is there a way I can split the base64 into smaller chunks, decode them and then join the data at the end?

I read somewhere that you can split base64 into strings of character length of mod 3 = 0. Is this true, if so how would I do this?

decode base64 python
1 Answers

I read somewhere that you can split base64 into strings of character length of mod 3 = 0. Is this true, if so how would I do this?

This is indeed true (an understanding of the way base64 encoding works and a little math will tell you why, if you want to know). One really easy thing you can do is split your string into divisible-by-3-length-pieces you know you can decode, decode the pieces, and then concatenate them. I'm going to choose 15 as my piece length (you'll likely want to choose a much larger number that's divisible by 3). So, perhaps:

PIECESIZE=15
text = ''
while(len(contents) > 0):
  text += b64decode(contents[:PIECESIZE])
  contents = contents[PIECESIZE:]
user_197581380
Related