How to make a tensorflow GAN model that attack an OCR model

Question

I was using the the attacks algorithm in the Cleverhans to do an OCR attacks. Basically, I am attacking a black-box mobile phone app that recognize the ID numbers from picture.

I found C++ tesseract model capable of doing this. So I was planning on a transfer attacks towards the tesseract model. The accuracy of tesseract model is quite low: 100 out of 157 images.

So my first question is that could you give me some suggestions of such OCR model that I can use directly, better if pre-trained

OCR model only need to recognize digits 0-9 and character X
I am able to get some confidence values

So currently I am developing the algorithm upon tesseract model. Since the tesseract api is written in c++, I have to write the prediction model as c++ file and compile it to binary file.

My solution of transferring images and predictions between Python and C++ is that:

I will first save the perturbed image locally
run the binary file to store output labels and confidence value in a .csv file
Python will then read the labels and confidence values from .csv file

The current problem I encountered is that Tensorflow does not consider the values after a sess.run() as part of the graph. So every time the code ran

grad = tf.gradient(y, x)

it says that the grad is None.

Below is my code of saving the image and then return the confidence values.

filenames = []
# save for tesseract api to use
for idx in range(images.get_shape()[0] ):
    if not os.isdir(MODEL_BUFFER):
        os.mkdir(MODEL_BUFFER)

    filename = os.path.join(MODEL_BUFFER, "temp_image_" + idx + ".JPG")
    filenames.append(filename )

    with tf.gfile.Open(os.path.join(MODEL_BUFFER, "temp_image_" + idx + ".JPG"), 'w') as f:
        img = (((images[idx, :, :, :] + 1.0) * 0.5) * 255.0)

        enc = tf.image.encode_jpeg(img)

        fname = tf.constant(filename)

        fwrite = tf.write_file(fname, enc)

        sess.run(fwrite)
# call tesseract
cmd = ["./ts_at"] + filenames
subprocess(cmd)

# read the predictions
y_preds = []
with open("output.csv", "r") as f:
    reader = csv.reader(f)
    header_row = next(reader)
    filename = header_row.index("filename")
    confidence = header_row.index("confidence")
    ID = header_row.index("ID")
    idx = 0
    for row in reader:

        y_preds.append(int(row[confidence]) )

return y_preds

Also, I have tried returning tf.constant(y_preds) but still no luck. One explanation online is that Tensorflow thinks that there is no connections between the confidence values and the input images.

So my second question is that how to solve this problem?

Thanks

How to make a tensorflow GAN model that attack an OCR model

0 Answers