How to customize a loss function that minimize the misclassification matrix cost in tensorflow CNN?

I need help on building tensorflow CNN model with my own loss function that minimize the misclassification matrix cost. For example: assume the misclassification matrix is below cost matrix cost = 4+6+2+3+5+1 I want this as my loss function. Can anyone help on this?

model = keras.models.Sequential()
model.add(keras.layers.experimental.preprocessing.Rescaling(1./255, input_shape=(IMG_HEIGHT,IMG_WIDTH, 3)))

model.add(keras.layers.Conv2D(filters=32,kernel_size=(3,3),padding='same',activation='relu') 
model.add(keras.layers.MaxPooling2D(pool_size=(2,2)))

model.add(keras.layers.Conv2D(filters=64,kernel_size=(3,3),padding='same',activation='relu') 
model.add(keras.layers.MaxPooling2D(pool_size=(2,2)))

model.add(keras.layers.Conv2D(filters=128,kernel_size=(3,3),padding='same',activation='relu')
model.add(keras.layers.MaxPooling2D(pool_size=(2,2)))

model.add(keras.layers.Flatten())
model.add(keras.layers.Dense(128,activation="relu")
model.add(keras.layers.Dense(64,"relu"))
model.add(keras.layers.Dense(5,"softmax"))
def custom_loss():
    **minimize cost matrix**

model.compile(loss=custom_loss,
              optimizer = "Adam",metrics=["accuracy"])
model.summary()