TL/DR: I have to optimize a problem to minimize a non-linear function who's gradient is unknown. Its 80 input variables are categorical. I am looking for the best optimization algorithm to solve the problem, that must be implementable in Python.
Full question:
I want to optimize a problem that consists in minimizing an Objective Function (OF) which internally executes a neural network (programmed in Python) and is therefore unkown. The inputs to this function are 80 positions, therefore categorical values. I would like to find the best optimization algorithm, and would like some guidance, for the moment I haven't found any compatible libraries or good books/texts/posts/slides that are applicable to my specific problem.
Current approach:
My first approach was to try the NLOpt Python library, although made for continual variables I tried to restrict inputs using the library's restriction options, inputing variables as one hot vectors and imposing:
- equality restrictions to these vectors (upper bound 1, lower bound 0),
- their remainder when divided by 1 is 0 (to make them categorical) and
- no two elements can be in the same position and all postiions must be occupied (each one-hot vector must add to one and the sum of each position in all one-hot vectors must also add to one)
This didn't work out, as NLOPT explores the space where the restrictions are not fullfilled in intermediate steps and for these my OF can't be executed. I am aware that integer programming and discrete optimization are harder computational problems than linear programming as integer variables make an optimization problem non-convex, however, this seemed like a good initial exploration of the problem.
After exploring several other Python Libraries, I haven't found any that can be used for non-linear function optimization that include discrete variables.
I am now thinking of programming a simple Hill Climbing algorithm by hand with custom built mutations (postion changes) as a first step to then program a full Genetic Algorithm (following this and other GA classic texts for context) with a larger population and am also looking at Simulated Annealing algorithms. Defining mutations by hand makes good sense in my problem as some position changes can be more disruptive than others.
Do you think this is the correct approach? Any guidance or resources would be of great help. Thanks in advance.