How to detect which values are allowable in a parameter grid?

user_5541865 0
Viewed 1047

I have started working on a project wherein I need to detect trainable parameters for a given scikit-learn estimator, and if possible, to find allowable values for categorical variables (and reasonable intervals for continuous ones).

I can fetch a dictionary with parameters usingestimator.get_params()and then set a value usingestimator.set_params(**{'var1':val1, 'var2':val2}), and so on.

For example, for a KNN-classifier we have the following dict of params: {'metric': 'minkowski', 'algorithm': 'auto', 'n_neighbors': 10, 'n_jobs': 1, 'p': 2, 'metric_params': None, 'weights': 'uniform', 'leaf_size': 30}.

Now, I can using the types of the values to infer which are categorical (str types), continuous (float), discrete (int) and so on. One possibly related problem is parameters for which the default is set to NoneType, but I might just not touch these anyway, for a good reason.

The challenge now becomes to infer and define a parameter grid for use in e.g. RandomizedSearchCV. For discrete and continuous variables the problem is tractable using e.g. a combination of try-except blocks together with the scipy.stats module, possible restricting the interval to lie in the "vicinity" around the default value (but at the same time being careful to not set e.g. n_jobs to some crazy value -- that might need to be hard-coded in, or explicitly set later). If you have experience with something similar, and have some tips/tricks up your sleeve, I would love to hear about them.

But the real problem now is: how to infer for e.g. algorithm that the allowable values actually are{‘auto’, ‘ball_tree’, ‘kd_tree’, ‘brute’}??

I have just started looking into the problem, and perhaps we can parse the error message we get if we try to set it to some un-allowable value? I am on the look out for good ideas here, as I want to avoid having to do this manually (I will if I have to, but it seems rather inelegant...)

cross-validation scikit-learn parameter-passing python
2 Answers

My attempt to get all this from the docstring (LinearSVC as the example algorithm), which was aided greatly by splitlines():

liner = str(LinearSVC().__doc__).split('Parameters\n    ----------\n')[1].split('\n\n    Attributes\n')[0].replace('\n        ', '\n').splitlines()

This does not create a dictionary, but is simple enough to extract just the explained "Parameters" section from the docstring, which has all of the params explained and have all their listed possible/expected/accepted value inputs, which are nicely indented by one, tab, and now we can use a simple loop with a conditional, using " : " as our anchor to identify the lines of possible/expected/accepted value inputs:

for i in liner:
   ...:     if " : " in i: #<<< the key is to use " : " as our anchor
   ...:         print(i)

The end result, prints out to:

    penalty : str, 'l1' or 'l2' (default='l2')
    loss : str, 'hinge' or 'squared_hinge' (default='squared_hinge')
    dual : bool, (default=True)
    tol : float, optional (default=1e-4)
    C : float, optional (default=1.0)
    multi_class : str, 'ovr' or 'crammer_singer' (default='ovr')
    fit_intercept : bool, optional (default=True)
    intercept_scaling : float, optional (default=1)
    class_weight : {dict, 'balanced'}, optional
    verbose : int, (default=0)
    random_state : int, RandomState instance or None, optional (default=None)
    max_iter : int, (default=1000)

So glad I can share, and if anyone else needs the full docstring parameter printout, just use:

print(str(LinearSVC().__doc__).split('Parameters\n    ----------\n')[1].split('\n\n    Attributes\n')[0].replace('\n        ', '\n'))

EDIT: If this is not intended to print out - the best way to have it as a string object is using a list comprehension, but it requires some ugly replaces, because there is extensive notation in the docstring:

docstring_short = str([i for i in liner.splitlines() if " : " in i]).replace('["    ', '').replace('    ', ',\n').replace('", "', '').replace('", \'', '').replace("', '", '').replace("', \"", '').replace(']', '')
user_60436140
Related