Minimizing negative log-likelihood of logistic regression, scipy returning warning: "Desired error not necessarily achieved due to precision loss."

Viewed 616

I'm trying to sort out why scipy optimize isn't converging on a solution for the minimum negative-log-likelihood of the logistic regression function (as implemented below).

It seems to converge for smaller data sets, but for the larger data sets scipy returns the warning: "Desired error not necessarily achieved due to precision loss."

I thought this was a well-behaved optimization problem, so I'm anxious that I'm missing an obvious mistake.

Can anyone spot a mistake in my implementation or make a suggestion that I might try?

I'm using the default method, but I have had little luck with the other various methods that miminize allows.

Many thanks!

Quick summary of the implementation. I'm minimizing the following statement:

enter image description here

with the caveat that since b is a constant, I'm using the exponent -(w*x + b). I think I've implemented that function correct, but maybe I'm not seeing something. Since the data are constants with respect to the function being minimized, I just output a function definition that retains the data within it; thus, the function to be minimized only accepts the weights.

The data is a pandas dataframe of the format: rows == samples, columns == attributes, but LAST column == label (0 or 1). I've transformed all the data to make sure it is continuous, and I've normalized it to have a mean of 0 and a standard deviation of 1. I'm also starting with random weights between [0, 0.1], treating the first weight as 'b'.

def get_optimization_func_call(data, sheepda):
    #
    # Extract pos/neg data without label
    pos_df = data[data[LABEL] == 1].as_matrix()[:, :-1]
    neg_df = data[data[LABEL] == 0].as_matrix()[:, :-1]

    #
    # Def evaluation of positive terms by row
    def eval_pos_row(pos_row, w, b):
        cur_exponent = np.dot(w, pos_row) + b
        cur_val = expit(cur_exponent)
        if cur_val == 0:
            print("pos", cur_exponent)
        return (-1 * np.log(cur_val))

    #
    # Def evaluation of positive terms by row
    def eval_neg_row(neg_row, w, b):
        cur_exponent = np.dot(w, neg_row) + b
        cur_val = 1.0 - expit(cur_exponent)
        if cur_val == 0:
            print("neg", cur_exponent)
        return (-1 * np.log(cur_val))

    #
    #  Define the function used for optimization
    def log_likelihood(weights):
        #
        # Separate weights
        w = weights[1:]
        b = weights[0]
        #
        # Ge the norm of weights
        w_norm = np.dot(w, w)
        #
        # Sum over positive examples
        pos_sum = np.sum(
            np.apply_along_axis(eval_pos_row, 1, pos_df, w, b)
        )
        neg_sum = np.sum(
            np.apply_along_axis(eval_neg_row, 1, neg_df, w, b)
        )
        #
        return (0.5 * w_norm) + sheepda * (pos_sum + neg_sum)

    return log_likelihood


w = uniform.rvs(size=20) / 10.0
LL = get_optimization_func_call(clean_test_data, 0.5)

res = minimize(LL, w, options={"maxiter": 1e4, "disp": True})
0 Answers
Related