Avoid overflow with softplus function in python

user_1880863 0
Viewed 8148

I am trying to implement the following softplus function:

log(1 + exp(x))

I've tried it with math/numpy and float64 as data type, but whenever x gets too large (e.g. x = 1000) the result is inf.

Can you assist me on how to successfully handle this function with large numbers?

machine-learning python
5 Answers

TLDR:

import numpy as np
import math

def softplus_np(x): return np.log1p(np.exp(-np.abs(x))) + np.maximum(x, 0)
def softplus_math(x): return math.log1p(math.exp(-abs(x))) + max(x, 0)

Explanation: There is a relation which one can use:

log(1+exp(x)) = log(1+exp(x)) - log(exp(x)) + x = log(1+exp(-x)) + x

So a safe implementation, as well as mathematically sound, would be:

log(1+exp(-abs(x))) + max(x,0)

This works both for math and numpy functions (use e.g.: np.log, np.exp, np.abs, np.maximum).

user_102208500

i use this code to work in arrays

def safe_softplus(x):
    inRanges = (x < 100)
    return np.log(1 + np.exp(x*inRanges))*inRanges + x*(1-inRanges)
user_86639320

What I'm currently using (slightly inefficient but clean and vectorized):

def safe_softplus(x, limit=30):
    return np.where(x>limit, x, np.log1p(np.exp(x)))
user_22511320

If you're using TensorFlow, you can simply use the tf.math.softplus.

user_70112870
Related