how local variables work inside nest function in python 3?

You need to tell the inner function that the res variable is nonlocal, like this:

def main(m):
    res = m
    def help(a, b):
        nonlocal res
        print(res)
        res = min(res, a*b)
    help(3,2)
    return res
main(3)

You can read from a non locally scoped variable without issue in a function, but if you try to write to a variable then python assumes that you want to create a new locally scoped variable. That local variable hides and prevents access to any similarly named variable that may exist in the namespace of higher scopes, like outer functions or global variables. To inform it that you do not want to create a local variable and want to access the outer scope variable you need to use the nonlocal keyword as in the above example.

Using global res does not fix it because global variables are at the top level of the namespace hierarchy and the res variable that you are trying to access is not. The res you want to access is in between the global scope and the local scope. That is why you need to use nonlocal instead of global. If res were at the top/external level then global would have been the solution.

The problem is that you are introducing a new, uninitialized local variable in the help() function, then trying to pass it to the min(res, a*b) function.

Instead of trying to deal with the scoping issues, you could just return min(res, a*b) and assign it to res outside of the help() function:

def main(m):
    res = m
    def help(a, b):
        # print(res)
        return min(res, a*b)
    res = help(3,2)
    return res

The relevant code here is

res = m
def help(a, b):
    print(res)
    res = min(res, a*b)  # may or may not be commented out

This is not doing what you think it is.

Let's go through this line-by-line:

1. res = m: this defines the variable res, with value m, which lives in the main function
2. def help(a, b): begins a function definition. Within the help function, you are able to define entirely new variables which are insulated from the variables in main
3. print(res): this prints the value of variable res. To find the value of res, Python will first look in help function a variable named res. If there is none, it will search the main function (i.e. the parent scope) for a variable named res. (If there were to be none there, it would continue to search higher and higher scopes.) As it turns out, in your example, whether or not there is a variable named res in the help function depends on whether or not line 4 is commented out!
4. res = min(res, a*b): This does not do what you think it does. What you want it to do is to change the variable value of the res variable in the main function. However, Python will instead create a new variable res within the help function and set its value to min(res, a*b).

This is why you are getting an UnboundLocalError. Because when line 4 is uncommented, then print(res) refers to the variable named res in the help function, not the main function. But when line 4 is commented out, there is no res variable in the help function, so print(res) instead uses the variable in the main function.

So, how to fix it? Actually pretty simple, just change the code to:

def help(a, b):
    nonlocal res
    print(res)
    res = min(res, a*b)

The nonlocal statement tells Python, "hey, the res variable is coming from an outer scope".