The lambda expression () -> counter++; is valid as counter++; is a statement expression. This is explicitly permitted in the JLS:
If the function type's result is void, the lambda body is either a statement expression (§14.8) or a void-compatible block.
And for the statement expression's definition:
An expression statement is executed by evaluating the expression; if the expression has a value, the value is discarded.
If you read the whole JLS 15.27.3, you'll see why () -> {return counter + 1;} is not void-compatible. A lambda body is not validated with the exact same rules as the simple expression statement.
In other words, counter++; and ++counter; are statement expressions, meaning that the expression can be evaluated and be treated as a statement whose result is simply discarded.
It's clearer (or rather more familiar-looking) when converted to a block body:
Runnable task5 = () -> {
counter++; //result discarded, but you can't use "return counter++;"
};
Runnable task6 = () -> {
++counter; //result discarded, but you can't use "return ++counter;"
};
This has nothing to do with UnaryOperator, at least as far as that functional interface is concerned. It just happens that counter++ is compatible with IntUnaryOperator, but the expression could have been anything else, including (but not limited to) the following, and your question would still apply as the statement yields a result:
Runnable task5 = () -> counter += 1;
Runnable task6 = () -> counter = counter + 1;
Runnable task7 = () -> Math.addExact(counter, 1); // no change in counter
Runnable task8 = () -> return2(); //return2 takes no input
All these are int-producing expressions that are congruent with Runnable.run(). Specifically, task8 takes no input but produces a result; and it is not even compatible with any unary operator functional interface.