How to capture error message in python that is not an exception?

The error inside the library is probably written to the standard error output sys.stderr. You could thus redirect the error output and check if the library is writing to it. See for instance this question on logging error output.

Once you have a logger (or in your case a class that records and parses any errors), use it like this:

import sys
_stderr = sys.stderr
sys.stderr = MyErrorCheckLogger()
try:
    myOrder = myAddress.sell(...)
finally:
    sys.stderr = _stderr

Of course, you can do the same thing with sys.stdout—the standard output channel.

Edit: Here is a full example where we catch any error messages directly printed to the standard output (stdout).

import sys

class ErrorChecker:
    def __init__(self, output):
        self.output = output               # Save the original `stdout` (or `stderr`)
        self.buffer = ""                   # The buffer for the message printed

    def write(self, message):
        self.buffer += message             # Add any message to our buffer
        self.output.write(message)         # Print it as original intended

    def flush(self):
        self.output.flush()

def throwError(f, *args, **kwargs):
    sys.stdout = ErrorChecker(sys.stdout)  # Create a new output-stream with buffer
    try:
        result = f(*args, **kwargs)        # Execute the code that might cause an error
    finally:
        buffer = sys.stdout.buffer         # Get the output buffer
        sys.stdout = sys.stdout.output     # Restore the original 'stdout'
    if buffer.startswith("[ERROR]"):       # If the output is an error message, raise
        raise Exception(buffer)            # it as an actual exception
    return result

myOrder = throwError(myAddress.sell, assetPair = PRO_WAVES, amount = 1491999999, price = proprice1, matcherFee = 700000, maxLifetime = 3600)

After reading through the actual code, it appears that you can use pywaves.setThrowOnError() to cause exceptions to be raised whenever errors are logged, such as in the situation you describe. However, for whatever reason that doesn't appear to be documented, so use at your own risk. You can use the issue tracking system to communicate with the developers about this.