catch syscall but exclude syscalls from libc

I need to catch mprotect syscalls that is not originating from libc

You could likely achieve almost the desired result by filtering out mprotect calls originating from the particular address in libc.so.6 -- most calls should be coming from that address:

(gdb) cond 2 $rip != 0x00007ffff74cde57

If that's not good enough, you should be able to use embedded Python to evaluate arbitrary conditions once catchpoint is reached, using the Breakpoint.stop function.