Sendmsg(UDP) with ancillary message to set source port

I have a UDP socket on which i have specified the IP_PKTINFO flag to pass a ancillary message containing a pktinfo struct to set a custom source address

struct in_pktinfo {
  unsigned int   ipi_ifindex;
  struct in_addr ipi_spec_dst;
   struct in_addr ipi_addr;
};

struct in_addr is just wrapping struct around a unsigned long

struct in_addr {
   unsigned long s_addr;  // load with inet_aton()
};

On my implementation:

Datagrams arriving on a transparent UDP socket(tproxy), analyzed, processed and then forwarded to a specific destination(Port Address Translation). So it is crucial for me to set the source IP as well as source port.

I can realize setting a specific source port if i open new socket and bind to it, because of the large amount of datagrams it is very inefficient.

Questions: is there anyway to set the source port without multiple sockets/binds or using a raw socket.