I'm creating a virtual serial port using socat.
socat -d -d pty,echo=0,raw pty,echo=0,raw
That works as expected so far. Using echo/cat I can send/receice text etc.
But what about signal lines like DTR or RTS? How would I get / set the state of these lines with a pty? Is that even possible? I couldn't find any mentions about it anywhere.
socat is a pipe handler, basically lets you tap in the Tx and Rx "lines" without you having to care about signaling when data is ready/received.
The RTS/CTS/DSR/DTR are actual pins in a serial connector that control what is going on on the Tx/Rx lines.
Off the top of my head, I haven't used socat nor tried to do anything similar, lowest possible level I got was the EMV interface and protocol and sometimes I also netcat stuff real quick between machines when I'm too lazy to cp to a directory within httpd home... anyway, if you are trying to connect two entities with socat (separate machines, or applications on the same machine) you'll either use the same pipe and specify some control characters so they end up talking at the same time (got to make a note of this and try to implement it somehow with my wife), or use two separate pipes, one for Rx and one for Tx: Tx of entity 1 goes into Rx of entity 2, Tx of entity 2 goes into Rx of entity 1.
From your comment, it sounds like you want to control RTS/CTS independent of your data stream. You will have to write an application to interact with the serial port using ioctls.
I found this helpful forum post (with an example application) https://www.linuxquestions.org/questions/programming-9/manually-controlling-rts-cts-326590/
You can use hardware flow control lines (RTS/CTS):
socat stdio file:/dev/ttyAMA0,crtscts=1,b9600
Nowadays, this is mostly useful when talking to an RS485 transceiver, since that's the most common example of half-duplex serial line that is still in use. On some commonly used transceivers, such as the 75HVD12, the DE (drive enable) pin is connected to the host's RTS. There's also an active-low /RE (receive enable) pin that is connected to either RTS or CTS.
If you have access to GPIO, such as on a Raspberry Pi, you might be able to assign DTR and DSR to an output and input pin respectively. Alternatively, an USB adapter such as FTDI232 will assert DTR if it is connected to a computer.