CSS pseudo-selector to select the current element within querySelector usable for the sibling (+) or general sibling (~) selector?

If I have this HTML:

<div class="elem">
   <div class="child"></div>
</div>
<div class="sibling">
</div>

and the JS

let elem = document.querySelector('.elem');

Using elem.querySelector(":scope ~ div") does not return anything even though there are div siblings of elem.

Using elem.querySelector(":scope > div") returns an element because also there are are div children of elem.

How could I select the current sibling of elem?