The Pandoc documentation says that cross references can be made to section headers in a number of ways. For example, you can create your own ID and reference that ID. For example:
# This is my header {#header}
Will create an ID with value '#header' that can be refenced in the text, as such:
[Link to header](#header)
Which will display the text 'Link to header' with a link to the header.
I couldn't find anywhere how to make the text of the link be the section number when compiled as a LaTeX document.
For example, if my header is compiled to '1.2.3 Section Header', I want my cross-reference to text to display as '1.2.3'.