Mount filesystem image placed in a static variable

There are several steps we'll want to take

1. Create a file system image
2. Embed this image into the binary
3. Mount the embedded image as a read-only file system.

It sounds like you already know how to perform stems 1 and 2, but not how to do step 3.

I prepared ab.sqfs image and a.out which contains that image at offset 0x3010. Here are the commands to mount this filesystem:

# optional, look at the bytes of the filesystem from step 1 
xxd -l 16 -g1 ab.sqfs
00000000: 68 73 71 73 07 00 00 00 6c 61 ce 60 00 00 02 00  hsqs....la.`....

# optional: confirm that we have the correct file offset to the start of FS image
xxd -l 16 -g1 -s 0x3010 a.out
00003010: 68 73 71 73 07 00 00 00 6c 61 ce 60 00 00 02 00  hsqs....la.`....

# create a loop device which "points" into the file:
sudo losetup -r -o 0x3010 loop0 a.out 
losetup: a.out: Warning: file does not fit into a 512-byte sector; the end of the file will be ignored.

# optional: confirm that (just created) /dev/loop0 contains expected bytes
sudo xxd -l 16 -g1 /dev/loop0
00000000: 68 73 71 73 07 00 00 00 6c 61 ce 60 00 00 02 00  hsqs....la.`....

# create directory on which the FS will be mounted
mkdir /tmp/mnt         

# finally mount the FS:
sudo mount -oro /dev/loop0 /tmp/mnt

# optional: verify contents of /tmp/mnt
ls -lR /tmp/mnt
... has exactly the files I've put into it.

what C API magic do I need?

You can run the losetup and mount commands under strace to observe what they do. The key steps for losetup are:

openat(AT_FDCWD, "/tmp/a.out", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/dev/loop0", O_RDONLY|O_CLOEXEC) = 4
ioctl(4, LOOP_SET_FD, 3)                = 0
ioctl(4, LOOP_SET_STATUS64, {lo_offset=0x3010, lo_number=0, lo_flags=LO_FLAGS_READ_ONLY, lo_file_name="/tmp/a.out", ...}) = 0

And for mount:

mount("/dev/loop0", "/tmp/mnt", "squashfs", MS_RDONLY, NULL) = 0

These calls can be performed by the application itself, or by "shelling out" to external losetup and mount commands.