How to merge multiple H.264 streams into a single H.264 stream

I encode camera images with several parallel H.264 baseline I-frame only encoder cores in FPGA (more parallel cores are required to get the encoding done fast enough without external memory). I use an opensource encoder. On simulator I use 352x288 yuv420 input image and I slice it to 2 horizontal halves.

The decoder is on another embedded device and I can decode the separated slices correctly. But, instead of several decoded video sequences, I would need to have just one vido sequence (with the whole image) from one H.264 byte stream. It is preferred to merge the H.264 streams on the FPGA (instead of doing this by post processing the decoded slice video sequences on the embedded host).

So, the input image data looks like this:

    .--.-----.--.--.-----.--.
    |MB|     |MB|MB|     |MB|
    | 0| ... |10|11| ... |21|
    |--.     '--|--'     '--|
    |MB|        |           |
    |22|        |           |
    |--'        |           |
    |   Slice   |   Slice   |
    |     0     |     1     |
    '-----------'----------'

And the encoder's block diagram is this (just one encoder core shown, without merging):

                                 YUV420 MacroBlocks (16*16 pixel)
                                         |       |
                                  chroma |       | luma
                                         |       |
                      .----------------.-------. |
                      |                | |     | |
                      |                v v     v v
                 reconstruct      intra8x8cc  intra4x4 ---> header
                      ^                |        |             |
                      |                v        v             |
                inv.transform        coretransform            |
                      ^                |     |                |
                      |                |     v                |
                  dequantise           | dctransform          |
                   ^       ^           |     |                |
                   |       |           v     v                |
        inv.dc.transform <-'---------- quantise               |
                                           |                  |
                                           v                  |
                                         buffer               |
                                           |                  |
                                           v                  |
                                          cavlc               |
                                           |                  |
                                           v                  |
                                         tobytes <------------'
                                           |
                                           v
                                         H.264 byte stream

The encoded h264 data from a singel core (parsed with H264Naked, see parse.txt, ref.yuv420 and ref.264) contains 3 NAL-s: in the first one there is a Sequence Parameter Set, in the second a Picture Parameter Set and in the third a Coded slice of an IDR picture. The SPS and PPS headers are static and these are generated on the host. The IDR header is generated by the encoder on the FPGA. Currently I use a constant Quantization Parameter (i.e. VBR, but it is planned to implement CBR, with variable QP later).

To get the encoded streams merged, I have removed the IDR header from the slices (except from slice 0) in the header block. Then I buffer the output of CAVLC blocks from each encoder for a whole MB row (this is the encoded MB data, MB headers can be identified from the output of header block). Then from this buffer I drive all the CAVLC data to a new instance of tobytes block in this sequence:

• CAVLC data of one MB row from slice 0
• CAVLC data of one MB row from slice 1
• the above is repeated till the end of the frame.

When I decode this merged H.264 stream, then I get the following error with ffmpeg (see merged_tobytes.264 and ffplay_error.txt):

[h264 @ 0x7f5f0c00ac00] dquant out of range (-112) at 13 1B f=0/0   
[h264 @ 0x7f5f0c00ac00] error while decoding MB 13 1
[h264 @ 0x7f5f0c00ac00] concealing 396 DC, 396 AC, 396 MV errors in I frame

With JM reference decoder I get this error (jm_dec_error.txt):

mb_qp_delta is out of range (-112)
illegal chroma intra pred mode!

I think the problem is that the decoder does not know about the slices and expects neighboring MB data from other slice which are not available in the encoded data. I have checked the Macroblock prediction syntax in the standard (chapter 7.3.5.1), but I do not see how should I correct this (I have very little knowledge about H.264). On the encoder side, I see that intra4x4 and intra8x8cc are using neighboring MB prediction and pixel data from the top and left MB neighbors when those are available (we have no top MB neighbors in the first MB row and we have no left MB neighbor at first MB of each MB row, in each slice).

I have also tried to use FMO and declared slice groups (slice_group_map_type=2) as described here, but FMO and slice groups are not supported by typical H.264 decoders (see ffmpeg issue, I think it is used only in broadcast equipments). In fact, I had also got error from JM reference decoder even when I have declared the slice groups (see merged_tobytes_with_slice_grops.264 and jm_dec_slice_groups_error.txt):

warning: Intra_8x8_Horizontal prediction mode not allowed at mb 0
illegal chroma intra pred mode!

Any help is appreciated.

Reference photo is taken from here.

PS: if someone can advise a deblocker which could be used with this FPGA core, that would be also nice.