Merge two streams from Firestore together in Dart/Flutter

I have a chat feature that gets Firestore documents for both the current user and the person he/she is chatting with. I'm trying to merge these streams together, but when I use mergeWith or Rx.merge, it only returns the results of one stream. Here is the function that gets the data:

fetchData(String userUid, String chatterUid) async {
var myChats = _messages
    .document(userUid)
    .collection('userMessages')
    .document(chatterUid)
    .collection('chats')
    .orderBy('time', descending: true)
    .snapshots();

var otherChats = _messages
    .document(chatterUid)
    .collection('userMessages')
    .document(userUid)
    .collection('chats')
    .orderBy('time', descending: true)
    .snapshots();

Rx.merge([otherChats, myChats]).listen((event) {
  if (event.documents.isNotEmpty) {
    _controller.add(event.documents.map((e) {
      return Chat.fromData(e.data);
    }).toList());
  } else {
    _controller.add(<Chat>[]);
  }
});

Is there anyway to combine the results so that all the results are filtered into one stream that I can return?