SnapshotListener not working offline when querying by server timestamp

In my chat app, I retrieve new chat messages via this query:

fun getNewChatMessages(chat: Chat, from: Date) =
    getCollectionForChat(chat)
        .document(chat.documentId!!)
        .collection("messages")
        .orderBy("timeStamp", Query.Direction.DESCENDING)
        .whereGreaterThanOrEqualTo("timeStamp", from)
        .addSnapshotListener( [...]

The "timestamp" field is a server timestamp:

data class ChatMessage(
    val text: String? = null,
    val senderUid: String? = null,
    val senderName: String? = null,
    val imageUrl: String? = null,

    @ServerTimestamp val timeStamp: Date? = null,
    @DocumentId val documentId: String? = null
)

When I add a new chat message to the collection while offline, I am not getting any live updates. I could track the problem down to this line of the query:

.whereGreaterThanOrEqualTo("timeStamp", from)

Without this line, I get updates immediately even while offline. Does equality queries on the server timestamp only work once the data is persisted on the server? If yes, what is my alternative? Using a local timestamp (i.e. Date() or System.currentTimeMillis())?