Lag in starting coroutine on Dispatchers.Main

ViewModel.kt

private val viewModelJob = Job()
private val defaultScope = CoroutineScope(Dispatchers.Default + viewModelJob)

fun sort() {

    defaultScope.launch {

        // sort logic (on tempList)

        Log.d("my_tag", "Sort time: " + (System.currentTimeMillis() - sortStart))

        withContext(Dispatchers.Main) {
            Log.d("my_tag", "Updating sorted list")
            sortedItemList.value = tempList
        }
    }
}

sortedItemList is a MutableLiveData in the viewModel, that is being observed in the activity, which in turn prints a log and uses the list to set a recycler view. But whenever I open the activity, I see a lag of around 80 - 300ms in setting the recycler view, which is quite noticeable. Here's my logcat:

2020-06-13 14:46:41.031 22004-22066 D/my_tag: Sort time: 19
2020-06-13 14:46:41.324 22004-22004 D/my_tag: Updating sorted list
2020-06-13 14:46:41.324 22004-22004 D/my_tag: Observe sorted list

I think this lag is because Dispatchers.Main consists of only the main thread, which may be doing something else, at that time. Isn't it so? And, how should I remove this lag, or have a faster workaround around this? Thanks.