How to use Kotlin coroutines with two for loops and channels that update each other?

lrc 0
Viewed 57

I am trying to learn more about Kotlin coroutines and channels by building a web crawler, modelled by this (gopl.io/ch8/crawl3)

The idea is that a set of coroutines are launched, which loop through a channel of links to visit linksToVisit, and push all found links to another channel, foundLinks.

A separate for loop then loops through foundLinks, checks if they have already been visited, and if not, pushes them back to linksToVisit for the coroutines to pick up.

My code so far seems to correctly visit all the links, but does not terminate - it hangs once all the links have been visited. What have I done wrong, and is it possible to use for loops in this way?

Here is the code:

fun crawl(startUrl: String) = runBlocking(CoroutineScope(Dispatchers.IO).coroutineContext) {

    val linksToVisit = Channel<String>()
    launch { linksToVisit.send(startUrl) }
    val foundLinks = Channel<List<String>>()

    repeat(20) {
        launch {
            for (channel in linksToVisit) {
                val links = findLinks(channel)
                launch { foundLinks.send(links) }
            }
        }
    }

    val visitedLinks = mutableMapOf<String, Boolean>()

    for (links in foundLinks) {
        for (link in links) {
            if (!visitedLinks.contains(link)) {
                visitedLinks[link] = true
                linksToVisit.send(link)
            }
        }
    }
}

Where findLinks(channel) retrieves the page (using JSoup) & returns a list of found links.

Side question: is JSoup compatible with coroutines?

channel kotlin-coroutines kotlin
1 Answers

You need to call close() on the channel once there are no more links to process. Once the channel is closed, and all remaining elements have been received, the for loop will finish.

userPsnj0
Related