How to map over async generators?

user_931559 0
Viewed 1270

Let's say we have an async generator:

exports.asyncGen = async function* (items) {
  for (const item of items) {
    const result = await someAsyncFunc(item)
    yield result;
  }
}

is it possible to map over this generator? Essentially I want to do this:

const { asyncGen } = require('./asyncGen.js')

exports.process = async function (items) {
  return asyncGen(items).map(item => {
    //... do something
  })
}

As of now .map fails to recognize async iterator.

The alternative is to use for await ... of but that's nowhere near elegant as with .map

async-iterator node.js map-function iterator javascript
3 Answers

The iterator methods proposal that would provide this method is still at stage 2 only. You can use some polyfill, or write your own map helper function though:

async function* map(asyncIterable, callback) {
    let i = 0;
    for await (const val of asyncIterable)
        yield callback(val, i++);
}

exports.process = function(items) {
    return map(asyncGen(items), item => {
       //... do something
    });
};
user_10485720

The alternative is to use for await ... of, but that's nowhere near elegant as with .map

For an elegant and efficient solution, here's one using iter-ops library:

import {pipe, map} from 'iter-ops';

const i = pipe(
    asyncGen(), // your async generator result
    map(value => /*map logic*/)
); //=> AsyncIterable
  • It is elegant, because the syntax is clean, simple, and applicable to any iterable or iterators, not just asynchronous generators.
  • It is more flexible and reusable, as you can add lots of other operators to the same pipeline.

Since it produces a standard JavaScript AsyncIterable, you can do:

for await(const a of i) {
    console.log(a); //=> print values
}

P.S. I'm the author of iter-ops.

user_11020510

TL;DR - If the mapping function is async:

To make asyncIter not wait for each mapping before producing the next value, do

async function asyncIterMap(asyncIter, asyncFunc) {
    const promises = [];
    for await (const value of asyncIter) {
        promises.push(asyncFunc(value))
    }
    return await Promise.all(promises)
}

// example - how to use:
const results = await asyncIterMap(myAsyncIter(), async (str) => {
    await sleep(3000)
    return str.toUpperCase()
});

More Demoing:

// dummy asyncIter for demonstration

const sleep = (ms) => new Promise(res => setTimeout(res, ms))

async function* myAsyncIter() {
    await sleep(1000)
    yield 'first thing'
    await sleep(1000)
    yield 'second thing'
    await sleep(1000)
    yield 'third thing'
}

Then

// THIS IS BAD! our asyncIter waits for each mapping.

for await (const thing of myAsyncIter()) {
    console.log('starting with', thing)
    await sleep(3000)
    console.log('finished with', thing)
}

// total run time: ~12 seconds

Better version:

// this is better.

const promises = [];

for await (const thing of myAsyncIter()) {
    const task = async () => {
        console.log('starting with', thing)
        await sleep(3000)
        console.log('finished with', thing)
    };
    promises.push(task())
}

await Promise.all(promises)

// total run time: ~6 seconds
user_83399600
Related