Sarp IŞIK brand logo.

How To Iterate Over An Array Asynchronously

2020-09-21 - by Sarp IŞIK

Photo by Tine Ivanič on Unsplash

Event loop is the heart of the Node js and because of the single-threaded nature of javascript, we need to handle things in an async way as much as possible to prevent event loop being blocked especially in a server-side environment. Today I will try to explain how to iterate over an array of elements asynchronously so that we don't block the event loop.

Prerequisites

Note: The following examples are executed in Node js v13.7.0 environment.

Table of Contents

Iterating Synchronously

For simplicity, I will create a small array of numbers (1, 2, 3) but in the real world case, this can be a huge array of elements.

const arr = Array(3)
    .fill(0)
    .map((_, i) => ++i);

Let's write our first function which is iterating synchronously so it blocks the event-loop. 

function syncLoop(arr, cb) {
    for (const element of arr) {
        cb(element);
    }
}

I will also execute the following nested "setImmediate" callbacks to understand that if the execution happens on the same iteration of the event-loop or not. The reason I use "setImmediate" callback here is it calls the passed callback function once on the related event-loop iteration. In short, each "setImmediate" represents the different event-loop iteration.

setImmediate(() => {
    console.log('Should logged "1" on the first iteration.');
    setImmediate(() => {
        console.log('Should logged "2" on the second iteration.');
        setImmediate(() => {
            console.log('Should logged "3" on the third iteration.');
        });
    });
});

When we run the following code:

const arr = Array(3)
    .fill(0)
    .map((_, i) => ++i);
setImmediate(() => {
    console.log('Should logged "1" on the first iteration.');
    setImmediate(() => {
        console.log('Should logged "2" on the second iteration.');
        setImmediate(() => {
            console.log('Should logged "3" on the third iteration.');
        });
    });
});
console.log('before loop');
syncLoop(arr, console.log);
console.log('after loop');
function syncLoop(arr, cb) {
    for (const element of arr) {
        cb(element);
    }
}

The result will be:

before loop
1
2
3
after loop
Should logged "1" on the first iteration.
Should logged "2" on the second iteration.
Should logged "3" on the third iteration.

The above log statements mean we are blocking the event-loop during the "syncLoop" execution. If the array was iterated asynchronously, each "Should logged..." statement would be logged sequentially like the following:

before loop
1
after loop
Should logged "1" on the first iteration.
2
Should logged "2" on the second iteration.
3
Should logged "3" on the third iteration.

Iterating Synchronously Inside Promise

This time let's define a new function which has the same functionality but returns a promise:

function promiseLoop(arr, cb) {
    return new Promise((resolve, reject) => {
        try {
            for (const element of arr) {
                cb(element);
            }
            resolve();
        } catch (error) {
            reject(error);
        }
    });
}

As you notice everything almost the same inside the promise callback except calling the "resolve" callback at the end of the for-loop iteration. Let's call the new function:

...
// syncLoop(arr, console.log);
promiseLoop(arr, console.log); 
...

After the above execution the result should be:

before loop
1
2
3
after loop
Should logged "1" on the first iteration.
Should logged "2" on the second iteration.
Should logged "3" on the third iteration.

Even though we wrapped our loop function inside a promise, the execution order is the still same so that we are still blocking the event-loop on the first iteration.

Iterating Synchronously Inside Async/Await

Wrapping by a promise had no effect on the order of execution. Let's see what happens if we execute the same function in an async/await way. Below is a new function:

async function asyncAwaitLoop(arr, cb) {
    for (const element of arr) {
        await cb(element);
    }
}

This is the updated execution:

...
console.log("before loop");
// syncLoop(arr, console.log);
// promiseLoop(arr, console.log);
asyncAwaitLoop(arr, console.log); 
console.log("after loop");
...

and the result is:

before loop
1
after loop
2
3
Should logged "1" on the first iteration.
Should logged "2" on the second iteration.
Should logged "3" on the third iteration.

Well, this time we can see some differences. The execution order has changed after the first iteration of the for-loop. That means the rest iterations executed as microtasks but we can see that all of the for-loop iterations still happening on the same iteration of the event-loop because the log statements of "setImmediate" callbacks are happening on the last.

Iterating Asynchronously

So far, we executed all for-loop iterations on the same iteration of the event-loop which is not what we wanted. It is time to implement the solution. First, let's define the functions to rescue:

function asyncLoop(arr, cb) {
    return new Promise((resolve, reject) => {
        recursivelyAsyncLoop(arr, cb, resolve).catch(reject);
    });
}
async function recursivelyAsyncLoop(arr, cb, resolve, i = 0) {
    if (i === arr.length) resolve();
    else {
        const element = arr[i];
        cb(element);
        setImmediate(() => {
            recursivelyAsyncLoop(arr, cb, resolve, ++i);
        });
    }
}

The above solution has two parts: First, we define a function named "asyncLoop" which is a promise-wrapper function so that we don't block the current stack. Second, we define a function named "recursivelyAsyncLoop" which -as its name suggests- recursively calls itself until the iteration of all array elements completed. The real difference from the above for-loop based functions is we are calling recursively inside the "setImmediate" callback so that we are telling the event-loop that "when you finish your phases on the current iteration, call the passed function inside the setImmediate on your next iteration". Let's see what does it mean on the action:

...
console.log("before loop");
// syncLoop(arr, console.log);
// promiseLoop(arr, console.log);
// asyncAwaitLoop(arr, console.log);
asyncLoop(arr, console.log); 
console.log("after loop");
...

The result:

before loop
1
after loop
Should logged "1" on the first iteration.
2
Should logged "2" on the second iteration.
3
Should logged "3" on the third iteration.

As stated before, instead of creating a blocking-loop via for-loop (this can be a native method like "forEach"), we are using the native loop which is happening inside the Node js environment thanks to "setImmediate" callback. The above solution can be extended to implement the async versions of array methods such as map, filter, some, every, etc...

References

© 2020, Sarp IŞIK