en.javascript.info/1-js/09-async/07-async-iteration-generators/article.md

Async iteration and generators

Asynchronous iterators allow to iterate over data that comes asynchronously, on-demand.

For instance, when we download something chunk-by-chunk, or just expect a events to come asynchronously and would like to iterate over that -- async iterators and generators may come in handy.

Let's see a simple example first, to grasp the syntax, and then build a real-life example.

Async iterators

Asynchronous iterators are totally similar to regular iterators, with a few syntactic differences.

Here's a small cheatsheet:

	Iterators	Async iterators
Object method to provide iteraterable	Symbol.iterator	Symbol.asyncIterator
next() return value is	any value	Promise
to loop, use	for..of	for await..of

Let's make an iterable range object, like we did in the chapter , but now it will return values asynchronously, one per second:

let range = {
  from: 1,
  to: 5,

  // for await..of calls this method once in the very beginning
*!*
  [Symbol.asyncIterator]() { // (1)
*/!*
    // ...it returns the iterator object:
    // onward, for await..of works only with that object, asking it for next values
    return {
      current: this.from,
      last: this.to,

      // next() is called on each iteration by the for..of loop
*!*
      async next() { // (2)
        // it should return the value as an object {done:.., value :...}
        // (automatically wrapped into a promise by async)
*/!*

        // can use await inside, do async stuff:
        await new Promise(resolve => setTimeout(resolve, 1000)); // (4)

        if (this.current <= this.last) {
          return { done: false, value: this.current++ };
        } else {
          return { done: true };
        }
      }
    };
  }
};

(async () => {

*!*
  for await (let value of range) { // (3)
    alert(value); // 1,2,3,4,5
  }
*/!*

})()

As we can see, the components are similar to regular iterators:

1. To make an object asynchronously iterable, it must have a method Symbol.asyncIterator (1).
2. To iterate, we use for await(let value of range) (3), namely add "await" after "for".
3. It calls range[Symbol.asyncIterator] and expects it to return an object with next (3) method returning a promise.
4. Then next() is called to obtain values. In that example values come with a delay of 1 second (4).

Features that require regular, synchronous iterators, don't work with asynchronous ones.

For instance, a spread operator won't work:
```js
alert( [...range] ); // Error, no Symbol.iterator
```

That's natural, as it expects to find `Symbol.iterator`, and there's none.

Async generators

Javascript also provides generators, that are also iterable.

Let's recall a sequence generator from the chapter :

function* generateSequence(start, end) {
  for (let i = start; i <= end; i++) {
    yield i;
  }
}

for(let value of generateSequence(1, 5)) {
  alert(value); // 1, then 2, then 3, then 4, then 5
}

Normally, we can't use await in generators. But what if we need to?

No problem, let's make an async generator, like this:

*!*async*/!* function* generateSequence(start, end) {

  for (let i = start; i <= end; i++) {

*!*
    // yay, can use await!
    await new Promise(resolve => setTimeout(resolve, 1000));
*/!*

    yield i;
  }

}

(async () => {

  let generator = generateSequence(1, 5);
  for *!*await*/!* (let value of generator) {
    alert(value); // 1, then 2, then 3, then 4, then 5
  }

})();

So simple, a little bit magical. We add the async keyword, and the generator now can use await inside of it, rely on promises and other async functions.

Technically, the difference of such generator is that generator.next() method is now asynchronous.

Instead of result = generator.next() for a regular, non-async generator, values can be obtained like this:

result = await generator.next(); // result = {value: ..., done: true/false}

Iterables via async generators

When we'd like to make an object iterable, we should add Symbol.iterator to it. A common practice is to implement it via a generator, just because that's simple.

Let's recall an example from the chapter :

let range = {
  from: 1,
  to: 5,

  *[Symbol.iterator]() { // a shorthand for [Symbol.iterator]: function*()
    for(let value = this.from; value <= this.to; value++) {
      yield value;
    }
  }
};

for(let value of range) {
  alert(value); // 1, then 2, then 3, then 4, then 5
}

Here a custom object range was made iterable, the generator *[Symbol.iterator] implements the logic behind listing values.

We can make it iterable asynchronously by replacing Symbol.iterator with Symbol.asyncIterator and marking it async:

let range = {
  from: 1,
  to: 5,

*!*
  async *[Symbol.asyncIterator]() { // same as [Symbol.asyncIterator]: async function*()
*/!*
    for(let value = this.from; value <= this.to; value++) {

      // make a pause between values, wait for something  
      await new Promise(resolve => setTimeout(resolve, 1000));

      yield value;
    }
  }
};

(async () => {

  for *!*await*/!* (let value of range) {
    alert(value); // 1, then 2, then 3, then 4, then 5
  }

})();

Real-life example

There are many online APIs that deliver paginated data. For instance, when we need a list of users, then we can fetch it page-by-page: a request returns a pre-defined count (e.g. 100 users), and provides an URL to the next page.

The pattern is very common, it's not about users, but just about anything. For instance, Github allows to retrieve commits in the same, paginated fasion:

• We should make a request to URL in the form https://api.github.com/repos/(repo)/commits.
• It responds with a JSON of 30 commits, and also provides a link to the next page in the Link header.
• Then we can use it for the next request, if need more commits, and so on.

What we'd like to have is an iterable source of commits, so that we could do it like this:

let repo = 'iliakan/javascript-tutorial-en'; // Github repository to get commits from

for await (let commit of fetchCommits(repo)) {
  // process commit
}

We'd like fetchCommits to get commits for us, making requests whenever needed. And let it care about all pagination stuff.

With async generators that's pretty easy to implement:

async function* fetchCommits(repo) {
  let url = `https://api.github.com/repos/${repo}/commits`;

  while (url) {
    const response = await fetch(url, { // (1)
      headers: {'User-Agent': 'Our script'}, // github requires user-agent header
    });

    const body = await response.json(); // (2) parses response as JSON (array of commits)

    // (3) the URL of the next page is in the headers, extract it
    let nextPage = response.headers.get('Link').match(/<(.*?)>; rel="next"/);
    nextPage = nextPage && nextPage[1];

    url = nextPage;

    for(let commit of body) { // (4) yield commits one by one, until the page ends
      yield commit;
    }
  }
}

1. We use the browser fetch method to download from a remote URL. It allows to supply authorization and other headers if needed (Github requires User-Agent).
2. The result is parsed as JSON.
3. ...And we extract the next page URL from the Link header of the response. It has a special format, so we use a regexp for that. The next page URL may look like this: https://api.github.com/repositories/93253246/commits?page=2, it's generatd by Github itself.
4. Then we yield all commits received, and when they finish -- the next while(url) iteration will trigger, making one more request.

An example of use (shows commit authors in console):

(async () => {

  let count = 0;

  for await (const commit of fetchCommits('iliakan/javascript-tutorial-en')) {

    console.log(commit.author.login);

    if (++count == 100) { // let's stop at 100 commits
      break;
    }
  }

})();

The internal mechanics is invisible from the outside. What we see -- is an async generator that returns commits, just what we need.

Summary

Regular iterators and generators work fine with the data that doesn't take time to generate.

When we expect the data to come asynchronously, with delays, their async counterparts can be used, and for await..of instead of for..of.

Syntax differences between async and regular iterators:

	Iterators	Async iterators
Object method to provide iteraterable	Symbol.iterator	Symbol.asyncIterator
next() return value is	any value	Promise

Syntax differences between async and regular generators:

	Generators	Async generators
Declaration	function*	async function*
generator.next() returns	{value:…, done: true/false}	Promise that resolves to {value:…, done: true/false}

In web-development we often meet streams of data, when it flows chunk-by-chunk. For instance, downloading or uploading a big file.

We could use async generators to process such data, but there's also another API called Streams, that may be more convenient. It's not a part of Javascript language standard though.

Streams and async generators complement each other, both are great ways to handle async data flows.