async generators
This commit is contained in:
Ilya Kantor
parent
2ee2751216
commit
1369332661
3 changed files with 322 additions and 10 deletions
|
|
@ -152,7 +152,7 @@ alert(sequence); // 0, 1, 2, 3
|
||||||
|
|
||||||
In the code above, `...generateSequence()` turns the iterable into array of items (read more about the spread operator in the chapter [](info:rest-parameters-spread-operator#spread-operator))
|
In the code above, `...generateSequence()` turns the iterable into array of items (read more about the spread operator in the chapter [](info:rest-parameters-spread-operator#spread-operator))
|
||||||
|
|
||||||
## Converting iterables to generators
|
## Using generators instead of iterables
|
||||||
|
|
||||||
Some time ago, in the chapter [](info:iterable) we created an iterable `range` object that returns values `from..to`.
|
Some time ago, in the chapter [](info:iterable) we created an iterable `range` object that returns values `from..to`.
|
||||||
|
|
||||||
|
|
@ -203,6 +203,8 @@ alert(sequence); // 1, 2, 3, 4, 5
|
||||||
|
|
||||||
...But what if we'd like to keep a custom `range` object?
|
...But what if we'd like to keep a custom `range` object?
|
||||||
|
|
||||||
|
## Converting Symbol.iterator to generator
|
||||||
|
|
||||||
We can get the best from both worlds by providing a generator as `Symbol.iterator`:
|
We can get the best from both worlds by providing a generator as `Symbol.iterator`:
|
||||||
|
|
||||||
```js run
|
```js run
|
||||||
|
|
@ -220,7 +222,16 @@ let range = {
|
||||||
alert( [...range] ); // 1,2,3,4,5
|
alert( [...range] ); // 1,2,3,4,5
|
||||||
```
|
```
|
||||||
|
|
||||||
The `range` object is now iterable. The last variant with a generator is much more concise than the original iterable code, and keeps the same functionality.
|
The `range` object is now iterable.
|
||||||
|
|
||||||
|
That works pretty well, because when `range[Symbol.iterator]` is called:
|
||||||
|
- it returns an object (now a generator)
|
||||||
|
- that has `.next()` method (yep, a generator has it)
|
||||||
|
- that returns values in the form `{value: ..., done: true/false}` (check, exactly what generator does).
|
||||||
|
|
||||||
|
That's not a coincidence, of course. Generators aim to make iterables easier, so we can see that.
|
||||||
|
|
||||||
|
The last variant with a generator is much more concise than the original iterable code, and keeps the same functionality.
|
||||||
|
|
||||||
```smart header="Generators may continue forever"
|
```smart header="Generators may continue forever"
|
||||||
In the examples above we generated finite sequences, but we can also make a generator that yields values forever. For instance, an unending sequence of pseudo-random numbers.
|
In the examples above we generated finite sequences, but we can also make a generator that yields values forever. For instance, an unending sequence of pseudo-random numbers.
|
||||||
|
|
|
||||||
|
|
@ -1,20 +1,297 @@
|
||||||
|
|
||||||
# Async iteration and generators
|
# Async iteration and generators
|
||||||
|
|
||||||
In web-programming, we often need to work with fragmented data that comes piece-by-piece.
|
Asynchronous iterators allow to iterate over data that comes asynchronously, on-demand.
|
||||||
|
|
||||||
That happens when we upload or download a file. Or we need to fetch paginated data.
|
For instance, when we download something chunk-by-chunk, or just expect a events to come asynchronously and would like to iterate over that -- async iterators and generators may come in handy.
|
||||||
|
|
||||||
|
Let's see a simple example first, to grasp the syntax, and then build a real-life example.
|
||||||
|
|
||||||
For
|
## Async iterators
|
||||||
Either we need to download or upload something, or we need
|
|
||||||
|
|
||||||
In the previous chapter we saw how `async/await` allows to write asynchronous code.
|
Asynchronous iterators are totally similar to regular iterators, with a few syntactic differences.
|
||||||
|
|
||||||
But they don't solve
|
Here's a small cheatsheet:
|
||||||
|
|
||||||
|
| | Iterators | Async iterators |
|
||||||
|
|-------|-----------|-----------------|
|
||||||
|
| Object method to provide iteraterable | `Symbol.iterator` | `Symbol.asyncIterator` |
|
||||||
|
| `next()` return value is | any value | `Promise` |
|
||||||
|
| to loop, use | `for..of` | `for await..of` |
|
||||||
|
|
||||||
|
Let's make an iterable `range` object, like we did in the chapter [](info:iterable), but now it will return values asynchronously, one per second:
|
||||||
|
|
||||||
Regular iterators work fine with the data that doesn't take time to generate.
|
```js run
|
||||||
|
let range = {
|
||||||
|
from: 1,
|
||||||
|
to: 5,
|
||||||
|
|
||||||
A `for..of` loop assumes that we
|
// for await..of calls this method once in the very beginning
|
||||||
|
*!*
|
||||||
|
[Symbol.asyncIterator]() { // (1)
|
||||||
|
*/!*
|
||||||
|
// ...it returns the iterator object:
|
||||||
|
// onward, for await..of works only with that object, asking it for next values
|
||||||
|
return {
|
||||||
|
current: this.from,
|
||||||
|
last: this.to,
|
||||||
|
|
||||||
|
// next() is called on each iteration by the for..of loop
|
||||||
|
*!*
|
||||||
|
async next() { // (2)
|
||||||
|
// it should return the value as an object {done:.., value :...}
|
||||||
|
// (automatically wrapped into a promise by async)
|
||||||
|
*/!*
|
||||||
|
|
||||||
|
// can use await inside, do async stuff:
|
||||||
|
await new Promise(resolve => setTimeout(resolve, 1000)); // (4)
|
||||||
|
|
||||||
|
if (this.current <= this.last) {
|
||||||
|
return { done: false, value: this.current++ };
|
||||||
|
} else {
|
||||||
|
return { done: true };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
};
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
(async () => {
|
||||||
|
|
||||||
|
*!*
|
||||||
|
for await (let value of range) { // (3)
|
||||||
|
alert(value); // 1,2,3,4,5
|
||||||
|
}
|
||||||
|
*/!*
|
||||||
|
|
||||||
|
})()
|
||||||
|
```
|
||||||
|
|
||||||
|
As we can see, the components are similar to regular iterators:
|
||||||
|
|
||||||
|
1. To make an object asynchronously iterable, it must have a method `Symbol.asyncIterator` `(1)`.
|
||||||
|
2. To iterate, we use `for await(let value of range)` `(3)`, namely add "await" after "for".
|
||||||
|
3. It calls `range[Symbol.asyncIterator]` and expects it to return an object with `next` `(3)` method returning a promise.
|
||||||
|
4. Then `next()` is called to obtain values. In that example values come with a delay of 1 second `(4)`.
|
||||||
|
|
||||||
|
````warn header="A spread operator doesn't work asynchronously"
|
||||||
|
Features that require regular, synchronous iterators, don't work with asynchronous ones.
|
||||||
|
|
||||||
|
For instance, a spread operator won't work:
|
||||||
|
```js
|
||||||
|
alert( [...range] ); // Error, no Symbol.iterator
|
||||||
|
```
|
||||||
|
|
||||||
|
That's natural, as it expects to find `Symbol.iterator`, and there's none.
|
||||||
|
````
|
||||||
|
|
||||||
|
## Async generators
|
||||||
|
|
||||||
|
Javascript also provides generators, that are also iterable.
|
||||||
|
|
||||||
|
Let's recall a sequence generator from the chapter [](info:generators):
|
||||||
|
|
||||||
|
```js run
|
||||||
|
function* generateSequence(start, end) {
|
||||||
|
for (let i = start; i <= end; i++) {
|
||||||
|
yield i;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
for(let value of generateSequence(1, 5)) {
|
||||||
|
alert(value); // 1, then 2, then 3, then 4, then 5
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Normally, we can't use `await` in generators. But what if we need to?
|
||||||
|
|
||||||
|
No problem, let's make an async generator, like this:
|
||||||
|
|
||||||
|
```js run
|
||||||
|
*!*async*/!* function* generateSequence(start, end) {
|
||||||
|
|
||||||
|
for (let i = start; i <= end; i++) {
|
||||||
|
|
||||||
|
*!*
|
||||||
|
// yay, can use await!
|
||||||
|
await new Promise(resolve => setTimeout(resolve, 1000));
|
||||||
|
*/!*
|
||||||
|
|
||||||
|
yield i;
|
||||||
|
}
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
(async () => {
|
||||||
|
|
||||||
|
let generator = generateSequence(1, 5);
|
||||||
|
for *!*await*/!* (let value of generator) {
|
||||||
|
alert(value); // 1, then 2, then 3, then 4, then 5
|
||||||
|
}
|
||||||
|
|
||||||
|
})();
|
||||||
|
```
|
||||||
|
|
||||||
|
So simple, a little bit magical. We add the `async` keyword, and the generator now can use `await` inside of it, rely on promises and other async functions.
|
||||||
|
|
||||||
|
Technically, the difference of such generator is that `generator.next()` method is now asynchronous.
|
||||||
|
|
||||||
|
Instead of `result = generator.next()` for a regular, non-async generator, values can be obtained like this:
|
||||||
|
|
||||||
|
```js
|
||||||
|
result = await generator.next(); // result = {value: ..., done: true/false}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Iterables via async generators
|
||||||
|
|
||||||
|
When we'd like to make an object iterable, we should add `Symbol.iterator` to it. A common practice is to implement it via a generator, just because that's simple.
|
||||||
|
|
||||||
|
Let's recall an example from the chapter [](info:generators):
|
||||||
|
|
||||||
|
```js run
|
||||||
|
let range = {
|
||||||
|
from: 1,
|
||||||
|
to: 5,
|
||||||
|
|
||||||
|
*[Symbol.iterator]() { // a shorthand for [Symbol.iterator]: function*()
|
||||||
|
for(let value = this.from; value <= this.to; value++) {
|
||||||
|
yield value;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
for(let value of range) {
|
||||||
|
alert(value); // 1, then 2, then 3, then 4, then 5
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Here a custom object `range` was made iterable, the generator `*[Symbol.iterator]` implements the logic behind listing values.
|
||||||
|
|
||||||
|
We can make it iterable asynchronously by replacing `Symbol.iterator` with `Symbol.asyncIterator` and marking it `async`:
|
||||||
|
|
||||||
|
```js run
|
||||||
|
let range = {
|
||||||
|
from: 1,
|
||||||
|
to: 5,
|
||||||
|
|
||||||
|
*!*
|
||||||
|
async *[Symbol.asyncIterator]() { // same as [Symbol.asyncIterator]: async function*()
|
||||||
|
*/!*
|
||||||
|
for(let value = this.from; value <= this.to; value++) {
|
||||||
|
|
||||||
|
// make a pause between values, wait for something
|
||||||
|
await new Promise(resolve => setTimeout(resolve, 1000));
|
||||||
|
|
||||||
|
yield value;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
(async () => {
|
||||||
|
|
||||||
|
for *!*await*/!* (let value of range) {
|
||||||
|
alert(value); // 1, then 2, then 3, then 4, then 5
|
||||||
|
}
|
||||||
|
|
||||||
|
})();
|
||||||
|
```
|
||||||
|
|
||||||
|
## Real-life example
|
||||||
|
|
||||||
|
There are many online APIs that deliver paginated data. For instance, when we need a list of users, then we can fetch it page-by-page: a request returns a pre-defined count (e.g. 100 users), and provides an URL to the next page.
|
||||||
|
|
||||||
|
The pattern is very common, it's not about users, but just about anything. For instance, Github allows to retrieve commits in the same, paginated fasion:
|
||||||
|
|
||||||
|
- We should make a request to URL in the form `https://api.github.com/repos/(repo)/commits`.
|
||||||
|
- It responds with a JSON of 30 commits, and also provides a link to the next page in the `Link` header.
|
||||||
|
- Then we can use it for the next request, if need more commits, and so on.
|
||||||
|
|
||||||
|
What we'd like to have is an iterable source of commits, so that we could do it like this:
|
||||||
|
|
||||||
|
```js
|
||||||
|
let repo = 'iliakan/javascript-tutorial-en'; // Github repository to get commits from
|
||||||
|
|
||||||
|
for await (let commit of fetchCommits(repo)) {
|
||||||
|
// process commit
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
We'd like `fetchCommits` to get commits for us, making requests whenever needed. And let it care about all pagination stuff.
|
||||||
|
|
||||||
|
With async generators that's pretty easy to implement:
|
||||||
|
|
||||||
|
```js
|
||||||
|
async function* fetchCommits(repo) {
|
||||||
|
let url = `https://api.github.com/repos/${repo}/commits`;
|
||||||
|
|
||||||
|
while (url) {
|
||||||
|
const response = await fetch(url, { // (1)
|
||||||
|
headers: {'User-Agent': 'Our script'}, // github requires user-agent header
|
||||||
|
});
|
||||||
|
|
||||||
|
const body = await response.json(); // (2) parses response as JSON (array of commits)
|
||||||
|
|
||||||
|
// (3) the URL of the next page is in the headers, extract it
|
||||||
|
let nextPage = response.headers.get('Link').match(/<(.*?)>; rel="next"/);
|
||||||
|
nextPage = nextPage && nextPage[1];
|
||||||
|
|
||||||
|
url = nextPage;
|
||||||
|
|
||||||
|
for(let commit of body) { // (4) yield commits one by one, until the page ends
|
||||||
|
yield commit;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
1. We use the browser `fetch` method to download from a remote URL. It allows to supply authorization and other headers if needed (Github requires User-Agent).
|
||||||
|
2. The result is parsed as JSON.
|
||||||
|
3. ...And we extract the next page URL from the `Link` header of the response. It has a special format, so we use a regexp for that. The next page URL may look like this: `https://api.github.com/repositories/93253246/commits?page=2`, it's generatd by Github itself.
|
||||||
|
4. Then we yield all commits received, and when they finish -- the next `while(url)` iteration will trigger, making one more request.
|
||||||
|
|
||||||
|
An example of use (shows commit authors in console):
|
||||||
|
|
||||||
|
```js run
|
||||||
|
(async () => {
|
||||||
|
|
||||||
|
let count = 0;
|
||||||
|
|
||||||
|
for await (const commit of fetchCommits('iliakan/javascript-tutorial-en')) {
|
||||||
|
|
||||||
|
console.log(commit.author.login);
|
||||||
|
|
||||||
|
if (++count == 100) { // let's stop at 100 commits
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
})();
|
||||||
|
```
|
||||||
|
|
||||||
|
The internal mechanics is invisible from the outside. What we see -- is an async generator that returns commits, just what we need.
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Regular iterators and generators work fine with the data that doesn't take time to generate.
|
||||||
|
|
||||||
|
When we expect the data to come asynchronously, with delays, their async counterparts can be used, and `for await..of` instead of `for..of`.
|
||||||
|
|
||||||
|
Syntax differences between async and regular iterators:
|
||||||
|
|
||||||
|
| | Iterators | Async iterators |
|
||||||
|
|-------|-----------|-----------------|
|
||||||
|
| Object method to provide iteraterable | `Symbol.iterator` | `Symbol.asyncIterator` |
|
||||||
|
| `next()` return value is | any value | `Promise` |
|
||||||
|
|
||||||
|
Syntax differences between async and regular generators:
|
||||||
|
|
||||||
|
| | Generators | Async generators |
|
||||||
|
|-------|-----------|-----------------|
|
||||||
|
| Declaration | `function*` | `async function*` |
|
||||||
|
| `generator.next()` returns | `{value:…, done: true/false}` | `Promise` that resolves to `{value:…, done: true/false}` |
|
||||||
|
|
||||||
|
In web-development we often meet streams of data, when it flows chunk-by-chunk. For instance, downloading or uploading a big file.
|
||||||
|
|
||||||
|
We could use async generators to process such data, but there's also another API called Streams, that may be more convenient. It's not a part of Javascript language standard though.
|
||||||
|
|
||||||
|
Streams and async generators complement each other, both are great ways to handle async data flows.
|
||||||
|
|
|
||||||
24
1-js/09-async/07-async-iteration-generators/head.html
Normal file
24
1-js/09-async/07-async-iteration-generators/head.html
Normal file
|
|
@ -0,0 +1,24 @@
|
||||||
|
<script>
|
||||||
|
async function* fetchCommits(repo) {
|
||||||
|
let url = `https://api.github.com/repos/${repo}/commits`;
|
||||||
|
|
||||||
|
while (url) {
|
||||||
|
const response = await fetch(url, {
|
||||||
|
headers: {'User-Agent': 'Our script'}, // github requires user-agent header
|
||||||
|
});
|
||||||
|
|
||||||
|
const body = await response.json(); // parses response as JSON (array of commits)
|
||||||
|
|
||||||
|
// the URL of the next page is in the headers, extract it
|
||||||
|
let nextPage = response.headers.get('Link').match(/<(.*?)>; rel="next"/);
|
||||||
|
nextPage = nextPage && nextPage[1];
|
||||||
|
|
||||||
|
url = nextPage;
|
||||||
|
|
||||||
|
// yield commits one by one, when they finish - fetch a new page url
|
||||||
|
for(let commit of body) {
|
||||||
|
yield commit;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
</script>
|
||||||
Loading…
Add table
Add a link
Reference in a new issue