This commit is contained in:
Ilya Kantor 2019-03-20 15:09:06 +03:00
parent 31f83a5e1b
commit 4ffd79e337
65 changed files with 1481 additions and 28000 deletions

View file

@ -204,7 +204,7 @@ We may have both default and named exports in a single module, but in practice p
**Another thing to note is that named exports must (naturally) have a name, while `export default` may be anonymous.** **Another thing to note is that named exports must (naturally) have a name, while `export default` may be anonymous.**
For instance, these are all perfecly valid default exports: For instance, these are all perfectly valid default exports:
```js ```js
export default class { // no class name export default class { // no class name

View file

@ -121,9 +121,34 @@ The example below correctly shows image sizes, because `window.onload` waits for
## window.onunload ## window.onunload
When a visitor leaves the page, the `unload` event triggers on `window`. We can do something there that doesn't involve a delay, like closing related popup windows. But we can't cancel the transition to another page. When a visitor leaves the page, the `unload` event triggers on `window`. We can do something there that doesn't involve a delay, like closing related popup windows.
For that we should use another event -- `onbeforeunload`. This event is a good place to send out analytics.
E.g. we have a script that gathers some data about mouse clicks, scrolls, viewed page areas -- the statistics that can help us to see what users want.
Then `onunload` is the best time to send it out. Regular networking methods such as [fetch](info:fetch-basics) or [XMLHttpRequest](info:xmlhttprequest) don't work well, because we're in the process of leaving the page.
So, there exist `navigator.sendBeacon(url, data)` method for such needs, described in the specification <https://w3c.github.io/beacon/>.
It sends the data in background. The transition to another page is not delayed: the browser leaves the page and performs `sendBeacon` in background.
Here's how to use it:
```js
let analyticsData = { /* object with gathered data */ };
window.addEventListener("unload", function() {
navigator.sendBeacon("/analytics", JSON.stringify(analyticsData));
};
```
- The request is sent as POST.
- We can send not only a string, but also forms and other formats, as described in the chapter <info:fetch-basics>, but usually it's a stringified object.
- The data is limited by 64kb.
When the `sendBeacon` request is finished, the browser probably has already left the document, so there's no way to get server response (which is usually empty for analytics).
If we want to cancel the transition to another page, we can't do it here. But we can use another event -- `onbeforeunload`.
## window.onbeforeunload [#window.onbeforeunload] ## window.onbeforeunload [#window.onbeforeunload]

View file

@ -1,17 +1,16 @@
# Cookies, document.cookie # Cookies, document.cookie
Cookies are small strings of data that are stored directly in the browser. They are not a part of Javascript, but rather a part of HTTP protocol, defined by [RFC 6265](https://tools.ietf.org/html/rfc6265) specification. Cookies are small strings of data that are stored directly in the browser. They are a part of HTTP protocol, defined by [RFC 6265](https://tools.ietf.org/html/rfc6265) specification.
Most of the time, cookies are set by a web server. Most of the time, cookies are set by a web server. Then they are automatically added to every request to the same domain.
One of the most widespread use of cookies is authentication: One of the most widespread use cases is authentication:
1. Upon sign in, the server uses `Set-Cookie` HTTP-header in a response to set a cookie with "session identifier". 1. Upon sign in, the server uses `Set-Cookie` HTTP-header in the response to set a cookie with "session identifier".
2. The browser stores the cookie. 2. Next time when the request is set to the same domain, the browser sends the over the net using `Cookie` HTTP-header.
3. Next time when the request is set to the same domain, the browser sends the over the net using `Cookie` HTTP-header. 3. So the server knows who made the request.
4. So the server knows who made the request.
The browser provides a special accessor `document.cookie` for cookies. We can also access cookies from the browser, using `document.cookie` property.
There are many tricky things about cookies and their options. In this chapter we'll cover them in detail. There are many tricky things about cookies and their options. In this chapter we'll cover them in detail.
@ -32,11 +31,11 @@ alert( document.cookie ); // cookie1=value1; cookie2=value2;...
``` ```
The string consist of `name=value` pairs, delimited by `; `. Each one is a separate cookie. The value of `document.cookie` consists of `name=value` pairs, delimited by `; `. Each one is a separate cookie.
To find a particular cookie, we can split `document.cookie` by `; `, and then find the right name. We can use either a regular expression or array functions to do that. To find a particular cookie, we can split `document.cookie` by `; `, and then find the right name. We can use either a regular expression or array functions to do that.
To make things simple, at the end of the chapter you'll find a few functions to manipulate cookies. We leave it as an excercise for the reader. Also, at the end of the chapter you'll find helper functions to manipulate cookies.
## Writing to document.cookie ## Writing to document.cookie
@ -51,26 +50,26 @@ document.cookie = "user=John"; // update only cookie named 'user'
alert(document.cookie); // show all cookies alert(document.cookie); // show all cookies
``` ```
If you run it, then probably you'll see multiple cookies. That's because `document.cookie=` operation does not overwrite all cookies, but only `user`. If you run it, then probably you'll see multiple cookies. That's because `document.cookie=` operation does not overwrite all cookies. It only sets the mentioned cookie `user`.
Technically, name and value can have any characters, but to keep the formatting valid they should be escaped using a built-in `encodeURIComponent` function: Technically, name and value can have any characters, but to keep the formatting valid they should be escaped using a built-in `encodeURIComponent` function:
```js run ```js run
// special values, need encoding // special values, need encoding
let name = "<>"; let name = "my name";
let value = "=" let value = "John Smith"
// encodes the cookie as %3C%3E=%3D // encodes the cookie as my%20name=John%20Smith
document.cookie = encodeURIComponent(name) + '=' + encodeURIComponent(value); document.cookie = encodeURIComponent(name) + '=' + encodeURIComponent(value);
alert(document.cookie); // ...; %3C%3E=%3D alert(document.cookie); // ...; my%20name=John%20Smith
``` ```
```warn header="Limitations" ```warn header="Limitations"
There are few limitations: There are few limitations:
- The `name=value` pair, after `encodeURIComponent`, should not exceed 4kb. So we can't store anything huge in a cookie. - The `name=value` pair, after `encodeURIComponent`, should not exceed 4kb. So we can't store anything huge in a cookie.
- The total number of cookies per domain is limited to 20+, depending on a browser. - The total number of cookies per domain is limited to around 20+, the exact limit depends on a browser.
``` ```
Cookies have several options, many of them are important and should be set. Cookies have several options, many of them are important and should be set.
@ -87,7 +86,7 @@ document.cookie = "user=John; path=/; expires=Tue, 19 Jan 2038 03:14:07 GMT"
The url path prefix, where the cookie is accessible. Must be absolute. By default, it's the current path. The url path prefix, where the cookie is accessible. Must be absolute. By default, it's the current path.
If a cookie is set with `path=/mypath`, it's visible at `/mypath` and `/mypath/*`, but not at `/page` or `/mypathpage`. If a cookie is set with `path=/admin`, it's visible at pages `/admin` and `/admin/something`, but not at `/home` or `/adminpage`.
Usually, we set `path=/` to make the cookie accessible from all website pages. Usually, we set `path=/` to make the cookie accessible from all website pages.
@ -95,11 +94,11 @@ Usually, we set `path=/` to make the cookie accessible from all website pages.
- **`domain=site.com`** - **`domain=site.com`**
Domain where the cookie is accessible. A domain where the cookie is accessible. In practice though, there are limitations. We can't set any domain.
By default, a cookie is accessible only at the domain that set it. So, if the cookie was set by `site.com`, we won't get it `other.com`. By default, a cookie is accessible only at the domain that set it. So, if the cookie was set by `site.com`, we won't get it `other.com`.
...But what's more tricky, we also won't get the cookie at a subdomain `forum.site.com`: ...But what's more tricky, we also won't get the cookie at a subdomain `forum.site.com`!
```js ```js
// at site.com // at site.com
@ -125,6 +124,8 @@ alert(document.cookie); // with user
For historical reasons, `domain=.site.com` (a dot at the start) also works this way, it might better to add the dot to support very old browsers. For historical reasons, `domain=.site.com` (a dot at the start) also works this way, it might better to add the dot to support very old browsers.
So, `domain` option allows to make a cookie accessible at subdomains.
## expires, max-age ## expires, max-age
By default, if a cookie doesn't have one of these options, it disappears when the browser is closed. Such cookies are called "session cookies" By default, if a cookie doesn't have one of these options, it disappears when the browser is closed. Such cookies are called "session cookies"
@ -144,13 +145,13 @@ date = date.toUTCString();
document.cookie = "user=John; expires=" + date; document.cookie = "user=John; expires=" + date;
``` ```
If we set `expires` to the past, the cookie will be deleted. If we set `expires` to a date in the past, the cookie is deleted.
- **`max-age=3600`** - **`max-age=3600`**
An alternative to `expires`, specifies the cookie expiration in seconds. An alternative to `expires`, specifies the cookie expiration in seconds from the current moment.
Can be either a number of seconds from the current moment, or zero/negative for immediate expiration (to remove the cookie): If zero or negative, then the cookie is deleted:
```js ```js
// cookie will die +1 hour from now // cookie will die +1 hour from now
@ -168,11 +169,12 @@ The cookie should be transferred only over HTTPS.
**By default if we set a cookie at `http://site.com`, then it also appears at `https://site.com` and vise versa.** **By default if we set a cookie at `http://site.com`, then it also appears at `https://site.com` and vise versa.**
That is, cookies only check the domain, they do not distinguish between the protocols. That is, cookies are domain-based, they do not distinguish between the protocols.
With this option, if a cookie is set by `https://site.com`, then it doesn't appear when the same site is accessed by HTTP, as `http://site.com`. So if a cookie has sensitive content that should never be sent over unencrypted HTTP, then the flag can prevent this. With this option, if a cookie is set by `https://site.com`, then it doesn't appear when the same site is accessed by HTTP, as `http://site.com`. So if a cookie has sensitive content that should never be sent over unencrypted HTTP, then the flag is the right thing.
```js ```js
// assuming we're on https:// now
// set the cookie secure (only accessible if over HTTPS) // set the cookie secure (only accessible if over HTTPS)
document.cookie = "user=John; secure"; document.cookie = "user=John; secure";
``` ```
@ -185,9 +187,9 @@ To understand when it's useful, let's introduce the following attack scenario.
### XSRF attack ### XSRF attack
Imagine, you are logged into the site `bank.com`. That is: you have an authentication cookie from that site. Your browser sends it to `bank.com` on every request, so that it recognizes you and performs all sensitive financial operations. Imagine, you are logged into the site `bank.com`. That is: you have an authentication cookie from that site. Your browser sends it to `bank.com` with every request, so that it recognizes you and performs all sensitive financial operations.
Now, while browsing the web in another window, you occasionally come to another site `evil.com`, that has a `<form action="https://bank.com/pay">` with hacker's account and JavaScript code that submits it automatically. Now, while browsing the web in another window, you occasionally come to another site `evil.com`, that automatically submits a form `<form action="https://bank.com/pay">` to `bank.com` with input fields that initiate a transaction to the hacker's account.
The form is submitted from `evil.com` directly to the bank site, and your cookie is also sent, just because it's sent every time you visit `bank.com`. So the bank recognizes you and actually performs the payment. The form is submitted from `evil.com` directly to the bank site, and your cookie is also sent, just because it's sent every time you visit `bank.com`. So the bank recognizes you and actually performs the payment.
@ -197,31 +199,33 @@ That's called a cross-site request forgery (or XSRF) attack.
Real banks are protected from it of course. All forms generated by `bank.com` have a special field, so called "xsrf protection token", that an evil page can't neither generate, nor somehow extract from a remote page (it can submit a form there, but can't get the data back). Real banks are protected from it of course. All forms generated by `bank.com` have a special field, so called "xsrf protection token", that an evil page can't neither generate, nor somehow extract from a remote page (it can submit a form there, but can't get the data back).
But that takes time to implement: we need to ensure that every form has the token field, and we must also check all requests.
### Enter cookie samesite option ### Enter cookie samesite option
Now, cookie `samesite` option provides another way to protect from such attacks, that (in theory) should not require "xsrf protection tokens". The cookie `samesite` option provides another way to protect from such attacks, that (in theory) should not require "xsrf protection tokens".
It has two possible values: It has two possible values:
- **`samesite=strict`, same as `samesite` without value** - **`samesite=strict` (same as `samesite` without value)**
A cookie with `samesite=strict` is never sent if the user comes from outside the site. A cookie with `samesite=strict` is never sent if the user comes from outside the site.
In other words, whether a user follows a link from the mail or submits a form from `evil.com`, or does any operation with the site that originates from another domain, the cookie is not sent. Then the XSRF attack will fail, as `bank.com` will not recognize the user without the cookie, and will not proceed with the payment. In other words, whether a user follows a link from their mail or submits a form from `evil.com`, or does any operation that originates from another domain, the cookie is not sent.
The protection is quite reliable. Only operations that come from `bank.com` will send the samesite cookie. If authentication cookies have `samesite` option, then XSRF attack has no chances to succeed, because a submission from `evil.com` comes without cookies. So `bank.com` will not recognize the user and will not proceed with the payment.
The protection is quite reliable. Only operations that come from `bank.com` will send the `samesite` cookie.
Although, there's a small inconvenience. Although, there's a small inconvenience.
When a user follows a legitimate link to `bank.com`, like from their own notes, they'll be surprised that `bank.com` does not recognize them. Indeed, `samesite=strict` cookies are not sent in that case. When a user follows a legitimate link to `bank.com`, like from their own notes, they'll be surprised that `bank.com` does not recognize them. Indeed, `samesite=strict` cookies are not sent in that case.
We could work around that by using two cookies: one for "general recognition", only for the purposes of saying: "Hello, John", and the other one for data-changing operations with `samesite=strict`. We could work around that by using two cookies: one for "general recognition", only for the purposes of saying: "Hello, John", and the other one for data-changing operations with `samesite=strict`. Then a person coming from outside of the site will see a welcome, but payments must be initiated from the bank website.
Then a person coming from outside of the site will see a welcome, but payments must be initiated from the bank website.
- **`samesite=lax`** - **`samesite=lax`**
Another approach to keep user experience is to use `samesite=lax`, a more relaxed value. A more relaxed approach that also protects from XSRF and doesn't break user experience.
Lax mode, just like `strict`, forbids the browser to send cookies when coming from outside the site, but adds an exception. Lax mode, just like `strict`, forbids the browser to send cookies when coming from outside the site, but adds an exception.
@ -234,22 +238,24 @@ A `samesite=lax` cookie is sent if both of these conditions are true:
That's usually true, but if the navigation is performed in an `<iframe>`, then it's not top-level. Also, AJAX requests do not perform any navigation, hence they don't fit. That's usually true, but if the navigation is performed in an `<iframe>`, then it's not top-level. Also, AJAX requests do not perform any navigation, hence they don't fit.
So, what `samesite=lax` does is basically allows a most common "open URL" operation to bring cookies. Something more complicated, like AJAX request from another site or a form submittion loses cookies. So, what `samesite=lax` does is basically allows a most common "go to URL" operation to have cookies. E.g. opening a website link from notes satisfies these conditions.
But anything more complicated, like AJAX request from another site or a form submittion loses cookies.
If that's fine for you, then adding `samesite=lax` will probably not break the user experience and add protection. If that's fine for you, then adding `samesite=lax` will probably not break the user experience and add protection.
Overall, `samesite` is great, but it has an important drawback: Overall, `samesite` is a great option, but it has an important drawback:
- `samesite` is not supported, ignored by old browsers (like year 2017). - `samesite` is ignored (not supported) by old browsers, year 2017 or so.
**So if we solely rely on `samesite` to provide protection, then old browsers will be totally vulnerable.** **So if we solely rely on `samesite` to provide protection, then old browsers will be vulnerable.**
But we surely can use `samesite` together with other protection measures, like xsrf tokens, to add an additional layer of defence. But we surely can use `samesite` together with other protection measures, like xsrf tokens, to add an additional layer of defence and then, in the future, when old browsers die out, we'll probably be able to drop xsrf tokens.
## httpOnly ## httpOnly
This option has nothing to do with Javascript, but we have to mention it for completeness. This option has nothing to do with Javascript, but we have to mention it for completeness.
Only the server, when it uses `Set-Cookie` to set a cookie, may set the `httpOnly` option. The web-server uses `Set-Cookie` header to set a cookie. And it may set the `httpOnly` option.
This option forbids any JavaScript access to the cookie. We can't see such cookie or manipulate it using `document.cookie`. This option forbids any JavaScript access to the cookie. We can't see such cookie or manipulate it using `document.cookie`.
@ -264,7 +270,7 @@ But if a cookie is `httpOnly`, then `document.cookie` doesn't see it, so it is p
Here's a small set of functions to work with cookies, more conveinent than a manual modification of `document.cookie`. Here's a small set of functions to work with cookies, more conveinent than a manual modification of `document.cookie`.
There exist many cookie libraries for that, so these are for demo purposes. Fully working though: There exist many cookie libraries for that, so these are for demo purposes. Fully working though.
### getCookie(name) ### getCookie(name)
@ -284,11 +290,9 @@ function getCookie(name) {
} }
``` ```
Here the regexp is generated dynamically, to match `; name=<value>`. Here `new RegExp` is generated dynamically, to match `; name=<value>`.
Please note that a cookie value can be an arbitrary string. If it contains characters that break the formatting, for instance spaces or `;`, such characters are encoded. Please note that a cookie value is encoded, so `getCookie` uses a built-in `decodeURIComponent` function to decode it.
To decode them, we need to use a built-in `decodeURIComponent` function, the function does it also.
### setCookie(name, value, options) ### setCookie(name, value, options)
@ -413,9 +417,9 @@ GDPR is not only about cookies, it's about other privacy-related issues too, but
Cookie options: Cookie options:
- `path=/`, by default current path, makes the cookie visible only under that path. - `path=/`, by default current path, makes the cookie visible only under that path.
- `domain=site.com`, by default a cookie is visible on current domain only, if set explicitly to the domain, makes the cookie visible on subdomains. - `domain=site.com`, by default a cookie is visible on current domain only, if set explicitly to the domain, makes the cookie visible on subdomains.
- `expires/max-age` set cookie expiration time, without them the cookie dies when the browser is closed. - `expires` or `max-age` sets cookie expiration time, without them the cookie dies when the browser is closed.
- `secure` makes the cookie HTTPS-only. - `secure` makes the cookie HTTPS-only.
- `samesite` forbids browser to send the cookie with requests coming from outside the site, helps to prevent XSRF attacks. - `samesite` forbids the browser to send the cookie with requests coming from outside the site, helps to prevent XSRF attacks.
Additionally: Additionally:
- Third-party cookies may be forbidden by the browser, e.g. Safari does that by default. - Third-party cookies may be forbidden by the browser, e.g. Safari does that by default.

View file

@ -1,17 +1,15 @@
# ArrayBuffer, binary arrays # ArrayBuffer, binary arrays
Binary data appears when we work with arbitrary files (uploading, downloading, creation). Or when we want to do image/audio processing. In web-development we meet binary data mostly while dealing with files (create, upload, download). Another typical use case is image processing.
That's all possible in Javascript, and binary operations are high-performant. That's all possible in Javascript, and binary operations are high-performant.
Although, there's a bit of confusion, because there are many classes. To name a few: Although, there's a bit of confusion, because there are many classes. To name a few:
- `ArrayBuffer`, `Uint8Array`, `DataView`, etc. - `ArrayBuffer`, `Uint8Array`, `DataView`, `Blob`, `File`, etc.
It may seem complex, but in fact it's not. Everything's fairly simple. Binary data in Javascript is implemented in a non-standard way, compared to other languages. But when we sort things out, everything becomes fairly simple.
Let's sort things out. **The basic binary object is `ArrayBuffer` -- a reference to a fixed-length contiguos memory area.**
**The basic binary object is `ArrayBuffer` -- a fixed-length raw sequence of bytes.**
We create it like this: We create it like this:
```js run ```js run
@ -21,62 +19,66 @@ alert(buffer.byteLength); // 16
This allocates a contiguous memory area of 16 bytes and pre-fills it with zeroes. This allocates a contiguous memory area of 16 bytes and pre-fills it with zeroes.
```warn header="`ArrayBuffer` is not `Array` at all" ```warn header="`ArrayBuffer` is not an array of something"
`ArrayBuffer` has nothing in common with `Array`: Let's eliminate a possible source of confusion. `ArrayBuffer` has nothing in common with `Array`:
- It has a fixed length, we can't increase or decrease it. - It has a fixed length, we can't increase or decrease it.
- It takes exactly that much space in the memory. - It takes exactly that much space in the memory.
- It can't store any value: only bytes, the exact number of them. - To access individual bytes, another "view" object is needed, not `buffer[index]`.
``` ```
`ArrayBuffer` is a raw sequence of bytes. What's stored in it? It has no clue. `ArrayBuffer` is a memory area. What's stored in it? It has no clue. Just a raw sequence of bytes.
**To manipulate an `ArrayBuffer`, we need to use a "view" object.** **To manipulate an `ArrayBuffer`, we need to use a "view" object.**
A view object does not store anything on it's own. It's just an "eyeglasses" that gives an interpretation of the bytes stored in the `ArrayBuffer`. A view object does not store anything on it's own. It's the "eyeglasses" that give an interpretation of the bytes stored in the `ArrayBuffer`.
For instance: For instance:
- **`Uint8Array`** -- treats each byte as a separate number, with possible values are from 0 to 255 (a byte is 8-bit, so can hold only that much). That's called a "8-bit unsigned integer". - **`Uint8Array`** -- treats each byte in `ArrayBuffer` as a separate number, with possible values are from 0 to 255 (a byte is 8-bit, so it can hold only that much). Such value is called a "8-bit unsigned integer".
- **`Uint16Array`** -- treats every 2 bytes as an integer, with possible values from 0 to 65535. That's called a "16-bit unsigned integer". - **`Uint16Array`** -- treats every 2 bytes as an integer, with possible values from 0 to 65535. That's called a "16-bit unsigned integer".
- **`Uint32Array`** -- treats every 4 bytes as an integer, with possible values from 0 to 4294967295. That's called a "32-bit unsigned integer". - **`Uint32Array`** -- treats every 4 bytes as an integer, with possible values from 0 to 4294967295. That's called a "32-bit unsigned integer".
- **`Float64Array`** -- treats every 8 bytes as a floating point number with possible values from <code>5.0x10<sup>-324</sup></code> to <code>1.8x10<sup>308</sup></code>. - **`Float64Array`** -- treats every 8 bytes as a floating point number with possible values from <code>5.0x10<sup>-324</sup></code> to <code>1.8x10<sup>308</sup></code>.
So, the binary data in an `ArrayBuffer` of 16 bytes can be interpreted as 16 "tiny numbers", or 8 bigger numbers, or 4 even bigger, or 2 floating-point values with high precision. So, the binary data in an `ArrayBuffer` of 16 bytes can be interpreted as 16 "tiny numbers", or 8 bigger numbers (2 bytes each), or 4 even bigger (4 bytes each), or 2 floating-point values with high precision (8 bytes each).
![](arraybuffer-views.png) ![](arraybuffer-views.png)
`ArrayBuffer` is the core object, the root of everything, the raw binary data. But if we're going to write into it, or iterate over it, basically for almost any operation we must use a view, e.g: `ArrayBuffer` is the core object, the root of everything, the raw binary data.
But if we're going to write into it, or iterate over it, basically for almost any operation we must use a view, e.g:
```js run ```js run
let buffer = new ArrayBuffer(16); // create a buffer of length 16 let buffer = new ArrayBuffer(16); // create a buffer of length 16
*!* *!*
let view = new Uint32Array(buffer); // treat every 4 bytes as an integer number let view = new Uint32Array(buffer); // treat buffer as a sequence of 32-bit integers
alert(Uint32Array.BYTES_PER_ELEMENT); // 4 bytes per integer
*/!* */!*
alert(view.length); // 4, it stores that many numbers alert(view.length); // 4, it stores that many integers
alert(view.byteLength); // 16, the length in bytes alert(view.byteLength); // 16, the size in bytes
// let's write a value // let's write a value
view[0] = 123456; view[0] = 123456;
// iterate over bytes // iterate over values
for(let num of view) { for(let num of view) {
alert(num); // 123456, then 0, 0, 0 (4 numbers total) alert(num); // 123456, then 0, 0, 0 (4 values total)
} }
alert(Uint32Array.BYTES_PER_ELEMENT); // 4 (that many bytes per number)
``` ```
The common term for all these views is [TypedArray](https://tc39.github.io/ecma262/#sec-typedarray-objects). They share the same set of methods and properities.
They are much more like regular arrays: have indexes and iterable.
## TypedArray ## TypedArray
The common term for all these views (`Uint8Array`, `Uint32Array`, etc) is [TypedArray](https://tc39.github.io/ecma262/#sec-typedarray-objects). They share the same set of methods and properities.
They are much more like regular arrays: have indexes and iterable.
A typed array constructor (be it `Int8Array` or `Float64Array`, doesn't matter) behaves differently depending on argument types. A typed array constructor (be it `Int8Array` or `Float64Array`, doesn't matter) behaves differently depending on argument types.
There are 5 variants: There are 5 variants of arguments:
```js ```js
new TypedArray(buffer, [byteOffset], [length]); new TypedArray(buffer, [byteOffset], [length]);
@ -86,13 +88,11 @@ new TypedArray(length);
new TypedArray(); new TypedArray();
``` ```
A view cannot exist without an underlying `ArrayBuffer`, so gets created automatically in all these calls except the first one.
1. If an `ArrayBuffer` argument is supplied, the view is created over it. We used that syntax already. 1. If an `ArrayBuffer` argument is supplied, the view is created over it. We used that syntax already.
Optionally we can provide `byteOffset` to start from (0 by default) and the `length` (till the end of the buffer by default). Optionally we can provide `byteOffset` to start from (0 by default) and the `length` (till the end of the buffer by default), then the view will cover only a part of the `buffer`.
2. If an `Array`, or any array-like object is given, it behaves as `Array.from`: creates a typed array of the same length and copies the content. 2. If an `Array`, or any array-like object is given, it creates a typed array of the same length and copies the content.
We can use it to pre-fill the array with the data: We can use it to pre-fill the array with the data:
```js run ```js run
@ -102,7 +102,7 @@ A view cannot exist without an underlying `ArrayBuffer`, so gets created automat
alert( arr.length ); // 4 alert( arr.length ); // 4
alert( arr[1] ); // 1 alert( arr[1] ); // 1
``` ```
3. If another `TypedArray` is supplied, it's contents is also copied. Values are converted to the new type in the process. The new array will have the same length. 3. If another `TypedArray` is supplied, it does the same: creates a typed array of the same length and copies values. Values are converted to the new type in the process.
```js run ```js run
let arr16 = new Uint16Array([1, 1000]); let arr16 = new Uint16Array([1, 1000]);
*!* *!*
@ -112,16 +112,31 @@ A view cannot exist without an underlying `ArrayBuffer`, so gets created automat
alert( arr8[1] ); // 232 (tried to copy 1000, but can't fit 1000 into 8 bits) alert( arr8[1] ); // 232 (tried to copy 1000, but can't fit 1000 into 8 bits)
``` ```
4. For a numeric argument `length` -- creates an `ArrayBuffer` to contain that many elements. Its byte length will be `length` multiplied by the number of bytes in a single item `TypedArray.BYTES_PER_ELEMENT`: 4. For a numeric argument `length` -- creates the typed array to contain that many elements. Its byte length will be `length` multiplied by the number of bytes in a single item `TypedArray.BYTES_PER_ELEMENT`:
```js run ```js run
let arr = new Uint16Array(4); let arr = new Uint16Array(4); // create typed array for 4 integers
alert( Uint16Array.BYTES_PER_ELEMENT ); // 2 bytes per number alert( Uint16Array.BYTES_PER_ELEMENT ); // 2 bytes per integer
alert( arr.byteLength ); // 8 bytes in the buffer, to contain four 2-byte numbers alert( arr.byteLength ); // 8 (size in bytes)
``` ```
5. Without arguments, creates an zero-length empty `ArrayBuffer`. 5. Without arguments, creates an zero-length typed array.
Here's the full list of typed arrays: We can create a `TypedArray` directly, without mentioning `ArrayBuffer`. But a view cannot exist without an underlying `ArrayBuffer`, so gets created automatically in all these cases except the first one (when provided).
To access the `ArrayBuffer`, there are properties:
- `arr.buffer` -- references the `ArrayBuffer`.
- `arr.byteLength` -- the length of the `ArrayBuffer`.
So, we can always move from one view to another:
```js
let arr8 = new Uint8Array([0, 1, 2, 3]);
// another view on the same data
let arr16 = new Uint16Array(arr8.buffer);
```
Here's the list of typed arrays:
- `Uint8Array`, `Uint16Array`, `Uint32Array` -- for integer numbers of 8, 16 and 32 bits. - `Uint8Array`, `Uint16Array`, `Uint32Array` -- for integer numbers of 8, 16 and 32 bits.
- `Uint8ClampedArray` -- for 8-bit integers, "clamps" them on assignment (see below). - `Uint8ClampedArray` -- for 8-bit integers, "clamps" them on assignment (see below).
@ -129,14 +144,14 @@ Here's the full list of typed arrays:
- `Float32Array`, `Float64Array` -- for signed floating-point numbers of 32 and 64 bits. - `Float32Array`, `Float64Array` -- for signed floating-point numbers of 32 and 64 bits.
```warn header="No `int8` or similar single-valued types" ```warn header="No `int8` or similar single-valued types"
Please note, despite of the names like `Int8Array`, there's no type like `int8`, or similar in Javascript. Please note, despite of the names like `Int8Array`, there's no single-value type like `int`, or `int8` in Javascript.
That's logical, as `Int8Array` is technically not an array, but rather a view on `ArrayBuffer`. That's logical, as `Int8Array` is not an array of these individual values, but rather a view on `ArrayBuffer`.
``` ```
### Out-of-bounds behavior ### Out-of-bounds behavior
What if we attempt to write an out-of-bounds value into a typed array? There will be no error. But the result may be not what we want. What if we attempt to write an out-of-bounds value into a typed array? There will be no error. But extra bits are cut-off.
For instance, let's try to put 256 into `Uint8Array`. In binary form, 256 is `100000000` (9 bits), but `Uint8Array` only provides 8 bits per value, that makes the available range from 0 to 255. For instance, let's try to put 256 into `Uint8Array`. In binary form, 256 is `100000000` (9 bits), but `Uint8Array` only provides 8 bits per value, that makes the available range from 0 to 255.
@ -155,8 +170,7 @@ In other words, the number modulo 2<sup>8</sup> is saved.
Here's the demo: Here's the demo:
```js run ```js run
let buffer = new ArrayBuffer(16); let uint8array = new Uint8Array(16);
let uint8array = new Uint8Array(buffer);
let num = 256; let num = 256;
alert(num.toString(2)); // 100000000 (binary representation) alert(num.toString(2)); // 100000000 (binary representation)
@ -168,7 +182,7 @@ alert(uint8array[0]); // 0
alert(uint8array[1]); // 1 alert(uint8array[1]); // 1
``` ```
`Uint8ClampedArray` is special in this aspect, its behavior is different. It saves 255 for any number that is greater than 255, and 0 for any negative number. That behavior is sometimes useful for image processing. `Uint8ClampedArray` is special in this aspect, its behavior is different. It saves 255 for any number that is greater than 255, and 0 for any negative number. That behavior is useful for image processing.
## TypedArray methods ## TypedArray methods
@ -181,28 +195,21 @@ There are few things we can't do though:
- No `splice` -- we can't "delete" a value, because typed arrays are views on a buffer, and these are fixed, contiguous areas of memory. All we can do is to assign a zero. - No `splice` -- we can't "delete" a value, because typed arrays are views on a buffer, and these are fixed, contiguous areas of memory. All we can do is to assign a zero.
- No `concat` method. - No `concat` method.
There are two additional properties: There are two additional methods:
- `arr.buffer` -- the reference to the underlying `ArrayBuffer`.
- `arr.byteLength` -- the byte size of the underlying `ArrayBuffer`.
...And methods:
- `arr.set(fromArr, [offset])` copies all elements from `fromArr` to the `arr`, starting at position `offset` (0 by default). - `arr.set(fromArr, [offset])` copies all elements from `fromArr` to the `arr`, starting at position `offset` (0 by default).
- `arr.subarray([begin, end])` creates a new view of the same type from `begin` to `end` (exclusive). That's similar to `arr.slice` method, but doesn't copy anything -- just creates a new view. - `arr.subarray([begin, end])` creates a new view of the same type from `begin` to `end` (exclusive). That's similar to `slice` method (that's also supported), but doesn't copy anything -- just creates a new view, to operate on the given piece of data.
These methods allow us to copy typed arrays one onto another, create new arrays from existing ones. These methods allow us to copy typed arrays, mix them, create new arrays from existing ones, and so on.
## DataView ## DataView
[DataView](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/DataView) is a special super-flexible "untyped" view over `ArrayBuffer`. [DataView](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/DataView) is a special super-flexible "untyped" view over `ArrayBuffer`. It allows to access the data on any offset in any format.
It allows to access the data on any offset in any format. - For typed arrays, the constructor dictates what the format is. The whole array is supposed to be uniform. The i-th number is `arr[i]`.
- With `DataView` we access the data with methods like `.getUint8(i)` or `.gteUint16(i)`. We choose the format at method call time instead of the construction time.
- For typed arrays, the constructor dictates what the format is, then we access i-th number as `arr[i]`. All array members are assumed to have the same format.
- With `DataView` we access the data with methods like `.getUint8(i)`. So we decide what the format is at method call time instead of the construction time.
The syntax: The syntax:
@ -233,11 +240,11 @@ alert( dataView.getUint32(0) ); // 4294967295 (biggest 32-bit unsigned int)
dataView.setUint32(0, 0); // set 4-byte number to zero dataView.setUint32(0, 0); // set 4-byte number to zero
``` ```
`DataView` is great when we store mixed-format data in the same buffer. E.g 8-bit integers followed by 32-bit floats, or more complex. `DataView` is great when we store mixed-format data in the same buffer. E.g we store a sequence of pairs (16-bit integer, 32-bit float). Then `DataView` allows to access them easily.
## Summary ## Summary
`ArrayBuffer` is the core object, a raw byte sequence. `ArrayBuffer` is the core object, a reference to the fixed-length contiguous memory area.
To do almost any operation on `ArrayBuffer`, we need a view. To do almost any operation on `ArrayBuffer`, we need a view.
@ -254,7 +261,8 @@ There are also two additional terms:
- `ArrayBufferView` is an umbrella term for all these kinds of views. - `ArrayBufferView` is an umbrella term for all these kinds of views.
- `BufferSource` is an umbrella term for `ArrayBuffer` or `ArrayBufferView`. - `BufferSource` is an umbrella term for `ArrayBuffer` or `ArrayBufferView`.
These are used in descriptions of methods that operate on binary data. `BufferSource` is one of the most common words, as it means "`ArrayBuffer` or a view over it" These are used in descriptions of methods that operate on binary data. `BufferSource` is one of the most common teerms, as it means "any kind of binary data" -- an `ArrayBuffer` or a view over it.
Here's a cheatsheet: Here's a cheatsheet:

View file

@ -1,6 +1,6 @@
# TextDecoder and TextEncoder # TextDecoder and TextEncoder
What if the binary data is actually a string? What if the binary data is actually a string? For instance, we received a file with textual data.
The build-in [TextDecoder](https://encoding.spec.whatwg.org/#interface-textdecoder) object allows to read the value into an an actual Javascript string, given the buffer and the encoding. The build-in [TextDecoder](https://encoding.spec.whatwg.org/#interface-textdecoder) object allows to read the value into an an actual Javascript string, given the buffer and the encoding.
@ -9,10 +9,10 @@ We first need to create it:
let decoder = new TextDecoder([label], [options]); let decoder = new TextDecoder([label], [options]);
``` ```
- **`label`** -- the encoding, `utf-8` by default. - **`label`** -- the encoding, `utf-8` by default, but `big5`, `windows-1251` and many other are also supported.
- **`options`** -- optional object: - **`options`** -- optional object:
- **`fatal`** -- boolean, if `true` then throw an exception for invalid string, otherwise (default) replace invalid byte sequences with character `\uFFFD`. - **`fatal`** -- boolean, if `true` then throw an exception for invalid (non-decodable) characters, otherwise (default) replace them with character `\uFFFD`.
- **`ignoreBOM`** -- boolean, if `true` then ignore BOM (an optional byte-order unicode mark), we usually don't set it. - **`ignoreBOM`** -- boolean, if `true` then ignore BOM (an optional byte-order unicode mark), rarely needed.
...And then decode: ...And then decode:
@ -39,7 +39,7 @@ let uint8Array = new Uint8Array([228, 189, 160, 229, 165, 189]);
alert( new TextDecoder().decode(uint8Array) ); // 你好 alert( new TextDecoder().decode(uint8Array) ); // 你好
``` ```
We can decode a part of the buffer by passing it a subarray: We can decode a part of the buffer by creating a subarray view for it:
```js run ```js run
@ -54,7 +54,7 @@ alert( new TextDecoder().decode(binaryString) ); // Hello
## TextEncoder ## TextEncoder
[TextEncoder](https://encoding.spec.whatwg.org/#interface-textencoder) does the reverse thing -- writes a string into a buffer. [TextEncoder](https://encoding.spec.whatwg.org/#interface-textencoder) does the reverse thing -- converts a string into bytes.
The syntax is: The syntax is:

View file

@ -4,7 +4,7 @@
In the browser, there are additional higher-level objects, described in [File API](https://www.w3.org/TR/FileAPI/). In the browser, there are additional higher-level objects, described in [File API](https://www.w3.org/TR/FileAPI/).
`Blob` consists of an optional string `type` (a MIME-type usually), plus `blobParts` this a sequence of other `Blob` objects, strings and `BufferSources`. `Blob` consists of an optional string `type` (a MIME-type usually), plus `blobParts` -- a sequence of other `Blob` objects, strings and `BufferSources`.
![](blob.png) ![](blob.png)
@ -16,12 +16,28 @@ The constructor syntax is:
new Blob(blobParts, options); new Blob(blobParts, options);
``` ```
- **`blobParts`** is an array of Blob/BufferSource/String values. - **`blobParts`** is an array of `Blob`/`BufferSource`/`String` values.
- **`options`** optional object: - **`options`** optional object:
- **`type`** -- blob type, usually MIME-type, e.g. `image/png`, - **`type`** -- blob type, usually MIME-type, e.g. `image/png`,
- **`endings`** -- whether to transform end-of-line `\r\n` to `\n` end vise versa, to make the blob correspond to current OS newlines. By default `"transparent"` (do nothing), but also can be `"native"` (transform). - **`endings`** -- whether to transform end-of-line to make the blob correspond to current OS newlines (`\r\n` or `\n`). By default `"transparent"` (do nothing), but also can be `"native"` (transform).
Then we can access blob as a whole, or extract slices with: For example:
```js
// create Blob from a string
let blob = new Blob(["<html></html>"], {type: 'text/html'});
// please note: the first argument must be an array [...]
```
```js
// create Blob from a typed array and strings
let hello = new Uint8Array([72, 101, 108, 108, 111]); // "hello" in binary form
let blob = new Blob([hello, ' ', 'world'], {type: 'text/plain'});
```
We can extract blob slices with:
```js ```js
blob.slice([byteStart], [byteEnd], [contentType]); blob.slice([byteStart], [byteEnd], [contentType]);
@ -33,11 +49,15 @@ blob.slice([byteStart], [byteEnd], [contentType]);
The arguments are similar to `array.slice`, negative numbers are allowed too. The arguments are similar to `array.slice`, negative numbers are allowed too.
We can't change data directly in a blob, but we can copy parts of blobs, create new blobs from them, mix them and so on. This behavior is similar to Javascript strings: we can't change a character in a string, but we can make a new corrected string. ```smart header="Blobs are immutable"
We can't change data directly in a blob, but we can slice parts of blobs, create new blobs from them, mix them into a new blob and so on.
This behavior is similar to Javascript strings: we can't change a character in a string, but we can make a new corrected string.
```
## Blob as URL ## Blob as URL
A Blob can be easily used in an URL for `<a>`, `<img>` or other tags. A Blob can be easily used as an URL for `<a>`, `<img>` or other tags, to show its contents.
Let's start with a simple example. By clicking on a link you download a dynamically-generated blob with `hello world` contents as a file: Let's start with a simple example. By clicking on a link you download a dynamically-generated blob with `hello world` contents as a file:
@ -46,18 +66,15 @@ Let's start with a simple example. By clicking on a link you download a dynamica
<a download="hello.txt" href='#' id="link">Download</a> <a download="hello.txt" href='#' id="link">Download</a>
<script> <script>
let binary = new Uint8Array([72, 101, 108, 108, 111]); // "hello" in binary form let blob = new Blob(["Hello, world!"], {type: 'text/plain'});
// create blob of multiple parts
let blob = new Blob([binary, ' ', 'world'], {type: 'text/plain'});
link.href = URL.createObjectURL(blob); link.href = URL.createObjectURL(blob);
</script> </script>
``` ```
We can also create a link in Javascript and simulate a click by `link.click()`, then download starts authomatically. We can also create a link dynamically in Javascript and simulate a click by `link.click()`, then download starts authomatically.
Here's the similar "on the fly" blob creation and download code, now without HTML: Here's the similar "on the fly" blob creation and download code, but without HTML:
```js run ```js run
let link = document.createElement('a'); let link = document.createElement('a');
@ -80,19 +97,19 @@ That's what the generated url looks like:
blob:https://javascript.info/1e67e00e-860d-40a5-89ae-6ab0cbee6273 blob:https://javascript.info/1e67e00e-860d-40a5-89ae-6ab0cbee6273
``` ```
The browser keeps an internal url -> blob mapping. For each url generated by `URL.createObjectURL`, it stores a reference to the corresponding blob. The browser for each url generated by `URL.createObjectURL` stores an the url -> blob mapping internally. So such urls are short, but allow to access the blob.
A generated url is only valid while the current document is open. And it allows to reference the blob in `<img>`, `<a>`, any other object that expects an URL. A generated url is only valid while the current document is open. And it allows to reference the blob in `<img>`, `<a>`, any other object that expects an url.
There's a side-effect though. While there's an mapping for a blob, the blob itself resides in the memory. The browser can't free it. There's a side-effect though. While there's an mapping for a blob, the blob itself resides in the memory. The browser can't free it.
The mapping is automatically cleared on document unload, so blobs are freed then. But if an app is long-living, then that doesn't happen. So if we create an URL, that blob will hang in memory, even if not needed any more. The mapping is automatically cleared on document unload, so blobs are freed then. But if an app is long-living, then that doesn't happen soon. So if we create an URL, that blob will hang in memory, even if not needed any more.
**`URL.revokeObjectURL(url)` removes the reference from the internal mapping, thus allowing the blob to be deleted (if there are no other references), and the memory to be freed.** **`URL.revokeObjectURL(url)` removes the reference from the internal mapping, thus allowing the blob to be deleted (if there are no other references), and the memory to be freed.**
In the last example, we intend the blob to be used only once, for instant downloading, so we call `URL.revokeObjectURL(link.href)` immediately. In the last example, we intend the blob to be used only once, for instant downloading, so we call `URL.revokeObjectURL(link.href)` immediately.
In the previous example though, with the clickable HTML-link, we dont't call `URL.revokeObjectURL(link.href)`, because that would make the blob url invalid. After the revocation, as the mapping is removed, the url doesn't work any more. In the previous example though, with the clickable HTML-link, we don't call `URL.revokeObjectURL(link.href)`, because that would make the blob url invalid. After the revocation, as the mapping is removed, the url doesn't work any more.
## Blob to base64 ## Blob to base64
@ -111,7 +128,7 @@ For instance, here's a smiley:
The browser will decode the string and show the image: <img src="data:image/png;base64,R0lGODlhDAAMAKIFAF5LAP/zxAAAANyuAP/gaP///wAAAAAAACH5BAEAAAUALAAAAAAMAAwAAAMlWLPcGjDKFYi9lxKBOaGcF35DhWHamZUW0K4mAbiwWtuf0uxFAgA7"> The browser will decode the string and show the image: <img src="data:image/png;base64,R0lGODlhDAAMAKIFAF5LAP/zxAAAANyuAP/gaP///wAAAAAAACH5BAEAAAUALAAAAAAMAAwAAAMlWLPcGjDKFYi9lxKBOaGcF35DhWHamZUW0K4mAbiwWtuf0uxFAgA7">
To transform a blob into base64 and vise-versa, we'll use built-in `FileReader/FileWriter` objects. In the [next chapter](info:file) we'll cover those more in-detail. For now, it's important only that they can read a blob as data url, that we can assign it to `link.href`. To transform a blob into base64, we'll use the built-in `FileReader` object. It can read data from Blobs in multiple formats. In the [next chapter](info:file) we'll cover it more in-depth.
Here's the demo of downloading a blob, now via base-64: Here's the demo of downloading a blob, now via base-64:
@ -123,16 +140,13 @@ let blob = new Blob(['Hello, world!'], {type: 'text/plain'});
*!* *!*
let reader = new FileReader(); let reader = new FileReader();
reader.readAsDataURL(blob); // converts the blob to base64 and calls onload
*/!* */!*
reader.onload = function() { reader.onload = function() {
link.href = reader.result; // data url link.href = reader.result; // data url
link.click(); link.click();
}; };
*!*
reader.readAsDataURL(blob); // converts the blob to base64 and calls onload
*/!*
``` ```
Both ways of making an URL of a blob are usable. But usually `URL.createObjectURL(blob)` is simpler and faster. Both ways of making an URL of a blob are usable. But usually `URL.createObjectURL(blob)` is simpler and faster.
@ -144,57 +158,70 @@ Both ways of making an URL of a blob are usable. But usually `URL.createObjectUR
- Performance and memory losses on big blobs for encoding. - Performance and memory losses on big blobs for encoding.
``` ```
## Image or a page to blob ## Image to blob
We can create a blob of an image, or even a page screenshot. We can create a blob of an image, an image part, or even make a page screenshot. That's handy to upload it somewhere.
That's usually done via `<canvas>` element: Image operations are done via `<canvas>` element:
1. We draw an existing image on canvas using [canvas.drawImage](https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D/drawImage). 1. Draw an image (or its part) on canvas using [canvas.drawImage](https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D/drawImage).
2. Canvas [.toBlob(callback, format, quality)](https://developer.mozilla.org/en-US/docs/Web/API/HTMLCanvasElement/toBlob) creates a blob and calls `callback` with it. 2. Call canvas method [.toBlob(callback, format, quality)](https://developer.mozilla.org/en-US/docs/Web/API/HTMLCanvasElement/toBlob) that creates a blob and runs `callback` with it when done.
In the example below, an image is just copied, but we could cut from it, or transform it on canvas prior to making a blob: In the example below, an image is just copied, but we could cut from it, or transform it on canvas prior to making a blob:
```js run ```js run
let img = document.querySelector('img'); // just any image // take any image
let img = document.querySelector('img');
// make <canvas> of the same size
let canvas = document.createElement('canvas'); let canvas = document.createElement('canvas');
canvas.width = img.clientWidth; canvas.width = img.clientWidth;
canvas.height = img.clientHeight; canvas.height = img.clientHeight;
let context = canvas.getContext('2d'); let context = canvas.getContext('2d');
context.drawImage(img, 0, 0); // allows to cut image // copy image to it (this method allows to cut image)
// we can context.rotate() as well, and do many other things context.drawImage(img, 0, 0);
// we can context.rotate(), and do many other things on canvas
// toBlob is async opereation, callback is called when done
canvas.toBlob(function(blob) { canvas.toBlob(function(blob) {
// download the blob // blob ready, download it
let link = document.createElement('a'); let link = document.createElement('a');
link.download = 'example.png'; link.download = 'example.png';
link.href = URL.createObjectURL(blob); link.href = URL.createObjectURL(blob);
link.click(); link.click();
// delete the internal blob reference, to let the browser clear memory from it
URL.revokeObjectURL(link.href); URL.revokeObjectURL(link.href);
}, 'image/png'); }, 'image/png');
``` ```
For screenshotting a page, we can use a library such as <https://github.com/niklasvh/html2canvas> to draw the screenshot on `<canvas>`, and then download it the same way. If we prefer `async/await` instead of callbacks:
```js
let blob = await new Promise(resolve => canvasElem.toBlob(resolve, 'image/png'));
```
For screenshotting a page, we can use a library such as <https://github.com/niklasvh/html2canvas>. What it does is just walks the page and draws it on `<canvas>`. Then we can get a blob of it the same way as above.
## From Blob to ArrayBuffer ## From Blob to ArrayBuffer
The `Blob` constructor allows to create a blob from almost anything. The `Blob` constructor allows to create a blob from almost anything, including any `BufferSource`.
But we can also revert back to lowest-level `ArrayBuffer` using `FileReader`: But if we need to perform low-level processing, we can get the lowest-level `ArrayBuffer` from it using `FileReader`:
```js ```js
// get arrayBuffer from blob // get arrayBuffer from blob
let fileReader = new FileReader(); let fileReader = new FileReader();
fileReader.onload = function(event) {
let arrayBuffer = fileReader.result;
};
*!* *!*
fileReader.readAsArrayBuffer(blob); fileReader.readAsArrayBuffer(blob);
*/!* */!*
fileReader.onload = function(event) {
let arrayBuffer = fileReader.result;
};
``` ```
@ -204,9 +231,9 @@ While `ArrayBuffer`, `Uint8Array` and other `BufferSource` are "binary data", a
That makes Blobs convenient for upload/download operations, that are so common in the browser. That makes Blobs convenient for upload/download operations, that are so common in the browser.
Methods that perform web-requests, such as [XMLHttpRequest](info:xmlhttprequest), [fetch](info:fetch) and so on, can work with `Blob` natively, as well as with other binary types. Methods that perform web-requests, such as [XMLHttpRequest](info:xmlhttprequest), [fetch](info:fetch-basics) and so on, can work with `Blob` natively, as well as with other binary types.
That said, it's still possible to convert betweeen `Blob` and low-level binary data types: We can easily convert betweeen `Blob` and low-level binary data types:
- We can make a Blob from a typed array using `new Blob(...)` constructor. - We can make a Blob from a typed array using `new Blob(...)` constructor.
- We can get back `ArrayBuffer` from a Blob using `FileReader`, and then create a view over it for low-level binary processing. - We can get back `ArrayBuffer` from a Blob using `FileReader`, and then create a view over it for low-level binary processing.

View file

@ -33,7 +33,7 @@ function showFile(input) {
``` ```
```smart ```smart
Please note: the input may select multiple files, so `input.files` is an array-like object with them. Here we have only one file, but it's still the same, so we just take `input.files[0]`. The input may select multiple files, so `input.files` is an array-like object with them. Here we have only one file, so we just take `input.files[0]`.
``` ```
## FileReader ## FileReader
@ -63,13 +63,13 @@ As the reading proceeds, there are events:
- `error` -- error has occured. - `error` -- error has occured.
- `loadend` -- reading finished with either success or failure. - `loadend` -- reading finished with either success or failure.
At the end: When the reading is finished, we can access the result as:
- `reader.result` is the result (if successful) - `reader.result` is the result (if successful)
- `reader.error` is the error (if failed). - `reader.error` is the error (if failed).
The most widely used events are for sure `load` and `error`. The most widely used events are for sure `load` and `error`.
Here's an example: Here's an example of reading a file:
```html run ```html run
<input type="file" onchange="readFile(this)"> <input type="file" onchange="readFile(this)">
@ -80,6 +80,8 @@ function readFile(input) {
let reader = new FileReader(); let reader = new FileReader();
reader.readAsText(file);
reader.onload = function() { reader.onload = function() {
console.log(reader.result); console.log(reader.result);
}; };
@ -88,16 +90,24 @@ function readFile(input) {
console.log(reader.error); console.log(reader.error);
}; };
reader.readAsText(file);
} }
</script> </script>
``` ```
```smart header="`FileReader` for blobs"
As mentioned in the chapter <info:blob>, `FileReader` works for any blobs, not just files.
So we can use it to convert a blob to another format:
- `readAsArrayBuffer(blob)` -- to `ArrayBuffer`,
- `readAsText(blob, [encoding])` -- to string (an alternative to `TextDecoder`),
- `readAsDataURL(blob)` -- to base64 data url.
```
```smart header="`FileReaderSync` is available for workers only" ```smart header="`FileReaderSync` is available for workers only"
For Web Workers, there also exists a synchronous variant of `FileReader`, called [FileReaderSync](https://www.w3.org/TR/FileAPI/#FileReaderSync). For Web Workers, there also exists a synchronous variant of `FileReader`, called [FileReaderSync](https://www.w3.org/TR/FileAPI/#FileReaderSync).
Reading methods `read*` do not generate events, but rather return the result, as regular functions do. Its reading methods `read*` do not generate events, but rather return a result, as regular functions do.
That's only inside a Web Worker though, because delays and hang-ups in Web Workers are less important, they do not affect the page. That's only inside a Web Worker though, because delays and hang-ups in Web Workers are less important, they do not affect the page.
``` ```
@ -113,8 +123,6 @@ In addition to `Blob` methods and properties, `File` objects also have `fileName
- `ArrayBuffer` (`readAsArrayBuffer`). - `ArrayBuffer` (`readAsArrayBuffer`).
- Data url, base-64 encoded (`readAsDataURL`). - Data url, base-64 encoded (`readAsDataURL`).
In many cases though, we don't have to read the file contents. In many cases though, we don't have to read the file contents. Just as we did with blobs, we can create a short url with `URL.createObjectURL(file)` and assign it to `<a>` or `<img>`. This way the file can be downloaded or shown up as an image, as a part of canvas etc.
We can create a blob url with `URL.createObjectURL(file)` and assign it to `<a>` or `<img>`. This way the file can be downloaded or show up as an image, as a part of canvas etc. And if we're going to send a `File` over a network, that's also easy, as network API like `XMLHttpRequest` or `fetch` natively accepts `File` objects.
And if we're going to send a `File` over a network, then it's also easy, as network API like `XMLHttpRequest` or `fetch` natively accepts `File` objects.

Binary file not shown.

Before

Width:  |  Height:  |  Size: 10 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 23 KiB

View file

@ -0,0 +1,361 @@
# Fetch: Basics
Method `fetch()` is the modern way of sending requests over HTTP.
Is evolved for several years and continues to improve, right now the support is pretty solid among browsers.
The basic syntax is:
```js
let promise = fetch(url, [options])
```
- **`url`** -- the URL to access.
- **`options`** -- optional parameters: method, headers etc.
The browser starts the request right away and returns a `promise`.
Getting a response is usually a two-stage process.
**The `promise` resolves with an object of the built-in [Response](https://fetch.spec.whatwg.org/#response-class) class as soon as the server responds with headers.**
So we can check HTTP status, to see whether it is successful or not, check headers, but don't have the body yet.
The promise rejects if the `fetch` was unable to make HTTP-request, e.g. network problems, or there's no such site. HTTP-errors, even such as 404 or 500, are considered a normal flow.
We can see them in response properties:
- **`ok`** -- boolean, `true` if the HTTP status code is 200-299.
- **`status`** -- HTTP status code.
For example:
```js
let response = await fetch(url);
if (response.ok) { // if HTTP-status is 200-299
// get the response body (see below)
let json = await response.json();
} else {
alert("HTTP-Error: " + response.status);
}
```
To get the response body, we need to use an additional method call.
`Response` provides multiple promise-based methods to access the body in various formats:
- **`response.json()`** -- parse the response as JSON object,
- **`response.text()`** -- return the response as text,
- **`response.formData()`** -- return the response as FormData object (form/multipart encoding),
- **`response.blob()`** -- return the response as [Blob](info:blob) (binary data with type),
- **`response.arrayBuffer()`** -- return the response as [ArrayBuffer](info:arraybuffer-binary-arrays) (pure binary data),
- additionally, `response.body` is a [ReadableStream](https://streams.spec.whatwg.org/#rs-class) object, it allows to read the body chunk-by-chunk, we'll see an example later.
For instance, here we get a JSON-object with latest commits from Github:
```js run async
let response = await fetch('https://api.github.com/repos/iliakan/javascript-tutorial-en/commits');
*!*
let commits = await response.json(); // read response body and parse as JSON
*/!*
alert(commits[0].author.login);
```
Or, the same using pure promises syntax:
```js run
fetch('https://api.github.com/repos/iliakan/javascript-tutorial-en/commits')
.then(response => response.json())
.then(commits => alert(commits[0].author.login));
```
To get the text:
```js
let text = await response.text();
```
And for the binary example, let's fetch and show an image (see chapter [Blob](info:blob) for details about operations on blobs):
```js async run
let response = await fetch('/article/fetch/logo-fetch.svg');
*!*
let blob = await response.blob(); // download as Blob object
*/!*
// create <img> for it
let img = document.createElement('img');
img.style = 'position:fixed;top:10px;left:10px;width:100px';
document.body.append(img);
// show it
img.src = URL.createObjectURL(blob);
setTimeout(() => { // hide after two seconds
img.remove();
URL.revokeObjectURL(img.src);
}, 2000);
```
````warn
We can choose only one body-parsing method.
If we got the response with `response.text()`, then `response.json()` won't work, as the body content has already been processed.
```js
let text = await response.text(); // response body consumed
let parsed = await response.json(); // fails (already consumed)
````
## Headers
There's a Map-like headers object in `response.headers`.
We can get individual headers or iterate over them:
```js run async
let response = await fetch('https://api.github.com/repos/iliakan/javascript-tutorial-en/commits');
// get one header
alert(response.headers.get('Content-Type')); // application/json; charset=utf-8
// iterate over all headers
for (let [key, value] of response.headers) {
alert(`${key} = ${value}`);
}
```
To set a header, we can use the `headers` option, like this:
```js
let response = fetch(protectedUrl, {
headers: {
Authentication: 'abcdef'
}
});
```
...But there's a list of [forbidden HTTP headers](https://fetch.spec.whatwg.org/#forbidden-header-name) that we can't set:
- `Accept-Charset`, `Accept-Encoding`
- `Access-Control-Request-Headers`
- `Access-Control-Request-Method`
- `Connection`
- `Content-Length`
- `Cookie`, `Cookie2`
- `Date`
- `DNT`
- `Expect`
- `Host`
- `Keep-Alive`
- `Origin`
- `Referer`
- `TE`
- `Trailer`
- `Transfer-Encoding`
- `Upgrade`
- `Via`
- `Proxy-*`
- `Sec-*`
These headers ensure proper and safe HTTP, so they are controlled exclusively by the browser.
## POST requests
To make a `POST` request, or a request with another method, we need to use `fetch` options:
- **`method`** -- HTTP-method, e.g. `POST`,
- **`body`** -- one of:
- a string (e.g. JSON),
- `FormData` object, to submit the data as `form/multipart`,
- `Blob`/`BufferSource` to send binary data,
- [URLSearchParams](info:url), to submit the data as `x-www-form-urlencoded`, rarely used.
Let's see examples.
## Submit JSON
This code submits a `user` object as JSON:
```js run async
let user = {
name: 'John',
surname: 'Smith'
};
*!*
let response = await fetch('/article/fetch-basics/post/user', {
method: 'POST',
headers: {
'Content-Type': 'application/json;charset=utf-8'
},
body: JSON.stringify(user)
});
*/!*
let result = await response.json();
alert(result.message);
```
Please note, if the body is a string, then `Content-Type` is set to `text/plain;charset=UTF-8` by default. So we use `headers` option to send `application/json` instead.
## Submit a form
Let's do the same with an HTML `<form>`.
```html run
<form id="formElem">
<input type="text" name="name" value="John">
<input type="text" name="surname" value="Smith">
</form>
<script>
(async () => {
let response = await fetch('/article/fetch-basics/post/user', {
method: 'POST',
*!*
body: new FormData(formElem)
*/!*
});
let result = await response.json();
alert(result.message);
})();
</script>
```
Here [FormData](https://xhr.spec.whatwg.org/#formdata) automatically encodes the form, `<input type="file">` fields are handled also, and sends it using `Content-Type: form/multipart`.
## Submit an image
We can also submit binary data directly using `Blob` or `BufferSource`.
For example, here's a `<canvas>` where we can draw by moving a mouse. A click on the "submit" button sends the image to server:
```html run autorun height="90"
<body style="margin:0">
<canvas id="canvasElem" width="100" height="80" style="border:1px solid"></canvas>
<input type="button" value="Submit" onclick="submit()">
<script>
canvasElem.onmousemove = function(e) {
let ctx = canvasElem.getContext('2d');
ctx.lineTo(e.clientX, e.clientY);
ctx.stroke();
};
async function submit() {
let blob = await new Promise(resolve => canvasElem.toBlob(resolve, 'image/png'));
let response = await fetch('/article/fetch-basics/post/image', {
method: 'POST',
body: blob
});
let result = await response.json();
alert(result.message);
}
</script>
</body>
```
Here we also didn't need to set `Content-Type` manually, because a `Blob` object has a built-in type (here `image/png`, as generated by `toBlob`).
The `submit()` function can be rewritten without `async/await` like this:
```js
function submit() {
canvasElem.toBlob(function(blob) {
fetch('/article/fetch-basics/post/image', {
method: 'POST',
body: blob
})
.then(response => response.json())
.then(result => alert(JSON.stringify(result, null, 2)))
}, 'image/png');
}
```
## Custom FormData with image
In practice though, it's often more convenient to send an image as a part of the form, with additional fields, such as "name" and other metadata.
Also, servers are usually more suited to accept multipart-encoded forms, rather than raw binary data.
```html run autorun height="90"
<body style="margin:0">
<canvas id="canvasElem" width="100" height="80" style="border:1px solid"></canvas>
<input type="button" value="Submit" onclick="submit()">
<script>
canvasElem.onmousemove = function(e) {
let ctx = canvasElem.getContext('2d');
ctx.lineTo(e.clientX, e.clientY);
ctx.stroke();
};
async function submit() {
let blob = await new Promise(resolve => canvasElem.toBlob(resolve, 'image/png'));
*!*
let formData = new FormData();
formData.append("name", "myImage");
formData.append("image", blob);
*/!*
let response = await fetch('/article/fetch-basics/post/image-form', {
method: 'POST',
body: formData
});
let result = await response.json();
alert(result.message);
}
</script>
</body>
```
Now, from the server standpoint, the image is a "file" in the form.
## Summary
A typical fetch request consists of two `awaits`:
```js
let response = await fetch(url, options); // resolves with response headers
let result = await response.json(); // read body as json
```
Or, promise-style:
```js
fetch(url, options)
.then(response => response.json())
.then(result => /* process result */)
```
Response properties:
- `response.status` -- HTTP code of the response,
- `response.ok` -- `true` is the status is 200-299.
- `response.headers` -- Map-like object with HTTP headers.
Methods to get response body:
- **`response.json()`** -- parse the response as JSON object,
- **`response.text()`** -- return the response as text,
- **`response.formData()`** -- return the response as FormData object (form/multipart encoding),
- **`response.blob()`** -- return the response as [Blob](info:blob) (binary data with type),
- **`response.arrayBuffer()`** -- return the response as [ArrayBuffer](info:arraybuffer-binary-arrays) (pure binary data),
Fetch options so far:
- `method` -- HTTP-method,
- `headers` -- an object with request headers (not any header is allowed),
- `body` -- string/FormData/BufferSource/Blob/UrlSearchParams data to submit.
In the next chapters we'll see more options and use cases.

View file

Before

Width:  |  Height:  |  Size: 290 B

After

Width:  |  Height:  |  Size: 290 B

Before After
Before After

View file

@ -0,0 +1,57 @@
const Koa = require('koa');
const app = new Koa();
const bodyParser = require('koa-bodyparser');
const getRawBody = require('raw-body')
const busboy = require('async-busboy');
const Router = require('koa-router');
let router = new Router();
router.post('/user', async (ctx) => {
ctx.body = {
message: "User saved."
};
});
router.post('/image', async (ctx) => {
let body = await getRawBody(ctx.req, {
limit: '1mb'
});
ctx.body = {
message: `Image saved, size:${body.length}.`
};
});
router.post('/image-form', async (ctx) => {
let files = [];
const { fields } = await busboy(ctx.req, {
onFile(fieldname, file, filename, encoding, mimetype) {
// read all file stream to continue
getRawBody(file, { limit: '1mb'}).then(body => {
files.push({
fieldname,
filename,
length: body.length
});
})
}
});
ctx.body = {
message: `Image saved, name: ${fields.name}, size:${files[0].length}.`
};
});
app
.use(bodyParser())
.use(router.routes())
.use(router.allowedMethods());
if (!module.parent) {
http.createServer(app.callback()).listen(8080);
} else {
exports.accept = app.callback();
}

View file

@ -0,0 +1,104 @@
# Fetch: Download progress
Fetch allows to track download progress, but not upload progress.
Please note: there's currently no way for fetch to track upload progress. For that purpose, please use [XMLHttpRequest](info:xmlhttprequest).
To track download progress, we can use `response.body` property. It's a "readable stream" -- a special object that provides body chunk-by-chunk, as it comes, so we can see how much is available at the moment.
Here's the sketch of code that uses it to read response:
```js
// instead of response.json() and other methods
const reader = response.body.getReader();
// infinite loop while the body is downloading
while(true) {
// done is true for the last chunk
// value is Uint8Array of the chunk bytes
const {done, value} = await reader.read();
if (done) {
break;
}
console.log(`Received ${value.length} bytes`)
}
```
So, we loop, while `await reader.read()` returns response chunks.
A chunk has two properties:
- **`done`** -- true when the reading is complete.
- **`value`** -- a typed array of bytes: `Uint8Array`.
To log the progress, we just need to count chunks.
Here's the full code to get response and log the progress, more explanations follow:
```js run async
// Step 1: start the fetch and obtain a reader
let response = await fetch('https://api.github.com/repos/iliakan/javascript-tutorial-en/commits?per_page=100');
const reader = response.body.getReader();
// Step 2: get total length
const contentLength = +response.headers.get('Content-Length');
// Step 3: read the data
let receivedLength = 0; // length at the moment
let chunks = []; // array of received binary chunks (comprises the body)
while(true) {
const {done, value} = await reader.read();
if (done) {
break;
}
chunks.push(value);
receivedLength += value.length;
console.log(`Received ${receivedLength} of ${contentLength}`)
}
// Step 4: concatenate chunks into single Uint8Array
let chunksAll = new Uint8Array(receivedLength); // (4.1)
let position = 0;
for(let chunk of chunks) {
chunksAll.set(chunk, position); // (4.2)
position += chunk.length;
}
// Step 5: decode into a string
let result = new TextDecoder("utf-8").decode(chunksAll);
// We're done!
let commits = JSON.parse(result);
alert(commits[0].author.login);
```
Let's explain that step-by-step:
1. We perform `fetch` as usual, but instead of calling `response.json()`, we obtain a stream reader `response.body.getReader()`.
Please note, we can't use both these methods to read the same response. Either use a reader or a response method to get the result.
2. Prior to reading, we can figure out the full response length from the `Content-Length` header.
It may be absent for cross-domain requests (see chapter <info:fetch-crossorigin>) and, well, technically a server doesn't have to set it. But usually it's at place.
3. Call `await reader.read()` until it's done.
We gather response `chunks` in the array. That's important, because after the response is consumed, we won't be able to "re-read" it using `response.json()` or another way (you can try, there'll be an error).
4. At the end, we have `chunks` -- an array of `Uint8Array` byte chunks. We need to join them into a single result. Unfortunately, there's no single method that concatenates those, so there's some code to do that:
1. We create `new Uint8Array(receivedLength)` -- a same-typed array with the combined length.
2. Then use `.set(chunk, position)` method to copy each `chunk` one after another in the resulting array.
5. We have the result in `chunksAll`. It's a byte array though, not a string.
To create a string, we need to interpret these bytes. The built-in [TextDecoder](info:text-decoder) does exactly that. Then we can `JSON.parse` it.
What if we need binary content instead of JSON? That's even simpler. Instead of steps 4 and 5, we could make a blob of all chunks:
```js
let blob = new Blob(chunks);
```
Once again, please note, that's not for upload progress (no way now), only for download progress.

View file

Before

Width:  |  Height:  |  Size: 290 B

After

Width:  |  Height:  |  Size: 290 B

Before After
Before After

View file

@ -0,0 +1,110 @@
# Fetch: Abort
Aborting a `fetch` is a little bit tricky. Remember, `fetch` returns a promise. And Javascript generally has no concept of "aborting" a promise. So how can we cancel a fetch?
There's a special built-in object for such purposes: `AbortController`.
The usage is pretty simple:
- Step 1: create a controller:
```js
let controller = new AbortController();
```
The controller is an extremely simple object. It has a single method `abort()`, and a single property `signal`, that generates an event when `abort()` is called:
We can even use it without `fetch` for our own purposes, like this:
```js run
let controller = new AbortController();
let signal = controller.signal;
// triggers when controller.abort() is called
signal.addEventListener('abort', () => alert("abort!"));
controller.abort(); // abort!
alert(signal.aborted); // true (after abort)
```
- Step 2: pass the `signal` property to `fetch` option:
```js
let controller = new AbortController();
fetch(url, {
signal: controller.signal
});
```
Now `fetch` listens to the signal.
- Step 3: to abort, call `controller.abort()`:
```js
controller.abort();
```
We're done: `fetch` gets the event from `signal` and aborts the request.
When a fetch is aborted, its promise rejects with an error named `AbortError`, so we should handle it:
```js run async
// abort in 1 second
let controller = new AbortController();
setTimeout(() => controller.abort(), 1000);
try {
let response = await fetch('/article/fetch-abort/demo/hang', {
signal: controller.signal
});
} catch(err) {
if (err.name == 'AbortError') { // handle abort()
alert("Aborted!");
} else {
throw err;
}
}
```
**`AbortController` is scalable, it allows to cancel multiple fetches at once.**
For instance, here we fetch many `urls` in parallel, and the controller aborts them all:
```js
let urls = [...]; // a list of urls to fetch in parallel
let controller = new AbortController();
let fetchJobs = urls.map(url => fetch(url, {
signal: controller.signal
}));
let results = await Promise.all(fetchJobs);
// from elsewhere:
// controller.abort() stops all fetches
```
If wee have our own jobs, different from `fetch`, we can use a single `AbortController` to stop those, together with fetches.
```js
let urls = [...];
let controller = new AbortController();
let ourJob = new Promise((resolve, reject) => {
...
controller.signal.addEventListener('abort', reject);
});
let fetchJobs = urls.map(url => fetch(url, {
signal: controller.signal
}));
let results = await Promise.all([...fetchJobs, ourJob]);
// from elsewhere:
// controller.abort() stops all fetches and ourJob
```

View file

@ -0,0 +1,21 @@
const Koa = require('koa');
const app = new Koa();
const Router = require('koa-router');
let router = new Router();
router.get('/hang', async (ctx) => {
await new Promise(() => {});
});
app
.use(router.routes())
.use(router.allowedMethods());
if (!module.parent) {
http.createServer(app.callback()).listen(8080);
} else {
exports.accept = app.callback();
}

View file

@ -0,0 +1,9 @@
We need `Origin`, because sometimes `Referer` is absent. For instance, when we `fetch` HTTP-page from HTTPS (access less secure from more secure), then there's no `Referer`.
The [Content Security Policy](http://en.wikipedia.org/wiki/Content_Security_Policy) may forbid sending a `Referer`.
As we'll see, `fetch` also has options that prevent sending the `Referer` and even allow to change it (within the same site).
By specification, `Referer` is an optional HTTP-header.
Exactly because `Referer` is unreliable, `Origin` was invented. The browser guarantees correct `Origin` for cross-origin requests.

View file

@ -0,0 +1,28 @@
importance: 5
---
# Why do we need Origin?
As you probably know, there's HTTP-header `Referer`, that usually contains an url of the page which initiated a network request.
For instance, when fetching `http://google.com` from `http://javascript.info/some/url`, the headers look like this:
```
Accept: */*
Accept-Charset: utf-8
Accept-Encoding: gzip,deflate,sdch
Connection: keep-alive
Host: google.com
*!*
Origin: http://javascript.info
Referer: http://javascript.info/some/url
*/!*
```
As you can see, both `Referer` and `Origin` are present.
The questions:
1. Why `Origin` is needed, if `Referer` has even more information?
2. If it possible that there's no `Referer` or `Origin`, or it's incorrect?

View file

@ -0,0 +1,370 @@
# Fetch: Cross-Origin Requests
If we make a `fetch` from an arbitrary web-site, that will probably fail.
The core concept here is *origin* -- a domain/port/protocol triplet.
Cross-origin requests -- those sent to another domain or protocol or port -- require special headers from the remote side. That policy is called "CORS": Cross-Origin Resource Sharing.
For instance, let's try fetching `http://example.com`:
```js run async
try {
await fetch('http://example.com');
} catch(err) {
alert(err); // Failed to fetch
}
```
Fetch fails, as expected.
## Why?
Because cross-origin restrictions protect the internet from evil hackers.
Seriously. Let's make a very brief historical digression.
For many years Javascript did not have any special methods to perform network requests.
**A script from one site could not access the content of another site.**
That simple, yet powerful rule was a foundation of the internet security. E.g. a script from the page `hacker.com` could not access user's mailbox at `gmail.com`. People felt safe.
But web developers demanded more power. A variety of tricks were invented to work around it.
One way to communicate with another server was to submit a `<form>` there. People submitted it into `<iframe>`, just to stay on the current page, like this:
```html
<!-- form target -->
<iframe name="iframe"></iframe>
<!-- a form could be dynamically generated and submited by Javascript -->
<form target="iframe" method="POST" action="http://another.com/…">
...
</form>
```
- So, it was possible to make a GET/POST request to another site, even without networking methods.
- But as it's forbidden to access the content of an `<iframe>` from another site, it wasn't possible to read the response.
So, `<form>` allowed to submit the data anywhere, but the response content was unaccessible.
Another trick was to use a `<script src="http://another.com/…">` tag. A script could have any `src`, from any domain. But again -- it was impossible to access the raw content of such script.
If `another.com` intended to expose data for this kind of access, then a so-called "JSONP (JSON with padding)" protocol was used.
Here's the flow:
1. First, in advance, we declare a global function to accept the data, e.g. `gotWeather`.
2. Then we make a `<script>` and pass its name as the `callback` query parameter, e.g. `src="http://another.com/weather.json?callback=gotWeather"`.
3. The remote server dynamically generates a response that wraps the data into `gotWeather(...)` call.
4. As the script executes, `gotWeather` runs, and, as it's our function, we have the data.
Here's an example of the code to receive the data in JSONP:
```js run
// 1. Declare the function to process the data
function gotWeather({ temperature, humidity }) {
alert(`temperature: ${temperature}, humidity: ${humidity}`);
}
// 2. Pass its name as the ?callback parameter for the script
let script = document.createElement('script');
script.src = `http://cors.javascript.local/article/fetch-crossorigin/demo/script?callback=gotWeather`;
document.body.append(script);
// 3. The expected answer from the server looks like this:
/*
gotWeather({
temperature: 25,
humidity: 78
});
*/
```
That works, and doesn't violate security, because both sides agreed to pass the data this way. And, when both sides agree, it's definitely not a hack. There are still services that provide such access, as it works even for very old browsers.
After a while, modern network methods appeared. At first, cross-origin requests were forbidden. But as a result of long discussions, cross-domain requests were allowed, in a way that does not add any capabilities unless explicitly allowed by the server.
## Simple requests
[Simple requests](http://www.w3.org/TR/cors/#terminology) must satisfy the following conditions:
1. [Simple method](http://www.w3.org/TR/cors/#simple-method): GET, POST or HEAD
2. [Simple headers](http://www.w3.org/TR/cors/#simple-header) -- only allowed:
- `Accept`,
- `Accept-Language`,
- `Content-Language`,
- `Content-Type` with the value `application/x-www-form-urlencoded`, `multipart/form-data` or `text/plain`.
Any other request is considered "non-simple". For instance, a request with `PUT` method or with an `API-Key` HTTP-header does not fit the limitations.
**The essential difference is that a "simple request" can be made with a `<form>` or a `<script>`, without any special methods.**
So, even a very old server should be ready to accept a simple request.
Contrary to that, requests with non-standard headers or e.g. method `DELETE` can't be created this way. For a long time Javascript was unable to do such requests. So an old server may assume that such requests come from a privileged source, "because a webpage is unable to send them".
When we try to make a non-simple request, the browser sends a special "preflight" request that asks the server -- does it agree to accept such cross-origin requests, or not?
And, unless the server explicitly confirms that with headers, a non-simple request is not sent.
Now we'll go into details. All of them serve a single purpose -- to ensure that new cross-origin capabilities are only accessible with an explicit permission from the server.
## CORS for simple requests
If a request is cross-origin, the browser always adds `Origin` header to it.
For instance, if we request `https://anywhere.com/request` from `https://javascript.info/page`, the headers will be like:
```
GET /request
Host: anywhere.com
*!*
Origin: https://javascript.info
*/!*
...
```
As you can see, `Origin` contains exactly the origin (domain/protocol/port), without a path.
The server can inspect the `Origin` and, if it agrees to accept such a request, adds a special header `Access-Control-Allow-Origin` to the response. That header should contain the allowed origin (in our case `https://javascript.info`), or a star `*`. Then the response is successful, otherwise an error.
The browser plays the role of a trusted mediator here:
1. It ensures that the corrent `Origin` is sent with a cross-domain request.
2. If checks for correct `Access-Control-Allow-Origin` in the response, if it is so, then Javascript access, otherwise forbids with an error.
![](xhr-another-domain.png)
Here's an example of an "accepting" response:
```
200 OK
Content-Type:text/html; charset=UTF-8
*!*
Access-Control-Allow-Origin: https://javascript.info
*/!*
```
## Response headers
For cross-origin request, by default Javascript may only access "simple response headers":
- `Cache-Control`
- `Content-Language`
- `Content-Type`
- `Expires`
- `Last-Modified`
- `Pragma`
Any other response header is forbidden.
```smart header="Please note: no `Content-Length`"
Please note: there's no `Content-Length` header in the list!
So, if we're downloading something and would like to track the percentage of progress, then an additional permission is required to access that header (see below).
```
To grant Javascript access to any other response header, the server must list it in the `Access-Control-Expose-Headers` header.
For example:
```
200 OK
Content-Type:text/html; charset=UTF-8
Content-Length: 12345
API-Key: 2c9de507f2c54aa1
Access-Control-Allow-Origin: https://javascript.info
*!*
Access-Control-Expose-Headers: Content-Length,API-Key
*/!*
```
With such `Access-Control-Expose-Headers` header, the script is allowed to access `Content-Length` and `API-Key` headers of the response.
## "Non-simple" requests
We can use any HTTP-method: not just `GET/POST`, but also `PATCH`, `DELETE` and others.
Some time ago no one could even assume that a webpage is able to do such requests. So there may exist webservices that treat a non-standard method as a signal: "That's not a browser". They can take it into account when checking access rights.
So, to avoid misunderstandings, any "non-simple" request -- that couldn't be done in the old times, the browser does not make such requests right away. Before it sends a preliminary, so-called "preflight" request, asking for permission.
A preflight request uses method `OPTIONS` and has no body.
- `Access-Control-Request-Method` header has the requested method.
- `Access-Control-Request-Headers` header provides a comma-separated list of non-simple HTTP-headers.
If the server agrees to serve the requests, then it should respond with status 200, without body.
- The response header `Access-Control-Allow-Method` must have the allowed method.
- The response header `Access-Control-Allow-Headers` must have a list of allowed headers.
- Additionally, the header `Access-Control-Max-Age` may specify a number of seconds to cache the permissions. So the browser won't have to send a preflight for subsequent requests that satisfy given permissions.
![](xhr-preflight.png)
Let's see how it works step-by-step on example, for a cross-domain `PATCH` request (this method is often used to update data):
```js
let response = await fetch('https://site.com/service.json', {
method: 'PATCH',
headers: {
'Content-Type': 'application/json'
'API-Key': 'secret'
}
});
```
There are three reasons why the request is not simple (one is enough):
- Method `PATCH`
- `Content-Type` is not one of: `application/x-www-form-urlencoded`, `multipart/form-data`, `text/plain`.
- Custom `API-Key` header.
### Step 1 (preflight request)
The browser, on its own, sends a preflight request that looks like this:
```
OPTIONS /service.json
Host: site.com
Origin: https://javascript.info
Access-Control-Request-Method: PATCH
Access-Control-Request-Headers: Content-Type,API-Key
```
- Method: `OPTIONS`.
- The path -- exactly the same as the main request: `/service.json`.
- Cross-origin special headers:
- `Origin` -- the source origin.
- `Access-Control-Request-Method` -- requested method.
- `Access-Control-Request-Headers` -- a comma-separated list of "non-simple" headers.
### Step 2 (preflight response)
The server should respond with status 200 and headers:
- `Access-Control-Allow-Method: PATCH`
- `Access-Control-Allow-Headers: Content-Type,API-Key`.
That would allow future communication, otherwise an error is triggered.
If the server expects other methods and headers, makes sense to list them all at once, e.g:
```
200 OK
Access-Control-Allow-Methods: PUT,PATCH,DELETE
Access-Control-Allow-Headers: API-Key,Content-Type,If-Modified-Since,Cache-Control
Access-Control-Max-Age: 86400
```
Now the browser can see that `PATCH` is in the list of allowed methods, and both headers are in the list too, so it sends out the main request.
Besides, the preflight response is cached for time, specified by `Access-Control-Max-Age` header (86400 seconds, one day), so subsequent requests will not cause a preflight. Assuming that they fit the allowances, they will be sent directly.
### Step 3 (actual request)
When the preflight is successful, the browser now makes the real request. Here the flow is the same as for simple requests.
The real request has `Origin` header (because it's cross-origin):
```
PATCH /service.json
Host: site.com
Content-Type: application/json
API-Key: secret
Origin: https://javascript.info
```
### Step 4 (actual response)
The server should not forget to add `Accept-Control-Allow-Origin` to the response. A successful preflight does not relieve from that:
```
Access-Control-Allow-Origin: https://javascript.info
```
Now everything's correct. Javascript is able to read the full response.
## Credentials
A cross-origin request by default does not bring any credentials (cookies or HTTP authentication).
That's uncommon for HTTP-requests. Usually, a request to `http://site.com` is accompanied by all cookies from that domain. But cross-domain requests made by Javascript methods are an exception.
For example, `fetch('http://another.com')` does not send any cookies, even those that belong to `another.com` domain.
Why?
That's because a request with credentials is much more powerful than an anonymous one. If allowed, it grants Javascript the full power to act and access sensitive information on behalf of a user.
Does the server really trust pages from `Origin` that much? A request with credentials needs an additional header to pass through.
To enable credentials, we need to add the option `credentials: "include"`, like this:
```js
fetch('http://another.com', {
credentials: "include"
});
```
Now `fetch` sends cookies originating from `another.com` with the request.
If the server wishes to accept the request with credentials, it should add a header `Access-Control-Allow-Credentials: true` to the response, in addition to `Access-Control-Allow-Origin`.
For example:
```
200 OK
Access-Control-Allow-Origin: https://javascript.info
Access-Control-Allow-Credentials: true
```
Please note: `Access-Control-Allow-Origin` is prohibited from using a star `*` for requests with credentials. There must be exactly the origin there, like above. That's an additional safety measure, to ensure that the server really knows who it trusts.
## Summary
Networking methods split cross-origin requests into two kinds: "simple" and all the others.
[Simple requests](http://www.w3.org/TR/cors/#terminology) must satisfy the following conditions:
- Method: GET, POST or HEAD.
- Headers -- we can set only:
- `Accept`
- `Accept-Language`
- `Content-Language`
- `Content-Type` to the value `application/x-www-form-urlencoded`, `multipart/form-data` or `text/plain`.
The essential difference is that simple requests were doable since ancient times using `<form>` or `<script>` tags, while non-simple were impossible for browsers for a long time.
So, practical difference is that simple requests are sent right away, with `Origin` header, but for other ones the browser makes a preliminary "preflight" request, asking for permission.
**For simple requests:**
- → The browser sends `Origin` header with the origin.
- ← For requests without credentials (default), the server should set:
- `Access-Control-Allow-Origin` to `*` or same as `Origin`
- ← For requests with credentials, the server should set:
- `Access-Control-Allow-Origin` to `Origin`
- `Access-Control-Allow-Credentials` to `true`
Additionally, if Javascript wants no access non-simple response headers:
- `Cache-Control`
- `Content-Language`
- `Content-Type`
- `Expires`
- `Last-Modified`
- `Pragma`
...Then the server should list the allowed ones in `Access-Control-Expose-Headers` header.
**For non-simple requests, a preliminary "preflight" request is issued before the requested one:**
- → The browser sends `OPTIONS` request to the same url, with headers:
- `Access-Control-Request-Method` has requested method.
- `Access-Control-Request-Headers` lists non-simple requested headers
- ← The server should respond with status 200 and headers:
- `Access-Control-Allow-Method` with a list of allowed methods,
- `Access-Control-Allow-Headers` with a list of allowed headers,
- `Access-Control-Max-Age` with a number of seconds to cache permissions.
- Then the actual request is sent, the previous "simple" scheme is applied.

View file

Before

Width:  |  Height:  |  Size: 37 KiB

After

Width:  |  Height:  |  Size: 37 KiB

Before After
Before After

View file

Before

Width:  |  Height:  |  Size: 88 KiB

After

Width:  |  Height:  |  Size: 88 KiB

Before After
Before After

Binary file not shown.

After

Width:  |  Height:  |  Size: 34 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 76 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 58 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 131 KiB

After

Width:  |  Height:  |  Size: 136 KiB

Before After
Before After

View file

@ -0,0 +1,187 @@
# Fetch API
So far, we know quite a bit about fetch.
Now let's see the rest of API, to cover all its abilities.
Here's the full list of all possible fetch options with their default values (alternatives in comments):
```js
let promise = fetch(url, {
method: "GET", // POST, PUT, DELETE, etc.
headers: {
"Content-Type": "text/plain;charset=UTF-8" // for a string body, depends on body
},
body: undefined // string, FormData, Blob, BufferSource, or URLSearchParams
referrer: "about:client", // "" for no-referrer, or an url from the current origin
referrerPolicy: "no-referrer-when-downgrade", // no-referrer, origin, same-origin...
mode: "cors", // same-origin, no-cors
credentials: "same-origin", // omit, include
cache: "default", // no-store, reload, no-cache, force-cache, or only-if-cached
redirect: "follow", // manual, error
integrity: "", // a hash, like "sha256-abcdef1234567890"
keepalive: false, // true
signal: undefined, // AbortController to abort request
window: window // null
});
```
An impressive list, right?
We fully covered `method`, `headers` and `body` in the chapter <info:fetch-basics>.
The `signal` option is covered in <info:fetch-abort>.
Now let's explore the rest of options.
## referrer, referrerPolicy
These options govern how `fetch` sets HTTP `Referer` header.
That header contains the url of the page that made the request. In most scenarios, it plays a very minor informational role, but sometimes, for security purposes, it makes sense to remove or modify it.
.
**The `referrer` option allows to set any `Referer` within the current origin) or disable it.**
To send no referer, set an empty string:
```js
fetch('/page', {
*!*
referrer: "" // no Referer header
*/!*
});
```
To set another url within the current origin:
```js
fetch('/page', {
// assuming we're on https://javascript.info
// we can set any Referer header, but only within the current origin
*!*
referrer: "https://javascript.info/anotherpage"
*/!*
});
```
**The `referrerPolicy` option sets general rules for `Referer`.**
Possible values are described in the [Referrer Policy specification](https://w3c.github.io/webappsec-referrer-policy/):
- **`"no-referrer-when-downgrade"`** -- default value: `Referer` is sent always, unless we send a request from HTTPS to HTTP (to less secure protocol).
- **`"no-referrer"`** -- never send `Referer`.
- **`"origin"`** -- only send the origin in `Referer`, not the full page URL, e.g. `http://site.com` instead of `http://site.com/path`.
- **`"origin-when-cross-origin"`** -- send full referrer to the same origin, but only the origin part for cross-origin requests.
- **`"same-origin"`** -- send full referrer to the same origin, but no referer for for cross-origin requests.
- **`"strict-origin"`** -- send only origin, don't send referrer for HTTPS→HTTP requests.
- **`"strict-origin-when-cross-origin"`** -- for same-origin send full referrer, for cross-origin send only origin, unless it's HTTPS→HTTP request, then send nothing.
- **`"unsafe-url"`** -- always send full url in `Referer`.
Let's say we have an admin zone with URL structure that shouldn't be visible from outside.
If we send a cross-origin `fetch`, then by default it sends the `Referer` header with the full url of our page (except when we request from HTTPS to HTTP, then no `Referer`).
E.g. `Referer: https://javascript.info/admin/secret/paths`.
If we'd like to totally hide the referrer:
```js
fetch('https://another.com/page', {
referrerPolicy: "no-referrer" // no Referer, same effect as referrer: ""
});
```
Otherwise, if we'd like the remote side to see where the request comes from, we can send only the "origin" part of the url:
```js
fetch('https://another.com/page', {
referrerPolicy: "strict-origin" // Referer: https://javascript.info
});
```
## mode
The `mode` option serves as a safe-guard that prevents cross-origin requests:
- **`"cors"`** -- the default, cross-origin requests are allowed, as described in <info:fetch-crossorigin>.
- **`"same-origin"`** -- cross-origin requests are forbidden
- **`"no-cors"`** -- only simple cross-origin requests are allowed.
That may be useful in contexts when the fetch url comes from 3rd-party, and we want a "power off switch" to limit cross-origin capabilities.
## credentials
The `credentials` option specifies whether `fetch` should send cookies and HTTP-Authorization headers with the request.
- **`"same-origin"`** -- the default, don't send for cross-origin requests,
- **`"include"`** -- always send, requires `Accept-Control-Allow-Credentials` from cross-origin server,
- **`"omit"`** -- never send.
## cache
By default, `fetch` requests make use of standard HTTP-caching. That is, it honors `Expires`, `Cache-Control` headers, sends `If-Modified-Since`, and so on. Just like regular HTTP-requests do.
The `cache` options allows to ignore HTTP-cache or fine-tune interaction its usage:
- **`"default"`** -- `fetch` uses standard HTTP-cache rules and headers,
- **`"no-store"`** -- totally ignore HTTP-cache, this mode becomes default if we set a header `If-Modified-Since`, `If-None-Match`, `If-Unmodified-Since`, `If-Match`, or `If-Range`.
- **`"reload"`** -- don't take the result from HTTP-cache (if any), but populate cache with the response (if response headers allow)
- **`"no-cache"`** -- create a conditional request if there is a cached response, and a normal request otherwise. Populate HTTP-cache with the response.
- **`"force-cache"`** -- use any response from HTTP-cache, even if it's stale. If there's no response in HTTP-cache, behave normally.
- **`"only-if-cached"`** -- use any response from HTTP-cache, even if it's stale. If there's no response in HTTP-cache, then error. Only works when `mode` is `"same-origin"`.
## redirect
Normally, `fetch` transparently follows HTTP-redirects, like 301, 302 etc.
The `redirect` option allows to change that:
- **`"follow"`** -- the default, follow HTTP-redirects,
- **`"error"`** -- error in case of HTTP-redirect,
- **`"manual"`** -- don't follow HTTP-redirect, but `response.url` will be the new URL, and `response.redirected` will be `true`, so that we can perform the redirect manually to the new URL (if needed).
## integrity
The `integrity` option allows to check if the response matches the known-ahead checksum.
As described in the [specification](https://w3c.github.io/webappsec-subresource-integrity/), supported hash-functions are SHA-256, SHA-384, and SHA-512, there might be others depending on a browser.
For example, we're downloading a file, and we know that it's SHA-256 checksum is "abc" (a real checksum is longer, of course).
We can put it in the `integrity` option, like this:
```js
fetch('http://site.com/file', {
integrity: 'sha256-abd'
});
```
Then `fetch` will calculate SHA-256 on its own and compare it with our string. In case of a mismatch, an error is triggered.
## keepalive
The `keepalive` option indicates that the request may outlive the page.
For example, we gather statistics about how the current visitor uses our page (mouse clicks, interesting page fragments), to improve user experience.
When the visitor leaves our page -- we'd like to save it on our server.
We can use `window.onunload` for that:
```js run
window.onunload = function() {
fetch('/analytics', {
method: 'POST',
body: "statistics",
*!*
keepalive: true
*/!*
});
};
```
Normally, when a document is unloaded, all associated network requests are aborted. But `keepalive` option tells the browser to perform the request in background, even after it leaves the page. So it's essential for our request to succeed.
- We can't send megabytes: the body limit for keepalive requests is 64kb. If we gather more data, we can send it out regularly, then there won't be a lot for the last request.
- In case of `onunload` the request is completed when the current document is already unloaded, so we don't get server response (which is usually empty).

View file

Before

Width:  |  Height:  |  Size: 290 B

After

Width:  |  Height:  |  Size: 290 B

Before After
Before After

View file

Before

Width:  |  Height:  |  Size: 16 KiB

After

Width:  |  Height:  |  Size: 16 KiB

Before After
Before After

View file

Before

Width:  |  Height:  |  Size: 35 KiB

After

Width:  |  Height:  |  Size: 35 KiB

Before After
Before After

View file

@ -1,109 +0,0 @@
# Fetch basics
Method `fetch()` is the modern way of sending requests over HTTP.
Is evolved for several years and continues to improve, right now its support is pretty solid among browsers.
The basic syntax is:
```js
let promise = fetch(url, [params])
```
- **`url`** -- the URL to access.
- **`params`** -- optional parameters: method, headers etc.
The browser starts the request right away and returns a `promise`.
Accepting a response is usually a two-stage process.
**The `promise` resolves with an object of the built-in [Response](https://fetch.spec.whatwg.org/#response-class) class as soon as the server responds with headers.**
So we can access the headers, we know HTTP status, whether it is successful, but don't have the body yet.
The main response properties are:
- **`ok`** -- boolean, `true` if the HTTP status code is 200-299.
- **`status`** -- HTTP status code.
- **`headers`** -- HTTP headers, a Map-like object.
We can iterate over headers the same way as over a `Map`:
```js run async
let response = await fetch('https://api.github.com/repos/iliakan/javascript-tutorial-en/commits');
// have headers already
for (let [key, value] of response.headers) {
alert(`${key} = ${value}`);
}
if (response.ok) {
// wait for the body
let json = await response.json();
} else {
// handle error
}
```
To get the response body, we need to use an additional method call.
`Response` allows to access the body in multiple formats, using following promise-based methods:
- **`json()`** -- parse as JSON object,
- **`text()`** -- as text,
- **`formData()`** -- as formData (form/multipart encoding),
- **`blob()`** -- as Blob (for binary data),
- **`arrayBuffer()`** -- as ArrayBuffer (for binary data)
- additionally, `response.body` is a [ReadableStream](https://streams.spec.whatwg.org/#rs-class) object, it allows to read the body chunk-by-chunk.
For instance, here we get the response as JSON:
```js run async
let response = await fetch('https://api.github.com/repos/iliakan/javascript-tutorial-en/commits');
*!*
let commits = await response.json();
*/!*
alert(commits[0].author.login);
```
Or, using pure promises:
```js run
fetch('https://api.github.com/repos/iliakan/javascript-tutorial-en/commits')
.then(response => response.json())
.then(commits => alert(commits[0].author.login));
```
To get text:
```js
let text = await response.text();
```
And for the binary example, let's fetch and show an image (see chapter [Blob](info:blob) for details about operations on blobs):
```js async run
let response = await fetch('/article/fetch/logo-fetch.svg');
*!*
let blob = await response.blob(); // download as Blob object
*/!*
// create <img> for it
let img = document.createElement('img');
img.style = 'position:fixed;top:10px;left:10px;width:100px';
document.body.append(img);
// show it
img.src = URL.createObjectURL(blob);
setTimeout(() => { // hide after two seconds
img.remove();
URL.revokeObjectURL(img.src);
}, 2000);
```
```warn
Please note: we can use only one of these methods.
If we get `response.text()`, then `response.json()` won't work, as the body content has already been processed.
```

View file

@ -1,9 +0,0 @@
`Origin` нужен, потому что `Referer` передаётся не всегда. В частности, при запросе с HTTPS на HTTP -- нет `Referer`.
Политика [Content Security Policy](http://en.wikipedia.org/wiki/Content_Security_Policy) может запрещать пересылку `Referer`.
По стандарту `Referer` является необязательным HTTP-заголовком, в некоторых браузерах есть настройки, которые запрещают его слать.
Именно поэтому, ввиду того, что на `Referer` полагаться нельзя, и придумали заголовок `Origin`, который гарантированно присылается при кросс-доменных запросах.
Что же касается "неправильного" `Referer` -- это из области фантастики. Когда-то, много лет назад, в браузерах были ошибки, которые позволяли подменить `Referer` из JavaScript, но они давно исправлены. Никакая "злая страница" не может его подменить.

View file

@ -1,30 +0,0 @@
importance: 5
---
# Зачем нужен Origin?
Как вы, наверняка, знаете, существует HTTP-заголовок `Referer`, в котором обычно указан адрес страницы, с которой инициирован запрос.
Например, при отправке `XMLHttpRequest` со страницы `http://javascript.ru/some/url` на `http://google.ru`, заголовки будут примерно такими:
```
Accept:*/*
Accept-Charset:windows-1251,utf-8;q=0.7,*;q=0.3
Accept-Encoding:gzip,deflate,sdch
Accept-Language:ru-RU,ru;q=0.8,en-US;q=0.6,en;q=0.4
Connection:keep-alive
Host:google.ru
*!*
Origin:http://javascript.ru
Referer:http://javascript.ru/some/url
*/!*
```
Как видно, здесь присутствуют и `Referer` и `Origin`.
Итак, вопросы:
1. Зачем нужен `Origin`, если `Referer` содержит даже более полную информацию?
2. Может ли быть такое, что заголовка `Referer` нет или он неправильный?

View file

@ -1,329 +0,0 @@
# Cross-Origin Requests
If we make a `fetch` from an arbitrary web-site, that will probably fail.
The core concept here is *origin* -- a domain/port/protocol triplet.
Cross-origin requests -- those sent to another domain or protocol or port -- require special headers from the remote side. That policy is called "CORS": Cross-Origin Resource Sharing.
For instance, let's try fetching from `http://example.com`:
```js run async
try {
await fetch('http://example.com');
} catch(err) {
alert(err); // Failed to fetch
}
```
Fetch fails, as expected.
## Why?
Because cross-origin restrictions protect the internet from evil hackers.
Seriously. Let's make a very brief historical digression.
For many years Javascript did not have any special methods to perform network requests.
One way to communicate with another server was to submit a `<form>` there. People submitted it into `<iframe>`, just to stay on the current page, like this:
```html
<iframe name="iframe"></iframe>
<form target="iframe" method="POST" action="http://another.com/…">
<!-- a form could be dynamically generated and submited by Javascript -->
</form>
```
- So, it was possible to make a GET/POST request to another site.
- But as it's forbidden to access the content of an `<iframe>` from another site, it wasn't possible to read the response.
**A script from one site could not access the content of another site.**
That simple, yet powerful rule was a foundation of the internet security. E.g. a script from the page `hacker.com` could not access user's mailbox at `gmail.com`. People felt safe.
But growing appetites of web developers demanded more power. A variety of tricks were invented.
One of them was to use a `<script src="http://another.com/…">` tag. It could request a script from anywhere, but again -- it was forbidden to access its content (if from another site).
So if `another.com` was going to send back the data, it would be wrapped in a callback function with the name known to the receiving side. Usually, an url parameter, such as `?callback=…`, was used to specify the name.
For instance:
1. We declare a global function, e.g. `gotWeather`.
1. Our page is requesting `http://another.com/weather.json?callback=gotWeather`.
2. The remote server dynamically generates a script wrapping the response data into `gotWeather(...)`. As `gotWeather` is our global function, it handles the response: we have the data.
Here's the demo:
```js run
function gotWeather({ temperature, humidity }) {
alert(`temperature: ${temperature}, humidity: ${humidity}`);
}
let script = document.createElement('script');
script.src = `http://cors.javascript.local/article/fetch-crossorigin/demo/script?callback=gotWeather`;
document.body.append(script);
```
The response looks like this:
```js
gotWeather({
temperature: 25,
humidity: 78
});
```
That works, and doesn't violate security, because both sides agreed to pass the data this way. And, when both sides agree, it's definitely not a hack.
Then `XMLHttpRequest` appeared, and `fetch` afterwards. After long discussions, it was decided that cross-domain requests would be possible, but in a way that does not add any capabilities unless explicitly allowed by the server.
## Simple requests
[Simple requests](http://www.w3.org/TR/cors/#terminology) must satisfy the following conditions:
1. [Simple method](http://www.w3.org/TR/cors/#simple-method): GET, POST or HEAD
2. [Simple headers](http://www.w3.org/TR/cors/#simple-header) -- only allowed:
- `Accept`
- `Accept-Language`
- `Content-Language`
- `Content-Type` with the value `application/x-www-form-urlencoded`, `multipart/form-data` or `text/plain`.
Any other request is considered "non-simple". For instance, a request with `PUT` method or with an `Authorization` HTTP-header does not fit the limitations.
**The essential difference is that a "simple request" can be made with a `<form>` or a `<script>`, without any special methods.**
So, even a very old server should be ready to accept a simple request.
Contrary to that, headers with non-standard headers or e.g. `DELETE` method can't be created this way. For a long time Javascript was unable to do such requests. So an old server may assume that such requests come from a privileged source, "because a webpage is unable to send them".
That's why we try to make a non-simple request, the browser sends a special "preflight" request that asks the server -- does it agree to accept such cross-origin requests, or not?
And, unless the server explicitly confirms that with headers, a non-simple request is not sent.
Now we'll go into details. All of them serve a single purpose -- to ensure that new cross-origin capabilities are only accessible with an explicit permission from the server.
## CORS for simple requests
A browser always adds to cross-origin requests header `Origin`, with the origin of the request.
For instance, if we request `https://anywhere.com/request` from `https://javascript.info/page`, the headers will be like:
```
GET /request
Host:anywhere.com
*!*
Origin:https://javascript.info
*/!*
...
```
Please note: `Origin` has no path, only domain/protocol/port.
The server can inspect the `Origin` and, if it agrees to accept such a request, add a special header `Access-Control-Allow-Origin` to the response. That header should contain the allowed origin (in our case `https://javascript.info`), or a star `*`.
The browser plays the role of a trusted mediator here:
1. It ensures that the corrent `Origin` is sent in any cross-domain request.
2. If checks for correct `Access-Control-Allow-Origin` in the response, grants Javascript access if it's so, forbids otherwise.
![](xhr-another-domain.png)
Here's an example of an "accepting" response:
```
200 OK
Content-Type:text/html; charset=UTF-8
*!*
Access-Control-Allow-Origin: https://javascript.info
*/!*
```
## Response headers
For cross-origin request, by default Javascript may only access "simple response headers":
- `Cache-Control`
- `Content-Language`
- `Content-Type`
- `Expires`
- `Last-Modified`
- `Pragma`
Any other header is forbidden.
```smart
Please note: there's no `Content-Length` header in the list! So, if we're downloading something, we can't track the percentage of progress.
```
To grant Javascript access to any other header, the server must list it in the `Access-Control-Expose-Headers` header.
For example:
```
200 OK
Content-Type:text/html; charset=UTF-8
Access-Control-Allow-Origin: https://javascript.info
Content-Length: 12345
Authorization: 2c9de507f2c54aa1
*!*
Access-Control-Expose-Headers: Content-Length, Authorization
*/!*
```
With such `Access-Control-Expose-Headers` header, the script is allowed to access `Content-Length` and `Authorization` headers of the response.
## "Non-simple" requests
We can use any HTTP-method: not just `GET/POST`, but also `PATCH`, `DELETE` and others.
Some time ago no one could even assume that a webpage is able to do such requests. So there may exist webservices that treat a non-standard method as a signal: "That's not a browser". They can take it into account when checking access rights.
So, to avoid misunderstandings, any "non-simple" request -- that couldn't be done in the old times, the browser does not make such requests right away. Before it sends a preliminary, so-called "preflight" request, asking for permission.
A preflight request uses method `OPTIONS` and has no body:
- `Access-Control-Request-Method` header has the requested method.
- `Access-Control-Request-Headers` header provides a comma-separated list of non-simple HTTP-headers.
If the server agrees to serve the requests, then it should respond with status 200, no body and:
- The response header `Access-Control-Allow-Method` must have the allowed method.
- The response header `Access-Control-Allow-Headers` must have a list of allowed headers.
- Additionally, the header `Access-Control-Max-Age` may specify a number of seconds to cache the permissions. So the browser won't have to send a preflight for subsequent requests that satisfy given permissions.
![](xhr-preflight.png)
Let's see how it works step-by-step for a cross-domain `PATCH` request:
```js
let response = await fetch('https://site.com/service.json', {
method: 'PATCH', // this method is often used to update data
headers: {
'Content-Type': 'application/json'
'API-Key': 'secret'
}
});
```
There are three reasons why the request is not simple (one is enough):
- Method `PATCH`
- `Content-Type` is not one of: `application/x-www-form-urlencoded`, `multipart/form-data`, `text/plain`.
- Custom `API-Key` header.
### Step 1 (preflight request)
The browser, on its own, sends a preflight request that looks like this:
```
OPTIONS /service.json
Host: site.com
Origin: https://javascript.info
Access-Control-Request-Method: PATCH
Access-Control-Request-Headers: Content-Type,API-Key
```
- Method: `OPTIONS`.
- The path -- exactly the same as the main request: `/service.json`.
- Cross-origin special headers:
- `Origin` -- the source origin.
- `Access-Control-Request-Method` -- requested method.
- `Access-Control-Request-Headers` -- a comma-separated list of "non-simple" headers.
### Step 2 (preflight response)
The server can respond with status 200 and headers:
- `Access-Control-Allow-Method: PATCH`
- `Access-Control-Allow-Headers: Content-Type,API-Key`.
...But if it expects other methods and headers, makes sense to list them all at once:
```
200 OK
Access-Control-Allow-Methods: PUT,PATCH,DELETE
Access-Control-Allow-Headers: API-Key,Content-Type,If-Modified-Since,Cache-Control
Access-Control-Max-Age: 86400
```
Now the browser can see that `PATCH` is an allowed method, and both headers are in the list too, so it sends the main request.
Besides, the preflight response is cached for 86400 seconds (one day), so subsequent requests will not cause a preflight. Assuming that they fit the allowances, they will be sent directly, without `OPTIONS`.
### Step 3 (actual request)
When the preflight is successful, the browser now makes the real request.
It has `Origin` header (because it's cross-origin):
```
PATCH /service.json
Host: site.com
Content-Type: application/json
API-Key: secret
Origin: https://javascript.info
```
### Step 4 (actual response)
The server should not forget `Accept-Control-Allow-Origin` in the response. A successful preflight does not relieve from that:
```
Access-Control-Allow-Origin: https://javascript.info
```
Now it's done. Because of `Access-Control-Max-Age: 86400`, preflight permissions are cached for 86400 seconds (1 day).
## Credentials
A cross-origin request by default does not bring any cookies.
for example, `fetch('http://another.com')` does not send any cookies, even those that belong to `another.com` domain.
To send cookies, we need to add the option `credentials: "include"`.
Like this:
```js
fetch('http://another.com', {
credentials: "include"
});
```
Now `fetch` sends cookies originating from `another.com` with the request.
A request with credentials is much more powerful than an anonymouse one. Does the server trust Javascript to work on behalf of a user? Then it will access sensitive information, these permissions are not for everyone.
The server should add a new header `Access-Control-Allow-Credentials: true` to the response, if it allows requests with credentials.
For example:
```
200 OK
Access-Control-Allow-Origin: https://javascript.info
Access-Control-Allow-Credentials: true
```
Please note: using a start `*` in `Access-Control-Allow-Origin` is forbidden in such case. The server must explicitly put the trusted origin there. That's an additional safety measure, to ensure that the server really knows who it trusts.
## Итого
- Все современные браузеры умеют делать кросс-доменные XMLHttpRequest.
- В IE8,9 для этого используется объект `XDomainRequest`, ограниченный по возможностям.
- Кросс-доменный запрос всегда содержит заголовок `Origin` с доменом запроса.
Порядок выполнения:
1. Для запросов с "непростым" методом или особыми заголовками браузер делает предзапрос `OPTIONS`, указывая их в `Access-Control-Request-Method` и `Access-Control-Request-Headers`.
Браузер ожидает ответ со статусом `200`, без тела, со списком разрешённых методов и заголовков в `Access-Control-Allow-Method` и `Access-Control-Allow-Headers`. Дополнительно можно указать `Access-Control-Max-Age` для кеширования предзапроса.
2. Браузер делает запрос и проверяет, есть ли в ответе `Access-Control-Allow-Origin`, равный `*` или `Origin`.
Для запросов с `withCredentials` может быть только `Origin` и дополнительно `Access-Control-Allow-Credentials: true`.
3. Если проверки пройдены, то вызывается `xhr.onload`, иначе `xhr.onerror`, без деталей ответа.
4. Дополнительно: названия нестандартных заголовков ответа сервер должен указать в `Access-Control-Expose-Headers`, если хочет, чтобы клиент мог их прочитать.
Детали и примеры мы разобрали выше.

Binary file not shown.

Before

Width:  |  Height:  |  Size: 34 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 75 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 56 KiB

View file

@ -1,165 +0,0 @@
# Fetch API
TODO: FormData in detail https://xhr.spec.whatwg.org/#formdata?
The second argument provides a lot of flexibility to `fetch` syntax.
Here's the full list of possible options with default values (alternatives commented out):
```js
let promise = fetch(url, {
method: "GET", // POST, PUT, DELETE, etc.
headers: {
"Content-Type": "text/plain;charset=UTF-8" // for a string body, depends on body
},
body: undefined // string, FormData, Blob, BufferSource, or URLSearchParams
referrer: "about:client", // "" for no-referrer, or an url from the current origin
referrerPolicy: "no-referrer-when-downgrade", // no-referrer, origin, same-origin...
mode: "cors", // same-origin, no-cors
credentials: "same-origin", // omit, include
cache: "default", // no-store, reload, no-cache, force-cache, or only-if-cached
redirect: "follow", // manual, error
integrity: "", // a hash, like "sha256-abcdef1234567890"
keepalive: false, // true
signal: undefined, // AbortController to abort request
window: window // null
});
```
An impressive list, right? A lot of capabilities.
Let's explore what `fetch` can do.
We'll explore the options one-by-one with examples.
## method, headers, body
These are the most widely used fields.
- **`method`** -- HTTP-method, e.g. `POST`,
- **`headers`** -- an object with HTTP headers,
- **`body`** -- a string, or:
- FormData object, to submit `form/multipart`
- Blob/BufferSource to send binary data
- URLSearchParams, to submit `x-www-form-urlencoded`
For instance, to submit a `user` object as JSON:
```js run async
let user = {
name: 'John',
surname: 'Smith'
};
*!*
let response = await fetch('/article/fetch-api/post/user', {
method: 'POST',
headers: {
'Content-Type': 'application/json;charset=utf-8'
},
body: JSON.stringify(user)
});
*/!*
let result = await response.json();
alert(JSON.stringify(result, null, 2));
```
Please note, for a string body `Content-Type` is `text/plain;charset=UTF-8` by default. So the correct header is set manually.
Let's do the same with an HTML `<form>`:
```html run
<form id="formElem">
<input type="text" name="name" value="John">
<input type="text" name="surname" value="Smith">
</form>
<script>
fetch('/article/fetch-api/post/user', {
method: 'POST',
body: new FormData(formElem)
})
.then(response => response.json())
.then(result => alert(JSON.stringify(result, null, 2)))
</script>
```
Here [FormData](https://xhr.spec.whatwg.org/#formdata) automatically encodes the form (with files if any), and the `Content-Type` is `form/multipart`.
We can also submit binary data directly using `Blob` or `BufferSource`.
For example, here's a `<canvas>` where we can draw by moving a mouse. A click on the "submit" button sends the image to server:
```html run autorun height="90"
<body style="margin:0">
<canvas id="canvasElem" width="100" height="80" style="border:1px solid"></canvas>
<input type="button" value="Submit" onclick="submit()">
<script>
canvasElem.onmousemove = function(e) {
let ctx = canvasElem.getContext('2d');
ctx.lineTo(e.clientX, e.clientY);
ctx.stroke();
};
function submit() {
canvasElem.toBlob(function(blob) {
fetch('/article/fetch-api/post/image', {
method: 'POST',
body: blob
})
.then(response => response.json())
.then(result => alert(JSON.stringify(result, null, 2)))
}, 'image/png');
}
</script>
</body>
```
Here we also didn't need to set `Content-Type` manually, because a `Blob` object has a built-in type (here `image/png`, as generated by `toBlob`).
```warn header="Not any header is allowed"
There's a list of [forbidden HTTP headers](https://fetch.spec.whatwg.org/#forbidden-header-name) that we can't set:
- `Accept-Charset`, `Accept-Encoding`
- `Access-Control-Request-Headers`
- `Access-Control-Request-Method`
- `Connection`
- `Content-Length`
- `Cookie`, `Cookie2`
- `Date`
- `DNT`
- `Expect`
- `Host`
- `Keep-Alive`
- `Origin`
- `Referer`
- `TE`
- `Trailer`
- `Transfer-Encoding`
- `Upgrade`
- `Via`
- `Proxy-*`
- `Sec-*`
All these headers are controlled solely by the browser, to ensure proper and safe HTTP. So we can't set them.
```
## referrer, referrerPolicy
These options govern how `fetch` sets HTTP `Referer` header (header has one `r`, for historical reasons).
That header tells the server which page made the request. In most scenarios, it plays a very minor informational role, so if you don't have specific reasons to care about it, like increased security, please skip to the next section.
- **`"no-referrer-when-downgrade"`** -- default value, the referer is set always, unless we send a request from HTTPS to HTTP (to less secure protocol).
- **`"no-referrer"`** -- don't set the referer.
- **`"origin"`** -- only set the domain, not the full page URL, e.g. `http://site.com` instead of `http://site.com/path`.
no-referrer
The Referer header will be omitted entirely. No referrer information is sent along with requests.
(default)
This is the user agent's default behavior if no policy is specified. The URL is sent as a referrer when the protocol security level stays the same (HTTP→HTTP, HTTPS→HTTPS), but isn't sent to a less secure destination (HTTPS→HTTP).

View file

@ -1,96 +0,0 @@
# Fetch: Track download progress
To track download progress, we need to use `response.body`.
It's a "readable stream" - a special object that provides access chunk-by-chunk.
Here's the code to do this:
```js
const reader = response.body.getReader();
while(true) {
// done is true for the last chunk
// value is Uint8Array of bytes
const chunk = await reader.read();
if (chunk.done) {
break;
}
console.log(`Received ${chunk.value.length} bytes`)
}
```
We do the infinite loop, while `await reader.read()` returns response chunks.
A chunk has two properties:
- **`done`** -- true when the reading is complete.
- **`value`** -- a typed array of bytes: `Uint8Array`.
The full code to get response and log the progress:
```js run async
// Step 1: start the request and obtain a reader
let response = await fetch('https://api.github.com/repos/iliakan/javascript-tutorial-en/commits?per_page=100');
const reader = response.body.getReader();
// Step 2: get total length
const contentLength = +response.headers.get('Content-Length');
// Step 3: read the data
let receivedLength = 0;
let chunks = [];
while(true) {
const {done, value} = await reader.read();
if (done) {
break;
}
chunks.push(value);
receivedLength += value.length;
console.log(`Received ${receivedLength} of ${contentLength}`)
}
// Step 4: join chunks into result
let chunksAll = new Uint8Array(receivedLength); // (4.1)
let position = 0;
for(let chunk of chunks) {
chunksAll.set(chunk, position); // (4.2)
position += chunk.length;
}
// Step 5: decode into a string
let result = new TextDecoder("utf-8").decode(chunksAll);
let commits = JSON.parse(result);
// We're done!
alert(commits[0].author.login);
```
Let's explain that step-by-step:
1. We perform `fetch` as usual, but instead of calling `response.json()`, we obtain a stream reader `response.body.getReader()`.
Please note, we can't use both these methods to read the same response. Either use a reader or a response method to get the result.
2. Prior to reading, we can figure out the full response length by its `Content-Length` header.
It may be absent for cross-domain requests (as in the example) and, well, technically a server doesn't have to set it. But usually it's at place.
3. Now `await reader.read()` until it's done.
We gather the `chunks` in the array. That's important, because after the response is consumed, we won't be able to "re-read" it using `response.json()` or another way (you can try, there'll be an error).
4. At the end, we have `chunks` -- an array of `Uint8Array` byte chunks. We need to join them into a single result. Unfortunately, there's no single method that concatenates those.
1. We create `new Uint8Array(receivedLength)` -- a same-type array with the combined length.
2. Then use `.set(chunk, position)` method that copies each `chunk` at the given `position` (one by one) in the resulting array.
5. We have the result in `chunksAll`. It's a byte array though, not a string.
To create a string, we need to interpret these bytes. The built-in `TextEncoder` does exactly that. Then we can `JSON.parse` it.
What if we need binary content instead of JSON? That's even simpler. Instead of steps 4 and 5, we could make a blob of all chunks:
```js
let blob = new Blob(chunks);
```

View file

@ -1,36 +0,0 @@
<!doctype html>
<script>
(async () {
const response = await fetch('long.txt');
const reader = response.body.getReader();
const contentLength = +response.headers.get('Content-Length');
let receivedLength = 0;
let chunks = [];
while(true) {
const chunk = await reader.read();
if (chunk.done) {
console.log("done!");
break;
}
chunks.push(chunk.value);
receivedLength += chunk.value.length;
console.log(`${receivedLength}/${contentLength} received`)
}
let chunksMerged = new Uint8Array(receivedLength);
let length = 0;
for(let chunk of chunks) {
chunksMerged.set(chunk, length);
length += chunk.length;
}
let result = new TextDecoder("utf-8").decode(chunksMerged);
console.log(result);
})();
</script>

View file

@ -1,55 +0,0 @@
let http = require('http');
let url = require('url');
let querystring = require('querystring');
let static = require('node-static');
let file = new static.Server('.', {
cache: 0
});
function accept(req, res) {
if (req.method == 'POST') {
let chunks = [];
let length = 0;
req.on('data', function (data) {
chunks.push(data);
length += data.length;
// Too much POST data, kill the connection!
if (length > 1e6) {
request.connection.destroy();
}
});
req.on('end', function() {
// let post = JSON.parse(chunks.join(''));
if (req.url == '/user') {
res.writeHead(200, { 'Content-Type': 'application/json' });
res.end(JSON.stringify({ message: 'User saved' }));
} else if (req.url == '/image') {
res.writeHead(200, { 'Content-Type': 'application/json' });
res.end(JSON.stringify({ message: "Image saved", imageSize: length }));
} else {
res.writeHead(404);
res.end("Not found");
}
});
} else {
file.serve(req, res);
}
}
// ------ запустить сервер -------
if (!module.parent) {
http.createServer(accept).listen(8080);
} else {
exports.accept = accept;
}

View file

@ -1,36 +0,0 @@
<!doctype html>
<script>
(async () {
const response = await fetch('long.txt');
const reader = response.body.getReader();
const contentLength = +response.headers.get('Content-Length');
let receivedLength = 0;
let chunks = [];
while(true) {
const chunk = await reader.read();
if (chunk.done) {
console.log("done!");
break;
}
chunks.push(chunk.value);
receivedLength += chunk.value.length;
console.log(`${receivedLength}/${contentLength} received`)
}
let chunksMerged = new Uint8Array(receivedLength);
let length = 0;
for(let chunk of chunks) {
chunksMerged.set(chunk, length);
length += chunk.length;
}
let result = new TextDecoder("utf-8").decode(chunksMerged);
console.log(result);
})();
</script>

File diff suppressed because it is too large Load diff

Binary file not shown.