test: Add e2e release gate (script + manual checklist)
Releases need more than eyeballs: scripts/e2e-api-check.sh now runs 18 assertions over the live instance (login/refresh, listings, private rows, subscriptions incl. remote channels, watching + stored position, file token, playlist reachability) and TESTS.md defines the manual D-pad checkout that a script cannot do. Also switch the channels screen to the user's own channels (the search-based local listing rejects empty queries) and resolve the real account name at login since logins accept email addresses.
This commit is contained in:
parent
764fa8b34c
commit
83dde8ba74
10 changed files with 284 additions and 8 deletions
95
TESTS.md
Normal file
95
TESTS.md
Normal file
|
|
@ -0,0 +1,95 @@
|
|||
# E2E tests — release gate
|
||||
|
||||
How to validate a release. Two layers:
|
||||
|
||||
1. **Scripted API smoke test** — `scripts/e2e-api-check.sh`, runs anywhere sops+jtube.env
|
||||
are available (container or any agent machine). Fast feedback on the backend contract.
|
||||
2. **Manual device test** — on an Android TV with a remote (D-pad!). Compose TV focus
|
||||
behavior can only be validated by hand; use the checklist below.
|
||||
|
||||
The manual checklist is a hard gate for releases; the script alone is not enough even
|
||||
when everything passes (playback, focus and UI states are untestable by script).
|
||||
|
||||
## Test environment / prerequisites
|
||||
|
||||
- Instance: any PeerTube (baseline: <https://tube.jeena.net>, v8.3.1, media on R2 object storage).
|
||||
- Account: `the test account` (sops: `~/.agents/secrets/jtube.env`). Credentials resolved
|
||||
at login; **do not store the password in the device during tests** — it's typed once
|
||||
on the setup screen (which is exactly feature 0).
|
||||
- Seeded content (tagged `jtube-tv-test`, channel `the test channel`):
|
||||
- `JTube TV Test 1 - Sintel Teaser` **Private** (~52 s)
|
||||
- `JTube TV Test 2 - Caminandes Llamigos` **Private** (~2:30)
|
||||
- `JTube TV Test 3 - Cosmos Laundromat` **Private** (~50 s)
|
||||
- the test account has watched entries in server history (public + test videos).
|
||||
- Subscription: `mediterranean_homestead@lone.earth` (remote channel — exercises
|
||||
federation in the subscription feed).
|
||||
- Public content: instance's own local videos (≥ 3000).
|
||||
- Device: Android TV with D-pad; also sanity-check a phone profile for the touch path.
|
||||
- Network conditions if available: Wi-Fi + (later) metered/lossy playback test.
|
||||
|
||||
## A. Scripted API checks (run before every release)
|
||||
|
||||
```bash
|
||||
./scripts/e2e-api-check.sh # exit 0 = backend contract intact
|
||||
```
|
||||
|
||||
Covers: login/refresh, local videos, my-videos incl. private rows, own channels,
|
||||
local channel search, search, subscription feed (incl. remote channel), history shape,
|
||||
watching POST → 204 → stored position, stream URL resolution, file-token endpoint,
|
||||
playlist reachability.
|
||||
|
||||
## B. Manual device checklist
|
||||
|
||||
Legend: [T] must pass for TV (D-pad). Pass criteria named per item; log failures with
|
||||
APK version + device.
|
||||
|
||||
### 0 — Credentials onboarding
|
||||
- [ ] T1 Fresh install: no URL/account stored → setup screen appears with URL prefilled to `https://tube.jeena.net`.
|
||||
- [ ] T2 Login with wrong password → app shows "Invalid credentials", user stays on setup, nothing persisted (clear app data → setup shows again). Repeat once to catch double-submit lockout via spinner + disabled button.
|
||||
- [ ] T3 Login with bad host (`x.invalid`) → reachable-error message, no crash.
|
||||
- [ ] T4 Successful login → home screen; force-stop app, relaunch → home screen directly (session persisted, encrypted). Tokens survive (no re-login).
|
||||
- [ ] T5 Long-run: 24 h later (token expiry ~86 400 s) relaunch → any authorized screen still works without re-login (refresh-token flow ran silently). If this can't be waited out, simulate by `adb shell` clearing only the access token? — skip; instead validate refresh via the API script (F0) and mark T5 after one real 24 h exposure per release cycle.
|
||||
|
||||
### 1 — Local videos / My videos
|
||||
- [ ] T6 "Videos" entry: list loads, newest public local videos first, thumbnails visible, D-pad scroll down past 25 rows loads page 2 (spinner at bottom disappears, list grows).
|
||||
- [ ] T7 "My videos": the 3 seeded **Private** test videos appear with a privacy badge; non-test public videos of the account also appear if any.
|
||||
- [ ] T8 Scroll from "Videos" to "My videos" and back → each screen keeps its own list (no cross-contamination of cached pages).
|
||||
- [ ] T9 Long video title truncates (2 lines max) without breaking the row layout; long channel names truncate.
|
||||
|
||||
### 2 — Channels
|
||||
- [ ] T10 "Channels" lists account's own channels (`the test channel`), avatar when present, video count.
|
||||
- [ ] T11 Click channel → that channel's videos list; open and play one; back returns to the channel's list (not the channels screen lost state).
|
||||
- [ ] T12 Remote channel via direct handle is out of MVP scope, but local-channel search (when implemented) must be re-tested: `searchTarget=local` ignores federated results.
|
||||
|
||||
### 3 — Search
|
||||
- [ ] T13 Search screen: field focusable via D-pad; on-screen/IME input works. Search "Sintel" → Test 1 private video appears even though it's private (search endpoint filters locally: verify it does — if private videos are NOT in search results, this is expected PeerTube behavior; adjust expectation: my-videos screen is the place private content is discoverable).
|
||||
- [ ] T14 Empty search string → no crash, stays on empty state. Gibberish → "Nothing here yet".
|
||||
|
||||
### 4 — Subscriptions
|
||||
- [ ] T15 Subscriptions feed shows the remote (`lone.earth`) channel's videos — mixed-feed behavior works; newest first.
|
||||
- [ ] T16 Feed pagination works beyond 25 items when subscribed to enough channels (skip if < 50 items total; note skip reason).
|
||||
- [ ] T17 Click remote video → plays or fails *gracefully* (a remote video's media is on the remote host; MVP may fail to resolve — must show error toast/message + back, never crash/freeze).
|
||||
|
||||
### 5 — History
|
||||
- [ ] T18 History lists server-side history newest-first, incl. both public and private watched entries.
|
||||
- [ ] T19 Play an entry to ≥ 20 s, leave (back) → entry is present/updated in history when re-opening the screen (server round-trip within ~1 s; app must not show stale cache as fresh — reload on screen entry).
|
||||
|
||||
### 6 — Playback
|
||||
- [ ] T20 Play a **public local** video → video+audio start < 5 s, D-pad pauses/plays, seek bar appears via center press, back closes player and lands on the list.
|
||||
- [ ] T21 Play a **private** test video → plays. (On tube.jeena.net media is publicly readable today; the app must still send the file token — verify in logs that `?videoFileToken=` was appended, since that is what private-bucket instances need.)
|
||||
- [ ] T22 Long watch: stay in a video > 60 s (watching reports every ~20 s) → server `userHistory.currentTime` updated; re-entering the video later resumes or at least proves the reports landed (curl check).
|
||||
- [ ] T23 Switch between two videos (from list, play, back, play another) → no audio ghosting/duplicate players (only one active ExoPlayer; release resources on exit).
|
||||
- [ ] T24 Airplane-mode / dead WiFi: opening player → error state shown, no crash; back works.
|
||||
- [ ] T25 Remote-hosted video (subscription item, media on lone.earth) → if it plays it must not break when network switches; at minimum no crash.
|
||||
- [ ] T26 Video without HLS (progressive-only instance): force by playing an item whose detail has no `streamingPlaylists` — if none exists on the test instance, note skip.
|
||||
|
||||
### 7 — Robustness / release hygiene
|
||||
- [ ] T27 App data cleared while logged out → setup screen, no crash loops.
|
||||
- [ ] T28 Rotate/screen-size changes are irrelevant on TV (landscape only), but app must not crash on phones when sideloaded.
|
||||
- [ ] T29 Kill server (or wrong URL after instance rename): any authorized screen → consistent error UI + retry; stale-token case must eventually surface "re-login" (authenticator gives up).
|
||||
- [ ] T30 APK is signed, minified build installs and boots (release build, `minifyEnabled` once enabled — re-run the whole checklist for it, R8 has bitten before).
|
||||
|
||||
## Release gate
|
||||
|
||||
All [T] items pass on the release build; script exits 0. Skips must be documented with
|
||||
reason. New features added in a release must append their checklist items to section B.
|
||||
Loading…
Add table
Add a link
Reference in a new issue