feat: Add update checker tooling

Add the initial dataset, version checker, tests, and project setup files so the update checker can be run and validated.
This commit is contained in:
Jeena 2026-03-12 12:49:28 +00:00
parent 1eddaca1ad
commit 95cd8e0906
10 changed files with 692 additions and 20 deletions

6
.env.sample Normal file
View file

@ -0,0 +1,6 @@
# Required for authenticated version checks
PAPERLESS_API_TOKEN=replace-with-api-token
RADICALE_BASIC_AUTH=base64-user-colon-pass
# Optional (use if upstream APIs need auth to avoid rate limits)
GITHUB_TOKEN=optional-github-token

4
.gitignore vendored Normal file
View file

@ -0,0 +1,4 @@
.venv/
__pycache__/
.pytest_cache/
*.pyc

26
AGENTS.md Normal file
View file

@ -0,0 +1,26 @@
# Agents
This project uses three cooperating agents to build the update checker.
## Research agent
Focus:
- Identify `current_version_url` for each webservice.
- Determine extraction rules for the running version (JSON path, regex, or text pattern).
- Identify `upstream_latest_version_url` and extraction rules for the latest release.
- Record notes for services without reliable version endpoints.
## Coding agent
Focus:
- Define the YAML schema and validation rules.
- Implement dataset loading and validation.
- Implement version fetching and extraction based on dataset rules.
- Implement version normalization, comparison, and output filtering.
## Testing agent
Focus:
- Add unit tests for extraction helpers and version comparison.
- Add fixtures for JSON/text/HTML response parsing.
- Validate behavior for unknown or missing versions.

View file

@ -1,40 +1,49 @@
# Backlog: check-for-updates
## US-01 - Parse Uptime Status Page
## US-01 - Curated Webservice Dataset
As a maintainer, I want the script to fetch and parse the services listed on `https://uptime.jeena.net/status/everything` so that I have a normalized list of hosted services to check.
As a maintainer, I want a curated YAML dataset of webservices to check so that the script does not rely on scraping a status page.
Acceptance criteria:
- Given the status page is reachable, the script extracts service name and URL for each monitored service.
- Parsed services are normalized for consistent URL handling (scheme and trailing slash).
- Failures to fetch or parse are reported with a clear error message and non-zero exit.
- A YAML file contains the full list of webservices (name + base URL).
- Each entry supports version endpoint and upstream metadata fields, even if left empty initially.
- The script can load the dataset without performing any network discovery.
## US-02 - Service Configuration Mapping
## US-01A - Sample Environment File
As a maintainer, I want to provide a config file that maps services to upstream sources and version detection strategies so that the script can work reliably across heterogeneous services.
As a maintainer, I want a `.env.sample` file listing required authentication variables so that I can configure the script quickly.
Acceptance criteria:
- The script accepts a `--config` CLI argument and reads a YAML/JSON/TOML file.
- Each config entry supports: service name, URL, upstream type, repo identifier, and version strategy.
- Invalid config formats are rejected with a clear error message.
- `.env.sample` exists and lists required auth environment variables.
- Each variable includes a short note about its expected format.
- Optional variables are clearly marked.
## US-03 - Version Detection Framework
## US-02 - Service Research Dataset
As a maintainer, I want a reusable framework of version detection strategies so that different services can report their running version reliably.
As a maintainer, I want to research and populate version endpoints and upstream release sources for each webservice so that checks are reliable without guesswork.
Acceptance criteria:
- The script supports a registry of detection strategies.
- At least one strategy uses common version endpoints (e.g., `/version`, `/health`, `/api/version`).
- The script records "unknown" when no version can be detected.
- Each webservice entry includes `current_version_url` and `upstream_latest_version_url`.
- Each entry includes extraction rules for both current and latest versions.
- Services without a viable version endpoint are clearly marked with notes and `current_version_url` left empty.
## US-04 - Service-Specific Version Strategies
## US-03 - Dataset Loader and Validation
As a maintainer, I want service-specific version detection for common tools (e.g., Gitea/Forgejo/Nextcloud/Miniflux) so that versions are detected accurately where possible.
As a maintainer, I want to load and validate the curated dataset so that the script can operate consistently on structured inputs.
Acceptance criteria:
- At least one named strategy queries a known service API endpoint.
- Strategies can be selected per service via config.
- If the service endpoint fails, the result is marked unknown without crashing the run.
- The script accepts a `--config` CLI argument and reads a YAML file.
- The dataset is validated for required fields and data types.
- Invalid datasets are rejected with a clear error message.
## US-04 - Version Fetching and Parsing
As a maintainer, I want to fetch versions using dataset-defined endpoints and extraction rules so that version detection is consistent across services.
Acceptance criteria:
- The script fetches `current_version_url` for each service.
- The script applies the dataset extraction rule to obtain the running version.
- Failures are logged per service without aborting the run.
## US-05 - Upstream Release Lookup
@ -89,3 +98,21 @@ Acceptance criteria:
- The script exits with code 0 when it completes a run.
- The script exits with code 1 on unrecoverable setup failures.
- Output is stable across runs for the same inputs.
## US-11 - Automated Tests
As a maintainer, I want tests for extraction and version comparison so that changes can be verified quickly.
Acceptance criteria:
- Tests cover JSONPath, regex, and header extraction.
- Tests cover version comparison including prerelease handling.
- Tests run with pytest.
## US-12 - Python Project Setup
As a maintainer, I want a virtual environment workflow with `pyproject.toml` so that dependencies are managed consistently.
Acceptance criteria:
- `pyproject.toml` defines runtime and dev dependencies.
- README documents venv setup and installation commands.
- `.venv` is ignored by git.

37
README.md Normal file
View file

@ -0,0 +1,37 @@
# check-for-updates
Small Python script to compare running service versions against upstream releases.
## Requirements
- Python 3.10+
## Setup
Create and activate a virtual environment, then install dependencies:
```bash
python -m venv .venv
source .venv/bin/activate
pip install -e .[dev]
```
Copy `.env.sample` to `.env` and fill required values. Export the variables before running the script:
```bash
export PAPERLESS_API_TOKEN=...
export RADICALE_BASIC_AUTH=...
```
## Usage
```bash
python3 check_updates.py --config services.yaml
python3 check_updates.py --config services.yaml --all
```
## Tests
```bash
python -m pytest
```

399
check_updates.py Normal file
View file

@ -0,0 +1,399 @@
#!/usr/bin/env python3
import argparse
import json
import os
import re
import sys
from dataclasses import dataclass
from typing import Any, Dict, Optional
from urllib.request import Request, urlopen
import yaml
@dataclass
class ExtractRule:
type: str
value: str
@dataclass
class ServiceConfig:
name: str
base_url: str
current_version_url: Optional[str]
current_version_extract: Optional[ExtractRule]
current_version_headers: Optional[Dict[str, str]]
upstream_latest_version_url: Optional[str]
upstream_latest_extract: Optional[ExtractRule]
upstream_latest_headers: Optional[Dict[str, str]]
notes: Optional[str]
def load_yaml(path: str) -> Dict[str, Any]:
with open(path, "r", encoding="utf-8") as handle:
return yaml.safe_load(handle) or {}
def parse_extract_rule(raw: Optional[Dict[str, Any]]) -> Optional[ExtractRule]:
if not raw:
return None
rule_type = raw.get("type")
value = raw.get("value")
if not rule_type or not value:
return None
allowed = {"jsonpath", "regex", "text", "header"}
if rule_type not in allowed:
raise ValueError(f"Unsupported extract rule type: {rule_type}")
return ExtractRule(type=rule_type, value=value)
def load_services(config: Dict[str, Any]) -> Dict[str, ServiceConfig]:
services = config.get("services")
if not isinstance(services, list):
raise ValueError("Config must include a 'services' list")
loaded: Dict[str, ServiceConfig] = {}
for entry in services:
if not isinstance(entry, dict):
raise ValueError("Each service entry must be a mapping")
name = entry.get("name")
base_url = entry.get("base_url")
if not name or not base_url:
raise ValueError("Each service must include name and base_url")
if not isinstance(name, str) or not isinstance(base_url, str):
raise ValueError("Service name and base_url must be strings")
current_headers = entry.get("current_version_headers")
upstream_headers = entry.get("upstream_latest_headers")
if current_headers is not None and not isinstance(current_headers, dict):
raise ValueError("current_version_headers must be a mapping")
if upstream_headers is not None and not isinstance(upstream_headers, dict):
raise ValueError("upstream_latest_headers must be a mapping")
current_url = entry.get("current_version_url")
upstream_url = entry.get("upstream_latest_version_url")
current_extract = parse_extract_rule(entry.get("current_version_extract"))
upstream_extract = parse_extract_rule(entry.get("upstream_latest_extract"))
if current_url and not current_extract:
raise ValueError(f"Service {name} must define current_version_extract")
if upstream_url and not upstream_extract:
raise ValueError(f"Service {name} must define upstream_latest_extract")
loaded[name] = ServiceConfig(
name=name,
base_url=base_url,
current_version_url=current_url,
current_version_extract=current_extract,
current_version_headers=current_headers,
upstream_latest_version_url=upstream_url,
upstream_latest_extract=upstream_extract,
upstream_latest_headers=upstream_headers,
notes=entry.get("notes"),
)
return loaded
def resolve_env_placeholders(value: str) -> str:
pattern = re.compile(r"\{env:([A-Z0-9_]+)\}")
def replace(match: re.Match[str]) -> str:
env_name = match.group(1)
env_value = os.getenv(env_name)
if env_value is None:
raise ValueError(f"Missing environment variable {env_name}")
return env_value
return pattern.sub(replace, value)
def build_headers(raw_headers: Optional[Dict[str, str]]) -> Dict[str, str]:
if not raw_headers:
return {}
resolved = {}
for key, value in raw_headers.items():
resolved[key] = resolve_env_placeholders(str(value))
return resolved
def fetch_response(
url: str,
timeout: float,
user_agent: str,
extra_headers: Optional[Dict[str, str]] = None,
) -> tuple[str, Dict[str, str]]:
headers = {"User-Agent": user_agent}
if extra_headers:
headers.update(extra_headers)
request = Request(url, headers=headers)
with urlopen(request, timeout=timeout) as response:
body = response.read().decode("utf-8", errors="replace")
response_headers = {k.lower(): v for k, v in response.headers.items()}
return body, response_headers
def extract_jsonpath(payload: Any, path: str) -> Optional[str]:
if not path.startswith("$."):
return None
current = payload
for part in path[2:].split("."):
if isinstance(current, list):
if not part.isdigit():
return None
index = int(part)
if index >= len(current):
return None
current = current[index]
continue
if not isinstance(current, dict):
return None
if part not in current:
return None
current = current[part]
if current is None:
return None
return str(current)
def extract_version(
body: str,
rule: ExtractRule,
headers: Optional[Dict[str, str]] = None,
) -> Optional[str]:
if rule.type == "jsonpath":
try:
payload = json.loads(body)
except json.JSONDecodeError:
return None
return extract_jsonpath(payload, rule.value)
if rule.type == "header":
if not headers:
return None
return headers.get(rule.value.lower())
if rule.type == "regex":
import re
match = re.search(rule.value, body)
if not match:
return None
if match.lastindex:
return match.group(1)
return match.group(0)
if rule.type == "text":
return rule.value if rule.value in body else None
return None
def normalize_version(raw: Optional[str]) -> Optional[str]:
if not raw:
return None
return raw.strip().lstrip("v")
def parse_version(value: str) -> Optional[Dict[str, Any]]:
match = re.match(
r"^(\d+)\.(\d+)\.(\d+)(?:-([0-9A-Za-z.-]+))?(?:\+([0-9A-Za-z.-]+))?$",
value,
)
if not match:
return None
major, minor, patch, prerelease, _build = match.groups()
prerelease_parts = None
if prerelease:
prerelease_parts = []
for part in prerelease.split("."):
if part.isdigit():
prerelease_parts.append((True, int(part)))
else:
prerelease_parts.append((False, part))
return {
"major": int(major),
"minor": int(minor),
"patch": int(patch),
"prerelease": prerelease_parts,
}
def compare_prerelease(a: Optional[list], b: Optional[list]) -> int:
if a is None and b is None:
return 0
if a is None:
return -1
if b is None:
return 1
for left, right in zip(a, b):
left_is_num, left_value = left
right_is_num, right_value = right
if left_is_num and right_is_num:
if left_value != right_value:
return 1 if left_value > right_value else -1
elif left_is_num != right_is_num:
return -1 if left_is_num else 1
else:
if left_value != right_value:
return 1 if left_value > right_value else -1
if len(a) == len(b):
return 0
return 1 if len(a) > len(b) else -1
def compare_versions(current: Optional[str], latest: Optional[str]) -> Optional[int]:
if not current or not latest:
return None
current_norm = normalize_version(current)
latest_norm = normalize_version(latest)
if not current_norm or not latest_norm:
return None
current_version = parse_version(current_norm)
latest_version = parse_version(latest_norm)
if current_version is None or latest_version is None:
return None
current_tuple = (current_version["major"], current_version["minor"], current_version["patch"])
latest_tuple = (latest_version["major"], latest_version["minor"], latest_version["patch"])
if latest_tuple > current_tuple:
return 1
if latest_tuple < current_tuple:
return -1
return compare_prerelease(current_version["prerelease"], latest_version["prerelease"])
def build_upstream_fallback(url: Optional[str]) -> Optional[Dict[str, Any]]:
if not url:
return None
if "api.github.com/repos/" in url and url.endswith("/releases/latest"):
tag_url = url.replace("/releases/latest", "/tags")
return {
"url": tag_url,
"extract": ExtractRule(type="jsonpath", value="$.0.name"),
}
if "codeberg.org/api/v1/repos/" in url and url.endswith("/releases/latest"):
tag_url = url.replace("/releases/latest", "/tags")
return {
"url": tag_url,
"extract": ExtractRule(type="jsonpath", value="$.0.name"),
}
return None
def check_service(service: ServiceConfig, timeout: float, user_agent: str) -> Dict[str, Any]:
result: Dict[str, Any] = {
"name": service.name,
"current": None,
"latest": None,
"current_error": None,
"latest_error": None,
"upstream_url": service.upstream_latest_version_url,
}
if service.current_version_url and service.current_version_extract:
try:
headers = build_headers(service.current_version_headers)
body, response_headers = fetch_response(
service.current_version_url,
timeout,
user_agent,
headers,
)
result["current"] = extract_version(
body,
service.current_version_extract,
response_headers,
)
except Exception as exc:
result["current_error"] = str(exc)
if service.upstream_latest_version_url and service.upstream_latest_extract:
headers = build_headers(service.upstream_latest_headers)
try:
body, response_headers = fetch_response(
service.upstream_latest_version_url,
timeout,
user_agent,
headers,
)
result["latest"] = extract_version(
body,
service.upstream_latest_extract,
response_headers,
)
if result["latest"] is None:
raise ValueError("Latest version extraction returned empty")
except Exception as exc:
fallback = build_upstream_fallback(service.upstream_latest_version_url)
if fallback:
try:
body, response_headers = fetch_response(
fallback["url"],
timeout,
user_agent,
headers,
)
result["latest"] = extract_version(
body,
fallback["extract"],
response_headers,
)
except Exception as fallback_exc:
result["latest_error"] = str(fallback_exc)
else:
result["latest_error"] = str(exc)
return result
def main() -> int:
parser = argparse.ArgumentParser(description="Check for webservice updates")
parser.add_argument("--config", default="services.yaml", help="Path to services YAML")
parser.add_argument("--all", action="store_true", help="Show all services")
parser.add_argument("--timeout", type=float, default=10.0, help="HTTP timeout in seconds")
parser.add_argument("--user-agent", default="check-for-updates/1.0", help="HTTP user agent")
args = parser.parse_args()
try:
config = load_yaml(args.config)
services = load_services(config)
except Exception as exc:
print(f"Failed to load config: {exc}", file=sys.stderr)
return 1
results = []
for service in services.values():
results.append(check_service(service, args.timeout, args.user_agent))
output_lines = []
for result in sorted(results, key=lambda item: item["name"].lower()):
comparison = compare_versions(result["current"], result["latest"])
has_update = comparison == 1
if not args.all and not has_update:
continue
current = result["current"] or "unknown"
latest = result["latest"] or "unknown"
upstream = result["upstream_url"] or "unknown"
notes = []
if comparison is None and result["current"] and result["latest"]:
notes.append("unparseable")
if result["current_error"]:
notes.append("current error")
if result["latest_error"]:
notes.append("latest error")
suffix = f" [{' '.join(notes)}]" if notes else ""
output_lines.append(f"{result['name']}: {current} -> {latest} ({upstream}){suffix}")
if output_lines:
print("\n".join(output_lines))
for result in sorted(results, key=lambda item: item["name"].lower()):
if result["current_error"]:
print(
f"{result['name']}: current version error: {result['current_error']}",
file=sys.stderr,
)
if result["latest_error"]:
print(
f"{result['name']}: latest version error: {result['latest_error']}",
file=sys.stderr,
)
return 0
if __name__ == "__main__":
raise SystemExit(main())

17
pyproject.toml Normal file
View file

@ -0,0 +1,17 @@
[build-system]
requires = ["setuptools>=68", "wheel"]
build-backend = "setuptools.build_meta"
[project]
name = "check-for-updates"
version = "0.1.0"
description = "Check running service versions against upstream releases"
requires-python = ">=3.10"
dependencies = [
"PyYAML>=6.0",
]
[project.optional-dependencies]
dev = [
"pytest>=8.0",
]

105
services.yaml Normal file
View file

@ -0,0 +1,105 @@
services:
- name: Firefox Sync
base_url: https://fxsync.jeena.net/
current_version_url: https://fxsync.jeena.net/__heartbeat__
current_version_extract:
type: jsonpath
value: $.version
upstream_latest_version_url: https://api.github.com/repos/mozilla-services/syncstorage-rs/releases/latest
upstream_latest_extract:
type: jsonpath
value: $.tag_name
- name: Grafana
base_url: https://monitoring.bundang.swierczyniec.info/
current_version_url: https://monitoring.bundang.swierczyniec.info/api/health
current_version_extract:
type: jsonpath
value: $.version
upstream_latest_version_url: https://api.github.com/repos/grafana/grafana/releases/latest
upstream_latest_extract:
type: jsonpath
value: $.tag_name
- name: Immich
base_url: https://photos.bundang.swierczyniec.info/
current_version_url:
current_version_extract:
upstream_latest_version_url: https://api.github.com/repos/immich-app/immich/releases/latest
upstream_latest_extract:
type: jsonpath
value: $.tag_name
notes: Instance version endpoint returned 404 for /api/server-info/version.
- name: PeerTube
base_url: https://tube.jeena.net/
current_version_url: https://tube.jeena.net/api/v1/config
current_version_extract:
type: jsonpath
value: $.serverVersion
upstream_latest_version_url: https://api.github.com/repos/peertube/peertube/releases/latest
upstream_latest_extract:
type: jsonpath
value: $.tag_name
- name: Mastodon
base_url: https://toot.jeena.net/
current_version_url: https://toot.jeena.net/api/v1/instance
current_version_extract:
type: jsonpath
value: $.version
upstream_latest_version_url: https://api.github.com/repos/mastodon/mastodon/releases/latest
upstream_latest_extract:
type: jsonpath
value: $.tag_name
- name: Open WebUI AI
base_url: https://ai.bundang.swierczyniec.info/
current_version_url: https://ai.bundang.swierczyniec.info/api/version
current_version_extract:
type: jsonpath
value: $.version
upstream_latest_version_url: https://api.github.com/repos/open-webui/open-webui/releases/latest
upstream_latest_extract:
type: jsonpath
value: $.tag_name
- name: Paperless
base_url: https://paperless.jeena.net/
current_version_url: https://paperless.jeena.net/api/documents/
current_version_extract:
type: header
value: x-version
current_version_headers:
Authorization: Token {env:PAPERLESS_API_TOKEN}
upstream_latest_version_url: https://api.github.com/repos/paperless-ngx/paperless-ngx/releases/latest
upstream_latest_extract:
type: jsonpath
value: $.tag_name
notes: Uses x-version response header from authenticated API request.
- name: PieFed
base_url: https://piefed.jeena.net/
current_version_url: https://piefed.jeena.net/api/v3/site
current_version_extract:
type: jsonpath
value: $.version
upstream_latest_version_url: https://codeberg.org/api/v1/repos/rimu/pyfedi/releases/latest
upstream_latest_extract:
type: jsonpath
value: $.tag_name
- name: Radicale WebDav
base_url: https://dav.jeena.net/.web/
current_version_url: https://dav.jeena.net/version
current_version_extract:
type: jsonpath
value: $.version
current_version_headers:
Authorization: Basic {env:RADICALE_BASIC_AUTH}
upstream_latest_version_url: https://api.github.com/repos/Kozea/Radicale/releases/latest
upstream_latest_extract:
type: jsonpath
value: $.tag_name
notes: /version returns 401 without auth; uses Basic auth.
- name: FreshRSS
base_url: https://rss.jeena.net/
current_version_url:
current_version_extract:
upstream_latest_version_url: https://api.github.com/repos/FreshRSS/FreshRSS/releases/latest
upstream_latest_extract:
type: jsonpath
value: $.tag_name
notes: No unauthenticated version endpoint found.

24
tests/test_extraction.py Normal file
View file

@ -0,0 +1,24 @@
import json
import sys
from pathlib import Path
sys.path.append(str(Path(__file__).resolve().parents[1]))
from check_updates import ExtractRule, extract_version
def test_extract_jsonpath():
body = json.dumps({"version": "1.2.3"})
rule = ExtractRule(type="jsonpath", value="$.version")
assert extract_version(body, rule) == "1.2.3"
def test_extract_header():
rule = ExtractRule(type="header", value="x-version")
headers = {"x-version": "2.3.4"}
assert extract_version("", rule, headers) == "2.3.4"
def test_extract_regex_group():
rule = ExtractRule(type="regex", value=r"Version: (\d+\.\d+\.\d+)")
assert extract_version("Version: 1.9.0", rule) == "1.9.0"

27
tests/test_versions.py Normal file
View file

@ -0,0 +1,27 @@
import sys
from pathlib import Path
sys.path.append(str(Path(__file__).resolve().parents[1]))
from check_updates import compare_versions
def test_compare_versions_newer():
assert compare_versions("1.2.3", "1.2.4") == 1
def test_compare_versions_equal():
assert compare_versions("2.0.0", "2.0.0") == 0
def test_compare_versions_older():
assert compare_versions("2.1.0", "2.0.9") == -1
def test_compare_versions_unparseable():
assert compare_versions("1.2", "1.2.3") is None
def test_compare_versions_prerelease():
assert compare_versions("1.2.3-alpha.1", "1.2.3") == 1
assert compare_versions("1.2.3", "1.2.3-alpha.1") == -1