From 9b792042e6dcb1e66e67344a0dd20b16f74cb085 Mon Sep 17 00:00:00 2001 From: thepra Date: Sun, 18 Oct 2026 09:18:26 +0200 Subject: [PATCH] R9 Web Push, then the honest version: VAPID key, push subscriptions, encrypted pushes, 4.5.0 advertised The server's VAPID key (PushKeys, made on first need, in Mongo) is the instance API's vapid.public_key and every app's vapid_key. /api/v1/push/subscription (POST, GET, PUT, DELETE) keeps one subscription per access token, so per persona; every notification kept (never a filtered one) is queued as a SendPush job for the subscriptions whose alert and policy admit it, and sent through the `push` client (the safe handler, https only) encrypted as RFC 8291 aes128gcm or the aesgcm draft with an ES256 VAPID JWT. 404/410 delete the subscription, 429/5xx/timeouts retry, other answers drop the push. Subscriptions end with their tokens (revocation, sessions ended, suspension, deletion, a restore); backups hold neither keys nor subscriptions. The walk of every Mastodon 4.3-4.5 client route (docs/MASTODON-API.md) added conversations/:id/unread, domain_blocks/preview, v2 notifications clear and PUT policy, /oauth/userinfo and stubs, so both instance APIs now say 4.5.0 (compatible; PrivaPub ). The pasture gets a push sink (pushsink.test) and scenarios/push.sh in batch A. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01WpVCoDeM1qUTRp4fXQdfof --- .gitignore | 1 + CLAUDE.md | 145 +++++- FEDERATION.md | 8 + PrivaPub/Api/Mastodon/Auth/OAuthSetup.cs | 17 + PrivaPub/Api/Mastodon/Auth/PersonaExchange.cs | 5 +- PrivaPub/Api/Mastodon/Auth/TokenController.cs | 2 +- .../Controllers/AccountsController.cs | 25 +- .../Mastodon/Controllers/AppsController.cs | 9 +- .../Controllers/InstanceController.cs | 21 +- .../NotificationRequestsController.cs | 2 +- .../Mastodon/Controllers/PushController.cs | 169 +++++++ .../Mastodon/Controllers/SearchController.cs | 35 +- .../Controllers/TimelinesController.cs | 14 +- .../Controllers/UserinfoController.cs | 40 ++ .../Infrastructure/MastodonController.cs | 3 +- .../Domain/Moderation/AccountModeration.cs | 1 + PrivaPub/Domain/Social/Notifications.cs | 21 + PrivaPub/Domain/Social/WebPush.cs | 156 ++++++ .../Domain/Statuses/ConversationStates.cs | 32 +- .../Infrastructure/Backup/ProtectiveMerge.cs | 3 + .../Infrastructure/Backup/ServerBackup.cs | 4 +- PrivaPub/Infrastructure/Cli/AdminCommands.cs | 1 + PrivaPub/Infrastructure/Data/Indexes.cs | 3 + .../Http/SafeHttpHandlerFactory.cs | 4 +- PrivaPub/Infrastructure/Push/Vapid.cs | 116 +++++ PrivaPub/Infrastructure/Push/WebPushCrypto.cs | 169 +++++++ PrivaPub/Infrastructure/Push/WebPushJob.cs | 264 ++++++++++ .../Middleware/SocialPubConfigurations.cs | 14 + PrivaPub/Models/Jobs/Job.cs | 3 +- PrivaPub/Models/PushKeys.cs | 18 + PrivaPub/Models/Social/ConversationState.cs | 1 + PrivaPub/Models/Social/PushSubscription.cs | 34 ++ PrivaPub/Services/PersonaRemoval.cs | 1 + PrivaPub/Services/RootSessions.cs | 7 +- docs/MASTODON-API.md | 70 +++ docs/ROADMAP.md | 24 +- tools/pasture/Caddyfile | 5 + tools/pasture/lib/interop.sh | 8 +- tools/pasture/peers/pushsink.sh | 22 + tools/pasture/pushsink/pushsink.cs | 465 ++++++++++++++++++ tools/pasture/scenarios/push.sh | 168 +++++++ tools/pasture/sweep.sh | 9 +- tools/smoke/mastodon-api.sh | 9 +- 43 files changed, 2079 insertions(+), 49 deletions(-) create mode 100644 PrivaPub/Api/Mastodon/Controllers/PushController.cs create mode 100644 PrivaPub/Api/Mastodon/Controllers/UserinfoController.cs create mode 100644 PrivaPub/Domain/Social/WebPush.cs create mode 100644 PrivaPub/Infrastructure/Push/Vapid.cs create mode 100644 PrivaPub/Infrastructure/Push/WebPushCrypto.cs create mode 100644 PrivaPub/Infrastructure/Push/WebPushJob.cs create mode 100644 PrivaPub/Models/PushKeys.cs create mode 100644 PrivaPub/Models/Social/PushSubscription.cs create mode 100644 docs/MASTODON-API.md create mode 100644 tools/pasture/peers/pushsink.sh create mode 100644 tools/pasture/pushsink/pushsink.cs create mode 100644 tools/pasture/scenarios/push.sh diff --git a/.gitignore b/.gitignore index 4479eae..9024341 100644 --- a/.gitignore +++ b/.gitignore @@ -407,6 +407,7 @@ PrivaPub/media-store-incoming/ PrivaPub/media-store-backups/ tools/pasture/.publish/ tools/pasture/.flood/ +tools/pasture/.pushsink/ .claude/worktrees/ tools/pasture/.ca/ tools/pasture/.state/ diff --git a/CLAUDE.md b/CLAUDE.md index 4937e80..1c7e09f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -94,6 +94,8 @@ PrivaPub/ ASP.NET Core Web API, net10.0 Infrastructure/ Http/ FederationHttp + SafeHttpHandlerFactory + IpRangeGuard: the only way out Jobs/ JobQueue (leases), JobWorker, Backoff, HostCircuitBreaker + Push/ Web Push (R9): Vapid (the server's key, ES256 JWTs), WebPushCrypto (aes128gcm, aesgcm), + WebPushJob (SendPush, through the `push` client) Ids/ PrivacyIds (day-only ids for personas and groups, arrival-ordered ids for remote posts) Data/ Indexes (created at start), EntityMaps.Warm, Migrations/_NNN_*.cs Cli/ AdminCommands (`PrivaPub admin promote|demote|role|create-root|move-persona|delete-root|reset-totp`, @@ -130,6 +132,7 @@ PrivaPub/ ASP.NET Core Web API, net10.0 Moderation/ ModerationLog (the staff's log, AdminAction), AccountModeration (silence, suspend, posts removed), Restrictions (who is held back, in memory), ModerationFlags (posts' flags again) Social/ FollowService (local in-process, remote Follow/Accept), Notifications, Announcements, + WebPush (who is pushed what; WebPushDispatcher on Notifications.Told), FeaturedTags (a persona's featured hashtags, sent and received), Suggestions (who to follow) Timelines/ Fanout (TimelineEntry rows, Mastodon's home rules), TimelineService Privacy/ VisibilityPolicy (IsPublic expression, CanSee) @@ -146,8 +149,8 @@ PrivaPub/ ASP.NET Core Web API, net10.0 Entities/ Mappers/ Mastodon entities; MastodonMapper (Account, Status), AccountSearch Controllers/ apps, instance, accounts, statuses, timelines, notifications, search, lists, filters, media, polls, reactions, reports, scheduled statuses, streaming, tags, trends, cdns, - provenance, custom emojis, featured tags, suggestions; the stubs are `StubsController` - in `SearchController.cs` + provenance, custom emojis, featured tags, suggestions, Web Push (`PushController`), OAuth + userinfo; the stubs are `StubsController` in `SearchController.cs` Web/Pages/ Razor: /@{user}, /@{user}/{id} (public posts only, strict CSP, noindex); OAuth/: /oauth/login (root password), /oauth/authorize (choose persona, consent) Services/ RootUsersService, GroupUsersService, PostsService, AppConfigurationService, RootRemoval, @@ -429,6 +432,87 @@ client's role route never does. and `ProtectiveMerge` deletes those of a persona deleted since. Mastodon's account archive has no featured tags, notes or endorsements, so the persona archive leaves them out. +## Web Push (completion program R9) + +- **The key** (`Models/PushKeys.cs`, `Infrastructure/Push/Vapid.cs`): one P-256 VAPID key pair (RFC 8292), made the first + time something needs it and kept in Mongo (one document, id `vapid`; two processes at once keep the first one saved). + Its public key, the uncompressed point in URL-safe base64 with its padding (as Mastodon writes `VAPID_PUBLIC_KEY`), is + v2's `configuration.vapid.public_key`, an app's `vapid_key` (`POST /api/v1/apps`, `apps/verify_credentials`) and every + subscription's `server_key`. +- **Subscriptions** (`Models/Social/PushSubscription.cs`, a new collection, unique on `TokenId`): one per Mastodon access + token, as on Mastodon, so one persona's (the token's subject) and never its login's. Each holds the endpoint, the + client's `p256dh` and `auth` as it sent them, `standard`, the alerts turned on, the policy, and the token itself, which + the payload carries back. The alerts are Mastodon 4.5's names (`WebPush.Alerts`: `mention`, `quote`, `status`, + `reblog`, `follow`, `follow_request`, `favourite`, `poll`, `update`, `quoted_update`, `severed_relationships`, + `moderation_warning`, `annual_report`, `admin.sign_up`, `admin.report`) and Pleroma's `pleroma:emoji_reaction`; those + PrivaPub never notifies (the admin ones, `quoted_update`, `severed_relationships`, `annual_report`) are kept and never fire. +- **Routes** (`Api/Mastodon/Controllers/PushController.cs`, the `push` scope, a persona's token; Mastodon's parameters as + a form or JSON): + - `POST /api/v1/push/subscription`: `subscription[endpoint]` (an https URL; plain http only where + `Federation:AllowPlainHttp`, which production refuses), `subscription[keys][p256dh]` (a point on P-256), + `subscription[keys][auth]` (16 bytes), `subscription[standard]`, `data[alerts][]`, `data[policy]` (`all` by + default, `followed`, `follower`, `none`). It replaces the token's subscription and answers it; 400 without an + endpoint, 422 for an endpoint that is no https URL, keys that are no Web Push keys, or another policy. + - `GET`: the token's subscription, or 404 `Record not found`. + - `PUT`: `data[alerts]` and `data[policy]` replace what it had, as Mastodon's update does (an alert left out is off, a + policy left out is `all`); 404 without one. + - `DELETE`: `{}`, whether there was one or not. + - The answer is Mastodon's `WebPushSubscription`: `id` (a string), `endpoint`, `standard`, `alerts` (every alert, true + or false), `server_key`, `policy`. +- **What is pushed** (`Domain/Social/WebPush.cs`): `Notifications.Add` and `Notifications.Warn` tell `Notifications.Told` + of each notification they keep, never one the notification policy filtered or dropped nor one blocks or mutes hid. + `WebPushDispatcher` (a hosted service: the command line pushes nothing) hangs there and queues one `SendPush` job + (`JobKind.SendPush`, the last member; dedupe `push||`, its host the endpoint's) for each + subscription of that persona whose alert for the type is on and whose policy admits the sender: `followed`, an account + the persona follows; `follower`, one that follows it; the persona itself (its poll ended, the staff's warning) under + every policy but `none`. Nothing for a persona `Restrictions` holds back (suspended, its login banned). +- **Sending** (`Infrastructure/Push/WebPushJob.cs`). Everything is read again when the job runs: the subscription, its + persona (not held back, not deleted), its token (valid, unexpired, its subject the subscription's persona, or the + subscription goes), the notification (still the persona's, still pushable, still shown to it by + `MastodonMapper.Notifications`). The payload is Mastodon's JSON, at most 3993 bytes (`WebPushCrypto.MaxPayload`): + - `access_token`, the subscription's own token, never another's; `preferred_locale`, the persona's default posting + language, else the server's first; `notification_id`, PrivaPub's id (a string); `notification_type`, its Mastodon + name; + - `icon`, the sender's avatar through the media proxy; `title`, Mastodon's English subjects ("You were mentioned by + …", "… favorited your post", …); `body`, the post's warning, else its words, else the sender's bio, without tags, 140 + characters at most. + - **Encryption** (`WebPushCrypto`, System.Security.Cryptography only: ECDiffieHellman on P-256, HKDF-SHA-256, AesGcm), + a fresh key pair and salt per message. A `standard` subscription gets RFC 8291's `aes128gcm` (RFC 8188's header: + salt, record size 4096, our public key as the key id; one record, delimiter 0x02) with `Authorization: vapid + t=, k=`. Any other gets the draft Mastodon still sends by default, `aesgcm` (two bytes of padding length + first) with `Encryption: salt=…`, `Crypto-Key: dh=;p256ecdsa=` and `Authorization: WebPush + `. Both carry `Content-Type: application/octet-stream`, `TTL: 172800` and `Urgency: normal`, as Mastodon's do. + - **The JWT** (`VapidKey.Token`): ES256, `aud` the endpoint's origin, `exp` twelve hours ahead (RFC 8292 allows a + day), `sub` `mailto:` the contact email the administrator gave, else the server's address (`mailto:privapub@` + for a test server on plain http). Never a persona. + - **Answers:** 2xx is done; 404 and 410 delete the subscription; 408, 429, a 5xx, a timeout or a broken connection retry + on the jobs' backoff (six attempts, about forty minutes); any other answer, a redirect, or an endpoint that does not + resolve to a public address drops the push, logged by the push service's host alone. Four run at a time, four per host. +- **decePub:** the first-party client's exchanged persona tokens may hold `push` too (`PersonaExchange.FirstPartyScopes`), + so it can subscribe like any other client. +- **Lifecycle:** a subscription goes with its token: `/oauth/revoke` (an OpenIddict handler after the revocation), + `RootSessions.Revoke` (a password changed or recovered, two-factor turned on, signing out everywhere, a ban, a deleted + login), `PersonaRemoval`, a suspension (`AccountModeration`), `admin move-persona`, and any token found unusable when a + push is sent. A new subscription made with the same token replaces it. +- **Backups:** `PushSubscription` and `PushKeys` are never in a backup (`ServerBackup.Excluded`). A restore leaves the + live key as it is, so the key the push services know never changes, and deletes every subscription with the tokens + (`ProtectiveMerge`); clients subscribe again once signed in. +- **The rest of the honest version** (`docs/MASTODON-API.md`, the walk of every Mastodon 4.3, 4.4 and 4.5 client route), + built with it: + - `POST /api/v1/conversations/:id/unread` (`ConversationState.MarkedUnread`, until it is read); + - `GET /api/v1/domain_blocks/preview` (`following_count` and `followers_count` the block would take, counted as + `BlockDomain` takes them); + - `POST /api/v2/notifications/clear` and `PUT /api/v2/notifications/policy`; + - `GET`/`POST /oauth/userinfo` (the `profile` scope: `iss`, `sub` the actor, `name`, `preferred_username`, `profile`, + `picture`; `userinfo_endpoint` in the server's metadata) and `verify_credentials` for a `profile` token + (`Scope("read:accounts|profile")`: a scope written `a|b` is either); + - stubs: `GET /api/v1/annual_reports/:year` (none) and `POST …/:year/read` (`{}`), `POST /api/v1/accounts` (403, as + Mastodon with registrations closed: they are cut), `POST /api/v1/emails/confirmations` (403) and + `GET /api/v1/emails/check_confirmation` (`true`), `GET /api/v1/peers/search` (`[]`), and 4.4's + `GET /api/v1_alpha/async_refreshes/:id` (404: PrivaPub never sends `Mastodon-Async-Refresh`). + + Both instance APIs then say `4.5.0 (compatible; PrivaPub )` (Mastodon invariant 6). + ## Custom emoji (completion program R7) - **The server's own emoji** (`Models/Post/LocalEmoji.cs`, a collection of its own): a shortcode (`[a-zA-Z0-9_]{2,64}`, @@ -487,8 +571,12 @@ client's role route never does. 1. **Every outbound request goes through `IFederationHttp`.** Its handler resolves the name itself and connects only to public addresses; redirects are followed by hand (three at most, each re-checked); bodies are capped at 1 MB; only JSON media types are read; a refused URL is not asked again for five minutes. Never create another - `HttpClient` for federation. The only other outbound traffic is SMTP and `GeoUpdater`'s monthly DB-IP Lite download - (its own `geo` client, a fixed HTTPS host, size-capped, the file checked before it is swapped in). + `HttpClient` for federation. The only other outbound traffic is SMTP, `GeoUpdater`'s monthly DB-IP Lite download + (its own `geo` client, a fixed HTTPS host, size-capped, the file checked before it is swapped in), and Web Push (R9): + the named `push` client, on `SafeHttpHandlerFactory`'s handler (it resolves the name itself and connects only to + public addresses, with federation's test-network exceptions; no redirects, cookies or proxy), https only (plain http + only where `Federation:AllowPlainHttp`), 5 s to connect and 15 s a request. It POSTs encrypted payloads to the + endpoints clients subscribed with, and reads nothing back but the status. 2. **Every fetch is signed by the instance actor** (`privapub`), never by a persona; deliveries are signed by the acting avatar or group. Both are draft-cavage rsa-sha256 over `(request-target) host date` (+ `digest` on bodies). Inbound, an RFC 9421 signature (`Signature-Input`) is verified too, its target against our public address. A persona's own @@ -605,9 +693,14 @@ client's role route never does. 5. Unsupported features answer empty lists or 422 with a message, never 404 or 500, so clients degrade. Any `/api` error that leaves without a body (a challenge, a 404 from routing, a 429) gets Mastodon's `{"error": ...}` from `UseMastodonErrorBodies`. -6. Advertise `4.2.0 (compatible; PrivaPub)` until streaming and Web Push exist too; `api_versions.mastodon = 7` stays, - because clients show quotes from it, so everything a 4.3+ client calls because of it must answer: grouped - notifications (`/api/v2/notifications`, its unread count, groups, accounts and dismiss) and the notification policy. +6. **Advertise `4.5.0 (compatible; PrivaPub )`** in both instance APIs (`InstanceController.Version`, since R9; + the release is the build's tag without its `v`, else its ref or commit), with `api_versions.mastodon = 7`, 4.5's own. + The number is the walk's (`docs/MASTODON-API.md`): every client route of Mastodon 4.3, 4.4 and 4.5 is served or + answers as a named stub, and none a client relies on is missing. A route Mastodon adds, or one PrivaPub stops serving, + goes there first, and the version follows the walk, never a wish; `tools/smoke/mastodon-api.sh` checks it on every + deploy. Clients branch on both numbers, so everything a 4.3+ client calls must answer: grouped notifications + (`/api/v2/notifications`, its unread count, groups, accounts, dismiss and clear), quotes, Web Push, streaming and the + notification policy. The policy is real (`Domain/Social/NotificationPolicies.cs`, applied in `Notifications.Add`): accepted by default, it filters (into a `NotificationRequest` per account, out of every list and count unless `include_filtered`) or drops what strangers, accounts that do not follow the persona (or only for three days), accounts newer than 30 days, @@ -624,6 +717,8 @@ client's role route never does. private notes (shown to it alone, never federated), dismissed suggestions. Suggestions never offer the persona itself or a sibling persona of its login, and `familiar_followers` answers no one for every account, since it would tell who follows whom here. +9. **A Web Push subscription is its token's** (R9): one per access token, read, changed and deleted only with that token, + pushed only its persona's notifications, and its payload carries only that token. It ends when the token does. ## Media invariants @@ -766,6 +861,12 @@ client's role route never does. decided in `docs/ROADMAP.md` ("Owner decisions on what PrivaPub reveals"). Anything new that tells another server something about an avatar gets the same treatment: ask, then record it there. - **Location-ranged posts never federate.** +- **Web Push tells a push service when, never what** (R9). Every payload is encrypted to the subscribing client's own + keys (RFC 8291, or the aesgcm draft), so the push service (Google's, Mozilla's, Apple's, a UnifiedPush distributor) + learns only that this server has something for that endpoint, when, and how big: never the words, the sender or the + persona. The VAPID JWT names the push service's origin and the server's contact (the contact email, or the server's + address), never a persona. Subscriptions are per token, so per persona: none receives a sibling's notifications, and + nothing stored pairs two personas of one login. - **Statistics name servers, never people** (owner decisions on statistics, 2026-10-03). Every interaction is recorded through `IInteractionLedger` (`Infrastructure/Statistics`) as an `InteractionEvent` (90 days), but an event never holds a root, persona or group id, an activity id, an inbox URL, a remote actor URI or a sender IP: @@ -868,7 +969,8 @@ client's role route never does. **Never in a backup** (`ServerBackup.Excluded`): `InteractionSalt` (owner rule), `Job` and `Delivery` (work in flight), `EmailRecovery` and `EmailVerification`, `openiddict.tokens` and `.authorizations` (sessions), `MaintenanceLock`, `ServerSetup` (a restore - never opens setup again), and `AppConfiguration` (its SMTP password; it is made again from appsettings at boot). One + never opens setup again), `AppConfiguration` (its SMTP password; it is made again from appsettings at boot), + `PushSubscription` (sessions' too, R9) and `PushKeys` (the VAPID key: a restore keeps the live one). One backup or restore runs at a time (`MaintenanceLock`, a heartbeat document; a holder silent for 2 minutes is taken over), and the media janitor purges nothing meanwhile. `BackupScheduler` backs up at `Backups:NightlyAt` (03:30 UTC), or at once when it missed the night; @@ -895,7 +997,8 @@ client's role route never does. nothing holds any more goes there too); the staff's log (`AdminAction`) is the union, so a restore never loses an entry, its own request included, and the server's settings and custom emoji (`LocalEmoji`, R7) are P's: the live settings and emoji outlive a restore, and an emoji deleted since stays deleted; - 5. every session ends (a new `SessionStamp` for each root, persona tokens deleted), circuits close, and a + 5. every session ends (a new `SessionStamp` for each root, persona tokens deleted, and every Web Push subscription with + them, R9: clients subscribe again, to the same VAPID key), circuits close, and a `RestoreRecord` (never in a backup) tells what happened (`admin restore --status`). Each attempt redoes everything. Refused before anything changed, it is abandoned and recorded, and the server boots as @@ -1154,6 +1257,25 @@ tools/pasture/run.sh down # removes e DM; her endorsement of him and her note on him are hers alone (feat_bob, a sibling persona of the same login, sees neither, and `GET /api/v1/accounts/:id/endorsements` answers nobody else), and feat_bob is never suggested to her; the stubs (translate 422, annual reports, the admin API 403) answer. +- **Web Push (`scenarios/push.sh`, needs mastodon and pushsink; after featured in batch A):** push_alice subscribes with + the sink's keys; mastouser's mention reaches the sink as aes128gcm, its VAPID JWT by the instance API's key for the + sink's origin with TTL and Urgency, and decrypts to a `mention` with her own access token and her notification's id; + the mention alert turned off stops it while his favourite is pushed; `policy=followed` stops him until she follows him; + `standard=false` is aesgcm (Encryption, Crypto-Key, `WebPush`) and decrypts too, and the new subscription replaces the + old; push_bob, a sibling persona of the same login, never gets hers nor she his; DELETE stops the pushes; the sink + answering 410 removes the subscription; a token revoked at `/oauth/revoke` is pushed nothing; no subscription is left. + Both instance APIs say 4.5.0 and give the key an app registration gives. +- **pushsink (`peers/pushsink.sh`, `pushsink/pushsink.cs`, as `pushsink.test`):** a Web Push service of the pasture's + own, a .NET file-based app published once into `.pushsink` like flood. It makes each subscription's keys itself, checks + what a push service checks (the VAPID JWT: ES256 by the server's key, the instance API's or the one the scenario gave + it; `aud` its own origin; `exp` ahead and at most a day; a `mailto:` or `https:` `sub`; TTL), decrypts `aes128gcm` and + `aesgcm` with its own code, not PrivaPub's, and keeps what it took. Its API, behind Caddy: `POST /subscriptions` + (`{"server_key"}` optional) answers `{id, endpoint, keys: {p256dh, auth}}`; `GET /subscriptions/{id}`; + `POST /subscriptions/{id}/answer` (`{"status": 410, "times": 1}`, for the next pushes); `GET` and `DELETE + /subscriptions/{id}/received` (each push: when, the status answered, `ok` and the errors, headers, the JWT's claims, the + decrypted payload); `POST /push/{id}` is the push service (201, 403 for a VAPID failure, 400 for the rest, 404 for no + such subscription); `/health`. PrivaPub reaches it through Caddy as every peer, since its `push` client honours + `appsettings.Pasture.json`'s Federation exceptions; nothing new is configured. In memory only. - **Mailpit (1.31.4, `peers/mailpit.sh`, as `mailpit.test`):** the pasture's mail server. `appsettings.Pasture.json` sends PrivaPub's mail to `pasture-mailpit:1025` (plain SMTP, any login accepted) and its recovery links to `https://decepub.test`. `scenarios/accounts.sh` reads what arrived through Mailpit's API: an address set with the @@ -1318,8 +1440,9 @@ the owner's GoToSocial account.** the server's own pre-deploy backup (`PrivaPub admin backup --kind pre-deploy --db-only`, run from the new build before its migrations, verified, and checked to hold no statistics salt; no deploy while `restore.json` waits), stop → rsync → start, a `127.0.0.1:6970/build.json` health loop with rollback, then public checks (actor, NodeInfo, - Swagger 404, inbox junk 400, unsigned 401) and `tools/smoke/mastodon-api.sh` (app registration, client credentials, - discovery, instance, public timeline). Then it checks that what production should be running is running: + Swagger 404, inbox junk 400, unsigned 401) and `tools/smoke/mastodon-api.sh` (app registration with its `vapid_key`, + client credentials, discovery, both instance APIs with the advertised version and the VAPID key, public timeline). + Then it checks that what production should be running is running: - NodeInfo and the instance API agree on registrations (closed, invitations only); - `/stargazing` says the crawler is on; - nothing signs in (owner decision 2026-10-17): the deploy's own login, `deploy-smoke`, gave its @thepra to the diff --git a/FEDERATION.md b/FEDERATION.md index 559277d..d33b5ed 100644 --- a/FEDERATION.md +++ b/FEDERATION.md @@ -456,6 +456,14 @@ PrivaPub keeps statistics about servers, never about accounts (see `/stargazing` - **Opting out:** in robots.txt, the crawler obeys the group `PrivaPub-Stargazer`, then `PrivaPub`, then `*`. A robots.txt that answers with a server error or times out also keeps it out, as do this server's domain blocks. +## Web Push + +Not federation, but traffic an operator may see: when a client signed in as a persona subscribes to Web Push (Mastodon's +`/api/v1/push/subscription`), PrivaPub POSTs its notifications to the push service the client named, encrypted to the +client (RFC 8291 `aes128gcm`, or the older `aesgcm` draft Mastodon still sends) and carrying a VAPID JWT (RFC 8292) signed +by the server's own P-256 key, the one `/api/v2/instance` gives as `configuration.vapid.public_key`. No HTTP signature +and no ActivityPub; https endpoints on public addresses only, no redirects. The push service learns when, never what. + ## Local-only posts Posts with a location (shown to nearby users of this server) never leave the server, in any form. diff --git a/PrivaPub/Api/Mastodon/Auth/OAuthSetup.cs b/PrivaPub/Api/Mastodon/Auth/OAuthSetup.cs index 935103d..ad122fb 100644 --- a/PrivaPub/Api/Mastodon/Auth/OAuthSetup.cs +++ b/PrivaPub/Api/Mastodon/Auth/OAuthSetup.cs @@ -72,6 +72,23 @@ namespace PrivaPub.Api.Mastodon.Auth options.AddEventHandler(builder => builder.UseInlineHandler(OutOfBandCode) .SetOrder(OpenIddictServerAspNetCoreHandlers.Authentication.ProcessFormPostResponse.Descriptor.Order - 1_000) .SetType(OpenIddictServerHandlerType.Custom)); + // a revoked token takes its Web Push subscription with it (R9), once OpenIddict has revoked it + options.AddEventHandler(builder => builder.UseInlineHandler(async context => + { + if (context.GenericTokenPrincipal?.GetTokenId() is { Length: > 0 } tokenId) + await PrivaPub.Domain.Social.WebPush.ForgetToken(tokenId); + }) + .SetOrder(OpenIddictServerHandlers.Revocation.RevokeToken.Descriptor.Order + 1_000) + .SetType(OpenIddictServerHandlerType.Custom)); + // Mastodon 4.4's userinfo (Controllers/UserinfoController), named in the authorization server's metadata + options.AddEventHandler(builder => builder.UseInlineHandler(context => + { + if (context.Issuer != default) + context.UserInfoEndpoint = new Uri(context.Issuer, Controllers.UserinfoController.Path.TrimStart('/')); + return default; + }) + .SetOrder(OpenIddictServerHandlers.Discovery.AttachEndpoints.Descriptor.Order + 1_000) + .SetType(OpenIddictServerHandlerType.Custom)); options.AddEventHandler(builder => builder.UseInlineHandler(context => { if (context.Response.AccessToken != default) diff --git a/PrivaPub/Api/Mastodon/Auth/PersonaExchange.cs b/PrivaPub/Api/Mastodon/Auth/PersonaExchange.cs index f5956ff..71c9163 100644 --- a/PrivaPub/Api/Mastodon/Auth/PersonaExchange.cs +++ b/PrivaPub/Api/Mastodon/Auth/PersonaExchange.cs @@ -27,6 +27,9 @@ namespace PrivaPub.Api.Mastodon.Auth public static string FirstPartyClientId(IConfiguration configuration) => configuration["ClientApi:FirstPartyClientId"] is { Length: > 0 } clientId ? clientId : "decepub"; + // what the first-party client's persona tokens may hold: push since R9, so decePub can subscribe to Web Push + public static readonly string[] FirstPartyScopes = { "read", "write", "follow", "push" }; + //the first-party application: public (a browser keeps no secret), token exchange only, no redirect public static async Task EnsureFirstPartyClient(IServiceProvider services, CancellationToken token) { @@ -46,7 +49,7 @@ namespace PrivaPub.Api.Mastodon.Auth Permissions.GrantTypes.TokenExchange } }; - foreach (var scopeName in new[] { "read", "write", "follow" }) + foreach (var scopeName in FirstPartyScopes) descriptor.Permissions.Add(Permissions.Prefixes.Scope + scopeName); var existing = await applications.FindByClientIdAsync(clientId, token); diff --git a/PrivaPub/Api/Mastodon/Auth/TokenController.cs b/PrivaPub/Api/Mastodon/Auth/TokenController.cs index 959704d..2cb74f4 100644 --- a/PrivaPub/Api/Mastodon/Auth/TokenController.cs +++ b/PrivaPub/Api/Mastodon/Auth/TokenController.cs @@ -70,7 +70,7 @@ namespace PrivaPub.Api.Mastodon.Auth var identity = new ClaimsIdentity(TokenValidationParameters.DefaultAuthenticationType, Claims.Name, Claims.Role); identity.SetClaim(Claims.Subject, avatar.ID); identity.SetClaim(Claims.Name, avatar.UserName); - identity.SetScopes(MastodonScopes.Parse(request.Scope).Intersect(new[] { "read", "write", "follow" })); + identity.SetScopes(MastodonScopes.Parse(request.Scope).Intersect(PersonaExchange.FirstPartyScopes)); identity.SetDestinations(_ => new[] { Destinations.AccessToken }); return SignIn(new ClaimsPrincipal(identity), OpenIddictServerAspNetCoreDefaults.AuthenticationScheme); } diff --git a/PrivaPub/Api/Mastodon/Controllers/AccountsController.cs b/PrivaPub/Api/Mastodon/Controllers/AccountsController.cs index 3e571f2..7e5fd8e 100644 --- a/PrivaPub/Api/Mastodon/Controllers/AccountsController.cs +++ b/PrivaPub/Api/Mastodon/Controllers/AccountsController.cs @@ -47,7 +47,8 @@ namespace PrivaPub.Api.Mastodon.Controllers _outbox = outbox; } - [HttpGet("/api/v1/accounts/verify_credentials"), Scope("read:accounts")] + // Mastodon 4.4: a token with the profile scope alone may read it too + [HttpGet("/api/v1/accounts/verify_credentials"), Scope("read:accounts|profile")] public async Task VerifyCredentials(CancellationToken token) => Json(await _mapper.Local(Me, withSource: true, token)); [HttpPatch("/api/v1/accounts/update_credentials"), Scope("write:accounts"), EnableRateLimiting(RateLimiting.Uploads), RequestSizeLimit(20 * 1024 * 1024), @@ -398,6 +399,28 @@ namespace PrivaPub.Api.Mastodon.Controllers public async Task DomainBlocks(CancellationToken token) => Json((await DB.Default.Find().Match(b => b.AvatarId == MyId).Sort(b => b.Domain, Order.Ascending).ExecuteAsync(token)).Select(b => b.Domain).ToList()); + // Mastodon's preview of a server block: how many of the accounts the persona follows, and of those following it, the + // block would take away, counted as BlockDomain takes them (an account on the server or a subdomain, or whose handle + // is) + [HttpGet("/api/v1/domain_blocks/preview"), Scope("write:blocks")] + public async Task PreviewDomainBlock(CancellationToken token) + { + var domain = Params.Get("domain")?.Trim().ToLowerInvariant(); + if (string.IsNullOrEmpty(domain)) + return Error(StatusCodes.Status400BadRequest, "Param is missing or the value is empty: domain"); + if (Uri.CheckHostName(domain) != UriHostNameType.Dns) + return Error(StatusCodes.Status422UnprocessableEntity, "Validation failed: Domain is invalid"); + var followings = await _dbEntities.Followings.Match(f => f.AvatarId == MyId && !f.TargetIsLocal && f.State == FollowState.Accepted) + .ExecuteAsync(token); + var followers = await _dbEntities.Followers.Match(f => f.LocalActorId == MyId && f.IsAccepted).ExecuteAsync(token); + var handles = await Hidden.HandleDomains(followings.Select(f => f.TargetActorURI).Concat(followers.Select(f => f.ActorURI)).ToList(), token); + return Json(new + { + following_count = followings.Count(f => Hidden.Blocks(domain, f.TargetActorURI, handles.GetValueOrDefault(f.TargetActorURI))), + followers_count = followers.Count(f => Hidden.Blocks(domain, f.ActorURI, handles.GetValueOrDefault(f.ActorURI))) + }); + } + [HttpPost("/api/v1/domain_blocks"), Scope("write:blocks")] public async Task BlockDomain(CancellationToken token) { diff --git a/PrivaPub/Api/Mastodon/Controllers/AppsController.cs b/PrivaPub/Api/Mastodon/Controllers/AppsController.cs index a323766..dee609f 100644 --- a/PrivaPub/Api/Mastodon/Controllers/AppsController.cs +++ b/PrivaPub/Api/Mastodon/Controllers/AppsController.cs @@ -16,10 +16,12 @@ namespace PrivaPub.Api.Mastodon.Controllers public class AppsController : MastodonController { readonly IOpenIddictApplicationManager _applications; + readonly PrivaPub.Infrastructure.Push.IVapid _vapid; - public AppsController(IOpenIddictApplicationManager applications) + public AppsController(IOpenIddictApplicationManager applications, PrivaPub.Infrastructure.Push.IVapid vapid) { _applications = applications; + _vapid = vapid; } [HttpPost("/api/v1/apps"), AllowAnonymous] @@ -69,7 +71,8 @@ namespace PrivaPub.Api.Mastodon.Controllers RedirectUris = redirects, RedirectUri = string.Join("\n", redirects), ClientId = descriptor.ClientId, - ClientSecret = descriptor.ClientSecret + ClientSecret = descriptor.ClientSecret, + VapidKey = (await _vapid.Key(token)).ForApi//Web Push's server key, as Mastodon still gives it here }); } @@ -88,7 +91,7 @@ namespace PrivaPub.Api.Mastodon.Controllers website = properties.TryGetValue("website", out var website) ? website.GetString() : default, scopes = User.GetScopes().ToList(), redirect_uris = (await _applications.GetRedirectUrisAsync(application, token)).ToList(), - vapid_key = string.Empty + vapid_key = (await _vapid.Key(token)).ForApi }); } diff --git a/PrivaPub/Api/Mastodon/Controllers/InstanceController.cs b/PrivaPub/Api/Mastodon/Controllers/InstanceController.cs index 735349e..5093d30 100644 --- a/PrivaPub/Api/Mastodon/Controllers/InstanceController.cs +++ b/PrivaPub/Api/Mastodon/Controllers/InstanceController.cs @@ -28,7 +28,19 @@ namespace PrivaPub.Api.Mastodon.Controllers // are enforced. public class InstanceController : MastodonController { - public const string Version = "4.2.0 (compatible; PrivaPub)"; + // The Mastodon version this API answers as, then PrivaPub's own release (its tag, or the commit a build came from). + // 4.5.0 since R9 (2026-10-18): every client route of Mastodon 4.3, 4.4 and 4.5 is served or answers as a documented + // stub (docs/MASTODON-API.md), streaming and Web Push included. api_versions.mastodon is 4.5's own, 7. + public const string MastodonVersion = "4.5.0"; + public static readonly string Version = Advertised(string.IsNullOrEmpty(BuildInfo.Ref) ? BuildInfo.Commit : BuildInfo.Ref); + + static string Advertised(string release) + { + var own = release?.Trim(); + if (own is { Length: > 1 } && own[0] == 'v' && char.IsDigit(own[1])) + own = own[1..];//a tag v1.27.0 is release 1.27.0 + return string.IsNullOrEmpty(own) ? $"{MastodonVersion} (compatible; PrivaPub)" : $"{MastodonVersion} (compatible; PrivaPub {own})"; + } const int ActivityWeeks = 12; static readonly TimeSpan ActivityLifetime = TimeSpan.FromHours(1); static volatile CachedActivity _activity; @@ -39,14 +51,16 @@ namespace PrivaPub.Api.Mastodon.Controllers readonly IOptionsMonitor _registrations; readonly IInstanceSettings _settings; readonly MastodonMapper _mapper; + readonly PrivaPub.Infrastructure.Push.IVapid _vapid; public InstanceController(ILocalActorService localActors, IOptionsMonitor registrations, IInstanceSettings settings, - MastodonMapper mapper) + MastodonMapper mapper, PrivaPub.Infrastructure.Push.IVapid vapid) { _localActors = localActors; _registrations = registrations; _settings = settings; _mapper = mapper; + _vapid = vapid; } bool Open => _registrations.CurrentValue.IsOpen; @@ -165,7 +179,8 @@ namespace PrivaPub.Api.Mastodon.Controllers statuses = Statuses, media_attachments = MediaAttachments, polls = Polls, - translation = new { enabled = false } + translation = new { enabled = false }, + vapid = new { public_key = (await _vapid.Key(token)).ForApi }//Web Push (R9): the key every push is signed with }, registrations = new { enabled = Open, approval_required = false, message = Open ? default : RegistrationOptions.ClosedMessage }, contact = new { email = Settings.ContactEmail ?? string.Empty, account = await ContactAccount(Settings, token) }, diff --git a/PrivaPub/Api/Mastodon/Controllers/NotificationRequestsController.cs b/PrivaPub/Api/Mastodon/Controllers/NotificationRequestsController.cs index 7cffccc..28f771c 100644 --- a/PrivaPub/Api/Mastodon/Controllers/NotificationRequestsController.cs +++ b/PrivaPub/Api/Mastodon/Controllers/NotificationRequestsController.cs @@ -69,7 +69,7 @@ namespace PrivaPub.Api.Mastodon.Controllers [HttpGet("/api/v2/notifications/policy"), Scope("read:notifications")] public async Task Policy(CancellationToken token) => Json(await V2(await NotificationPolicies.Of(MyId, token), token)); - [HttpPatch("/api/v2/notifications/policy"), Scope("write:notifications")] + [HttpPatch("/api/v2/notifications/policy"), HttpPut("/api/v2/notifications/policy"), Scope("write:notifications")] public async Task SetPolicy(CancellationToken token) { var policy = await NotificationPolicies.Of(MyId, token); diff --git a/PrivaPub/Api/Mastodon/Controllers/PushController.cs b/PrivaPub/Api/Mastodon/Controllers/PushController.cs new file mode 100644 index 0000000..034edfa --- /dev/null +++ b/PrivaPub/Api/Mastodon/Controllers/PushController.cs @@ -0,0 +1,169 @@ +using Microsoft.AspNetCore.Authentication; +using Microsoft.AspNetCore.Mvc; +using Microsoft.Extensions.Options; + +using MongoDB.Driver; +using MongoDB.Entities; + +using OpenIddict.Abstractions; +using OpenIddict.Validation.AspNetCore; + +using PrivaPub.Api.Mastodon.Infrastructure; +using PrivaPub.Domain.Social; +using PrivaPub.Infrastructure.Http; +using PrivaPub.Infrastructure.Push; +using PrivaPub.Models.Social; + +namespace PrivaPub.Api.Mastodon.Controllers +{ + // Mastodon's Web Push API (completion program R9): one subscription per access token, so per persona, made, read, changed + // and deleted with the token itself (the `push` scope). Its parameters as Mastodon takes them, as a form or JSON: + // subscription[endpoint], subscription[keys][p256dh], subscription[keys][auth], subscription[standard] (4.4: RFC 8291 + // and RFC 8292, else the aesgcm draft), data[alerts][] and data[policy] (all, followed, follower, none). Its answer is + // Mastodon's WebPushSubscription. What is pushed, and how, is WebPushDispatcher's and WebPushJob's. + public class PushController : MastodonController + { + const int MaxEndpointLength = 2048; + + readonly IVapid _vapid; + readonly IOptionsMonitor _federation; + + public PushController(IVapid vapid, IOptionsMonitor federation) + { + _vapid = vapid; + _federation = federation; + } + + string TokenId => User.GetTokenId(); + + // a new subscription replaces the one this token had, as on Mastodon + [HttpPost("/api/v1/push/subscription"), Scope("push")] + public async Task Create(CancellationToken token) + { + var endpoint = Params.Get("subscription[endpoint]")?.Trim(); + if (string.IsNullOrEmpty(endpoint)) + return Error(StatusCodes.Status400BadRequest, "Param is missing or the value is empty: subscription"); + if (endpoint.Length > MaxEndpointLength || !Uri.TryCreate(endpoint, UriKind.Absolute, out var uri) || !WebPushJob.IsAllowed(uri, _federation.CurrentValue)) + return Error(StatusCodes.Status422UnprocessableEntity, "Validation failed: Endpoint is not a valid URL"); + var p256dh = Params.Get("subscription[keys][p256dh]")?.Trim(); + var auth = Params.Get("subscription[keys][auth]")?.Trim(); + if (string.IsNullOrEmpty(p256dh) || string.IsNullOrEmpty(auth)) + return Error(StatusCodes.Status422UnprocessableEntity, "Validation failed: Key p256dh can't be blank, Key auth can't be blank"); + if (!WebPushCrypto.AreUsable(WebPushCrypto.FromBase64Url(p256dh), WebPushCrypto.FromBase64Url(auth))) + return Error(StatusCodes.Status422UnprocessableEntity, "Validation failed: Keys are not valid Web Push keys"); + var (alerts, policy, invalid) = Data(); + if (invalid != default) + return invalid; + var tokenId = TokenId; + var accessToken = await AccessToken(); + if (string.IsNullOrEmpty(tokenId) || string.IsNullOrEmpty(accessToken)) + return Error(StatusCodes.Status401Unauthorized, "The access token is invalid"); + + var subscription = new PushSubscription + { + AvatarId = MyId, + TokenId = tokenId, + AccessToken = accessToken, + Endpoint = endpoint, + P256dh = p256dh, + Auth = auth, + Standard = Params.Bool("subscription[standard]") == true, + Alerts = alerts, + Policy = policy + }; + // (two at once with one token: the unique index on the token keeps one, and the later replaces it) + for (var attempt = 0; ; attempt++) + { + await WebPush.ForgetToken(tokenId); + try + { + subscription.ID = default; + await DB.Default.SaveAsync(subscription, token); + break; + } + catch (MongoWriteException ex) when (ex.WriteError?.Category == ServerErrorCategory.DuplicateKey && attempt < 2) + { + } + } + return Json(await View(subscription, token)); + } + + [HttpGet("/api/v1/push/subscription"), Scope("push")] + public async Task Show(CancellationToken token) => + await Mine(token) is { } subscription ? Json(await View(subscription, token)) : NotFoundError(); + + // Mastodon's update: the data given replaces what was there (alerts left out are off, the policy left out is all) + [HttpPut("/api/v1/push/subscription"), Scope("push")] + public async Task Update(CancellationToken token) + { + var subscription = await Mine(token); + if (subscription == default) + return NotFoundError(); + var (alerts, policy, invalid) = Data(); + if (invalid != default) + return invalid; + subscription.Alerts = alerts; + subscription.Policy = policy; + subscription.UpdatedAt = DateTime.UtcNow; + await DB.Default.Update().MatchID(subscription.ID) + .Modify(s => s.Alerts, alerts) + .Modify(s => s.Policy, policy) + .Modify(s => s.UpdatedAt, subscription.UpdatedAt) + .ExecuteAsync(token); + return Json(await View(subscription, token)); + } + + [HttpDelete("/api/v1/push/subscription"), Scope("push")] + public async Task Delete(CancellationToken token) + { + await WebPush.ForgetToken(TokenId); + return Json(new { }); + } + + async Task Mine(CancellationToken token) + { + var tokenId = TokenId; + var avatarId = MyId; + return string.IsNullOrEmpty(tokenId) ? default + : await DB.Default.Find().Match(s => s.TokenId == tokenId && s.AvatarId == avatarId).ExecuteFirstAsync(token); + } + + // data[alerts][] (each true or false; unknown types are ignored, as Mastodon permits only its own) and + // data[policy] + (List Alerts, PushPolicy Policy, IActionResult Invalid) Data() + { + var alerts = WebPush.Alerts.Where(alert => Params.Bool($"data[alerts][{alert}]") == true).ToList(); + var given = Params.Get("data[policy]"); + if (string.IsNullOrWhiteSpace(given)) + return (alerts, PushPolicy.All, default); + return WebPush.ParsePolicy(given) is { } policy + ? (alerts, policy, default) + : (alerts, PushPolicy.All, Error(StatusCodes.Status422UnprocessableEntity, "Validation failed: Policy is not included in the list")); + } + + // the bearer token of this request, wherever the client put it (OpenIddict takes the header, a form field or the + // query): the payload carries it back, as Mastodon's does + async Task AccessToken() + { + var stored = await HttpContext.GetTokenAsync(OpenIddictValidationAspNetCoreDefaults.AuthenticationScheme, + OpenIddictValidationAspNetCoreConstants.Tokens.AccessToken); + if (!string.IsNullOrEmpty(stored)) + return stored; + var header = Request.Headers.Authorization.ToString(); + if (header.StartsWith("Bearer ", StringComparison.OrdinalIgnoreCase)) + return header["Bearer ".Length..].Trim(); + return Params.Get("access_token"); + } + + // Mastodon's WebPushSubscription + async Task View(PushSubscription subscription, CancellationToken token) => new + { + id = subscription.ID, + endpoint = subscription.Endpoint, + standard = subscription.Standard, + alerts = WebPush.Alerts.ToDictionary(alert => alert, alert => subscription.Alerts.Contains(alert)), + server_key = (await _vapid.Key(token)).ForApi, + policy = WebPush.Name(subscription.Policy) + }; + } +} diff --git a/PrivaPub/Api/Mastodon/Controllers/SearchController.cs b/PrivaPub/Api/Mastodon/Controllers/SearchController.cs index 7aaee9e..3de059b 100644 --- a/PrivaPub/Api/Mastodon/Controllers/SearchController.cs +++ b/PrivaPub/Api/Mastodon/Controllers/SearchController.cs @@ -140,9 +140,6 @@ namespace PrivaPub.Api.Mastodon.Controllers public class StubsController : MastodonController { - [HttpGet("/api/v1/push/subscription"), Scope("push")] - public IActionResult PushSubscription() => Error(StatusCodes.Status404NotFound, "Record not found"); - // Routes Mastodon answers that we have nothing behind: an empty or default answer lets clients degrade, where a // 404 breaks some of them (Mastodon invariant 5) [HttpGet("/api/v1/timelines/link"), Microsoft.AspNetCore.Authorization.AllowAnonymous] @@ -163,9 +160,35 @@ namespace PrivaPub.Api.Mastodon.Controllers [HttpPost("/api/v1/statuses/{id}/translate"), Scope("read:statuses")] public IActionResult Translate(string id) => Error(StatusCodes.Status422UnprocessableEntity, "Translation is not available on this server"); - // Mastodon 4.3's yearly reports (Wrapstodon): none are made here, so there are never any to show - [HttpGet("/api/v1/annual_reports"), Scope("read:accounts")] - public IActionResult AnnualReports() => Json(new { annual_reports = Array.Empty(), accounts = Array.Empty(), statuses = Array.Empty() }); + // Mastodon 4.3's yearly reports (Wrapstodon): none are made here, so there are never any to show, for any year (4.4's + // show), and marking one read changes nothing + [HttpGet("/api/v1/annual_reports"), HttpGet("/api/v1/annual_reports/{year}"), Scope("read:accounts")] + public IActionResult AnnualReports(string year) => Json(new { annual_reports = Array.Empty(), accounts = Array.Empty(), statuses = Array.Empty() }); + + [HttpPost("/api/v1/annual_reports/{year}/read"), Scope("write:accounts")] + public IActionResult ReadAnnualReport(string year) => Json(new { }); + + // Registration through the API is cut (owner decision 2026-10-17): a login is made by setup or an invitation, on + // /clientapi. Mastodon answers this when its registrations are closed; its e-mail confirmation routes follow, since no + // account here waits for one + [HttpPost("/api/v1/accounts"), Microsoft.AspNetCore.Authorization.AllowAnonymous] + public IActionResult Register() => Error(StatusCodes.Status403Forbidden, "This action is not allowed"); + + [HttpPost("/api/v1/emails/confirmations"), Scope("write:accounts")] + public IActionResult ResendConfirmation() => + Error(StatusCodes.Status403Forbidden, "This method is only available while the e-mail is awaiting confirmation"); + + [HttpGet("/api/v1/emails/check_confirmation"), Scope("read:accounts")] + public IActionResult CheckConfirmation() => Json(true); + + // the peers' names are not published (/instance/peers stays empty), so a search of them finds none + [HttpGet("/api/v1/peers/search"), Microsoft.AspNetCore.Authorization.AllowAnonymous] + public IActionResult SearchPeers() => Json(Array.Empty()); + + // Mastodon 4.4's asynchronous refreshes are announced by a Mastodon-Async-Refresh header, which PrivaPub never sends + // (a thread's replies are read again in the background and simply show up): there is never one to ask about + [HttpGet("/api/v1_alpha/async_refreshes/{id}"), Scope("read")] + public IActionResult AsyncRefresh(string id) => NotFoundError(); // the Mastodon admin API is cut: moderation stays on /clientapi (ModeratorController, AdminController), so every // admin route answers as Mastodon answers an account that may not use it, whoever asks diff --git a/PrivaPub/Api/Mastodon/Controllers/TimelinesController.cs b/PrivaPub/Api/Mastodon/Controllers/TimelinesController.cs index 84845d8..52511f7 100644 --- a/PrivaPub/Api/Mastodon/Controllers/TimelinesController.cs +++ b/PrivaPub/Api/Mastodon/Controllers/TimelinesController.cs @@ -177,6 +177,18 @@ namespace PrivaPub.Api.Mastodon.Controllers return Json(conversation ?? new Conversation { Id = group.ID }); } + // Mastodon's unread (R9): shown unread again until it is read + [HttpPost("/api/v1/conversations/{id}/unread"), Scope("write:conversations")] + public async Task UnreadConversation(string id, CancellationToken token) + { + var group = await MyConversation(id, token); + if (group == default) + return NotFoundError(); + await ConversationStates.Unread(MyId, group.ID, token); + var conversation = (await ConversationsOf(new List { group }, token)).FirstOrDefault(); + return Json(conversation ?? new Conversation { Id = group.ID, Unread = true }); + } + // off the persona's list until a newer message; the messages themselves stay [HttpDelete("/api/v1/conversations/{id}"), Scope("write:conversations")] public async Task RemoveConversation(string id, CancellationToken token) @@ -410,7 +422,7 @@ namespace PrivaPub.Api.Mastodon.Controllers return mapped == default ? NotFoundError() : Json(mapped); } - [HttpPost("/api/v1/notifications/clear"), Scope("write:notifications")] + [HttpPost("/api/v1/notifications/clear"), HttpPost("/api/v2/notifications/clear"), Scope("write:notifications")] public async Task Clear(CancellationToken token) { await DB.Default.DeleteAsync(n => n.AvatarId == MyId); diff --git a/PrivaPub/Api/Mastodon/Controllers/UserinfoController.cs b/PrivaPub/Api/Mastodon/Controllers/UserinfoController.cs new file mode 100644 index 0000000..a8e109d --- /dev/null +++ b/PrivaPub/Api/Mastodon/Controllers/UserinfoController.cs @@ -0,0 +1,40 @@ +using Microsoft.AspNetCore.Mvc; + +using PrivaPub.Api.Mastodon.Infrastructure; +using PrivaPub.Api.Mastodon.Mappers; +using PrivaPub.Federation.Actors; + +namespace PrivaPub.Api.Mastodon.Controllers +{ + // Mastodon 4.4's OAuth userinfo (OpenID Connect's shape, GET or POST, the `profile` scope): who the token's persona is, + // for a site that signs in with it. Only what its public account says already; never its login. The authorization + // server's metadata names it (OAuthSetup). + public class UserinfoController : MastodonController + { + public const string Path = "/oauth/userinfo"; + + readonly MastodonMapper _mapper; + readonly ILocalActorService _localActors; + + public UserinfoController(MastodonMapper mapper, ILocalActorService localActors) + { + _mapper = mapper; + _localActors = localActors; + } + + [HttpGet(Path), HttpPost(Path), Scope("profile")] + public async Task Show(CancellationToken token) + { + var account = await _mapper.Local(Me, false, token); + return Json(new + { + iss = _localActors.BaseAddress + "/", + sub = Me.Uri, + name = account.DisplayName, + preferred_username = account.Username, + profile = account.Url, + picture = account.Avatar + }); + } + } +} diff --git a/PrivaPub/Api/Mastodon/Infrastructure/MastodonController.cs b/PrivaPub/Api/Mastodon/Infrastructure/MastodonController.cs index 552427a..0de6804 100644 --- a/PrivaPub/Api/Mastodon/Infrastructure/MastodonController.cs +++ b/PrivaPub/Api/Mastodon/Infrastructure/MastodonController.cs @@ -48,7 +48,8 @@ namespace PrivaPub.Api.Mastodon.Infrastructure var scope = context.ActionDescriptor.EndpointMetadata.OfType().LastOrDefault(); if (User.Identity?.IsAuthenticated == true) { - if (scope != default && !MastodonScopes.Grants(User.GetScopes(), scope.Scope)) + // (a scope written a|b is either) + if (scope != default && !scope.Scope.Split('|').Any(one => MastodonScopes.Grants(User.GetScopes(), one))) { context.Result = Error(StatusCodes.Status403Forbidden, "This action is outside the authorized scopes"); return; diff --git a/PrivaPub/Domain/Moderation/AccountModeration.cs b/PrivaPub/Domain/Moderation/AccountModeration.cs index 7a6168b..e1c96d4 100644 --- a/PrivaPub/Domain/Moderation/AccountModeration.cs +++ b/PrivaPub/Domain/Moderation/AccountModeration.cs @@ -154,6 +154,7 @@ namespace PrivaPub.Domain.Moderation // nothing acts as it from now on: its Mastodon tokens end and its open streams close await _tokens.RevokeBySubjectAsync(avatar.ID, CancellationToken.None); await _authorizations.RevokeBySubjectAsync(avatar.ID, CancellationToken.None); + await Social.WebPush.Forget(new[] { avatar.ID });//and its Web Push subscriptions with them (R9): it is pushed nothing Streams.CloseFor(avatar.ID); } if (action is ModerationActions.Silence or ModerationActions.Suspend) diff --git a/PrivaPub/Domain/Social/Notifications.cs b/PrivaPub/Domain/Social/Notifications.cs index 819db78..2d66f85 100644 --- a/PrivaPub/Domain/Social/Notifications.cs +++ b/PrivaPub/Domain/Social/Notifications.cs @@ -7,6 +7,25 @@ namespace PrivaPub.Domain.Social { public static class Notifications { + // told of every notification once it is kept for the persona's lists and streams: never one the notification policy + // filtered or dropped, nor one its blocks and mutes hid. Web Push hangs here (R9, WebPushDispatcher, set while the + // host runs; the command line sets nothing). Whatever it does never fails the notification. + public static Func Told { get; set; } + + static async Task Tell(Notification notification, CancellationToken token) + { + if (Told is not { } told) + return; + try + { + await told(notification, token); + } + catch (Exception) + { + //the hook logs its own failures + } + } + public static async Task Add(string avatarId, NotificationType type, string fromAccountId, string fromActorUri, string postId, CancellationToken token, string emoji = default, string emojiUrl = default, string text = default) { @@ -42,6 +61,7 @@ namespace PrivaPub.Domain.Social var told = new Timelines.Streams.Event("notification", notification.ID); Timelines.Streams.ToPersona(avatarId, new Timelines.Streams.Key(Timelines.Streams.User), told); Timelines.Streams.ToPersona(avatarId, new Timelines.Streams.Key(Timelines.Streams.UserNotification), told); + await Tell(notification, token); } catch (MongoWriteException ex) when (ex.WriteError?.Category == ServerErrorCategory.DuplicateKey) { @@ -73,6 +93,7 @@ namespace PrivaPub.Domain.Social var told = new Timelines.Streams.Event("notification", notification.ID); Timelines.Streams.ToPersona(avatarId, new Timelines.Streams.Key(Timelines.Streams.User), told); Timelines.Streams.ToPersona(avatarId, new Timelines.Streams.Key(Timelines.Streams.UserNotification), told); + await Tell(notification, CancellationToken.None); } } } diff --git a/PrivaPub/Domain/Social/WebPush.cs b/PrivaPub/Domain/Social/WebPush.cs new file mode 100644 index 0000000..cc01e15 --- /dev/null +++ b/PrivaPub/Domain/Social/WebPush.cs @@ -0,0 +1,156 @@ +using MongoDB.Entities; + +using PrivaPub.Api.Mastodon.Controllers; +using PrivaPub.Infrastructure.Jobs; +using PrivaPub.Models.Federation; +using PrivaPub.Models.Jobs; +using PrivaPub.Models.Social; + +using System.Text.Json; + +namespace PrivaPub.Domain.Social +{ + public sealed record PushPayload(string SubscriptionId, string NotificationId); + + // What Web Push sends, and to whom (completion program R9). A notification is pushed to each subscription of its persona + // whose alert for its type is on and whose policy admits its sender, as Mastodon's Web::PushSubscription#pushable? says; + // a filtered one never is, nor anything for a persona held back (suspended, or its login banned). + public static class WebPush + { + // Mastodon 4.5's alerts (its Notification::TYPES, which data[alerts] takes), then Pleroma's for emoji reactions. + // admin.sign_up and admin.report are kept but never fire: moderation is on /clientapi, and it notifies nobody. + public static readonly IReadOnlyList Alerts = new[] + { + "mention", "quote", "status", "reblog", "follow", "follow_request", "favourite", "poll", "update", "quoted_update", + "severed_relationships", "moderation_warning", "annual_report", "admin.sign_up", "admin.report", "pleroma:emoji_reaction" + }; + + /// The alert a notification's type answers to: its Mastodon name. + public static string AlertOf(NotificationType type) => NotificationsController.Names.TryGetValue(type, out var name) ? name : default; + + public static string Name(PushPolicy policy) => policy switch + { + PushPolicy.Followed => "followed", + PushPolicy.Follower => "follower", + PushPolicy.None => "none", + _ => "all" + }; + + public static PushPolicy? ParsePolicy(string value) => value?.Trim().ToLowerInvariant() switch + { + "all" => PushPolicy.All, + "followed" => PushPolicy.Followed, + "follower" => PushPolicy.Follower, + "none" => PushPolicy.None, + _ => (PushPolicy?)null + }; + + public static async Task Pushable(PushSubscription subscription, Notification notification, CancellationToken token) + { + if (notification.Filtered || notification.AvatarId != subscription.AvatarId) + return false; + var alert = AlertOf(notification.Type); + if (alert == default || !subscription.Alerts.Contains(alert)) + return false; + return await Admits(subscription, notification, token); + } + + // Mastodon's policy_allows_notification?: everyone, nobody, the accounts the persona follows, those that follow it. What + // the persona tells itself (its own poll ended, a warning from the staff, which Mastodon sends from the account itself) + // passes every policy but none. + static async Task Admits(PushSubscription subscription, Notification notification, CancellationToken token) + { + if (subscription.Policy == PushPolicy.None) + return false; + if (subscription.Policy == PushPolicy.All || notification.FromAccountId == subscription.AvatarId) + return true; + var avatarId = subscription.AvatarId; + var fromId = notification.FromAccountId; + var fromUri = notification.FromActorURI; + if (subscription.Policy == PushPolicy.Followed) + { + if (!string.IsNullOrEmpty(fromId) && await DB.Default.Find() + .Match(f => f.AvatarId == avatarId && f.FollowerKind == null && f.State == FollowState.Accepted && f.TargetAccountId == fromId).ExecuteAnyAsync(token)) + return true; + return !string.IsNullOrEmpty(fromUri) && await DB.Default.Find() + .Match(f => f.AvatarId == avatarId && f.FollowerKind == null && f.State == FollowState.Accepted && f.TargetActorURI == fromUri).ExecuteAnyAsync(token); + } + if (!string.IsNullOrEmpty(fromUri) && await DB.Default.Find() + .Match(f => f.LocalActorId == avatarId && f.LocalActorKind == LocalActorKind.Person && f.IsAccepted && f.ActorURI == fromUri).ExecuteAnyAsync(token)) + return true; + return !string.IsNullOrEmpty(fromId) && await DB.Default.Find() + .Match(f => f.AvatarId == fromId && f.FollowerKind == null && f.State == FollowState.Accepted && f.TargetAccountId == avatarId).ExecuteAnyAsync(token); + } + + /// Every subscription of these personas goes (their tokens ended, or they did). + public static async Task Forget(IReadOnlyCollection avatarIds) + { + var ids = avatarIds?.Where(id => !string.IsNullOrEmpty(id)).Distinct().ToList() ?? new List(); + if (ids.Count > 0) + await DB.Default.DeleteAsync(s => ids.Contains(s.AvatarId));//a delete takes no token outside a transaction + } + + /// The subscription made with this token goes (the token was revoked). + public static async Task ForgetToken(string tokenId) + { + if (!string.IsNullOrEmpty(tokenId)) + await DB.Default.DeleteAsync(s => s.TokenId == tokenId); + } + } + + // Hangs on Notifications.Told while the host runs and queues one SendPush job per subscription the notification goes to + // (WebPushJob sends it, encrypted). Queueing is a few indexed reads and inserts; nothing leaves the server here. + public class WebPushDispatcher : IHostedService + { + readonly IJobQueue _jobs; + readonly ILogger _logger; + + public WebPushDispatcher(IJobQueue jobs, ILogger logger) + { + _jobs = jobs; + _logger = logger; + } + + public Task StartAsync(CancellationToken cancellationToken) + { + Notifications.Told = Dispatch; + return Task.CompletedTask; + } + + public Task StopAsync(CancellationToken cancellationToken) + { + Notifications.Told = default; + return Task.CompletedTask; + } + + public async Task Dispatch(Notification notification, CancellationToken token) + { + try + { + if (notification == default || notification.Filtered || Moderation.Restrictions.IsSuspended(notification.AvatarId)) + return; + var alert = WebPush.AlertOf(notification.Type); + if (alert == default) + return; + var avatarId = notification.AvatarId; + var subscriptions = await DB.Default.Find().Match(s => s.AvatarId == avatarId && s.Alerts.Contains(alert)).ExecuteAsync(token); + var jobs = new List(); + foreach (var subscription in subscriptions) + if (await WebPush.Pushable(subscription, notification, token)) + jobs.Add(new Job + { + Kind = JobKind.SendPush, + Payload = JsonSerializer.Serialize(new PushPayload(subscription.ID, notification.ID)), + Host = Uri.TryCreate(subscription.Endpoint, UriKind.Absolute, out var endpoint) ? endpoint.Host : default, + DedupeKey = $"push|{subscription.ID}|{notification.ID}" + }); + if (jobs.Count > 0) + await _jobs.EnqueueMany(jobs, token); + } + catch (Exception ex) when (ex is not OperationCanceledException || !token.IsCancellationRequested) + { + _logger.LogWarning(ex, "{Service} could not queue the pushes of a notification", nameof(WebPushDispatcher)); + } + } + } +} diff --git a/PrivaPub/Domain/Statuses/ConversationStates.cs b/PrivaPub/Domain/Statuses/ConversationStates.cs index ab94343..bd8fab4 100644 --- a/PrivaPub/Domain/Statuses/ConversationStates.cs +++ b/PrivaPub/Domain/Statuses/ConversationStates.cs @@ -21,14 +21,40 @@ namespace PrivaPub.Domain.Statuses await Read(authorAvatarId, conversationId, postId, token); } - public static Task Read(string avatarId, string conversationId, string postId, CancellationToken token) => - Mark(avatarId, conversationId, s => s.LastReadPostId, postId, token); + public static async Task Read(string avatarId, string conversationId, string postId, CancellationToken token) + { + await Mark(avatarId, conversationId, s => s.LastReadPostId, postId, token); + await DB.Default.Update() + .Match(s => s.AvatarId == avatarId && s.ConversationId == conversationId && s.MarkedUnread) + .Modify(s => s.MarkedUnread, false) + .ExecuteAsync(token); + } + + // Mastodon's unread: the conversation shows unread, whoever wrote last, until the persona reads it again (R9) + public static async Task Unread(string avatarId, string conversationId, CancellationToken token) + { + Task Write() => DB.Default.Update() + .Match(s => s.AvatarId == avatarId && s.ConversationId == conversationId) + .Modify(s => s.MarkedUnread, true) + .Modify(s => s.UpdatedAt, DateTime.UtcNow) + .Option(o => o.IsUpsert = true) + .ExecuteAsync(token); + try + { + await Write(); + } + catch (MongoWriteException ex) when (ex.WriteError?.Category == ServerErrorCategory.DuplicateKey) + { + await Write(); + } + } public static Task Hide(string avatarId, string conversationId, string postId, CancellationToken token) => Mark(avatarId, conversationId, s => s.HiddenUpTo, postId, token); public static bool IsUnread(ConversationState state, string lastPostId, string lastAuthorId, string avatarId) => - lastPostId != default && lastAuthorId != avatarId && string.CompareOrdinal(lastPostId, state?.LastReadPostId) > 0; + state?.MarkedUnread == true + || lastPostId != default && lastAuthorId != avatarId && string.CompareOrdinal(lastPostId, state?.LastReadPostId) > 0; public static bool IsHidden(ConversationState state, string lastPostId) => state?.HiddenUpTo != default && string.CompareOrdinal(lastPostId, state.HiddenUpTo) <= 0; diff --git a/PrivaPub/Infrastructure/Backup/ProtectiveMerge.cs b/PrivaPub/Infrastructure/Backup/ProtectiveMerge.cs index deb40c0..b50aa10 100644 --- a/PrivaPub/Infrastructure/Backup/ProtectiveMerge.cs +++ b/PrivaPub/Infrastructure/Backup/ProtectiveMerge.cs @@ -168,6 +168,9 @@ namespace PrivaPub.Infrastructure.Backup .Set("CredentialsChangedAt", now), cancellationToken: token); foreach (var collection in new[] { "openiddict.tokens", "openiddict.authorizations" }) report.SessionsEnded += (int)(await database.GetCollection(collection).DeleteManyAsync(FilterDefinition.Empty, token)).DeletedCount; + // and the Web Push subscriptions made with those tokens (R9; never in a backup): clients subscribe again once signed + // in. The VAPID key (PushKeys, never in a backup either) stays the live one. + await database.GetCollection("PushSubscription").DeleteManyAsync(FilterDefinition.Empty, token); // what the backup remembered of servers being down is long past await database.GetCollection("RemoteInstance").UpdateManyAsync(FilterDefinition.Empty, diff --git a/PrivaPub/Infrastructure/Backup/ServerBackup.cs b/PrivaPub/Infrastructure/Backup/ServerBackup.cs index 14b6898..1a24a2f 100644 --- a/PrivaPub/Infrastructure/Backup/ServerBackup.cs +++ b/PrivaPub/Infrastructure/Backup/ServerBackup.cs @@ -42,7 +42,9 @@ namespace PrivaPub.Infrastructure.Backup ["AppConfiguration"] = "the configuration's copy holds the SMTP password, and is made again from appsettings at boot", [nameof(RestoreRecord)] = "what restores did outlives what they restore", ["PersonaArchive"] = "personas' archives being made or imported belong to the moment", - ["ServerSetup"] = "the server is set up once: a restore never opens its setup again" + ["ServerSetup"] = "the server is set up once: a restore never opens its setup again", + ["PushSubscription"] = "Web Push subscriptions belong to sessions: a restore ends every one, and clients subscribe again", + ["PushKeys"] = "the VAPID key every push subscription was made with: a restore keeps the live one" }; static readonly JsonWriterSettings Json = new() { OutputMode = JsonOutputMode.CanonicalExtendedJson, Indent = false }; diff --git a/PrivaPub/Infrastructure/Cli/AdminCommands.cs b/PrivaPub/Infrastructure/Cli/AdminCommands.cs index 0491f98..7a261e1 100644 --- a/PrivaPub/Infrastructure/Cli/AdminCommands.cs +++ b/PrivaPub/Infrastructure/Cli/AdminCommands.cs @@ -290,6 +290,7 @@ namespace PrivaPub.Infrastructure.Cli await DB.Default.Update().MatchID(link.ID).Modify(ra => ra.RootId, root.ID).ExecuteAsync(); await services.GetRequiredService().RevokeBySubjectAsync(persona.ID); await services.GetRequiredService().RevokeBySubjectAsync(persona.ID); + await Domain.Social.WebPush.Forget(new[] { persona.ID });//its Web Push subscriptions go with its tokens (R9) await Log(services, AdminActions.PersonaMove, AdminTargets.Persona, persona.ID, "@" + personaName, "moved to another login"); output.WriteLine($"@{personaName}: moved to {login}"); return 0; diff --git a/PrivaPub/Infrastructure/Data/Indexes.cs b/PrivaPub/Infrastructure/Data/Indexes.cs index 379f42e..81c2da6 100644 --- a/PrivaPub/Infrastructure/Data/Indexes.cs +++ b/PrivaPub/Infrastructure/Data/Indexes.cs @@ -73,6 +73,9 @@ namespace PrivaPub.Infrastructure.Data await Plain(token, n => n.TargetAccountId); await DB.Default.Index().Key(d => d.AvatarId, KeyType.Ascending).Key(d => d.AccountId, KeyType.Ascending) .Option(o => o.Unique = true).CreateAsync(token); + // Web Push (a new collection, R9): one subscription per access token, a persona's read when it is notified + await Unique(s => s.TokenId, Builders.Filter.Gt(s => s.TokenId, ""), token); + await Plain(token, s => s.AvatarId); // the staff's log, read newest first await DB.Default.Index().Key(a => a.At, KeyType.Descending).CreateAsync(token); diff --git a/PrivaPub/Infrastructure/Http/SafeHttpHandlerFactory.cs b/PrivaPub/Infrastructure/Http/SafeHttpHandlerFactory.cs index 18717aa..cff7382 100644 --- a/PrivaPub/Infrastructure/Http/SafeHttpHandlerFactory.cs +++ b/PrivaPub/Infrastructure/Http/SafeHttpHandlerFactory.cs @@ -11,7 +11,7 @@ namespace PrivaPub.Infrastructure.Http public static class SafeHttpHandlerFactory { - public static SocketsHttpHandler Create(FederationOptions options, IConnectedAddresses connected = default) => new() + public static SocketsHttpHandler Create(FederationOptions options, IConnectedAddresses connected = default, TimeSpan? connectTimeout = default) => new() { SslOptions = options.AcceptAnyCertificate ? new SslClientAuthenticationOptions { RemoteCertificateValidationCallback = (_, _, _, _) => true } @@ -20,7 +20,7 @@ namespace PrivaPub.Infrastructure.Http UseProxy = false, UseCookies = false, AutomaticDecompression = DecompressionMethods.All, - ConnectTimeout = TimeSpan.FromSeconds(10), + ConnectTimeout = connectTimeout ?? TimeSpan.FromSeconds(10), PooledConnectionLifetime = TimeSpan.FromMinutes(2), MaxResponseHeadersLength = 64, ConnectCallback = (context, token) => Connect(context.DnsEndPoint, options.AllowPrivateNetworks, token, connected) diff --git a/PrivaPub/Infrastructure/Push/Vapid.cs b/PrivaPub/Infrastructure/Push/Vapid.cs new file mode 100644 index 0000000..36443df --- /dev/null +++ b/PrivaPub/Infrastructure/Push/Vapid.cs @@ -0,0 +1,116 @@ +using MongoDB.Driver; +using MongoDB.Entities; + +using PrivaPub.Models; + +using System.Globalization; +using System.Security.Cryptography; +using System.Text; +using System.Text.Json; + +namespace PrivaPub.Infrastructure.Push +{ + // The server's VAPID key (RFC 8292, completion program R9) and the JWT every push carries. The key is made the first time + // it is needed and kept in Mongo (PushKeys, one document); every process reads that one. + public interface IVapid + { + Task Key(CancellationToken token); + } + + public sealed class VapidKey + { + // RFC 8292 2: no more than 24 hours ahead; half of it leaves room for a push service's clock + public static readonly TimeSpan TokenLifetime = TimeSpan.FromHours(12); + + readonly ECParameters _parameters; + + public VapidKey(byte[] publicPoint, byte[] d) + { + PublicPoint = publicPoint; + _parameters = new ECParameters + { + Curve = ECCurve.NamedCurves.nistP256, + D = d, + Q = new ECPoint { X = publicPoint[1..33], Y = publicPoint[33..] } + }; + } + + public byte[] PublicPoint { get; } + + /// As the APIs give it (the instance's vapid.public_key, an app's vapid_key, a subscription's server_key): + /// URL-safe base64 with its padding, as Mastodon's VAPID_PUBLIC_KEY is written. + public string ForApi => Convert.ToBase64String(PublicPoint).Replace('+', '-').Replace('/', '_'); + + /// As headers take it (RFC 8292's k, the draft's Crypto-Key p256ecdsa): base64url without padding. + public string ForHeader => WebPushCrypto.Base64Url(PublicPoint); + + /// The ES256 JWT for one push service (aud: its origin), signed by this key. + public string Token(string audience, string subject, DateTime now) + { + var header = WebPushCrypto.Base64Url("{\"typ\":\"JWT\",\"alg\":\"ES256\"}"u8); + var claims = WebPushCrypto.Base64Url(JsonSerializer.SerializeToUtf8Bytes(new Dictionary + { + ["aud"] = audience, + ["exp"] = new DateTimeOffset(now + TokenLifetime).ToUnixTimeSeconds(), + ["sub"] = subject + })); + var signed = header + "." + claims; + using var key = ECDsa.Create(_parameters); + var signature = key.SignData(Encoding.ASCII.GetBytes(signed), HashAlgorithmName.SHA256, DSASignatureFormat.IeeeP1363FixedFieldConcatenation); + return signed + "." + WebPushCrypto.Base64Url(signature); + } + + /// A push service's origin, the JWT's aud: scheme, host and a port that is not the scheme's own. + public static string Audience(Uri endpoint) => endpoint.GetLeftPart(UriPartial.Authority).ToLower(CultureInfo.InvariantCulture); + } + + public class Vapid : IVapid + { + readonly SemaphoreSlim _gate = new(1, 1); + volatile VapidKey _key; + + public async Task Key(CancellationToken token) + { + if (_key is { } known) + return known; + await _gate.WaitAsync(token); + try + { + if (_key is { } meanwhile) + return meanwhile; + var stored = await Stored(token) ?? await Make(token); + _key = new VapidKey(WebPushCrypto.FromBase64Url(stored.PublicKey), WebPushCrypto.FromBase64Url(stored.PrivateKey)); + return _key; + } + finally + { + _gate.Release(); + } + } + + static Task Stored(CancellationToken token) => + DB.Default.Find().Match(k => k.ID == PushKeys.Name).ExecuteFirstAsync(token); + + // the first need makes it; two processes at once keep the first one saved + static async Task Make(CancellationToken token) + { + using var key = ECDsa.Create(ECCurve.NamedCurves.nistP256); + var parameters = key.ExportParameters(true); + var keys = new PushKeys + { + ID = PushKeys.Name, + PublicKey = WebPushCrypto.Base64Url([0x04, .. parameters.Q.X, .. parameters.Q.Y]), + PrivateKey = WebPushCrypto.Base64Url(parameters.D) + }; + try + { + await DB.Default.InsertAsync(keys, token); + return keys; + } + catch (MongoWriteException ex) when (ex.WriteError?.Category == ServerErrorCategory.DuplicateKey) + { + return await Stored(token); + } + } + } +} diff --git a/PrivaPub/Infrastructure/Push/WebPushCrypto.cs b/PrivaPub/Infrastructure/Push/WebPushCrypto.cs new file mode 100644 index 0000000..c29442c --- /dev/null +++ b/PrivaPub/Infrastructure/Push/WebPushCrypto.cs @@ -0,0 +1,169 @@ +using System.Buffers.Binary; +using System.Security.Cryptography; +using System.Text; + +namespace PrivaPub.Infrastructure.Push +{ + // Web Push message encryption, on System.Security.Cryptography alone (completion program R9). Two encodings, as Mastodon + // sends them: + // - RFC 8291 (`aes128gcm`, in RFC 8188's record), for a subscription made with `standard` (Mastodon 4.4); + // - the draft before it (`aesgcm`: draft-ietf-webpush-encryption-04 over draft-ietf-httpbis-encryption-encoding-03), for + // every other one, with its salt in the Encryption header and our public key in Crypto-Key. + // Both: a fresh P-256 key pair and a fresh 16-byte salt per message, ECDH with the client's p256dh, HKDF-SHA-256 keyed by + // its auth secret, AES-128-GCM, one record. Only the client can read the message; the push service carries it blind. + public static class WebPushCrypto + { + public const int RecordSize = 4096; + // the largest payload a 4096-byte aes128gcm body holds: 86 bytes of header, the delimiter and the tag + public const int MaxPayload = RecordSize - 86 - 1 - 16; + const int SaltLength = 16; + const int TagLength = 16; + + public sealed record Message(byte[] Body, string ContentEncoding, byte[] Salt, byte[] ServerPublicKey); + + public static Message Aes128Gcm(ReadOnlySpan plaintext, byte[] clientPublicKey, byte[] authSecret) + { + using var server = ECDiffieHellman.Create(ECCurve.NamedCurves.nistP256); + return Aes128Gcm(plaintext, clientPublicKey, authSecret, server, RandomNumberGenerator.GetBytes(SaltLength)); + } + + public static Message Aes128Gcm(ReadOnlySpan plaintext, byte[] clientPublicKey, byte[] authSecret, ECDiffieHellman server, byte[] salt) + { + if (plaintext.Length > MaxPayload) + throw new ArgumentException($"a push payload holds {MaxPayload} bytes at most", nameof(plaintext)); + var serverPublic = PublicPoint(server); + var shared = SharedSecret(server, clientPublicKey); + // RFC 8291 3.3 and 3.4: PRK_key = HMAC-SHA-256(auth, ecdh); IKM = HKDF-Expand(PRK_key, "WebPush: info" 0x00 + // ua_public as_public, 32) + var keyInfo = Concat(Encoding.ASCII.GetBytes("WebPush: info\0"), clientPublicKey, serverPublic); + var ikm = HKDF.Expand(HashAlgorithmName.SHA256, HKDF.Extract(HashAlgorithmName.SHA256, shared, authSecret), 32, keyInfo); + // RFC 8188 2.2 and 2.3: the content encryption key and the nonce, from the salt + var prk = HKDF.Extract(HashAlgorithmName.SHA256, ikm, salt); + var cek = HKDF.Expand(HashAlgorithmName.SHA256, prk, 16, Encoding.ASCII.GetBytes("Content-Encoding: aes128gcm\0")); + var nonce = HKDF.Expand(HashAlgorithmName.SHA256, prk, 12, Encoding.ASCII.GetBytes("Content-Encoding: nonce\0")); + // one record, the last: the payload, its delimiter 0x02, no padding + var record = new byte[plaintext.Length + 1]; + plaintext.CopyTo(record); + record[^1] = 0x02; + var encrypted = Seal(cek, nonce, record); + // RFC 8188 2.1's header: the salt, the record size, the key id's length and the key id (RFC 8291: our public key) + var body = new byte[SaltLength + 4 + 1 + serverPublic.Length + encrypted.Length]; + salt.CopyTo(body, 0); + BinaryPrimitives.WriteUInt32BigEndian(body.AsSpan(SaltLength), RecordSize); + body[SaltLength + 4] = (byte)serverPublic.Length; + serverPublic.CopyTo(body, SaltLength + 5); + encrypted.CopyTo(body, SaltLength + 5 + serverPublic.Length); + return new Message(body, "aes128gcm", salt, serverPublic); + } + + public static Message AesGcm(ReadOnlySpan plaintext, byte[] clientPublicKey, byte[] authSecret) + { + using var server = ECDiffieHellman.Create(ECCurve.NamedCurves.nistP256); + return AesGcm(plaintext, clientPublicKey, authSecret, server, RandomNumberGenerator.GetBytes(SaltLength)); + } + + public static Message AesGcm(ReadOnlySpan plaintext, byte[] clientPublicKey, byte[] authSecret, ECDiffieHellman server, byte[] salt) + { + if (plaintext.Length > MaxPayload) + throw new ArgumentException($"a push payload holds {MaxPayload} bytes at most", nameof(plaintext)); + var serverPublic = PublicPoint(server); + var shared = SharedSecret(server, clientPublicKey); + // the draft's: IKM = HKDF(salt: auth, ecdh, "Content-Encoding: auth" 0x00, 32), then the key and nonce from the salt + // over a context of both public keys, each after its length + var ikm = HKDF.Expand(HashAlgorithmName.SHA256, HKDF.Extract(HashAlgorithmName.SHA256, shared, authSecret), 32, + Encoding.ASCII.GetBytes("Content-Encoding: auth\0")); + var context = Concat(Encoding.ASCII.GetBytes("P-256\0"), Length16(clientPublicKey), clientPublicKey, Length16(serverPublic), serverPublic); + var prk = HKDF.Extract(HashAlgorithmName.SHA256, ikm, salt); + var cek = HKDF.Expand(HashAlgorithmName.SHA256, prk, 16, Concat(Encoding.ASCII.GetBytes("Content-Encoding: aesgcm\0"), context)); + var nonce = HKDF.Expand(HashAlgorithmName.SHA256, prk, 12, Concat(Encoding.ASCII.GetBytes("Content-Encoding: nonce\0"), context)); + // two bytes saying how much padding follows (none), then the payload + var padded = new byte[2 + plaintext.Length]; + plaintext.CopyTo(padded.AsSpan(2)); + return new Message(Seal(cek, nonce, padded), "aesgcm", salt, serverPublic); + } + + /// A client's public key: an uncompressed point on P-256, or a CryptographicException. + public static ECDiffieHellman ImportPublic(byte[] point) + { + if (point is not { Length: 65 } || point[0] != 0x04) + throw new CryptographicException("not an uncompressed P-256 point"); + var key = ECDiffieHellman.Create(new ECParameters + { + Curve = ECCurve.NamedCurves.nistP256, + Q = new ECPoint { X = point[1..33], Y = point[33..] } + }); + return key; + } + + /// Whether a subscription's keys can be encrypted to: a point on P-256 and a 16-byte secret. + public static bool AreUsable(byte[] p256dh, byte[] auth) + { + if (auth is not { Length: 16 }) + return false; + try + { + using var key = ImportPublic(p256dh); + using var probe = ECDiffieHellman.Create(ECCurve.NamedCurves.nistP256); + probe.DeriveRawSecretAgreement(key.PublicKey); + return true; + } + catch (CryptographicException) + { + return false; + } + } + + public static byte[] PublicPoint(ECAlgorithm key) + { + var parameters = key.ExportParameters(false); + return [0x04, .. parameters.Q.X, .. parameters.Q.Y]; + } + + public static string Base64Url(ReadOnlySpan bytes) => + Convert.ToBase64String(bytes).TrimEnd('=').Replace('+', '-').Replace('/', '_'); + + /// base64url or base64, padded or not, as clients send keys; null when it is neither. + public static byte[] FromBase64Url(string text) + { + if (string.IsNullOrWhiteSpace(text)) + return default; + var normal = text.Trim().TrimEnd('=').Replace('-', '+').Replace('_', '/'); + normal = normal.PadRight(normal.Length + (4 - normal.Length % 4) % 4, '='); + var bytes = new byte[normal.Length * 3 / 4]; + return Convert.TryFromBase64String(normal, bytes, out var written) ? bytes[..written] : default; + } + + static byte[] SharedSecret(ECDiffieHellman server, byte[] clientPublicKey) + { + using var client = ImportPublic(clientPublicKey); + return server.DeriveRawSecretAgreement(client.PublicKey); + } + + static byte[] Seal(byte[] key, byte[] nonce, byte[] plaintext) + { + var output = new byte[plaintext.Length + TagLength]; + using var aes = new AesGcm(key, TagLength); + aes.Encrypt(nonce, plaintext, output.AsSpan(0, plaintext.Length), output.AsSpan(plaintext.Length)); + return output; + } + + static byte[] Length16(byte[] value) + { + var length = new byte[2]; + BinaryPrimitives.WriteUInt16BigEndian(length, (ushort)value.Length); + return length; + } + + static byte[] Concat(params byte[][] parts) + { + var all = new byte[parts.Sum(p => p.Length)]; + var at = 0; + foreach (var part in parts) + { + part.CopyTo(all, at); + at += part.Length; + } + return all; + } + } +} diff --git a/PrivaPub/Infrastructure/Push/WebPushJob.cs b/PrivaPub/Infrastructure/Push/WebPushJob.cs new file mode 100644 index 0000000..7aa6a58 --- /dev/null +++ b/PrivaPub/Infrastructure/Push/WebPushJob.cs @@ -0,0 +1,264 @@ +using Microsoft.Extensions.Localization; +using Microsoft.Extensions.Options; + +using MongoDB.Entities; + +using OpenIddict.Abstractions; + +using PrivaPub.Api.Mastodon.Controllers; +using PrivaPub.Api.Mastodon.Mappers; +using PrivaPub.Domain.Instance; +using PrivaPub.Domain.Moderation; +using PrivaPub.Domain.Social; +using PrivaPub.Federation.Actors; +using PrivaPub.Infrastructure.Http; +using PrivaPub.Infrastructure.Jobs; +using PrivaPub.Models.Jobs; +using PrivaPub.Models.Social; +using PrivaPub.Models.User; +using PrivaPub.Resources; + +using System.Net; +using System.Net.Http.Headers; +using System.Net.Sockets; +using System.Text.Json; +using System.Text.RegularExpressions; + +using static OpenIddict.Abstractions.OpenIddictConstants; + +namespace PrivaPub.Infrastructure.Push +{ + // Sends one Web Push notification to one subscription (JobKind.SendPush, queued by WebPushDispatcher; completion + // program R9), as Mastodon's Web::PushNotificationWorker does: + // - the payload is Mastodon's (access_token, preferred_locale, notification_id, notification_type, icon, title, body), built + // when it is sent, as the persona sees the notification then (one it can no longer see, or dismissed meanwhile, is not + // sent), its access_token the subscription's own: the token is checked to be valid and the persona's first; + // - encrypted for the client alone (WebPushCrypto): RFC 8291 aes128gcm with RFC 8292's `vapid t=, k=` for a standard + // subscription, otherwise the aesgcm draft with Encryption and Crypto-Key and `WebPush `; the JWT (Vapid) names the + // push service's origin (aud), lasts twelve hours (exp) and the server's contact (sub: mailto: the contact email, or + // the server's address; never a persona); + // - TTL two days and Urgency normal, as Mastodon's; + // - through the `push` client: public addresses only, https only, no redirects, small timeouts (the federation client's + // safe handler, with the pasture's exceptions where Federation says so). + // A 404 or 410 means the subscription is gone: it is deleted. 408, 429, a 5xx, a timeout or a broken connection retry on + // the jobs' backoff; any other answer drops the push (logged by the push service's host alone). + public class WebPushJob : IJobHandler + { + public const string ClientName = "push"; + public static readonly TimeSpan RequestTimeout = TimeSpan.FromSeconds(15); + public static readonly TimeSpan ConnectTimeout = TimeSpan.FromSeconds(5); + // Mastodon's Web::PushNotificationWorker::TTL and URGENCY + const int TimeToLive = 48 * 3600; + const string Urgency = "normal"; + const int BodyLength = 140; + + readonly IHttpClientFactory _http; + readonly IVapid _vapid; + readonly IServiceScopeFactory _scopes; + readonly IInstanceSettings _settings; + readonly ILocalActorService _localActors; + readonly IOptionsMonitor _federation; + readonly IStringLocalizer _localizer; + readonly ILogger _logger; + + public WebPushJob(IHttpClientFactory http, IVapid vapid, IServiceScopeFactory scopes, IInstanceSettings settings, ILocalActorService localActors, + IOptionsMonitor federation, IStringLocalizer localizer, ILogger logger) + { + _http = http; + _vapid = vapid; + _scopes = scopes; + _settings = settings; + _localActors = localActors; + _federation = federation; + _localizer = localizer; + _logger = logger; + } + + public JobKind Kind => JobKind.SendPush; + public int Concurrency => 4; + public int MaxAttempts => 6; + public int PerHostLimit => 4; + + public async Task Handle(Job job, CancellationToken token) + { + var payload = JsonSerializer.Deserialize(job.Payload ?? "null"); + if (string.IsNullOrEmpty(payload?.SubscriptionId) || string.IsNullOrEmpty(payload.NotificationId)) + return JobOutcome.Done; + var subscriptionId = payload.SubscriptionId; + var subscription = await DB.Default.Find().Match(s => s.ID == subscriptionId).ExecuteFirstAsync(token); + if (subscription == default) + return JobOutcome.Done; + // a persona held back (suspended, its login banned) gets no push + if (Restrictions.IsSuspended(subscription.AvatarId)) + return JobOutcome.Done; + var notificationId = payload.NotificationId; + var avatarId = subscription.AvatarId; + var notification = await DB.Default.Find().Match(n => n.ID == notificationId && n.AvatarId == avatarId).ExecuteFirstAsync(token); + if (notification == default || !await WebPush.Pushable(subscription, notification, token)) + return JobOutcome.Done; + if (!Uri.TryCreate(subscription.Endpoint, UriKind.Absolute, out var endpoint) || !IsAllowed(endpoint, _federation.CurrentValue)) + return JobOutcome.Dead("endpoint"); + + byte[] json; + using (var scope = _scopes.CreateScope()) + { + // the token the subscription was made with must still be valid, and the persona's: it goes into the payload + if (!await TokenHolds(scope.ServiceProvider.GetRequiredService(), subscription, token)) + { + await WebPush.ForgetToken(subscription.TokenId); + return JobOutcome.Done; + } + var avatar = await DB.Default.Find().Match(a => a.ID == avatarId && a.DeletionAt == null).ExecuteFirstAsync(token); + if (avatar == default) + { + await WebPush.Forget(new[] { avatarId }); + return JobOutcome.Done; + } + var mapped = (await scope.ServiceProvider.GetRequiredService().Notifications(new List { notification }, avatarId, token)) + .FirstOrDefault(); + if (mapped == default) + return JobOutcome.Done; + json = JsonSerializer.SerializeToUtf8Bytes(new Dictionary + { + ["access_token"] = subscription.AccessToken, + ["preferred_locale"] = Locale(avatar, scope.ServiceProvider), + ["notification_id"] = notification.ID, + ["notification_type"] = WebPush.AlertOf(notification.Type), + ["icon"] = mapped.Account?.AvatarStatic ?? mapped.Account?.Avatar, + ["title"] = Title(notification, mapped), + ["body"] = Body(mapped) + }); + } + + var key = await _vapid.Key(token); + var p256dh = WebPushCrypto.FromBase64Url(subscription.P256dh); + var auth = WebPushCrypto.FromBase64Url(subscription.Auth); + if (!WebPushCrypto.AreUsable(p256dh, auth)) + return JobOutcome.Dead("keys"); + var message = subscription.Standard ? WebPushCrypto.Aes128Gcm(json, p256dh, auth) : WebPushCrypto.AesGcm(json, p256dh, auth); + var jwt = key.Token(VapidKey.Audience(endpoint), Subject(), DateTime.UtcNow); + + using var request = new HttpRequestMessage(HttpMethod.Post, endpoint) { Content = new ByteArrayContent(message.Body) }; + request.Content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream"); + request.Content.Headers.ContentEncoding.Add(message.ContentEncoding); + request.Headers.TryAddWithoutValidation("TTL", TimeToLive.ToString(System.Globalization.CultureInfo.InvariantCulture)); + request.Headers.TryAddWithoutValidation("Urgency", Urgency); + if (subscription.Standard) + request.Headers.TryAddWithoutValidation("Authorization", $"vapid t={jwt}, k={key.ForHeader}"); + else + { + request.Headers.TryAddWithoutValidation("Authorization", $"WebPush {jwt}"); + request.Headers.TryAddWithoutValidation("Encryption", $"salt={WebPushCrypto.Base64Url(message.Salt)}"); + request.Headers.TryAddWithoutValidation("Crypto-Key", $"dh={WebPushCrypto.Base64Url(message.ServerPublicKey)};p256ecdsa={key.ForHeader}"); + } + + try + { + using var response = await _http.CreateClient(ClientName).SendAsync(request, HttpCompletionOption.ResponseHeadersRead, token); + var status = (int)response.StatusCode; + if (status is >= 200 and < 300) + return JobOutcome.Done; + if (response.StatusCode is HttpStatusCode.NotFound or HttpStatusCode.Gone) + { + await DB.Default.DeleteAsync(subscription.ID); + _logger.LogInformation("{Kind}: {Host} answered {Status}, the subscription is gone", Kind, endpoint.Host, status); + return JobOutcome.Done; + } + if (response.StatusCode is HttpStatusCode.RequestTimeout or HttpStatusCode.TooManyRequests || status >= 500) + return JobOutcome.Retry($"status {status}"); + _logger.LogWarning("{Kind}: {Host} refused a push with {Status}; it is dropped", Kind, endpoint.Host, status); + return JobOutcome.Dead($"status {status}"); + } + catch (BlockedDestinationException) + { + _logger.LogWarning("{Kind}: {Host} does not resolve to a public address; the push is dropped", Kind, endpoint.Host); + return JobOutcome.Dead("blocked destination"); + } + catch (TaskCanceledException) when (!token.IsCancellationRequested) + { + return JobOutcome.Retry("timeout"); + } + catch (HttpRequestException ex) when (ex.InnerException is BlockedDestinationException) + { + _logger.LogWarning("{Kind}: {Host} does not resolve to a public address; the push is dropped", Kind, endpoint.Host); + return JobOutcome.Dead("blocked destination"); + } + catch (Exception ex) when (ex is HttpRequestException or IOException or SocketException) + { + return JobOutcome.Retry(ex.GetType().Name); + } + } + + /// https, a host, no user information; plain http only on a test network (Federation:AllowPlainHttp, which + /// production refuses at start). + public static bool IsAllowed(Uri endpoint, FederationOptions options) => + endpoint != default && endpoint.IsAbsoluteUri && !string.IsNullOrEmpty(endpoint.Host) && string.IsNullOrEmpty(endpoint.UserInfo) + && (endpoint.Scheme == Uri.UriSchemeHttps || options?.AllowPlainHttp == true && endpoint.Scheme == Uri.UriSchemeHttp); + + static async Task TokenHolds(IOpenIddictTokenManager tokens, PushSubscription subscription, CancellationToken token) + { + if (string.IsNullOrEmpty(subscription.TokenId) || string.IsNullOrEmpty(subscription.AccessToken)) + return false; + var stored = await tokens.FindByIdAsync(subscription.TokenId, token); + if (stored == default || !await tokens.HasStatusAsync(stored, Statuses.Valid, token)) + return false; + if (await tokens.GetExpirationDateAsync(stored, token) is { } expires && expires < DateTimeOffset.UtcNow) + return false; + return await tokens.GetSubjectAsync(stored, token) == subscription.AvatarId; + } + + // RFC 8292's sub: how the push service reaches the server's operator. The contact email the administrator gave, or + // the server's own address; never a persona. (A server on plain http, a test network, has no https: address to give, + // so it names a mailbox at its host: RFC 8292 takes only mailto: and https:.) + string Subject() + { + var email = _settings.Current.ContactEmail?.Trim(); + if (!string.IsNullOrEmpty(email) && !email.Any(char.IsWhiteSpace) && email.Contains('@')) + return "mailto:" + email; + var address = _localActors.BaseAddress; + return Uri.TryCreate(address, UriKind.Absolute, out var base_) && base_.Scheme != Uri.UriSchemeHttps + ? "mailto:privapub@" + base_.Host + : address; + } + + // the persona's language (its default for posts), else the server's first + static string Locale(Avatar avatar, IServiceProvider services) + { + var language = avatar.Settings?.DefaultLanguage; + return string.IsNullOrWhiteSpace(language) ? InstanceController.LanguageCodes(services)[0] : language.Trim(); + } + + // Mastodon's notification_mailer subjects + string Title(Notification notification, Api.Mastodon.Entities.Notification mapped) + { + var name = string.IsNullOrWhiteSpace(mapped.Account?.DisplayName) ? mapped.Account?.Username ?? string.Empty : mapped.Account.DisplayName; + return notification.Type switch + { + NotificationType.Mention => string.Format(_localizer["You were mentioned by {0}"], name), + NotificationType.Status => string.Format(_localizer["{0} just posted"], name), + NotificationType.Reblog => string.Format(_localizer["{0} boosted your post"], name), + NotificationType.Follow => string.Format(_localizer["{0} is now following you"], name), + NotificationType.FollowRequest => string.Format(_localizer["Pending follower: {0}"], name), + NotificationType.Favourite => string.Format(_localizer["{0} favorited your post"], name), + NotificationType.Poll => string.Format(_localizer["A poll by {0} has ended"], name), + NotificationType.Update => string.Format(_localizer["{0} edited a post"], name), + NotificationType.Quote => string.Format(_localizer["{0} quoted your post"], name), + NotificationType.Reaction => string.Format(_localizer["{0} reacted to your post with {1}"], name, notification.Emoji), + NotificationType.ModerationWarning => _localizer["You have received a moderation warning"], + _ => name + }; + } + + // Mastodon's: the post's warning, else its words, else the account's bio, without tags, 140 characters at most + static string Body(Api.Mastodon.Entities.Notification mapped) + { + var source = !string.IsNullOrWhiteSpace(mapped.Status?.SpoilerText) ? mapped.Status.SpoilerText + : mapped.Status?.Content ?? mapped.ModerationWarning?.Text ?? mapped.Account?.Note ?? string.Empty; + var text = WebUtility.HtmlDecode(Tags.Replace(Breaks.Replace(source, "\n"), string.Empty)).Trim(); + return text.Length <= BodyLength ? text : text[..(BodyLength - 3)] + "..."; + } + + static readonly Regex Breaks = new(@"|

\s*]*>", RegexOptions.IgnoreCase | RegexOptions.Compiled); + static readonly Regex Tags = new("<[^>]*>", RegexOptions.Compiled); + } +} diff --git a/PrivaPub/Middleware/SocialPubConfigurations.cs b/PrivaPub/Middleware/SocialPubConfigurations.cs index 1e49039..9721d4e 100644 --- a/PrivaPub/Middleware/SocialPubConfigurations.cs +++ b/PrivaPub/Middleware/SocialPubConfigurations.cs @@ -57,7 +57,21 @@ namespace PrivaPub.Middleware .ConfigurePrimaryHttpMessageHandler(provider => SafeHttpHandlerFactory.Create(provider.GetRequiredService>().Value, provider.GetRequiredService())) .AddHttpMessageHandler(provider => new EdgeHintsHandler(provider.GetRequiredService())); + // not federation: Web Push (R9), to the push services clients subscribed with. The same safe handler (public + // addresses only, the pasture's exceptions where Federation says so, no redirects), https only (WebPushJob), + // small timeouts, and no statistics of where it connects + service.AddHttpClient(Infrastructure.Push.WebPushJob.ClientName, (provider, client) => + { + var baseAddress = provider.GetRequiredService>().CurrentValue.BackendBaseAddress?.TrimEnd('/'); + client.Timeout = Infrastructure.Push.WebPushJob.RequestTimeout; + client.DefaultRequestHeaders.UserAgent.ParseAdd($"PrivaPub/{BuildInfo.Ref} (+{baseAddress}/)"); + }) + .ConfigurePrimaryHttpMessageHandler(provider => + SafeHttpHandlerFactory.Create(provider.GetRequiredService>().Value, connectTimeout: Infrastructure.Push.WebPushJob.ConnectTimeout)); return service + .AddSingleton() + .AddSingleton() + .AddHostedService() .AddSingleton() .AddSingleton() .AddSingleton() diff --git a/PrivaPub/Models/Jobs/Job.cs b/PrivaPub/Models/Jobs/Job.cs index fd8b4fc..7654da6 100644 --- a/PrivaPub/Models/Jobs/Job.cs +++ b/PrivaPub/Models/Jobs/Job.cs @@ -40,7 +40,8 @@ namespace PrivaPub.Models.Jobs ExportArchive, ImportArchive, SyncFeeds, - SendEmailVerification + SendEmailVerification, + SendPush//a Web Push notification to one subscription (R9, WebPushJob) } public enum JobState diff --git a/PrivaPub/Models/PushKeys.cs b/PrivaPub/Models/PushKeys.cs new file mode 100644 index 0000000..6e28ef7 --- /dev/null +++ b/PrivaPub/Models/PushKeys.cs @@ -0,0 +1,18 @@ +using MongoDB.Entities; + +namespace PrivaPub.Models +{ + // The server's VAPID key pair (RFC 8292, completion program R9): one P-256 key, made the first time something needs it + // (Infrastructure/Push/Vapid) and kept here, one document with the fixed id `vapid`. Every push carries a JWT it signs, + // and the push services tie each subscription to its public key, which the instance API and app registrations + // advertise. Never in a backup and never dropped by a restore (ServerBackup.Excluded): a restored key would break every + // subscription a client made since. + public class PushKeys : Entity + { + public const string Name = "vapid"; + + public string PublicKey { get; set; }//the uncompressed point (0x04 || X || Y), base64url without padding + public string PrivateKey { get; set; }//the private scalar D, base64url without padding + public DateTime CreatedAt { get; set; } = DateTime.UtcNow; + } +} diff --git a/PrivaPub/Models/Social/ConversationState.cs b/PrivaPub/Models/Social/ConversationState.cs index ce0e3cf..f19971c 100644 --- a/PrivaPub/Models/Social/ConversationState.cs +++ b/PrivaPub/Models/Social/ConversationState.cs @@ -10,6 +10,7 @@ namespace PrivaPub.Models.Social public string ConversationId { get; set; } public string LastReadPostId { get; set; } public string HiddenUpTo { get; set; } + public bool MarkedUnread { get; set; }//Mastodon's POST /api/v1/conversations/:id/unread, until it is read again (R9) public DateTime UpdatedAt { get; set; } = DateTime.UtcNow; } } diff --git a/PrivaPub/Models/Social/PushSubscription.cs b/PrivaPub/Models/Social/PushSubscription.cs new file mode 100644 index 0000000..1aac536 --- /dev/null +++ b/PrivaPub/Models/Social/PushSubscription.cs @@ -0,0 +1,34 @@ +using MongoDB.Entities; + +namespace PrivaPub.Models.Social +{ + // A Web Push subscription (completion program R9). One per Mastodon access token, as on Mastodon (a new one made with the + // same token replaces it), and so one persona's, the token's subject: never its login's, so nothing pairs sibling + // personas. It holds the token it was made with, which the encrypted payload carries back to the client as Mastodon's + // does (`access_token`), so it goes the moment the token does: revoked (/oauth/revoke, its login's sessions ended), its + // persona suspended, moved or deleted, a restore. Never in a backup (ServerBackup.Excluded). + public class PushSubscription : Entity + { + public string AvatarId { get; set; } + public string TokenId { get; set; }//the OpenIddict token's id; unique + public string AccessToken { get; set; }//the bearer token as the client sent it when it subscribed + public string Endpoint { get; set; }//the push service's URL for this client + public string P256dh { get; set; }//the client's ECDH public key: an uncompressed P-256 point, base64url as it sent it + public string Auth { get; set; }//its 16-byte authentication secret, base64url as it sent it + public bool Standard { get; set; }//RFC 8291 aes128gcm and RFC 8292 `vapid` when true (Mastodon 4.4), else the aesgcm draft + public List Alerts { get; set; } = new();//the alerts turned on, by Mastodon's names (WebPush.Alerts) + public PushPolicy Policy { get; set; } = PushPolicy.All; + public DateTime CreatedAt { get; set; } = DateTime.UtcNow; + public DateTime UpdatedAt { get; set; } = DateTime.UtcNow; + } + + // whose notifications are pushed (Mastodon's data[policy]): everyone's, the accounts the persona follows, the accounts + // that follow it, nobody's + public enum PushPolicy + { + All, + Followed, + Follower, + None + } +} diff --git a/PrivaPub/Services/PersonaRemoval.cs b/PrivaPub/Services/PersonaRemoval.cs index c8114b1..d74b234 100644 --- a/PrivaPub/Services/PersonaRemoval.cs +++ b/PrivaPub/Services/PersonaRemoval.cs @@ -95,6 +95,7 @@ namespace PrivaPub.Services // nothing acts as it while it goes await _tokens.RevokeBySubjectAsync(avatar.ID, token); await _authorizations.RevokeBySubjectAsync(avatar.ID, token); + await Domain.Social.WebPush.Forget(new[] { avatar.ID });//its Web Push subscriptions with its tokens (R9) Streams.CloseFor(avatar.ID); foreach (var group in await _dbEntities.Groups.Match(g => g.OwnerAvatarId == avatar.ID && !g.DeletionAt.HasValue).ExecuteAsync(token)) diff --git a/PrivaPub/Services/RootSessions.cs b/PrivaPub/Services/RootSessions.cs index 9fa3068..b8a620a 100644 --- a/PrivaPub/Services/RootSessions.cs +++ b/PrivaPub/Services/RootSessions.cs @@ -15,7 +15,7 @@ namespace PrivaPub.Services } // Ends everything a root is signed in with: its /clientapi tokens, through a new SessionStamp (checked by JwtEvents), - // and the Mastodon API tokens and authorizations of each of its personas. Used when its password is changed or + // and the Mastodon API tokens and authorizations of each of its personas, with their Web Push subscriptions. Used when its password is changed or // recovered, when it signs out everywhere and when it is deleted. End signs out one session only: the device that // asked (its Mastodon tokens it revokes itself). public class RootSessions : IRootSessions @@ -40,11 +40,14 @@ namespace PrivaPub.Services .Modify(u => u.SessionStamp, Guid.NewGuid().ToString("N")) .Modify(u => u.EndedSessions, new List())//the new stamp ends those too .ExecuteAsync(token); - foreach (var link in await _dbEntities.RootToAvatars.Match(ra => ra.RootId == rootId).ExecuteAsync(token)) + var links = await _dbEntities.RootToAvatars.Match(ra => ra.RootId == rootId).ExecuteAsync(token); + foreach (var link in links) { await _tokens.RevokeBySubjectAsync(link.AvatarId, token); await _authorizations.RevokeBySubjectAsync(link.AvatarId, token); } + // their Web Push subscriptions go with their tokens (R9) + await Domain.Social.WebPush.Forget(links.Select(l => l.AvatarId).ToList()); } // remembered as long as one of its tokens could still be valid; the ones that have expired are forgotten meanwhile diff --git a/docs/MASTODON-API.md b/docs/MASTODON-API.md new file mode 100644 index 0000000..113f83f --- /dev/null +++ b/docs/MASTODON-API.md @@ -0,0 +1,70 @@ +# The Mastodon client API: what PrivaPub answers + +The walk behind the version PrivaPub advertises (completion program R9, 2026-10-18). Every client route of Mastodon 4.3.0, +4.4.0 and 4.5.0, as their `config/routes/api.rb` and `config/routes.rb` define them, set against PrivaPub's controllers +(`PrivaPub/Api/Mastodon/Controllers`). Since nothing a client relies on is missing, both instance APIs say +`4.5.0 (compatible; PrivaPub )` (`InstanceController.Version`; Mastodon invariant 6 in CLAUDE.md), and +`api_versions.mastodon` is 7, 4.5's own. + +- **Served:** the route does what Mastodon's does, within PrivaPub's rules (a token is one persona, per-persona state, + located posts outside the Mastodon API). +- **Stub:** the route answers as Mastodon answers when the feature is off or empty here (an empty list, `{}`, 403, 404 or + 422 with Mastodon's message), so clients degrade instead of breaking (Mastodon invariant 5). Each is a decision, named. +- **Missing:** no route. None of these is the client API. + +When a route is added to Mastodon, add it here, and keep the advertised version honest. + +## Served + +| Area | Routes | +|---|---| +| Statuses | `GET /api/v1/statuses` (`id[]`), `POST`, `GET`/`PUT`/`DELETE /api/v1/statuses/:id`, `context`, `history`, `source`, `reblogged_by`, `favourited_by`, `reblog`/`unreblog`, `favourite`/`unfavourite`, `bookmark`/`unbookmark`, `mute`/`unmute`, `pin`/`unpin`; 4.5's `GET …/quotes`, `POST …/quotes/:id/revoke` and `PUT …/interaction_policy` | +| Timelines | `home`, `public`, `tag/:hashtag`, `list/:id` | +| Streaming | `/api/v1/streaming` (WebSocket) and `/api/v1/streaming/*` (server-sent events), `health`: `user`, `user:notification`, `public`, `public:local`, `public:remote`, `hashtag`, `hashtag:local`, `list`, `direct` | +| Accounts | `verify_credentials` (4.4: the `profile` scope too), `update_credentials`, `search`, `lookup`, `relationships`, `familiar_followers` (empty for every account by design: R8), `GET /api/v1/accounts` (`id[]`), `:id`, `:id/statuses`, `followers`, `following`, `lists`, `featured_tags`, `endorsements` (4.4; to the persona itself), `follow`, `unfollow`, `remove_from_followers`, `block`, `unblock`, `mute`, `unmute`, `pin`/`unpin` and 4.4's `endorse`/`unendorse`, `note` | +| Relationships | `blocks`, `mutes`, `domain_blocks` (`GET`, `POST`, `DELETE`), `domain_blocks/preview` (R9: how many follows and followers the block would take), `follow_requests` with `authorize` and `reject`, `endorsements` | +| Notifications | v1 `notifications` (index, `:id`, `clear`, `unread_count`, `:id/dismiss`), `notifications/requests` (index, `:id`, `accept`, `dismiss`, the bulk `accept` and `dismiss`, `merged`), `notifications/policy` (`GET`, `PUT`, `PATCH`); v2 grouped `notifications` (index, `:group_key`, `:group_key/accounts`, `:group_key/dismiss`, `unread_count`, `clear` since R9), `notifications/policy` (`GET`, `PATCH`, `PUT` since R9) | +| Web Push | `POST`, `GET`, `PUT`, `DELETE /api/v1/push/subscription` (R9; CLAUDE.md, Web Push) | +| Conversations | `conversations` (index), `DELETE :id`, `:id/read`, `:id/unread` (R9) | +| Media and polls | `POST /api/v1/media`, `/api/v2/media`, `GET`/`PUT`/`DELETE /api/v1/media/:id`; `polls/:id`, `polls/:id/votes` | +| Lists, filters, tags | `lists` (CRUD, `:id/accounts` `GET`/`POST`/`DELETE`); v1 `filters` and v2 `filters` with `keywords` and `statuses` (both nested and by id); `tags/:name` with `follow`, `unfollow` and 4.4's `feature`, `unfeature`; `followed_tags`; `featured_tags` (index, create, destroy) and `featured_tags/suggestions` | +| Discovery | `/api/v2/search`, `trends` and `trends/tags`, `trends/statuses`, `trends/links`, `directory`, `suggestions` (v1 index and destroy, v2 index) | +| The persona's own | `bookmarks`, `favourites`, `markers` (`GET`, `POST`), `preferences`, `scheduled_statuses` (index, show, update, destroy), `profile/avatar` and `profile/header` (`DELETE`), `reports` | +| The server | v1 and v2 `instance`, `instance/peers` (empty by design: which servers this one talks to is not published), `rules`, `domain_blocks` (the published ones), `extended_description`, `privacy_policy`, 4.4's `terms_of_service` and `terms_of_service/:date`, `languages`, `translation_languages` (none: translation is cut), `activity` (empty unless the administrator publishes it), `custom_emojis`, `announcements` with `dismiss` and `reactions` | +| Apps and OAuth | `POST /api/v1/apps`, `apps/verify_credentials`, `/oauth/authorize`, `/oauth/token`, `/oauth/revoke`, `/.well-known/oauth-authorization-server`, 4.4's `/oauth/userinfo` (`GET` and `POST`, the `profile` scope; R9) | + +## Stubs + +| Route | Answer | Why | +|---|---|---| +| `POST /api/v1/statuses/:id/translate` | 422 `Translation is not available on this server` | translation is cut (owner decision 2026-10-17) | +| `GET /api/v1/timelines/link` | `[]` | 4.3's timeline of a trending link: not built (trending links themselves are served) | +| `GET /api/v1/annual_reports`, 4.4's `GET /api/v1/annual_reports/:year` | `{"annual_reports": [], "accounts": [], "statuses": []}` | no yearly reports are made | +| 4.4's `POST /api/v1/annual_reports/:year/read` | `{}` | there is none to read | +| `POST /api/v1/accounts` | 403 `This action is not allowed` (Mastodon's answer with registrations closed) | registration through the API is cut: logins are made by setup or invitation on `/clientapi` | +| `POST /api/v1/emails/confirmations` | 403 `This method is only available while the e-mail is awaiting confirmation` | no account here waits for a confirmation | +| `GET /api/v1/emails/check_confirmation` | `true` | the same | +| `GET /api/v1/peers/search` | `[]` | peers are not published, as `instance/peers` | +| 4.4's `GET /api/v1_alpha/async_refreshes/:id` | 404 `Record not found` | PrivaPub never sends `Mastodon-Async-Refresh`: a thread's replies are read in the background and show up | +| `GET /api/v1/accounts/:id/identity_proofs` | `[]` | gone from Mastodon long ago; still asked by old clients | +| every verb under `/api/v1/admin/**`, `GET`/`POST` under `/api/v2/admin/**` | 403 `This action is not allowed`, signed in or not | the Mastodon admin API is cut: moderation is on `/clientapi` | + +## Missing, and not the client API + +| Route | Why it is not served | +|---|---| +| `GET /api/oembed` | embedding a post in another website; PrivaPub's public pages have no embed view | +| `/api/web/*` (`settings`, `embeds`, `push_subscriptions`) | Mastodon's own web interface, signed in by its session cookie; PrivaPub's client is decePub on `/clientapi` | +| `/oauth/authorized_applications` | Doorkeeper's HTML page of a user's applications, in Mastodon's web interface | +| `/invite/:code` (JSON) | Mastodon's app shows an invitation before registering through the API, which is cut; PrivaPub's invitations are `/clientapi`'s | +| `/.well-known/host-meta` | OStatus-era discovery; every client and server here uses WebFinger directly | +| `POST /api/v1/polls` | in 4.3's route list with no action behind it; gone in 4.4 | + +Beyond routes: the `public:media` streams (and their `:local` and `:remote` forms) are not offered; the media-only public +timelines (`only_media`) are, so a client polls them. + +## PrivaPub's own, on the same API + +`/api/v1/pleroma/statuses/:id/reactions` (Pleroma's emoji reactions), `/api/v1/statuses/:id/downvote` and `undownvote`, +`/api/v1/feeds` and `/api/v1/timelines/feed/:id` (PieFed's and Lemmy's feeds), and `/api/privapub/v1/*` (provenance, +servers, CDNs, events' join and leave): Mastodon invariant 7. diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 7fc2eb3..179344c 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -154,6 +154,24 @@ Written 2026-10-01 from the original 2023 code, the decePubClient UI, a federati settings form; a `Move` carries the persona's endorsement and note to the new account; featured tags, notes and endorsements are left out of the persona archive (Mastodon's layout has none), its `following_accounts.csv` now carries notify and languages. + - R9 Web Push, then the honest version (built 2026-10-18, not yet deployed; `tools/pasture/scenarios/push.sh` with the + pasture's own push service, `tools/pasture/pushsink`): the server's VAPID key (`PushKeys`, made on first need, in + Mongo, never in a backup), Mastodon's `/api/v1/push/subscription` (create, show, update, delete; one subscription + per access token, so per persona), every notification kept (never a filtered one) pushed to the subscriptions whose + alert and policy admit it, as `SendPush` jobs through the `push` client, encrypted as RFC 8291 `aes128gcm` or the + `aesgcm` draft, signed with an ES256 VAPID JWT; subscriptions end with their tokens (revocation, sessions ended, + suspension, deletion, a restore). Then the walk of every Mastodon 4.3, 4.4 and 4.5 client route + (`docs/MASTODON-API.md`), with what it found missing built (`conversations/:id/unread`, `domain_blocks/preview`, v2 + `notifications/clear` and `PUT` policy, `/oauth/userinfo` and the `profile` scope on `verify_credentials`) or answered + as stubs (annual reports by year, API registration 403, e-mail confirmation, peer search, async refreshes), and both + instance APIs advertising `4.5.0 (compatible; PrivaPub )`. Chosen while building it, for the owner to + confirm: the payload carries the subscription's own access token, as Mastodon's does (encrypted to that client + alone); what a persona tells itself (its poll ended, the staff's warning) passes the `followed` and `follower` policies + (Mastodon would hold it back); a push service's 4xx other than 404, 410, 408 and 429 drops that push and keeps the + subscription (Mastodon deletes it); the JWT lasts twelve hours (Mastodon's a day); titles are Mastodon's English + subjects and `preferred_locale` is the persona's posting language (no persona has an interface language); six + attempts on the jobs' backoff (Mastodon retries five times); `alerts` answers every alert true or false (Mastodon + echoes those sent); the key lives in Mongo, not in the configuration; `/oauth/userinfo` is served (Mastodon 4.4's). ## Intent @@ -600,7 +618,8 @@ The first refactor commit is a pure move with namespaces only. Logic changes fol - `Link` paging on `max_id`, `since_id` and `min_id`; CORS exposes `Link`. - `Idempotency-Key` honoured on posting statuses. - **Endpoints:** - - **Instance:** v1 and v2, advertising version `4.2.0 (compatible; PrivaPub)`. + - **Instance:** v1 and v2, advertising version `4.2.0 (compatible; PrivaPub)`. *`4.5.0 (compatible; PrivaPub + )` since R9 (2026-10-18), after the walk in `docs/MASTODON-API.md`.* - **Accounts:** `verify_credentials`, `update_credentials`, `:id`, `:id/statuses`, `lookup`, `relationships`, `search`, followers and following, follow and unfollow, `follow_requests`. - **Statuses:** CRUD, edit, `context`, `history`, `source`, favourite and reblog plus their undos (outbound Like and @@ -829,7 +848,8 @@ it, raw where it doesn't. - **Mastodon API:** ~~streaming WebSocket~~ (done 2026-10-05: `/api/v1/streaming` as a WebSocket and as server-sent events, user, notification, public, hashtag and list streams, each event mapped for its persona; a deletion reaches only the streams that showed the post), Web Push (gated: outbound traffic to push services; approved 2026-10-17, C R9), grouped - notifications. Then advertise an honest version. + notifications. Then advertise an honest version. *Done in R9 (2026-10-18): Web Push, then `4.5.0 (compatible; PrivaPub + )` once every 4.3, 4.4 and 4.5 client route was served or a named stub (`docs/MASTODON-API.md`).* - **Backfill:** an author's outbox after following them: **done 2026-10-05** (`OutboxBackfill`, the first page's public posts, twenty at most, once a day; checked live against Mastodon). - **Long tail:** diff --git a/tools/pasture/Caddyfile b/tools/pasture/Caddyfile index 55dba70..3258430 100644 --- a/tools/pasture/Caddyfile +++ b/tools/pasture/Caddyfile @@ -207,6 +207,11 @@ lemmy19.test { reverse_proxy pasture-lemmy19:8536 } +pushsink.test { + tls internal + reverse_proxy pasture-pushsink:8080 +} + flood1.test, flood2.test, flood3.test, flood4.test, flood5.test, flood6.test, flood7.test, flood8.test, flood9.test, flood10.test, flood11.test, flood12.test, flood13.test, flood14.test, flood15.test, flood16.test, flood17.test, flood18.test, flood19.test, flood20.test { tls internal reverse_proxy pasture-flood:8080 diff --git a/tools/pasture/lib/interop.sh b/tools/pasture/lib/interop.sh index 7621506..a4aa571 100644 --- a/tools/pasture/lib/interop.sh +++ b/tools/pasture/lib/interop.sh @@ -42,14 +42,14 @@ privapub_root() { echo "$root" | j "print(d['token'])" } -# privapub_token : creates the persona under the pasture root if needed and returns a Mastodon token for it, -# through the real OAuth flow (tools/smoke/oauth.sh). +# privapub_token [scopes]: creates the persona under the pasture root if needed and returns a Mastodon token for +# it, through the real OAuth flow (tools/smoke/oauth.sh); "read write follow" unless other scopes are asked for privapub_token() { - local persona=$1 jwt + local persona=$1 scopes=${2:-read write follow} jwt jwt=$(privapub_root) curl -s -o /dev/null -X POST $P/clientapi/avatar/private/insert -H 'Content-Type: application/json' -H "Authorization: Bearer $jwt" \ -d "{\"userName\":\"$persona\",\"name\":\"$persona of PrivaPub\",\"biography\":\"testing federation\"}" - "$here/../smoke/oauth.sh" "$P" "$ROOT_USER" "$ROOT_PASS" "$persona" "read write follow" | cut -d' ' -f1 + "$here/../smoke/oauth.sh" "$P" "$ROOT_USER" "$ROOT_PASS" "$persona" "$scopes" | cut -d' ' -f1 } # p_report : a persona reports an account's posts and asks for diff --git a/tools/pasture/peers/pushsink.sh b/tools/pasture/peers/pushsink.sh new file mode 100644 index 0000000..653993b --- /dev/null +++ b/tools/pasture/peers/pushsink.sh @@ -0,0 +1,22 @@ +# pushsink: a Web Push service as pushsink.test, for scenarios/push.sh (tools/pasture/pushsink/pushsink.cs, a .NET +# file-based app published once into .pushsink and run on the runtime image PrivaPub uses). It makes each subscription's +# keys itself, takes PrivaPub's pushes as a push service would (the VAPID JWT checked against the instance API's key, the +# aes128gcm or aesgcm body decrypted) and keeps what it took for the scenario to read: https://pushsink.test:6443/ +# subscriptions//received. It can be told to answer the next push otherwise (/subscriptions//answer, a 410). +# PrivaPub reaches it through Caddy as it reaches every peer: its push client honours appsettings.Pasture.json's +# Federation exceptions (AcceptAnyCertificate for Caddy's CA, AllowPrivateNetworks), which production refuses at start. +pushsink_up() { + local dotnet=${DOTNET:-$(command -v dotnet || echo ~/.dotnet/dotnet)} + if [ ! -x "$here/.pushsink/pushsink" ] || [ "$here/pushsink/pushsink.cs" -nt "$here/.pushsink/pushsink" ]; then + "$dotnet" publish "$here/pushsink/pushsink.cs" -r linux-x64 --self-contained true -o "$here/.pushsink" -v quiet + fi + podman run -d --replace --name pasture-pushsink --network $net --label pasture=1 -w /app -v "$here/.pushsink:/app:Z,ro" \ + -v "$ca/bundle.pem:/etc/ssl/certs/ca-certificates.crt:z,ro" -e SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt \ + -e PUSHSINK_BASE=https://pushsink.test -e PUSHSINK_INSTANCE=https://privapub.test \ + mcr.microsoft.com/dotnet/runtime-deps:10.0 /app/pushsink >/dev/null + for _ in $(seq 1 60); do + site pushsink.test -s -o /dev/null -w '%{http_code}' https://pushsink.test:6443/health 2>/dev/null | grep -q 200 && break + sleep 1 + done + echo "pushsink: https://pushsink.test:6443" +} diff --git a/tools/pasture/pushsink/pushsink.cs b/tools/pasture/pushsink/pushsink.cs new file mode 100644 index 0000000..307ef6c --- /dev/null +++ b/tools/pasture/pushsink/pushsink.cs @@ -0,0 +1,465 @@ +#:sdk Microsoft.NET.Sdk.Web +#:property PublishAot=false +#:property InvariantGlobalization=true +#:property Nullable=disable + +// pushsink: a Web Push service for the pasture (tools/pasture/scenarios/push.sh), behind Caddy as pushsink.test, which +// takes PrivaPub's pushes as a browser's push service would and keeps what it took for the scenario to read. It makes the +// keys of each subscription itself, so it can read what PrivaPub encrypted to them, and it checks what a push service +// checks: the VAPID JWT (ES256 by the server's key, the push service's origin as aud, exp in the future and at most a day +// ahead, a mailto: or https: sub) and the encryption (RFC 8291 aes128gcm, or the aesgcm draft with Encryption and +// Crypto-Key). Its own code, not PrivaPub's, so a round trip proves the two agree. +// POST /subscriptions {"server_key": ""} (optional; without it the +// key is read from $PUSHSINK_INSTANCE/api/v2/instance): a new subscription, its +// keys made here: {"id", "endpoint", "keys": {"p256dh", "auth"}} +// GET /subscriptions/{id} the subscription, with what it answers next +// POST /subscriptions/{id}/answer {"status": 410, "times": 1}: the next push(es) are answered so, after being read +// GET /subscriptions/{id}/received every push it took or refused, oldest first: when, the status it answered, +// whether every check passed (ok) and the errors, the headers, the JWT's claims, +// the decrypted payload (parsed JSON) +// DELETE /subscriptions/{id}/received forgets them +// POST /push/{id} the push service: 201 when every check passes, 403 for the VAPID's failures, 400 +// for the rest, 404 for no such subscription, or the status asked for +// GET /health +// Everything is in memory: a new container starts empty. +using System.Buffers.Binary; +using System.Collections.Concurrent; +using System.Security.Cryptography; +using System.Text; +using System.Text.Json; +using System.Text.Json.Nodes; + +var publicBase = (Environment.GetEnvironmentVariable("PUSHSINK_BASE") ?? "https://pushsink.test").TrimEnd('/'); +var instance = (Environment.GetEnvironmentVariable("PUSHSINK_INSTANCE") ?? "https://privapub.test").TrimEnd('/'); +var listen = Environment.GetEnvironmentVariable("PUSHSINK_LISTEN") ?? "http://0.0.0.0:8080"; +var subscriptions = new ConcurrentDictionary(); +var http = new HttpClient { Timeout = TimeSpan.FromSeconds(10) }; + +var builder = WebApplication.CreateBuilder(); +builder.Logging.SetMinimumLevel(LogLevel.Warning); +builder.WebHost.UseUrls(listen); +builder.WebHost.ConfigureKestrel(k => k.Limits.MaxRequestBodySize = 64 * 1024); +var app = builder.Build(); + +app.MapGet("/health", () => Results.Json(new { ok = true, subscriptions = subscriptions.Count })); + +app.MapPost("/subscriptions", async (HttpContext context) => +{ + var request = context.Request.ContentLength > 0 ? await JsonNode.ParseAsync(context.Request.Body) : null; + var key = ECDiffieHellman.Create(ECCurve.NamedCurves.nistP256); + var subscription = new Subscription(Convert.ToHexStringLower(RandomNumberGenerator.GetBytes(12)), key, RandomNumberGenerator.GetBytes(16), + request?["server_key"]?.GetValue()); + subscriptions[subscription.Id] = subscription; + return Results.Json(subscription.View(publicBase)); +}); + +app.MapGet("/subscriptions/{id}", (string id) => + subscriptions.TryGetValue(id, out var subscription) ? Results.Json(subscription.View(publicBase)) : Results.NotFound()); + +app.MapPost("/subscriptions/{id}/answer", async (HttpContext context, string id) => +{ + if (!subscriptions.TryGetValue(id, out var subscription)) + return Results.NotFound(); + var request = await JsonNode.ParseAsync(context.Request.Body); + var status = request?["status"]?.GetValue() ?? 0; + if (status is < 200 or > 599) + return Results.BadRequest(new { error = "status: 200 to 599" }); + subscription.Answer(status, request?["times"]?.GetValue() ?? 1); + return Results.Json(subscription.View(publicBase)); +}); + +app.MapGet("/subscriptions/{id}/received", (string id) => + subscriptions.TryGetValue(id, out var subscription) ? Results.Text(subscription.Received().ToJsonString(), "application/json") : Results.NotFound()); + +app.MapDelete("/subscriptions/{id}/received", (string id) => +{ + if (!subscriptions.TryGetValue(id, out var subscription)) + return Results.NotFound(); + subscription.Forget(); + return Results.Json(new { }); +}); + +app.MapPost("/push/{id}", async (HttpContext context, string id) => +{ + if (!subscriptions.TryGetValue(id, out var subscription)) + return Results.NotFound(); + using var buffer = new MemoryStream(); + await context.Request.Body.CopyToAsync(buffer); + var check = new Check(context.Request, buffer.ToArray(), subscription, publicBase); + var serverKey = subscription.ServerKey ?? await InstanceKey(); + check.Run(serverKey); + var forced = subscription.NextAnswer(); + var status = forced ?? (check.Errors.Count == 0 ? 201 : check.VapidFailed ? 403 : 400); + subscription.Keep(check.Record(status)); + if (status == 201) + context.Response.Headers.Location = $"{publicBase}/messages/{Guid.NewGuid():N}"; + return Results.StatusCode(status); +}); + +app.Run(); +return 0; + +// the server's key as its instance API gives it (configuration.vapid.public_key), read once a minute at most +async Task InstanceKey() +{ + if (InstanceCache.Key is { } known && DateTime.UtcNow - InstanceCache.At < TimeSpan.FromMinutes(1)) + return known; + try + { + var document = JsonNode.Parse(await http.GetStringAsync(instance + "/api/v2/instance")); + InstanceCache.Key = document?["configuration"]?["vapid"]?["public_key"]?.GetValue(); + InstanceCache.At = DateTime.UtcNow; + } + catch (Exception) + { + InstanceCache.Key = null; + } + return InstanceCache.Key; +} + +static class InstanceCache +{ + public static string Key; + public static DateTime At; +} + +sealed class Subscription(string id, ECDiffieHellman key, byte[] auth, string serverKey) +{ + readonly Lock _gate = new(); + readonly List _received = new(); + int _answer; + int _answerTimes; + + public string Id => id; + public ECDiffieHellman Key => key; + public byte[] Auth => auth; + public string ServerKey => serverKey; + public byte[] PublicPoint + { + get + { + var parameters = key.ExportParameters(false); + return [0x04, .. parameters.Q.X, .. parameters.Q.Y]; + } + } + + public JsonObject View(string publicBase) + { + lock (_gate) + return new JsonObject + { + ["id"] = id, + ["endpoint"] = $"{publicBase}/push/{id}", + ["keys"] = new JsonObject { ["p256dh"] = B64.Url(PublicPoint), ["auth"] = B64.Url(auth) }, + ["server_key"] = serverKey, + ["next_answer"] = _answerTimes > 0 ? _answer : null, + ["next_answer_times"] = _answerTimes, + ["received"] = _received.Count + }; + } + + public void Answer(int status, int times) + { + lock (_gate) + { + _answer = status; + _answerTimes = Math.Max(1, times); + } + } + + public int? NextAnswer() + { + lock (_gate) + { + if (_answerTimes <= 0) + return null; + _answerTimes--; + return _answer; + } + } + + public void Keep(JsonObject record) + { + lock (_gate) + _received.Add(record); + } + + public JsonArray Received() + { + lock (_gate) + return new JsonArray(_received.Select(r => (JsonNode)r.DeepClone()).ToArray()); + } + + public void Forget() + { + lock (_gate) + _received.Clear(); + } +} + +// what a push service checks of one push, and what the sink keeps of it +sealed class Check(HttpRequest request, byte[] body, Subscription subscription, string publicBase) +{ + public List Errors { get; } = new(); + public bool VapidFailed { get; private set; } + JsonObject _claims; + string _scheme; + string _k; + JsonNode _payload; + string _encoding; + + public void Run(string expectedServerKey) + { + _encoding = request.Headers.ContentEncoding.ToString().Trim().ToLowerInvariant(); + if (_encoding is not ("aes128gcm" or "aesgcm")) + Errors.Add($"Content-Encoding is '{_encoding}', neither aes128gcm nor aesgcm"); + if (!int.TryParse(request.Headers["TTL"].ToString(), out var ttl) || ttl < 0) + Errors.Add("no TTL header (RFC 8030 5.2)"); + Vapid(expectedServerKey); + try + { + var plaintext = _encoding switch + { + "aes128gcm" => Decrypt.Aes128Gcm(body, subscription.Key, subscription.PublicPoint, subscription.Auth), + "aesgcm" => Decrypt.AesGcm(body, Param(request.Headers["Encryption"].ToString(), "salt"), Param(request.Headers["Crypto-Key"].ToString(), "dh"), + subscription.Key, subscription.PublicPoint, subscription.Auth), + _ => null + }; + if (plaintext != null) + _payload = JsonNode.Parse(plaintext); + } + catch (Exception ex) when (ex is CryptographicException or FormatException or JsonException or ArgumentException or IndexOutOfRangeException) + { + Errors.Add("the body could not be decrypted: " + ex.Message); + } + } + + // RFC 8292 3: Authorization: vapid t=, k=; the draft's: Authorization: WebPush and Crypto-Key's p256ecdsa + void Vapid(string expectedServerKey) + { + var authorization = request.Headers.Authorization.ToString().Trim(); + var space = authorization.IndexOf(' '); + _scheme = space > 0 ? authorization[..space] : authorization; + var rest = space > 0 ? authorization[(space + 1)..].Trim() : string.Empty; + string jwt; + if (_scheme.Equals("vapid", StringComparison.OrdinalIgnoreCase)) + { + jwt = Param(rest, "t"); + _k = Param(rest, "k"); + } + else if (_scheme.Equals("WebPush", StringComparison.OrdinalIgnoreCase)) + { + jwt = rest; + _k = Param(request.Headers["Crypto-Key"].ToString(), "p256ecdsa"); + } + else + { + Fail($"Authorization's scheme is '{_scheme}', neither vapid nor WebPush"); + return; + } + if (_encoding == "aes128gcm" && !_scheme.Equals("vapid", StringComparison.OrdinalIgnoreCase)) + Fail("an aes128gcm push carries RFC 8292's vapid authorization"); + var key = B64.Decode(_k); + if (key is not { Length: 65 } || key[0] != 0x04) + { + Fail("the VAPID key (k or p256ecdsa) is no uncompressed P-256 point"); + return; + } + if (string.IsNullOrEmpty(expectedServerKey)) + Fail("the server's key is unknown (no server_key given, and the instance API did not say)"); + else if (!B64.Decode(expectedServerKey).AsSpan().SequenceEqual(key)) + Fail("the VAPID key is not the server's (the instance API's vapid.public_key)"); + var parts = jwt?.Split('.') ?? []; + if (parts.Length != 3) + { + Fail("the VAPID token is no JWT"); + return; + } + JsonNode header; + try + { + header = JsonNode.Parse(B64.Decode(parts[0])); + _claims = JsonNode.Parse(B64.Decode(parts[1])) as JsonObject; + } + catch (Exception) + { + Fail("the JWT's header or claims are not JSON"); + return; + } + if (header?["alg"]?.GetValue() != "ES256") + Fail($"the JWT's alg is '{header?["alg"]}', not ES256"); + using (var verifier = ECDsa.Create(new ECParameters { Curve = ECCurve.NamedCurves.nistP256, Q = new ECPoint { X = key[1..33], Y = key[33..] } })) + { + var signature = B64.Decode(parts[2]); + if (signature is not { Length: 64 } || !verifier.VerifyData(Encoding.ASCII.GetBytes(parts[0] + "." + parts[1]), signature, HashAlgorithmName.SHA256, + DSASignatureFormat.IeeeP1363FixedFieldConcatenation)) + Fail("the JWT's ES256 signature does not verify with the VAPID key"); + } + var audience = new Uri($"{publicBase}/push/{subscription.Id}").GetLeftPart(UriPartial.Authority); + if (_claims?["aud"]?.GetValue() != audience) + Fail($"the JWT's aud is '{_claims?["aud"]}', not the push service's origin {audience}"); + var now = DateTimeOffset.UtcNow.ToUnixTimeSeconds(); + var exp = _claims?["exp"]?.GetValue() ?? 0; + if (exp <= now) + Fail("the JWT has expired (exp)"); + else if (exp > now + 24 * 3600 + 60) + Fail("the JWT's exp is more than 24 hours ahead (RFC 8292 2)"); + var sub = _claims?["sub"]?.GetValue() ?? string.Empty; + if (!sub.StartsWith("mailto:", StringComparison.Ordinal) && !sub.StartsWith("https:", StringComparison.Ordinal)) + Fail($"the JWT's sub is '{sub}', neither mailto: nor https:"); + } + + void Fail(string error) + { + VapidFailed = true; + Errors.Add(error); + } + + // a parameter of a header written a=b, c=d or a=b;c=d (Crypto-Key may hold several) + static string Param(string header, string name) + { + foreach (var part in (header ?? string.Empty).Split([',', ';'], StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries)) + { + var equals = part.IndexOf('='); + if (equals > 0 && part[..equals].Trim().Equals(name, StringComparison.OrdinalIgnoreCase)) + return part[(equals + 1)..].Trim().Trim('"'); + } + return null; + } + + public JsonObject Record(int status) => new() + { + ["at"] = DateTime.UtcNow.ToString("O"), + ["status"] = status, + ["ok"] = Errors.Count == 0, + ["errors"] = new JsonArray(Errors.Select(e => (JsonNode)e).ToArray()), + ["encoding"] = _encoding, + ["headers"] = new JsonObject + { + ["ttl"] = request.Headers["TTL"].ToString(), + ["urgency"] = request.Headers["Urgency"].ToString(), + ["content_type"] = request.ContentType, + ["content_encoding"] = request.Headers.ContentEncoding.ToString(), + ["authorization_scheme"] = _scheme, + ["encryption"] = request.Headers["Encryption"].ToString(), + ["crypto_key"] = request.Headers["Crypto-Key"].ToString(), + ["user_agent"] = request.Headers.UserAgent.ToString() + }, + ["vapid"] = new JsonObject + { + ["k"] = _k, + ["aud"] = _claims?["aud"]?.DeepClone(), + ["exp"] = _claims?["exp"]?.DeepClone(), + ["sub"] = _claims?["sub"]?.DeepClone() + }, + ["bytes"] = body.Length, + ["payload"] = _payload?.DeepClone() + }; +} + +// the client's side of RFC 8291 and of the aesgcm draft +static class Decrypt +{ + public static byte[] Aes128Gcm(byte[] body, ECDiffieHellman client, byte[] clientPublic, byte[] auth) + { + // RFC 8188 2.1: salt (16), record size (4), key id length (1), key id: the sender's public key + var salt = body[..16]; + var recordSize = (int)BinaryPrimitives.ReadUInt32BigEndian(body.AsSpan(16, 4)); + var idLength = body[20]; + var serverPublic = body[21..(21 + idLength)]; + if (recordSize < 18) + throw new CryptographicException($"record size {recordSize} is below 18"); + var shared = Agree(client, serverPublic); + var keyInfo = Concat(Encoding.ASCII.GetBytes("WebPush: info\0"), clientPublic, serverPublic); + var ikm = HKDF.Expand(HashAlgorithmName.SHA256, HKDF.Extract(HashAlgorithmName.SHA256, shared, auth), 32, keyInfo); + var prk = HKDF.Extract(HashAlgorithmName.SHA256, ikm, salt); + var cek = HKDF.Expand(HashAlgorithmName.SHA256, prk, 16, Encoding.ASCII.GetBytes("Content-Encoding: aes128gcm\0")); + var nonce = HKDF.Expand(HashAlgorithmName.SHA256, prk, 12, Encoding.ASCII.GetBytes("Content-Encoding: nonce\0")); + var content = body[(21 + idLength)..]; + var plaintext = new List(); + var records = (content.Length + recordSize - 1) / recordSize; + for (var i = 0; i < records; i++) + { + var record = content[(i * recordSize)..Math.Min(content.Length, (i + 1) * recordSize)]; + var recordNonce = (byte[])nonce.Clone(); + for (var b = 0; b < 8; b++)//the record's sequence number, XORed into the nonce's last bytes + recordNonce[11 - b] ^= (byte)((long)i >> (8 * b)); + var opened = Open(cek, recordNonce, record); + var end = opened.Length - 1; + while (end >= 0 && opened[end] == 0) + end--; + var last = i == records - 1; + if (end < 0 || opened[end] != (last ? 0x02 : 0x01)) + throw new CryptographicException("a record's delimiter is wrong"); + plaintext.AddRange(opened[..end]); + } + return plaintext.ToArray(); + } + + public static byte[] AesGcm(byte[] body, string salt, string serverKey, ECDiffieHellman client, byte[] clientPublic, byte[] auth) + { + var saltBytes = B64.Decode(salt) ?? throw new FormatException("no salt in the Encryption header"); + var serverPublic = B64.Decode(serverKey) ?? throw new FormatException("no dh in the Crypto-Key header"); + var shared = Agree(client, serverPublic); + var ikm = HKDF.Expand(HashAlgorithmName.SHA256, HKDF.Extract(HashAlgorithmName.SHA256, shared, auth), 32, Encoding.ASCII.GetBytes("Content-Encoding: auth\0")); + var context = Concat(Encoding.ASCII.GetBytes("P-256\0"), Length16(clientPublic), clientPublic, Length16(serverPublic), serverPublic); + var prk = HKDF.Extract(HashAlgorithmName.SHA256, ikm, saltBytes); + var cek = HKDF.Expand(HashAlgorithmName.SHA256, prk, 16, Concat(Encoding.ASCII.GetBytes("Content-Encoding: aesgcm\0"), context)); + var nonce = HKDF.Expand(HashAlgorithmName.SHA256, prk, 12, Concat(Encoding.ASCII.GetBytes("Content-Encoding: nonce\0"), context)); + var opened = Open(cek, nonce, body); + var padding = BinaryPrimitives.ReadUInt16BigEndian(opened); + return opened[(2 + padding)..]; + } + + static byte[] Agree(ECDiffieHellman client, byte[] serverPublic) + { + if (serverPublic is not { Length: 65 } || serverPublic[0] != 0x04) + throw new CryptographicException("the sender's key is no uncompressed P-256 point"); + using var server = ECDiffieHellman.Create(new ECParameters + { + Curve = ECCurve.NamedCurves.nistP256, + Q = new ECPoint { X = serverPublic[1..33], Y = serverPublic[33..] } + }); + return client.DeriveRawSecretAgreement(server.PublicKey); + } + + static byte[] Open(byte[] key, byte[] nonce, byte[] sealedRecord) + { + if (sealedRecord.Length < 16) + throw new CryptographicException("a record shorter than its tag"); + var plaintext = new byte[sealedRecord.Length - 16]; + using var aes = new AesGcm(key, 16); + aes.Decrypt(nonce, sealedRecord.AsSpan(0, plaintext.Length), sealedRecord.AsSpan(plaintext.Length), plaintext); + return plaintext; + } + + static byte[] Length16(byte[] value) + { + var length = new byte[2]; + BinaryPrimitives.WriteUInt16BigEndian(length, (ushort)value.Length); + return length; + } + + static byte[] Concat(params byte[][] parts) => parts.SelectMany(p => p).ToArray(); +} + +static class B64 +{ + public static string Url(byte[] bytes) => Convert.ToBase64String(bytes).TrimEnd('=').Replace('+', '-').Replace('/', '_'); + + public static byte[] Decode(string text) + { + if (string.IsNullOrWhiteSpace(text)) + return null; + var normal = text.Trim().TrimEnd('=').Replace('-', '+').Replace('_', '/'); + normal = normal.PadRight(normal.Length + (4 - normal.Length % 4) % 4, '='); + try + { + return Convert.FromBase64String(normal); + } + catch (FormatException) + { + return null; + } + } +} diff --git a/tools/pasture/scenarios/push.sh b/tools/pasture/scenarios/push.sh new file mode 100644 index 0000000..8e662d6 --- /dev/null +++ b/tools/pasture/scenarios/push.sh @@ -0,0 +1,168 @@ +# Web Push (completion program R9), with Mastodon and the pushsink. push_alice subscribes with keys the sink made for +# her: a mention from mastouser reaches the sink encrypted (aes128gcm), signed by the server's VAPID key for the sink's +# origin, and decrypts to Mastodon's payload with notification_type mention and her own access token. Turning the mention +# alert off stops it while a favourite still comes; policy=followed stops mastouser's mention until she follows him; +# standard=false is the aesgcm draft and decrypts too, and a new subscription replaces the old; push_bob, a sibling persona +# of the same login, never gets hers, nor she his; DELETE stops the pushes; the sink answering 410 removes the +# subscription; a revoked token takes its subscription with it. Needs the mastodon and pushsink peers (batch A); run +# after featured. +M=https://mastodon.test:6443 +mcurl() { curl -sk --resolve mastodon.test:6443:127.0.0.1 "$@"; } +PS=https://pushsink.test:6443 +scurl() { curl -sk --resolve pushsink.test:6443:127.0.0.1 "$@"; } +. "$here/peers/mastodon.sh" +code() { curl -s -o /dev/null -w '%{http_code}' "$@"; } +# sink_new: a subscription of the sink's own, its keys made there, told the server's key to check the VAPID JWT with +sink_new() { scurl -X POST "$PS/subscriptions" -H 'Content-Type: application/json' -d "{\"server_key\":\"$server_key\"}"; } +sink_count() { scurl "$PS/subscriptions/$1" | j "print(d['received'])"; } +# sink_last +sink_last() { scurl "$PS/subscriptions/$1/received" | j "$2"; } +sink_waits() { local sid=$1 want=$2; until_true 30 '[ "$(sink_count "$sid")" -ge "$want" ]'; } +# subscribe [policy=]: Mastodon's form +subscribe() { + local auth=$1 sink=$2 standard=$3 args=() a + shift 3 + for a in "$@"; do + case $a in policy=*) args+=(-d "data[policy]=${a#policy=}") ;; *) args+=(-d "data[alerts][$a]=true") ;; esac + done + curl -s -X POST -H "$auth" "$P/api/v1/push/subscription" \ + --data-urlencode "subscription[endpoint]=$(echo "$sink" | j "print(d['endpoint'])")" \ + --data-urlencode "subscription[keys][p256dh]=$(echo "$sink" | j "print(d['keys']['p256dh'])")" \ + --data-urlencode "subscription[keys][auth]=$(echo "$sink" | j "print(d['keys']['auth'])")" \ + -d "subscription[standard]=$standard" "${args[@]}" +} +# m_mention : mastouser's public post naming a persona here +m_mention() { mcurl -o /dev/null -X POST -H "$MH" "$M/api/v1/statuses" --data-urlencode "status=@$1@privapub.test $2" -d visibility=public; } +# notified : the persona holds a mention with these words (it arrived, pushed or not) +notified() { curl -s -H "$1" "$P/api/v1/notifications?types[]=mention&limit=30" | j "print(any('$2' in (n.get('status') or {}).get('content', '') for n in d))"; } +# arrives_unpushed : the mention arrived and the sink got nothing +arrives_unpushed() { + local auth=$1 words=$2 sid=$3 had=$4 + until_true 30 '[ "$(notified "$auth" "$words")" = True ]' || return 1 + sleep 6 + [ "$(sink_count "$sid")" = "$had" ] +} + +echo "push" +run=$(date +%s) +AT=$(privapub_token push_alice "read write follow push"); AH="Authorization: Bearer $AT" +BT=$(privapub_token push_bob "read write follow push"); BH="Authorization: Bearer $BT" +MT=$(mastodon_token); MH="Authorization: Bearer $MT" +[ -n "$AT" ] && [ -n "$BT" ] && [ -n "$MT" ] && ok "tokens for push_alice, push_bob (her sibling, the same login) and mastouser" \ + || { ko "a token is missing"; return 1; } + +echo " the server's key" +server_key=$(curl -s "$P/api/v2/instance" | j "print(d['configuration']['vapid']['public_key'])") +[ "$(python3 -c "import base64,sys; k=base64.urlsafe_b64decode(sys.argv[1]); print(len(k) == 65 and k[0] == 4)" "$server_key" 2>/dev/null)" = True ] \ + && ok "the instance API gives the VAPID key (an uncompressed P-256 point)" || ko "configuration.vapid: $server_key" +[ "$(curl -s -X POST "$P/api/v1/apps" -d client_name=push-check -d redirect_uris=urn:ietf:wg:oauth:2.0:oob -d scopes=push | j "print(d['vapid_key'])")" = "$server_key" ] \ + && ok "an app's vapid_key is the same key" || ko "the app's vapid_key differs" +[ "$(curl -s "$P/api/v1/instance" | j "print(d['version'].startswith('4.5.0 (compatible; PrivaPub'))")" = True ] \ + && [ "$(curl -s "$P/api/v2/instance" | j "print(d['version'].startswith('4.5.0 (compatible; PrivaPub') and d['api_versions']['mastodon'] == 7)")" = True ] \ + && ok "both instance APIs say 4.5.0 (compatible; PrivaPub ...)" || ko "the version: $(curl -s "$P/api/v2/instance" | j "print(d['version'])")" + +echo " a mention, pushed" +m_on_p=$(curl -s -H "$AH" "$P/api/v2/search?q=mastouser@mastodon.test&resolve=true&type=accounts" | j "print(d['accounts'][0]['id'])") +# (a run before may have left the follow the policy check needs gone) +curl -s -o /dev/null -X POST -H "$AH" "$P/api/v1/accounts/$m_on_p/unfollow" +for persona in push_alice push_bob; do + mcurl -o /dev/null -H "$MH" "$M/api/v2/search?q=@$persona@privapub.test&resolve=true&type=accounts" +done +sa=$(sink_new); sa_id=$(echo "$sa" | j "print(d['id'])") +made=$(subscribe "$AH" "$sa" true mention favourite policy=all) +[ "$(echo "$made" | j "print(d['standard'] is True and d['alerts']['mention'] is True and d['alerts']['reblog'] is False and d['policy'] == 'all' and d['server_key'] == '$server_key' and d['endpoint'].startswith('https://pushsink.test/push/'))")" = True ] \ + && ok "push_alice subscribes with the sink's keys (standard)" || ko "the subscription: $(echo "$made" | head -c 400)" +[ "$(curl -s -H "$AH" "$P/api/v1/push/subscription" | j "print(d['id'])")" = "$(echo "$made" | j "print(d['id'])")" ] \ + && ok "GET answers it" || ko "GET: $(curl -s -H "$AH" "$P/api/v1/push/subscription" | head -c 300)" +[ "$(code -H "$BH" "$P/api/v1/push/subscription")" = 404 ] && ok "push_bob's token has none" || ko "push_bob's token sees a subscription" +m_mention push_alice "first push $run" +sink_waits "$sa_id" 1 \ + && [ "$(sink_last "$sa_id" "r = d[-1]; p = r['payload'] or {}; print(r['ok'] and r['status'] == 201 and r['encoding'] == 'aes128gcm' and p.get('notification_type') == 'mention' and 'first push $run' in p.get('body', ''))")" = True ] \ + && ok "mastouser's mention reaches the sink, aes128gcm, and decrypts to a mention" || ko "the push: $(scurl "$PS/subscriptions/$sa_id/received" | head -c 800)" +[ "$(sink_last "$sa_id" "print(d[-1]['payload']['access_token'] == '$AT')")" = True ] && ok "it carries push_alice's own access token" \ + || ko "the push's access_token is not hers" +[ "$(sink_last "$sa_id" "r = d[-1]; v = r['vapid']; print(v['aud'] == 'https://pushsink.test' and v['sub'].startswith(('mailto:', 'https:')) and 'push_alice' not in v['sub'] and r['headers']['authorization_scheme'] == 'vapid' and int(r['headers']['ttl']) > 0 and r['headers']['urgency'] == 'normal')")" = True ] \ + && ok "signed for the sink's origin by the server, TTL and Urgency set, no persona in the sub" || ko "the VAPID claims: $(sink_last "$sa_id" "print(d[-1]['vapid'], d[-1]['headers'])")" +pushed_id=$(sink_last "$sa_id" "print(d[-1]['payload']['notification_id'])") +[ "$(curl -s -H "$AH" "$P/api/v1/notifications/$pushed_id" | j "print(d['type'] == 'mention' and 'first push $run' in d['status']['content'])")" = True ] \ + && ok "its notification_id is her notification" || ko "notification $pushed_id: $(curl -s -H "$AH" "$P/api/v1/notifications/$pushed_id" | head -c 200)" +[ "$(sink_last "$sa_id" "p = d[-1]['payload']; print(p['icon'].startswith('https://privapub.test/') and 'mastouser' in p['title'] and p['preferred_locale'] != '')")" = True ] \ + && ok "its icon goes through PrivaPub, its title names mastouser" || ko "icon and title: $(sink_last "$sa_id" "print(d[-1]['payload'])")" + +echo " alerts and policy" +changed=$(curl -s -X PUT -H "$AH" "$P/api/v1/push/subscription" -d "data[alerts][favourite]=true") +[ "$(echo "$changed" | j "print(d['alerts']['mention'] is False and d['alerts']['favourite'] is True and d['policy'] == 'all')")" = True ] \ + && ok "PUT turns the mention alert off" || ko "PUT: $(echo "$changed" | head -c 300)" +before=$(sink_count "$sa_id") +m_mention push_alice "unwanted $run" +arrives_unpushed "$AH" "unwanted $run" "$sa_id" "$before" && ok "with the alert off, the mention arrives unpushed" || ko "a mention was pushed with its alert off" +alice_post=$(curl -s -X POST -H "$AH" "$P/api/v1/statuses" --data-urlencode "status=push_alice posts $run" -d visibility=public | j "print(d['uri'])") +m_post=$(mcurl -H "$MH" "$M/api/v2/search?q=$alice_post&resolve=true&type=statuses" | j "print(d['statuses'][0]['id'])") +mcurl -o /dev/null -X POST -H "$MH" "$M/api/v1/statuses/$m_post/favourite" +sink_waits "$sa_id" $((before + 1)) && [ "$(sink_last "$sa_id" "print(d[-1]['ok'] and d[-1]['payload']['notification_type'])")" = favourite ] \ + && ok "his favourite of her post is pushed" || ko "no favourite push: $(sink_last "$sa_id" "print(d[-1])")" +curl -s -o /dev/null -X PUT -H "$AH" "$P/api/v1/push/subscription" -d "data[alerts][mention]=true" -d "data[policy]=followed" +before=$(sink_count "$sa_id") +m_mention push_alice "a stranger $run" +arrives_unpushed "$AH" "a stranger $run" "$sa_id" "$before" && ok "policy=followed: mastouser, whom she does not follow, is not pushed" \ + || ko "a non-followed sender was pushed" +curl -s -o /dev/null -X POST -H "$AH" "$P/api/v1/accounts/$m_on_p/follow" +until_true 30 '[ "$(curl -s -H "$AH" "$P/api/v1/accounts/relationships?id[]=$m_on_p" | j "print(d[0][\"following\"])")" = True ]' \ + && ok "push_alice follows mastouser" || ko "her follow was not accepted" +m_mention push_alice "a friend now $run" +sink_waits "$sa_id" $((before + 1)) && [ "$(sink_last "$sa_id" "print('a friend now $run' in d[-1]['payload']['body'])")" = True ] \ + && ok "once followed, his mention is pushed" || ko "a followed sender was not pushed" + +echo " the aesgcm draft" +sl=$(sink_new); sl_id=$(echo "$sl" | j "print(d['id'])") +[ "$(subscribe "$AH" "$sl" false mention | j "print(d['standard'])")" = False ] && ok "push_alice subscribes again, standard=false" || ko "the legacy subscription" +before=$(sink_count "$sa_id") +m_mention push_alice "the draft $run" +sink_waits "$sl_id" 1 \ + && [ "$(sink_last "$sl_id" "r = d[-1]; print(r['ok'] and r['encoding'] == 'aesgcm' and r['headers']['authorization_scheme'] == 'WebPush' and 'p256ecdsa=' in r['headers']['crypto_key'] and r['headers']['encryption'].startswith('salt=') and r['payload']['notification_type'] == 'mention' and r['payload']['access_token'] == '$AT')")" = True ] \ + && ok "aesgcm, with Encryption and Crypto-Key, decrypts to her mention" || ko "the aesgcm push: $(scurl "$PS/subscriptions/$sl_id/received" | head -c 800)" +sleep 3 +[ "$(sink_count "$sa_id")" = "$before" ] && ok "the subscription it replaced gets nothing" || ko "the replaced subscription is still pushed" + +echo " siblings" +sb=$(sink_new); sb_id=$(echo "$sb" | j "print(d['id'])") +subscribe "$BH" "$sb" true mention policy=all >/dev/null +alice_before=$(sink_count "$sl_id") +m_mention push_alice "for alice only $run" +sink_waits "$sl_id" $((alice_before + 1)); sleep 4 +[ "$(sink_count "$sb_id")" = 0 ] && ok "push_bob's subscription never gets push_alice's mention" || ko "a sibling's subscription got her push" +m_mention push_bob "for bob only $run" +sink_waits "$sb_id" 1 && [ "$(sink_last "$sb_id" "p = d[-1]['payload']; print(p['access_token'] == '$BT' and 'for bob only $run' in p['body'])")" = True ] \ + && ok "push_bob's own mention, with his own token" || ko "push_bob's push: $(scurl "$PS/subscriptions/$sb_id/received" | head -c 400)" +sleep 3 +[ "$(sink_count "$sl_id")" = "$((alice_before + 1))" ] && [ "$(sink_last "$sl_id" "print(all(r['payload']['access_token'] == '$AT' for r in d))")" = True ] \ + && ok "push_alice gets nothing of his, and only her token" || ko "push_alice's subscription got push_bob's" +[ "$(curl -s -X DELETE -H "$BH" "$P/api/v1/push/subscription")" = "{}" ] && ok "push_bob unsubscribes" || ko "push_bob's DELETE" + +echo " DELETE, 410, revocation" +[ "$(curl -s -X DELETE -H "$AH" "$P/api/v1/push/subscription")" = "{}" ] && [ "$(code -H "$AH" "$P/api/v1/push/subscription")" = 404 ] \ + && ok "DELETE: the subscription is gone" || ko "DELETE left it" +before=$(sink_count "$sl_id") +m_mention push_alice "after delete $run" +arrives_unpushed "$AH" "after delete $run" "$sl_id" "$before" && ok "nothing is pushed after DELETE" || ko "pushed after DELETE" +sg=$(sink_new); sg_id=$(echo "$sg" | j "print(d['id'])") +subscribe "$AH" "$sg" true mention >/dev/null +scurl -o /dev/null -X POST "$PS/subscriptions/$sg_id/answer" -H 'Content-Type: application/json' -d '{"status": 410}' +m_mention push_alice "gone $run" +sink_waits "$sg_id" 1 && until_true 15 '[ "$(code -H "$AH" "$P/api/v1/push/subscription")" = 404 ]' \ + && ok "the sink answering 410 removes the subscription" || ko "after a 410: $(curl -s -H "$AH" "$P/api/v1/push/subscription" | head -c 200)" +before=$(sink_count "$sg_id") +m_mention push_alice "still gone $run" +arrives_unpushed "$AH" "still gone $run" "$sg_id" "$before" && ok "and nothing more goes there" || ko "pushed after the 410" +oauth=$("$here/../smoke/oauth.sh" "$P" "$ROOT_USER" "$ROOT_PASS" push_alice "read write follow push") +RT=$(echo "$oauth" | cut -d' ' -f1); rcid=$(echo "$oauth" | cut -d' ' -f2); rcs=$(echo "$oauth" | cut -d' ' -f3) +sr=$(sink_new); sr_id=$(echo "$sr" | j "print(d['id'])") +subscribe "Authorization: Bearer $RT" "$sr" true mention >/dev/null +curl -s -o /dev/null -X POST "$P/oauth/revoke" -d "token=$RT" -d "client_id=$rcid" -d "client_secret=$rcs" +[ "$(code -H "Authorization: Bearer $RT" "$P/api/v1/push/subscription")" = 401 ] && ok "the revoked token is refused" || ko "the revoked token still works" +m_mention push_alice "revoked $run" +arrives_unpushed "$AH" "revoked $run" "$sr_id" 0 && ok "a revoked token's subscription is pushed nothing" || ko "pushed to a revoked token's subscription" +[ "$(podman exec pasture-mongo mongosh --quiet PrivaPub --eval 'db.PushSubscription.countDocuments()' 2>/dev/null | tail -1)" = 0 ] \ + && ok "no subscription is left" || ko "subscriptions left: $(podman exec pasture-mongo mongosh --quiet PrivaPub --eval 'db.PushSubscription.countDocuments()' 2>/dev/null | tail -1)" + +curl -s -o /dev/null -X POST -H "$AH" "$P/api/v1/accounts/$m_on_p/unfollow" diff --git a/tools/pasture/sweep.sh b/tools/pasture/sweep.sh index fb1dcb7..7c75b3d 100755 --- a/tools/pasture/sweep.sh +++ b/tools/pasture/sweep.sh @@ -6,11 +6,12 @@ # PrivaPub runs with SecureMode on, as in production, unless PRIVAPUB_ENV says otherwise; DOTNET and PASTURE_PORT reach # run.sh as they are. Each batch's output is kept in out/sweep-.log. # usage: tools/pasture/sweep.sh [A|B|C|D ...] (default: every batch, in that order) -# A gts mastodon misskey sharkey akkoma, with relay, aoderelay and mailpit; scenarios: those five, threads followsync +# A gts mastodon misskey sharkey akkoma, with relay, aoderelay, mailpit and pushsink; scenarios: those five, threads followsync # pins moves relay (relay's covers both relays) accounts (mailpit's) instance (the server's own settings: it changes # them all and puts them back) moderation (gts and mastodon; after instance, whose log check expects only the # administrator's acts) emoji (gts, mastodon, misskey and akkoma; after moderation) featured (mastodon: featured -# tags, notify, the direct stream, endorsements and notes per persona; after emoji) +# tags, notify, the direct stream, endorsements and notes per persona; after emoji) push (mastodon and pushsink: Web +# Push, encrypted both ways, alerts, policies, siblings, DELETE, 410 and revocation; after featured) # B lemmy lemmy19 piefed mbin nodebb # C peertube pixelfed vernissage wordpress ghost writefreely owncast castopod funkwhale # D friendica mobilizon gancio bookwyrm smithereen mitra snac ktistec forte hubzilla @@ -22,7 +23,7 @@ export PRIVAPUB_ENV=${PRIVAPUB_ENV:-Federation__SecureMode=true} batch_peers() { case "$1" in - A) echo "gts mastodon misskey sharkey akkoma relay aoderelay mailpit" ;; + A) echo "gts mastodon misskey sharkey akkoma relay aoderelay mailpit pushsink" ;; B) echo "lemmy lemmy19 piefed mbin nodebb" ;; C) echo "peertube pixelfed vernissage wordpress ghost writefreely owncast castopod funkwhale" ;; D) echo "friendica mobilizon gancio bookwyrm smithereen mitra snac ktistec forte hubzilla" ;; @@ -30,7 +31,7 @@ batch_peers() { } batch_scenarios() { case "$1" in - A) echo "gts mastodon misskey sharkey akkoma threads followsync pins moves relay accounts instance moderation emoji featured" ;; + A) echo "gts mastodon misskey sharkey akkoma threads followsync pins moves relay accounts instance moderation emoji featured push" ;; *) batch_peers "$1" ;; esac } diff --git a/tools/smoke/mastodon-api.sh b/tools/smoke/mastodon-api.sh index 03a3b17..e670b3f 100755 --- a/tools/smoke/mastodon-api.sh +++ b/tools/smoke/mastodon-api.sh @@ -6,12 +6,19 @@ BASE="${1:?base url}"; TOKEN="${2:-}" fail() { echo "::error::$*"; exit 1; } json() { python3 -c "import sys,json; d=json.load(sys.stdin); $1"; } +# the version clients branch on (Mastodon invariant 6): Mastodon 4.5's, then PrivaPub's release version=$(curl -fsS "$BASE/api/v1/instance" | json "print(d['version'])") || fail "instance v1" -curl -fsS "$BASE/api/v2/instance" | json "assert d['configuration']['statuses']['max_characters'] > 0" || fail "instance v2" +case "$version" in "4.5.0 (compatible; PrivaPub"*) ;; *) fail "instance v1 says version '$version', not 4.5.0 (compatible; PrivaPub ...)" ;; esac +v2=$(curl -fsS "$BASE/api/v2/instance") || fail "instance v2" +echo "$v2" | json "assert d['configuration']['statuses']['max_characters'] > 0" || fail "instance v2" +echo "$v2" | json "assert d['version'] == '$version' and d['api_versions']['mastodon'] == 7" || fail "instance v2's version or api_versions" +vapid=$(echo "$v2" | json "import base64; k = d['configuration']['vapid']['public_key']; assert len(base64.urlsafe_b64decode(k)) == 65; print(k)") \ + || fail "instance v2 gives no VAPID key (configuration.vapid.public_key)" curl -fsS "$BASE/.well-known/oauth-authorization-server" | json "assert d['token_endpoint'].endswith('/oauth/token')" || fail "oauth discovery" app=$(curl -fsS -X POST "$BASE/api/v1/apps" -d 'client_name=privapub-smoke&redirect_uris=urn:ietf:wg:oauth:2.0:oob&scopes=read') || fail "app registration" id=$(echo "$app" | json "print(d['client_id'])"); secret=$(echo "$app" | json "print(d['client_secret'])") +[ "$(echo "$app" | json "print(d['vapid_key'])")" = "$vapid" ] || fail "the app's vapid_key is not the instance's VAPID key" app_token=$(curl -fsS -X POST "$BASE/oauth/token" -d "grant_type=client_credentials&client_id=$id&client_secret=$secret&scope=read" | json "print(d['access_token'])") || fail "client credentials" curl -fsS -H "Authorization: Bearer $app_token" "$BASE/api/v1/apps/verify_credentials" | json "assert d['name'] == 'privapub-smoke'" || fail "app verify_credentials" code=$(curl -s -o /dev/null -w '%{http_code}' -H "Authorization: Bearer $app_token" "$BASE/api/v1/accounts/verify_credentials")