Compare commits

..
9 Commits
Author SHA1 Message Date
thepraandClaude Opus 5.5 125a49a1a0 T4: nothing answers 500
Build / Build (push) Successful in 2m24s
Deploy / privapub.thepra.dev (push) Successful in 2m39s
NeverFiveHundredTests walks every endpoint the server maps and fills each route parameter
with junk ('x', zeroed and random ObjectIds, a dot-dot segment, 5000 characters). It calls
each one anonymously, as a persona, and with a junk token, and asserts that nothing
answers 5xx or throws, and that every /api error carries a Mastodon error body.

It found two bugs:
- A junk 'Authorization: Bearer' on any non-/api route, anonymous ones included
  (/build.json, /peasants/*, inboxes), broke the response. JwtEvents.AuthenticationFailed
  wrote a body during authentication and the endpoint then wrote its own. The 401 body now
  comes from the Challenge event, which runs only when an endpoint needs a user.
- /api answered 401, 404 and 429 with no body. UseMastodonErrorBodies gives any /api error
  that leaves without a body Mastodon's {"error": ...}.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELjqpznMFMNrJoJUj6K5p2
2026-10-03 10:46:20 +02:00
thepraandClaude Opus 5.5 5e34517e73 T3: the whole server under test
PrivaPubHost is a WebApplicationFactory<Program> on the fixture's database, configured only
through UseSetting (visible before Build, unlike ConfigureAppConfiguration). It drops the
background workers so tests run the jobs they queue (Jobs.Run, RunInbox), gives each client
its own address for the rate limiter, and has a SecureMode variant. Accounts signs up roots,
adds personas and gets Mastodon tokens through the real /oauth code flow. RemoteActor signs
HttpRequestMessages for the real /peasants routes; Peer records bodies and headers and
serves files with ranges and text pages.

Program registers the Guid serializer with TryRegisterSerializer, so a second host in one
process starts; the fixture runs the migrations in production's order before any test.

HostBootTests: the host shares the fixture database; the harness handles exactly the
activities the server registers; every job kind has one handler; every controller and page
model can be made; the service graph validates with ValidateOnBuild and ValidateScopes;
Swagger is 404 outside Development; a persona's token never names its root; a signed DM
through the real /mouth route is queued, processed and stored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELjqpznMFMNrJoJUj6K5p2
2026-10-03 10:43:03 +02:00
thepraandClaude Opus 5.5 85fdff606d T2: CI runs every test against a throwaway mongod
tools/ci/with-test-mongod.sh starts mongod (or podman's mongo:8) on a random localhost port
with a temporary data directory, runs the command, and removes both. build.yml and deploy.yml
test through it, so the ~85 integration tests stop being skipped in CI. MongoFixture refuses
production's port and data directory, and in CI anything but the wrapper's mongod; with
PRIVAPUB_TEST_REQUIRE_MONGOD=1 a missing mongod fails instead of skipping.

The deploy now passes the PRIVAPUB_SMOKE_TOKEN secret to the Mastodon smoke check, so its
signed-in half runs once the owner creates the persona and the secret.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELjqpznMFMNrJoJUj6K5p2
2026-10-03 10:37:42 +02:00
thepraandClaude Opus 5.5 00b2685cf4 T1: tests stop sharing state they don't own
- JobQueue takes an optional scope, so a test's worker leases and reaps only its own jobs.
- The dead-host delivery test runs alone (Exclusive), on its own jobs, and cleans up the
  breaker rows it trips; the breaker has tests of its own on unique hosts.
- Index and migration tests run alone: they drop indexes and rewrite every post.
- DomainBlocks.Load replaces reflection and a database-wide block in tests.
- Harness.Outgoing sees only deliveries queued since the harness started: Peer ports are
  reused within a run, which made the circle test flaky.
- Two pure-logic tests leave Mongo-gated classes, so CI runs them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELjqpznMFMNrJoJUj6K5p2
2026-10-03 10:36:15 +02:00
thepraandClaude Opus 5.5 7c6fe80f1b docs: owner decisions on statistics, and the sweep and statistics plan
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELjqpznMFMNrJoJUj6K5p2
2026-10-03 10:30:46 +02:00
thepraandClaude Opus 5.5 e6729c77e7 P6 done: personas can be quoted, under the owner's default of anyone, automatically
Build / Build (push) Successful in 1m7s
Deploy / privapub.thepra.dev (push) Successful in 1m15s
- Our public and unlisted posts state interactionPolicy.canQuote. The policy comes from the post, then the persona
  (`source[quote_policy]`: public, followers or nobody; default public as the owner chose), and is always nobody for
  followers-only posts and DMs.
- QuoteRequests are answered with Accept{result} naming a parrot-licence at /peasants/{name}/parrot-licences/{id}
  (the route name the owner chose), or with Reject. Followers-only checks that the requester really follows.
- A licence is a QuoteAuthorization naming both posts; revoking it (POST /api/v1/statuses/:id/quotes/:quoting_id/revoke)
  marks it 410, sends Delete{licence} to the quoter and the persona's followers, and revokes our own copy of the quote.
- A quote that arrives with one of our licences is accepted only if that licence is ours, unrevoked and names exactly
  that quoting post. One persona quoting another gets a licence too.
- Mastodon API: quote_approval for our posts (automatic, followers, current_user), `quote_approval_policy` when posting,
  PUT /api/v1/statuses/:id/interaction_policy, `source.quote_policy`.

Checked live: GoToSocial still accepts our posts with the policy stated, and leaves likes, replies and boosts open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012CzABvBkbcFqoHdmi8b9WB
2026-10-01 19:53:13 +02:00
thepraandClaude Opus 5.5 f9658a8e7a P6: remote video and audio play through the proxy
Build / Build (push) Successful in 37s
Deploy / privapub.thepra.dev (push) Successful in 57s
- The media proxy streams ranged requests from the origin (passing the range on, never caching), downloads and caches
  whole files otherwise, and serves cached files with range support. A PeerTube video is never fetched whole for one
  viewer, and clients still never contact the remote host.
- PeerTube's fragmented MP4 files inside an HLS entry are read as variants, so HLS-only instances play too.
- A remote Video or Audio post becomes one playable Mastodon attachment: the best MP4 up to 720p that carries both
  sound and picture, with its poster and duration; the card is kept only when nothing is playable.
- nginx: /media/proxy/ with proxy_buffering off and a 600 s read timeout (applied on the box, with a backup).

Checked live: a GoToSocial image through the proxy answers 206 with exactly the asked range when streamed, 200 when
cached, and 206 with the right Content-Range from the cache.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012CzABvBkbcFqoHdmi8b9WB
2026-10-01 18:59:14 +02:00
thepraandClaude Opus 5.5 6f1ab0073c P6: quote posts, received and sent (FEP-044f and the older keys)
Build / Build (push) Successful in 35s
Deploy / privapub.thepra.dev (push) Successful in 54s
- Received quotes: read from `quote`, `quoteUrl`, `quoteUri`, `_misskey_quote` or a FEP-e232 Link tag; the quoted post
  is fetched once; a quoteAuthorization stamp is verified field by field on the quoted author's origin; a consent
  quote without a stamp is pending; an older-key quote of a public post is shown; Delete of a stamp revokes. Counts
  and a `quote` notification follow the accepted state. The `quote-inline` fallback survives sanitising and is removed
  from content when the real quote is shown.
- Personas quote through `quoted_status_id`: posts that state a quote policy get a QuoteRequest and stay pending until
  an Accept brings a stamp we can verify, then an Update adds quoteAuthorization; posts that state none are quoted
  the older way, without `quote`; another persona's posts cannot be quoted yet (we issue no stamps). Quoting posts
  are delivered to the quoted author too.
- Mastodon API: Status.quote (with the quoted status one level deep), quotes_count, quote_approval from the remote
  policy, GET /api/v1/statuses/:id/quotes, `quote` notifications, and api_versions.mastodon = 7.

Checked live: GoToSocial's author-only quote policy is respected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012CzABvBkbcFqoHdmi8b9WB
2026-10-01 18:53:28 +02:00
thepraandClaude Opus 5.5 001fdac3be P6: link previews read by the server, as the owner decided
Build / Build (push) Successful in 35s
Deploy / privapub.thepra.dev (push) Successful in 56s
- A public post that links to a page gets a preview card: from the post's own data first (FEP-8967 Link preview, the
  object's image, title and summary); otherwise the server reads the page once, 0-60 s after the post arrives, never
  when someone reads it. OpenGraph and Twitter tags give title, description and image; one LinkPreview per address
  is cached for 7 days and shared by the whole server, so a fetch never points at a persona.
- Lemmy link posts, which carry no title or description, get them filled in.
- The page fetch uses the guarded client (public addresses, three redirects, HTML only, first 512 KB).
- Federation:FetchLinkPreviews switches page fetching off.
- Local public posts get cards too.

Checked live: a link to a GoToSocial profile page becomes a card with its title, description and proxied image.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012CzABvBkbcFqoHdmi8b9WB
2026-10-01 18:43:39 +02:00
70 changed files with 2675 additions and 209 deletions

No files matched your search

+2 -2
View File
@@ -15,5 +15,5 @@ jobs:
- name: Build
run: dotnet build PrivaPub.sln -c Release
- name: Test
run: dotnet test PrivaPub.sln -c Release --no-build
- name: Test (unit and integration, on a throwaway mongod)
run: tools/ci/with-test-mongod.sh dotnet test PrivaPub.sln -c Release --no-build
+6 -3
View File
@@ -28,8 +28,8 @@ jobs:
echo "BUILD_REF=${GITHUB_REF_NAME:-master}" >> "$GITHUB_ENV"
echo "BUILD_TIME=$(date -u +%Y-%m-%dT%H:%M:%SZ)" >> "$GITHUB_ENV"
- name: Test
run: dotnet test PrivaPub.sln -c Release
- name: Test (unit and integration, on a throwaway mongod)
run: tools/ci/with-test-mongod.sh dotnet test PrivaPub.sln -c Release
- name: Publish
run: |
@@ -90,6 +90,8 @@ jobs:
fi
- name: Verify what is being served
env:
SMOKE_TOKEN: ${{ secrets.PRIVAPUB_SMOKE_TOKEN }}
run: |
served=$(curl -fsS "$PUBLIC_URL/build.json" | python3 -c "import json,sys; print(json.load(sys.stdin).get('commit',''))")
[ "$served" = "$BUILD_COMMIT" ] || { echo "::error::served build is '$served', expected '$BUILD_COMMIT'"; exit 1; }
@@ -104,5 +106,6 @@ jobs:
code=$(curl -s -o /dev/null -w '%{http_code}' -X POST -H 'Content-Type: application/activity+json' \
--data '{"type":"Follow","actor":"https://example.org/users/x","object":"'"$PUBLIC_URL"'/peasants/privapub"}' "$PUBLIC_URL/human-centipede")
[ "$code" = "401" ] || { echo "::error::an unsigned inbox POST answered $code"; exit 1; }
tools/smoke/mastodon-api.sh "$PUBLIC_URL"
tools/smoke/mastodon-api.sh "$PUBLIC_URL" "$SMOKE_TOKEN"
[ -n "$SMOKE_TOKEN" ] || echo "::warning::PRIVAPUB_SMOKE_TOKEN is not set, so the signed-in API was not checked"
echo "::notice::serving $served"
+54 -5
View File
@@ -48,6 +48,7 @@ actor-scoped route gets a name in the same spirit, agreed with the owner, and a
| Notes | `/peasants/{name}/scribbles/{id}` |
| Activities | `/peasants/{name}/grunts/{id}` (`create-{postId}` resolves) |
| DM context | `/peasants/{name}/whispers/{id}` |
| Quote permission (FEP-044f `QuoteAuthorization`) | `/peasants/{name}/parrot-licences/{id}` |
| Token refresh | `/clientapi/user/sniff/again` |
Agreed for later phases: `/gossip`, `/drool`, `/echoes` (replies, likes, shares), `/trophies` (featured), `/tattoos`
@@ -123,6 +124,7 @@ dotnet build PrivaPub.sln -c Release
dotnet test PrivaPub.sln # unit tests; integration tests skip
PRIVAPUB_TEST_MONGOD=1 dotnet test PrivaPub.sln # all of them, against mongod on 127.0.0.1:27017
# (PRIVAPUB_TEST_MONGO overrides; a fresh database per run, dropped after)
tools/ci/with-test-mongod.sh dotnet test PrivaPub.sln # all of them, on a throwaway mongod, as CI runs them
cd PrivaPub && ASPNETCORE_ENVIRONMENT=Development \
Kestrel__Endpoints__Http__Url=http://127.0.0.1:6970 Kestrel__Endpoints__Http__Protocols=Http1AndHttp2 \
AppConfiguration__BackendBaseAddress=http://127.0.0.1:6970 MongoSettings__Database=PrivaPubTest \
@@ -197,7 +199,10 @@ cd /var/www/privapub.thepra.dev && sudo -u www-data ASPNETCORE_ENVIRONMENT=Produ
3. Read parameters through `Params` (query, form and JSON merged Rails-style), never MVC binding. A value type read
from a conditional must say `(int?)null`, not `default`: that bug once made every list one item long.
4. Answer with `Json(...)` (snake_case, explicit nulls) or `Error(status, message)`; page lists with `Page` and `Link`.
5. Unsupported features answer empty lists or 422 with a message, never 404 or 500, so clients degrade.
5. Unsupported features answer empty lists or 422 with a message, never 404 or 500, so clients degrade. Any `/api`
error that leaves without a body (a challenge, a 404 from routing, a 429) gets Mastodon's `{"error": ...}` from
`UseMastodonErrorBodies`. `NeverFiveHundredTests` walks every route with junk ids, anonymously, as a persona and
with a junk token.
6. Advertise `4.2.0 (compatible; PrivaPub)` until grouped notifications exist.
7. **What a Mastodon `Status` cannot say goes in `Status.privapub`** (`PrivaPubStatus`): object type, title, excerpt,
cover, the author's source, link, video, audio and event details, and up/down votes. Every media URL in it goes
@@ -212,6 +217,16 @@ cd /var/www/privapub.thepra.dev && sudo -u www-data ASPNETCORE_ENVIRONMENT=Produ
sibling `media-proxy`, which `/media/files` does not serve.
3. **A client never contacts a remote server for media:** every remote URL the API returns goes through
`IMediaProxy.Wrap`, an HMAC-signed `/media/proxy/` URL fetched by `IFederationHttp.GetMedia`.
4. **The proxy serves three ways:**
- **Cached:** a file already cached is served from disk, ranges included.
- **Downloaded:** a request without a `Range` is downloaded whole, up to `Media:MaxProxiedBytes`, then cached.
- **Streamed:** a ranged request, or anything too big to cache, is streamed from the origin with the range passed on,
and never cached. That is how remote video plays.
nginx has a `/media/proxy/` location with `proxy_buffering off` and a 600 s read timeout for those streams.
5. **Remote video and audio become one playable attachment** in the Mastodon API (`MastodonMapper.Playable`): the best MP4
up to 720p that carries both sound and picture, including PeerTube's fragmented files inside an HLS entry. HLS
playlists themselves are not rewritten.
## Privacy invariants
@@ -258,6 +273,21 @@ cd /var/www/privapub.thepra.dev && sudo -u www-data ASPNETCORE_ENVIRONMENT=Produ
- **A poll vote is a `Note` with a `name`, an `inReplyTo` that is a poll we hold, and no content.** `CreateHandler`
hands it to `PollService.Receive` before anything else, so it never becomes a reply. Our votes on other servers' polls
go only to the poll's author, without `published` (PieFed counts a vote only then).
- **Link previews follow owner decision 1** (`Domain/Content/LinkPreviews.cs`): only public posts, queued on arrival with
0–60 s of jitter, one cached `LinkPreview` per address for the whole server, never fetched when someone reads.
`LinkPreviews.Wanted` is called where posts are saved (`CreateHandler`, `StoreContext`, `StatusService.Publish`).
- **Quote states come from `QuoteService.Resolve`:**
- with a stamp: accepted only if `Verified` (fetched, on the quoted author's origin, naming both posts exactly);
- a FEP-044f `quote` without a stamp: pending;
- older keys only: accepted when the quoted post is public.
`QuotesCount` moves with the accepted state, never with the raw key. When a persona quotes, `QuotePermission` decides:
asking first for posts that state a policy, quoting at once for posts that state none, refusing otherwise. Only `quote`
of a post that asks for consent puts `quote` in our JSON; older-key quotes leave it out, as Sharkey learned they must.
- **Our quote policy is `ActivityPubRenderer.QuotableBy`:** the post's `LocalQuotePolicy`, falling back to the persona's
`Settings.QuotePolicy` (default `public`, owner decision), and always `nobody` for anything but public and unlisted. The
same rule writes `canQuote`, answers `QuoteRequest`s (`QuoteService.ReceiveRequest`) and fills `quote_approval`, so they
cannot disagree. A persona quoting another persona gets a parrot-licence too, so other servers see an approved quote.
- **A server is described on arrival, never on read.** The first record from a host enqueues `DescribeInstance` (its
NodeInfo, at most once a week, into `RemoteInstance`), so opening the details view tells nobody anything.
- **Post ids are the timeline order, so they follow arrival, not `published`.** `PrivacyIds.Arrived` gives a remote post
@@ -290,18 +320,37 @@ cd /var/www/privapub.thepra.dev && sudo -u www-data ASPNETCORE_ENVIRONMENT=Produ
stored as an edit date on every remote post. Write `(T?)null`. It has shipped four times (`MastodonParams.Bool/Int`,
`NoteParser.Int`, `NoteParser.Time`).
- ActivityPub output is built with `System.Text.Json.Nodes` in `ActivityPubRenderer`, not typed models. Inbound
documents are read through `InboxService.Id`/`Value`, which handle string, object and array.
documents are read through `ActivityJson.Id`/`Value`, which handle string, object and array.
- Libraries chosen for the roadmap: HtmlSanitizer, Markdig (`DisableHtml`), NSign (RFC 9421 inbound), OpenIddict +
OpenIddict.MongoDb, NetVips, Blurhash.Core, FFMpegCore. No ImageSharp (licence key enforced), no MassTransit.
## Testing
`PrivaPub.Tests` (xUnit v3). Unit tests need nothing; tests marked `Category=Integration` need a mongod and skip
without `PRIVAPUB_TEST_MONGOD=1`. CI runs the unit tests only (the box's mongods are production).
without `PRIVAPUB_TEST_MONGOD=1`. CI (`build.yml` and `deploy.yml`) runs all of them through
`tools/ci/with-test-mongod.sh`, which starts a throwaway mongod on a random localhost port and deletes it afterwards.
The box's own mongods are production, so `MongoFixture` refuses port 27022, a data directory under `/var/lib/privapub`,
and in CI anything but the wrapper's mongod. With `PRIVAPUB_TEST_REQUIRE_MONGOD=1`, which the wrapper sets, a missing
mongod fails the run instead of silently skipping half the tests.
- `Support/Peer` is an in-process HTTP server answering on two origins (`127.0.0.1` and `localhost`), so origin rules
can be tested; `Support/RemoteActor` signs real deliveries with its own key.
- Inbox scenarios go through `InboxService.Receive` with a signed request, not through the private handlers.
- Inbox scenarios go through `InboxReceiver.Receive` with a signed request, not through the private handlers.
- `Support/Host/PrivaPubHost` is the whole server under test (`WebApplicationFactory<Program>`, environment `Testing`,
configured only through `UseSetting`, on the fixture's database). `PrivaPubHost.Shared()` boots it once per run;
`SecureModeHost` is the same with `Federation:SecureMode`. Background workers are removed, so a test runs the jobs it
queued with `host.Run(j => ...)` or `host.RunInbox(activityId)`. `Accounts` signs a root up, adds personas, and gets a
Mastodon token through the real `/oauth` code flow; `RemoteActor.SignedPost`/`SignedGet` sign `HttpRequestMessage`s
for the real `/peasants` routes. Each client gets its own `X-Test-Client` address, so rate limits don't collide.
- All test classes share one database and run in parallel, so a test touches only rows it made:
- random names and GUIDs;
- `Harness.Outgoing` sees only deliveries queued since that harness started, because Peer ports are reused;
- a worker gets a scoped `new JobQueue(j => ...)` so it never leases another test's jobs;
- domain blocks are set with `DomainBlocks.Load`, never written to the database.
A test that must change something database-wide (drop indexes, run a migration over every post, let deliveries to
`localhost` fail and trip its breaker) goes in `[Xunit.Collection(nameof(Exclusive))]`, which runs alone, and
cleans up after itself. Pure logic belongs in an unconditional unit test, not in a Mongo-gated class.
Beyond the tests, verify by building, running locally, and exercising:
- the client API (sign up, create an avatar, a group, a post);
@@ -311,7 +360,7 @@ Interop is checked against real servers, starting with the workstation's own pas
```bash
DOTNET=~/.dotnet/dotnet tools/pasture/run.sh up # podman: PrivaPub + the latest GoToSocial + Mongo, behind Caddy
tools/pasture/interop.sh # 25 checks, each side driven through its own Mastodon API
tools/pasture/interop.sh # 33 checks, each side driven through its own Mastodon API
tools/pasture/run.sh down
```
+36 -1
View File
@@ -50,6 +50,7 @@ actor document, a collection, NodeInfo or a delivery relates two avatars of the
| Followers / following | `/peasants/{name}/groupies`, `/peasants/{name}/stalking` |
| Objects | `/peasants/{name}/scribbles/{id}` |
| Activities | `/peasants/{name}/grunts/{id}` |
| Quote permissions (`QuoteAuthorization`) | `/peasants/{name}/parrot-licences/{id}` |
| Direct-message context | `/peasants/{name}/whispers/{id}` |
| Profile and post pages | `/@{name}`, `/@{name}/{id}` |
@@ -117,6 +118,27 @@ content, one per choice, counted once per voter. Counts are refreshed with an `U
minutes. Our own votes on other servers' polls are sent the same way to the poll's author only, without `published`.
An incoming `Update` without a newer `updated` only refreshes counts and details; it is never recorded as an edit.
**Quotes** (FEP-044f, with the older keys):
- **Received.** A quote is read from `quote`, `quoteUrl`, `quoteUri`, `_misskey_quote` or a FEP-e232 `Link` tag.
- With a `quoteAuthorization`, the stamp is fetched and must be a `QuoteAuthorization` on the quoted author's origin,
attributed to them, naming exactly the two posts.
- A FEP-044f `quote` without a stamp is pending until an `Update` brings one.
- A quote made with the older keys only, by software that asks nobody, is shown when the quoted post is public.
- A `Delete` of a stamp revokes the quote.
- **Sent.** Our accounts quote with every key, plus the `RE:` fallback.
- When the quoted post states an `interactionPolicy.canQuote` that allows us, `quote` is included and a `QuoteRequest`
(with the quoting post as `instrument`) goes to the quoted author. The quote stays pending until their `Accept` brings a
stamp we can verify; the post is then updated with `quoteAuthorization`.
- Posts that state no policy are quoted the older way, without `quote`.
- Quoting posts are also delivered to the quoted author.
- **Quoting our accounts.** Our public and unlisted posts state `interactionPolicy.canQuote`. By default anyone may quote,
automatically (as on Mastodon); an account can choose followers only or nobody, and so can each post.
- A `QuoteRequest` is answered with `Accept{result}` naming a stamp at `/peasants/{name}/parrot-licences/{id}`, or
with `Reject`.
- A stamp is a `QuoteAuthorization` naming both posts. A revoked stamp answers 410 and is announced with
`Delete{stamp}`.
- Followers-only posts and DMs can never be quoted.
**Custom emoji** (`Emoji` tags) are read on posts, display names, bios and profile fields, at most 64 per object.
**Profiles** keep their header, profile fields, `manuallyApprovesFollowers`, `published`, `movedTo`, `indexable`,
`memorial` and avatar and header descriptions. A post's title becomes `name` and is also the first, bold line of `content`, because Mastodon does not show
@@ -128,6 +150,18 @@ Received `summary` is a content warning only on a `Note` or `Question`, or when
Visibility is expressed in `to`/`cc` the way Mastodon does it: public, unlisted, followers-only and direct. Inbound
followers-only posts are recognised by the author's own `followers` collection.
## Link previews
For a public post that links to a page, PrivaPub builds a preview card. It uses, in order:
1. a Link attachment's own `preview` (FEP-8967) or the post's own image, title and summary;
2. otherwise, the linked page, read once by the server between 0 and 60 seconds after the post arrives (never when
someone reads it), with an `Accept: text/html` request that reads at most 512 KB, through the same address checks as
every other fetch.
The page's OpenGraph and Twitter tags give the title, description and image; the result is cached per address for 7
days and shared by every account on the server. Images are served to clients only through PrivaPub's media proxy.
`Federation:FetchLinkPreviews=false` turns page fetching off.
## Local-only posts
Posts with a location (shown to nearby users of this server) never leave the server, in any form.
@@ -154,6 +188,7 @@ Posts with a location (shown to nearby users of this server) never leave the ser
## Known limitations
- Our own posts carry no custom emoji: emoji are received and shown, not offered.
- Remote media is fetched through this server's proxy when a local client displays it.
- Remote media is fetched through this server's proxy when a local client displays it. Video and audio are streamed
through it with byte ranges passed on, so a PeerTube file is never fetched whole for one viewer.
- Collections expose counts, not members.
- Only `rsa-sha256`-style keys are verified. RFC 9421 signatures are planned.
+43
View File
@@ -0,0 +1,43 @@
using PrivaPub.Domain.Content;
using PrivaPub.Models.Post;
namespace PrivaPub.Tests.Domain
{
public class LinkPreviewTests
{
[Fact]
public void Reads_open_graph_then_twitter_then_the_page_title()
{
var preview = LinkPreviews.Read("https://news.example/a?utm_source=x", new Uri("https://news.example/a"), """
<html><head><title>Fallback title</title>
<meta property="og:title" content="Rain &amp; sun">
<meta name="twitter:description" content="A <b>long</b> story">
<meta property="og:image" content="/img/cover.jpg">
<meta property="og:site_name" content="News">
</head><body></body></html>
""");
Assert.Equal("Rain & sun", preview.Title);
Assert.Equal("A long story", preview.Description);
Assert.Equal("https://news.example/img/cover.jpg", preview.ImageURL);
Assert.Equal("News", preview.SiteName);
Assert.False(preview.Failed);
Assert.True(LinkPreviews.Read("https://x.example", new Uri("https://x.example"), "<html><body>nothing</body></html>").Failed);
}
[Fact]
public void Picks_the_first_real_link_and_skips_mentions_tags_quotes_and_ourselves() =>
Assert.Equal("https://blog.example/post", LinkPreviews.FirstLink(new Post
{
Url = "https://m.example/@a/1",
ContentHtml = """
<p><span class="h-card"><a href="https://m.example/@bob" class="u-url mention">@bob</a></span>
<a href="https://m.example/tags/x" class="mention hashtag" rel="tag">#x</a>
<span class="quote-inline">RE: <a href="https://k.example/notes/1">link</a></span>
<a href="https://privapub.test/@me">me</a>
<a href="https://m.example/@a/1">self</a>
<a href="https://blog.example/post">read this</a></p>
"""
}, "https://privapub.test"));
}
}
@@ -0,0 +1,37 @@
using Microsoft.Extensions.Logging.Abstractions;
using PrivaPub.Federation.Moderation;
using PrivaPub.Models.Federation;
namespace PrivaPub.Tests.Federation
{
public class DomainBlocksTests
{
[Fact]
public void A_block_covers_subdomains_but_not_lookalikes()
{
var blocks = new DomainBlocks(NullLogger<DomainBlocks>.Instance);
blocks.Load(new[] { new DomainBlock { Domain = "evil.example", Severity = DomainBlockSeverity.Silence } });
Assert.NotNull(blocks.Find("evil.example"));
Assert.NotNull(blocks.Find("A.Evil.Example."));
Assert.Null(blocks.Find("notevil.example"));
Assert.False(blocks.IsSuspended("evil.example"));
}
[Fact]
public void A_suspension_is_found_from_any_subdomain()
{
var blocks = new DomainBlocks(NullLogger<DomainBlocks>.Instance);
blocks.Load(new[]
{
new DomainBlock { Domain = " Spam.Example. ", Severity = DomainBlockSeverity.Suspend },
new DomainBlock { Domain = "", Severity = DomainBlockSeverity.Suspend }
});
Assert.True(blocks.IsSuspended("deep.inside.spam.example"));
Assert.False(blocks.IsSuspended("example"));
Assert.Null(blocks.Find(default));
}
}
}
@@ -52,13 +52,15 @@ namespace PrivaPub.Tests.Federation
var db = new DbEntities();
_blocks = new DomainBlocks(NullLogger<DomainBlocks>.Instance);
_receiver = new InboxReceiver(_local, remote, queue, _blocks, NullLogger<InboxReceiver>.Instance);
var remotePosts = new RemotePosts(db, _local, remote, _blocks, queue, new ObjectRecords(queue), new NoPreviews());
var quotes = new QuoteService(db, remote, remotePosts, _local, delivery, new OutboxPublisher(db, _local, delivery));
_processor = new InboxProcessor(remote, new IActivityHandler[]
{
new FollowHandler(db, _local, remote, delivery),
new UndoHandler(db, _local, new Reactions(db, delivery)),
new CreateHandler(db, _local, remote, delivery, _blocks, new Fanout(db), new RemotePosts(db, _local, remote, _blocks, queue, new ObjectRecords(queue)), new GroupDistributor(delivery), new ObjectRecords(queue), new PollService(db, _local, delivery, queue)),
new DeleteHandler(db, _local, remote, delivery, new GroupDistributor(delivery)),
new UpdateHandler(db, _local, remote, new GroupDistributor(delivery), new ObjectRecords(queue))
new CreateHandler(db, _local, remote, delivery, _blocks, new Fanout(db), remotePosts, new GroupDistributor(delivery), new ObjectRecords(queue), new PollService(db, _local, delivery, queue), new NoPreviews(), quotes),
new DeleteHandler(db, _local, remote, delivery, new GroupDistributor(delivery), quotes),
new UpdateHandler(db, _local, remote, new GroupDistributor(delivery), new ObjectRecords(queue), quotes)
}, NullLogger<InboxProcessor>.Instance);
}
@@ -267,10 +269,7 @@ namespace PrivaPub.Tests.Federation
var token = TestContext.Current.CancellationToken;
var alice = await LocalAvatar("alice");
var bob = new RemoteActor(_peer, "bob", _peer.B);
await DB.Default.SaveAsync(new DomainBlock { Domain = "localhost", Severity = DomainBlockSeverity.Suspend }, token);
try
{
await _blocks.Reload(token);
_blocks.Load(new[] { new DomainBlock { Domain = "localhost", Severity = DomainBlockSeverity.Suspend } });
var before = _peer.Requests.Count;
var result = await Deliver(bob, $"/peasants/{alice.UserName}/mouth", DirectCreate(bob, alice.Uri));
@@ -279,27 +278,6 @@ namespace PrivaPub.Tests.Federation
Assert.Equal(before, _peer.Requests.Count);
Assert.False(await DB.Default.Find<Post>().Match(p => p.ActorURI == bob.Id).ExecuteAnyAsync(token));
}
finally
{
await DB.Default.DeleteAsync<DomainBlock>(b => b.Domain == "localhost");
await _blocks.Reload(token);
}
}
[Fact]
public void A_block_covers_subdomains_but_not_lookalikes()
{
var blocks = new DomainBlocks(NullLogger<DomainBlocks>.Instance);
typeof(DomainBlocks).GetField("_blocks", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance)!
.SetValue(blocks, new Dictionary<string, DomainBlock> { ["evil.example"] = new() { Domain = "evil.example", Severity = DomainBlockSeverity.Silence } });
typeof(DomainBlocks).GetField("_loadedAt", System.Reflection.BindingFlags.NonPublic | System.Reflection.BindingFlags.Instance)!
.SetValue(blocks, DateTime.UtcNow);
Assert.NotNull(blocks.Find("evil.example"));
Assert.NotNull(blocks.Find("A.Evil.Example."));
Assert.Null(blocks.Find("notevil.example"));
Assert.False(blocks.IsSuspended("evil.example"));
}
[Fact]
public async Task A_bad_signature_is_a_401()
@@ -0,0 +1,36 @@
using PrivaPub.Federation.Objects;
using System.Text.Json.Nodes;
namespace PrivaPub.Tests.Federation
{
public class ObjectRecordsTests
{
[Fact]
public void An_object_larger_than_the_cap_keeps_only_its_hash()
{
var big = new JsonObject { ["content"] = new string('x', ObjectRecords.MaxRawBytes + 1) };
var (text, hash, bytes, truncated) = ObjectRecords.Capture(big);
Assert.Null(text);
Assert.True(truncated);
Assert.True(bytes > ObjectRecords.MaxRawBytes);
Assert.Equal(64, hash.Length);
}
[Fact]
public void A_small_object_is_kept_whole_with_its_hash()
{
var small = new JsonObject { ["type"] = "Note", ["content"] = "hi" };
var (text, hash, bytes, truncated) = ObjectRecords.Capture(small);
Assert.Equal(small.ToJsonString(), text);
Assert.False(truncated);
Assert.Equal(text.Length, bytes);
Assert.Equal(64, hash.Length);
Assert.Equal(default, ObjectRecords.Capture(default));
}
}
}
@@ -214,6 +214,30 @@ namespace PrivaPub.Tests.Federation
Assert.Equal(new DateTime(2026, 9, 30, 12, 0, 0, DateTimeKind.Utc), closed.Poll.ClosedAt);
}
[Fact]
public void Finds_playable_files_inside_an_hls_only_peertube_video_and_picks_one_with_sound()
{
var note = Parse("""
{
"id": "https://tube.example/videos/watch/2", "type": "Video", "name": "HLS only", "attributedTo": "https://tube.example/accounts/ann",
"url": [
{ "type": "Link", "mediaType": "text/html", "href": "https://tube.example/w/def" },
{ "type": "Link", "mediaType": "application/x-mpegURL", "href": "https://tube.example/master.m3u8", "tag": [
{ "type": "Link", "mediaType": "video/mp4", "href": "https://tube.example/1080-fragmented.mp4", "height": 1080,
"attachment": [{ "type": "PropertyValue", "name": "ffprobe_codec_type", "value": "video" }, { "type": "PropertyValue", "name": "ffprobe_codec_type", "value": "audio" }] },
{ "type": "Link", "mediaType": "video/mp4", "href": "https://tube.example/720-fragmented.mp4", "height": 720,
"attachment": [{ "type": "PropertyValue", "name": "ffprobe_codec_type", "value": "video" }] },
{ "type": "Link", "mediaType": "video/mp4", "href": "https://tube.example/480-fragmented.mp4", "height": 480,
"attachment": [{ "type": "PropertyValue", "name": "ffprobe_codec_type", "value": "video" }, { "type": "PropertyValue", "name": "ffprobe_codec_type", "value": "audio" }] }
] }
]
}
""");
Assert.Equal(3, note.Video.Variants.Count);
Assert.Equal("https://tube.example/480-fragmented.mp4", MastodonMapper.BestVideo(note.Video.Variants).Url);
}
[Fact]
public void Normalises_hashtags_like_mastodon() =>
Assert.Equal("fedi".Normalize(System.Text.NormalizationForm.FormKC).ToLowerInvariant(), Assert.Single(Parse("""
@@ -68,18 +68,5 @@ namespace PrivaPub.Tests.Federation
Assert.Equal("<p>hi again</p>", JsonNode.Parse(Assert.Single(record.Revisions).Raw)!["content"]!.GetValue<string>());
Assert.True(await DB.Default.Find<Job>().Match(j => j.Kind == JobKind.DescribeInstance && j.Payload == record.Host).ExecuteAnyAsync(token));
}
[Fact]
public void An_object_larger_than_the_cap_keeps_only_its_hash()
{
var big = new JsonObject { ["content"] = new string('x', ObjectRecords.MaxRawBytes + 1) };
var (text, hash, bytes, truncated) = ObjectRecords.Capture(big);
Assert.Null(text);
Assert.True(truncated);
Assert.True(bytes > ObjectRecords.MaxRawBytes);
Assert.Equal(64, hash.Length);
}
}
}
+221
View File
@@ -0,0 +1,221 @@
using MongoDB.Entities;
using PrivaPub.ClientModels.Post;
using PrivaPub.Domain.Statuses;
using PrivaPub.Federation.Objects;
using PrivaPub.Models.Post;
using PrivaPub.Tests.Support;
using System.Text.Json.Nodes;
namespace PrivaPub.Tests.Federation
{
[Trait("Category", "Integration")]
public sealed class QuoteTests : IAsyncLifetime
{
Harness _harness;
public async ValueTask InitializeAsync()
{
Assert.SkipUnless(MongoFixture.Enabled, MongoFixture.Skip);
_harness = await Harness.Start();
}
public async ValueTask DisposeAsync()
{
if (_harness != default)
await _harness.DisposeAsync();
}
static string Origin(RemoteActor actor) => new Uri(actor.Id).GetLeftPart(UriPartial.Authority);
static string NewId(RemoteActor actor, string kind) => $"{Origin(actor)}/{kind}/{Guid.NewGuid():N}";
async Task<Post> Delivered(RemoteActor author, string mentioned, Action<JsonObject> shape = default)
{
var note = new JsonObject
{
["id"] = NewId(author, "notes"), ["type"] = "Note", ["attributedTo"] = author.Id, ["content"] = "<p>hello</p>",
["to"] = new JsonArray(Addressing.Public), ["cc"] = new JsonArray(mentioned),
["tag"] = new JsonArray(new JsonObject { ["type"] = "Mention", ["href"] = mentioned, ["name"] = "@someone" })
};
shape?.Invoke(note);
await _harness.Deliver(author, "/human-centipede", new JsonObject { ["id"] = NewId(author, "activities"), ["type"] = "Create", ["actor"] = author.Id, ["object"] = note });
return await DB.Default.Find<Post>().Match(p => p.ObjectURI == note["id"]!.GetValue<string>()).ExecuteFirstAsync(TestContext.Current.CancellationToken);
}
string Stamp(RemoteActor quotedAuthor, string quoting, string quoted)
{
var path = $"/stamps/{Guid.NewGuid():N}";
var id = Origin(quotedAuthor) + path;
_harness.Peer.Serve(path, new JsonObject
{
["@context"] = "https://www.w3.org/ns/activitystreams", ["id"] = id, ["type"] = "QuoteAuthorization",
["attributedTo"] = quotedAuthor.Id, ["interactingObject"] = quoting, ["interactionTarget"] = quoted
}.ToJsonString());
return id;
}
[Fact]
public async Task A_misskey_quote_of_a_public_post_is_shown_and_counted()
{
var token = TestContext.Current.CancellationToken;
var (_, alice) = await _harness.Persona("alice");
var ann = new RemoteActor(_harness.Peer, "ann");
var kitty = new RemoteActor(_harness.Peer, "kitty");
var original = await Delivered(ann, alice.Uri);
var quoting = await Delivered(kitty, alice.Uri, n =>
{
n["_misskey_quote"] = original.ObjectURI;
n["content"] = $"<p>look<span class=\"quote-inline\"><br>RE: <a href=\"{original.ObjectURI}\">{original.ObjectURI}</a></span></p>";
});
Assert.Equal(QuoteState.Accepted, quoting.QuoteState);
Assert.Equal(original.ID, quoting.QuotedPostId);
Assert.Equal(1, (await DB.Default.Find<Post>().OneAsync(original.ID, token)).QuotesCount);
var status = await new PrivaPub.Api.Mastodon.Mappers.MastodonMapper(_harness.Db, _harness.Local).Status(quoting, alice.Id, token);
Assert.Equal("accepted", status.Quote.State);
Assert.Equal(original.ID, status.Quote.QuotedStatus.Id);
Assert.DoesNotContain("RE:", status.Content);
}
[Fact]
public async Task A_stamped_quote_is_accepted_a_forged_one_is_not_and_revoking_takes_it_back()
{
var token = TestContext.Current.CancellationToken;
var (_, alice) = await _harness.Persona("alice");
var ann = new RemoteActor(_harness.Peer, "ann");
var bob = new RemoteActor(_harness.Peer, "bob");
var original = await Delivered(ann, alice.Uri);
var quotingId = NewId(bob, "notes");
var stamp = Stamp(ann, quotingId, original.ObjectURI);
var forged = Stamp(ann, NewId(bob, "notes"), original.ObjectURI);
var quoting = await Delivered(bob, alice.Uri, n => { n["id"] = quotingId; n["quote"] = original.ObjectURI; n["quoteAuthorization"] = stamp; });
var cheat = await Delivered(bob, alice.Uri, n => { n["quote"] = original.ObjectURI; n["quoteAuthorization"] = forged; });
var waiting = await Delivered(bob, alice.Uri, n => n["quote"] = original.ObjectURI);
Assert.Equal(QuoteState.Accepted, quoting.QuoteState);
Assert.Equal(QuoteState.Unauthorized, cheat.QuoteState);
Assert.Equal(QuoteState.Pending, waiting.QuoteState);
await _harness.Deliver(ann, "/human-centipede", new JsonObject
{
["id"] = NewId(ann, "activities"), ["type"] = "Delete", ["actor"] = ann.Id, ["object"] = stamp
});
Assert.Equal(QuoteState.Revoked, (await DB.Default.Find<Post>().OneAsync(quoting.ID, token)).QuoteState);
Assert.Equal(0, (await DB.Default.Find<Post>().OneAsync(original.ID, token)).QuotesCount);
}
[Fact]
public async Task A_persona_asks_before_quoting_and_is_stamped_when_accepted()
{
var token = TestContext.Current.CancellationToken;
var (_, alice) = await _harness.Persona("alice");
var ann = new RemoteActor(_harness.Peer, "ann");
var original = await Delivered(ann, alice.Uri, n => n["interactionPolicy"] = new JsonObject
{
["canQuote"] = new JsonObject { ["automaticApproval"] = new JsonArray(Addressing.Public) }
});
var outcome = await _harness.Statuses.Publish(alice, new StatusDraft { Text = "this", QuotedStatusId = original.ID }, token);
Assert.Equal(QuoteState.Pending, outcome.Post.QuoteState);
var request = Assert.Single(await _harness.Outgoing(ann.Id + "/inbox"), a => a["type"]!.GetValue<string>() == "QuoteRequest");
Assert.Equal(original.ObjectURI, request["object"]!.GetValue<string>());
Assert.Equal(original.ObjectURI, request["instrument"]!["quote"]!.GetValue<string>());
await _harness.Deliver(ann, "/human-centipede", new JsonObject
{
["id"] = NewId(ann, "activities"), ["type"] = "Accept", ["actor"] = ann.Id,
["object"] = request["id"]!.GetValue<string>(), ["result"] = Stamp(ann, outcome.Post.ObjectURI, original.ObjectURI)
});
var quoting = await DB.Default.Find<Post>().OneAsync(outcome.Post.ID, token);
Assert.Equal(QuoteState.Accepted, quoting.QuoteState);
Assert.NotNull(quoting.QuoteAuthorizationURI);
Assert.Contains(await _harness.Outgoing(ann.Id + "/inbox"), a => a["type"]!.GetValue<string>() == "Update"
&& a["object"]!["quoteAuthorization"]?.GetValue<string>() == quoting.QuoteAuthorizationURI);
}
[Fact]
public async Task A_legacy_post_is_quoted_without_asking_and_a_persona_that_allows_nobody_cannot_be_quoted()
{
var token = TestContext.Current.CancellationToken;
var (aliceRoot, alice) = await _harness.Persona("alice");
var (bobRoot, bob) = await _harness.Persona("bob");
var kitty = new RemoteActor(_harness.Peer, "kitty");
var legacy = await Delivered(kitty, alice.Uri);
bob.Settings.QuotePolicy = PrivaPub.Models.User.QuotePolicies.Nobody;
var bobs = (await _harness.Statuses.Publish(bob, new StatusDraft { Text = "mine" }, token)).Post;
var outcome = await _harness.Statuses.Publish(alice, new StatusDraft { Text = "nice", QuotedStatusId = legacy.ID }, token);
Assert.Equal(QuoteState.Accepted, outcome.Post.QuoteState);
Assert.False(outcome.Post.QuoteByConsent);
Assert.False((await _harness.Statuses.Publish(alice, new StatusDraft { Text = "and this", QuotedStatusId = bobs.ID }, token)).Ok);
Assert.True((await _harness.Statuses.Publish(bob, new StatusDraft { Text = "me again", QuotedStatusId = bobs.ID }, token)).Ok);
}
JsonObject Request(RemoteActor quoter, string quotedUri, string quotingId) => new()
{
["id"] = NewId(quoter, "activities"), ["type"] = "QuoteRequest", ["actor"] = quoter.Id, ["object"] = quotedUri,
["instrument"] = new JsonObject { ["id"] = quotingId, ["type"] = "Note", ["attributedTo"] = quoter.Id, ["quote"] = quotedUri, ["content"] = "<p>so true</p>" }
};
[Fact]
public async Task Anyone_may_quote_a_persona_by_default_and_the_licence_can_be_revoked()
{
var token = TestContext.Current.CancellationToken;
var (_, alice) = await _harness.Persona("alice");
var mallory = new RemoteActor(_harness.Peer, "mallory");
await _harness.Deliver(mallory, "/human-centipede", new JsonObject { ["id"] = NewId(mallory, "follows"), ["type"] = "Follow", ["actor"] = mallory.Id, ["object"] = alice.Uri });
var original = (await _harness.Statuses.Publish(alice, new StatusDraft { Text = "quote me" }, token)).Post;
var created = (await _harness.Outgoing(mallory.Id + "/inbox")).Single(a => a["type"]!.GetValue<string>() == "Create")["object"]!;
Assert.Equal(Addressing.Public, created["interactionPolicy"]!["canQuote"]!["automaticApproval"]![0]!.GetValue<string>());
var quotingId = NewId(mallory, "notes");
await _harness.Deliver(mallory, "/human-centipede", Request(mallory, original.ObjectURI, quotingId));
var accept = Assert.Single(await _harness.Outgoing(mallory.Id + "/inbox"), a => a["type"]!.GetValue<string>() == "Accept" && a["result"] != null);
var licence = accept["result"]!.GetValue<string>();
Assert.StartsWith(alice.Uri + "/parrot-licences/", licence);
var quoting = await Delivered(mallory, alice.Uri, n => { n["id"] = quotingId; n["quote"] = original.ObjectURI; n["quoteAuthorization"] = licence; });
Assert.Equal(QuoteState.Accepted, quoting.QuoteState);
Assert.Equal(1, (await DB.Default.Find<Post>().OneAsync(original.ID, token)).QuotesCount);
Assert.True(await _harness.Quotes.RevokeLicence(original, quoting, token));
Assert.Equal(QuoteState.Revoked, (await DB.Default.Find<Post>().OneAsync(quoting.ID, token)).QuoteState);
Assert.Contains(await _harness.Outgoing(mallory.Id + "/inbox"), a => a["type"]!.GetValue<string>() == "Delete" && a["object"]!.GetValue<string>() == licence);
}
[Fact]
public async Task A_followers_only_persona_turns_a_stranger_down()
{
var token = TestContext.Current.CancellationToken;
var (_, alice) = await _harness.Persona("alice");
alice.Settings.QuotePolicy = PrivaPub.Models.User.QuotePolicies.Followers;
var original = (await _harness.Statuses.Publish(alice, new StatusDraft { Text = "friends only quoting" }, token)).Post;
var stranger = new RemoteActor(_harness.Peer, "stranger");
await _harness.Deliver(stranger, "/human-centipede", Request(stranger, original.ObjectURI, NewId(stranger, "notes")));
Assert.Equal("Reject", Assert.Single(await _harness.Outgoing(stranger.Id + "/inbox"))["type"]!.GetValue<string>());
}
[Fact]
public async Task One_persona_quotes_another_with_a_licence()
{
var token = TestContext.Current.CancellationToken;
var (_, alice) = await _harness.Persona("alice");
var (_, bob) = await _harness.Persona("bob");
var original = (await _harness.Statuses.Publish(alice, new StatusDraft { Text = "public thought" }, token)).Post;
var outcome = await _harness.Statuses.Publish(bob, new StatusDraft { Text = "agreed", QuotedStatusId = original.ID }, token);
Assert.True(outcome.Ok);
Assert.Equal(QuoteState.Accepted, outcome.Post.QuoteState);
Assert.StartsWith(alice.Uri + "/parrot-licences/", outcome.Post.QuoteAuthorizationURI);
Assert.Equal(1, (await DB.Default.Find<Post>().OneAsync(original.ID, token)).QuotesCount);
}
}
}
+146
View File
@@ -0,0 +1,146 @@
using Microsoft.AspNetCore.Mvc.Controllers;
using Microsoft.AspNetCore.Mvc.Infrastructure;
using Microsoft.AspNetCore.Mvc.RazorPages;
using Microsoft.Extensions.DependencyInjection;
using MongoDB.Entities;
using PrivaPub.Federation.Inbox;
using PrivaPub.Infrastructure.Jobs;
using PrivaPub.Models;
using PrivaPub.Models.Jobs;
using PrivaPub.Models.Post;
using PrivaPub.Tests.Support;
using PrivaPub.Tests.Support.Host;
using System.Net;
using System.Text.Json.Nodes;
namespace PrivaPub.Tests.Http
{
[Trait("Category", "Integration")]
public sealed class HostBootTests : IAsyncLifetime
{
PrivaPubHost _host;
public async ValueTask InitializeAsync()
{
Assert.SkipUnless(MongoFixture.Enabled, MongoFixture.Skip);
_host = await PrivaPubHost.Shared();
}
public ValueTask DisposeAsync() => ValueTask.CompletedTask;
[Fact]
public async Task The_host_shares_the_fixture_database()
{
Assert.True(await DB.Default.Find<AppConfiguration>().ExecuteAnyAsync(TestContext.Current.CancellationToken));
using var client = _host.Client();
Assert.Equal(HttpStatusCode.OK, (await client.GetAsync("/build.json", TestContext.Current.CancellationToken)).StatusCode);
}
[Fact]
public async Task The_harness_handles_the_same_activities_as_the_server()
{
await using var harness = await Harness.Start();
var registered = _host.Services.GetServices<IActivityHandler>().Select(h => h.GetType()).OrderBy(t => t.Name);
var harnessed = harness.Handlers.Select(h => h.GetType()).OrderBy(t => t.Name);
Assert.Equal(registered, harnessed);
}
[Fact]
public void Every_job_kind_has_exactly_one_handler()
{
var handlers = _host.Services.GetServices<IJobHandler>().GroupBy(h => h.Kind).ToDictionary(g => g.Key, g => g.Count());
foreach (var kind in Enum.GetValues<JobKind>())
Assert.True(handlers.GetValueOrDefault(kind) == 1, $"{kind} has {handlers.GetValueOrDefault(kind)} handlers");
}
[Fact]
public void Every_controller_and_page_model_can_be_made()
{
using var scope = _host.Services.CreateScope();
var controllers = _host.Get<IActionDescriptorCollectionProvider>().ActionDescriptors.Items
.OfType<ControllerActionDescriptor>()
.Select(d => d.ControllerTypeInfo.AsType())
.Distinct()
.ToList();
var pages = typeof(Program).Assembly.GetTypes().Where(t => t.IsSubclassOf(typeof(PageModel)) && !t.IsAbstract).ToList();
Assert.NotEmpty(controllers);
Assert.NotEmpty(pages);
foreach (var type in controllers.Concat(pages))
Assert.NotNull(ActivatorUtilities.CreateInstance(scope.ServiceProvider, type));
}
[Fact]
public void The_service_graph_is_valid()
{
var services = new ServiceCollection();
foreach (var descriptor in _host.Registered)
((ICollection<ServiceDescriptor>)services).Add(descriptor);
using var provider = services.BuildServiceProvider(new ServiceProviderOptions { ValidateOnBuild = true, ValidateScopes = true });
}
[Fact]
public async Task Swagger_is_not_served_outside_development()
{
using var client = _host.Client();
Assert.Equal(HttpStatusCode.NotFound, (await client.GetAsync("/swagger/index.html", TestContext.Current.CancellationToken)).StatusCode);
}
[Fact]
public async Task A_persona_gets_a_mastodon_token_through_the_real_code_flow()
{
var root = await _host.SignUp();
var persona = await _host.Persona(root, "boot");
var token = await _host.MastodonToken(persona);
using var client = _host.As(token);
var account = await client.GetStringAsync("/api/v1/accounts/verify_credentials", TestContext.Current.CancellationToken);
Assert.Contains($"\"id\":\"{persona.Id}\"", account);
Assert.DoesNotContain(root.Id, account);
Assert.DoesNotContain(root.UserName, account);
}
[Fact]
public async Task A_signed_delivery_through_the_real_route_is_queued_and_stored()
{
var token = TestContext.Current.CancellationToken;
await using var peer = await Peer.Start();
var persona = await _host.Persona(await _host.SignUp(), "mouth");
var bob = new RemoteActor(peer, "bob");
var noteId = $"{peer.A}/notes/{Guid.NewGuid():N}";
var activity = new JsonObject
{
["id"] = noteId + "/activity",
["type"] = "Create",
["actor"] = bob.Id,
["to"] = new JsonArray($"{PrivaPubHost.Base}/peasants/{persona.UserName}"),
["object"] = new JsonObject
{
["id"] = noteId,
["type"] = "Note",
["attributedTo"] = bob.Id,
["to"] = new JsonArray($"{PrivaPubHost.Base}/peasants/{persona.UserName}"),
["content"] = "<p>hello through the front door</p>",
["published"] = DateTime.UtcNow.ToString("O")
}
};
using var client = _host.Client();
var response = await client.SendAsync(bob.SignedPost($"/peasants/{persona.UserName}/mouth", activity), token);
Assert.Equal(HttpStatusCode.Accepted, response.StatusCode);
Assert.Equal(1, await _host.RunInbox(noteId + "/activity", token));
var stored = await DB.Default.Find<Post>().Match(p => p.ObjectURI == noteId).ExecuteFirstAsync(token);
Assert.Equal(PostVisibility.Direct, stored.Visibility);
Assert.Contains("front door", stored.ContentHtml);
}
}
}
@@ -0,0 +1,127 @@
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Routing;
using Microsoft.AspNetCore.Routing.Patterns;
using Microsoft.Extensions.DependencyInjection;
using PrivaPub.Tests.Support;
using PrivaPub.Tests.Support.Host;
using System.Net;
using System.Net.Http.Headers;
using System.Text;
using System.Text.Json.Nodes;
namespace PrivaPub.Tests.Http
{
[Trait("Category", "Integration")]
public sealed class NeverFiveHundredTests : IAsyncLifetime
{
static readonly string[] Junk = { "x", "000000000000000000000000", "%2e%2e", new string('a', 5000), "5f0c1a2b3c4d5e6f70819203" };
PrivaPubHost _host;
public async ValueTask InitializeAsync()
{
Assert.SkipUnless(MongoFixture.Enabled, MongoFixture.Skip);
_host = await PrivaPubHost.Shared();
}
public ValueTask DisposeAsync() => ValueTask.CompletedTask;
public static TheoryData<string> Credentials() => new() { "anonymous", "persona", "junk" };
[Theory]
[MemberData(nameof(Credentials))]
public async Task No_route_answers_500_to_junk(string credentials)
{
var token = TestContext.Current.CancellationToken;
var bearer = credentials switch
{
"persona" => await _host.MastodonToken(await _host.Persona(await _host.SignUp(), "junk")),
"junk" => "not-a-token",
_ => default
};
var jwt = credentials == "persona" ? (await _host.SignUp()).Jwt : bearer;
var failures = new List<string>();
foreach (var (method, template) in Routes())
foreach (var value in Junk)
{
var path = Fill(template, value);
using var request = new HttpRequestMessage(new HttpMethod(method), path);
request.Headers.Add(PrivaPubHost.ClientHeader, $"10.9.{Random.Shared.Next(256)}.{Random.Shared.Next(1, 255)}");
var auth = path.StartsWith("/api/", StringComparison.Ordinal) || path.StartsWith("/oauth/", StringComparison.Ordinal) ? bearer : jwt;
if (auth != default)
request.Headers.Authorization = new AuthenticationHeaderValue("Bearer", auth);
if (method is "POST" or "PUT" or "PATCH")
request.Content = new StringContent("{}", Encoding.UTF8, "application/json");
using var client = _host.Client();
HttpResponseMessage response;
try
{
response = await client.SendAsync(request, token);
}
catch (Exception ex)
{
failures.Add($"{method} {Short(path)} threw {ex.GetType().Name}: {ex.Message}");
continue;
}
using (response)
{
if ((int)response.StatusCode >= 500)
failures.Add($"{method} {Short(path)} answered {(int)response.StatusCode}");
else if (path.StartsWith("/api/", StringComparison.Ordinal) && (int)response.StatusCode >= 400 && !await IsMastodonError(response))
failures.Add($"{method} {Short(path)} answered {(int)response.StatusCode} without a Mastodon error body");
}
}
Assert.True(failures.Count == 0, $"{failures.Count} problems as {credentials}:\n" + string.Join("\n", failures.Distinct().Take(80)));
}
IEnumerable<(string Method, string Template)> Routes()
{
var endpoints = _host.Services.GetRequiredService<EndpointDataSource>().Endpoints.OfType<RouteEndpoint>();
foreach (var endpoint in endpoints)
{
var methods = endpoint.Metadata.GetMetadata<IHttpMethodMetadata>()?.HttpMethods;
var template = "/" + endpoint.RoutePattern.RawText?.TrimStart('/');
foreach (var method in methods is { Count: > 0 } ? methods : new[] { "GET", "POST" })
yield return (method, template);
}
}
static string Fill(string template, string value)
{
var pattern = RoutePatternFactory.Parse(template);
var builder = new StringBuilder();
foreach (var segment in pattern.PathSegments)
{
builder.Append('/');
foreach (var part in segment.Parts)
builder.Append(part switch
{
RoutePatternLiteralPart literal => literal.Content,
RoutePatternParameterPart => value,
RoutePatternSeparatorPart separator => separator.Content,
_ => string.Empty
});
}
return builder.Length == 0 ? "/" : builder.ToString();
}
static async Task<bool> IsMastodonError(HttpResponseMessage response)
{
var body = await response.Content.ReadAsStringAsync();
try
{
return JsonNode.Parse(body) is JsonObject error && error["error"] is JsonValue;
}
catch (System.Text.Json.JsonException)
{
return false;
}
}
static string Short(string path) => path.Length > 120 ? path[..120] + "…" : path;
}
}
@@ -0,0 +1,95 @@
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Logging.Abstractions;
using MongoDB.Entities;
using PrivaPub.Federation.Actors;
using PrivaPub.Federation.Outbox;
using PrivaPub.Infrastructure.Jobs;
using PrivaPub.Models;
using PrivaPub.Models.Jobs;
using PrivaPub.Models.User;
using PrivaPub.StaticServices;
using PrivaPub.Tests.Support;
using System.Text.Json.Nodes;
namespace PrivaPub.Tests.Infrastructure
{
[Xunit.Collection(nameof(Exclusive))]
[Trait("Category", "Integration")]
public sealed class DeliveryWorkerTests : IAsyncLifetime
{
static readonly string[] PeerHosts = { "localhost", "127.0.0.1" };
Peer _peer;
public async ValueTask InitializeAsync()
{
Assert.SkipUnless(MongoFixture.Enabled, MongoFixture.Skip);
_peer = await Peer.Start();
await DB.Default.DeleteAsync<RemoteInstance>(i => PeerHosts.Contains(i.Host));
}
public async ValueTask DisposeAsync()
{
if (_peer == default)
return;
await _peer.DisposeAsync();
await DB.Default.DeleteAsync<RemoteInstance>(i => PeerHosts.Contains(i.Host));
}
[Fact]
public async Task A_dead_host_does_not_hold_up_deliveries_to_live_ones()
{
var token = TestContext.Current.CancellationToken;
var (privateKey, publicKey) = Keys.NewKeyPair();
var avatar = new Avatar { UserName = $"sender{Guid.NewGuid():N}"[..20], PrivateKey = privateKey, PublicKey = publicKey };
await DB.Default.SaveAsync(avatar, token);
var local = new LocalActorService(new DbEntities(), new StaticOptions<AppConfiguration>(new AppConfiguration { BackendBaseAddress = "https://privapub.test" }));
var sender = local.FromAvatar(avatar);
var run = $"https://privapub.test/a/{Guid.NewGuid():N}/";
var queue = new JobQueue(j => j.DedupeKey.StartsWith(run));
var delivery = new DeliveryService(new DbEntities(), queue);
_peer.Answer("/dead/inbox", 503, TimeSpan.FromMilliseconds(300));
_peer.Answer("/live/inbox", 202);
var deadInboxes = Enumerable.Range(0, 60).Select(i => $"{_peer.B}/dead/inbox?{i}");
await delivery.Enqueue(sender, deadInboxes, new JsonObject { ["id"] = run + "dead", ["type"] = "Create" }, token);
await delivery.Enqueue(sender, new[] { $"{_peer.A}/live/inbox" }, new JsonObject { ["id"] = run + "live", ["type"] = "Create" }, token);
var handler = new DeliveryJobHandler(local, Peer.Http(), new HostCircuitBreaker(new MemoryCache(new MemoryCacheOptions())), NullLogger<DeliveryJobHandler>.Instance);
using var worker = new JobWorker(queue, new IJobHandler[] { handler }, NullLogger<JobWorker>.Instance);
await worker.StartAsync(token);
try
{
await Until(() => Task.FromResult(_peer.Requests.Any(r => r.Path == "/live/inbox")), token);
var deadSoFar = _peer.Requests.Count(r => r.Path == "/dead/inbox");
var live = Assert.Single(_peer.Requests, r => r.Path == "/live/inbox");
Assert.Contains($"keyId=\"{sender.KeyId}\"", live.Signature);
Assert.True(deadSoFar < 30, $"{deadSoFar} dead deliveries were tried before the live one");
await Until(() => DB.Default.Find<Job>().Match(j => j.DedupeKey.StartsWith(run + "dead") && j.State == JobState.Pending && j.Attempts > 0).ExecuteAnyAsync(token), token);
}
finally
{
await worker.StopAsync(CancellationToken.None);
}
var retried = await DB.Default.Find<Job>().Match(j => j.DedupeKey.StartsWith(run + "dead") && j.Attempts > 0 && j.State == JobState.Pending).ExecuteFirstAsync(token);
Assert.Equal("503", retried.LastError?.Split(' ')[0]);
Assert.True(retried.RunAt > DateTime.UtcNow);
Assert.True((await DB.Default.Find<RemoteInstance>().Match(i => i.Host == "localhost").ExecuteFirstAsync(token))?.ConsecutiveFailures > 0);
}
static async Task Until(Func<Task<bool>> condition, CancellationToken token)
{
var deadline = DateTime.UtcNow.AddSeconds(30);
while (!await condition())
{
Assert.True(DateTime.UtcNow < deadline, "timed out");
await Task.Delay(50, token);
}
}
}
}
@@ -0,0 +1,67 @@
using Microsoft.Extensions.Caching.Memory;
using MongoDB.Entities;
using PrivaPub.Infrastructure.Jobs;
using PrivaPub.Models.Jobs;
using PrivaPub.Tests.Support;
namespace PrivaPub.Tests.Infrastructure
{
[Trait("Category", "Integration")]
public sealed class HostCircuitBreakerTests : IAsyncLifetime
{
readonly string _host = $"breaker{Guid.NewGuid():N}.example";
public ValueTask InitializeAsync()
{
Assert.SkipUnless(MongoFixture.Enabled, MongoFixture.Skip);
return ValueTask.CompletedTask;
}
public async ValueTask DisposeAsync()
{
if (MongoFixture.Enabled)
await DB.Default.DeleteAsync<RemoteInstance>(i => i.Host == _host);
}
[Fact]
public async Task A_host_is_quarantined_only_past_the_threshold_and_freed_by_a_success()
{
var token = TestContext.Current.CancellationToken;
var breaker = new HostCircuitBreaker(new MemoryCache(new MemoryCacheOptions()));
for (var i = 1; i < HostCircuitBreaker.Threshold; i++)
await breaker.Failed(_host, "503", token);
Assert.Null(await breaker.UnavailableUntil(_host, token));
await breaker.Failed(_host, "503", token);
var until = await breaker.UnavailableUntil(_host, token);
Assert.NotNull(until);
Assert.InRange(until.Value, DateTime.UtcNow.AddMinutes(55), DateTime.UtcNow.AddMinutes(65));
var instance = await DB.Default.Find<RemoteInstance>().Match(i => i.Host == _host).ExecuteFirstAsync(token);
Assert.Equal(HostCircuitBreaker.Threshold, instance.ConsecutiveFailures);
Assert.Equal("503", instance.LastError);
await breaker.Succeeded(_host, token);
Assert.Null(await breaker.UnavailableUntil(_host, token));
instance = await DB.Default.Find<RemoteInstance>().Match(i => i.Host == _host).ExecuteFirstAsync(token);
Assert.Equal(0, instance.ConsecutiveFailures);
Assert.NotNull(instance.LastSuccessAt);
}
[Fact]
public async Task A_healthy_host_is_written_at_most_once_an_hour()
{
var token = TestContext.Current.CancellationToken;
var breaker = new HostCircuitBreaker(new MemoryCache(new MemoryCacheOptions()));
await breaker.Succeeded(_host, token);
var first = (await DB.Default.Find<RemoteInstance>().Match(i => i.Host == _host).ExecuteFirstAsync(token)).LastSuccessAt;
await breaker.Succeeded(_host, token);
var second = (await DB.Default.Find<RemoteInstance>().Match(i => i.Host == _host).ExecuteFirstAsync(token)).LastSuccessAt;
Assert.Equal(first, second);
}
}
}
@@ -12,6 +12,7 @@ using PrivaPub.Tests.Support;
namespace PrivaPub.Tests.Infrastructure
{
[Xunit.Collection(nameof(Exclusive))]
[Trait("Category", "Integration")]
public class IndexTests
{
+4 -55
View File
@@ -1,19 +1,9 @@
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Logging.Abstractions;
using MongoDB.Entities;
using PrivaPub.Federation.Actors;
using PrivaPub.Federation.Outbox;
using PrivaPub.Infrastructure.Jobs;
using PrivaPub.Models;
using PrivaPub.Models.Jobs;
using PrivaPub.Models.User;
using PrivaPub.StaticServices;
using PrivaPub.Tests.Support;
using System.Text.Json.Nodes;
namespace PrivaPub.Tests.Infrastructure
{
public class BackoffTests
@@ -38,19 +28,13 @@ namespace PrivaPub.Tests.Infrastructure
[Trait("Category", "Integration")]
public sealed class JobQueueTests : IAsyncLifetime
{
Peer _peer;
public async ValueTask InitializeAsync()
public ValueTask InitializeAsync()
{
Assert.SkipUnless(MongoFixture.Enabled, MongoFixture.Skip);
_peer = await Peer.Start();
return ValueTask.CompletedTask;
}
public async ValueTask DisposeAsync()
{
if (_peer != default)
await _peer.DisposeAsync();
}
public ValueTask DisposeAsync() => ValueTask.CompletedTask;
static Job NewJob(string host = default, string dedupe = default) => new()
{
@@ -102,44 +86,9 @@ namespace PrivaPub.Tests.Infrastructure
var job = new Job { Kind = (JobKind)98, State = JobState.Running, LeasedUntil = DateTime.UtcNow.AddMinutes(-1) };
await DB.Default.SaveAsync(job, token);
await new JobQueue().Reap(token);
await new JobQueue(j => j.ID == job.ID).Reap(token);
Assert.Equal(JobState.Pending, (await DB.Default.Find<Job>().OneAsync(job.ID, token)).State);
}
[Fact]
public async Task A_dead_host_does_not_hold_up_deliveries_to_live_ones()
{
var token = TestContext.Current.CancellationToken;
var (privateKey, publicKey) = Keys.NewKeyPair();
var avatar = new Avatar { UserName = $"sender{Guid.NewGuid():N}"[..20], PrivateKey = privateKey, PublicKey = publicKey };
await DB.Default.SaveAsync(avatar, token);
var local = new LocalActorService(new DbEntities(), new StaticOptions<AppConfiguration>(new AppConfiguration { BackendBaseAddress = "https://privapub.test" }));
var sender = local.FromAvatar(avatar);
var queue = new JobQueue();
var delivery = new DeliveryService(new DbEntities(), queue);
_peer.Answer("/dead/inbox", 503, TimeSpan.FromMilliseconds(300));
_peer.Answer("/live/inbox", 202);
var deadInboxes = Enumerable.Range(0, 60).Select(i => $"{_peer.B}/dead/inbox?{i}");
await delivery.Enqueue(sender, deadInboxes, new JsonObject { ["id"] = $"https://privapub.test/a/{Guid.NewGuid():N}", ["type"] = "Create" }, token);
await delivery.Enqueue(sender, new[] { $"{_peer.A}/live/inbox" }, new JsonObject { ["id"] = $"https://privapub.test/a/{Guid.NewGuid():N}", ["type"] = "Create" }, token);
var handler = new DeliveryJobHandler(local, Peer.Http(), new HostCircuitBreaker(new MemoryCache(new MemoryCacheOptions())), NullLogger<DeliveryJobHandler>.Instance);
using var worker = new JobWorker(queue, new IJobHandler[] { handler }, NullLogger<JobWorker>.Instance);
await worker.StartAsync(token);
var deadline = DateTime.UtcNow.AddSeconds(5);
while (DateTime.UtcNow < deadline && !_peer.Requests.Any(r => r.Path == "/live/inbox"))
await Task.Delay(50, token);
var live = Assert.Single(_peer.Requests, r => r.Path == "/live/inbox");
Assert.True(_peer.Requests.Count(r => r.Path == "/dead/inbox") < 30);
while (DateTime.UtcNow < deadline && !await DB.Default.Find<RemoteInstance>().Match(i => i.Host == "localhost").ExecuteAnyAsync(token))
await Task.Delay(50, token);
await worker.StopAsync(token);
Assert.Contains($"keyId=\"{sender.KeyId}\"", live.Signature);
var instance = await DB.Default.Find<RemoteInstance>().Match(i => i.Host == "localhost").ExecuteFirstAsync(token);
Assert.True(instance.ConsecutiveFailures > 0);
}
}
}
@@ -6,6 +6,7 @@ using PrivaPub.Tests.Support;
namespace PrivaPub.Tests.Infrastructure
{
[Xunit.Collection(nameof(Exclusive))]
[Trait("Category", "Integration")]
public class MigrationTests
{
+1
View File
@@ -9,6 +9,7 @@
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.AspNetCore.Mvc.Testing" Version="10.0.9" />
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="18.10.1" />
<PackageReference Include="xunit.v3" Version="3.2.2" />
<PackageReference Include="xunit.runner.visualstudio" Version="3.1.5" />
+7
View File
@@ -0,0 +1,7 @@
namespace PrivaPub.Tests.Support
{
[CollectionDefinition(nameof(Exclusive), DisableParallelization = true)]
public sealed class Exclusive
{
}
}
+25 -11
View File
@@ -34,6 +34,8 @@ namespace PrivaPub.Tests.Support
public const string Host = "privapub.test";
public const string Base = "https://" + Host;
readonly DateTime _started = Millisecond(DateTime.UtcNow);
Harness(Peer peer)
{
Peer = peer;
@@ -46,30 +48,33 @@ namespace PrivaPub.Tests.Support
Records = new ObjectRecords(Queue);
Polls = new PollService(Db, Local, Delivery, Queue);
Reactions = new Reactions(Db, Delivery);
RemotePosts = new RemotePosts(Db, Local, Remote, new NoBlocks(), Queue, Records);
RemotePosts = new RemotePosts(Db, Local, Remote, new NoBlocks(), Queue, Records, new NoPreviews());
Outbox = new OutboxPublisher(Db, Local, Delivery);
Quotes = new QuoteService(Db, Remote, RemotePosts, Local, Delivery, Outbox);
Receiver = new InboxReceiver(Local, Remote, Queue, new NoBlocks(), NullLogger<InboxReceiver>.Instance);
Processor = new InboxProcessor(Remote, new IActivityHandler[]
Handlers = new IActivityHandler[]
{
new FollowHandler(Db, Local, Remote, Delivery),
new AcceptHandler(Db, Local),
new RejectHandler(Db, Local),
new AcceptHandler(Db, Local, Quotes),
new RejectHandler(Db, Local, Quotes),
new UndoHandler(Db, Local, Reactions),
new LikeHandler(Db, Reactions),
new EmojiReactHandler(Db, Reactions),
new QuoteRequestHandler(Quotes),
new DislikeHandler(Db),
new JoinHandler(Db, Local, Delivery),
new AnnounceHandler(Db, Local, RemotePosts, Fanout, Remote),
new CreateHandler(Db, Local, Remote, Delivery, new NoBlocks(), Fanout, RemotePosts, Groups, Records, Polls),
new DeleteHandler(Db, Local, Remote, Delivery, Groups),
new UpdateHandler(Db, Local, Remote, Groups, Records),
new CreateHandler(Db, Local, Remote, Delivery, new NoBlocks(), Fanout, RemotePosts, Groups, Records, Polls, new NoPreviews(), Quotes),
new DeleteHandler(Db, Local, Remote, Delivery, Groups, Quotes),
new UpdateHandler(Db, Local, Remote, Groups, Records, Quotes),
new FlagHandler(Db, Local)
}, NullLogger<InboxProcessor>.Instance);
};
Processor = new InboxProcessor(Remote, Handlers, NullLogger<InboxProcessor>.Instance);
Follows = new FollowService(Db, Local, Remote, Delivery, new KeyLocalizer<GenericRes>(), NullLogger<FollowService>.Instance);
Content = new ContentRenderer(Local, Remote);
Outbox = new OutboxPublisher(Db, Local, Delivery);
Media = new MediaService(new StaticOptions<MediaOptions>(new MediaOptions { Root = Path.Combine(Path.GetTempPath(), $"privapub-media-{Guid.NewGuid():N}") }),
Local, default, NullLogger<MediaService>.Instance);
Statuses = new StatusService(Db, Local, Remote, Delivery, Content, Outbox, Fanout, Media, Groups, Polls);
Statuses = new StatusService(Db, Local, Remote, Delivery, Content, Outbox, Fanout, Media, Groups, Polls, new NoPreviews(), Quotes);
Posts = new PostsService(Db, Local, Statuses, new KeyLocalizer<GenericRes>(), NullLogger<PostsService>.Instance);
Timelines = new TimelineService(Db, new KeyLocalizer<GenericRes>());
Relationships = new RelationshipService(Db, Follows, Delivery);
@@ -84,6 +89,7 @@ namespace PrivaPub.Tests.Support
public DeliveryService Delivery { get; }
public InboxReceiver Receiver { get; }
public InboxProcessor Processor { get; }
public IReadOnlyList<IActivityHandler> Handlers { get; }
public FollowService Follows { get; }
public ContentRenderer Content { get; }
public OutboxPublisher Outbox { get; }
@@ -96,6 +102,7 @@ namespace PrivaPub.Tests.Support
public ObjectRecords Records { get; }
public PollService Polls { get; }
public Reactions Reactions { get; }
public QuoteService Quotes { get; }
public TimelineService Timelines { get; }
public RelationshipService Relationships { get; }
public ReportService Reports { get; }
@@ -133,13 +140,15 @@ namespace PrivaPub.Tests.Support
}
public async Task<List<JsonObject>> Outgoing(string inbox) =>
(await DB.Default.Find<Job>().Match(j => j.Kind == JobKind.Deliver).ExecuteAsync())
(await DB.Default.Find<Job>().Match(j => j.Kind == JobKind.Deliver && j.CreatedAt >= _started).ExecuteAsync())
.Select(j => JsonSerializer.Deserialize<DeliveryPayload>(j.Payload))
.Where(p => p.Inbox == inbox)
.Select(p => JsonNode.Parse(p.Body)!.AsObject())
.ToList();
public async ValueTask DisposeAsync() => await Peer.DisposeAsync();
static DateTime Millisecond(DateTime time) => new(time.Ticks - time.Ticks % TimeSpan.TicksPerMillisecond, DateTimeKind.Utc);
}
public sealed class KeyLocalizer<T> : IStringLocalizer<T>
@@ -148,4 +157,9 @@ namespace PrivaPub.Tests.Support
public LocalizedString this[string name, params object[] arguments] => new(name, string.Format(name, arguments));
public IEnumerable<LocalizedString> GetAllStrings(bool includeParentCultures) => Enumerable.Empty<LocalizedString>();
}
public sealed class NoPreviews : ILinkPreviews
{
public Task Wanted(PrivaPub.Models.Post.Post post, CancellationToken token) => Task.CompletedTask;
}
}
+117
View File
@@ -0,0 +1,117 @@
using PrivaPub.Infrastructure.Cli;
using System.Net;
using System.Net.Http.Headers;
using System.Net.Http.Json;
using System.Text.Json.Nodes;
using System.Text.RegularExpressions;
namespace PrivaPub.Tests.Support.Host
{
public sealed record Root(string Id, string UserName, string Password, string Jwt);
public sealed record Persona(string Id, string UserName, Root Root);
public static partial class Accounts
{
public const string Password = "Test-Pass-1!";
const string OutOfBand = "urn:ietf:wg:oauth:2.0:oob";
public static async Task<Root> SignUp(this PrivaPubHost host, string name = "root")
{
var userName = $"{name}{Guid.NewGuid():N}"[..24];
using var client = host.Client();
var response = await client.PostAsJsonAsync("/clientapi/user/signup", new { userName, password = Password });
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
var jwt = (await response.Content.ReadFromJsonAsync<JsonObject>())!;
return new Root(jwt["userId"]!.GetValue<string>(), userName, Password, jwt["token"]!.GetValue<string>());
}
public static async Task<Root> LogIn(this PrivaPubHost host, Root root)
{
using var client = host.Client();
var response = await client.PostAsJsonAsync("/clientapi/user/login", new { userName = root.UserName, password = root.Password });
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
var jwt = (await response.Content.ReadFromJsonAsync<JsonObject>())!;
return root with { Jwt = jwt["token"]!.GetValue<string>() };
}
public static async Task<Root> Admin(this PrivaPubHost host)
{
var root = await host.SignUp("admin");
Assert.Equal(0, await AdminCommands.Run(new[] { "promote", root.UserName }));
return await host.LogIn(root);
}
public static async Task<Persona> Persona(this PrivaPubHost host, Root root, string name = "persona")
{
var userName = $"{name}{Guid.NewGuid():N}"[..20];
using var client = host.As(root.Jwt);
var response = await client.PostAsJsonAsync("/clientapi/avatar/private/insert", new { userName, name, biography = "testing" });
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
var avatar = (await response.Content.ReadFromJsonAsync<JsonObject>())!;
return new Persona(avatar["id"]!.GetValue<string>(), userName, root);
}
public static async Task<string> MastodonToken(this PrivaPubHost host, Persona persona, string scopes = "read write follow")
{
using var client = host.Client(cookies: true);
var app = await Form(client, "/api/v1/apps", ("client_name", "privapub-tests"), ("redirect_uris", OutOfBand), ("scopes", scopes));
var clientId = app["client_id"]!.GetValue<string>();
var clientSecret = app["client_secret"]!.GetValue<string>();
var query = $"client_id={Uri.EscapeDataString(clientId)}&redirect_uri={Uri.EscapeDataString(OutOfBand)}&response_type=code&scope={Uri.EscapeDataString(scopes)}";
var code = await Authorize(client, persona, query);
var token = await Form(client, "/oauth/token", ("grant_type", "authorization_code"), ("code", code), ("client_id", clientId),
("client_secret", clientSecret), ("redirect_uri", OutOfBand));
return token["access_token"]!.GetValue<string>();
}
public static async Task<string> Authorize(HttpClient client, Persona persona, string query, string decision = "allow")
{
var returnUrl = "/oauth/authorize?" + query;
var login = await client.GetStringAsync("/oauth/login?returnUrl=" + Uri.EscapeDataString(returnUrl));
var antiforgery = AntiforgeryToken().Match(login).Groups[1].Value;
var signedIn = await client.PostAsync("/oauth/login", new FormUrlEncodedContent(new Dictionary<string, string>
{
["returnUrl"] = returnUrl,
["__RequestVerificationToken"] = antiforgery,
["userName"] = persona.Root.UserName,
["password"] = persona.Root.Password
}));
Assert.Equal(HttpStatusCode.Redirect, signedIn.StatusCode);
var choose = await client.GetStringAsync(returnUrl + "&signed_in=1");
var fields = HiddenInput().Matches(choose).Select(m => new KeyValuePair<string, string>(m.Groups[1].Value, WebUtility.HtmlDecode(m.Groups[2].Value))).ToList();
fields.Add(new("avatarId", persona.Id));
fields.Add(new("decision", decision));
var answer = await client.PostAsync("/oauth/authorize", new FormUrlEncodedContent(fields));
var page = await answer.Content.ReadAsStringAsync();
return Code().Match(page) is { Success: true } match ? match.Groups[1].Value : default;
}
public static HttpClient As(this PrivaPubHost host, string bearer)
{
var client = host.Client();
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", bearer);
return client;
}
static async Task<JsonObject> Form(HttpClient client, string path, params (string Key, string Value)[] fields)
{
var response = await client.PostAsync(path, new FormUrlEncodedContent(fields.Select(f => new KeyValuePair<string, string>(f.Key, f.Value))));
var body = await response.Content.ReadAsStringAsync();
Assert.True(response.IsSuccessStatusCode, $"{path} answered {(int)response.StatusCode}: {body}");
return JsonNode.Parse(body)!.AsObject();
}
[GeneratedRegex("name=\"__RequestVerificationToken\" type=\"hidden\" value=\"([^\"]*)\"")]
private static partial Regex AntiforgeryToken();
[GeneratedRegex("<input type=\"hidden\" name=\"([^\"]*)\" value=\"([^\"]*)\"")]
private static partial Regex HiddenInput();
[GeneratedRegex("<code>([^<]*)</code>")]
private static partial Regex Code();
}
}
+64
View File
@@ -0,0 +1,64 @@
using Microsoft.Extensions.DependencyInjection;
using MongoDB.Entities;
using PrivaPub.Federation.Outbox;
using PrivaPub.Infrastructure.Jobs;
using PrivaPub.Models.Jobs;
using System.Linq.Expressions;
using System.Text.Json;
using System.Text.Json.Nodes;
namespace PrivaPub.Tests.Support.Host
{
public static class Jobs
{
public static async Task<int> Run(this PrivaPubHost host, Expression<Func<Job, bool>> which, CancellationToken token = default)
{
var handlers = host.Services.GetServices<IJobHandler>().ToDictionary(h => h.Kind);
var queue = host.Get<IJobQueue>();
var ran = 0;
for (var round = 0; round < 50; round++)
{
var pending = await DB.Default.Find<Job>().Match(which).Match(j => j.State == JobState.Pending).ExecuteAsync(token);
if (pending.Count == 0)
return ran;
foreach (var candidate in pending)
{
var job = await DB.Default.UpdateAndGet<Job>()
.Match(j => j.ID == candidate.ID && j.State == JobState.Pending)
.Modify(j => j.State, JobState.Running)
.Modify(j => j.LeasedUntil, DateTime.UtcNow + JobQueue.LeaseTime)
.Modify(b => b.Inc(j => j.Attempts, 1))
.ExecuteAsync(token);
if (job == default)
continue;
var handler = handlers[job.Kind];
JobOutcome outcome;
try
{
outcome = await handler.Handle(job, token);
}
catch (Exception ex)
{
outcome = JobOutcome.Dead(ex.GetType().Name + ": " + ex.Message);
}
await queue.Finish(job, outcome, handler.MaxAttempts, token);
ran++;
}
}
return ran;
}
public static Task<int> RunInbox(this PrivaPubHost host, string activityId, CancellationToken token = default) =>
host.Run(j => j.Kind == JobKind.ProcessInbox && j.DedupeKey == "inbox|" + activityId, token);
public static async Task<List<JsonObject>> Deliveries(string inbox, DateTime since, CancellationToken token = default) =>
(await DB.Default.Find<Job>().Match(j => j.Kind == JobKind.Deliver && j.CreatedAt >= since).ExecuteAsync(token))
.Select(j => JsonSerializer.Deserialize<DeliveryPayload>(j.Payload))
.Where(p => p.Inbox == inbox)
.Select(p => JsonNode.Parse(p.Body)!.AsObject())
.ToList();
}
}
+160
View File
@@ -0,0 +1,160 @@
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.DataProtection;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc.Testing;
using Microsoft.AspNetCore.TestHost;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using PrivaPub.Api.Mastodon.Auth;
using PrivaPub.Domain.Media;
using PrivaPub.Infrastructure.Jobs;
using System.Net;
namespace PrivaPub.Tests.Support.Host
{
public class PrivaPubHost : WebApplicationFactory<Program>
{
public const string Host = "privapub.test";
public const string Base = "https://" + Host;
public const string ClientHeader = "X-Test-Client";
static readonly SemaphoreSlim Boot = new(1, 1);
static readonly Type[] Unwanted = { typeof(JobWorker), typeof(MediaJanitor), typeof(OAuthPruner) };
static PrivaPubHost _shared;
readonly string _mediaRoot = Path.Combine(Path.GetTempPath(), $"privapub-tests-{Guid.NewGuid():N}");
public IReadOnlyList<ServiceDescriptor> Registered { get; private set; }
public static async Task<PrivaPubHost> Shared()
{
await Boot.WaitAsync();
try
{
if (_shared == default)
{
var host = new PrivaPubHost();
_ = host.Services;
_shared = host;
}
return _shared;
}
finally
{
Boot.Release();
}
}
protected virtual IEnumerable<KeyValuePair<string, string>> Settings() => new Dictionary<string, string>
{
["MongoSettings:ConnectionString"] = MongoFixture.Connection,
["MongoSettings:Database"] = MongoFixture.Database,
["MongoSettings:LogsDatabase"] = "logs",
["AppConfiguration:Version"] = "0.0.0-test",
["AppConfiguration:BackendBaseAddress"] = Base,
["AppConfiguration:MaxAllowedUploadFiles"] = "3",
["AppConfiguration:MaxAllowedFileSize"] = "2097152",
["AppConfiguration:SupportedLanguages:0"] = "en",
["AppConfiguration:SupportedLanguages:1"] = "it",
["AppConfiguration:HashingOptions:Iterations"] = "1000",
["AppConfiguration:Jwt:Key"] = "privapub-tests-only-signing-key-0123456789abcdef0123456789abcdef",
["AppConfiguration:Jwt:Issuer"] = Base,
["AppConfiguration:Jwt:Audience"] = Base,
["AppConfiguration:Jwt:HoursTimeout"] = "1",
["AppConfiguration:EmailConfiguration:SmtpServer"] = "127.0.0.1",
["AppConfiguration:EmailConfiguration:SmtpPort"] = "9",
["AppConfiguration:EmailConfiguration:UseSSL"] = "false",
["AppConfiguration:EmailConfiguration:SmtpUsername"] = "nobody@privapub.test",
["AppConfiguration:EmailConfiguration:SmtpPassword"] = "none",
["Federation:AllowPrivateNetworks"] = "true",
["Federation:AllowPlainHttp"] = "true",
["Federation:FetchLinkPreviews"] = "false",
["Media:Root"] = _mediaRoot,
["Logging:LogLevel:Default"] = "Warning",
["Serilog:MinimumLevel:Default"] = Environment.GetEnvironmentVariable("PRIVAPUB_TEST_LOGS") == "1" ? "Information" : "Fatal"
};
protected override void ConfigureWebHost(IWebHostBuilder builder)
{
builder.UseEnvironment("Testing");
foreach (var (key, value) in Settings())
builder.UseSetting(key, value);
builder.ConfigureTestServices(services =>
{
foreach (var hosted in services.Where(s => s.ServiceType == typeof(IHostedService) && Unwanted.Contains(s.ImplementationType)).ToList())
services.Remove(hosted);
services.AddDataProtection().UseEphemeralDataProtectionProvider();
services.AddSingleton<IStartupFilter, ClientAddressFilter>();
Registered = services.ToList();
});
}
protected override void ConfigureClient(HttpClient client)
{
client.BaseAddress = new Uri(Base + "/");
client.DefaultRequestHeaders.Add(ClientHeader, $"10.{Random.Shared.Next(256)}.{Random.Shared.Next(256)}.{Random.Shared.Next(1, 255)}");
}
public HttpClient Client(bool cookies = false) => CreateClient(new WebApplicationFactoryClientOptions
{
BaseAddress = new Uri(Base + "/"),
AllowAutoRedirect = false,
HandleCookies = cookies
});
public T Get<T>() where T : notnull => Services.GetRequiredService<T>();
protected override void Dispose(bool disposing)
{
base.Dispose(disposing);
if (disposing && Directory.Exists(_mediaRoot))
Directory.Delete(_mediaRoot, recursive: true);
}
sealed class ClientAddressFilter : IStartupFilter
{
public Action<IApplicationBuilder> Configure(Action<IApplicationBuilder> next) => app =>
{
app.Use(async (context, call) =>
{
if (IPAddress.TryParse(context.Request.Headers[ClientHeader].ToString(), out var address))
context.Connection.RemoteIpAddress = address;
await call(context);
});
next(app);
};
}
}
public sealed class SecureModeHost : PrivaPubHost
{
static readonly SemaphoreSlim Boot = new(1, 1);
static SecureModeHost _shared;
public static new async Task<SecureModeHost> Shared()
{
await PrivaPubHost.Shared();
await Boot.WaitAsync();
try
{
if (_shared == default)
{
var host = new SecureModeHost();
_ = host.Services;
_shared = host;
}
return _shared;
}
finally
{
Boot.Release();
}
}
protected override IEnumerable<KeyValuePair<string, string>> Settings() =>
base.Settings().Append(new KeyValuePair<string, string>("Federation:SecureMode", "true"));
}
}
+40 -3
View File
@@ -13,15 +13,24 @@ namespace PrivaPub.Tests.Support
public sealed class MongoFixture : IAsyncLifetime
{
public const string Skip = "set PRIVAPUB_TEST_MONGOD=1 (and optionally PRIVAPUB_TEST_MONGO) to run the tests that need a mongod";
const int ProductionPort = 27022;
const int DefaultPort = 27017;
public static bool Enabled => Environment.GetEnvironmentVariable("PRIVAPUB_TEST_MONGOD") == "1";
static bool Required => Environment.GetEnvironmentVariable("PRIVAPUB_TEST_REQUIRE_MONGOD") == "1";
static bool InCi => !string.IsNullOrEmpty(Environment.GetEnvironmentVariable("CI")) || !string.IsNullOrEmpty(Environment.GetEnvironmentVariable("GITEA_ACTIONS"));
public string Database { get; } = $"PrivaPubTests_{Guid.NewGuid():N}";
public static string Connection { get; } = Environment.GetEnvironmentVariable("PRIVAPUB_TEST_MONGO") ?? $"mongodb://127.0.0.1:{DefaultPort}";
public static string Database { get; } = $"PrivaPubTests_{Guid.NewGuid():N}";
public async ValueTask InitializeAsync()
{
if (Required && !Enabled)
throw new InvalidOperationException("PRIVAPUB_TEST_REQUIRE_MONGOD=1 but PRIVAPUB_TEST_MONGOD is not 1: the integration tests would be skipped");
if (!Enabled)
return;
var settings = MongoClientSettings.FromConnectionString(Connection);
Refuse(settings);
try
{
BsonSerializer.RegisterSerializer(new GuidSerializer(GuidRepresentation.Standard));
@@ -29,9 +38,10 @@ namespace PrivaPub.Tests.Support
catch (BsonSerializationException)
{
}
var connection = Environment.GetEnvironmentVariable("PRIVAPUB_TEST_MONGO") ?? "mongodb://127.0.0.1:27017";
await DB.InitAsync(Database, MongoClientSettings.FromConnectionString(connection));
await DB.InitAsync(Database, settings);
await RefuseProductionData();
EntityMaps.Warm();
await DB.Default.MigrateAsync<Program>();
await Indexes.Create();
}
@@ -40,5 +50,32 @@ namespace PrivaPub.Tests.Support
if (Enabled)
await DB.Default.Database().Client.DropDatabaseAsync(Database);
}
static void Refuse(MongoClientSettings settings)
{
var ports = settings.Servers.Select(s => s.Port).ToList();
if (ports.Contains(ProductionPort))
throw new InvalidOperationException($"the tests never run against port {ProductionPort}: that is production's mongod");
if (InCi && (Environment.GetEnvironmentVariable("PRIVAPUB_TEST_MONGO") == default || ports.Contains(DefaultPort)))
throw new InvalidOperationException("in CI the tests run only against a throwaway mongod: use tools/ci/with-test-mongod.sh");
}
static async Task RefuseProductionData()
{
BsonDocument options;
try
{
options = await DB.Default.Database().Client.GetDatabase("admin").RunCommandAsync<BsonDocument>(new BsonDocument("getCmdLineOpts", 1));
}
catch (MongoCommandException)
{
return;
}
var dbPath = options.GetValue("parsed", new BsonDocument()).AsBsonDocument
.GetValue("storage", new BsonDocument()).AsBsonDocument
.GetValue("dbPath", BsonNull.Value);
if (dbPath.IsString && dbPath.AsString.StartsWith("/var/lib/privapub", StringComparison.Ordinal))
throw new InvalidOperationException($"the mongod at {Connection} keeps its data in {dbPath.AsString}: that is production's");
}
}
}
+14 -8
View File
@@ -17,7 +17,7 @@ namespace PrivaPub.Tests.Support
public sealed class Peer : IAsyncDisposable
{
readonly WebApplication _app;
readonly ConcurrentDictionary<string, string> _documents = new();
readonly ConcurrentDictionary<string, (string Text, string ContentType)> _documents = new();
readonly ConcurrentDictionary<string, (int Status, TimeSpan Delay)> _answers = new();
readonly ConcurrentDictionary<string, (byte[] Bytes, string ContentType)> _files = new();
@@ -40,12 +40,15 @@ namespace PrivaPub.Tests.Support
Peer peer = default;
app.Run(async context =>
{
peer.Requests.Enqueue(new(context.Request.Method, context.Request.Path, context.Request.Headers["Signature"].ToString()));
var body = context.Request.ContentLength > 0 || context.Request.Headers.TransferEncoding.Count > 0
? await new StreamReader(context.Request.Body).ReadToEndAsync()
: default;
peer.Requests.Enqueue(new(context.Request.Method, context.Request.Path, context.Request.Headers["Signature"].ToString(), body,
context.Request.Headers.ToDictionary(h => h.Key, h => h.Value.ToString(), StringComparer.OrdinalIgnoreCase)));
var key = context.Request.Path.Value;
if (peer._files.TryGetValue(key, out var file))
{
context.Response.ContentType = file.ContentType;
await context.Response.Body.WriteAsync(file.Bytes);
await Results.Bytes(file.Bytes, file.ContentType, enableRangeProcessing: true).ExecuteAsync(context);
return;
}
if (peer._answers.TryGetValue(key, out var answer))
@@ -60,15 +63,17 @@ namespace PrivaPub.Tests.Support
context.Response.StatusCode = StatusCodes.Status404NotFound;
return;
}
context.Response.ContentType = "application/activity+json";
await context.Response.WriteAsync(document.Replace("{A}", peer.A).Replace("{B}", peer.B));
context.Response.ContentType = document.ContentType;
await context.Response.WriteAsync(document.Text.Replace("{A}", peer.A).Replace("{B}", peer.B));
});
await app.StartAsync();
peer = new Peer(app, new Uri(app.Urls.First()).Port);
return peer;
}
public void Serve(string path, string json) => _documents[path] = json;
public void Serve(string path, string json) => _documents[path] = (json, "application/activity+json");
public void ServeText(string path, string text, string contentType) => _documents[path] = (text, contentType);
public void ServeFile(string path, byte[] bytes, string contentType) => _files[path] = (bytes, contentType);
@@ -95,7 +100,8 @@ namespace PrivaPub.Tests.Support
public Task Reload(CancellationToken token) => Task.CompletedTask;
}
public sealed record HttpRequestRecord(string Method, string Path, string Signature);
public sealed record HttpRequestRecord(string Method, string Path, string Signature, string Body = default,
IReadOnlyDictionary<string, string> Headers = default);
public sealed class StaticOptions<T> : IOptionsMonitor<T>
{
+32 -7
View File
@@ -44,18 +44,46 @@ namespace PrivaPub.Tests.Support
}
};
public HttpRequestMessage SignedGet(string path, string host = "privapub.test")
{
var date = DateTimeOffset.UtcNow.ToString("r", CultureInfo.InvariantCulture);
var request = new HttpRequestMessage(HttpMethod.Get, path);
request.Headers.TryAddWithoutValidation("Date", date);
request.Headers.TryAddWithoutValidation("Accept", "application/activity+json");
request.Headers.TryAddWithoutValidation("Signature", Signature($"(request-target): get {path}\nhost: {host}\ndate: {date}", "(request-target) host date"));
return request;
}
public HttpRequestMessage SignedPost(string path, JsonNode activity, string host = "privapub.test", string date = default)
{
var body = Encoding.UTF8.GetBytes(activity.ToJsonString());
date ??= DateTimeOffset.UtcNow.ToString("r", CultureInfo.InvariantCulture);
var digest = HttpSignatures.Digest(body);
var request = new HttpRequestMessage(HttpMethod.Post, path) { Content = new ByteArrayContent(body) };
request.Content.Headers.TryAddWithoutValidation("Content-Type", "application/activity+json");
request.Headers.TryAddWithoutValidation("Date", date);
request.Headers.TryAddWithoutValidation("Digest", digest);
request.Headers.TryAddWithoutValidation("Signature",
Signature($"(request-target): post {path}\nhost: {host}\ndate: {date}\ndigest: {digest}", "(request-target) host date digest"));
return request;
}
string Signature(string signingString, string headers)
{
var signature = Convert.ToBase64String(_key.SignData(Encoding.UTF8.GetBytes(signingString), HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1));
return $"keyId=\"{KeyId}\",algorithm=\"rsa-sha256\",headers=\"{headers}\",signature=\"{signature}\"";
}
public HttpRequest Get(string host, string path)
{
var date = DateTimeOffset.UtcNow.ToString("r", CultureInfo.InvariantCulture);
var signingString = $"(request-target): get {path}\nhost: {host}\ndate: {date}";
var signature = Convert.ToBase64String(_key.SignData(Encoding.UTF8.GetBytes(signingString), HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1));
var context = new DefaultHttpContext();
context.Request.Method = "GET";
context.Request.Host = new HostString(host);
context.Request.Path = path;
context.Features.Get<IHttpRequestFeature>().RawTarget = path;
context.Request.Headers["Date"] = date;
context.Request.Headers["Signature"] = $"keyId=\"{KeyId}\",algorithm=\"rsa-sha256\",headers=\"(request-target) host date\",signature=\"{signature}\"";
context.Request.Headers["Signature"] = Signature($"(request-target): get {path}\nhost: {host}\ndate: {date}", "(request-target) host date");
return context.Request;
}
@@ -64,9 +92,6 @@ namespace PrivaPub.Tests.Support
var body = Encoding.UTF8.GetBytes(activity.ToJsonString());
var date = DateTimeOffset.UtcNow.ToString("r", CultureInfo.InvariantCulture);
var digest = HttpSignatures.Digest(body);
var signingString = $"(request-target): post {path}\nhost: {host}\ndate: {date}\ndigest: {digest}";
var signature = Convert.ToBase64String(_key.SignData(Encoding.UTF8.GetBytes(signingString), HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1));
var context = new DefaultHttpContext();
context.Request.Method = "POST";
context.Request.Host = new HostString(host);
@@ -75,7 +100,7 @@ namespace PrivaPub.Tests.Support
context.Request.Headers["Date"] = date;
context.Request.Headers["Digest"] = digest;
context.Request.Headers["Content-Type"] = "application/activity+json";
context.Request.Headers["Signature"] = $"keyId=\"{KeyId}\",algorithm=\"rsa-sha256\",headers=\"(request-target) host date digest\",signature=\"{signature}\"";
context.Request.Headers["Signature"] = Signature($"(request-target): post {path}\nhost: {host}\ndate: {date}\ndigest: {digest}", "(request-target) host date digest");
context.Request.Body = new MemoryStream(body);
context.Request.ContentLength = body.Length;
return context.Request;
@@ -79,6 +79,8 @@ namespace PrivaPub.Api.Mastodon.Controllers
avatar.Settings.DefaultSensitive = sensitive;
if (Params.Has("source[language]"))
avatar.Settings.DefaultLanguage = Params.Get("source[language]");
if (Params.Get("source[quote_policy]") is { } quotePolicy && QuotePolicies.IsKnown(quotePolicy))
avatar.Settings.QuotePolicy = quotePolicy;
var fields = new Dictionary<string, string>();
for (var i = 0; i < 4; i++)
{
@@ -73,6 +73,7 @@ namespace PrivaPub.Api.Mastodon.Controllers
domain = Domain,
title = "PrivaPub",
version = Version,
api_versions = new { mastodon = 7 },
source_url = "https://git.thepra.dev/thepra/SocialPub",
description = Description,
usage = new { users = new { active_month = 0 } },
@@ -60,13 +60,38 @@ namespace PrivaPub.Api.Mastodon.Controllers
[HttpGet("/media/proxy/{signature}/{encoded}"), AllowAnonymous, ApiExplorerSettings(IgnoreApi = true)]
public async Task<IActionResult> Proxy(string signature, string encoded, CancellationToken token)
{
var (path, contentType) = await _proxy.Fetch(signature, encoded, token);
if (path == default)
var url = _proxy.Verified(signature, encoded);
if (url == default)
return NotFound();
Response.Headers["X-Content-Type-Options"] = "nosniff";
Response.Headers["Content-Security-Policy"] = "default-src 'none'; sandbox";
Response.Headers["Cache-Control"] = "public, max-age=604800";
return PhysicalFile(path, contentType);
if (_proxy.Cached(url) is { Path: not null } cached)
return PhysicalFile(cached.Path, cached.ContentType, enableRangeProcessing: true);
if (Request.Headers.Range.Count == 0)
{
var (path, contentType) = await _proxy.Fetch(signature, encoded, token);
if (path != default)
return PhysicalFile(path, contentType, enableRangeProcessing: true);
}
return await Stream(url, token);
}
async Task<IActionResult> Stream(string url, CancellationToken token)
{
var range = System.Net.Http.Headers.RangeHeaderValue.TryParse(Request.Headers.Range.ToString(), out var asked) ? asked : default;
using var upstream = await _proxy.Open(url, range, token);
if (upstream == default)
return NotFound();
Response.StatusCode = (int)upstream.StatusCode;
Response.ContentType = upstream.Content.Headers.ContentType?.ToString() ?? "application/octet-stream";
if (upstream.Content.Headers.ContentLength is { } length)
Response.ContentLength = length;
if (upstream.Content.Headers.ContentRange is { } contentRange)
Response.Headers.ContentRange = contentRange.ToString();
Response.Headers.AcceptRanges = "bytes";
await upstream.Content.CopyToAsync(Response.Body, token);
return new EmptyResult();
}
object View(MediaAttachment attachment) => new
@@ -9,6 +9,9 @@ using PrivaPub.Api.Mastodon.Mappers;
using PrivaPub.Domain.Privacy;
using PrivaPub.Domain.Statuses;
using PrivaPub.Models.Post;
using PrivaPub.Models.User;
using PrivaPub.Federation.Rendering;
using PrivaPub.Federation.Outbox;
using PrivaPub.StaticServices;
using PostEntity = PrivaPub.Models.Post.Post;
@@ -24,9 +27,14 @@ namespace PrivaPub.Api.Mastodon.Controllers
readonly MastodonMapper _mapper;
readonly DbEntities _dbEntities;
readonly IMemoryCache _cache;
readonly IQuoteService _quotes;
readonly IOutboxPublisher _outbox;
public StatusesController(IStatusService statuses, MastodonMapper mapper, DbEntities dbEntities, IMemoryCache cache)
public StatusesController(IStatusService statuses, MastodonMapper mapper, DbEntities dbEntities, IMemoryCache cache, IQuoteService quotes,
IOutboxPublisher outbox)
{
_quotes = quotes;
_outbox = outbox;
_statuses = statuses;
_mapper = mapper;
_dbEntities = dbEntities;
@@ -54,6 +62,8 @@ namespace PrivaPub.Api.Mastodon.Controllers
InReplyTo = Params.Get("in_reply_to_id"),
Language = Params.Get("language") ?? Me.Settings.DefaultLanguage,
MediaIds = Params.List("media_ids"),
QuotedStatusId = Params.Get("quoted_status_id"),
QuotePolicy = Params.Get("quote_approval_policy"),
Poll = Params.Has("poll[options]")
? new PollDraft(Params.List("poll[options]"), Params.Int("poll[expires_in]") ?? 0, Params.Bool("poll[multiple]") == true,
Params.Bool("poll[hide_totals]") == true)
@@ -226,6 +236,47 @@ namespace PrivaPub.Api.Mastodon.Controllers
return Json(reblogs.Select(r => accounts.GetValueOrDefault(MastodonMapper.AuthorOf(r))).Where(a => a != default).ToList());
}
[HttpGet("/api/v1/statuses/{id}/quotes"), Scope("read:statuses", requiresUser: false), Microsoft.AspNetCore.Authorization.AllowAnonymous]
public async Task<IActionResult> Quotes(string id, CancellationToken token)
{
if (await Visible(id, token) == default)
return NotFoundError();
var query = _dbEntities.Posts.Match(p => p.QuotedPostId == id && p.QuoteState == QuoteState.Accepted && !p.DeletedAt.HasValue);
var quotes = await Page.From(Params, Limit()).Fetch(query, p => p.ID, token);
var visible = new List<PostEntity>();
foreach (var quote in quotes)
if (quote.Visibility != PostVisibility.LocalGeo && await VisibilityPolicy.CanSee(quote, MyId, token))
visible.Add(quote);
Link($"/api/v1/statuses/{id}/quotes", quotes.LastOrDefault()?.ID, quotes.FirstOrDefault()?.ID);
return Json(await _mapper.Statuses(visible, MyId, token));
}
[HttpPost("/api/v1/statuses/{id}/quotes/{quotingId}/revoke"), Scope("write:statuses")]
public async Task<IActionResult> RevokeQuote(string id, string quotingId, CancellationToken token)
{
var quoted = await _dbEntities.Posts.Match(p => p.ID == id && p.GroupUserId == MyId && !p.IsFederatedCopy && !p.DeletedAt.HasValue).ExecuteFirstAsync(token);
var quoting = quoted == default ? default : await _dbEntities.Posts.Match(p => p.ID == quotingId && p.QuotedPostId == quoted.ID).ExecuteFirstAsync(token);
if (quoting == default || !await _quotes.RevokeLicence(quoted, quoting, token))
return NotFoundError();
return Json(await _mapper.Status(await _dbEntities.Posts.MatchID(quoting.ID).ExecuteFirstAsync(token), MyId, token));
}
[HttpPut("/api/v1/statuses/{id}/interaction_policy"), Scope("write:statuses")]
public async Task<IActionResult> InteractionPolicy(string id, CancellationToken token)
{
var post = await _dbEntities.Posts.Match(p => p.ID == id && p.GroupUserId == MyId && !p.IsFederatedCopy && !p.DeletedAt.HasValue).ExecuteFirstAsync(token);
if (post == default)
return NotFoundError();
var policy = Params.Get("quote_approval_policy");
if (!QuotePolicies.IsKnown(policy))
return Error(StatusCodes.Status422UnprocessableEntity, "Validation failed: Unknown quote approval policy");
await DB.Default.Update<PostEntity>().MatchID(post.ID).Modify(p => p.LocalQuotePolicy, policy).ExecuteAsync(token);
post.LocalQuotePolicy = policy;
if (!post.IsLocalOnly)
await _outbox.Publish(Me, post, ActivityPubRenderer.UpdateOf(post, Me, $"policy-{DateTime.UtcNow.Ticks}"), token);
return Json(await _mapper.Status(post, MyId, token));
}
[HttpPost("/api/v1/statuses/{id}/bookmark"), Scope("write:bookmarks")]
public async Task<IActionResult> Bookmark(string id, CancellationToken token)
{
@@ -164,7 +164,8 @@ namespace PrivaPub.Api.Mastodon.Controllers
[NotificationType.Reblog] = "reblog",
[NotificationType.Update] = "update",
[NotificationType.Poll] = "poll",
[NotificationType.Reaction] = "pleroma:emoji_reaction"
[NotificationType.Reaction] = "pleroma:emoji_reaction",
[NotificationType.Quote] = "quote"
};
readonly MastodonMapper _mapper;
@@ -59,6 +59,7 @@ namespace PrivaPub.Api.Mastodon.Entities
public List<Field> Fields { get; set; } = new();
public int FollowRequestsCount { get; set; }
public string AttributionDomains { get; set; }
public string QuotePolicy { get; set; } = "public";
}
public class PrivaPubStatus
@@ -161,6 +162,20 @@ namespace PrivaPub.Api.Mastodon.Entities
public int Down { get; set; }
}
public class QuoteEntity
{
public string State { get; set; }
public Status QuotedStatus { get; set; }
public string QuotedStatusId { get; set; }
}
public class QuoteApprovalEntity
{
public List<string> Automatic { get; set; } = new();
public List<string> Manual { get; set; } = new();
public string CurrentUser { get; set; } = "denied";
}
public class EmojiReactionEntity
{
public string Name { get; set; }
@@ -253,6 +268,9 @@ namespace PrivaPub.Api.Mastodon.Entities
public PrivaPubStatus Privapub { get; set; }
public List<EmojiReactionEntity> EmojiReactions { get; set; } = new();
public PleromaStatus Pleroma { get; set; }
public QuoteEntity Quote { get; set; }
public int QuotesCount { get; set; }
public QuoteApprovalEntity QuoteApproval { get; set; }
}
public class MediaAttachment
@@ -0,0 +1,29 @@
using System.Text.Json;
namespace PrivaPub.Api.Mastodon.Infrastructure
{
public static class MastodonErrors
{
public static IApplicationBuilder UseMastodonErrorBodies(this IApplicationBuilder app) =>
app.UseWhen(context => context.Request.Path.StartsWithSegments("/api"), api => api.UseStatusCodePages(async pages =>
{
var response = pages.HttpContext.Response;
response.ContentType = "application/json; charset=utf-8";
await response.WriteAsync(JsonSerializer.Serialize(new { error = Message(response.StatusCode) }, MastodonJson.Options));
}));
public static string Message(int status) => status switch
{
StatusCodes.Status401Unauthorized => "The access token is invalid",
StatusCodes.Status403Forbidden => "This action is outside the authorized scopes",
StatusCodes.Status404NotFound => "Record not found",
StatusCodes.Status405MethodNotAllowed => "Method not allowed",
StatusCodes.Status413PayloadTooLarge => "Payload too large",
StatusCodes.Status415UnsupportedMediaType => "Unsupported media type",
StatusCodes.Status422UnprocessableEntity => "Validation failed",
StatusCodes.Status429TooManyRequests => "Too many requests",
>= 500 => "Something went wrong.",
_ => "Bad request"
};
}
}
+118 -5
View File
@@ -7,6 +7,8 @@ using PrivaPub.Federation.Actors;
using PrivaPub.Federation.Rendering;
using PrivaPub.Models.Federation;
using PrivaPub.Models.Group;
using PrivaPub.Domain.Privacy;
using PrivaPub.Federation.Objects;
using PrivaPub.Models.Post;
using PrivaPub.Domain.Statuses;
using PrivaPub.Models.Social;
@@ -91,6 +93,7 @@ namespace PrivaPub.Api.Mastodon.Mappers
Privacy = actor.Settings.DefaultVisibility ?? "public",
Sensitive = actor.Settings.DefaultSensitive,
Language = actor.Settings.DefaultLanguage,
QuotePolicy = actor.Settings.QuotePolicy ?? QuotePolicies.Public,
Note = actor.Summary ?? string.Empty,
Fields = actor.Fields.Select(f => new Field { Name = f.Key, Value = f.Value }).ToList(),
FollowRequestsCount = (int)await DB.Default.CountAsync<Follower>(f => f.LocalActorId == actor.Id && !f.IsAccepted, token)
@@ -146,7 +149,10 @@ namespace PrivaPub.Api.Mastodon.Mappers
public async Task<Status> Status(PostEntity post, string viewerId, CancellationToken token) =>
(await Statuses(new[] { post }, viewerId, token)).FirstOrDefault();
public async Task<List<Status>> Statuses(IReadOnlyCollection<PostEntity> posts, string viewerId, CancellationToken token)
public Task<List<Status>> Statuses(IReadOnlyCollection<PostEntity> posts, string viewerId, CancellationToken token) =>
Statuses(posts, viewerId, nested: false, token);
async Task<List<Status>> Statuses(IReadOnlyCollection<PostEntity> posts, string viewerId, bool nested, CancellationToken token)
{
var originalIds = posts.Where(p => p.ReblogOfPostId != default).Select(p => p.ReblogOfPostId).Distinct().ToList();
var originals = originalIds.Count == 0
@@ -220,7 +226,7 @@ namespace PrivaPub.Api.Mastodon.Mappers
SpoilerText = post.SpoilerText ?? (post.HasContentWarning ? post.Title ?? ActivityPubRenderer.ContentWarning : string.Empty),
Visibility = Visibility(post.Visibility),
Language = post.Language,
MediaAttachments = post.Media.Select(Media).ToList(),
MediaAttachments = post.Media.Count > 0 ? post.Media.Select(Media).ToList() : Playable(post),
Card = Card(post),
Poll = Poll(post, viewerId, ownVotes.GetValueOrDefault(post.ID)),
Emojis = Emojis(post.Emojis),
@@ -233,6 +239,34 @@ namespace PrivaPub.Api.Mastodon.Mappers
return status;
}
var localAuthors = all.Where(p => !p.IsFederatedCopy && p.GroupUserId != default).Select(p => p.GroupUserId).Distinct().ToList();
var followsAuthor = viewerId == default || localAuthors.Count == 0
? new HashSet<string>()
: await FollowedLocalAuthors(viewerId, localAuthors, token);
var quotedIds = nested ? new List<string>() : all.Where(p => p.QuoteState == QuoteState.Accepted && p.QuotedPostId != default).Select(p => p.QuotedPostId).Distinct().ToList();
var quotedPosts = new List<PostEntity>();
foreach (var quotedPost in quotedIds.Count == 0 ? new List<PostEntity>() : await _dbEntities.Posts.Match(p => quotedIds.Contains(p.ID) && !p.DeletedAt.HasValue).ExecuteAsync(token))
if (quotedPost.Visibility != PostVisibility.LocalGeo && await VisibilityPolicy.CanSee(quotedPost, viewerId, token))
quotedPosts.Add(quotedPost);
var quotedStatuses = quotedPosts.Count == 0
? new Dictionary<string, Status>()
: (await Statuses(quotedPosts, viewerId, nested: true, token)).ToDictionary(q => q.Id);
void Quote(Status status, PostEntity post)
{
status.QuotesCount = post.QuotesCount;
status.QuoteApproval = post.IsFederatedCopy ? Approval(post, viewerId) : LocalApproval(post, viewerId, followsAuthor.Contains(post.GroupUserId));
if (post.QuoteState == QuoteState.None)
return;
var state = post.QuoteState.ToString().ToLowerInvariant();
var quoted = post.QuotedPostId != default && quotedStatuses.TryGetValue(post.QuotedPostId, out var shown) ? shown : default;
status.Quote = nested
? new QuoteEntity { State = state, QuotedStatusId = post.QuotedPostId }
: new QuoteEntity { State = state, QuotedStatus = post.QuoteState == QuoteState.Accepted ? quoted : default, QuotedStatusId = post.QuotedPostId };
if (!nested && post.QuoteState == QuoteState.Accepted && quoted != default)
status.Content = WithoutQuoteFallback(status.Content);
}
var mapped = new List<Status>();
foreach (var post in posts)
{
@@ -242,10 +276,12 @@ namespace PrivaPub.Api.Mastodon.Mappers
var status = Map(post);
if (status == default)
continue;
Quote(status, post);
if (post.ReblogOfPostId != default)
{
if (!originals.TryGetValue(post.ReblogOfPostId, out var original) || Map(original) is not { } inner)
continue;
Quote(inner, original);
status.Reblog = inner;
status.Content = string.Empty;
status.Reblogged = reblogged.Contains(original.ID);
@@ -257,6 +293,54 @@ namespace PrivaPub.Api.Mastodon.Mappers
public static string AuthorOf(PostEntity post) => post.AuthorAccountId ?? post.GroupUserId;
static readonly AngleSharp.Html.Parser.HtmlParser Fragments = new();
public static string WithoutQuoteFallback(string html)
{
if (string.IsNullOrEmpty(html) || !html.Contains("quote-inline", StringComparison.Ordinal))
return html;
var document = Fragments.ParseDocument($"<body>{html}</body>");
foreach (var fallback in document.QuerySelectorAll(".quote-inline").ToList())
fallback.Remove();
return document.Body!.InnerHtml;
}
async Task<HashSet<string>> FollowedLocalAuthors(string viewerId, List<string> authorIds, CancellationToken token)
{
var uris = new Dictionary<string, string>();
foreach (var avatar in await _dbEntities.Avatars.Match(a => authorIds.Contains(a.ID)).ExecuteAsync(token))
uris[_localActors.FromAvatar(avatar).Uri] = avatar.ID;
var targets = uris.Keys.ToList();
return (await _dbEntities.Followings.Match(f => f.AvatarId == viewerId && targets.Contains(f.TargetActorURI) && f.State == FollowState.Accepted).ExecuteAsync(token))
.Select(f => uris[f.TargetActorURI]).ToHashSet();
}
static QuoteApprovalEntity LocalApproval(PostEntity post, string viewerId, bool viewerFollows)
{
var policy = ActivityPubRenderer.QuotableBy(post);
var everyone = policy switch { QuotePolicies.Public => new List<string> { "public" }, QuotePolicies.Followers => new List<string> { "followers" }, _ => new List<string>() };
var current = viewerId == default ? "unknown"
: viewerId == post.GroupUserId && post.Visibility is PostVisibility.Public or PostVisibility.Unlisted ? "automatic"
: policy == QuotePolicies.Public || policy == QuotePolicies.Followers && viewerFollows ? "automatic"
: "denied";
return new QuoteApprovalEntity { Automatic = everyone, CurrentUser = current };
}
static QuoteApprovalEntity Approval(PostEntity post, string viewerId)
{
if (!post.IsFederatedCopy || post.Visibility is not (PostVisibility.Public or PostVisibility.Unlisted))
return new QuoteApprovalEntity();
if (post.QuotePolicy is not { } policy)
return new QuoteApprovalEntity { Automatic = new List<string> { "public" }, CurrentUser = viewerId == default ? "unknown" : "automatic" };
static List<string> Named(IEnumerable<string> who) => who.Select(w => Addressing.IsPublic(w) ? "public" : w.EndsWith("/followers") ? "followers" : "unsupported_policy").Distinct().ToList();
return new QuoteApprovalEntity
{
Automatic = Named(policy.Automatic),
Manual = Named(policy.Manual),
CurrentUser = viewerId == default ? "unknown" : policy.Automatic.Any(Addressing.IsPublic) ? "automatic" : policy.Manual.Any(Addressing.IsPublic) ? "manual" : "denied"
};
}
public static string Content(PostEntity post)
{
var content = post.ContentHtml ?? ActivityPubRenderer.Html(post.Text);
@@ -387,13 +471,42 @@ namespace PrivaPub.Api.Mastodon.Mappers
Streams = variant.Streams
};
List<MediaAttachment> Playable(PostEntity post)
{
var (variant, kind, duration) = post.Video is { } video ? (BestVideo(video.Variants), "video", video.DurationSeconds)
: post.Audio is { } audio ? (audio.Variants.FirstOrDefault(), "audio", audio.DurationSeconds)
: default;
if (variant == default)
return new List<MediaAttachment>();
return new List<MediaAttachment>
{
Media(new PostMedia
{
Id = new Guid(System.Security.Cryptography.MD5.HashData(System.Text.Encoding.UTF8.GetBytes(variant.Url))),
RemoteURL = variant.Url,
ContentType = variant.MediaType ?? kind + "/*",
RemotePreviewURL = post.CoverURL,
Width = variant.Width,
Height = variant.Height,
DurationSeconds = duration,
Description = post.Title
})
};
}
public static MediaVariant BestVideo(IReadOnlyCollection<MediaVariant> variants)
{
var playable = variants.Where(v => v.MediaType?.StartsWith("video/") == true && (v.Streams.Count == 0 || v.Streams.Contains("audio") && v.Streams.Contains("video"))).ToList();
return playable.Where(v => v.Height <= 720).OrderByDescending(v => v.Height).FirstOrDefault()
?? playable.OrderBy(v => v.Height ?? int.MaxValue).FirstOrDefault();
}
PreviewCard Card(PostEntity post)
{
if (!post.IsFederatedCopy)
return default;
if (post.Link is { } link)
return Card(link.Href, link.Title ?? post.Title, link.Description ?? (post.Media.Count == 0 ? post.Excerpt : default), link.ImageURL);
if (post.ObjectType is "Article" or "Event" or "Video" or "Audio" && post.Url != default && post.Media.Count == 0)
if (post.IsFederatedCopy && post.ObjectType is "Article" or "Event" && post.Url != default && post.Media.Count == 0
|| post.IsFederatedCopy && post.ObjectType is "Video" or "Audio" && post.Url != default && post.Media.Count == 0 && Playable(post).Count == 0)
return Card(post.Url, post.Title, post.Excerpt, post.CoverURL);
return default;
}
+155
View File
@@ -0,0 +1,155 @@
using AngleSharp.Html.Parser;
using Microsoft.Extensions.Options;
using MongoDB.Entities;
using PrivaPub.Federation.Actors;
using PrivaPub.Federation.Objects;
using PrivaPub.Infrastructure.Http;
using PrivaPub.Infrastructure.Jobs;
using PrivaPub.Models.Jobs;
using PrivaPub.Models.Post;
using PrivaPub.StaticServices;
using System.Security.Cryptography;
using PostEntity = PrivaPub.Models.Post.Post;
namespace PrivaPub.Domain.Content
{
public class LinkPreview : Entity
{
public string Url { get; set; }
public string Title { get; set; }
public string Description { get; set; }
public string ImageURL { get; set; }
public string SiteName { get; set; }
public bool Failed { get; set; }
public DateTime FetchedAt { get; set; } = DateTime.UtcNow;
}
public interface ILinkPreviews
{
Task Wanted(PostEntity post, CancellationToken token);
}
public class LinkPreviews : ILinkPreviews, IJobHandler
{
const int MaxJitterSeconds = 60;
static readonly TimeSpan Freshness = TimeSpan.FromDays(7);
static readonly HtmlParser Parser = new();
readonly DbEntities _dbEntities;
readonly ILocalActorService _localActors;
readonly IFederationHttp _http;
readonly IJobQueue _queue;
readonly IOptionsMonitor<FederationOptions> _options;
public LinkPreviews(DbEntities dbEntities, ILocalActorService localActors, IFederationHttp http, IJobQueue queue,
IOptionsMonitor<FederationOptions> options)
{
_dbEntities = dbEntities;
_localActors = localActors;
_http = http;
_queue = queue;
_options = options;
}
public JobKind Kind => JobKind.FetchPreview;
public int Concurrency => 2;
public int MaxAttempts => 1;
public int PerHostLimit => 1;
public async Task Wanted(PostEntity post, CancellationToken token)
{
if (!_options.CurrentValue.FetchLinkPreviews || post.Visibility is not (PostVisibility.Public or PostVisibility.Unlisted) || post.IsLocalOnly
|| post.ReblogOfPostId != default || post.Link?.Title != default || post.Link == default && post.Media.Count > 0)
return;
var url = post.Link?.Href ?? FirstLink(post, _localActors.BaseAddress);
if (url == default)
return;
await _queue.EnqueueMany(new[]
{
new Job
{
Kind = JobKind.FetchPreview, Payload = post.ID, Host = new Uri(url).Host.ToLowerInvariant(), DedupeKey = $"preview|{post.ID}",
RunAt = DateTime.UtcNow.AddSeconds(RandomNumberGenerator.GetInt32(0, MaxJitterSeconds + 1))
}
}, token);
}
public async Task<JobOutcome> Handle(Job job, CancellationToken token)
{
var post = await _dbEntities.Posts.MatchID(job.Payload).ExecuteFirstAsync(token);
if (post == default || post.DeletedAt.HasValue || post.Link?.Title != default)
return JobOutcome.Done;
var url = post.Link?.Href ?? FirstLink(post, _localActors.BaseAddress);
if (url == default)
return JobOutcome.Done;
var preview = await DB.Default.Find<LinkPreview>().Match(p => p.Url == url).ExecuteFirstAsync(token);
if (preview == default || preview.FetchedAt < DateTime.UtcNow - Freshness)
preview = await Fetch(url, token);
if (preview == default || preview.Failed)
return JobOutcome.Done;
var link = post.Link ?? new PostLink { Href = url };
link.Title ??= preview.Title;
link.Description ??= preview.Description;
link.ImageURL ??= preview.ImageURL;
await DB.Default.Update<PostEntity>().MatchID(post.ID).Modify(p => p.Link, link).ExecuteAsync(token);
return JobOutcome.Done;
}
async Task<LinkPreview> Fetch(string url, CancellationToken token)
{
var (finalUri, html) = await _http.GetPage(url, token);
var preview = html == default ? new LinkPreview { Url = url, Failed = true } : Read(url, finalUri, html);
await DB.Default.Update<LinkPreview>()
.Match(p => p.Url == url)
.Modify(p => p.Title, preview.Title)
.Modify(p => p.Description, preview.Description)
.Modify(p => p.ImageURL, preview.ImageURL)
.Modify(p => p.SiteName, preview.SiteName)
.Modify(p => p.Failed, preview.Failed)
.Modify(p => p.FetchedAt, DateTime.UtcNow)
.Option(o => o.IsUpsert = true)
.ExecuteAsync(token);
return preview;
}
public static LinkPreview Read(string url, Uri finalUri, string html)
{
var document = Parser.ParseDocument(html);
string Meta(params string[] names) => names
.Select(name => document.QuerySelector($"meta[property='{name}'], meta[name='{name}']")?.GetAttribute("content"))
.FirstOrDefault(value => !string.IsNullOrWhiteSpace(value));
var image = Meta("og:image:secure_url", "og:image", "og:image:url", "twitter:image", "twitter:image:src");
var imageUri = image != default && Uri.TryCreate(finalUri, image, out var resolved) && resolved.Scheme is "https" or "http" ? resolved.AbsoluteUri : default;
var preview = new LinkPreview
{
Url = url,
Title = ObjectShapes.Text(Meta("og:title", "twitter:title") ?? document.Title, 300),
Description = ObjectShapes.Text(Meta("og:description", "twitter:description", "description"), 1000),
SiteName = ObjectShapes.Text(Meta("og:site_name"), 200),
ImageURL = imageUri
};
preview.Failed = preview.Title == default && preview.Description == default && preview.ImageURL == default;
return preview;
}
public static string FirstLink(PostEntity post, string ownBase)
{
if (string.IsNullOrEmpty(post.ContentHtml))
return default;
var document = Parser.ParseDocument(post.ContentHtml);
return document.QuerySelectorAll("a[href]")
.Where(a => !(a.ClassList.Contains("mention") || a.ClassList.Contains("hashtag") || a.GetAttribute("rel")?.Contains("tag") == true
|| a.Closest(".quote-inline") != default))
.Select(a => a.GetAttribute("href"))
.FirstOrDefault(href => Uri.TryCreate(href, UriKind.Absolute, out var uri) && uri.Scheme is "https" or "http"
&& href != post.Url && href != post.ObjectURI && !href.StartsWith(ownBase + "/", StringComparison.OrdinalIgnoreCase));
}
}
}
+34 -12
View File
@@ -15,6 +15,9 @@ namespace PrivaPub.Domain.Media
{
string Wrap(string remoteUrl);
Task<(string Path, string ContentType)> Fetch(string signature, string encodedUrl, CancellationToken token);
string Verified(string signature, string encodedUrl);
(string Path, string ContentType) Cached(string url);
Task<HttpResponseMessage> Open(string url, System.Net.Http.Headers.RangeHeaderValue range, CancellationToken token);
}
public class MediaProxy : IMediaProxy
@@ -43,7 +46,7 @@ namespace PrivaPub.Domain.Media
return $"{_localActors.BaseAddress}/media/proxy/{Sign(remoteUrl)}/{encoded}";
}
public async Task<(string Path, string ContentType)> Fetch(string signature, string encodedUrl, CancellationToken token)
public string Verified(string signature, string encodedUrl)
{
string url;
try
@@ -54,19 +57,38 @@ namespace PrivaPub.Domain.Media
{
return default;
}
if (!CryptographicOperations.FixedTimeEquals(Encoding.ASCII.GetBytes(signature ?? string.Empty), Encoding.ASCII.GetBytes(Sign(url))))
return default;
var name = Convert.ToHexStringLower(SHA256.HashData(Encoding.UTF8.GetBytes(url)));
var directory = System.IO.Path.Combine(_media.ProxyRoot, name[..2]);
var path = System.IO.Path.Combine(directory, name);
var typePath = path + ".type";
if (File.Exists(path) && File.Exists(typePath))
{
File.SetLastWriteTimeUtc(path, DateTime.UtcNow);
return (path, await File.ReadAllTextAsync(typePath, token));
return CryptographicOperations.FixedTimeEquals(Encoding.ASCII.GetBytes(signature ?? string.Empty), Encoding.ASCII.GetBytes(Sign(url))) ? url : default;
}
public (string Path, string ContentType) Cached(string url)
{
var (path, typePath) = CachePaths(url);
if (!File.Exists(path) || !File.Exists(typePath))
return default;
File.SetLastWriteTimeUtc(path, DateTime.UtcNow);
return (path, File.ReadAllText(typePath));
}
public Task<HttpResponseMessage> Open(string url, System.Net.Http.Headers.RangeHeaderValue range, CancellationToken token) =>
_http.OpenMedia(url, range, token);
(string Path, string TypePath) CachePaths(string url)
{
var name = Convert.ToHexStringLower(SHA256.HashData(Encoding.UTF8.GetBytes(url)));
var path = System.IO.Path.Combine(_media.ProxyRoot, name[..2], name);
return (path, path + ".type");
}
public async Task<(string Path, string ContentType)> Fetch(string signature, string encodedUrl, CancellationToken token)
{
var url = Verified(signature, encodedUrl);
if (url == default)
return default;
if (Cached(url) is { Path: not null } cached)
return cached;
var (path, typePath) = CachePaths(url);
var directory = System.IO.Path.GetDirectoryName(path);
var (bytes, contentType) = await _http.GetMedia(url, _options.CurrentValue.MaxProxiedBytes, token);
if (bytes == default)
return default;
+1 -20
View File
@@ -233,30 +233,11 @@ namespace PrivaPub.Domain.Statuses
return JobOutcome.Done;
await Closing(post, author, token);
if (!post.IsLocalOnly)
await _outbox.Publish(author, post, Update(post, author), token);
await _outbox.Publish(author, post, ActivityPubRenderer.UpdateOf(post, author, $"poll-{DateTime.UtcNow.Ticks}"), token);
return JobOutcome.Done;
}
protected virtual Task Closing(PostEntity post, LocalActor author, CancellationToken token) => Task.CompletedTask;
static JsonObject Update(PostEntity post, LocalActor author)
{
var question = post.Visibility == PostVisibility.Direct
? ActivityPubRenderer.DirectNote(post, author, Array.Empty<(string, string)>(), post.ContextURI)
: ActivityPubRenderer.Note(post, author, default, post.InReplyToURI);
question["to"] = new JsonArray(post.To.Select(t => (JsonNode)t).ToArray());
question["cc"] = new JsonArray(post.Cc.Select(c => (JsonNode)c).ToArray());
return new JsonObject
{
["@context"] = ActivityPubRenderer.Context(),
["id"] = author.ActivityUri($"update-{post.ID}-poll-{DateTime.UtcNow.Ticks}"),
["type"] = "Update",
["actor"] = author.Uri,
["to"] = question["to"]!.DeepClone(),
["cc"] = question["cc"]!.DeepClone(),
["object"] = question
};
}
}
public class PollCloseJob : PollRefreshJob
+304
View File
@@ -0,0 +1,304 @@
using MongoDB.Entities;
using PrivaPub.Domain.Social;
using PrivaPub.Federation.Actors;
using PrivaPub.Federation.Inbox;
using PrivaPub.Federation.Objects;
using PrivaPub.Federation.Outbox;
using PrivaPub.Federation.Rendering;
using PrivaPub.Models.Federation;
using PrivaPub.Models.User;
using PrivaPub.Models.Post;
using PrivaPub.Models.Social;
using PrivaPub.StaticServices;
using System.Text.Json.Nodes;
using static PrivaPub.Federation.Objects.ActivityJson;
using PostEntity = PrivaPub.Models.Post.Post;
namespace PrivaPub.Domain.Statuses
{
public interface IQuoteService
{
Task Resolve(PostEntity post, NoteDocument note, CancellationToken token);
Task Revoke(string stampUri, CancellationToken token);
Task<bool> Verified(string stampUri, string quotingUri, PostEntity quoted, CancellationToken token);
Task<QuotePermission> Permission(PostEntity quoted, LocalActor author, CancellationToken token);
Task<string> Grant(LocalActor author, PostEntity quoted, string quotingUri, string quoterUri, CancellationToken token);
Task ReceiveRequest(JsonNode request, ForeignAvatar actor, CancellationToken token);
Task<bool> RevokeLicence(PostEntity quoted, PostEntity quoting, CancellationToken token);
Task Request(LocalActor author, PostEntity post, PostEntity quoted, JsonObject note, CancellationToken token);
Task<bool> Answered(JsonNode answer, ForeignAvatar actor, bool accepted, CancellationToken token);
}
public enum QuotePermission
{
Denied,
Granted,
AskFirst
}
public class QuoteService : IQuoteService
{
readonly DbEntities _dbEntities;
readonly IRemoteActorService _remoteActors;
readonly IRemotePosts _remotePosts;
readonly ILocalActorService _localActors;
readonly IDeliveryService _delivery;
readonly IOutboxPublisher _outbox;
public QuoteService(DbEntities dbEntities, IRemoteActorService remoteActors, IRemotePosts remotePosts, ILocalActorService localActors,
IDeliveryService delivery, IOutboxPublisher outbox)
{
_dbEntities = dbEntities;
_remoteActors = remoteActors;
_remotePosts = remotePosts;
_localActors = localActors;
_delivery = delivery;
_outbox = outbox;
}
const string RequestPrefix = "quote-request-";
const string LicencePath = "/parrot-licences/";
public static string LicenceUri(LocalActor author, string licenceId) => author.Uri + LicencePath + licenceId;
async Task<bool> MayQuote(PostEntity quoted, string quoterUri, CancellationToken token)
{
var author = await _localActors.FindById(LocalActorKind.Person, quoted.GroupUserId, token);
if (author == default)
return false;
if (quoterUri == author.Uri)
return quoted.Visibility is PostVisibility.Public or PostVisibility.Unlisted;
return ActivityPubRenderer.QuotableBy(quoted) switch
{
QuotePolicies.Public => true,
QuotePolicies.Followers => await Follows(quoterUri, author, token),
_ => false
};
}
async Task<bool> Follows(string followerUri, LocalActor author, CancellationToken token)
{
if (await _dbEntities.Followers.Match(f => f.LocalActorId == author.Id && f.ActorURI == followerUri && f.IsAccepted).ExecuteAnyAsync(token))
return true;
var local = await _localActors.FindByUri(followerUri, token);
return local != default
&& await _dbEntities.Followings.Match(f => f.AvatarId == local.Id && f.TargetActorURI == author.Uri && f.State == FollowState.Accepted).ExecuteAnyAsync(token);
}
public async Task<string> Grant(LocalActor author, PostEntity quoted, string quotingUri, string quoterUri, CancellationToken token)
{
var existing = await DB.Default.Find<QuoteLicence>().Match(l => l.PostId == quoted.ID && l.QuotingObjectURI == quotingUri && l.RevokedAt == null).ExecuteFirstAsync(token);
if (existing != default)
return LicenceUri(author, existing.ID);
var licence = new QuoteLicence { PostId = quoted.ID, AuthorAvatarId = author.Id, QuotingObjectURI = quotingUri, QuoterActorURI = quoterUri };
await DB.Default.SaveAsync(licence, token);
return LicenceUri(author, licence.ID);
}
public async Task ReceiveRequest(JsonNode request, ForeignAvatar actor, CancellationToken token)
{
var objectUri = Id(request["object"]);
var instrument = request["instrument"];
var quotingUri = Id(instrument);
if (objectUri == default || quotingUri == default || !Origin.Same(quotingUri, actor.ActorURI)
|| instrument is JsonObject embedded && Id(embedded["attributedTo"]) is { } by && by != actor.ActorURI)
return;
var quoted = await _dbEntities.Posts.Match(p => p.ObjectURI == objectUri && !p.IsFederatedCopy && !p.DeletedAt.HasValue).ExecuteFirstAsync(token);
var author = quoted == default ? default : await _localActors.FindById(LocalActorKind.Person, quoted.GroupUserId, token);
if (author == default || string.IsNullOrEmpty(actor.InboxURL))
return;
var answer = new JsonObject
{
["@context"] = ActivityPubRenderer.Context(),
["actor"] = author.Uri,
["object"] = Id(request),
["to"] = new JsonArray(actor.ActorURI)
};
if (await MayQuote(quoted, actor.ActorURI, token))
{
answer["type"] = "Accept";
answer["result"] = await Grant(author, quoted, quotingUri, actor.ActorURI, token);
answer["id"] = author.ActivityUri($"accept-quote-{Guid.NewGuid():N}");
}
else
{
answer["type"] = "Reject";
answer["id"] = author.ActivityUri($"reject-quote-{Guid.NewGuid():N}");
}
await _delivery.Enqueue(author, new[] { actor.InboxURL }, answer, token);
}
public async Task<bool> RevokeLicence(PostEntity quoted, PostEntity quoting, CancellationToken token)
{
var licence = await DB.Default.Find<QuoteLicence>().Match(l => l.PostId == quoted.ID && l.QuotingObjectURI == quoting.ObjectURI && l.RevokedAt == null).ExecuteFirstAsync(token);
var author = licence == default ? default : await _localActors.FindById(LocalActorKind.Person, quoted.GroupUserId, token);
if (author == default)
return false;
await DB.Default.Update<QuoteLicence>().MatchID(licence.ID).Modify(l => l.RevokedAt, DateTime.UtcNow).ExecuteAsync(token);
var uri = LicenceUri(author, licence.ID);
await Revoke(uri, token);
var quoter = quoting.IsFederatedCopy ? await _dbEntities.ForeignAvatars.Match(f => f.ActorURI == quoting.ActorURI).ExecuteFirstAsync(token) : default;
var inboxes = (await _delivery.FollowerInboxes(author, token)).Append(quoter?.InboxURL).Where(i => !string.IsNullOrEmpty(i)).Distinct().ToList();
if (inboxes.Count > 0)
await _delivery.Enqueue(author, inboxes, new JsonObject
{
["@context"] = ActivityPubRenderer.Context(),
["id"] = author.ActivityUri($"revoke-quote-{licence.ID}"),
["type"] = "Delete",
["actor"] = author.Uri,
["object"] = uri,
["to"] = new JsonArray(Addressing.Public)
}, token);
return true;
}
async Task<bool> OurLicence(string stampUri, PostEntity quoting, PostEntity quoted, CancellationToken token)
{
var marker = stampUri?.LastIndexOf(LicencePath, StringComparison.Ordinal) ?? -1;
if (marker < 0 || !stampUri.StartsWith(_localActors.BaseAddress + "/", StringComparison.OrdinalIgnoreCase))
return false;
var id = stampUri[(marker + LicencePath.Length)..];
return await DB.Default.Find<QuoteLicence>().Match(l => l.ID == id && l.PostId == quoted.ID && l.QuotingObjectURI == quoting.ObjectURI && l.RevokedAt == null)
.ExecuteAnyAsync(token);
}
public async Task<QuotePermission> Permission(PostEntity quoted, LocalActor author, CancellationToken token)
{
if (!quoted.IsFederatedCopy)
return await MayQuote(quoted, author.Uri, token) ? QuotePermission.Granted : QuotePermission.Denied;
if (quoted.Visibility is not (PostVisibility.Public or PostVisibility.Unlisted))
return QuotePermission.Denied;
if (quoted.QuotePolicy is not { } policy)
return QuotePermission.Granted;
return policy.Automatic.Concat(policy.Manual).Any(Addressing.IsPublic) || policy.Automatic.Concat(policy.Manual).Contains(author.Uri)
? QuotePermission.AskFirst
: QuotePermission.Denied;
}
public async Task Request(LocalActor author, PostEntity post, PostEntity quoted, JsonObject note, CancellationToken token)
{
var owner = await _dbEntities.ForeignAvatars.Match(f => f.ActorURI == quoted.ActorURI).ExecuteFirstAsync(token);
if (string.IsNullOrEmpty(owner?.InboxURL))
return;
await _delivery.Enqueue(author, new[] { owner.InboxURL }, new JsonObject
{
["@context"] = ActivityPubRenderer.Context(),
["id"] = author.ActivityUri(RequestPrefix + post.ID),
["type"] = "QuoteRequest",
["actor"] = author.Uri,
["object"] = quoted.ObjectURI,
["instrument"] = note.DeepClone(),
["to"] = new JsonArray(quoted.ActorURI)
}, token);
}
public async Task<bool> Answered(JsonNode answer, ForeignAvatar actor, bool accepted, CancellationToken token)
{
var request = answer["object"];
var requestId = Id(request);
var marker = requestId?.LastIndexOf("/grunts/" + RequestPrefix, StringComparison.Ordinal) ?? -1;
if (marker < 0 && !(request is JsonObject && Value(request, "type") == "QuoteRequest"))
return false;
if (marker < 0)
return true;
var postId = requestId[(marker + "/grunts/".Length + RequestPrefix.Length)..];
var post = await _dbEntities.Posts.Match(p => p.ID == postId && !p.IsFederatedCopy && !p.DeletedAt.HasValue).ExecuteFirstAsync(token);
var author = post == default ? default : await _localActors.FindById(LocalActorKind.Person, post.GroupUserId, token);
if (author == default || author.ActivityUri(RequestPrefix + post.ID) != requestId || post.QuoteState != QuoteState.Pending)
return true;
var quoted = await _dbEntities.Posts.MatchID(post.QuotedPostId).ExecuteFirstAsync(token);
if (quoted?.ActorURI != actor.ActorURI)
return true;
if (!accepted)
{
await DB.Default.Update<PostEntity>().MatchID(post.ID).Modify(p => p.QuoteState, QuoteState.Rejected).ExecuteAsync(token);
return true;
}
var stamp = Id(answer["result"]);
if (stamp == default || !await Verified(stamp, post.ObjectURI, quoted, token))
return true;
post.QuoteAuthorizationURI = stamp;
post.QuoteState = QuoteState.Accepted;
await DB.Default.Update<PostEntity>().MatchID(post.ID)
.Modify(p => p.QuoteAuthorizationURI, stamp)
.Modify(p => p.QuoteState, QuoteState.Accepted)
.ExecuteAsync(token);
await DB.Default.Update<PostEntity>().MatchID(quoted.ID).Modify(b => b.Inc(p => p.QuotesCount, 1)).ExecuteAsync(token);
if (!post.IsLocalOnly)
await _outbox.Publish(author, post, ActivityPubRenderer.UpdateOf(post, author, "quote-approved"), token);
return true;
}
public async Task Resolve(PostEntity post, NoteDocument note, CancellationToken token)
{
if (note.QuoteUri == default && !note.QuoteDeleted)
return;
var quoted = note.QuoteUri == default
? default
: await _dbEntities.Posts.Match(p => p.ObjectURI == note.QuoteUri && !p.DeletedAt.HasValue).ExecuteFirstAsync(token)
?? await _remotePosts.StoreContext(note.QuoteUri, RemotePosts.MaxDepth, token);
var state = note.QuoteDeleted ? QuoteState.Deleted
: quoted == default ? QuoteState.Pending
: note.QuoteAuthorization != default
? (quoted.IsFederatedCopy ? await Verified(note.QuoteAuthorization, post.ObjectURI, quoted, token) : await OurLicence(note.QuoteAuthorization, post, quoted, token))
? QuoteState.Accepted
: QuoteState.Unauthorized
: note.QuotesByConsent ? QuoteState.Pending
: quoted.Visibility is PostVisibility.Public or PostVisibility.Unlisted ? QuoteState.Accepted
: QuoteState.Unauthorized;
var wasCounted = post.QuoteState == QuoteState.Accepted ? post.QuotedPostId : default;
await DB.Default.Update<PostEntity>().MatchID(post.ID)
.Modify(p => p.QuoteURI, note.QuoteUri)
.Modify(p => p.QuotedPostId, quoted?.ID)
.Modify(p => p.QuoteState, state)
.Modify(p => p.QuoteAuthorizationURI, state == QuoteState.Accepted ? note.QuoteAuthorization : default)
.ExecuteAsync(token);
post.QuotedPostId = quoted?.ID;
post.QuoteState = state;
var nowCounted = state == QuoteState.Accepted ? quoted?.ID : default;
if (wasCounted == nowCounted)
return;
if (wasCounted != default)
await DB.Default.Update<PostEntity>().MatchID(wasCounted).Modify(b => b.Inc(p => p.QuotesCount, -1)).ExecuteAsync(token);
if (nowCounted != default)
{
await DB.Default.Update<PostEntity>().MatchID(nowCounted).Modify(b => b.Inc(p => p.QuotesCount, 1)).ExecuteAsync(token);
if (!quoted.IsFederatedCopy)
await Notifications.Add(quoted.GroupUserId, NotificationType.Quote, post.AuthorAccountId, post.ActorURI, post.ID, token);
}
}
public async Task Revoke(string stampUri, CancellationToken token)
{
var revoked = await _dbEntities.Posts.Match(p => p.QuoteAuthorizationURI == stampUri && p.QuoteState == QuoteState.Accepted).ExecuteAsync(token);
foreach (var post in revoked)
{
await DB.Default.Update<PostEntity>().MatchID(post.ID).Modify(p => p.QuoteState, QuoteState.Revoked).ExecuteAsync(token);
if (post.QuotedPostId != default)
await DB.Default.Update<PostEntity>().MatchID(post.QuotedPostId).Modify(b => b.Inc(p => p.QuotesCount, -1)).ExecuteAsync(token);
}
}
public async Task<bool> Verified(string stampUri, string quotingUri, PostEntity quoted, CancellationToken token)
{
if (!Origin.Same(stampUri, quoted.ActorURI))
return false;
using var fetched = await _remoteActors.FetchObject(stampUri, token);
if (fetched == default)
return false;
var stamp = JsonNode.Parse(fetched.Root.GetRawText());
return Value(stamp, "type") == "QuoteAuthorization"
&& Id(stamp["attributedTo"]) == quoted.ActorURI
&& Id(stamp["interactingObject"]) == quotingUri
&& Id(stamp["interactionTarget"]) == quoted.ObjectURI;
}
}
}
+35 -2
View File
@@ -13,6 +13,7 @@ using PrivaPub.Models.Federation;
using PrivaPub.Models.Group;
using PrivaPub.Models.Media;
using PrivaPub.Models.Post;
using PrivaPub.Models.User;
using PrivaPub.Models.Social;
using PrivaPub.StaticServices;
@@ -40,6 +41,8 @@ namespace PrivaPub.Domain.Statuses
public double? Longitude { get; init; }
public double? RangeKm { get; init; }
public PollDraft Poll { get; init; }
public string QuotedStatusId { get; init; }
public string QuotePolicy { get; init; }
}
public sealed record StatusOutcome(PostEntity Post, int Status = StatusCodes.Status200OK, string Error = default)
@@ -71,10 +74,15 @@ namespace PrivaPub.Domain.Statuses
readonly IMediaService _media;
readonly IGroupDistributor _groups;
readonly IPollService _polls;
readonly ILinkPreviews _previews;
readonly IQuoteService _quotes;
public StatusService(DbEntities dbEntities, ILocalActorService localActors, IRemoteActorService remoteActors, IDeliveryService delivery,
IContentRenderer content, IOutboxPublisher outbox, IFanout fanout, IMediaService media, IGroupDistributor groups, IPollService polls)
IContentRenderer content, IOutboxPublisher outbox, IFanout fanout, IMediaService media, IGroupDistributor groups, IPollService polls,
ILinkPreviews previews, IQuoteService quotes)
{
_previews = previews;
_quotes = quotes;
_polls = polls;
_media = media;
_groups = groups;
@@ -112,6 +120,18 @@ namespace PrivaPub.Domain.Statuses
if (parent != default && !await VisibilityPolicy.CanSee(parent, author.Id, token))
return StatusOutcome.Fail(StatusCodes.Status404NotFound, "Record not found");
PostEntity quoted = default;
var quotePermission = QuotePermission.Denied;
if (!string.IsNullOrEmpty(draft.QuotedStatusId))
{
quoted = await _dbEntities.Posts.Match(p => p.ID == draft.QuotedStatusId && !p.DeletedAt.HasValue && p.ReblogOfPostId == null).ExecuteFirstAsync(token);
if (quoted == default || !await VisibilityPolicy.CanSee(quoted, author.Id, token))
return StatusOutcome.Fail(StatusCodes.Status404NotFound, "Record not found");
quotePermission = await _quotes.Permission(quoted, author, token);
if (quotePermission == QuotePermission.Denied)
return StatusOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: This post cannot be quoted");
}
var located = draft.Latitude.HasValue || draft.Longitude.HasValue;
if (located && (draft.Latitude is not (>= -90 and <= 90) || draft.Longitude is not (>= -180 and <= 180) || group != default
|| draft.Visibility == PostVisibility.Direct))
@@ -163,11 +183,19 @@ namespace PrivaPub.Domain.Statuses
InReplyToAccountId = parent?.AuthorAccountId ?? parent?.GroupUserId,
IsLocalOnly = isLocalOnly,
ActorURI = author.Uri,
Poll = draft.Poll == default ? default : _polls.Create(draft.Poll)
Poll = draft.Poll == default ? default : _polls.Create(draft.Poll),
QuoteURI = quoted?.ObjectURI,
QuotedPostId = quoted?.ID,
QuoteByConsent = quoted != default && (quoted.QuotePolicy != default || !quoted.IsFederatedCopy),
QuoteState = quoted == default ? QuoteState.None : quotePermission == QuotePermission.Granted ? QuoteState.Accepted : QuoteState.Pending
};
post.ID = (string)post.GenerateNewID();
post.ObjectURI = author.PostUri(post.ID);
post.Url = author.PostHtmlUrl(post.ID);
post.LocalQuotePolicy = QuotePolicies.IsKnown(draft.QuotePolicy) ? draft.QuotePolicy : author.Settings.QuotePolicy ?? QuotePolicies.Public;
if (quoted is { IsFederatedCopy: false } && post.QuoteState == QuoteState.Accepted
&& await _localActors.FindById(LocalActorKind.Person, quoted.GroupUserId, token) is { } quotedAuthor)
post.QuoteAuthorizationURI = await _quotes.Grant(quotedAuthor, quoted, post.ObjectURI, author.Uri, token);
if (located)
{
post.Geo = new Coordinates2D(Math.Round(draft.Longitude.Value, 2), Math.Round(draft.Latitude.Value, 2));
@@ -207,6 +235,11 @@ namespace PrivaPub.Domain.Statuses
await DB.Default.Update<PostEntity>().MatchID(parent.ID).Modify(b => b.Inc(p => p.RepliesCount, 1)).ExecuteAsync(token);
await _fanout.Distribute(post, token);
await _polls.Scheduled(post, token);
if (post.QuoteState == QuoteState.Accepted)
await DB.Default.Update<PostEntity>().MatchID(quoted.ID).Modify(b => b.Inc(p => p.QuotesCount, 1)).ExecuteAsync(token);
if (post.QuoteState == QuoteState.Pending && create?["object"] is JsonObject quotingNote)
await _quotes.Request(author, post, quoted, quotingNote, token);
await _previews.Wanted(post, token);
if (create != default)
await _outbox.Publish(author, post, create, token);
if (create != default && group is { IsCircle: false } && visibility is PostVisibility.Public or PostVisibility.Unlisted)
@@ -199,6 +199,33 @@ namespace PrivaPub.Federation.Controllers
return Activity(note);
}
[HttpGet, Route("{actor}/parrot-licences/{licenceId}")]
public async Task<IActionResult> ParrotLicence(string actor, string licenceId, CancellationToken token)
{
var author = await _localActors.FindByUserName(actor, token);
var licence = author == default ? default : await DB.Default.Find<QuoteLicence>().Match(l => l.ID == licenceId && l.AuthorAvatarId == author.Id).ExecuteFirstAsync(token);
var quoted = licence == default ? default : await _dbEntities.Posts.MatchID(licence.PostId).ExecuteFirstAsync(token);
if (quoted == default)
return NotFound();
var uri = Domain.Statuses.QuoteService.LicenceUri(author, licence.ID);
if (licence.RevokedAt.HasValue || quoted.DeletedAt.HasValue)
return new ContentResult
{
Content = new JsonObject { ["@context"] = ActivityPubRenderer.ActivityStreams, ["id"] = uri, ["type"] = "Tombstone", ["formerType"] = "QuoteAuthorization" }.ToJsonString(),
ContentType = ActivityContentType,
StatusCode = StatusCodes.Status410Gone
};
return Activity(new JsonObject
{
["@context"] = ActivityPubRenderer.Context(),
["id"] = uri,
["type"] = "QuoteAuthorization",
["attributedTo"] = author.Uri,
["interactingObject"] = licence.QuotingObjectURI,
["interactionTarget"] = quoted.ObjectURI
});
}
[HttpGet, Route("{actor}/grunts/{activityId}")]
public async Task<IActionResult> Grunt(string actor, string activityId, CancellationToken token)
{
@@ -1,5 +1,6 @@
using MongoDB.Entities;
using PrivaPub.Domain.Statuses;
using PrivaPub.Federation.Actors;
using PrivaPub.Models.Social;
using PrivaPub.Models.User;
@@ -15,17 +16,21 @@ namespace PrivaPub.Federation.Inbox.Handlers
{
readonly DbEntities _dbEntities;
readonly ILocalActorService _localActors;
readonly IQuoteService _quotes;
public AcceptHandler(DbEntities dbEntities, ILocalActorService localActors)
public AcceptHandler(DbEntities dbEntities, ILocalActorService localActors, IQuoteService quotes)
{
_dbEntities = dbEntities;
_localActors = localActors;
_quotes = quotes;
}
public virtual string Type => "Accept";
public async Task Handle(JsonNode activity, ForeignAvatar actor, CancellationToken token)
{
if (await _quotes.Answered(activity, actor, accepted: Type == "Accept", token))
return;
var following = await FindFollowing(activity["object"], actor, _dbEntities, _localActors, token);
if (following == default)
return;
@@ -56,7 +61,7 @@ namespace PrivaPub.Federation.Inbox.Handlers
public class RejectHandler : AcceptHandler
{
public RejectHandler(DbEntities dbEntities, ILocalActorService localActors) : base(dbEntities, localActors)
public RejectHandler(DbEntities dbEntities, ILocalActorService localActors, IQuoteService quotes) : base(dbEntities, localActors, quotes)
{
}
@@ -4,6 +4,7 @@ using MongoDB.Entities;
using PrivaPub.Domain.Timelines;
using PrivaPub.Federation.Actors;
using PrivaPub.Federation.Moderation;
using PrivaPub.Domain.Content;
using PrivaPub.Domain.Statuses;
using PrivaPub.Federation.Objects;
using PrivaPub.Federation.Outbox;
@@ -37,10 +38,14 @@ namespace PrivaPub.Federation.Inbox.Handlers
readonly IGroupDistributor _groups;
readonly IObjectRecords _records;
readonly IPollService _polls;
readonly ILinkPreviews _previews;
readonly IQuoteService _quotes;
public CreateHandler(DbEntities dbEntities, ILocalActorService localActors, IRemoteActorService remoteActors, IDeliveryService delivery,
IDomainBlocks domainBlocks, IFanout fanout, IRemotePosts remotePosts, IGroupDistributor groups, IObjectRecords records, IPollService polls)
IDomainBlocks domainBlocks, IFanout fanout, IRemotePosts remotePosts, IGroupDistributor groups, IObjectRecords records, IPollService polls, ILinkPreviews previews, IQuoteService quotes)
{
_quotes = quotes;
_previews = previews;
_records = records;
_polls = polls;
_groups = groups;
@@ -148,6 +153,8 @@ namespace PrivaPub.Federation.Inbox.Handlers
return;
}
await _records.Record(note, post, ObjectPath.Delivered, refetched, token);
await _previews.Wanted(post, token);
await _quotes.Resolve(post, note, token);
if (parent != default)
await DB.Default.Update<PostEntity>().MatchID(parent.ID).Modify(b => b.Inc(p => p.RepliesCount, 1)).ExecuteAsync(token);
@@ -1,5 +1,6 @@
using MongoDB.Entities;
using PrivaPub.Domain.Statuses;
using PrivaPub.Federation.Actors;
using PrivaPub.Federation.Objects;
using PrivaPub.Federation.Outbox;
@@ -27,10 +28,12 @@ namespace PrivaPub.Federation.Inbox.Handlers
readonly IRemoteActorService _remoteActors;
readonly IDeliveryService _delivery;
readonly IGroupDistributor _groups;
readonly IQuoteService _quotes;
public DeleteHandler(DbEntities dbEntities, ILocalActorService localActors, IRemoteActorService remoteActors, IDeliveryService delivery,
IGroupDistributor groups)
IGroupDistributor groups, IQuoteService quotes)
{
_quotes = quotes;
_groups = groups;
_dbEntities = dbEntities;
_localActors = localActors;
@@ -72,7 +75,10 @@ namespace PrivaPub.Federation.Inbox.Handlers
.ExecuteAsync(token);
var post = await _dbEntities.Posts.Match(p => p.ObjectURI == objectUri && p.ActorURI == actor.ActorURI).ExecuteFirstAsync(token);
if (post == default)
{
await _quotes.Revoke(objectUri, token);
return;
}
await DB.Default.DeleteAsync<PostEntity>(post.ID);
await DB.Default.DeleteAsync<ObjectRecord>(r => r.PostId == post.ID || r.ObjectURI == objectUri);
await DB.Default.DeleteAsync<TimelineEntry>(e => e.PostId == post.ID || e.ReblogOfPostId == post.ID);
@@ -0,0 +1,21 @@
using PrivaPub.Domain.Statuses;
using PrivaPub.Models.User;
using System.Text.Json.Nodes;
namespace PrivaPub.Federation.Inbox.Handlers
{
public class QuoteRequestHandler : IActivityHandler
{
readonly IQuoteService _quotes;
public QuoteRequestHandler(IQuoteService quotes)
{
_quotes = quotes;
}
public string Type => "QuoteRequest";
public Task Handle(JsonNode activity, ForeignAvatar actor, CancellationToken token) => _quotes.ReceiveRequest(activity, actor, token);
}
}
@@ -1,6 +1,7 @@
using MongoDB.Entities;
using PrivaPub.Federation.Actors;
using PrivaPub.Domain.Statuses;
using PrivaPub.Federation.Objects;
using PrivaPub.Federation.Outbox;
using PrivaPub.Models.Post;
@@ -24,10 +25,12 @@ namespace PrivaPub.Federation.Inbox.Handlers
readonly IRemoteActorService _remoteActors;
readonly IGroupDistributor _groups;
readonly IObjectRecords _records;
readonly IQuoteService _quotes;
public UpdateHandler(DbEntities dbEntities, ILocalActorService localActors, IRemoteActorService remoteActors, IGroupDistributor groups,
IObjectRecords records)
IObjectRecords records, IQuoteService quotes)
{
_quotes = quotes;
_records = records;
_groups = groups;
_dbEntities = dbEntities;
@@ -56,6 +59,7 @@ namespace PrivaPub.Federation.Inbox.Handlers
return;
post.Poll = note.Poll ?? post.Poll;
post.QuotePolicy = note.QuotePolicy;
post.Video = note.Video ?? post.Video;
post.Audio = note.Audio ?? post.Audio;
post.Event = note.Event ?? post.Event;
@@ -63,6 +67,7 @@ namespace PrivaPub.Federation.Inbox.Handlers
{
await DB.Default.SaveAsync(post, token);
await _records.Revise(note, Id(activity), token);
await Requote(post, note, token);
return;
}
@@ -102,11 +107,18 @@ namespace PrivaPub.Federation.Inbox.Handlers
post.UpdateDate = DateTime.UtcNow;
await DB.Default.SaveAsync(post, token);
await _records.Revise(note, Id(activity), token);
await Requote(post, note, token);
if (!string.IsNullOrEmpty(post.GroupId) && await _localActors.FindById(Models.Federation.LocalActorKind.Group, post.GroupId, token) is { IsCircle: false } community)
await _groups.Announce(community, activity.AsObject(), post.ObjectURI, isNewPost: false, token);
}
static bool IsEdit(NoteDocument note, PostEntity post) =>
note.Updated is { } updated && updated > (post.EditedAt ?? post.CreationDate);
async Task Requote(PostEntity post, NoteDocument note, CancellationToken token)
{
if (note.QuoteUri != post.QuoteURI || note.QuoteAuthorization != post.QuoteAuthorizationURI || post.QuoteState != QuoteState.Accepted)
await _quotes.Resolve(post, note, token);
}
}
}
+6 -1
View File
@@ -1,6 +1,7 @@
using MongoDB.Driver;
using MongoDB.Entities;
using PrivaPub.Domain.Content;
using PrivaPub.Federation.Actors;
using PrivaPub.Federation.Moderation;
using PrivaPub.Federation.Objects;
@@ -39,10 +40,12 @@ namespace PrivaPub.Federation.Inbox
readonly IDomainBlocks _domainBlocks;
readonly IJobQueue _queue;
readonly IObjectRecords _records;
readonly ILinkPreviews _previews;
public RemotePosts(DbEntities dbEntities, ILocalActorService localActors, IRemoteActorService remoteActors, IDomainBlocks domainBlocks,
IJobQueue queue, IObjectRecords records)
IJobQueue queue, IObjectRecords records, ILinkPreviews previews)
{
_previews = previews;
_records = records;
_dbEntities = dbEntities;
_localActors = localActors;
@@ -79,6 +82,7 @@ namespace PrivaPub.Federation.Inbox
Excerpt = note.Excerpt,
Source = note.Source,
Poll = note.Poll,
QuotePolicy = note.QuotePolicy,
Emojis = note.Emojis.ToList(),
CoverURL = note.CoverURL,
Link = note.Link,
@@ -143,6 +147,7 @@ namespace PrivaPub.Federation.Inbox
return await _dbEntities.Posts.Match(p => p.ObjectURI == objectUri).ExecuteFirstAsync(token);
}
await _records.Record(note, post, ObjectPath.Fetched, refetched: false, token);
await _previews.Wanted(post, token);
if (grandparent == default && !string.IsNullOrEmpty(note.InReplyTo) && depth < MaxDepth)
await _queue.Enqueue(JobKind.FetchAncestors, JsonSerializer.Serialize(new AncestorsPayload(post.ID, depth + 1)), new Uri(note.InReplyTo).Host,
@@ -44,9 +44,11 @@ namespace PrivaPub.Federation.Moderation
public bool IsSuspended(string host) => Find(host)?.Severity == DomainBlockSeverity.Suspend;
public async Task Reload(CancellationToken token)
public async Task Reload(CancellationToken token) =>
Load(await DB.Default.Find<DomainBlock>().ExecuteAsync(token));
public void Load(IEnumerable<DomainBlock> blocks)
{
var blocks = await DB.Default.Find<DomainBlock>().ExecuteAsync(token);
_blocks = blocks.Where(b => !string.IsNullOrEmpty(b.Domain))
.GroupBy(b => Normalise(b.Domain))
.ToDictionary(g => g.Key, g => g.First());
@@ -38,7 +38,7 @@ namespace PrivaPub.Federation.Objects
{
KeepChildNodes = true
};
foreach (var allowed in new[] { "mention", "hashtag", "ellipsis", "invisible" })
foreach (var allowed in new[] { "mention", "hashtag", "ellipsis", "invisible", "quote-inline" })
sanitizer.AllowedClasses.Add(allowed);
sanitizer.RemovingCssClass += (_, e) => e.Cancel = MicroformatClass().IsMatch(e.CssClass);
sanitizer.RemovingTag += (_, e) =>
+15 -1
View File
@@ -25,6 +25,9 @@ namespace PrivaPub.Federation.Objects
public string Context { get; init; }
public string Audience { get; init; }
public string QuoteUri { get; init; }
public string QuoteAuthorization { get; init; }
public bool QuotesByConsent { get; init; }//FEP-044f `quote`, as opposed to the legacy keys that ask nobody
public bool QuoteDeleted { get; init; }
public DateTime Published { get; init; }
public DateTime? Updated { get; init; }
public IReadOnlyList<string> To { get; init; } = Array.Empty<string>();
@@ -34,6 +37,7 @@ namespace PrivaPub.Federation.Objects
public IReadOnlyList<PostMedia> Attachments { get; init; } = Array.Empty<PostMedia>();
public PostSource Source { get; init; }
public PostPoll Poll { get; init; }
public InteractionRule QuotePolicy { get; init; }
public IReadOnlyList<CustomEmoji> Emojis { get; init; } = Array.Empty<CustomEmoji>();
public string CoverURL { get; init; }
public PostLink Link { get; init; }
@@ -79,7 +83,10 @@ namespace PrivaPub.Federation.Objects
InReplyTo = Id(note["inReplyTo"]),
Context = Id(note["context"]) ?? Value(note, "conversation"),
Audience = Id(note["audience"]),
QuoteUri = Value(note, "quote") ?? Value(note, "quoteUrl") ?? Value(note, "quoteUri") ?? Value(note, "_misskey_quote"),
QuoteUri = Id(note["quote"]) ?? Value(note, "quoteUrl") ?? Value(note, "quoteUri") ?? Value(note, "_misskey_quote") ?? QuoteLink(note),
QuoteAuthorization = Id(note["quoteAuthorization"]),
QuotesByConsent = note.ContainsKey("quote"),
QuoteDeleted = note["quote"] is JsonObject quoted && Value(quoted, "type") == "Tombstone",
Published = Time(Value(note, "published")) ?? DateTime.UtcNow,
Updated = Time(Value(note, "updated")),
To = List(note["to"]),
@@ -97,6 +104,7 @@ namespace PrivaPub.Federation.Objects
Attachments = Attachments(note),
Source = ObjectShapes.Source(note),
Poll = ObjectShapes.Poll(note),
QuotePolicy = ObjectShapes.QuotePolicy(note),
Emojis = ObjectShapes.Emojis(note["tag"]),
CoverURL = ObjectShapes.Cover(note),
Link = ObjectShapes.Link(note),
@@ -169,6 +177,12 @@ namespace PrivaPub.Federation.Objects
};
}
static string QuoteLink(JsonObject note) =>
Tags(note, "Link")
.Where(t => Value(t, "mediaType") is { } type && (type.StartsWith("application/activity+json") || type.StartsWith("application/ld+json")))
.Select(t => Value(t, "href"))
.FirstOrDefault(href => Origin.Of(href) != default);
static string FirstOf(JsonNode map) =>
map is JsonObject languages ? languages.Select(l => l.Value is JsonValue v && v.TryGetValue<string>(out var text) ? text : default).FirstOrDefault(t => t != default) : default;
+22 -1
View File
@@ -18,6 +18,7 @@ namespace PrivaPub.Federation.Objects
const int CoverMaxWidth = 1280;
const int MaxEmojis = 64;
const int MaxPollOptions = 20;
const int MaxAudience = 32;
static readonly Dictionary<string, string> MediaTypesByExtension = new(StringComparer.OrdinalIgnoreCase)
{
@@ -97,6 +98,23 @@ namespace PrivaPub.Federation.Objects
.Take(MaxEmojis)
.ToList();
public static InteractionRule QuotePolicy(JsonObject note)
{
if (note["interactionPolicy"] is not JsonObject policy || policy["canQuote"] is not JsonObject canQuote)
return default;
static List<string> Who(JsonNode node) => node switch
{
JsonArray array => array.Select(Id).Where(id => id != default).Take(MaxAudience).ToList(),
JsonNode single when Id(single) is { } id => new List<string> { id },
_ => new List<string>()
};
return new InteractionRule
{
Automatic = Who(canQuote["automaticApproval"] ?? canQuote["always"]),
Manual = Who(canQuote["manualApproval"] ?? canQuote["approvalRequired"])
};
}
public static PostPoll Poll(JsonObject note)
{
if (Value(note, "type") != "Question")
@@ -199,10 +217,13 @@ namespace PrivaPub.Federation.Objects
if (Value(note, "type") != "Video")
return default;
var links = Objects(note["url"]).ToList();
var nested = links.Where(l => Value(l, "mediaType")?.Equals("application/x-mpegURL", StringComparison.OrdinalIgnoreCase) == true)
.SelectMany(l => Objects(l["tag"]))
.ToList();
return new VideoDetails
{
DurationSeconds = Seconds(note["duration"]),
Variants = Variants(links, "video/", "audio/"),
Variants = Variants(links.Concat(nested).ToList(), "video/", "audio/"),
PlaylistURL = links.Where(l => Value(l, "mediaType")?.Equals("application/x-mpegURL", StringComparison.OrdinalIgnoreCase) == true)
.Select(l => Value(l, "href")).FirstOrDefault(IsWeb),
IsLive = Bool(note, "isLiveBroadcast") == true,
@@ -60,6 +60,16 @@ namespace PrivaPub.Federation.Outbox
inboxes.Add(parentAuthor.InboxURL);
}
if (post.Visibility is PostVisibility.Public or PostVisibility.Unlisted && !string.IsNullOrEmpty(post.QuotedPostId))
{
var quoted = await _dbEntities.Posts.MatchID(post.QuotedPostId).ExecuteFirstAsync(token);
var quotedAuthor = quoted is { IsFederatedCopy: true }
? await _dbEntities.ForeignAvatars.Match(a => a.ActorURI == quoted.ActorURI).ExecuteFirstAsync(token)
: default;
if (!string.IsNullOrEmpty(quotedAuthor?.InboxURL))
inboxes.Add(quotedAuthor.InboxURL);
}
return inboxes.Where(i => !string.IsNullOrEmpty(i)).Distinct(StringComparer.Ordinal).ToList();
}
@@ -39,6 +39,22 @@ namespace PrivaPub.Federation.Rendering
["blurhash"] = "toot:blurhash",
["votersCount"] = "toot:votersCount",
["Emoji"] = "toot:Emoji",
["quote"] = new JsonObject { ["@id"] = "https://w3id.org/fep/044f#quote", ["@type"] = "@id" },
["quoteAuthorization"] = new JsonObject { ["@id"] = "https://w3id.org/fep/044f#quoteAuthorization", ["@type"] = "@id" },
["QuoteRequest"] = "https://w3id.org/fep/044f#QuoteRequest",
["QuoteAuthorization"] = "https://w3id.org/fep/044f#QuoteAuthorization",
["interactingObject"] = new JsonObject { ["@id"] = "gts:interactingObject", ["@type"] = "@id" },
["interactionTarget"] = new JsonObject { ["@id"] = "gts:interactionTarget", ["@type"] = "@id" },
["misskey"] = "https://misskey-hub.net/ns#",
["_misskey_quote"] = "misskey:_misskey_quote",
["fedibird"] = "http://fedibird.com/ns#",
["quoteUri"] = "fedibird:quoteUri",
["quoteUrl"] = "as:quoteUrl",
["gts"] = "https://gotosocial.org/ns#",
["interactionPolicy"] = new JsonObject { ["@id"] = "gts:interactionPolicy", ["@type"] = "@id" },
["canQuote"] = new JsonObject { ["@id"] = "gts:canQuote", ["@type"] = "@id" },
["automaticApproval"] = new JsonObject { ["@id"] = "gts:automaticApproval", ["@type"] = "@id" },
["manualApproval"] = new JsonObject { ["@id"] = "gts:manualApproval", ["@type"] = "@id" },
["litepub"] = "http://litepub.social/ns#",
["EmojiReact"] = "litepub:EmojiReact",
["focalPoint"] = new JsonObject { ["@container"] = "@list", ["@id"] = "toot:focalPoint" },
@@ -160,6 +176,50 @@ namespace PrivaPub.Federation.Rendering
return note;
}
public static JsonObject UpdateOf(PostEntity post, LocalActor author, string reason)
{
var note = post.Visibility == PostVisibility.Direct
? DirectNote(post, author, Array.Empty<(string, string)>(), post.ContextURI)
: Note(post, author, default, post.InReplyToURI);
note["to"] = new JsonArray(post.To.Select(t => (JsonNode)t).ToArray());
note["cc"] = new JsonArray(post.Cc.Select(c => (JsonNode)c).ToArray());
return new JsonObject
{
["@context"] = Context(),
["id"] = author.ActivityUri($"update-{post.ID}-{reason}"),
["type"] = "Update",
["actor"] = author.Uri,
["to"] = note["to"]!.DeepClone(),
["cc"] = note["cc"]!.DeepClone(),
["object"] = note
};
}
public static string QuotableBy(PostEntity post) =>
post.Visibility is PostVisibility.Public or PostVisibility.Unlisted ? post.LocalQuotePolicy ?? Models.User.QuotePolicies.Nobody : Models.User.QuotePolicies.Nobody;
static void AddQuote(JsonObject note, PostEntity post, ref string content)
{
if (string.IsNullOrEmpty(post.QuoteURI))
return;
var link = WebUtility.HtmlEncode(post.QuoteURI);
content += $"<p class=\"quote-inline\">RE: <a href=\"{link}\">{link}</a></p>";
if (post.QuoteByConsent)
note["quote"] = post.QuoteURI;
if (!string.IsNullOrEmpty(post.QuoteAuthorizationURI))
note["quoteAuthorization"] = post.QuoteAuthorizationURI;
note["_misskey_quote"] = post.QuoteURI;
note["quoteUri"] = post.QuoteURI;
note["quoteUrl"] = post.QuoteURI;
(note["tag"] as JsonArray)?.Add(new JsonObject
{
["type"] = "Link",
["mediaType"] = "application/ld+json; profile=\"https://www.w3.org/ns/activitystreams\"",
["href"] = post.QuoteURI,
["name"] = $"RE: {post.QuoteURI}"
});
}
static void AddPoll(JsonObject note, PostPoll poll)
{
note["type"] = "Question";
@@ -211,6 +271,21 @@ namespace PrivaPub.Federation.Rendering
}))
.ToArray())
};
AddQuote(note, post, ref content);
if (!post.IsFederatedCopy)
note["interactionPolicy"] = new JsonObject
{
["canQuote"] = new JsonObject
{
["automaticApproval"] = new JsonArray((JsonNode)(QuotableBy(post) switch
{
Models.User.QuotePolicies.Public => Objects.Addressing.Public,
Models.User.QuotePolicies.Followers => author.Followers,
_ => author.Uri
}))
}
};
note["content"] = content;
if (!string.IsNullOrEmpty(post.Language))
note["contentMap"] = new JsonObject { [post.Language] = content };
if (post.Poll is { } poll)
+2
View File
@@ -116,6 +116,8 @@ namespace PrivaPub.Infrastructure.Data
await DB.Default.Index<Reaction>().Key(r => r.PostId, KeyType.Ascending).Key(r => r.ActorURI, KeyType.Ascending).Key(r => r.Emoji, KeyType.Ascending)
.Option(o => o.Unique = true).CreateAsync(token);
await Plain<Reaction>(token, r => r.ActivityURI);
await DB.Default.Index<QuoteLicence>().Key(l => l.PostId, KeyType.Ascending).Key(l => l.QuotingObjectURI, KeyType.Ascending).CreateAsync(token);
await Unique<Domain.Content.LinkPreview>(p => p.Url, Builders<Domain.Content.LinkPreview>.Filter.Type(p => p.Url, BsonType.String), token);
}
static async Task Unique<T>(System.Linq.Expressions.Expression<Func<T, object>> key, FilterDefinition<T> partial,
@@ -22,6 +22,8 @@ namespace PrivaPub.Infrastructure.Http
bool IsAllowed(Uri target);
Task<FetchedJson> GetJson(string url, string accept, Action<HttpRequestMessage> sign, CancellationToken token);
bool FailedTemporarily(string url);
Task<(Uri FinalUri, string Html)> GetPage(string url, CancellationToken token);
Task<HttpResponseMessage> OpenMedia(string url, System.Net.Http.Headers.RangeHeaderValue range, CancellationToken token);
Task<HttpResponseMessage> Send(HttpRequestMessage request, CancellationToken token);
Task<(byte[] Bytes, string ContentType)> GetMedia(string url, long maxBytes, CancellationToken token);
}
@@ -181,6 +183,106 @@ namespace PrivaPub.Infrastructure.Http
}
}
public async Task<(Uri FinalUri, string Html)> GetPage(string url, CancellationToken token)
{
if (!Uri.TryCreate(url, UriKind.Absolute, out var target) || !IsAllowed(target))
return default;
using var timeout = CancellationTokenSource.CreateLinkedTokenSource(token);
timeout.CancelAfter(RequestTimeout);
try
{
for (var hop = 0; hop <= MaxRedirects; hop++)
{
using var request = new HttpRequestMessage(HttpMethod.Get, target);
request.Headers.Accept.ParseAdd("text/html, application/xhtml+xml");
using var response = await _httpClientFactory.CreateClient(ClientName).SendAsync(request, HttpCompletionOption.ResponseHeadersRead, timeout.Token);
if (IsRedirect(response.StatusCode))
{
var location = response.Headers.Location;
var next = location == default ? default : location.IsAbsoluteUri ? location : new Uri(target, location);
if (!IsAllowed(next))
return default;
target = next;
continue;
}
var mediaType = response.Content.Headers.ContentType?.MediaType?.ToLowerInvariant();
if (!response.IsSuccessStatusCode || mediaType is not ("text/html" or "application/xhtml+xml"))
return default;
var bytes = await ReadPrefix(response.Content, MaxPageBytes, timeout.Token);
var charset = response.Content.Headers.ContentType?.CharSet?.Trim('"');
var encoding = System.Text.Encoding.UTF8;
try
{
if (!string.IsNullOrEmpty(charset))
encoding = System.Text.Encoding.GetEncoding(charset);
}
catch (ArgumentException)
{
}
return (target, encoding.GetString(bytes));
}
return default;
}
catch (Exception ex) when (ex is HttpRequestException or BlockedDestinationException or OperationCanceledException && !token.IsCancellationRequested)
{
_logger.LogInformation("Page {Url} refused: {Reason}", url, ex.Message);
return default;
}
}
public async Task<HttpResponseMessage> OpenMedia(string url, System.Net.Http.Headers.RangeHeaderValue range, CancellationToken token)
{
if (!Uri.TryCreate(url, UriKind.Absolute, out var target) || !IsAllowed(target))
return default;
try
{
for (var hop = 0; hop <= MaxRedirects; hop++)
{
using var request = new HttpRequestMessage(HttpMethod.Get, target);
request.Headers.Accept.ParseAdd("video/*, audio/*, image/*");
request.Headers.Range = range;
var response = await _httpClientFactory.CreateClient(ClientName).SendAsync(request, HttpCompletionOption.ResponseHeadersRead, token);
if (IsRedirect(response.StatusCode))
{
var location = response.Headers.Location;
response.Dispose();
var next = location == default ? default : location.IsAbsoluteUri ? location : new Uri(target, location);
if (!IsAllowed(next))
return default;
target = next;
continue;
}
var mediaType = response.Content.Headers.ContentType?.MediaType?.ToLowerInvariant();
if (!response.IsSuccessStatusCode || mediaType == default || mediaType.Contains("svg")
|| !(mediaType.StartsWith("video/") || mediaType.StartsWith("audio/") || mediaType.StartsWith("image/") || mediaType == "application/octet-stream"))
{
response.Dispose();
return default;
}
return response;
}
return default;
}
catch (Exception ex) when (ex is HttpRequestException or BlockedDestinationException or OperationCanceledException && !token.IsCancellationRequested)
{
_logger.LogInformation("Media stream {Url} refused: {Reason}", url, ex.Message);
return default;
}
}
const int MaxPageBytes = 512 * 1024;
static async Task<byte[]> ReadPrefix(HttpContent content, int limit, CancellationToken token)
{
await using var stream = await content.ReadAsStreamAsync(token);
var buffer = new byte[limit];
var total = 0;
int read;
while (total < limit && (read = await stream.ReadAsync(buffer.AsMemory(total, limit - total), token)) > 0)
total += read;
return buffer[..total];
}
public async Task<HttpResponseMessage> Send(HttpRequestMessage request, CancellationToken token)
{
if (!IsAllowed(request.RequestUri))
@@ -6,5 +6,6 @@ namespace PrivaPub.Infrastructure.Http
public bool AllowPlainHttp { get; set; }
public bool SecureMode { get; set; }
public bool AcceptAnyCertificate { get; set; }
public bool FetchLinkPreviews { get; set; } = true;//owner decision 1: the server reads linked pages of public posts
}
}
+10 -2
View File
@@ -4,6 +4,7 @@ using MongoDB.Entities;
using PrivaPub.Models.Jobs;
using System.Collections.Concurrent;
using System.Linq.Expressions;
namespace PrivaPub.Infrastructure.Jobs
{
@@ -39,6 +40,13 @@ namespace PrivaPub.Infrastructure.Jobs
readonly string _owner = $"{Environment.MachineName}:{Environment.ProcessId}";
readonly ConcurrentDictionary<JobKind, SemaphoreSlim> _signals = new();
readonly FilterDefinition<Job> _scope = Builders<Job>.Filter.Empty;
public JobQueue()
{
}
public JobQueue(Expression<Func<Job, bool>> scope) => _scope = Builders<Job>.Filter.Where(scope);
public async Task<bool> Enqueue(JobKind kind, string payload, string host, string dedupeKey, CancellationToken token) =>
await EnqueueMany(new[] { new Job { Kind = kind, Payload = payload, Host = host, DedupeKey = dedupeKey } }, token) == 1;
@@ -66,7 +74,7 @@ namespace PrivaPub.Infrastructure.Jobs
var now = DateTime.UtcNow;
var busy = busyHosts?.ToList() ?? new List<string>();
return await DB.Default.UpdateAndGet<Job>()
.Match(j => j.Kind == kind && j.State == JobState.Pending && j.RunAt <= now && !busy.Contains(j.Host))
.Match(f => f.Where(j => j.Kind == kind && j.State == JobState.Pending && j.RunAt <= now && !busy.Contains(j.Host)) & _scope)
.Modify(j => j.State, JobState.Running)
.Modify(j => j.LeasedUntil, now + LeaseTime)
.Modify(j => j.LeaseOwner, _owner)
@@ -106,7 +114,7 @@ namespace PrivaPub.Infrastructure.Jobs
{
var now = DateTime.UtcNow;
var result = await DB.Default.Update<Job>()
.Match(j => j.State == JobState.Running && j.LeasedUntil < now)
.Match(f => f.Where(j => j.State == JobState.Running && j.LeasedUntil < now) & _scope)
.Modify(j => j.State, JobState.Pending)
.Modify(j => j.RunAt, now)
.Modify(j => j.LeasedUntil, null)
@@ -78,7 +78,11 @@ namespace PrivaPub.Middleware
.AddSingleton<IActivityHandler, LikeHandler>()
.AddSingleton<IActivityHandler, DislikeHandler>()
.AddSingleton<IActivityHandler, EmojiReactHandler>()
.AddSingleton<IActivityHandler, QuoteRequestHandler>()
.AddSingleton<IReactions, Reactions>()
.AddSingleton<LinkPreviews>()
.AddSingleton<ILinkPreviews>(services => services.GetRequiredService<LinkPreviews>())
.AddSingleton<IJobHandler>(services => services.GetRequiredService<LinkPreviews>())
.AddSingleton<IActivityHandler, JoinHandler>()
.AddSingleton<IActivityHandler, AnnounceHandler>()
.AddSingleton<IActivityHandler, FlagHandler>()
@@ -89,6 +93,7 @@ namespace PrivaPub.Middleware
.AddSingleton<IRemotePosts, RemotePosts>()
.AddSingleton<IObjectRecords, ObjectRecords>()
.AddSingleton<IPollService, PollService>()
.AddSingleton<IQuoteService, QuoteService>()
.AddSingleton<IJobHandler, PollRefreshJob>()
.AddSingleton<IJobHandler, PollCloseJob>()
.AddSingleton<IJobHandler, InstanceDescriber>()
@@ -0,0 +1,14 @@
using MongoDB.Entities;
namespace PrivaPub.Models.Federation
{
public class QuoteLicence : Entity
{
public string PostId { get; set; }//the quoted local post
public string AuthorAvatarId { get; set; }
public string QuotingObjectURI { get; set; }
public string QuoterActorURI { get; set; }
public DateTime CreatedAt { get; set; } = DateTime.UtcNow;
public DateTime? RevokedAt { get; set; }
}
}
+2 -1
View File
@@ -25,7 +25,8 @@ namespace PrivaPub.Models.Jobs
FetchAncestors,
DescribeInstance,
PollRefresh,
PollClose
PollClose,
FetchPreview
}
public enum JobState
+19
View File
@@ -57,6 +57,14 @@ namespace PrivaPub.Models.Post
public string Url { get; set; }
public string ContextURI { get; set; }
public string QuoteURI { get; set; }
public string QuotedPostId { get; set; }//our copy of the quoted post, once fetched
public QuoteState QuoteState { get; set; }
public string QuoteAuthorizationURI { get; set; }//FEP-044f: the quoted author's stamp
public bool QuoteByConsent { get; set; }
public string LocalQuotePolicy { get; set; }//who may quote this local post: QuotePolicies.Public, Followers or Nobody//our quote of a post whose server asks for consent (FEP-044f), so `quote` is sent
public int QuotesCount { get; set; }
[BsonIgnoreIfNull]
public InteractionRule QuotePolicy { get; set; }//a remote post's interactionPolicy.canQuote; null when it states none
public List<string> To { get; set; } = new();
public List<string> Cc { get; set; } = new();
@@ -73,6 +81,17 @@ namespace PrivaPub.Models.Post
public string UpdateReason { get; set; }
}
public enum QuoteState
{
None,
Pending,
Accepted,
Rejected,
Revoked,
Deleted,
Unauthorized
}
public enum PostVisibility
{
Public,
+6
View File
@@ -23,6 +23,12 @@ namespace PrivaPub.Models.Post
public int Votes { get; set; }
}
public class InteractionRule
{
public List<string> Automatic { get; set; } = new();//actor or collection URIs; Public for anyone
public List<string> Manual { get; set; } = new();
}
public class PostSource
{
public string Content { get; set; }
+2 -1
View File
@@ -25,6 +25,7 @@ namespace PrivaPub.Models.Social
Reblog,
Update,
Poll,
Reaction
Reaction,
Quote
}
}
+10
View File
@@ -110,6 +110,16 @@ namespace PrivaPub.Models.User
public string DefaultVisibility { get; set; } = "public";
public bool DefaultSensitive { get; set; }
public string DefaultLanguage { get; set; }
public string QuotePolicy { get; set; } = QuotePolicies.Public;//owner decision: anyone may quote a public post, automatically
}
public static class QuotePolicies
{
public const string Public = "public";
public const string Followers = "followers";
public const string Nobody = "nobody";
public static bool IsKnown(string value) => value is Public or Followers or Nobody;
}
public enum AvatarServer
+3 -1
View File
@@ -12,6 +12,7 @@ using Serilog;
using PrivaPub.Data;
using PrivaPub.Extensions;
using PrivaPub.Api.Mastodon.Auth;
using PrivaPub.Api.Mastodon.Infrastructure;
using PrivaPub.Infrastructure;
using PrivaPub.Infrastructure.Cli;
using PrivaPub.Infrastructure.Data;
@@ -78,7 +79,7 @@ try
try
{
BsonSerializer.RegisterSerializer(new GuidSerializer(GuidRepresentation.Standard));
BsonSerializer.TryRegisterSerializer(new GuidSerializer(GuidRepresentation.Standard));
var mongoSettings = builder.Configuration.GetSection(nameof(MongoSettings)).Get<MongoSettings>();
await DB.InitAsync(mongoSettings.Database, MongoClientSettings.FromConnectionString(mongoSettings.ConnectionString));
EntityMaps.Warm();
@@ -128,6 +129,7 @@ try
app.UseHttpsRedirection();
app.UseCors("DefaultCORS");
app.UseMastodonErrorBodies();
app.UseStaticFiles();
var mediaRoot = app.Services.GetRequiredService<PrivaPub.Domain.Media.IMediaService>().Root;
+6 -3
View File
@@ -14,8 +14,11 @@ namespace PrivaPub.Services
ILogger<JwtEvents> _logger { get; set; }
const string contentType = "application/json";
public override async Task AuthenticationFailed(AuthenticationFailedContext context)
public override async Task Challenge(JwtBearerChallengeContext context)
{
context.HandleResponse();
context.Response.StatusCode = StatusCodes.Status401Unauthorized;
context.Response.Headers.WWWAuthenticate = context.AuthenticateFailure == default ? "Bearer" : "Bearer error=\"invalid_token\"";
try
{
var localizer = context.HttpContext.RequestServices.GetRequiredService<IStringLocalizer<GenericRes>>();
@@ -26,7 +29,7 @@ namespace PrivaPub.Services
catch (Exception ex)
{
_logger = context.HttpContext.RequestServices.GetRequiredService<ILogger<JwtEvents>>();
_logger.LogError(ex, "Error at AuthenticationFailed()");
_logger.LogError(ex, "Error at Challenge()");
}
}
@@ -42,7 +45,7 @@ namespace PrivaPub.Services
catch (Exception ex)
{
_logger = context.HttpContext.RequestServices.GetRequiredService<ILogger<JwtEvents>>();
_logger.LogError(ex, "Error at AuthenticationFailed()");
_logger.LogError(ex, "Error at Forbidden()");
}
}
}
+12
View File
@@ -48,6 +48,18 @@ server {
proxy_read_timeout 300s;
}
location ^~ /media/proxy/ {
proxy_buffering off;
proxy_pass http://127.0.0.1:6970;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 600s;
}
location / {
proxy_pass http://127.0.0.1:6970;
proxy_http_version 1.1;
+54 -10
View File
@@ -11,7 +11,8 @@ Written 2026-10-01 from the original 2023 code, the decePubClient UI, a federati
- [x] P3 Social features: v1.5.0, deployed 2026-10-01; media, proxy, blocks, mutes, bookmarks, pins and reports verified by tests and locally (no upload on production yet)
- [x] P4 Groups and privacy features: v1.6.0, deployed 2026-10-01; communities, circles and local-only located posts verified by tests. v1.6.1 adds the pasture (`tools/pasture/`): live interop with GoToSocial 0.22.1 passes all 25 checks, three runs in a row. Lemmy and a live Mastodon circle member are not run yet; the pasture has GoToSocial only
- [x] P5 Lose nothing: v1.7.0 to v1.9.1, deployed 2026-10-01; parsing, typed details, provenance, downvotes, tombstones, federated blocks, all checked live against GoToSocial. Book reviews and forum threads keep their raw form only (typed in P7 and P8)
- [ ] P6 Emoji, polls, quotes, reactions, cards, players
- [x] P6 Emoji, polls, quotes, reactions, cards, players: v1.10.0 to v1.15.0, deployed 2026-10-01; polls, link cards, ranged video streaming and quote policies checked live against GoToSocial
- [ ] T/M Test sweep, the full pasture and the interaction ledger (see "Sweep and statistics" below)
- [ ] P7 Threads, communities, moderation, the social graph
- [ ] P8 Signatures, discovery, the long tail
@@ -128,6 +129,22 @@ and circles (see Owner decisions).
| PeerTube views | **Never sent.** The remote video file is still downloaded through our proxy when it is played; that cannot be avoided without pre-downloading video. |
| Bluesky bridging (Bridgy Fed) | **Allowed per persona**, with a clear warning that the posts become far more widely copied. Leaving the bridge must work, through a federated `Block` sent to the bridge. Persona creation dates are moved back by a random number of days, so personas created the same day no longer share a date. |
| Reactions and votes | **Public, with a one-time notice.** The client tells each persona once, before its first reaction or vote, that these are public and visible as that persona. |
| Who may quote a persona (default) | **Anyone, automatically**, as on Mastodon, for public and unlisted posts only. Each persona can change its default (followers only, or nobody) and each post can be changed; a granted quote can be revoked. Followers-only posts and DMs can never be quoted. |
| Quote permission address | `/peasants/{name}/parrot-licences/{id}` |
### Owner decisions on statistics (2026-10-03)
| Question | Decision |
|---|---|
| Fediverse statistics | **Recorded now, published later as per-server aggregates only.** The following are each kept for 90 days as an event that names the remote *server*, never a remote account or a local persona: every inbox answer, every processed activity, every delivery attempt and every outbound request. Each day folds into per-server counters kept indefinitely, plus weekly server snapshots. A future public page, `/stargazing`, shows per-server aggregates for educational use, never per account and never per persona. |
| Remote accounts in statistics | **Never stored.** Distinct accounts per server per day are counted with a keyed hash. Its key is made for that day, kept only until the day's rollup and then destroyed, so the hashes can be neither reversed nor linked across days. |
| Local side in statistics | **Only the kind of local actor** (person, group, application), **and only on public and unlisted traffic.** DMs, followers-only and circle traffic are one "private" class, never broken out per server in public. Circles are never named, whether as a kind or as a reason. Fetches of our own documents are counted per day, never per server. |
| Reading-driven traffic | **Counted per day, never logged per event:** the media proxy, lookups a client asks for, and the client API per endpoint group (admin only). Client app names are not recorded. |
| Describing servers | **Every server we exchange activities with is described weekly**, from its NodeInfo (including the user counts it publishes) and its Mastodon instance API, never its contact account. These requests are unsigned, because they are not ActivityPub documents. Never on read. |
| Server locations | **City and network (ASN) from the offline DB-IP Lite databases** (CC BY 4.0, attributed), downloaded monthly outside the app. The location comes from the address we connected to; an inbound sender's address is never recorded, and no address is stored. In public: city and network only for servers reporting at least 10 users and not behind a CDN; the country otherwise; only the CDN's name for CDN-fronted servers. The admin sees everything. |
| Crawler | **Off by default** (`Statistics:Crawler:Enabled`). When on, it identifies as `PrivaPub-Stargazer/<version> (+https://privapub.thepra.dev/stargazing)`, where `/stargazing` explains it and how to opt out. It honours robots.txt (an unreachable robots.txt means "keep out") and domain blocks. It visits one server a minute, each at most weekly, and at most 5000 servers. It reads only robots.txt, NodeInfo, the instance API and the peers list, never accounts, posts or directories. Crawled servers stay marked as crawled. |
| A remote account deletes itself | **Its posts are kept but hidden everywhere** (`Post.AuthorGone`): from timelines, profiles, search and lookups by id. Its follows and timeline rows go, as before. |
| Signed-in smoke check in production | **An undiscoverable persona**, whose read-only token is the Gitea secret `PRIVAPUB_SMOKE_TOKEN`; the deploy checks `verify_credentials`, home and notifications with it. The owner creates both. |
## Libraries (researched; no maintained .NET ActivityPub library exists, so Letterbook and Iceshrimp.NET both wrote their own)
@@ -171,6 +188,7 @@ were agreed on 2026-10-01.
| Group members | `/peasants/{group}/flock` | *new* |
| Group moderators | `/peasants/{group}/wardens` | *new* |
| DM conversation context | `/peasants/{name}/whispers/{id}` | *new*, replaces `/conversations/{id}` |
| Quote permission (FEP-044f stamp) | `/peasants/{name}/parrot-licences/{id}` | chosen by the owner, 2026-10-01 |
Routes outside the actor namespace stay conventional, because other software looks for them by name:
- `/.well-known/*` and `/nodeinfo/*`;
@@ -453,19 +471,26 @@ it, raw where it doesn't.
- **Polls** in and out, with the Misskey, Pleroma and PieFed vote shapes. A count refresh is never an edit (done in
v1.10.0; checked live both ways against GoToSocial).
- **Quotes (FEP-044f):**
- read every key; verify `QuoteAuthorization` on all of its fields; handle revocation;
- be quotable: `canQuote`, answer `QuoteRequest`, serve and revoke stamps;
- then advertise `api_versions.mastodon ≥ 7` (4.5.0).
- read every key; verify `QuoteAuthorization` on all of its fields; handle revocation (done in v1.13.0);
- personas quote others: `quoted_status_id`, `QuoteRequest`, `Accept{result}` verified, `Update` with
`quoteAuthorization`; legacy quotes for posts that state no policy (done in v1.13.0); `api_versions.mastodon = 7`;
- be quotable (done in v1.15.0): `canQuote` on our posts (owner's default: anyone, automatically), `QuoteRequest`
answered with a parrot-licence or `Reject`, licences served and revoked; personas set `source[quote_policy]`, posts
`quote_approval_policy`; `POST /statuses/:id/quotes/:quoting_id/revoke`, `PUT /statuses/:id/interaction_policy`.
- **Emoji reactions** in all three inbound forms, plus outbound `EmojiReact`, exposed as `emoji_reactions` (done in
v1.11.0; Pleroma's `/api/v1/pleroma/statuses/:id/reactions` endpoints and `pleroma:emoji_reaction` notifications).
- **Link cards:**
- **Link cards** (done in v1.12.0):
- from the object, or from FEP-8967 `preview`, without fetching;
- fetching the page itself is the owner's decision (INTEROP §6.1).
- **Media:**
- video playback through the proxy (Range requests, HLS playlist rewriting, the poster), with a `video` card;
- otherwise the server reads the page (owner decision 1): public posts only, 0–60 s after arrival, cached per address
for 7 days across the whole server, `Federation:FetchLinkPreviews` to switch it off. Checked live against a
GoToSocial profile page.
- **Media** (done in v1.14.0 except where noted):
- video playback through the proxy: range requests streamed from the origin, the poster, a playable attachment picked
from MP4 or fragmented MP4 files (HLS playlists are not rewritten; not needed while PeerTube publishes the files);
- audio attachments;
- an article reader view;
- JPEG/PNG renditions kept for Pixelfed.
- JPEG/PNG for Pixelfed: uploads were already re-encoded to JPEG, PNG or GIF;
- an article reader view: the data is in `Status.privapub` (title, excerpt, cover, full HTML in `content`); the view
itself belongs to the client.
#### P7 Threads, communities and the social graph
- **Thread backfill:**
@@ -511,6 +536,25 @@ it, raw where it doesn't.
- Pixelfed `place` (display only, never re-federated);
- Bluesky bridging per persona.
### Sweep and statistics (T and M, planned 2026-10-03, between P6 and P7)
Two tracks, interleaved so that the test host exists before the ledger, and the ledger starts collecting early, because
statistics gain value with every day recorded.
| Step | Content | Tag |
|---|---|---|
| T1–T4 | Tests stop sharing state they don't own. CI runs every test against a throwaway mongod. The whole server runs under test (`WebApplicationFactory`). Nothing answers 500. | v1.15.1 |
| M1–M6 | The interaction ledger: inbox answers, handler verdicts, delivery attempts, outbound requests, served and client traffic. Provenance fixes (fetched records, stored extensions, group-wrapped Update/Delete). | v1.16.0 |
| T5–T8 | Coverage sweep over HTTP: OAuth, `/clientapi`, the Mastodon API, federation GETs, inbox gaps, jobs, migrations, pages. | v1.17.0 |
| M7–M10 | Daily rollups (`InstanceDay`, `ServerDay`). Every touched server described weekly (NodeInfo usage, instance API, snapshots). Geolocation (DB-IP Lite city and ASN). Admin statistics API under `/clientapi/admin/statistics`. | v1.18.0 |
| T9–T14 | The pasture as plugins. GoToSocial gaps, then Mastodon, Misskey/Sharkey, Akkoma and Lemmy 1.0, each run also checking that peer's statistics. | v1.18.x |
| M11 | The opt-in crawler (`PrivaPub-Stargazer`) and the `/stargazing` explainer. | v1.19.0 |
Later, and not part of these steps: the public `/stargazing` statistics. It is anonymous, cached and rate-limited, and
covers per-server software, self-published counts, location as projected by the rule above, availability buckets and
the public activity mix. Per-server activity is shown only as order-of-magnitude buckets, and only for servers that
report at least 10 users; smaller ones fold into one "small servers" aggregate.
### Cut or deferred (deliberately)
- **Cut:**
- Link-preview cards *fetched from the linked page* by default. Cards built from the object, or from FEP-8967
+57
View File
@@ -0,0 +1,57 @@
#!/usr/bin/env bash
# Runs a command (normally `dotnet test`) with a throwaway mongod: a fresh data directory on a random localhost port,
# stopped and deleted when the command ends. The box's own mongods (27017 shared, 27022 PrivaPub's) are never touched.
# usage: tools/ci/with-test-mongod.sh dotnet test PrivaPub.sln -c Release
set -euo pipefail
root="${RUNNER_TEMP:-${TMPDIR:-/tmp}}/privapub-test-mongod"
mkdir -p "$root"
# Leftovers of a cancelled run: anything here older than two hours.
find "$root" -mindepth 1 -maxdepth 1 -type d -mmin +120 -print0 2>/dev/null | while IFS= read -r -d '' old; do
[ -f "$old/pid" ] && kill "$(cat "$old/pid")" 2>/dev/null || true
[ -f "$old/container" ] && podman rm -f "$(cat "$old/container")" >/dev/null 2>&1 || true
rm -rf "$old"
done
listening() { (exec 3<>"/dev/tcp/127.0.0.1/$1") 2>/dev/null; }
port=""
for _ in $(seq 1 20); do
candidate=$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1]); s.close()')
case "$candidate" in 27017|27022) continue ;; esac
listening "$candidate" && continue
port=$candidate; break
done
[ -n "$port" ] || { echo "::error::no free port for the test mongod"; exit 1; }
dir=$(mktemp -d "$root/run-XXXXXX")
cleanup() {
status=$?
[ -f "$dir/pid" ] && kill "$(cat "$dir/pid")" 2>/dev/null || true
[ -f "$dir/container" ] && podman rm -f "$(cat "$dir/container")" >/dev/null 2>&1 || true
for _ in $(seq 1 20); do listening "$port" || break; sleep 0.5; done
rm -rf "$dir"
exit $status
}
trap cleanup EXIT INT TERM
if command -v mongod >/dev/null 2>&1; then
mkdir -p "$dir/db"
timeout 7200 mongod --dbpath "$dir/db" --port "$port" --bind_ip 127.0.0.1 --noauth \
--wiredTigerCacheSizeGB 0.25 --quiet --logpath "$dir/mongod.log" &
echo $! > "$dir/pid"
how="mongod $(mongod --version | head -1)"
elif command -v podman >/dev/null 2>&1; then
podman run -d --rm -p "127.0.0.1:$port:27017" docker.io/library/mongo:8 --quiet > "$dir/container"
how="podman mongo:8"
else
echo "::error::neither mongod nor podman is available for the test mongod"; exit 1
fi
for _ in $(seq 1 60); do listening "$port" && break; sleep 0.5; done
listening "$port" || { echo "::error::the test mongod did not start"; cat "$dir/mongod.log" 2>/dev/null | tail -20; exit 1; }
echo "test mongod: $how on 127.0.0.1:$port"
export PRIVAPUB_TEST_MONGOD=1 PRIVAPUB_TEST_REQUIRE_MONGOD=1 PRIVAPUB_TEST_MONGO="mongodb://127.0.0.1:$port"
"$@"
+9
View File
@@ -114,6 +114,15 @@ gts_poll_on_pp=$(curl -s -H "$PH" "$P/api/v1/timelines/home" | j "print(next(s['
curl -s -o /dev/null -X POST -H "$PH" "$P/api/v1/polls/$gts_poll_on_pp/votes" -d 'choices[]=0'
until_true 20 '[ "$(gcurl -s -H "$GH" "$G/api/v1/statuses/$gts_poll" | j "print(d[\"poll\"][\"options\"][0][\"votes_count\"])")" = "1" ]' && ok "alice's vote counts on GoToSocial" || ko "vote not counted on GoToSocial"
echo "quotes"
quote_target=$(curl -s -H "$PH" "$P/api/v1/timelines/home" | j "print(next(s['id'] for s in d if 'red or blue' in s['content']))")
[ "$(curl -s -o /dev/null -w '%{http_code}' -X POST -H "$PH" $P/api/v1/statuses -d "status=quoting&quoted_status_id=$quote_target")" = "422" ] \
&& ok "GoToSocial's author-only quote policy is respected" || ko "quoted a post whose author forbids it"
echo "link previews"
linked=$(curl -s -X POST -H "$PH" $P/api/v1/statuses --data-urlencode 'status=have a look https://gts.test/@gtsuser' -d 'visibility=public' | j "print(d['id'])")
until_true 45 '[ -n "$(curl -s -H "$PH" "$P/api/v1/statuses/$linked" | j "print((d[\"card\"] or {}).get(\"title\") or \"\")")" ]' && ok "the server builds a card for a linked page" || ko "no card for the linked page"
echo "edits and deletes"
curl -s -o /dev/null -X PUT -H "$PH" $P/api/v1/statuses/$pp_post -d 'status=Hello from PrivaPub, edited'
until_true 20 'gcurl -s -H "$GH" "$G/api/v1/statuses/$pp_on_gts" | grep -q "edited"' && ok "alice's edit reaches GoToSocial" || ko "edit not applied"