Compare commits

...
31 Commits
Author SHA1 Message Date
thepraandClaude Opus 5.5 088c54ddf3 ROADMAP: P7 so far, deployed as v1.24.0
Build / Build (push) Successful in 8m5s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012CzABvBkbcFqoHdmi8b9WB
2026-10-08 02:33:50 +02:00
thepraandClaude Opus 5.5 6fccb6fe35 P7: downvotes go out, and feeds are read through the server's reader
Build / Build (push) Successful in 8m45s
Deploy / privapub.thepra.dev (push) Successful in 9m0s
Votes out. A persona's downvote is a Dislike (POST /api/v1/statuses/:id/downvote and /undownvote, the viewer's own as
privapub.votes.downvoted). One vote a post: a changed vote is sent as the new vote alone, as Lemmy sends one, since an
Undo of the old one beside it could arrive after it and take the new one away; Undo goes only when a vote is taken back.
A vote on a post in a remote community goes to the community, which counts it, and to the author only when on another
server: one copy a server, since PieFed drops a second copy of an activity it has just seen. Mbin keeps a favourite
apart from a vote, so there a favourite stays after a switch to a downvote.

Feeds followed (owner decision 2026-10-07: through the server). Following a feed (Lemmy's multi-community, PieFed's feed)
keeps a FeedSubscription for the persona and nothing else; the new Service privapub_feeds (LocalActorKind.Reader,
reserved by migration _017) follows every community of the feeds read here, reconciled when a persona follows or leaves
one, when a feed's list is read again (kept as it was when it cannot be read) and every six hours. Its Following rows
carry FollowerKind, so nobody's home gets what it brings in and its unanswered follows are sent again as its own. The
persona reads GET /api/v1/timelines/feed/:id (the feed's threads, ours included) and lists its feeds at GET /api/v1/feeds.

Checked in the pasture, every scenario: 873 pass. The 14 failures are Hubzilla's (identical on the previous commit:
Hubzilla no longer answers a follow in this pasture since its restore) and two activities Smithereen never sent;
followsync passes once the pasture's restore is older than the 14-day pause. Live: alice's downvotes count as downvotes
on Lemmy 1.0 and PieFed, replacing her upvote; Lemmy 1.0 and PieFed take privapub_feeds' follows and their threads reach
the feed timelines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012CzABvBkbcFqoHdmi8b9WB
2026-10-08 02:15:28 +02:00
thepraandClaude Opus 5.5 3b3f1f6b93 P7: feeds read as accounts, and PieFed's post Move
Build / Build (push) Successful in 7m50s
Lemmy 1.0's multi-communities and PieFed's feeds (`type: Feed`) resolve as accounts: the feed's `following` is kept
once a day with its counts, each community in it is read in turn, and `/api/v1/accounts/:id/following` lists them.
Lemmy's feed carries the instance's key and its text in `description`; both are taken as they are. Following a feed
answers 422 until the owner decides how a feed is followed.

A post moved to another community (PieFed's Move{object: post, origin, target}), relayed by the community it leaves,
goes there with its thread; a move into a community we host is not taken.

Checked in the pasture: a Lemmy 1.0 feed and a PieFed feed list their communities (ours among them); a PieFed post moved
from its web code arrives moved, with its comment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012CzABvBkbcFqoHdmi8b9WB
2026-10-07 21:42:54 +02:00
thepraandClaude Opus 5.5 62189bf11d P7: thread backfill reads conversation containers and asks with If-None-Match
Build / Build (push) Failing after 7m52s
The FetchReplies job reads a thread's FEP-171b contextHistory before its FEP-7888 context: the posts each Add (or
Forte's plain Create) brought in, read from their own servers. A thread collection read whole keeps its ETag when the
document itself changes with every post (it counts them, or holds them all with no further page); the next read sends
it as If-None-Match and a 304 ends the job. The ETag is kept as sent, since NodeBB's has no quotes and the typed header
drops it. A context naming a post we hold (Forte's first post) is not fetched.

Checked in the pasture: Mastodon 4.7.3 answers the second read 304; NodeBB 4.16's unquoted ETag is kept (23/23 in its
scenario); a Forte thread completes through its replies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012CzABvBkbcFqoHdmi8b9WB
2026-10-07 21:33:49 +02:00
thepraandClaude Opus 5.5 fb4949b511 P7: FEP-268d searchableBy decides what search finds
Build / Build (push) Successful in 7m58s
A remote account's searchableBy, and a post's own, outrank indexable in status search: Public lets anyone find a
public post, the author's followers only those who follow it, anything else nobody but those it already reaches
(their own, named, favourited, bookmarked or boosted posts). Read on actors and posts, kept through edits; PrivaPub
does not emit it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012CzABvBkbcFqoHdmi8b9WB
2026-10-07 21:17:36 +02:00
thepraandClaude Opus 5.5 d4b1788a4f P7: community flairs, from Lemmy and PieFed
Build / Build (push) Successful in 8m2s
- A post's flairs are read in both dialects: Lemmy 1.0's CommunityPostTag (id, slug, name, description, colour slot)
  and PieFed's lemmy:CommunityTag (display name, text and background colours, whether to blur images). Colours are
  kept only as a colour slot or a hex value.
- A community's own list comes from its `tag` and PieFed's older `lemmy:tagsForPosts`, and fills in a post that names
  only a flair's id and slug, as Lemmy's posts do. Edits and refreshes keep them current.
- Clients get them as `privapub.flairs`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012CzABvBkbcFqoHdmi8b9WB
2026-10-07 21:14:32 +02:00
thepraandClaude Opus 5.5 587997dd67 P7: Lemmy's Warn and Resolve{Flag}
Build / Build (push) Successful in 8m37s
- A Warn from a community's moderator about a persona's own post there becomes that persona's moderation_warning
  notification (Mastodon's AccountWarning, with the reason and the post), believed from the community's own server or
  from an account the community's moderators collection lists. Remote communities keep that collection's address
  (attributedTo) as ForeignAvatar.ModeratorsURL; it is read only when a warning needs it.
- A Resolve{Flag} for one of our reports (`/grunts/flag-<report id>`) is kept as the report's remote resolution when it
  comes from the server holding what was reported, or the community it was reported to; our own moderators'
  resolution is never replaced. The moderators' report list shows both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012CzABvBkbcFqoHdmi8b9WB
2026-10-07 21:10:58 +02:00
thepraandClaude Opus 5.5 6fc2d001b2 ROADMAP: media, backups and archives deployed (v1.23.0, decePub v1.5.0)
Build / Build (push) Successful in 7m41s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
2026-10-07 17:55:58 +02:00
thepraandClaude Opus 5.5 4eda3da75d mongod's oplog size flag is --oplogSize on the command line
Build / Build (push) Successful in 8m26s
Deploy / privapub.thepra.dev (push) Successful in 8m53s
--oplogSizeMB is the configuration file's key (replication.oplogSizeMB); mongod refuses it as a flag, so setup.sh's
replica-set step left the unit restarting until the flag was fixed on Max.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
2026-10-07 17:32:02 +02:00
thepraandClaude Opus 5.5 163f7e4e2b An import reads the archive's followers once, finds threads in linear time, and imports a located post once
A post's visibility no longer parses actor.json again for every outbox item; a reply's conversation is its thread's
first post's, found from what the first pass read instead of searching every planned post (quadratic on a large
archive); and a located post always records where it came from, so importing a persona's own archive twice made it
twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
2026-10-07 12:51:58 +02:00
thepraandClaude Opus 5.5 44a5895b0c ROADMAP: the owner's decisions on media, backups and archives, and what the next deploy needs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
2026-10-07 12:50:17 +02:00
thepraandClaude Opus 5.5 3d7d406834 A persona's archive: exported in Mastodon's layout, imported without telling anyone
A root exports one of its personas (a job) as Mastodon's account archive, so other servers' importers read it: the
actor with its public key only, its own posts and boosts with their media, likes, bookmarks and Mastodon's CSV files,
plus PrivaPub's filters, followed hashtags, notification policy, pins, scheduled and located posts; nothing of its root,
its siblings, its keys or anyone's token. A ticket link downloads it, for a week.

An archive (PrivaPub's or Mastodon's) uploaded in pieces is imported into a persona in a job, the parts the root picks,
with progress and a stop. SafeArchive refuses links, escaping paths, duplicates, bombs and oversized items, and reads the
outbox one item at a time. Imported posts are delivered to no one, put in no home and notify nobody, yet show on the
profile, outbox, hashtags and search; back home a post keeps its id, from another actor it gets one of its date and
ImportedFromURI, so importing twice changes nothing. Relationships go through the existing services; located, scheduled
and likes only when asked; followers never.

tools/pasture/scenarios/persona-archive.sh imports mastouser's real Mastodon archive (156 posts, 24 pictures) into a
persona Mastodon follows: Mastodon receives none of it, and a second import changes nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
2026-10-07 12:43:12 +02:00
thepraandClaude Opus 5.5 bdc8be4508 A restore on the live pasture: its own scenario, and two fixes it found
tools/pasture/scenarios/restore.sh backs the pasture up from the administrator's endpoint, then alice_restore deletes a
post and makes another, mastouser follows her, she blocks bob_restore and carol_restore is made; the restore, asked for
from the endpoint, keeps every protective act (19 checks). town.sh renew <peer> signs a peer's town accounts in again,
since a restore ends every session.

It found that a backup listed media files already missing when it was made, so verifying it failed and the restore was
refused: a manifest now lists only the files it holds (refusals are cut to five lines). And a local post made after the
backup now comes back as a deleted row, as a deletion leaves it, so it answers 410 and its id is never given again;
DeletedObject holds remote tombstones only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
2026-10-07 12:24:23 +02:00
thepraandClaude Opus 5.5 9ad96f560d FEP-8fcf rests for two weeks after a restore
What a restore lost of either side is not the other server's to mend: for RestoreRecord.FollowersGrace (14 days)
deliveries carry no Collection-Synchronization header, and a follow only the remote remembers is adopted rather than
undone. Without a restore nothing changes; the interop sweep (gts, mastodon, misskey, sharkey, akkoma, pixelfed,
smithereen, followsync, pins, moves) passed, Smithereen's poll check on a second run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
2026-10-07 12:16:35 +02:00
thepraandClaude Opus 5.5 bc5f2beaf7 The administrator's page backs up, downloads, uploads and restores
/clientapi/admin/backups (owner decision 2026-10-07, approving these endpoints in production): the list with what runs,
the restore waiting and the last restore's report; back up now; delete; a download for the password, through a ticket
good for ten minutes in the link's path, as one tar whose length is known first and which honours Range (BackupTar);
an upload in pieces of at most 32 MB, each at the offset already received or refused with it, so a broken upload
resumes, read into a backup only when it holds nothing but plain files under one backup's folder; and a restore for
the password and the host typed out. Every call checks the administrator against the database. nginx streams downloads
for an hour and takes upload pieces unbuffered, rate-limited.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
2026-10-07 12:01:03 +02:00
thepraandClaude Opus 5.5 fba57318fa A backup is restored at boot, and never undoes a protective act
PrivaPub admin restore <id> (and soon the administrator's page) checks the backup (same host, a format and newest
migration this build reads, every hash) and writes restore.json; the running service sees it within seconds and stops,
and the next start restores it in MaintenanceGate, before migrations, indexes and hosted services: a pre-restore backup
taken once, every collection dropped and imported raw with its indexes, the media the live directory lacks brought
back, then the protective merge from the pre-restore backup. Followers and follows are the live ones; blocks, mutes,
domain blocks, reserved names, tombstones, reports, filters and OAuth applications are the union; deletions win;
accounts made since become tombstones and local posts made since answer 410; every session ends.

Each attempt redoes everything; one refused before any change is abandoned and recorded, one failed midway exits 1 for
systemd to retry, and after three it exits 75, which the unit no longer restarts. Commands wait (exit 75) while a
restore is pending. RestoreRecord tells what happened (admin restore --status).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
2026-10-07 11:55:17 +02:00
thepraandClaude Opus 5.5 12bb75809f The server backs itself up: every collection byte for byte, the media linked
A backup is a directory <stamp>-<kind> under Backups:Root (/var/lib/privapub/backups, 2770, files 0640), written as
.partial and renamed once whole: a manifest (host, build, newest migration, each collection's count, size, sha256 and
indexes, what was left out and why, the media list), each collection as gzipped canonical Extended JSON read raw, and
hard links to the files of untrashed media rows (copies where a link can't be made). On a replica set every collection
is read in one snapshot session. Never in a backup: the statistics salt, jobs, recovery codes, sessions, the
maintenance lock and the configuration's copy with its SMTP password.

One backup or restore at a time (MaintenanceLock, a heartbeat document), and the janitor purges nothing meanwhile.
BackupScheduler backs up nightly at 03:30 UTC (or at once after missing a night); rotation keeps 7 daily, 4 weekly,
3 pre-deploy and 3 pre-restore backups. CLI: admin backup [--kind] [--db-only], admin backups, admin backup verify;
these run before migrations, so the deploy's own pre-deploy backup, which replaces mongodump, is of the database as the
live build left it. EntityMaps.Warm runs once under a lock, since test hosts now boot side by side.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
2026-10-07 11:40:06 +02:00
thepraandClaude Opus 5.5 37b12c5fee Mongo is a one-member replica set
Owner decision 2026-10-07: production's mongod becomes a one-member replica set (rs0), so that a backup can read every
collection at one instant (a snapshot read session), which a standalone mongod can't. The unit runs mongod with
--replSet rs0 and a 990 MB oplog; setup.sh converts it once, idempotently (PrivaPub stopped, mongod restarted,
rs.initiate, the primary awaited, PrivaPub started); every connection string says directConnection=true, which works
against the standalone too, so this code can deploy before the conversion. The CI's throwaway mongod and the pasture's
are replica sets as well, so the snapshot path is what the tests exercise; MongoTopology tells which a mongod is, and
TopologyTests holds the test mongod to it. The suite passes on it (906).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
2026-10-07 11:21:13 +02:00
thepraandClaude Opus 5.5 830385ea2b An empty media_ids takes an edited post's media away
A JSON body's empty array flattened to nothing, so `media_ids: []` on an edit read as "media not sent" and the
post kept every attachment, where Mastodon removes them. An empty array is now a parameter with no values: the edit
leaves the post without media, and they go to the trash. MediaLifecycleTests checks it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
2026-10-07 11:07:17 +02:00
thepraandClaude Opus 5.5 dc63fa57b8 A login's storage is counted, and can have a quota
MediaAttachment.Size was stored and never summed: nobody, the administrator included, could tell what media took,
and nothing bounded it. Counted.Media and MediaOfRoot sum what is kept (not trashed). A login sees what its personas'
uploads and pictures take at /clientapi/user/storage (ViewStorage), with its quota when the server sets one;
Media:QuotaBytesPerRoot (0, no quota, by default) refuses an upload or a picture over it with 422, whichever persona
sends it; the statistics overview gains the media totals, the proxy cache and the trash (ViewMediaTotals). Tests: a
login's storage counts what its personas keep and forgets what is trashed; two uploads from two personas of one login
are refused together past the quota.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
2026-10-07 11:03:25 +02:00
thepraandClaude Opus 5.5 5252b8f320 A posted picture is described again by editing its post
PUT /api/v1/media/:id on a picture already posted changed the upload but never the post, which keeps its own copy, so
the new description or focal point reached nobody. As on Mastodon, PUT now takes only media not posted yet (404
otherwise), and a status edit takes media_attributes[] (id, description, focus; from a JSON body or a form): the
uploads and the post's copies change, and the Update the edit federates carries them. An edit that sends no media
keeps them as they were described. No pasture scenario edits a description, so the sweep could not see this.
MediaLifecycleTests: PUT on a posted picture is refused, an edit describes it again and keeps its warning, flag and
language as sent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
2026-10-07 11:00:04 +02:00
thepraandClaude Opus 5.5 a4ced664cf The proxy's shared download is started once
ConcurrentDictionary.GetOrAdd may run its factory more than once when asked at the same time, and each run started a
download: five clients asking at once for a new file made three upstream requests in one run of
Clients_asking_at_once_share_one_download. The factory now makes a Lazy, of which only the one kept is ever started,
and a download that begins just after another cached the file uses it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
2026-10-07 11:00:04 +02:00
thepraandClaude Opus 5.5 8fecf7de2d A persona's avatar and header can be removed
Mastodon's DELETE /api/v1/profile/avatar and /api/v1/profile/header: the picture goes to the trash (its file stops
being served at once), the profile shows the default again, followers are sent the updated profile, and the answer is
the account with its source. The profile Update is the one a persona that never had a picture sends, so nothing new
leaves PrivaPub. MediaLifecycleTests: the avatar, then the header, removed and no longer served.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
2026-10-07 10:55:12 +02:00
thepraandClaude Opus 5.5 3ca29603ed The media proxy is bounded
Anyone could mint signed proxy URLs (a remote account changes its icon, an anonymous lookup returns the URL), and each
anonymous request held up to 40 MB in memory; the cache grew without bound between hourly trims; two clients asking
for the same new file downloaded it twice and wrote over each other in place, so a reader could get half a file with a
7-day cache header; a file over the limit was downloaded twice on every request; a failed fetch, a 404, was cached by
browsers for a week; cached media of a server suspended later were still served, and RejectMedia skipped avatars,
emoji, covers, video variants, link cards and remote edits; /clientapi/group/members returned remote pictures raw.

Now a download is shared by everyone asking at once, streamed into a .part file and renamed into place
(FederationHttp.DownloadMedia copies bounded, never into memory), at most eight at a time; a file too big to cache is
remembered for an hour and only streamed, a failure for five minutes; the cache's size is counted as it grows and
trimmed as soon as it passes the cap; a cached file is opened before it is answered; browsers may cache only a
success; nothing of a suspended server, or of one whose media are rejected, is proxied (everything remote a client
sees goes through the proxy, so that covers every kind), and blocking one purges its cache; the proxy has its own rate
limit per client address; group members' pictures are proxied; the proxy's key is loaded once, the oldest if two
were made. This changes what PrivaPub serves its clients, not what it sends to other servers.

Tests: clients asking at once share one download, a failure isn't cached by browsers, an over-limit file is fetched
three times for two requests instead of four, a blocked server's media are refused and its cache purged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
2026-10-07 10:54:33 +02:00
thepraandClaude Opus 5.5 8e59145826 Audio and video are processed off the request
Every audio and video upload was probed and remuxed inside the request, whatever its length; ffmpeg would read any
protocol and probe any format; `-map 0` kept the data tracks iPhones add, which mp4 refuses; nothing was ever
transcoded, so HEVC or MPEG-4 Part 2 reached browsers that can't play them, and the advertised video_matrix_limit
and frame rate limit were never applied; the output was read whole into memory, the video was saved before its
poster could fail, and the poster's frame leaked in /tmp. FLAC uploads were served as 404.

Now an upload sent to /api/v2/media is stored as sent in media-incoming (beside the media root, never served) and
answered with 202 and no url, while a ProcessMedia job, one at a time, does the work; GET /api/v1/media/:id answers 206
until it is ready, or 422 with why, and media still processing can't be posted. v1 processes before answering.
ffmpeg reads only that file (protocol whitelist, format forced from the probe) and drops data and subtitle tracks. A
video browsers play as it is (H.264, VP8, VP9, AV1 within Media:MaxVideoPixels and MaxFrameRate) is remuxed, anything
else transcoded to H.264 that fits, as Mastodon does; longer than Media:MaxSeconds is refused. Outputs move into place
only once everything succeeded, every temporary file goes, durations are kept, FLAC is served as audio/flac, and the
unit gets PrivateTmp. The instance API advertises the limits that are now applied.

No pasture scenario uploads audio or video through PrivaPub, so the sweep could not see this. MastodonMediaTests: v2
answers 202 then the job makes it playable (and an unreadable file 422 once processed), media still processing can't
be posted, MPEG-4 Part 2 becomes H.264, a video over the limit is made smaller, FLAC is served.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
2026-10-07 10:48:56 +02:00
thepraandClaude Opus 5.5 e4f9d0b61e An image is checked before it is decoded
An upload went straight to libvips: whatever loader recognised the bytes ran (an SVG sent as image/png was rasterised),
nothing bounded how many pixels it would decode to (a small PNG could decode to gigabytes, three times over), a GIF
was loaded frame by frame and never resized, and all of it ran inside the request with nothing limiting how many at
once. A GIF was typed gifv but stayed a .gif, which a gifv player can't play; its metadata was kept; colours lost
their ICC profile without being converted; HEIC was advertised but the bundled libvips can't decode it.

Now:
- only libvips' JPEG, PNG, GIF, WebP and HEIF loaders ever run on an upload (every other loader is blocked);
- the header alone says how big an image would decode, refused above Media:MaxPixels (40 MP) or MaxFrames;
- a still image is shrunk on load, turned by its orientation and brought into sRGB (thumbnail), then written without
  metadata, a profile picture the same way;
- an animated GIF becomes a looping silent H.264 mp4 typed gifv, as on Mastodon (PostMedia.Kind keeps it a gifv),
  and a remote GIF is an image;
- processing runs Media:Concurrency at a time, and uploads have their own rate limit per credential;
- HEIC and HEIF are no longer offered.

Tests: only the upload formats load, the header tells the size, an SVG posing as a PNG and an image too large are
refused before decoding, an animated GIF becomes a gifv and a still one an image, HEIC isn't advertised. The media
scenarios against the pasture (GoToSocial, Mastodon, Misskey, Akkoma, Pixelfed, Smithereen, Vernissage, Castopod,
PeerTube) pass: 329 checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
2026-10-07 10:48:44 +02:00
thepraandClaude Opus 5.5 65938bb2a0 Media are reserved for one post, atomically
An upload's owner checked that it was free, then attached it in a second step, so two posts asking for the same
upload at once could both get it; a post now, or an edit, could take media a scheduled post held, which then failed
when its time came (only logged) and was deleted; and a persona deleted or banned after scheduling a post still
published it.

Now a post claims its media in one conditional update before anything is written (its id is minted first), and a
post that loses the race is refused with 422, the media it took put back. Scheduling reserves media the same way.
Media held by a scheduled post are that post's alone, and its job publishes only for a persona still there whose root
is neither deleted nor banned; otherwise what it held is trashed. MastodonScheduledStatusesTests: a post now and an
edit can't take scheduled media, two racing posts never both get an upload, a gone persona's scheduled post never
publishes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
2026-10-07 10:34:16 +02:00
thepraandClaude Opus 5.5 bb680e8cb8 A file lives exactly as long as something holds it
Deleting a post, editing media out, replacing an avatar or a header, and removing a whole root deleted no file: every
one stayed on disk and publicly served from /media/files with a year-long immutable cache, its row orphaned. Now every
upload is a row, profile pictures too (Kind avatar or header, ProfileOfAvatarId), and each of those acts trashes what
it held, as does an upload never posted for a day and a dropped scheduled post. A trashed row is marked in one
conditional update (an upload attached meanwhile is left alone), its files move into media-trash, beside the media
root and outside what /media/files serves, and the janitor deletes them a day later. Nothing is deleted for looking
unused.

Along the way: a profile picture that isn't an image, or can't be read, answers 422 instead of being silently ignored
with a 200; a removed root's scheduled posts are dropped, so nothing of it publishes later; media rows get indexes
(they had none), and the janitor's first pass comes five minutes after boot instead of an hour.

`PrivaPub admin media audit [--fix]` compares the disk with the database. With --fix (as www-data) it gives the
pictures personas show today a row, and trashes media of deleted posts or personas, rows whose files are missing, and
files nothing holds: the leftovers of every deletion until now. MediaLifecycleTests covers each act, that the trash is
never served, and the audit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
2026-10-07 10:31:14 +02:00
thepraandClaude Opus 5.5 52d201eee9 A focal point is two finite numbers
float.TryParse takes "NaN" and "Infinity", and Math.Clamp keeps a NaN, so `focus=NaN,NaN` on an upload or a media PUT
was stored as given. Once the media was posted, every status and timeline response holding the post, and the Note
and its delivery, failed to serialise, for every viewer. FocalPoint.Parse takes finite numbers only (anything else is
ignored, as an unreadable focus was); the mapper and the renderer leave out a focus that isn't sound, and migration
_016 removes the ones stored before, from the uploads and from the copy each post keeps. What PrivaPub sends is
unchanged for any focus that could be sent before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
2026-10-07 10:23:09 +02:00
thepraandClaude Opus 5.5 402f9e0d75 A long job keeps its lease, and runs once
A lease lasted two minutes and was never renewed, so the reaper gave any longer job to a second worker while the first
still ran it, and both finished it. The worker now renews the lease every third of its length while the handler runs;
a lease found taken (reaped and leased again) cancels the handler. Each lease carries its own owner stamp, since every
worker of a process shares one name, and Finish only counts for the lease it was given. Media processing and persona
archives will run longer than two minutes. JobQueueTests: a job three times its lease runs once with the reaper finding
nothing, and a stolen lease stops its handler and drops its outcome.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
2026-10-07 10:20:24 +02:00
thepraandClaude Opus 5.5 8492f24064 The deploy's dump leaves out the statistics salt
The pre-deploy mongodump kept InteractionSalt, the day's salt that the owner decided must be destroyed at each rollup:
a dump kept seven deploys long let the day's actor hashes be reversed. It is excluded now, and the deploy refuses a
dump that still names it (mongorestore's dry run lists every collection). The dumps already on the box still hold
old salts; removing them waits for the owner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
2026-10-07 10:16:35 +02:00
141 changed files with 9554 additions and 363 deletions

No files matched your search

+18 -7
View File
@@ -12,8 +12,7 @@ env:
BACKUPS: /var/backups/privapub.thepra.dev
LOCAL_URL: http://127.0.0.1:6970
PUBLIC_URL: https://privapub.thepra.dev
MONGO_URI: mongodb://127.0.0.1:27022
MONGO_DB: PrivaPub
SERVER_BACKUPS: /var/lib/privapub/backups
jobs:
site:
@@ -57,12 +56,24 @@ jobs:
echo "SNAPSHOT=$BACKUPS/site-$STAMP" >> "$GITHUB_ENV"
ls -1dt "$BACKUPS"/site-* 2>/dev/null | tail -n +4 | xargs -r rm -rf || true
- name: Dump the database
# The server's own backup (PrivaPub admin backup, owner decision 2026-10-07) of the database as the live build left it:
# the new build's command runs before its migrations. The media are listed, not linked (the runner may not link
# www-data's files; the nightly backups hold them). Never the statistics salt, which must not outlive its day (owner
# rule), nor the configuration's copy with its secrets. No deploy while a restore waits for its boot.
- name: Back up the database
run: |
DUMP="$BACKUPS/mongo-$(date +%Y%m%d-%H%M%S).archive.gz"
mongodump --quiet --uri "$MONGO_URI" --db "$MONGO_DB" --gzip --archive="$DUMP"
echo "::notice::database dumped to $DUMP ($(du -h "$DUMP" | cut -f1))"
ls -1t "$BACKUPS"/mongo-*.archive.gz 2>/dev/null | tail -n +8 | xargs -r rm -f || true
[ ! -e "$SERVER_BACKUPS/restore.json" ] || { echo "::error::a restore is pending or running ($SERVER_BACKUPS/restore.json): deploy after it"; exit 1; }
out=$(cd "$GITHUB_WORKSPACE/publish" && ASPNETCORE_ENVIRONMENT=Production ./PrivaPub admin backup --kind pre-deploy --db-only)
echo "$out"
id=$(echo "$out" | grep -oE '^[0-9]{8}-[0-9]{6}-pre-deploy' | tail -1)
[ -d "$SERVER_BACKUPS/$id" ] || { echo "::error::the backup was not made"; exit 1; }
(cd "$GITHUB_WORKSPACE/publish" && ASPNETCORE_ENVIRONMENT=Production ./PrivaPub admin backup verify "$id")
if [ -e "$SERVER_BACKUPS/$id/db/InteractionSalt.jsonl.gz" ]; then
echo "::error::the backup holds the statistics salt"
exit 1
fi
echo "BACKUP_ID=$id" >> "$GITHUB_ENV"
echo "::notice::database backed up as $id"
- name: Stop, sync, start
run: |
+4
View File
@@ -257,6 +257,7 @@ Generated_Code/
# because we have git ;-)
_UpgradeReport_Files/
Backup*/
!PrivaPub/Infrastructure/Backup/
UpgradeLog*.XML
UpgradeLog*.htm
ServiceFabricBackup/
@@ -401,6 +402,9 @@ FodyWeavers.xsd
PrivaPub/media-store/
PrivaPub/media-store-proxy/
PrivaPub/media-store-trash/
PrivaPub/media-store-incoming/
PrivaPub/media-store-backups/
tools/pasture/.publish/
tools/pasture/.flood/
.claude/worktrees/
+162 -14
View File
@@ -22,8 +22,12 @@ Its defining idea: **one private login owns several public personas.**
- **`DmGroup` is a direct-message conversation.**
- **`privapub` is the instance actor** (type Application). It signs fetches no persona should be tied to.
- **`privapub_reports` is the reporter** (type Service, `LocalActorKind.Reporter`). It carries reports to Lemmy, which
takes none from an Application, and to PieFed and Mbin, and names nobody. Both are server actors
(`LocalActor.IsServerActor`): never followed, mentioned or shown as accounts.
takes none from an Application, and to PieFed and Mbin, and names nobody.
- **`privapub_feeds` is the reader of feeds** (type Service, `LocalActorKind.Reader`). When a persona follows a feed
(Lemmy's multi-community, PieFed's feed), it follows the feed's communities in the server's name
(`Domain/Social/FeedFollows.cs`, owner decision 2026-10-07); its `Following` rows carry `FollowerKind`, and nobody's
home gets what it brings in. All three are server actors (`LocalActor.IsServerActor`): never followed, mentioned or
shown as accounts.
Privacy features in the model:
- location-ranged posts (`Post.Location`, `RangeKm`), to become local-only and never federated;
@@ -226,7 +230,9 @@ group www-data and reaches the private mongod; `sudo -u www-data` works too.
marks (`Domain/Statuses/ConversationStates.cs`; writing in a conversation reads it).
10. **Nothing slow happens inside a request.** Deliveries and inbox processing are `Job`s (`Infrastructure/Jobs`):
leased, retried on Mastodon's curve, at most two per host, paused per host by `RemoteInstance`. The inbox answers
202 once it has verified and queued; a handler must be idempotent (unique `ObjectURI`, job `DedupeKey`).
202 once it has verified and queued; a handler must be idempotent (unique `ObjectURI`, job `DedupeKey`). A lease
(2 minutes, stamped with its own owner) is renewed every third of its length while the handler runs, so a long job
runs once; a lease found taken cancels the handler, and `Finish` only counts for the lease it was given.
11. **Every number about posts and users goes through `Domain/Privacy/Counted`** (owner decision 2026-10-04: one
answer everywhere): a persona's `statuses_count`, its outbox `totalItems`, NodeInfo `localPosts` and the instance
`status_count` all count Mastodon's way (not deleted, not a DM, boosts included, circle and located posts too);
@@ -286,7 +292,9 @@ group www-data and reaches the private mongod; `sudo -u www-data` works too.
(`NotificationPermission`).
7. **What a Mastodon `Status` cannot say goes in `Status.privapub`** (`PrivaPubStatus`): object type, title, excerpt,
cover, the author's source, link, video, audio and event details, a remote post's own place (Pixelfed's `location`,
never federated again), and up/down votes. Every media URL in it goes
never federated again), up/down votes and the viewer's own downvote (`votes.downvoted`, set through PrivaPub's
`POST /api/v1/statuses/:id/downvote` and `/undownvote`; a favourite is the upvote), and its community's flairs
(`ObjectShapes.Flairs`, both dialects). Every media URL in it goes
through the proxy; only page links (`link.url`, an event's online link) point at the remote site, because following
one is the reader's choice.
@@ -294,18 +302,72 @@ group www-data and reaches the private mongod; `sudo -u www-data` works too.
1. **No upload keeps its metadata.** Images are re-encoded by libvips with `keep=none`; audio and video are remuxed with
`-map_metadata -1`. `MediaProcessingTests` checks EXIF and XMP are gone.
2. Files live under `Media:Root` (`/var/lib/privapub/media`), never in the published directory; the proxy cache is the
sibling `media-proxy`, which `/media/files` does not serve.
3. **A client never contacts a remote server for media:** every remote URL the API returns goes through
2. **An image is checked before it is decoded.**
- **Loaders:** only libvips' JPEG, PNG, GIF, WebP and HEIF loaders ever run on an upload (`MediaService`'s static
constructor blocks every other loader), so an SVG, PDF or TIFF claiming another type never loads.
- **Size:** the header alone tells width × height (× frames for a GIF), refused above `Media:MaxPixels` (40 MP) or
`Media:MaxFrames`.
- **Then:** the image is shrunk on load to `MaxImageSide`, turned by its orientation, and its colours brought into
sRGB (`thumbnail`).
- **GIFs:** an animated GIF becomes a looping silent H.264 mp4 typed `gifv`, as on Mastodon (ffmpeg with libx264;
`PostMedia.Kind` says so), and a still GIF is an image.
- **Types:** HEIC and HEIF are not accepted, since the bundled libvips has no HEVC decoder.
- **Load:** processing runs `Media:Concurrency` at a time, and uploads have their own rate limit (`uploads`, per
credential).
3. **Audio and video are processed off the request when the client allows it.**
- **v2:** `POST /api/v2/media` stores them as sent in `media-incoming` (beside the root, never served) and answers
202 with no url, while a `ProcessMedia` job (one at a time, its lease renewed) does the work. Until then
`GET /api/v1/media/:id` answers 206, and 422 with the reason if it failed. Media still processing can't be posted.
- **v1:** processes before answering.
- **ffmpeg** reads only the stored file: `-protocol_whitelist file`, and the input format is forced from the probe.
Data and subtitle tracks are dropped (`-dn -sn`, which iPhone MOVs need).
- **Remux or transcode:** a video browsers play as it is (H.264, VP8, VP9 or AV1, within `Media:MaxVideoPixels` and
`MaxFrameRate`) is remuxed; anything else is transcoded to H.264 that fits, as Mastodon does.
- **Limits:** longer than `Media:MaxSeconds` is refused.
- **Saving:** nothing is saved until everything succeeded; outputs move into place rather than being read into
memory, and every temporary file goes. FLAC is served as `audio/flac`, and the unit runs with `PrivateTmp`.
4. Files live under `Media:Root` (`/var/lib/privapub/media`), never in the published directory; the proxy cache is the
sibling `media-proxy` and the trash the sibling `media-trash`, neither of which `/media/files` serves.
5. **A file lives exactly as long as something holds it.** Every upload is a `MediaAttachment` row, profile pictures
too (`Kind` avatar or header, `ProfileOfAvatarId`). Deleting a post, an edit leaving media out, a replaced picture,
a dropped scheduled post, a removed root, and an upload never posted for a day each trash theirs
(`IMediaService.Trash`):
- the row gets `TrashedAt` in one conditional update, so a row attached meanwhile is left alone;
- its files move into `media-trash` at once, so `/media/files` stops serving them;
- `MediaJanitor` deletes them a day later (`TrashGrace`).
What a login's media take (every persona's, trashed ones no longer) is `Counted.MediaOfRoot`, shown to the login at
`/clientapi/user/storage`; `Media:QuotaBytesPerRoot` (0, none, by default) refuses an upload over it with 422; the
administrator's statistics overview has the totals, the proxy cache and the trash.
Nothing is deleted because it looks unused. `PrivaPub admin media audit [--fix]` compares disk and database: with
`--fix` (as www-data) it gives pictures shown from before their rows a row, and trashes media of deleted posts or
personas, rows whose files are missing, and files nothing holds.
6. **A client never contacts a remote server for media:** every remote URL the API returns goes through
`IMediaProxy.Wrap`, an HMAC-signed `/media/proxy/` URL fetched by `IFederationHttp.GetMedia`.
4. **The proxy serves three ways:**
- **Cached:** a file already cached is served from disk, ranges included.
- **Downloaded:** a request without a `Range` is downloaded whole, up to `Media:MaxProxiedBytes`, then cached.
7. **The proxy serves three ways:**
- **Cached:** a file already cached is served from disk, ranges included. It is opened before the answer, so a trim
that deletes it meanwhile breaks nothing.
- **Downloaded:** a request without a `Range` is downloaded whole, up to `Media:MaxProxiedBytes`, then cached:
- once for everyone asking for it at the same time;
- streamed into a `.part` file beside its place and renamed there, never held in memory;
- with its host in its `.type` sidecar, so that blocking a server can purge it.
- **Streamed:** a ranged request, or anything too big to cache, is streamed from the origin with the range passed on,
and never cached. That is how remote video plays.
and never cached. That is how remote video plays. A file found too big is remembered for an hour, so it isn't
fetched twice; one that failed is remembered for five minutes.
Also:
- The cache's size is counted as it grows and trimmed as soon as it passes `ProxyCacheBytes`; the janitor
recounts hourly.
- Browsers may cache only a success.
- Nothing of a suspended server, or of one whose media are rejected, is proxied (avatars, emoji, covers and link
cards included, since they all go through it), and blocking one purges its cache.
- The `proxy` rate limit counts per client address.
nginx has a `/media/proxy/` location with `proxy_buffering off` and a 600 s read timeout for those streams.
5. **Remote video and audio become one playable attachment** in the Mastodon API (`MastodonMapper.Playable`): the best MP4
8. **A focal point is two finite numbers** within -1..1 (`FocalPoint.Parse`); anything else is ignored. A stored NaN made
every status, timeline and Note holding its post fail to serialise; migration `_016` removed the ones stored before.
9. **Remote video and audio become one playable attachment** in the Mastodon API (`MastodonMapper.Playable`): the best MP4
up to 720p that carries both sound and picture, including PeerTube's fragmented files inside an HLS entry. HLS
playlists themselves are not rewritten.
@@ -424,7 +486,89 @@ group www-data and reaches the private mongod; `sudo -u www-data` works too.
`_NNN_` order, each runs once), then `Indexes.Create()`. A new entity needs nothing; a new unique index needs a
dedupe migration before it.
- Production runs its **own mongod** on 127.0.0.1:27022 (unit `privapub-mongod`, no auth, data in
`/var/lib/privapub/mongo`). The box's shared mongod needs credentials nobody here has.
`/var/lib/privapub/mongo`). The box's shared mongod needs credentials nobody here has. It is a one-member replica set
(`--replSet rs0`, a 990 MB oplog; owner decision 2026-10-07), so a backup reads every collection at one instant;
`setup.sh` converts it once (PrivaPub stopped, mongod restarted, `rs.initiate`), and every connection string says
`directConnection=true`, which also works against a standalone. The pasture's mongo is one too.
- **Backups** (`Infrastructure/Backup/`; owner decisions 2026-10-07: the whole server, plain files protected by their
permissions, run from the CLI, nightly and from the administrator's page). Each backup is a directory
`<yyyyMMdd-HHmmss>-<kind>` under `Backups:Root` (`/var/lib/privapub/backups`, www-data 2770, files 0640), written as
`.partial` and renamed once whole:
- `manifest.json` (`ArchiveManifest`): host, build, newest migration, whether it was read at one instant, each
collection's count, size, sha256 and indexes, what was left out and why, and the media list;
- `db/<collection>.jsonl.gz`: each document as one line of canonical Extended JSON, read raw, so every BSON type comes
back byte for byte; on a replica set every collection is read in one snapshot session
(`minSnapshotHistoryWindowInSeconds` is raised to an hour only while it reads);
- `media/`: hard links to the files of untrashed `MediaAttachment` rows (no disk spent; a deleted file stays until the
backup rotates out), copies where a link can't be made. `--db-only` lists them instead.
**Never in a backup** (`ServerBackup.Excluded`): `InteractionSalt` (owner rule), `Job` and `Delivery` (work in flight),
`EmailRecovery`, `openiddict.tokens` and `.authorizations` (sessions), `MaintenanceLock`, and `AppConfiguration`
(its SMTP password; it is made again from appsettings at boot). One backup or restore runs at a time
(`MaintenanceLock`, a heartbeat document; a holder silent for 2 minutes is taken over), and the media janitor purges
nothing meanwhile. `BackupScheduler` backs up at `Backups:NightlyAt` (03:30 UTC), or at once when it missed the night;
rotation keeps 7 daily and 4 weekly nightly backups, 3 pre-deploy, 3 pre-restore, and manual and uploaded ones until
deleted. CLI: `PrivaPub admin backup [--kind manual|pre-deploy] [--db-only]`, `admin backups`, `admin backup verify
<id>`; these run before migrations, so the deploy's backup is of the database as the live build left it.
- **Restores** (`ServerRestore`, `ProtectiveMerge`; owner decision 2026-10-07). Asking (`PrivaPub admin restore <id>`, or
the administrator's page) checks the backup (same host, a format and newest migration this build reads, every hash)
and writes `restore.json` in the backups' root; the running service sees it within seconds (`RestoreWatcher`) and
stops, and systemd starts it again. `MaintenanceGate` (in `Program`, right after the build, before migrations, indexes
and hosted services) carries it out:
1. a pre-restore backup P, taken once (a retry reuses it, never a half-restored database);
2. every collection the backup holds dropped and imported raw with its indexes, every other one dropped, except what a
backup never holds, which stays as it is;
3. media files the live directory lacks linked back from the backup (or moved from the trash); a restore deletes no file;
4. **the protective merge from P: a restore never undoes a protective act.** Followers and follows are P's; blocks,
mutes, domain blocks, being blocked, reserved names, `DeletedObject`, reports, filters and OAuth applications are
the union, P's row winning; roots, personas, groups, posts and remote accounts deleted since are deleted again, with
what their deletion takes away; P's moderation of a persona and a root's password, e-mail, ban and policies stay;
roots, personas and groups made since become tombstones (deleted, names kept), local posts made since answer 410,
media made or trashed since go to the trash;
5. every session ends (a new `SessionStamp` for each root, persona tokens deleted), circuits close, and a
`RestoreRecord` (never in a backup) tells what happened (`admin restore --status`).
Each attempt redoes everything. Refused before anything changed, it is abandoned and recorded, and the server boots as
it was; failed midway, the process exits 1 and the next start tries again; after 3 failures it exits 75, which the
unit's `RestartPreventExitStatus` leaves down for someone to look. While `restore.json` exists, commands exit 75 (but
`admin restore --status`) and the deploy refuses to run. For `RestoreRecord.FollowersGrace` (14 days) after a restore,
FEP-8fcf rests: no `Collection-Synchronization` header goes out, and a follow only the remote remembers is adopted,
not undone.
- **The administrator's page** (`BackupController`, `/clientapi/admin/backups`; owner decision 2026-10-07, which approved
these endpoints in production): the list (with what runs, the restore waiting, the last restore's report, logins
deleted since each backup), back up now (202; the page polls), delete, and:
- **download** for the password: a ticket good for ten minutes in the link's path (`/download/{ticket}`, anonymous, so a
browser saves it to disk), the backup as one tar (`BackupTar`: its exact length known first, written from the
files, `Range` honoured);
- **upload** in pieces of at most 32 MB (`TransferStore`): each `PUT ?offset=` must start where the upload stands, else
409 with what was received, so a broken upload resumes; `finish` reads it into a backup (`kind` uploaded), refusing
anything but plain files under one backup's folder, a path leaving it, or more than the disk holds;
- **restore** for the password and the server's host typed out.
Every call checks the administrator against the database, not only the token. nginx streams downloads for an hour and
takes upload pieces unbuffered (`deploy/nginx`, applied by `setup.sh`).
- **A persona's archive** (`Domain/Portability/`, `PersonaArchiveController` at `/clientapi/persona/{avatarId}/archive`;
owner decision 2026-10-07), for the persona's own root only, never a banned one. State per persona in `PersonaArchive`
(never in a backup), files in `<backups>/.personas/<avatarId>/`, a week.
- **Export** (`ExportArchiveJob`): Mastodon's account-archive layout, so other servers' importers read it: `actor.json`
(its public key only), `outbox.json` (its own posts as Creates, addressed as delivered, but circles' and located
posts; its boosts as Announces), `media_attachments/files/` (what the attachments' urls name), avatar and header,
`likes.json`, `bookmarks.json` and Mastodon's CSV files; then `privapub/`: filters, followed hashtags, the
notification policy, pins, scheduled and located posts, `archive.json`. Nothing of its root, its siblings, its keys,
anyone's token. Downloaded through a ten-minute ticket in the link's path.
- **Import** (`ImportArchiveJob`, the parts the root picks, with progress, stoppable): uploaded in pieces like a backup,
read through `SafeArchive`, which refuses links, paths leaving it, names twice, too many or too large files, a file
compressed over 100:1, and streams the outbox an item at a time (none over 1 MB). Only the archive actor's own
Creates become posts; boosts, direct and circle posts are counted. **Imported posts are delivered to no one**, put in
no home and notify nobody (an exception next to located posts), yet show on the profile, the outbox, hashtags and
search. Back home (same actor) a post keeps its id and address, and one already here, even deleted, or tombstoned,
stays as it is; from another actor it gets an id of its date, its address here and `ImportedFromURI` (unique per
persona), so importing twice changes nothing. Replies and self-quotes inside the archive point at their copies; polls
come closed without votes; mentions stay links; HTML is sanitized; media go through `MediaService` and the quota.
The other parts go through the existing services: follows asked again (never on a blocked server), blocks, mutes,
blocked servers, lists (members only if followed), bookmarks (by signed fetch), filters, followed hashtags, the
notification policy, pins (no `Add` delivered), the profile (announced as Settings would). Located and scheduled
posts (PrivaPub's archives) and likes (each tells its author) only when asked. Followers are never imported.
## Code style
@@ -449,6 +593,8 @@ group www-data and reaches the private mongod; `sudo -u www-data` works too.
`PrivaPub.Tests` (xUnit v3). Unit tests need nothing; tests marked `Category=Integration` need a mongod and skip
without `PRIVAPUB_TEST_MONGOD=1`. CI (`build.yml` and `deploy.yml`) runs all of them through
`tools/ci/with-test-mongod.sh`, which starts a throwaway mongod on a random localhost port and deletes it afterwards.
Like production's, it is a one-member replica set (`rs0`, reached with `directConnection=true`), so backups' snapshot
reads are what the tests exercise (`TopologyTests`).
The box's own mongods are production, so `MongoFixture` refuses port 27022, a data directory under `/var/lib/privapub`,
and in CI anything but the wrapper's mongod. With `PRIVAPUB_TEST_REQUIRE_MONGOD=1`, which the wrapper sets, a missing
mongod fails the run instead of silently skipping half the tests.
@@ -779,7 +925,9 @@ the owner's GoToSocial account.**
## Deploy
- **CI/CD:** push to `master` runs `build.yml` (build + tests) on the instance-wide `build` runner. A `v*` tag runs
`deploy.yml`: tests, self-contained linux-x64 publish, snapshot and `mongodump` to `/var/backups/privapub.thepra.dev`,
`deploy.yml`: tests, self-contained linux-x64 publish, a snapshot of the site to `/var/backups/privapub.thepra.dev`,
the server's own pre-deploy backup (`PrivaPub admin backup --kind pre-deploy --db-only`, run from the new build before
its migrations, verified, and checked to hold no statistics salt; no deploy while `restore.json` waits),
stop → rsync → start, a `127.0.0.1:6970/build.json` health loop with rollback, then public checks (actor, NodeInfo,
Swagger 404, inbox junk 400, unsigned 401) and `tools/smoke/mastodon-api.sh` (app registration, client credentials,
discovery, instance, public timeline). Then it checks that what production should be running is running:
+29 -8
View File
@@ -136,6 +136,9 @@ The names are the project's own and are stable; resolve actors through WebFinger
- The reporter is `/peasants/privapub_reports` (type `Service`, "Reports from <host>"), one for the whole server with
its own key. It sends the reports Lemmy, PieFed and Mbin take in Lemmy's shape (see **Reports** below) and does
nothing else: nobody follows or mentions it, the Mastodon API has no account for it, and it has no page.
- The reader of feeds is `/peasants/privapub_feeds` (type `Service`, "Feeds read on <host>"), one for the whole server
with its own key. When a persona here follows a feed, the reader follows each community in it, and stops following one
no feed read here holds any more; no community and no feed learns which persona reads it. It does nothing else.
## Groups
@@ -167,7 +170,14 @@ A group is either a **community** or a **circle**.
and from a community elsewhere once the post's origin answers it gone. A lock sent as it is (`Lock`, Mbin) is taken
from the post's own server only. A lock (`Announce{Lock}`, or `commentsEnabled: false` on the post) refuses replies,
ours included, until `Undo{Lock}`. A ban of a persona (`Announce{Block}` with the community as `target`) shows as
`blocked_by` on the community and refuses the persona's posts and replies there until the `Undo`.
`blocked_by` on the community and refuses the persona's posts and replies there until the `Undo`. A post moved to
another community (PieFed's `Announce{Move{object: post, origin, target}}`, relayed by the community it leaves) goes
there with its thread; a move into a community this server hosts is not taken.
- **Feeds** (Lemmy 1.0's multi-communities and PieFed's feeds, `type: Feed`) are read as accounts whose following is
their communities: the feed's `following` collection, read once a day with its counts, and each community in it read in
turn. A persona following a feed is followed through the server (owner decision 2026-10-07): the reader of feeds
follows its communities, the feed itself is sent nothing, and the persona reads the feed's threads in a timeline of
its own (`GET /api/v1/timelines/feed/:id`; the feeds it reads are `GET /api/v1/feeds`), not in its home.
## Activities
@@ -185,6 +195,8 @@ Received:
| `Like` | counted and notified, on posts the liker could see |
| `Dislike` | counted as a downvote (Lemmy, PieFed, Mbin, Friendica); `Undo` takes it back |
| `EmojiReact`, `Like` with an emoji `content` | an emoji reaction (FEP-c0e0; Pleroma, Akkoma, Iceshrimp.NET, Misskey, Sharkey), Unicode or a custom emoji from its `tag`; a `Like` whose content is ❤ stays a favourite; `Undo` takes it back |
| `Warn` (Lemmy 1.0) | a community moderator's warning about a persona's post there becomes the persona's `moderation_warning` notification, believed from the community's own server or an account its moderators collection lists |
| `Resolve{Flag}` (Lemmy 1.0) | a report we sent is marked resolved by the server holding what was reported (kept beside our own moderators' resolution) |
| `Join` | answered with `Ignore`: PrivaPub hosts no events of its own yet (FEP-8a8e) |
| `Announce` | counted and notified for local posts; shown to followers of the announcer, with the original refetched from its origin |
| `Delete` | deletes the object, or the actor and its follows; a deleted object id is remembered for 90 days, so a late `Create` cannot bring it back, though a new post its server publishes under that id after the deletion (Gancio numbers events from the last one kept) is kept |
@@ -194,8 +206,14 @@ Received:
| `Block` of a persona | the follows between them end, the blocker's posts and notifications are hidden from the persona, nothing of the persona's is addressed to the blocker, and the relationship says `blocked_by`; `Undo{Block}` lifts it |
Sent: `Follow`, `Undo{Follow}`, `Create{Note}`, `Create{Question}` and poll votes, `EmojiReact` and its `Undo`, `Update{Note}`, `Update{Person}`, `Delete{Tombstone}`, `Accept{Follow}`,
`Reject{Follow}`, `Like`, `Announce` and their `Undo`, `Flag`, `Join` and `Leave` of a remote event, and the replies to a
persona's posts passed on to its followers. A deleted post answers 410 with a `Tombstone`.
`Reject{Follow}`, `Like`, `Dislike`, `Announce` and their `Undo`, `Flag`, `Join` and `Leave` of a remote event, and the
replies to a persona's posts passed on to its followers. A deleted post answers 410 with a `Tombstone`.
- **Votes.** A favourite is a `Like`, a downvote a `Dislike`; a persona has one vote a post, and a changed vote is sent
as the new vote alone, as Lemmy sends one (an `Undo` of the old one beside it could arrive after it and take the new
one away). `Undo` is sent only when the vote is taken back. A vote on a post in a remote community goes to the
community, which counts it and passes it on, as Lemmy sends one, and to the post's author only when it lives on
another server: one copy a server, since PieFed drops a second copy of an activity it has just seen.
- **Attachments** are `Document`s with `mediaType`, `name` (alt text), `blurhash`, `focalPoint`, `width` and `height`.
Uploaded files have all metadata removed.
@@ -392,12 +410,15 @@ Posts with a location (shown to nearby users of this server) never leave the ser
- **Fetching.** All fetches are signed by the instance actor. They go only to public addresses, follow at most three
redirects and read at most 1 MB.
- **Threads.** A reply's missing parents are fetched, up to 10 levels. When someone here opens a public remote thread,
its replies are read from their servers, at most once an hour: the thread's `context` collection (FEP-7888) if it has
one, else its `replies` (PeerTube's `comments`) and theirs, two levels down; 5 pages and 100 posts at most. Only
public and unlisted replies are kept, each fetched from its own origin.
its replies are read from their servers, at most once an hour: the thread's conversation container (FEP-171b's
`contextHistory`: the posts its owner's `Add`s brought in) or its `context` collection (FEP-7888) if it has one, else
its `replies` (PeerTube's `comments`) and theirs, two levels down; 5 pages and 100 posts at most. Only public and
unlisted replies are kept, each fetched from its own origin. A thread collection read whole, which counts its posts or
holds them all, is asked again with `If-None-Match` and its last ETag; a 304 ends the read.
- **Reading our documents (SecureMode).** privapub.thepra.dev answers ActivityPub GETs only when they are signed, like
Mastodon's authorized fetch; the server's own actors (the instance actor `/peasants/privapub` and the reporter
`/peasants/privapub_reports`) are the exception, since their keys are needed first.
Mastodon's authorized fetch; the server's own actors (the instance actor `/peasants/privapub`, the reporter
`/peasants/privapub_reports` and the reader of feeds `/peasants/privapub_feeds`) are the exception, since their keys
are needed first.
A browser asking for HTML is redirected to the public page instead.
- **Posts that are not public** (followers-only, direct, circle) are served to a signed request from someone they were
for, or from the instance actor of a server where someone they were for lives, and to nobody else (404). Once deleted
+103
View File
@@ -0,0 +1,103 @@
using PrivaPub.ClientModels.Resources;
using System.ComponentModel.DataAnnotations;
namespace PrivaPub.ClientModels.Admin
{
// the server's backups as the administrator's page shows them, with what is running and the last restore
public class ViewBackups
{
public List<ViewBackup> Backups { get; set; } = [];
public string Running { get; set; }//"backup" or "restore" while one runs
public ViewRestoreWaiting Waiting { get; set; }
public ViewRestore LastRestore { get; set; }
public string Host { get; set; }//what a restore asks to be typed
public long FreeBytes { get; set; }
}
public class ViewBackup
{
public string Id { get; set; }
public string Kind { get; set; }//nightly, manual, pre-deploy, pre-restore, uploaded
public DateTime CreatedAt { get; set; }
public long Bytes { get; set; }//the database's files
public long MediaBytes { get; set; }
public int MediaFiles { get; set; }
public int MediaMissing { get; set; }
public bool DbOnly { get; set; }
public bool Consistent { get; set; }
public int Collections { get; set; }
public long Documents { get; set; }
public string AppCommit { get; set; }
public int MigrationNumber { get; set; }
public int RootsDeletedSince { get; set; }//logins deleted after it, which a restore deletes again
public string NotRestorable { get; set; }//why not, from its manifest; hashes are checked when asked
}
public class ViewRestoreWaiting
{
public string Backup { get; set; }
public string State { get; set; }
public int Attempts { get; set; }
public string RequestedBy { get; set; }
public DateTime RequestedAt { get; set; }
public string Error { get; set; }
}
public class ViewRestore
{
public string Backup { get; set; }
public DateTime BackupCreatedAt { get; set; }
public string PreRestore { get; set; }
public string RequestedBy { get; set; }
public DateTime RestoredAt { get; set; }
public bool Abandoned { get; set; }
public string Error { get; set; }
public long Documents { get; set; }
public int Collections { get; set; }
public int MediaRestored { get; set; }
public int MediaMissing { get; set; }
public int RootsDeleted { get; set; }
public int PersonasDeleted { get; set; }
public int GroupsDeleted { get; set; }
public int PostsDeleted { get; set; }
public List<string> RootsTombstoned { get; set; } = [];
public List<string> PersonasTombstoned { get; set; } = [];
public List<string> GroupsTombstoned { get; set; } = [];
public int PostsGone { get; set; }
public int MediaTrashed { get; set; }
public int ProtectiveKept { get; set; }
public int SessionsEnded { get; set; }
}
// a backup downloaded or restored: the administrator's password again, and for a restore the server's host typed out
public class BackupPasswordForm
{
[Required(ErrorMessageResourceName = "Required", ErrorMessageResourceType = typeof(ErrorsResource)),
StringLength(200, ErrorMessageResourceName = "MaxLengthString", ErrorMessageResourceType = typeof(ErrorsResource))]
public string Password { get; set; }
}
public class RestoreBackupForm : BackupPasswordForm
{
[Required(ErrorMessageResourceName = "Required", ErrorMessageResourceType = typeof(ErrorsResource)),
StringLength(253, ErrorMessageResourceName = "MaxLengthString", ErrorMessageResourceType = typeof(ErrorsResource))]
public string Host { get; set; }
}
// a download link good for a while, for one backup
public class ViewBackupTicket
{
public string Url { get; set; }
public DateTime ExpiresAt { get; set; }
public long Bytes { get; set; }
}
// a backup being uploaded in pieces: how much has arrived
public class ViewBackupUpload
{
public string Id { get; set; }
public long Received { get; set; }
public long ChunkBytes { get; set; }//the largest piece accepted
}
}
@@ -16,5 +16,7 @@ namespace PrivaPub.ClientModels.Admin
public bool IsResolved { get; set; }
public DateTime CreatedAt { get; set; }
public DateTime? ResolvedAt { get; set; }
public DateTime? RemoteResolvedAt { get; set; }//the server holding what was reported said it dealt with it
public string RemoteResolvedBy { get; set; }
}
}
@@ -12,6 +12,16 @@ namespace PrivaPub.ClientModels.Admin
public ViewHostCounts Hosts { get; set; } = new();
public int Accounts { get; set; }
public ViewLedgerCounts Ledger { get; set; } = new();
public ViewMediaTotals Media { get; set; } = new();
}
// what media take on the server: the uploads and pictures kept, the remote media cache, the trash waiting its grace
public class ViewMediaTotals
{
public long MediaBytes { get; set; }
public long MediaFiles { get; set; }
public long ProxyCacheBytes { get; set; }
public long TrashBytes { get; set; }
}
public class ViewLatency
@@ -0,0 +1,48 @@
namespace PrivaPub.ClientModels.User
{
// A persona's archive (owner decision 2026-10-07): made for it to take away, in Mastodon's account-archive layout, and
// restored into it. States: none, queued, running, ready (export) or done (import), failed, stopped.
public class ViewPersonaArchive
{
public string ExportState { get; set; }
public DateTime? ExportAskedAt { get; set; }
public DateTime? ExportReadyAt { get; set; }
public DateTime? ExportExpiresAt { get; set; }
public long ExportBytes { get; set; }
public string ExportError { get; set; }
public string ImportState { get; set; }
public DateTime? ImportAskedAt { get; set; }
public DateTime? ImportEndedAt { get; set; }
public List<string> ImportParts { get; set; } = [];
public long ImportDone { get; set; }
public long ImportTotal { get; set; }
public Dictionary<string, int> ImportCounts { get; set; } = [];//what was imported and what skipped, by why
public string ImportError { get; set; }
public string ImportFrom { get; set; }//the archive's actor
public bool ImportSameActor { get; set; }
}
// the parts of an archive to import: posts (with their media), profile, follows, blocks, mutes, domainblocks, lists,
// bookmarks, filters, tags, notificationpolicy, pins; and only when asked, located, scheduled, likes (which tell their
// authors)
public class ImportPersonaArchiveForm
{
public string UploadId { get; set; }
public List<string> Parts { get; set; } = [];
}
public class ViewArchiveTicket
{
public string Url { get; set; }
public DateTime ExpiresAt { get; set; }
public long Bytes { get; set; }
}
public class ViewArchiveUpload
{
public string Id { get; set; }
public long Received { get; set; }
public long ChunkBytes { get; set; }
}
}
+11
View File
@@ -0,0 +1,11 @@
namespace PrivaPub.ClientModels.User
{
// /clientapi/user/storage: what the login's media take (every persona's uploads and pictures), and its quota when the
// server sets one
public class ViewStorage
{
public long MediaBytes { get; set; }
public long MediaFiles { get; set; }
public long? QuotaBytes { get; set; }
}
}
@@ -71,6 +71,7 @@ namespace PrivaPub.Tests.Domain
public Task<LocalActor> FindByAddress(string address, CancellationToken token) => throw new NotSupportedException();
public Task<LocalActor> GetInstanceActor(CancellationToken token) => throw new NotSupportedException();
public Task<LocalActor> GetReporterActor(CancellationToken token) => throw new NotSupportedException();
public Task<LocalActor> GetReaderActor(CancellationToken token) => throw new NotSupportedException();
public Task<bool> IsUserNameTaken(string userName, CancellationToken token) => throw new NotSupportedException();
public Task<bool> TryReserveUserName(string userName, LocalActorKind kind, string ownerId, CancellationToken token) => throw new NotSupportedException();
public LocalActor FromAvatar(Avatar avatar) => throw new NotSupportedException();
+60 -2
View File
@@ -1,3 +1,4 @@
using PrivaPub.Models.Federation;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Caching.Memory;
@@ -46,7 +47,7 @@ namespace PrivaPub.Tests.Domain
var (_, alice) = await _harness.Persona("alice");
var (_, mallory) = await _harness.Persona("mallory");
var upload = await _harness.Media.Upload(alice, Upload(Png(300, 200), "image/png"), "a blue square", "0.25,-0.5", token);
var upload = await _harness.Media.Upload(alice, Upload(Png(300, 200), "image/png"), "a blue square", "0.25,-0.5", false, token);
Assert.True(upload.Ok);
Assert.True(File.Exists(Path.Combine(_harness.Media.Root, upload.Attachment.FilePath)));
@@ -73,7 +74,7 @@ namespace PrivaPub.Tests.Domain
{
var (_, alice) = await _harness.Persona("alice");
var outcome = await _harness.Media.Upload(alice, Upload(System.Text.Encoding.UTF8.GetBytes("<svg/>"), "image/svg+xml"), default, default, TestContext.Current.CancellationToken);
var outcome = await _harness.Media.Upload(alice, Upload(System.Text.Encoding.UTF8.GetBytes("<svg/>"), "image/svg+xml"), default, default, false, TestContext.Current.CancellationToken);
Assert.False(outcome.Ok);
Assert.Equal(422, outcome.Status);
@@ -98,5 +99,62 @@ namespace PrivaPub.Tests.Domain
Assert.StartsWith(_harness.Media.ProxyRoot, file);
Assert.Null(tampered);
}
// a login over its quota uploads nothing more, whichever persona tries
[Fact]
public async Task A_login_over_its_quota_uploads_nothing_more()
{
var token = TestContext.Current.CancellationToken;
var (root, alice) = await _harness.Persona("alice");
var (_, sibling) = await _harness.Persona("sibling", root);
var quota = new MediaService(new StaticOptions<MediaOptions>(new MediaOptions { Root = _harness.Media.Root, QuotaBytesPerRoot = 3000 }),
_harness.Local, default, Microsoft.Extensions.Logging.Abstractions.NullLogger<MediaService>.Instance);
// noise: its PNG is about as big as its pixels, so two of them are over the quota and one is not
static byte[] Noise()
{
var pixels = new byte[30 * 20 * 3];
Random.Shared.NextBytes(pixels);
using var image = NetVips.Image.NewFromMemory(pixels, 30, 20, 3, NetVips.Enums.BandFormat.Uchar);
return image.WriteToBuffer(".png");
}
Assert.True((await quota.Upload(alice, Upload(Noise(), "image/png"), default, default, false, token)).Ok);
var full = await quota.Upload(sibling, Upload(Noise(), "image/png"), default, default, false, token);
Assert.False(full.Ok);
Assert.Contains("storage is full", full.Error);
}
// nothing of a suspended server, or of one whose media are rejected, is proxied; blocking one purges what was cached
[Fact]
public async Task A_blocked_servers_media_are_not_proxied_and_their_cache_goes()
{
var token = TestContext.Current.CancellationToken;
var path = $"/files/{Guid.NewGuid():N}.png";
_harness.Peer.ServeFile(path, Png(10, 10), "image/png");
var remote = _harness.Peer.A + path;
var host = new Uri(remote).Host;
var blocks = new Blocks();
var proxy = new MediaProxy(_harness.Local, Peer.Http(), _harness.Media, new StaticOptions<MediaOptions>(new MediaOptions()), blocks);
Assert.Equal(ProxyOutcome.Cached, (await proxy.Download(remote, token)).Outcome);
blocks.Blocked[host] = new DomainBlock { Domain = host, Severity = DomainBlockSeverity.Silence, RejectMedia = true };
Assert.True(proxy.Refuses(remote));
Assert.True(proxy.Purge(host) >= 1);
Assert.Equal(default, proxy.Cached(remote));
blocks.Blocked[host] = new DomainBlock { Domain = host, Severity = DomainBlockSeverity.Silence };
Assert.False(proxy.Refuses(remote));
blocks.Blocked[host] = new DomainBlock { Domain = host, Severity = DomainBlockSeverity.Suspend };
Assert.True(proxy.Refuses(remote));
}
sealed class Blocks : PrivaPub.Federation.Moderation.IDomainBlocks
{
public Dictionary<string, DomainBlock> Blocked { get; } = new();
public DomainBlock Find(string host) => Blocked.GetValueOrDefault(host);
public bool IsSuspended(string host) => Find(host)?.Severity == DomainBlockSeverity.Suspend;
public Task Reload(CancellationToken token) => Task.CompletedTask;
}
}
}
+34 -3
View File
@@ -19,6 +19,37 @@ namespace PrivaPub.Tests.Domain
return tagged.WriteToBuffer(".jpg");
}
// SVG, PDF and the other formats libvips could read never load from an upload, whatever it claims to be
[Fact]
public void Only_the_upload_formats_are_ever_loaded()
{
var svg = "<svg xmlns='http://www.w3.org/2000/svg' width='20000' height='20000'><rect width='10' height='10'/></svg>"u8.ToArray();
Assert.Throws<VipsException>(() => MediaService.Inspect(svg));
Assert.Throws<VipsException>(() => MediaService.Inspect("%PDF-1.4\n1 0 obj<<>>endobj\ntrailer<<>>\n%%EOF"u8.ToArray()));
using var colour = (Image.Black(8, 8, bands: 3) + 100).Cast(Enums.BandFormat.Uchar);
foreach (var format in new[] { ".jpg", ".png", ".gif", ".webp" })
Assert.Equal(8, MediaService.Inspect(colour.WriteToBuffer(format)).Width);
Assert.Throws<VipsException>(() => MediaService.Inspect(colour.WriteToBuffer(".tif")));
}
// the header tells how big an image would decode before anything is decoded
[Fact]
public void The_header_says_how_big_an_image_would_decode()
{
using var huge = Image.Black(8000, 6000);
var header = MediaService.Inspect(huge.WriteToBuffer(".png"));
Assert.Equal(48_000_000, header.Pixels);
Assert.False(header.Animated);
using var frame = (Image.Black(40, 30, bands: 3) + 60).Cast(Enums.BandFormat.Uchar);
using var frames = Image.Arrayjoin(new[] { frame, frame + 80, frame + 160 }, across: 1).Cast(Enums.BandFormat.Uchar);
using var paged = frames.Mutate(m => m.Set(GValue.GIntType, "page-height", 30));
var gif = MediaService.Inspect(paged.WriteToBuffer(".gif"));
Assert.True(gif.Animated);
Assert.Equal(3, gif.Pages);
Assert.Equal(40 * 30 * 3, gif.Pixels);
}
[Fact]
public void Uploaded_images_lose_every_kind_of_metadata()
{
@@ -26,7 +57,7 @@ namespace PrivaPub.Tests.Domain
using (var original = Image.NewFromBuffer(input))
Assert.Contains("exif-data", original.GetFields());
var processed = MediaService.ProcessImage(input, 4096, 640, animated: false);
var processed = MediaService.ProcessImage(input, 4096, 640);
using var output = Image.NewFromBuffer(processed.Bytes);
var fields = output.GetFields();
@@ -44,7 +75,7 @@ namespace PrivaPub.Tests.Domain
[Fact]
public void Large_images_are_capped()
{
var processed = MediaService.ProcessImage(JpegWithMetadata(5000, 2500), 4096, 640, animated: false);
var processed = MediaService.ProcessImage(JpegWithMetadata(5000, 2500), 4096, 640);
Assert.Equal((4096, 2048), (processed.Width, processed.Height));
}
@@ -52,7 +83,7 @@ namespace PrivaPub.Tests.Domain
[Fact]
public void A_blurhash_is_well_formed()
{
var processed = MediaService.ProcessImage(JpegWithMetadata(64, 64), 4096, 640, animated: false);
var processed = MediaService.ProcessImage(JpegWithMetadata(64, 64), 4096, 640);
Assert.Equal(28, processed.Blurhash.Length);
Assert.Equal('L', processed.Blurhash[0]);
@@ -0,0 +1,115 @@
using MongoDB.Entities;
using PrivaPub.ClientModels.Post;
using PrivaPub.Models.Post;
using PrivaPub.Models.Social;
using PrivaPub.Tests.Support;
using System.Text.Json.Nodes;
namespace PrivaPub.Tests.Federation
{
[Trait("Category", "Integration")]
public sealed class CommunityModerationTests : IAsyncLifetime
{
Harness _harness;
public async ValueTask InitializeAsync()
{
Assert.SkipUnless(MongoFixture.Enabled, MongoFixture.Skip);
_harness = await Harness.Start();
}
public async ValueTask DisposeAsync()
{
if (_harness != default)
await _harness.DisposeAsync();
}
static string Origin(string id) => new Uri(id).GetLeftPart(UriPartial.Authority);
// a Lemmy community on the peer's first origin whose moderators collection lists the given accounts
RemoteActor Community(params RemoteActor[] moderators)
{
var community = new RemoteActor(_harness.Peer, "comm", type: "Group");
var document = community.Document();
document["attributedTo"] = community.Id + "/moderators";
_harness.Peer.Serve($"/users/{community.Name}", document.ToJsonString());
_harness.Peer.Serve($"/users/{community.Name}/moderators", new JsonObject
{
["id"] = community.Id + "/moderators", ["type"] = "OrderedCollection",
["orderedItems"] = new JsonArray(moderators.Select(m => (JsonNode)m.Id).ToArray())
}.ToJsonString());
return community;
}
async Task<(Post Post, PrivaPub.Federation.Actors.LocalActor Alice)> PostIn(RemoteActor community)
{
var token = TestContext.Current.CancellationToken;
var (root, alice) = await _harness.Persona("alice");
await _harness.Posts.InsertPost(root, new InsertPostForm { AvatarId = alice.Id, Text = "in the community" }, token);
var post = await DB.Default.Find<Post>().Match(p => p.GroupUserId == alice.Id).ExecuteFirstAsync(token);
await DB.Default.Update<Post>().MatchID(post.ID).Modify(p => p.AudienceURI, community.Id).ExecuteAsync(token);
await _harness.Remote.GetActor(community.Id, refresh: true, token);
return (post, alice);
}
JsonObject Warn(RemoteActor moderator, RemoteActor community, string persona, string post) => new()
{
["id"] = $"{Origin(moderator.Id)}/activities/warn/{Guid.NewGuid():N}", ["type"] = "Warn", ["actor"] = moderator.Id,
["to"] = new JsonArray(persona), ["object"] = post, ["summary"] = "please keep it civil", ["audience"] = community.Id
};
[Fact]
public async Task A_warning_from_the_communitys_moderators_reaches_the_persona_and_a_strangers_does_not()
{
var token = TestContext.Current.CancellationToken;
var listed = new RemoteActor(_harness.Peer, "listedmod", origin: _harness.Peer.B);
var stranger = new RemoteActor(_harness.Peer, "stranger", origin: _harness.Peer.B);
var community = Community(listed);
var admin = new RemoteActor(_harness.Peer, "admin");
var (post, alice) = await PostIn(community);
await _harness.Deliver(stranger, "/human-centipede", Warn(stranger, community, alice.Uri, post.ObjectURI));
Assert.False(await DB.Default.Find<Notification>().Match(n => n.AvatarId == alice.Id && n.Type == NotificationType.ModerationWarning).ExecuteAnyAsync(token));
await _harness.Deliver(listed, "/human-centipede", Warn(listed, community, alice.Uri, post.ObjectURI));
var warning = await DB.Default.Find<Notification>().Match(n => n.AvatarId == alice.Id && n.Type == NotificationType.ModerationWarning).ExecuteFirstAsync(token);
Assert.Equal("please keep it civil", warning.Text);
Assert.Equal(post.ID, warning.PostId);
var adminWarning = Warn(admin, community, alice.Uri, post.ObjectURI);
adminWarning["summary"] = "from the server's admin";
await DB.Default.DeleteAsync<Notification>(n => n.AvatarId == alice.Id);
await _harness.Deliver(admin, "/human-centipede", adminWarning);
Assert.Equal("from the server's admin", (await DB.Default.Find<Notification>().Match(n => n.AvatarId == alice.Id && n.Type == NotificationType.ModerationWarning).ExecuteFirstAsync(token)).Text);
}
[Fact]
public async Task A_reports_resolution_is_kept_only_from_the_server_holding_what_was_reported()
{
var token = TestContext.Current.CancellationToken;
var community = Community();
var (post, _) = await PostIn(community);
var report = new Report { PostIds = new() { post.ID }, Comment = "spam" };
await DB.Default.SaveAsync(report, token);
var flag = $"{_harness.Local.BaseAddress}/peasants/privapub_reports/grunts/flag-{report.ID}-1";
JsonObject Resolve(RemoteActor by) => new()
{
["id"] = $"{Origin(by.Id)}/activities/resolve/{Guid.NewGuid():N}", ["type"] = "Resolve", ["actor"] = by.Id, ["to"] = new JsonArray(community.Id),
["object"] = new JsonObject { ["id"] = flag, ["type"] = "Flag", ["actor"] = $"{_harness.Local.BaseAddress}/peasants/privapub_reports", ["object"] = post.ObjectURI }
};
var elsewhere = new RemoteActor(_harness.Peer, "elsewhere", origin: _harness.Peer.B);
var moderator = new RemoteActor(_harness.Peer, "moderator");
await _harness.Deliver(elsewhere, "/human-centipede", Resolve(elsewhere));
Assert.Null((await DB.Default.Find<Report>().OneAsync(report.ID, token)).RemoteResolvedAt);
await _harness.Deliver(moderator, "/human-centipede", Resolve(moderator));
var resolved = await DB.Default.Find<Report>().OneAsync(report.ID, token);
Assert.NotNull(resolved.RemoteResolvedAt);
Assert.Equal(moderator.Id, resolved.RemoteResolvedBy);
Assert.False(resolved.IsResolved);
}
}
}
+68
View File
@@ -0,0 +1,68 @@
using PrivaPub.Federation.Actors;
using PrivaPub.Federation.Objects;
using System.Text.Json;
using System.Text.Json.Nodes;
namespace PrivaPub.Tests.Federation
{
public class FlairTests
{
[Fact]
public void A_lemmy_posts_flair_is_described_by_its_community()
{
using var group = JsonDocument.Parse("""
{
"id": "https://lemmy.example/c/tenforward", "type": "Group", "preferredUsername": "tenforward", "inbox": "https://lemmy.example/c/tenforward/inbox",
"attributedTo": "https://lemmy.example/c/tenforward/moderators",
"tag": [{ "type": "CommunityPostTag", "id": "https://lemmy.example/c/tenforward/tag/news", "preferredUsername": "news",
"name": "News", "content": "What happened", "color": "color03" }]
}
""");
var community = ActorDocument.Parse(group.RootElement);
var page = NoteParser.Parse(JsonNode.Parse("""
{
"id": "https://lemmy.example/post/1", "type": "Page", "name": "Big news", "attributedTo": "https://lemmy.example/u/bo",
"tag": [{ "type": "CommunityPostTag", "id": "https://lemmy.example/c/tenforward/tag/news", "preferredUsername": "news" },
{ "type": "Mention", "href": "https://lemmy.example/c/tenforward", "name": "@tenforward@lemmy.example" }]
}
"""));
Assert.Equal("https://lemmy.example/c/tenforward/moderators", community.Moderators);
var flair = Assert.Single(ObjectShapes.Described(page.Flairs, community.Flairs));
Assert.Equal("News", flair.Name);
Assert.Equal("news", flair.Slug);
Assert.Equal("color03", flair.Color);
Assert.Equal("What happened", flair.Description);
}
[Fact]
public void Reads_piefeds_flairs_in_both_dialects_and_its_older_community_list()
{
var post = NoteParser.Parse(JsonNode.Parse("""
{
"id": "https://piefed.example/post/9", "type": "Page", "name": "A question", "attributedTo": "https://piefed.example/u/pf",
"tag": [{ "type": "lemmy:CommunityTag", "id": "https://piefed.example/c/ask/tag/3", "display_name": "Solved",
"text_color": "#ffffff", "background_color": "#2e7d32", "blur_images": false },
{ "type": "Hashtag", "href": "https://piefed.example/tag/q", "name": "#q" }]
}
"""));
using var group = JsonDocument.Parse("""
{
"id": "https://piefed.example/c/ask", "type": "Group", "preferredUsername": "ask", "inbox": "https://piefed.example/c/ask/inbox",
"lemmy:tagsForPosts": [{ "type": "lemmy:CommunityTag", "id": "https://piefed.example/c/ask/tag/4", "display_name": "Spoiler",
"text_color": "#000", "background_color": "red; background: url(x)", "blur_images": true }]
}
""");
var solved = Assert.Single(post.Flairs);
Assert.Equal("Solved", solved.Name);
Assert.Equal("#2e7d32", solved.BackgroundColor);
Assert.Equal("q", Assert.Single(post.Tags));
var spoiler = Assert.Single(ActorDocument.Parse(group.RootElement).Flairs);
Assert.True(spoiler.BlurImages);
Assert.Equal("#000", spoiler.TextColor);
Assert.Null(spoiler.BackgroundColor);
}
}
}
@@ -412,6 +412,48 @@ namespace PrivaPub.Tests.Federation
Assert.Equal(0, (await DB.Default.Find<Post>().OneAsync(reply.ID, token)).FavouritesCount);
}
// PieFed moves a post to another community, and the community it leaves relays the Move: the post's thread goes with it
[Fact]
public async Task A_post_moved_to_another_community_takes_its_thread_along()
{
var token = TestContext.Current.CancellationToken;
var (root, alice) = await _harness.Persona("alice");
var community = new RemoteActor(_harness.Peer, "cats", type: "Group");
var other = new RemoteActor(_harness.Peer, "dogs", type: "Group");
var poster = new RemoteActor(_harness.Peer, "poster");
await _harness.Follows.Follow(root, new FollowForm { AvatarId = alice.Id, Target = community.Id }, token);
await DB.Default.Update<Following>().Match(f => f.AvatarId == alice.Id).Modify(f => f.State, FollowState.Accepted).ExecuteAsync(token);
async Task<Post> Announced(JsonObject note)
{
note["audience"] = community.Id;
_harness.Peer.Serve(new Uri(IdOf(note)).AbsolutePath, note.ToJsonString());
await _harness.Deliver(community, "/human-centipede", Activity(community, "Announce", Create(poster, note)));
return await DB.Default.Find<Post>().Match(p => p.ObjectURI == IdOf(note)).ExecuteSingleAsync(token);
}
var page = PublicNote(poster, "<p>misplaced</p>", community.Id);
page["type"] = "Page";
page["name"] = "misplaced";
var post = await Announced(page);
var comment = PublicNote(poster, "<p>a comment</p>", community.Id);
comment["inReplyTo"] = IdOf(page);
var answer = await Announced(comment);
var move = new JsonObject
{
["id"] = NewId(poster, "moves"), ["type"] = "Move", ["actor"] = poster.Id, ["object"] = post.ObjectURI, ["origin"] = community.Id, ["target"] = other.Id
};
// only the community it leaves speaks for it
await _harness.Deliver(community, "/human-centipede", Activity(community, "Announce", new JsonObject
{
["id"] = NewId(poster, "moves"), ["type"] = "Move", ["actor"] = poster.Id, ["object"] = post.ObjectURI, ["origin"] = other.Id, ["target"] = community.Id
}));
Assert.Equal(("dropped", "unsupported"), (Processed("Announce").Outcome, Processed("Announce").Reason));
await _harness.Deliver(community, "/human-centipede", Activity(community, "Announce", move));
Assert.Equal(("accepted", "moved"), (Processed("Announce").Outcome, Processed("Announce").Reason));
Assert.Equal(other.Id, (await DB.Default.Find<Post>().OneAsync(post.ID, token)).AudienceURI);
Assert.Equal(other.Id, (await DB.Default.Find<Post>().OneAsync(answer.ID, token)).AudienceURI);
}
// Lemmy's moderators lock a community's post and ban members, and the community relays both inside its Announce
// (G-0006): a locked post takes no reply, a banned persona posts nothing there, until the Undo
[Fact]
+106 -1
View File
@@ -216,6 +216,103 @@ namespace PrivaPub.Tests.Federation
Assert.All(_harness.Peer.Requests.Where(r => r.Path.EndsWith("/replies")), r => Assert.False(string.IsNullOrEmpty(r.Signature)));
}
[Fact]
public async Task A_conversation_container_is_read_before_the_context_and_only_its_added_posts_taken()
{
var owner = new RemoteActor(_harness.Peer, "owner");
var answerer = new RemoteActor(_harness.Peer, "answerer");
var root = PublicNote(owner, "<p>a conversation</p>");
var history = NewId(owner, "conversations");
root["contextHistory"] = history;
root["context"] = NewId(owner, "never");
Served(root);
var embedded = Served(Reply(answerer, "added whole", root));
var referenced = Served(Reply(owner, "added by reference", embedded));
var liked = Served(Reply(answerer, "only liked", root));
var target = new JsonObject { ["type"] = "OrderedCollection", ["id"] = history, ["attributedTo"] = owner.Id };
JsonObject Add(JsonNode activity) => new()
{
["id"] = NewId(owner, "activities"), ["type"] = "Add", ["actor"] = owner.Id, ["object"] = activity, ["target"] = target.DeepClone()
};
var create = Served(Create(owner, referenced));
create["object"] = IdOf(referenced);
Served(create);
var byReference = Served(Add(IdOf(create)));
Served(new JsonObject
{
["id"] = history, ["type"] = "OrderedCollection", ["attributedTo"] = owner.Id, ["collectionOf"] = "Activity",
["orderedItems"] = new JsonArray(Create(owner, root), Add(Create(answerer, embedded)), IdOf(byReference), Add(Activity(answerer, "Like", JsonValue.Create(IdOf(liked)))))
});
var held = await Held(root);
Assert.Equal(1, await RunReplies(held));
var stored = await Stored(embedded, referenced, liked);
Assert.Equal(new[] { IdOf(embedded), IdOf(referenced) }.Order(), stored.Keys.Order());
Assert.Equal(held.ID, stored[IdOf(embedded)].AnsweringToPostId);
Assert.Equal(stored[IdOf(embedded)].ID, stored[IdOf(referenced)].AnsweringToPostId);
Assert.DoesNotContain(_harness.Peer.Requests, r => r.Path == new Uri(root["context"]!.GetValue<string>()).AbsolutePath);
}
[Fact]
public async Task An_unchanged_context_is_not_read_again_while_it_counts_the_thread()
{
var token = TestContext.Current.CancellationToken;
var poster = new RemoteActor(_harness.Peer, "poster");
var answerer = new RemoteActor(_harness.Peer, "answerer");
var root = PublicNote(poster, "<p>a topic</p>");
var context = NewId(poster, "topics");
var path = new Uri(context).AbsolutePath;
root["context"] = context;
Served(root);
var first = Served(Reply(answerer, "first", root));
var later = Served(Reply(poster, "later", root));
// NodeBB's: every post inline, counted, and an ETag without quotes
JsonObject Thread(params JsonObject[] notes) => new()
{
["id"] = context, ["type"] = "OrderedCollection", ["totalItems"] = notes.Length,
["orderedItems"] = new JsonArray(notes.Select(n => (JsonNode)IdOf(n)).ToArray())
};
_harness.Peer.ServeTagged(path, Thread(root, first).ToJsonString(), "4f930186eb");
var held = await Held(root);
var handler = new RepliesJobHandler(_harness.Db, _harness.Remote, _harness.RemotePosts, _harness.Queue);
await handler.Handle(RepliesJobHandler.For(held, 1), token);
Assert.Equal("4f930186eb", (await DB.Default.Find<ThreadCollection>().Match(c => c.URI == context).ExecuteFirstAsync(token)).ETag);
// the same version: a 304, and nothing read
_harness.Peer.ServeTagged(path, Thread(root, first, later).ToJsonString(), "4f930186eb");
await handler.Handle(RepliesJobHandler.For(held, 1), token);
Assert.Equal("4f930186eb", _harness.Peer.Requests.Last(r => r.Path == path).Headers["If-None-Match"]);
Assert.Single(await Stored(first, later));
// a new one
_harness.Peer.ServeTagged(path, Thread(root, first, later).ToJsonString(), "5a0b");
await handler.Handle(RepliesJobHandler.For(held, 1), token);
Assert.Equal(2, (await Stored(first, later)).Count);
Assert.Equal("5a0b", (await DB.Default.Find<ThreadCollection>().Match(c => c.URI == context).ExecuteFirstAsync(token)).ETag);
}
[Fact]
public async Task A_paged_contexts_etag_is_not_kept_since_its_first_page_hides_what_is_added_later()
{
var token = TestContext.Current.CancellationToken;
var poster = new RemoteActor(_harness.Peer, "poster");
var root = PublicNote(poster, "<p>a long thread</p>");
var context = NewId(poster, "contexts");
root["context"] = context;
Served(root);
var answer = Served(Reply(poster, "on the second page", root));
_harness.Peer.ServeTagged(new Uri(context).AbsolutePath, new JsonObject
{
["id"] = context, ["type"] = "Collection",
["first"] = new JsonObject { ["type"] = "CollectionPage", ["items"] = new JsonArray(IdOf(root)), ["next"] = context + "/2" }
}.ToJsonString(), "W/\"first\"");
Served(new JsonObject { ["id"] = context + "/2", ["type"] = "CollectionPage", ["items"] = new JsonArray(IdOf(answer)) });
var held = await Held(root);
await new RepliesJobHandler(_harness.Db, _harness.Remote, _harness.RemotePosts, _harness.Queue).Handle(RepliesJobHandler.For(held, 1), token);
Assert.Single(await Stored(answer));
Assert.False(await DB.Default.Find<ThreadCollection>().Match(c => c.URI == context && c.ETag != null).ExecuteAnyAsync(token));
}
[Fact]
public async Task A_threads_context_collection_brings_every_reply_at_once()
{
@@ -489,9 +586,17 @@ namespace PrivaPub.Tests.Federation
await new MediaJanitor(media, options, NullLogger<MediaJanitor>.Instance).Sweep(token);
Assert.False(await DB.Default.Find<MediaAttachment>().Match(m => m.ID == stale.ID).ExecuteAnyAsync(token));
// never posted for a day: trashed, its files out of what /media/files serves at once
Assert.NotNull((await DB.Default.Find<MediaAttachment>().OneAsync(stale.ID, token)).TrashedAt);
Assert.False(File.Exists(Path.Combine(root, stale.FilePath)));
Assert.False(File.Exists(Path.Combine(root, stale.PreviewPath)));
Assert.True(File.Exists(Path.Combine(media.TrashRoot, stale.FilePath)));
// and once its grace has passed, deleted with its row
await DB.Default.Update<MediaAttachment>().MatchID(stale.ID).Modify(m => m.TrashedAt, DateTime.UtcNow - MediaJanitor.TrashGrace - TimeSpan.FromMinutes(1)).ExecuteAsync(token);
await new MediaJanitor(media, options, NullLogger<MediaJanitor>.Instance).Sweep(token);
Assert.False(await DB.Default.Find<MediaAttachment>().Match(m => m.ID == stale.ID).ExecuteAnyAsync(token));
Assert.False(File.Exists(Path.Combine(media.TrashRoot, stale.FilePath)));
Assert.False(File.Exists(Path.Combine(media.TrashRoot, stale.PreviewPath)));
Assert.True(await DB.Default.Find<MediaAttachment>().Match(m => m.ID == fresh.ID).ExecuteAnyAsync(token));
Assert.True(await DB.Default.Find<MediaAttachment>().Match(m => m.ID == attached.ID).ExecuteAnyAsync(token));
Assert.True(File.Exists(Path.Combine(root, fresh.FilePath)));
+200
View File
@@ -0,0 +1,200 @@
using MongoDB.Entities;
using PrivaPub.ClientModels;
using PrivaPub.Infrastructure.Backup;
using PrivaPub.Models.User;
using PrivaPub.Tests.Support;
using PrivaPub.Tests.Support.Host;
using System.Formats.Tar;
using System.Net;
using System.Net.Http.Headers;
using System.Net.Http.Json;
using System.Text.Json.Nodes;
namespace PrivaPub.Tests.Http
{
// The administrator's page backs up, downloads, uploads and restores (owner decision 2026-10-07): only an
// administrator the database still says is one; a download and a restore ask for the password again, a restore for
// the host typed out; an upload resumes where it stopped, and one that would write outside its backup is refused.
[Trait("Category", "Integration")]
[Xunit.Collection(nameof(Exclusive))]
public sealed class BackupEndpointTests : IAsyncLifetime
{
const string Base = "/clientapi/admin/backups";
PrivaPubHost _host;
static CancellationToken Token => TestContext.Current.CancellationToken;
public async ValueTask InitializeAsync()
{
Assert.SkipUnless(MongoFixture.Enabled, MongoFixture.Skip);
_host = await PrivaPubHost.Shared();
}
public ValueTask DisposeAsync()
{
RestoreMarker.Clear(_host?.Get<Backups>().Root ?? Path.GetTempPath());
return ValueTask.CompletedTask;
}
async Task<string> BackedUp(HttpClient admin)
{
var before = _host.Get<Backups>().List().Select(b => b.Id).ToHashSet();
Assert.Equal(HttpStatusCode.Accepted, (await admin.PostAsync(Base, null, Token)).StatusCode);
var made = default(string);
Assert.True(await Wait(async () =>
{
var list = (await admin.GetFromJsonAsync<JsonObject>(Base, Token))!;
made = list["backups"]!.AsArray().Select(b => b!["id"]!.GetValue<string>()).FirstOrDefault(id => !before.Contains(id));
return made != default && list["running"] is null;
}), "the backup was never made");
return made;
}
static async Task<bool> Wait(Func<Task<bool>> done)
{
for (var i = 0; i < 120; i++)
{
if (await done())
return true;
await Task.Delay(250, Token);
}
return false;
}
[Fact]
public async Task Only_an_administrator_the_database_still_knows()
{
var root = await _host.SignUp("plain");
using (var plain = _host.As(root.Jwt))
Assert.Equal(HttpStatusCode.Forbidden, (await plain.GetAsync(Base, Token)).StatusCode);
var admin = await _host.Admin();
using var client = _host.As(admin.Jwt);
Assert.Equal(HttpStatusCode.OK, (await client.GetAsync(Base, Token)).StatusCode);
await DB.Default.Update<RootUser>().MatchID(admin.Id).Modify(u => u.Policies, [Policies.IsUser]).ExecuteAsync(Token);
Assert.Equal(HttpStatusCode.Forbidden, (await client.GetAsync(Base, Token)).StatusCode);
Assert.Equal(HttpStatusCode.Forbidden, (await client.PostAsync(Base, null, Token)).StatusCode);
}
[Fact]
public async Task A_backup_is_made_downloaded_whole_or_in_part_and_uploaded_again()
{
var admin = await _host.Admin();
using var client = _host.As(admin.Jwt);
var id = await BackedUp(client);
var listed = (await client.GetFromJsonAsync<JsonObject>(Base, Token))!;
Assert.Equal("privapub.test", listed["host"]!.GetValue<string>());
Assert.Equal("manual", listed["backups"]!.AsArray().Single(b => b!["id"]!.GetValue<string>() == id)!["kind"]!.GetValue<string>());
Assert.Equal(HttpStatusCode.UnprocessableEntity, (await client.PostAsJsonAsync($"{Base}/{id}/ticket", new { password = "wrong" }, Token)).StatusCode);
var ticket = (await (await client.PostAsJsonAsync($"{Base}/{id}/ticket", new { password = admin.Password }, Token)).Content.ReadFromJsonAsync<JsonObject>(Token))!;
var url = ticket["url"]!.GetValue<string>();
// a plain link: no token needed, the ticket is the credential
using var anonymous = _host.Client();
var whole = await anonymous.GetAsync(url, Token);
Assert.Equal(HttpStatusCode.OK, whole.StatusCode);
var bytes = await whole.Content.ReadAsByteArrayAsync(Token);
Assert.Equal(ticket["bytes"]!.GetValue<long>(), bytes.Length);
Assert.Equal(bytes.Length, whole.Content.Headers.ContentLength);
var names = new List<string>();
await using (var reader = new TarReader(new MemoryStream(bytes)))
while (await reader.GetNextEntryAsync(cancellationToken: Token) is { } entry)
names.Add(entry.Name);
Assert.Contains($"{id}/manifest.json", names);
Assert.Contains($"{id}/db/Avatar.jsonl.gz", names);
var part = new HttpRequestMessage(HttpMethod.Get, url);
part.Headers.Range = new RangeHeaderValue(700, 2747);
var partial = await anonymous.SendAsync(part, Token);
Assert.Equal(HttpStatusCode.PartialContent, partial.StatusCode);
Assert.Equal($"bytes 700-2747/{bytes.Length}", partial.Content.Headers.ContentRange!.ToString());
Assert.Equal(bytes.AsSpan(700, 2048).ToArray(), await partial.Content.ReadAsByteArrayAsync(Token));
Assert.Equal(HttpStatusCode.NotFound, (await anonymous.GetAsync($"{Base}/download/{new string('0', 64)}", Token)).StatusCode);
// gone, then uploaded again in three pieces, one sent at the wrong place and refused with where to go on
Assert.Equal(HttpStatusCode.OK, (await client.DeleteAsync($"{Base}/{id}", Token)).StatusCode);
Assert.Equal(HttpStatusCode.NotFound, (await client.DeleteAsync($"{Base}/{id}", Token)).StatusCode);
var upload = (await (await client.PostAsync($"{Base}/uploads", null, Token)).Content.ReadFromJsonAsync<JsonObject>(Token))!["id"]!.GetValue<string>();
var third = bytes.Length / 3;
Assert.Equal(HttpStatusCode.OK, (await Piece(client, upload, 0, bytes[..third])).StatusCode);
var gap = await Piece(client, upload, third + 10, bytes[(third + 10)..]);
Assert.Equal(HttpStatusCode.Conflict, gap.StatusCode);
Assert.Equal(third, (await gap.Content.ReadFromJsonAsync<JsonObject>(Token))!["received"]!.GetValue<long>());
Assert.Equal(HttpStatusCode.OK, (await Piece(client, upload, third, bytes[third..(2 * third)])).StatusCode);
Assert.Equal(HttpStatusCode.OK, (await Piece(client, upload, 2 * third, bytes[(2 * third)..])).StatusCode);
var finished = await client.PostAsync($"{Base}/uploads/{upload}/finish", null, Token);
Assert.Equal(HttpStatusCode.OK, finished.StatusCode);
Assert.Equal("uploaded", (await finished.Content.ReadFromJsonAsync<JsonObject>(Token))!["kind"]!.GetValue<string>());
Assert.Empty(await _host.Get<Backups>().Verify(id, Token));
Assert.Equal(HttpStatusCode.UnprocessableEntity, (await client.PostAsync($"{Base}/uploads/{upload}/finish", null, Token)).StatusCode);
_host.Get<Backups>().Delete(id);
}
Task<HttpResponseMessage> Piece(HttpClient client, string upload, long offset, byte[] bytes) =>
client.PutAsync($"{Base}/uploads/{upload}?offset={offset}", new ByteArrayContent(bytes), Token);
[Theory]
[InlineData("escape")]
[InlineData("link")]
[InlineData("elsewhere")]
[InlineData("junk")]
public async Task An_upload_that_is_not_a_plain_backup_is_refused_and_leaves_nothing(string how)
{
var admin = await _host.Admin();
using var client = _host.As(admin.Jwt);
const string id = "20260101-000000-manual";
var tar = new MemoryStream();
if (how == "junk")
tar.Write("not a tar at all, not even close"u8);
else
await using (var writer = new TarWriter(tar, TarEntryFormat.Pax, leaveOpen: true))
{
await writer.WriteEntryAsync(new PaxTarEntry(TarEntryType.RegularFile, $"{id}/manifest.json") { DataStream = new MemoryStream("{}"u8.ToArray()) }, Token);
await writer.WriteEntryAsync(how switch
{
"escape" => new PaxTarEntry(TarEntryType.RegularFile, $"{id}/../../evil") { DataStream = new MemoryStream("x"u8.ToArray()) },
"link" => new PaxTarEntry(TarEntryType.SymbolicLink, $"{id}/media/x.jpg") { LinkName = "/etc/passwd" },
_ => new PaxTarEntry(TarEntryType.RegularFile, "20260101-000000-nightly/db/Post.jsonl.gz") { DataStream = new MemoryStream("x"u8.ToArray()) }
}, Token);
}
var upload = (await (await client.PostAsync($"{Base}/uploads", null, Token)).Content.ReadFromJsonAsync<JsonObject>(Token))!["id"]!.GetValue<string>();
Assert.Equal(HttpStatusCode.OK, (await Piece(client, upload, 0, tar.ToArray())).StatusCode);
Assert.Equal(HttpStatusCode.UnprocessableEntity, (await client.PostAsync($"{Base}/uploads/{upload}/finish", null, Token)).StatusCode);
var root = _host.Get<Backups>().Root;
Assert.False(Directory.Exists(Path.Combine(root, id)));
Assert.False(Directory.Exists(Path.Combine(root, id + ServerBackup.PartialSuffix)));
Assert.False(File.Exists(Path.Combine(root, "..", "evil")));
}
// asked for: written for the next start (the test host has no watcher to stop it, and the marker is taken away)
[Fact]
public async Task A_restore_asks_for_the_password_and_the_host_typed_out()
{
var admin = await _host.Admin();
using var client = _host.As(admin.Jwt);
var id = await BackedUp(client);
var root = _host.Get<Backups>().Root;
try
{
Assert.Equal(HttpStatusCode.UnprocessableEntity, (await client.PostAsJsonAsync($"{Base}/{id}/restore", new { password = "wrong", host = "privapub.test" }, Token)).StatusCode);
Assert.Equal(HttpStatusCode.UnprocessableEntity, (await client.PostAsJsonAsync($"{Base}/{id}/restore", new { password = admin.Password, host = "elsewhere.test" }, Token)).StatusCode);
Assert.Null(RestoreMarker.Read(root));
Assert.Equal(HttpStatusCode.Accepted, (await client.PostAsJsonAsync($"{Base}/{id}/restore", new { password = admin.Password, host = "PrivaPub.test" }, Token)).StatusCode);
var marker = RestoreMarker.Read(root);
Assert.Equal((id, "pending", admin.UserName), (marker.Backup, marker.State, marker.RequestedBy));
Assert.Equal(id, (await client.GetFromJsonAsync<JsonObject>(Base, Token))!["waiting"]!["backup"]!.GetValue<string>());
Assert.Equal(HttpStatusCode.Conflict, (await client.DeleteAsync($"{Base}/{id}", Token)).StatusCode);
}
finally
{
RestoreMarker.Clear(root);
_host.Get<Backups>().Delete(id);
}
}
}
}
+140
View File
@@ -0,0 +1,140 @@
using MongoDB.Entities;
using PrivaPub.Federation.Actors;
using PrivaPub.Models.Federation;
using PrivaPub.Models.Jobs;
using PrivaPub.Models.Post;
using PrivaPub.Models.Social;
using PrivaPub.Tests.Support;
using PrivaPub.Tests.Support.Host;
using System.Net;
using System.Text.Json.Nodes;
namespace PrivaPub.Tests.Http
{
// Lemmy 1.0's multi-communities and PieFed's feeds are `Feed` actors whose `following` is their communities
[Trait("Category", "Integration")]
public sealed class MastodonFeedTests : IAsyncLifetime
{
PrivaPubHost _host;
Peer _peer;
public async ValueTask InitializeAsync()
{
Assert.SkipUnless(MongoFixture.Enabled, MongoFixture.Skip);
_host = await PrivaPubHost.Shared();
_peer = await Peer.Start();
}
public async ValueTask DisposeAsync()
{
if (_peer != default)
await _peer.DisposeAsync();
}
[Fact]
public async Task A_feed_shows_as_an_account_whose_following_is_its_communities()
{
var token = TestContext.Current.CancellationToken;
var reader = await _host.Mastodon("feeder");
var cats = new RemoteActor(_peer, "cats", type: "Group");
var dogs = new RemoteActor(_peer, "dogs", type: "Group");
var name = $"feed{Guid.NewGuid():N}"[..12];
var feed = $"{_peer.A}/m/{name}";
// as Lemmy serves one: the instance's inbox and key, its text in `description`
_peer.Serve($"/m/{name}", new JsonObject
{
["@context"] = new JsonArray("https://join-lemmy.org/context.json", "https://www.w3.org/ns/activitystreams"),
["type"] = "Feed", ["id"] = feed, ["inbox"] = $"{_peer.A}/inbox",
["publicKey"] = new JsonObject { ["id"] = $"{_peer.A}/#main-key", ["owner"] = $"{_peer.A}/", ["publicKeyPem"] = "-----BEGIN PUBLIC KEY-----\n-----END PUBLIC KEY-----\n" },
["following"] = $"{feed}/following", ["preferredUsername"] = name, ["name"] = "Pets", ["description"] = "every pet",
["attributedTo"] = $"{_peer.A}/users/owner"
}.ToJsonString());
_peer.Serve($"/m/{name}/following", new JsonObject
{
["type"] = "Collection", ["id"] = $"{feed}/following", ["totalItems"] = 2, ["items"] = new JsonArray(cats.Id, dogs.Id)
}.ToJsonString());
var account = await _host.Get<IRemoteActorService>().GetActor(feed, refresh: true, token);
Assert.NotNull(account);
await _host.Run(j => j.Kind == JobKind.CountAccount && j.Payload == feed, token);
var shown = (await reader.Client.Get($"/api/v1/accounts/{account.ID}")).Ok().Body;
Assert.Equal(("Pets", 2), (shown.Text("display_name"), shown.Number("following_count")));
Assert.Contains("every pet", shown.Text("note"));
var communities = (await reader.Client.Get($"/api/v1/accounts/{account.ID}/following")).Ok().Array;
Assert.Equal(new[] { cats.Id, dogs.Id }, communities.Select(c => c.Text("uri")));
Assert.All(communities, c => Assert.True(c!["group"]!.GetValue<bool>()));
}
// owner decision 2026-10-07: a feed is read through the server, whose reader follows its communities; the persona
// follows nothing, and reads the feed in a timeline of its own
[Fact]
public async Task A_feed_is_followed_by_the_servers_reader_and_read_in_its_own_timeline()
{
var token = TestContext.Current.CancellationToken;
var since = DateTime.UtcNow.AddSeconds(-1);
var persona = await _host.Mastodon("feedreader");
var cats = new RemoteActor(_peer, "cats", type: "Group");
var dogs = new RemoteActor(_peer, "dogs", type: "Group");
var poster = new RemoteActor(_peer, "poster");
var name = $"feed{Guid.NewGuid():N}"[..12];
var feed = $"{_peer.A}/m/{name}";
_peer.Serve($"/m/{name}", new JsonObject
{
["type"] = "Feed", ["id"] = feed, ["inbox"] = $"{_peer.A}/inbox", ["following"] = $"{feed}/following", ["preferredUsername"] = name, ["name"] = "Pets"
}.ToJsonString());
_peer.Serve($"/m/{name}/following", new JsonObject
{
["type"] = "Collection", ["id"] = $"{feed}/following", ["totalItems"] = 2, ["items"] = new JsonArray(cats.Id, dogs.Id)
}.ToJsonString());
var account = await _host.Get<IRemoteActorService>().GetActor(feed, refresh: true, token);
await _host.Run(j => j.Kind == JobKind.CountAccount && j.Payload == feed, token);
var followed = (await persona.Client.Post($"/api/v1/accounts/{account.ID}/follow")).Ok().Body;
Assert.True(followed.Flag("following"));
Assert.Equal(new[] { account.ID }, (await persona.Client.Get("/api/v1/feeds")).Ok().Ids);
await _host.Run(j => j.Kind == JobKind.SyncFeeds, token);
var readerActor = await _host.Get<ILocalActorService>().GetReaderActor(token);
var follows = await DB.Default.Find<Following>().Match(f => f.AvatarId == readerActor.Id).ExecuteAsync(token);
Assert.Equal(new[] { cats.Id, dogs.Id }.Order(), follows.Select(f => f.TargetActorURI).Order());
Assert.All(follows, f => Assert.Equal(LocalActorKind.Reader, f.FollowerKind));
Assert.False(await DB.Default.Find<Following>().Match(f => f.AvatarId == persona.Id).ExecuteAnyAsync(token));
var asked = Assert.Single(await cats.Delivered(since), d => d.Type() == "Follow");
Assert.Equal(readerActor.Uri, asked.Text("actor"));
var document = (await _host.Client().Fetch(new Uri(readerActor.Uri).AbsolutePath)).Json;
Assert.Equal(("Service", LocalActorService.ReaderUserName), (document.Text("type"), document.Text("preferredUsername")));
await _host.Deliver(cats, "/human-centipede", new JsonObject
{
["id"] = $"{cats.Id}/accepts/{Guid.NewGuid():N}", ["type"] = "Accept", ["actor"] = cats.Id, ["object"] = asked.Text("id")
});
var create = poster.Create("<p>a thread among cats</p>", new[] { "https://www.w3.org/ns/activitystreams#Public", cats.Id }, shape: note =>
{
note["type"] = "Page";
note["name"] = "Cats";
note["audience"] = cats.Id;
});
_peer.Serve(new Uri(create["object"]!.Text("id")).AbsolutePath, create["object"]!.ToJsonString());
await _host.Deliver(cats, "/human-centipede", new JsonObject
{
["id"] = $"{cats.Id}/announces/{Guid.NewGuid():N}", ["type"] = "Announce", ["actor"] = cats.Id, ["object"] = create,
["to"] = new JsonArray("https://www.w3.org/ns/activitystreams#Public")
});
var thread = await DB.Default.Find<Post>().Match(p => p.ObjectURI == create["object"]!.Text("id")).ExecuteSingleAsync(token);
Assert.Equal(new[] { thread.ID }, (await persona.Client.Get($"/api/v1/timelines/feed/{account.ID}")).Ok().Ids);
Assert.DoesNotContain(thread.ID, (await persona.Client.Get("/api/v1/timelines/home")).Ok().Ids);
Assert.False(await DB.Default.Find<TimelineEntry>().Match(e => e.PostId == thread.ID).ExecuteAnyAsync(token));
var stranger = await _host.Mastodon("notreading");
Assert.Equal(HttpStatusCode.NotFound, (await stranger.Client.Get($"/api/v1/timelines/feed/{account.ID}")).Status);
Assert.False((await persona.Client.Post($"/api/v1/accounts/{account.ID}/unfollow")).Ok().Body.Flag("following"));
await _host.Run(j => j.Kind == JobKind.SyncFeeds, token);
Assert.False(await DB.Default.Find<Following>().Match(f => f.AvatarId == readerActor.Id).ExecuteAnyAsync(token));
Assert.Contains(await cats.Delivered(since), d => d.Type() == "Undo" && d["object"].Text("type") == "Follow");
}
}
}
@@ -45,6 +45,8 @@ namespace PrivaPub.Tests.Http
Assert.Equal($"wss://{PrivaPubHost.Host}", v1.Body["urls"]!.Text("streaming_api"));
Assert.Equal(PrivaPub.Api.Mastodon.Controllers.StatusesController.MaxPins, v2.Body["configuration"]!["accounts"].Number("max_pinned_statuses"));
Assert.Contains("image/avif", v2.Body["configuration"]!["media_attachments"]!["supported_mime_types"]!.AsArray().Select(t => t!.GetValue<string>()));
// what the bundled libvips can't decode is not offered
Assert.DoesNotContain("image/heic", v2.Body["configuration"]!["media_attachments"]!["supported_mime_types"]!.AsArray().Select(t => t!.GetValue<string>()));
Assert.True(v2.Body["registrations"].Flag("enabled"));
Assert.True((await anonymous.Get("/nodeinfo/2.1")).Ok().Body.Flag("openRegistrations"));
Assert.False(v2.Body["configuration"]!["translation"].Flag("enabled"));
+139 -4
View File
@@ -1,3 +1,4 @@
using PrivaPub.Models.Jobs;
using PrivaPub.Domain.Media;
using PrivaPub.Tests.Support;
using PrivaPub.Tests.Support.Host;
@@ -31,6 +32,14 @@ namespace PrivaPub.Tests.Http
static CancellationToken Token => TestContext.Current.CancellationToken;
// what GET /api/v1/media/:id answers once the job processing an upload sent to v2 has run
async Task<ApiAnswer> Processed(Mastodon account, ApiAnswer accepted)
{
var id = accepted.Body.Text("id");
await _host.Run(j => j.Kind == JobKind.ProcessMedia && j.Payload == id, Token);
return await account.Client.Get($"/api/v1/media/{id}");
}
static Task<ApiAnswer> Upload(Mastodon account, string path, byte[] bytes, string contentType, string fileName, params (string Key, string Value)[] fields)
{
var form = MastodonHelpers.Multipart(("file", bytes, contentType, fileName));
@@ -136,6 +145,25 @@ namespace PrivaPub.Tests.Http
Assert.Null((await alice.Client.Put($"/api/v1/media/{id}", ("description", ""))).Ok().Body.Text("description"));
}
// a focal point is two finite numbers: NaN or Infinity is ignored, never stored (it broke every timeline holding the post)
[Fact]
public async Task A_focal_point_that_is_not_a_number_is_ignored()
{
var alice = await _host.Mastodon("alice");
var uploaded = (await Upload(alice, "/api/v2/media", MastodonHelpers.JpegWithMetadata(32, 32), "image/jpeg", "a.jpg", ("focus", "NaN,NaN"))).Ok();
Assert.Null(uploaded.Body["meta"]?["focus"]);
var id = uploaded.Body.Text("id");
(await alice.Client.Put($"/api/v1/media/{id}", ("focus", "0.5,0.25"))).Ok();
foreach (var unsound in new[] { "NaN,0", "Infinity,1", "0,-Infinity" })
{
var focus = (await alice.Client.Put($"/api/v1/media/{id}", ("focus", unsound))).Ok().Body["meta"]!["focus"]!;
Assert.Equal((0.5, 0.25), (focus["x"]!.GetValue<double>(), focus["y"]!.GetValue<double>()));
}
var status = (await alice.Client.Post("/api/v1/statuses", ("status", "focused"), ("media_ids[]", id))).Ok();
Assert.Equal(0.5, status.Body["media_attachments"]![0]!["meta"]!["focus"]!["x"]!.GetValue<double>());
}
[Fact]
public async Task Unsupported_unreadable_and_missing_files_are_refused_with_422()
{
@@ -145,12 +173,50 @@ namespace PrivaPub.Tests.Http
Assert.Equal(HttpStatusCode.UnprocessableEntity, text.Status);
Assert.Contains("not supported", text.Body.Text("error"));
Assert.Equal(HttpStatusCode.UnprocessableEntity, (await Upload(alice, "/api/v2/media", Encoding.UTF8.GetBytes("not a jpeg"), "image/jpeg", "a.jpg")).Status);
Assert.Equal(HttpStatusCode.UnprocessableEntity, (await Upload(alice, "/api/v2/media", Encoding.UTF8.GetBytes("not a video"), "video/mp4", "a.mp4")).Status);
Assert.Equal(HttpStatusCode.UnprocessableEntity, (await Upload(alice, "/api/v1/media", Encoding.UTF8.GetBytes("not a video"), "video/mp4", "a.mp4")).Status);
// sent to v2, it is taken, then refused once processed
var later = await Upload(alice, "/api/v2/media", Encoding.UTF8.GetBytes("not a video"), "video/mp4", "a.mp4");
Assert.Equal(HttpStatusCode.Accepted, later.Status);
Assert.Equal(HttpStatusCode.UnprocessableEntity, (await Processed(alice, later)).Status);
var empty = new MultipartFormDataContent { { new StringContent("no file"), "description" } };
Assert.Equal(HttpStatusCode.UnprocessableEntity, (await alice.Client.Exchange(new HttpRequestMessage(HttpMethod.Post, "/api/v2/media") { Content = empty })).Status);
Assert.Equal(HttpStatusCode.UnprocessableEntity, (await alice.Client.Post("/api/v1/media")).Status);
}
// refused before anything is decoded: an SVG claiming to be a PNG, and an image that would decode to too many pixels
[Fact]
public async Task An_image_that_is_not_one_or_too_large_is_refused_before_decoding()
{
var alice = await _host.Mastodon("alice");
var svg = "<svg xmlns='http://www.w3.org/2000/svg' width='30000' height='30000'><rect width='1' height='1'/></svg>"u8.ToArray();
Assert.Equal(HttpStatusCode.UnprocessableEntity, (await Upload(alice, "/api/v2/media", svg, "image/png", "a.png")).Status);
using var huge = NetVips.Image.Black(8000, 6000);
var tooLarge = await Upload(alice, "/api/v2/media", huge.WriteToBuffer(".png"), "image/png", "huge.png");
Assert.Equal(HttpStatusCode.UnprocessableEntity, tooLarge.Status);
Assert.Contains("too large", tooLarge.Body.Text("error"));
}
// an animated GIF becomes what Mastodon makes of one, a looping mp4 typed gifv; a still GIF stays an image
[Fact]
public async Task An_animated_gif_becomes_a_gifv_and_a_still_one_an_image()
{
var alice = await _host.Mastodon("alice");
using var frame = (NetVips.Image.Black(64, 48, bands: 3) + 60).Cast(NetVips.Enums.BandFormat.Uchar);
using var frames = NetVips.Image.Arrayjoin(new[] { frame, frame + 80, frame + 160 }, across: 1).Cast(NetVips.Enums.BandFormat.Uchar);
using var animated = frames.Mutate(m => m.Set(NetVips.GValue.GIntType, "page-height", 48));
var gifv = (await Upload(alice, "/api/v2/media", animated.WriteToBuffer(".gif"), "image/gif", "dance.gif")).Ok();
Assert.Equal("gifv", gifv.Body.Text("type"));
Assert.EndsWith(".mp4", gifv.Body.Text("url"));
Assert.Equal(64, gifv.Body["meta"]!["original"]!["width"]!.GetValue<int>());
var status = (await alice.Client.Post("/api/v1/statuses", ("status", "dancing"), ("media_ids[]", gifv.Body.Text("id")))).Ok();
Assert.Equal("gifv", status.Body["media_attachments"]![0]!.Text("type"));
var still = (await Upload(alice, "/api/v2/media", frame.WriteToBuffer(".gif"), "image/gif", "still.gif")).Ok();
Assert.Equal("image", still.Body.Text("type"));
}
[Fact]
public async Task Video_is_remuxed_without_its_metadata()
{
@@ -160,7 +226,13 @@ namespace PrivaPub.Tests.Http
"-c:v", "mpeg4", "-c:a", "aac", "-shortest");
Assert.Contains("secret title", Encoding.Latin1.GetString(video));
var uploaded = (await Upload(alice, "/api/v2/media", video, "video/mp4", "clip.mp4")).Ok();
// v2 answers before it is processed: 202, no url yet, and it can't be posted until it is
var accepted = await Upload(alice, "/api/v2/media", video, "video/mp4", "clip.mp4");
Assert.Equal(HttpStatusCode.Accepted, accepted.Status);
Assert.Null(accepted.Body["url"]);
Assert.Equal(HttpStatusCode.PartialContent, (await alice.Client.Get($"/api/v1/media/{accepted.Body.Text("id")}")).Status);
Assert.Equal(HttpStatusCode.UnprocessableEntity, (await alice.Client.Post("/api/v1/statuses", ("status", "too soon"), ("media_ids[]", accepted.Body.Text("id")))).Status);
var uploaded = (await Processed(alice, accepted)).Ok();
Assert.Equal("video", uploaded.Body.Text("type"));
Assert.Equal(320, uploaded.Body["meta"]!["original"].Number("width"));
@@ -168,6 +240,8 @@ namespace PrivaPub.Tests.Http
Assert.EndsWith(".jpg", uploaded.Body.Text("preview_url"));
var probe = await Probe(uploaded.Body.Text("url"));
Assert.Contains("\"codec_type\": \"video\"", probe);
// MPEG-4 Part 2, which browsers don't play, made H.264
Assert.Contains("\"codec_name\": \"h264\"", probe);
Assert.DoesNotContain("secret title", probe);
Assert.DoesNotContain("filmed at home", probe);
Assert.DoesNotContain("hidden handler", probe);
@@ -180,7 +254,8 @@ namespace PrivaPub.Tests.Http
var audio = await Made("m4a", "-f", "lavfi", "-i", "sine=frequency=330:duration=1", "-metadata", "title=secret song", "-metadata", "artist=Alice Smith",
"-c:a", "aac");
var uploaded = (await Upload(alice, "/api/v2/media", audio, "audio/mp4", "song.m4a")).Ok();
// v1 waits for it
var uploaded = (await Upload(alice, "/api/v1/media", audio, "audio/mp4", "song.m4a")).Ok();
Assert.Equal("audio", uploaded.Body.Text("type"));
var probe = await Probe(uploaded.Body.Text("url"));
@@ -189,6 +264,35 @@ namespace PrivaPub.Tests.Http
Assert.DoesNotContain("Alice Smith", probe);
}
// FLAC is served as what it is (ASP.NET's map has no entry for it, so it was a 404)
[Fact]
public async Task Flac_is_served()
{
var alice = await _host.Mastodon("alice");
var flac = await Made("flac", "-f", "lavfi", "-i", "sine=frequency=500:duration=1", "-c:a", "flac");
var uploaded = (await Upload(alice, "/api/v1/media", flac, "audio/flac", "song.flac")).Ok();
var served = await _host.Client().GetAsync(new Uri(uploaded.Body.Text("url")).PathAndQuery, Token);
Assert.Equal(HttpStatusCode.OK, served.StatusCode);
Assert.Equal("audio/flac", served.Content.Headers.ContentType?.MediaType);
}
// a video larger than video_matrix_limit is made smaller, its shape kept
[Fact]
public async Task A_video_larger_than_the_limit_is_made_smaller()
{
var alice = await _host.Mastodon("alice");
var video = await Made("mp4", "-f", "lavfi", "-i", "testsrc=duration=1:size=2000x1500:rate=5", "-c:v", "libx264", "-preset", "ultrafast", "-pix_fmt", "yuv420p");
var uploaded = (await Upload(alice, "/api/v1/media", video, "video/mp4", "big.mp4")).Ok();
var width = uploaded.Body["meta"]!["original"].Number("width");
var height = uploaded.Body["meta"]!["original"].Number("height");
Assert.True((long)width * height <= 2_304_000, $"{width}x{height}");
Assert.InRange((double)width / height, 1.3, 1.37);
}
static byte[] Bytes(int length)
{
var bytes = new byte[length];
@@ -311,6 +415,36 @@ namespace PrivaPub.Tests.Http
Assert.Single(_peer.Requests);
}
// clients asking at once for a file not cached yet share one download
[Fact]
public async Task Clients_asking_at_once_share_one_download()
{
var proxy = _host.Get<IMediaProxy>();
var bytes = Bytes(200_000);
var remote = Served(bytes, "image/png");
var path = new Uri(remote).AbsolutePath;
using var client = _host.Client();
var answers = await Task.WhenAll(Enumerable.Range(0, 5).Select(_ => client.GetAsync(proxy.Wrap(remote), Token)));
foreach (var answer in answers)
Assert.Equal(bytes, await answer.Content.ReadAsByteArrayAsync(Token));
Assert.Single(_peer.Requests, r => r.Path == path);
}
// a remote file that can't be had is a 404 no browser keeps
[Fact]
public async Task A_remote_file_that_fails_is_not_cached_by_browsers()
{
var proxy = _host.Get<IMediaProxy>();
using var client = _host.Client();
using var missing = await client.GetAsync(proxy.Wrap($"{_peer.A}/media/{Guid.NewGuid():N}.png"), Token);
Assert.Equal(HttpStatusCode.NotFound, missing.StatusCode);
Assert.Null(missing.Headers.CacheControl);
}
[Fact]
public async Task Anything_over_the_proxy_limit_is_streamed_and_never_cached()
{
@@ -328,7 +462,8 @@ namespace PrivaPub.Tests.Http
Assert.Equal(default, proxy.Cached(remote));
using var again = await client.GetAsync(proxy.Wrap(remote), Token);
Assert.Equal(bytes, await again.Content.ReadAsByteArrayAsync(Token));
Assert.Equal(4, _peer.Requests.Count);
// the first time, an attempt to cache it and the stream; then it is known to be too big, and only streamed
Assert.Equal(3, _peer.Requests.Count);
var small = Served(Bytes(SmallProxyHost.Limit / 2));
using (var fits = await client.GetAsync(proxy.Wrap(small), Token))
Assert.Equal(HttpStatusCode.OK, fits.StatusCode);
@@ -99,5 +99,60 @@ namespace PrivaPub.Tests.Http
Assert.Equal(published.Text("id"), media.PostId);
Assert.Null(media.ScheduledStatusId);
}
async Task<string> Picture(Mastodon account)
{
var upload = MastodonHelpers.Multipart(("file", MastodonHelpers.JpegWithMetadata(16, 16), "image/jpeg", "a.jpg"));
return (await account.Client.Exchange(new HttpRequestMessage(HttpMethod.Post, "/api/v2/media") { Content = upload })).Ok().Body.Text("id");
}
// a scheduled post's media are its alone: a post now, or an edit, can't take them
[Fact]
public async Task A_scheduled_posts_media_cannot_be_taken_by_a_post_now()
{
var alice = await _host.Mastodon("alice");
var mediaId = await Picture(alice);
(await alice.Client.Post("/api/v1/statuses", ("status", "later"), ("media_ids[]", mediaId), ("scheduled_at", In(TimeSpan.FromMinutes(10))))).Ok();
Assert.Equal(HttpStatusCode.UnprocessableEntity, (await alice.Client.Post("/api/v1/statuses", ("status", "now"), ("media_ids[]", mediaId))).Status);
var plain = (await alice.Client.Post("/api/v1/statuses", ("status", "plain"))).Ok().Body.Text("id");
Assert.Equal(HttpStatusCode.UnprocessableEntity, (await alice.Client.Put($"/api/v1/statuses/{plain}", ("status", "edited"), ("media_ids[]", mediaId))).Status);
var media = await DB.Default.Find<MediaAttachment>().OneAsync(mediaId, Token);
Assert.Null(media.PostId);
Assert.NotNull(media.ScheduledStatusId);
}
// two posts asking for the same upload at once: exactly one gets it
[Fact]
public async Task Two_posts_racing_for_one_upload_never_both_get_it()
{
var alice = await _host.Mastodon("alice");
for (var round = 0; round < 5; round++)
{
var mediaId = await Picture(alice);
var answers = await Task.WhenAll(Enumerable.Range(0, 2).Select(i =>
alice.Client.Post("/api/v1/statuses", ("status", $"race {round} {i}"), ("media_ids[]", mediaId))));
Assert.Single(answers, a => a.Status == HttpStatusCode.OK);
Assert.Single(answers, a => a.Status == HttpStatusCode.UnprocessableEntity);
}
}
// a persona deleted since it scheduled a post publishes nothing, and what the post held is trashed
[Fact]
public async Task A_gone_personas_scheduled_post_never_publishes()
{
var gone = await _host.Mastodon($"gone{Guid.NewGuid():N}"[..12]);
var mediaId = await Picture(gone);
var id = (await gone.Client.Post("/api/v1/statuses", ("status", "from beyond"), ("media_ids[]", mediaId), ("scheduled_at", In(TimeSpan.FromMinutes(10))))).Ok().Object.Text("id");
await DB.Default.Update<Models.User.Avatar>().MatchID(gone.Persona.Id).Modify(a => a.DeletionAt, DateTime.UtcNow).ExecuteAsync(Token);
await DB.Default.Update<ScheduledStatus>().MatchID(id).Modify(s => s.ScheduledAt, DateTime.UtcNow.AddSeconds(-1)).ExecuteAsync(Token);
await DB.Default.Update<Job>().Match(j => j.Kind == JobKind.PublishScheduled && j.Payload == id).Modify(j => j.RunAt, DateTime.UtcNow).ExecuteAsync(Token);
await _host.Run(j => j.Kind == JobKind.PublishScheduled && j.Payload == id, Token);
Assert.False(await DB.Default.Find<Models.Post.Post>().Match(p => p.GroupUserId == gone.Persona.Id && p.Text == "from beyond").ExecuteAnyAsync(Token));
Assert.False(await DB.Default.Find<ScheduledStatus>().Match(s => s.ID == id).ExecuteAnyAsync(Token));
Assert.NotNull((await DB.Default.Find<MediaAttachment>().OneAsync(mediaId, Token)).TrashedAt);
}
}
}
@@ -388,6 +388,47 @@ namespace PrivaPub.Tests.Http
Assert.True((await alice.Client.Post($"/api/v1/statuses/{quiet.Text("id")}/favourite")).Ok().Body.Flag("favourited"));
}
// a downvote is a Dislike, one vote a post: a downvote replaces the favourite and a favourite the downvote, each by its
// own activity alone; a vote on a post in a community goes to the community, which counts it, and to its author only
// when on another server
[Fact]
public async Task A_downvote_replaces_the_favourite_and_votes_reach_the_posts_community()
{
var since = DateTime.UtcNow.AddSeconds(-1);
var alice = await _host.Mastodon("alice");
var bob = new RemoteActor(_peer, "bob", origin: _peer.B);
var community = new RemoteActor(_peer, "pies", type: "Group");
await _host.Get<PrivaPub.Federation.Actors.IRemoteActorService>().GetActor(community.Id, refresh: true, TestContext.Current.CancellationToken);
var remote = await _host.PublicPostFrom(bob, alice, shape: note => note["audience"] = community.Id);
var neighbour = new RemoteActor(_peer, "neighbour");
var nearby = await _host.PublicPostFrom(neighbour, alice, shape: note => note["audience"] = community.Id);
(await alice.Client.Post($"/api/v1/statuses/{nearby.ID}/favourite")).Ok();
Assert.Empty(await neighbour.Delivered(since));
var path = $"/api/v1/statuses/{remote.ID}";
(await alice.Client.Post(path + "/favourite")).Ok();
var down = (await alice.Client.Post(path + "/downvote")).Ok().Body;
Assert.False(down.Flag("favourited"));
Assert.Equal((0, 1, true), (down["privapub"]!["votes"].Number("up"), down["privapub"]!["votes"].Number("down"), down["privapub"]!["votes"].Flag("downvoted")));
Assert.Equal(1, (await alice.Client.Post(path + "/downvote")).Ok().Body["privapub"]!["votes"].Number("down"));
var up = (await alice.Client.Post(path + "/favourite")).Ok().Body;
Assert.Equal((true, 0, false), (up.Flag("favourited"), up["privapub"]!["votes"].Number("down"), up["privapub"]!["votes"].Flag("downvoted")));
Assert.Null((await _host.Client().Get(path)).Ok().Body["privapub"]!["votes"]!["downvoted"]);
(await alice.Client.Post(path + "/downvote")).Ok();
var none = (await alice.Client.Post(path + "/undownvote")).Ok().Body;
Assert.Equal((false, 0, 0), (none.Flag("favourited"), none.Number("favourites_count"), none["privapub"]!["votes"].Number("down")));
foreach (var inbox in new[] { bob, community })
{
var sent = (await inbox.Delivered(since)).Where(d => (d.Type() == "Undo" ? d["object"].Text("object") : d.Text("object")) == remote.ObjectURI).ToList();
Assert.Equal(new[] { "Like", "Dislike", "Like", "Dislike", "Undo:Dislike" },
sent.Select(d => d.Type() == "Undo" ? "Undo:" + d["object"].Text("type") : d.Type()));
var dislike = sent.Last(d => d.Type() == "Dislike");
Assert.Equal((alice.Uri, remote.ObjectURI), (dislike.Text("actor"), dislike.Text("object")));
Assert.Equal(dislike.Text("id"), sent.Single(d => d.Type() == "Undo")["object"].Text("id"));
}
}
[Fact]
public async Task A_favourite_or_a_boost_given_again_after_its_undo_is_a_new_activity_that_is_delivered()
{
@@ -1,6 +1,12 @@
using MongoDB.Entities;
using PrivaPub.Federation.Objects;
using PrivaPub.Models.Social;
using PrivaPub.Tests.Support;
using PrivaPub.Tests.Support.Host;
using System.Text.Json.Nodes;
namespace PrivaPub.Tests.Http
{
// Status search by words, as Mastodon scopes it: the persona's own posts and those it boosted, favourited, bookmarked
@@ -47,6 +53,67 @@ namespace PrivaPub.Tests.Http
(await reader.Client.Get($"/api/v2/search?q={word}&type=statuses&limit=1&offset=1")).Ok().Body["statuses"]!.AsArray().Select(s => s.Text("id")));
}
// FEP-268d: a remote account's searchableBy outranks its indexable, and a post's own outranks its author's
[Fact]
public async Task A_remote_searchable_by_decides_before_indexable()
{
var token = TestContext.Current.CancellationToken;
await using var peer = await Peer.Start();
var reader = await _host.Mastodon("reader");
var keeper = await _host.Mastodon("keeper");
var word = $"s{Guid.NewGuid():N}"[..12];
async Task<RemoteActor> Author(string name, bool indexable, Func<RemoteActor, JsonArray> searchableBy)
{
var author = new RemoteActor(peer, name);
var document = author.Document();
document["indexable"] = indexable;
if (searchableBy != default)
document["searchableBy"] = searchableBy(author);
peer.Serve($"/users/{author.Name}", document.ToJsonString());
await DB.Default.SaveAsync(new Following { AvatarId = keeper.Id, TargetActorURI = author.Id, State = FollowState.Accepted }, token);
return author;
}
async Task<string> Says(RemoteActor author, string text, JsonArray searchableBy = default)
{
var noteId = $"{peer.A}/notes/{Guid.NewGuid():N}";
var note = new JsonObject
{
["id"] = noteId, ["type"] = "Note", ["attributedTo"] = author.Id, ["content"] = $"<p>{text}</p>",
["to"] = new JsonArray(Addressing.Public), ["cc"] = new JsonArray(author.Id + "/followers"), ["published"] = DateTime.UtcNow.ToString("O")
};
if (searchableBy != default)
note["searchableBy"] = searchableBy;
using var client = _host.Client();
await client.SendAsync(author.SignedPost("/human-centipede", new JsonObject
{
["id"] = noteId + "/activity", ["type"] = "Create", ["actor"] = author.Id,
["to"] = new JsonArray(Addressing.Public), ["cc"] = new JsonArray(author.Id + "/followers"), ["object"] = note
}), token);
Assert.Equal(1, await _host.RunInbox(noteId + "/activity", token));
return (await DB.Default.Find<Models.Post.Post>().Match(p => p.ObjectURI == noteId).ExecuteFirstAsync(token)).ID;
}
var open = await Says(await Author("open", indexable: false, _ => new JsonArray(Addressing.Public)), $"open {word}");
var closed = await Says(await Author("closed", indexable: true, _ => new JsonArray()), $"closed {word}");
var friendly = await Author("friendly", indexable: false, a => new JsonArray(a.Id + "/followers"));
var friends = await Says(friendly, $"friends {word}");
var quiet = await Author("quiet", indexable: false, default);
var perPost = await Says(quiet, $"per post {word}", new JsonArray(Addressing.Public));
var plain = await Says(quiet, $"plain {word}");
static List<string> Found(JsonNode body) => body["statuses"]!.AsArray().Select(s => s.Text("id")).ToList();
var before = Found((await reader.Client.Get($"/api/v2/search?q={word}&type=statuses&limit=40")).Ok().Body);
Assert.Contains(open, before);
Assert.Contains(perPost, before);
Assert.DoesNotContain(closed, before);
Assert.DoesNotContain(friends, before);
Assert.DoesNotContain(plain, before);
await DB.Default.SaveAsync(new Following { AvatarId = reader.Id, TargetActorURI = friendly.Id, State = FollowState.Accepted }, token);
Assert.Contains(friends, Found((await reader.Client.Get($"/api/v2/search?q={word}&type=statuses&limit=40")).Ok().Body));
}
// a handle asks for its account: no posts sharing its words, and no hashtag made of it
[Fact]
public async Task A_handle_finds_its_account_and_neither_posts_nor_a_hashtag()
+234
View File
@@ -0,0 +1,234 @@
using MongoDB.Entities;
using PrivaPub.Domain.Media;
using PrivaPub.Models.Media;
using PrivaPub.Services;
using PrivaPub.Tests.Support;
using PrivaPub.Tests.Support.Host;
using System.Net;
namespace PrivaPub.Tests.Http
{
// A file lives exactly as long as something holds it: a deleted post, an edit leaving media out, a replaced picture and
// a removed root each trash theirs, which stops /media/files serving them at once; the janitor deletes them later.
[Trait("Category", "Integration")]
public sealed class MediaLifecycleTests : IAsyncLifetime
{
PrivaPubHost _host;
public async ValueTask InitializeAsync()
{
Assert.SkipUnless(MongoFixture.Enabled, MongoFixture.Skip);
_host = await PrivaPubHost.Shared();
}
public ValueTask DisposeAsync() => ValueTask.CompletedTask;
static CancellationToken Token => TestContext.Current.CancellationToken;
async Task<string> Upload(Mastodon account, string name)
{
var form = MastodonHelpers.Multipart(("file", MastodonHelpers.JpegWithMetadata(64, 48), "image/jpeg", name));
return (await account.Client.Exchange(new HttpRequestMessage(HttpMethod.Post, "/api/v2/media") { Content = form })).Ok().Body.Text("id");
}
async Task<HttpStatusCode> Served(string url) =>
(await _host.Client().GetAsync(new Uri(url).PathAndQuery, Token)).StatusCode;
[Fact]
public async Task A_deleted_posts_media_stop_being_served()
{
var alice = await _host.Mastodon("alice");
var id = await Upload(alice, "a.jpg");
var status = (await alice.Client.Post("/api/v1/statuses", ("status", "with a picture"), ("media_ids[]", id))).Ok();
var url = status.Body["media_attachments"]![0]!["url"]!.GetValue<string>();
var preview = status.Body["media_attachments"]![0]!["preview_url"]!.GetValue<string>();
Assert.Equal(HttpStatusCode.OK, await Served(url));
(await alice.Client.Delete($"/api/v1/statuses/{status.Body.Text("id")}")).Ok();
Assert.Equal(HttpStatusCode.NotFound, await Served(url));
Assert.Equal(HttpStatusCode.NotFound, await Served(preview));
Assert.Equal("post deleted", (await DB.Default.Find<MediaAttachment>().OneAsync(id, Token)).TrashReason);
// the file waits in the trash, beside what /media/files serves
var relative = url[(url.IndexOf("/media/files/", StringComparison.Ordinal) + "/media/files/".Length)..];
Assert.True(File.Exists(Path.Combine(_host.Get<IMediaService>().TrashRoot, relative)));
Assert.Equal(HttpStatusCode.NotFound, await Served(url.Replace("/media/files/", "/media/files/.trash/")));
}
[Fact]
public async Task An_edit_that_leaves_media_out_trashes_them()
{
var alice = await _host.Mastodon("alice");
var kept = await Upload(alice, "kept.jpg");
var dropped = await Upload(alice, "dropped.jpg");
var status = (await alice.Client.Post("/api/v1/statuses", ("status", "two pictures"), ("media_ids[]", kept), ("media_ids[]", dropped))).Ok();
var urls = status.Body["media_attachments"]!.AsArray().ToDictionary(m => m!["id"]!.GetValue<string>(), m => m!["url"]!.GetValue<string>());
(await alice.Client.Put($"/api/v1/statuses/{status.Body.Text("id")}", ("status", "one picture"), ("media_ids[]", kept))).Ok();
Assert.Equal(HttpStatusCode.OK, await Served(urls[kept]));
Assert.Equal(HttpStatusCode.NotFound, await Served(urls[dropped]));
Assert.Null((await DB.Default.Find<MediaAttachment>().OneAsync(kept, Token)).TrashedAt);
Assert.Equal("edited out", (await DB.Default.Find<MediaAttachment>().OneAsync(dropped, Token)).TrashReason);
}
// a posted picture's description and focus change by editing its post (media_attributes), not by PUT, as on Mastodon
[Fact]
public async Task An_edit_describes_its_media_again_and_keeps_what_it_is_not_sent()
{
var alice = await _host.Mastodon("alice");
var id = await Upload(alice, "a.jpg");
var status = (await alice.Client.Post("/api/v1/statuses", ("status", "described later"), ("media_ids[]", id),
("spoiler_text", "a warning"), ("sensitive", "true"), ("language", "it"))).Ok();
var statusId = status.Body.Text("id");
Assert.Equal(HttpStatusCode.NotFound, (await alice.Client.Put($"/api/v1/media/{id}", ("description", "too late"))).Status);
var edited = (await alice.Client.Json(HttpMethod.Put, $"/api/v1/statuses/{statusId}", new System.Text.Json.Nodes.JsonObject
{
["status"] = "described now",
["spoiler_text"] = "a warning",
["sensitive"] = true,
["media_ids"] = new System.Text.Json.Nodes.JsonArray(id),
["media_attributes"] = new System.Text.Json.Nodes.JsonArray(new System.Text.Json.Nodes.JsonObject { ["id"] = id, ["description"] = "a red square", ["focus"] = "0.5,0.5" })
})).Ok();
Assert.Equal("a red square", edited.Body["media_attachments"]![0]!.Text("description"));
Assert.Equal(0.5, edited.Body["media_attachments"]![0]!["meta"]!["focus"]!["x"]!.GetValue<double>());
Assert.Equal("a warning", edited.Body.Text("spoiler_text"));
Assert.True(edited.Body["sensitive"]!.GetValue<bool>());
Assert.Equal("it", edited.Body.Text("language"));
Assert.Equal("a red square", (await DB.Default.Find<MediaAttachment>().OneAsync(id, Token)).Description);
// an edit that doesn't send media keeps them, described as they were; an empty media_ids takes them away
var kept = (await alice.Client.Put($"/api/v1/statuses/{statusId}", ("status", "media kept"))).Ok();
Assert.Equal("a red square", kept.Body["media_attachments"]![0]!.Text("description"));
var bare = (await alice.Client.Json(HttpMethod.Put, $"/api/v1/statuses/{statusId}", new System.Text.Json.Nodes.JsonObject
{
["status"] = "no picture now",
["media_ids"] = new System.Text.Json.Nodes.JsonArray()
})).Ok();
Assert.Empty(bare.Body["media_attachments"]!.AsArray());
Assert.Equal("edited out", (await DB.Default.Find<MediaAttachment>().OneAsync(id, Token)).TrashReason);
}
[Fact]
public async Task A_replaced_avatar_is_trashed_and_a_picture_that_is_not_one_is_refused()
{
var alice = await _host.Mastodon("alice");
async Task<ApiAnswer> Picture(byte[] bytes, string type, string name) =>
await alice.Client.Exchange(new HttpRequestMessage(HttpMethod.Patch, "/api/v1/accounts/update_credentials")
{
Content = MastodonHelpers.Multipart(("avatar", bytes, type, name))
});
var first = (await Picture(MastodonHelpers.JpegWithMetadata(300, 300), "image/jpeg", "one.jpg")).Ok().Body.Text("avatar");
var second = (await Picture(MastodonHelpers.JpegWithMetadata(320, 300), "image/jpeg", "two.jpg")).Ok().Body.Text("avatar");
Assert.NotEqual(first, second);
Assert.Equal(HttpStatusCode.NotFound, await Served(first));
Assert.Equal(HttpStatusCode.OK, await Served(second));
var row = await DB.Default.Find<MediaAttachment>().Match(m => m.ProfileOfAvatarId == alice.Persona.Id && m.Kind == "avatar" && m.TrashedAt == null).ExecuteSingleAsync(Token);
Assert.EndsWith(row.FilePath, second);
// not an image: 422, and the avatar stays as it was
var refused = await Picture("<svg xmlns='http://www.w3.org/2000/svg'/>"u8.ToArray(), "image/svg+xml", "x.svg");
Assert.Equal(HttpStatusCode.UnprocessableEntity, refused.Status);
Assert.Equal(second, (await alice.Client.Get("/api/v1/accounts/verify_credentials")).Ok().Body.Text("avatar"));
}
// Mastodon's DELETE /api/v1/profile/avatar and /header: the picture goes, the default shows again
[Fact]
public async Task A_picture_is_removed_and_the_default_shows_again()
{
var alice = await _host.Mastodon($"pic{Guid.NewGuid():N}"[..12]);
var form = MastodonHelpers.Multipart(("avatar", MastodonHelpers.JpegWithMetadata(200, 200), "image/jpeg", "me.jpg"),
("header", MastodonHelpers.JpegWithMetadata(900, 300), "image/jpeg", "top.jpg"));
var set = (await alice.Client.Exchange(new HttpRequestMessage(HttpMethod.Patch, "/api/v1/accounts/update_credentials") { Content = form })).Ok();
var avatar = set.Body.Text("avatar");
var header = set.Body.Text("header");
var removed = (await alice.Client.Delete("/api/v1/profile/avatar")).Ok();
Assert.Contains("missing", removed.Body.Text("avatar"));
Assert.Equal(header, removed.Body.Text("header"));
Assert.Equal(HttpStatusCode.NotFound, await Served(avatar));
Assert.Equal(HttpStatusCode.OK, await Served(header));
Assert.Contains("missing", (await alice.Client.Delete("/api/v1/profile/header")).Ok().Body.Text("header"));
Assert.Equal(HttpStatusCode.NotFound, await Served(header));
Assert.False(await DB.Default.Find<MediaAttachment>().Match(m => m.ProfileOfAvatarId == alice.Persona.Id && m.TrashedAt == null).ExecuteAnyAsync(Token));
}
[Fact]
public async Task A_removed_roots_media_and_pictures_are_trashed_and_its_scheduled_posts_dropped()
{
var gone = await _host.Mastodon($"gone{Guid.NewGuid():N}"[..12]);
var posted = await Upload(gone, "posted.jpg");
var url = (await gone.Client.Post("/api/v1/statuses", ("status", "soon gone"), ("media_ids[]", posted))).Ok()
.Body["media_attachments"]![0]!["url"]!.GetValue<string>();
var held = await Upload(gone, "held.jpg");
(await gone.Client.Post("/api/v1/statuses", ("status", "later"), ("media_ids[]", held),
("scheduled_at", DateTime.UtcNow.AddHours(2).ToString("O")))).Ok();
var avatar = (await gone.Client.Exchange(new HttpRequestMessage(HttpMethod.Patch, "/api/v1/accounts/update_credentials")
{
Content = MastodonHelpers.Multipart(("avatar", MastodonHelpers.JpegWithMetadata(200, 200), "image/jpeg", "me.jpg"))
})).Ok().Body.Text("avatar");
Assert.True(await _host.Get<IRootRemoval>().Remove(gone.Persona.Root.Id, Token));
Assert.Equal(HttpStatusCode.NotFound, await Served(url));
Assert.Equal(HttpStatusCode.NotFound, await Served(avatar));
Assert.False(await DB.Default.Find<MediaAttachment>().Match(m => m.OwnerAvatarId == gone.Persona.Id && m.TrashedAt == null).ExecuteAnyAsync(Token));
Assert.False(await DB.Default.Find<Models.Social.ScheduledStatus>().Match(s => s.AvatarId == gone.Persona.Id).ExecuteAnyAsync(Token));
}
// what a login's media take: every persona's uploads and pictures, trashed ones no longer
[Fact]
public async Task A_logins_storage_counts_what_its_personas_keep()
{
var alice = await _host.Mastodon($"store{Guid.NewGuid():N}"[..12]);
using var root = _host.As(alice.Persona.Root.Jwt);
async Task<System.Text.Json.Nodes.JsonObject> Storage() => await (await root.GetAsync("/clientapi/user/storage", Token)).JsonBody();
Assert.Equal(0, Storage().Result["mediaBytes"]!.GetValue<long>());
var id = await Upload(alice, "a.jpg");
var stored = await Storage();
Assert.True(stored["mediaBytes"]!.GetValue<long>() > 0);
Assert.Equal(1, stored["mediaFiles"]!.GetValue<long>());
Assert.Null(stored["quotaBytes"]);
await _host.Get<IMediaService>().Trash(m => m.ID == id, "test", Token);
Assert.Equal(0, (await Storage())["mediaFiles"]!.GetValue<long>());
}
[Fact]
public async Task The_audit_adopts_todays_pictures_and_trashes_what_nothing_holds()
{
var media = _host.Get<IMediaService>();
var bob = await _host.Mastodon($"bob{Guid.NewGuid():N}"[..12]);
// a picture shown from before pictures had rows, and a leftover nothing holds
var picture = $"2020/01/{Guid.NewGuid():N}.jpg";
var leftover = $"2020/01/{Guid.NewGuid():N}.jpg";
foreach (var relative in new[] { picture, leftover })
{
Directory.CreateDirectory(Path.GetDirectoryName(Path.Combine(media.Root, relative))!);
await File.WriteAllBytesAsync(Path.Combine(media.Root, relative), new byte[] { 1, 2, 3 }, Token);
}
await DB.Default.Update<Models.User.Avatar>().MatchID(bob.Persona.Id).Modify(a => a.PictureURL, media.Url(picture)).ExecuteAsync(Token);
var looked = await MediaAudit.Run(media, fix: false, Token);
Assert.True(looked.Adopted >= 1);
Assert.True(looked.Unheld >= 1);
Assert.True(File.Exists(Path.Combine(media.Root, leftover)));
await MediaAudit.Run(media, fix: true, Token);
Assert.True(await DB.Default.Find<MediaAttachment>().Match(m => m.ProfileOfAvatarId == bob.Persona.Id && m.FilePath == picture && m.TrashedAt == null).ExecuteAnyAsync(Token));
Assert.Equal(HttpStatusCode.OK, await Served(media.Url(picture)));
Assert.Equal(HttpStatusCode.NotFound, await Served(media.Url(leftover)));
Assert.True(File.Exists(Path.Combine(media.TrashRoot, leftover)));
Assert.True(await DB.Default.Find<MediaAttachment>().Match(m => m.FilePath == leftover && m.TrashedAt != null).ExecuteAnyAsync(Token));
}
}
}
+346
View File
@@ -0,0 +1,346 @@
using MongoDB.Entities;
using PrivaPub.Models.Jobs;
using PrivaPub.Models.Media;
using PrivaPub.Models.Social;
using PrivaPub.Models.User;
using PrivaPub.Tests.Support;
using PrivaPub.Tests.Support.Host;
using System.IO.Compression;
using System.Net;
using System.Net.Http.Json;
using System.Text;
using System.Text.Json.Nodes;
using PostEntity = PrivaPub.Models.Post.Post;
namespace PrivaPub.Tests.Http
{
// A persona's archive (owner decision 2026-10-07): exported in Mastodon's layout with nothing of its root, its siblings
// or its keys; imported back home without bringing anything back, or into another persona as posts nobody is told of,
// once; Mastodon's own archives read too; and hostile ones refused.
[Trait("Category", "Integration")]
public sealed class PersonaArchiveTests : IAsyncLifetime
{
PrivaPubHost _host;
static CancellationToken Token => TestContext.Current.CancellationToken;
public async ValueTask InitializeAsync()
{
Assert.SkipUnless(MongoFixture.Enabled, MongoFixture.Skip);
_host = await PrivaPubHost.Shared();
}
public ValueTask DisposeAsync() => ValueTask.CompletedTask;
static string Base(Persona persona) => $"/clientapi/persona/{persona.Id}/archive";
async Task<JsonObject> Status(Persona persona)
{
using var client = _host.As(persona.Root.Jwt);
return (await client.GetFromJsonAsync<JsonObject>(Base(persona), Token))!;
}
async Task<byte[]> Exported(Persona persona)
{
using var client = _host.As(persona.Root.Jwt);
Assert.Equal(HttpStatusCode.Accepted, (await client.PostAsync(Base(persona), null, Token)).StatusCode);
await _host.Run(j => j.Kind == JobKind.ExportArchive && j.Payload == persona.Id, Token);
Assert.Equal("ready", (await Status(persona))["exportState"]!.GetValue<string>());
var ticket = (await (await client.PostAsync(Base(persona) + "/ticket", null, Token)).Content.ReadFromJsonAsync<JsonObject>(Token))!;
using var anonymous = _host.Client();
var response = await anonymous.GetAsync(ticket["url"]!.GetValue<string>(), Token);
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
return await response.Content.ReadAsByteArrayAsync(Token);
}
// uploaded in two pieces, then imported with the parts asked for, the job run: the import's state
async Task<(HttpStatusCode Status, JsonObject State)> Imported(Persona persona, byte[] archive, params string[] parts)
{
using var client = _host.As(persona.Root.Jwt);
var upload = (await (await client.PostAsync("/clientapi/persona/archive/uploads", null, Token)).Content.ReadFromJsonAsync<JsonObject>(Token))!["id"]!.GetValue<string>();
var half = archive.Length / 2;
Assert.Equal(HttpStatusCode.OK, (await client.PutAsync($"/clientapi/persona/archive/uploads/{upload}?offset=0", new ByteArrayContent(archive[..half]), Token)).StatusCode);
Assert.Equal(HttpStatusCode.OK, (await client.PutAsync($"/clientapi/persona/archive/uploads/{upload}?offset={half}", new ByteArrayContent(archive[half..]), Token)).StatusCode);
var asked = await client.PostAsJsonAsync(Base(persona) + "/import", new { uploadId = upload, parts }, Token);
if (asked.StatusCode != HttpStatusCode.Accepted)
return (asked.StatusCode, default);
await _host.Run(j => j.Kind == JobKind.ImportArchive && j.Payload == persona.Id, Token);
return (asked.StatusCode, await Status(persona));
}
static int Count(JsonObject state, string what) => state["importCounts"]?[what]?.GetValue<int>() ?? 0;
static Dictionary<string, string> Texts(byte[] zip)
{
using var archive = new ZipArchive(new MemoryStream(zip));
return archive.Entries.Where(e => e.Name.EndsWith(".json") || e.Name.EndsWith(".csv"))
.ToDictionary(e => e.FullName, e => new StreamReader(e.Open()).ReadToEnd());
}
[Fact]
public async Task An_archive_holds_the_persona_and_nothing_of_its_root_its_siblings_or_its_keys()
{
var alice = await _host.Mastodon("alice");
var sibling = await _host.Persona(alice.Persona.Root, "sibling");
var bob = await _host.Mastodon("bob");
var picture = (await alice.Client.Exchange(new HttpRequestMessage(HttpMethod.Post, "/api/v2/media")
{
Content = MastodonHelpers.Multipart(("file", MastodonHelpers.JpegWithMetadata(64, 48), "image/jpeg", "dot.jpg"))
})).Ok().Body.Text("id");
var first = await alice.Status("a public post with a picture #archives", ("media_ids[]", picture));
await alice.Status("for my followers only", ("visibility", "private"));
await alice.Status("a reply to myself", ("in_reply_to_id", first.Text("id")));
await alice.Client.Post($"/api/v1/statuses/{first.Text("id")}/bookmark");
await alice.Client.Post($"/api/v1/statuses/{first.Text("id")}/pin");
await alice.Client.Post($"/api/v1/accounts/{bob.Persona.Id}/block");
await alice.Client.Post("/api/v1/domain_blocks", ("domain", "spam.example"));
await alice.Client.Post("/api/v1/tags/archives/follow");
var zip = await Exported(alice.Persona);
var texts = Texts(zip);
foreach (var file in new[] { "actor.json", "outbox.json", "likes.json", "bookmarks.json", "following_accounts.csv", "blocked_accounts.csv",
"muted_accounts.csv", "blocked_domains.csv", "lists.csv", "bookmarks.csv", "privapub/archive.json", "privapub/pins.json",
"privapub/followed_tags.json", "privapub/filters.json", "privapub/notification_policy.json" })
Assert.True(texts.ContainsKey(file), $"{file} is missing");
using (var archive = new ZipArchive(new MemoryStream(zip)))
Assert.Contains(archive.Entries, e => e.FullName.StartsWith("media_attachments/files/", StringComparison.Ordinal));
var outbox = JsonNode.Parse(texts["outbox.json"])!["orderedItems"]!.AsArray();
Assert.Equal(3, outbox.Count);
Assert.Contains(outbox, i => i!["object"]!["attachment"]?[0]?["url"]?.GetValue<string>().StartsWith("/media_attachments/files/") == true);
Assert.Contains(bob.Persona.UserName, texts["blocked_accounts.csv"]);
Assert.Contains("spam.example", texts["blocked_domains.csv"]);
Assert.Contains(first.Text("uri"), texts["privapub/pins.json"]);
var avatar = await DB.Default.Find<Avatar>().MatchID(alice.Persona.Id).ExecuteFirstAsync(Token);
var root = await DB.Default.Find<RootUser>().MatchID(alice.Persona.Root.Id).ExecuteFirstAsync(Token);
var all = string.Concat(texts.Values);
Assert.DoesNotContain(avatar.PrivateKey.Split('\n')[1], all);
Assert.DoesNotContain(avatar.SigningKey, all);
Assert.DoesNotContain(root.UserName, all);
Assert.DoesNotContain(root.HashedPassword, all);
Assert.DoesNotContain(sibling.UserName, all);
Assert.DoesNotContain(alice.Token, all);
}
[Fact]
public async Task Back_home_an_archive_brings_nothing_back()
{
var alice = await _host.Mastodon("alice");
await alice.Status("kept");
var regretted = await alice.Status("regretted");
var zip = await Exported(alice.Persona);
await alice.Client.Delete($"/api/v1/statuses/{regretted.Text("id")}");
var before = await DB.Default.CountAsync<PostEntity>(p => p.GroupUserId == alice.Persona.Id, Token);
var (status, state) = await Imported(alice.Persona, zip, "posts", "pins");
Assert.Equal(HttpStatusCode.Accepted, status);
Assert.Equal("done", state["importState"]!.GetValue<string>());
Assert.True(state["importSameActor"]!.GetValue<bool>());
Assert.Equal((0, 1, 1), (Count(state, "posts"), Count(state, "posts already here"), Count(state, "posts deleted here")));
Assert.Equal(before, await DB.Default.CountAsync<PostEntity>(p => p.GroupUserId == alice.Persona.Id, Token));
Assert.NotNull((await DB.Default.Find<PostEntity>().MatchID(regretted.Text("id")).ExecuteFirstAsync(Token)).DeletedAt);
}
[Fact]
public async Task Into_another_persona_posts_arrive_once_and_nobody_is_told()
{
var alice = await _host.Mastodon("alice");
var follower = await _host.Mastodon("follower");
var picture = (await alice.Client.Exchange(new HttpRequestMessage(HttpMethod.Post, "/api/v2/media")
{
Content = MastodonHelpers.Multipart(("file", MastodonHelpers.JpegWithMetadata(64, 48), "image/jpeg", "dot.jpg"))
})).Ok().Body.Text("id");
var first = await alice.Status("the first words #moving", ("media_ids[]", picture));
await alice.Status("and a reply to them", ("in_reply_to_id", first.Text("id")));
await _host.Located(alice, "here and nowhere else");
var zip = await Exported(alice.Persona);
var carol = await _host.Mastodon("carol");
await follower.Client.Post($"/api/v1/accounts/{carol.Persona.Id}/follow");
var actor = $"/peasants/{carol.Persona.UserName}";
var jobs = await DB.Default.CountAsync<Job>(j => j.Kind == JobKind.Deliver && j.Payload.Contains(actor), Token);
var (_, state) = await Imported(carol.Persona, zip, "posts", "pins", "tags", "located");
Assert.Equal("done", state["importState"]!.GetValue<string>());
Assert.False(state["importSameActor"]!.GetValue<bool>());
Assert.Equal(2, Count(state, "posts"));
Assert.Equal(1, Count(state, "media"));
Assert.Equal(1, Count(state, "located posts"));
var copies = await DB.Default.Find<PostEntity>().Match(p => p.GroupUserId == carol.Persona.Id && p.Visibility != Models.Post.PostVisibility.LocalGeo).ExecuteAsync(Token);
Assert.Equal(2, copies.Count);
var copy = copies.Single(p => p.ImportedFromURI == first.Text("uri"));
Assert.StartsWith($"{PrivaPubHost.Base}/peasants/{carol.Persona.UserName}/scribbles/", copy.ObjectURI);
Assert.Equal(["moving"], copy.Tags);
Assert.Single(copy.Media);
Assert.Equal(carol.Persona.Id, (await DB.Default.Find<MediaAttachment>().MatchID(copy.Media[0].AttachmentId).ExecuteFirstAsync(Token)).OwnerAvatarId);
var reply = copies.Single(p => p.ID != copy.ID);
Assert.Equal(copy.ID, reply.AnsweringToPostId);
Assert.Equal(copy.ObjectURI + "/context", reply.ContextURI);
// nobody told: no delivery, no home, no notification; on the profile all the same
Assert.Equal(jobs, await DB.Default.CountAsync<Job>(j => j.Kind == JobKind.Deliver && j.Payload.Contains(actor), Token));
var ids = copies.Select(p => p.ID).ToList();
Assert.False(await DB.Default.Find<TimelineEntry>().Match(e => ids.Contains(e.PostId)).ExecuteAnyAsync(Token));
Assert.Contains((await carol.Client.Get($"/api/v1/accounts/{carol.Persona.Id}/statuses")).Ok().Body!.AsArray(), s => s!.Text("id") == copy.ID);
// twice changes nothing
var (_, again) = await Imported(carol.Persona, zip, "posts", "located");
Assert.Equal((0, 2, 1), (Count(again, "posts"), Count(again, "posts already here"), Count(again, "located posts already here")));
Assert.Equal(3, await DB.Default.CountAsync<PostEntity>(p => p.GroupUserId == carol.Persona.Id, Token));
}
// Mastodon's own archive: its posts, not its boosts nor its direct messages
[Fact]
public async Task A_mastodon_archive_is_read()
{
var dave = await _host.Mastodon("dave");
const string actor = "https://mastodon.example/users/dave";
const string Public = "https://www.w3.org/ns/activitystreams#Public";
var outbox = new JsonObject
{
["@context"] = "https://www.w3.org/ns/activitystreams",
["id"] = "outbox.json",
["type"] = "OrderedCollection",
["orderedItems"] = new JsonArray(
Create(actor, "1", "<p>hello from <b>Mastodon</b><script>alert(1)</script></p>", [Public], [actor + "/followers"], "2023-04-01T10:00:00Z"),
Create(actor, "2", "<p>only for one</p>", ["https://elsewhere.example/users/x"], [], "2023-04-02T10:00:00Z"),
new JsonObject { ["type"] = "Announce", ["actor"] = actor, ["object"] = "https://elsewhere.example/notes/9" })
};
var zip = Zip(("actor.json", new JsonObject { ["id"] = actor, ["type"] = "Person", ["followers"] = actor + "/followers" }.ToJsonString()),
("outbox.json", outbox.ToJsonString()));
var (_, state) = await Imported(dave.Persona, zip, "posts");
Assert.Equal("done", state["importState"]!.GetValue<string>());
Assert.Equal((1, 1, 1), (Count(state, "posts"), Count(state, "boosts skipped"), Count(state, "direct or circle posts skipped")));
var post = await DB.Default.Find<PostEntity>().Match(p => p.GroupUserId == dave.Persona.Id).ExecuteFirstAsync(Token);
Assert.Equal(actor + "/statuses/1", post.ImportedFromURI);
Assert.Equal(new DateTime(2023, 4, 1, 10, 0, 0, DateTimeKind.Utc), post.CreationDate);
Assert.DoesNotContain("<script", post.ContentHtml);
Assert.Contains("Mastodon", post.ContentHtml);
}
static JsonObject Create(string actor, string id, string content, string[] to, string[] cc, string published) => new()
{
["type"] = "Create",
["actor"] = actor,
["object"] = new JsonObject
{
["id"] = $"{actor}/statuses/{id}",
["type"] = "Note",
["attributedTo"] = actor,
["content"] = content,
["published"] = published,
["to"] = new JsonArray(to.Select(t => (JsonNode)t).ToArray()),
["cc"] = new JsonArray(cc.Select(c => (JsonNode)c).ToArray())
}
};
static byte[] Zip(params (string Name, string Text)[] files)
{
var stream = new MemoryStream();
using (var archive = new ZipArchive(stream, ZipArchiveMode.Create, leaveOpen: true))
foreach (var (name, text) in files)
{
using var entry = archive.CreateEntry(name).Open();
entry.Write(Encoding.UTF8.GetBytes(text));
}
return stream.ToArray();
}
[Theory]
[InlineData("escape")]
[InlineData("twice")]
[InlineData("bomb")]
[InlineData("link")]
[InlineData("junk")]
public async Task A_hostile_archive_is_refused(string how)
{
var erin = await _host.Mastodon("erin");
var actorJson = new JsonObject { ["id"] = "https://elsewhere.example/users/erin" }.ToJsonString();
var zip = how switch
{
"escape" => Zip(("actor.json", actorJson), ("../escape.txt", "x")),
"twice" => Zip(("actor.json", actorJson), ("outbox.json", "{}"), ("outbox.json", "{}")),
"bomb" => Zip(("actor.json", actorJson), ("outbox.json", new string(' ', 30 * 1024 * 1024))),
"link" => Link(actorJson),
_ => "not a zip at all"u8.ToArray()
};
var (status, _) = await Imported(erin.Persona, zip, "posts");
Assert.Equal(HttpStatusCode.UnprocessableEntity, status);
Assert.False(await DB.Default.Find<PostEntity>().Match(p => p.GroupUserId == erin.Persona.Id).ExecuteAnyAsync(Token));
}
// a symbolic link, as zip on Unix stores one
static byte[] Link(string actorJson)
{
var stream = new MemoryStream();
using (var archive = new ZipArchive(stream, ZipArchiveMode.Create, leaveOpen: true))
{
using (var actor = archive.CreateEntry("actor.json").Open())
actor.Write(Encoding.UTF8.GetBytes(actorJson));
var link = archive.CreateEntry("media_attachments/files/passwd");
link.ExternalAttributes = unchecked((int)(0xA1FFu << 16));
using var target = link.Open();
target.Write("/etc/passwd"u8);
}
return stream.ToArray();
}
[Fact]
public async Task An_outbox_item_too_large_stops_the_import()
{
var frank = await _host.Mastodon("frank");
const string actor = "https://elsewhere.example/users/frank";
// (random, so it is no compression bomb: that is refused before the import starts)
var words = Convert.ToBase64String(System.Security.Cryptography.RandomNumberGenerator.GetBytes(1536 * 1024));
var huge = Create(actor, "1", words, ["https://www.w3.org/ns/activitystreams#Public"], [], "2024-01-01T00:00:00Z");
var zip = Zip(("actor.json", new JsonObject { ["id"] = actor }.ToJsonString()),
("outbox.json", new JsonObject { ["orderedItems"] = new JsonArray(huge) }.ToJsonString()));
var (_, state) = await Imported(frank.Persona, zip, "posts");
Assert.Equal("failed", state["importState"]!.GetValue<string>());
Assert.Contains("larger than", state["importError"]!.GetValue<string>());
}
[Fact]
public async Task Only_the_personas_own_root_and_never_a_banned_one()
{
var grace = await _host.Mastodon("grace");
var stranger = await _host.SignUp("stranger");
using (var other = _host.As(stranger.Jwt))
Assert.Equal(HttpStatusCode.NotFound, (await other.GetAsync(Base(grace.Persona), Token)).StatusCode);
Assert.Equal(HttpStatusCode.OK, (await _host.As(grace.Persona.Root.Jwt).GetAsync(Base(grace.Persona), Token)).StatusCode);
await DB.Default.Update<RootUser>().MatchID(grace.Persona.Root.Id).Modify(u => u.IsBanned, true).ExecuteAsync(Token);
Assert.NotEqual(HttpStatusCode.OK, (await _host.As(grace.Persona.Root.Jwt).GetAsync(Base(grace.Persona), Token)).StatusCode);
}
[Fact]
public async Task An_import_asked_to_stop_stops()
{
var henry = await _host.Mastodon("henry");
const string actor = "https://elsewhere.example/users/henry";
var zip = Zip(("actor.json", new JsonObject { ["id"] = actor }.ToJsonString()),
("outbox.json", new JsonObject { ["orderedItems"] = new JsonArray(Create(actor, "1", "one", ["https://www.w3.org/ns/activitystreams#Public"], [], "2024-01-01T00:00:00Z")) }.ToJsonString()));
using var client = _host.As(henry.Persona.Root.Jwt);
var upload = (await (await client.PostAsync("/clientapi/persona/archive/uploads", null, Token)).Content.ReadFromJsonAsync<JsonObject>(Token))!["id"]!.GetValue<string>();
await client.PutAsync($"/clientapi/persona/archive/uploads/{upload}?offset=0", new ByteArrayContent(zip), Token);
Assert.Equal(HttpStatusCode.Accepted, (await client.PostAsJsonAsync(Base(henry.Persona) + "/import", new { uploadId = upload, parts = new[] { "posts" } }, Token)).StatusCode);
Assert.Equal(HttpStatusCode.Conflict, (await client.PostAsJsonAsync(Base(henry.Persona) + "/import", new { uploadId = upload, parts = new[] { "posts" } }, Token)).StatusCode);
Assert.Equal(HttpStatusCode.OK, (await client.PostAsync(Base(henry.Persona) + "/import/stop", null, Token)).StatusCode);
await _host.Run(j => j.Kind == JobKind.ImportArchive && j.Payload == henry.Persona.Id, Token);
Assert.Equal("stopped", (await Status(henry.Persona))["importState"]!.GetValue<string>());
Assert.False(await DB.Default.Find<PostEntity>().Match(p => p.GroupUserId == henry.Persona.Id).ExecuteAnyAsync(Token));
}
}
}
@@ -0,0 +1,287 @@
using MongoDB.Bson;
using MongoDB.Bson.IO;
using MongoDB.Driver;
using MongoDB.Entities;
using PrivaPub.Infrastructure.Backup;
using PrivaPub.Infrastructure.Cli;
using PrivaPub.Tests.Support;
using PrivaPub.Tests.Support.Host;
using System.IO.Compression;
namespace PrivaPub.Tests.Infrastructure
{
// The server's backup: every collection as it was, byte for byte, read at one instant on a replica set, without what
// belongs to the moment (the statistics salt above all), with the media rows' files linked, checkable and rotated.
// Each test backs up a database of its own; alone, since the maintenance lock is the server's one.
[Trait("Category", "Integration")]
[Xunit.Collection(nameof(Exclusive))]
public sealed class BackupTests : IAsyncLifetime
{
readonly string _scratch = Path.Combine(Path.GetTempPath(), $"privapub-backup-tests-{Guid.NewGuid():N}");
IMongoDatabase _database;
string Backups => Path.Combine(_scratch, "backups");
string Media => Path.Combine(_scratch, "media");
string Trash => Path.Combine(_scratch, "media-trash");
public async ValueTask InitializeAsync()
{
Assert.SkipUnless(MongoFixture.Enabled, MongoFixture.Skip);
await PrivaPubHost.Shared();
_database = DB.Default.Database().Client.GetDatabase($"PrivaPubBackup_{Guid.NewGuid():N}");
Directory.CreateDirectory(Media);
Directory.CreateDirectory(Trash);
}
public async ValueTask DisposeAsync()
{
if (_database != default)
await _database.Client.DropDatabaseAsync(_database.DatabaseNamespace.DatabaseName);
if (Directory.Exists(_scratch))
Directory.Delete(_scratch, recursive: true);
}
BackupContext Context(BackupOptions options = default) => new(_database, Backups, Media, Trash, "https://privapub.test", options ?? new BackupOptions());
static CancellationToken Token => TestContext.Current.CancellationToken;
static List<BsonDocument> Read(string file)
{
using var gzip = new GZipStream(File.OpenRead(file), CompressionMode.Decompress);
using var reader = new StreamReader(gzip);
var documents = new List<BsonDocument>();
while (reader.ReadLine() is { } line)
documents.Add(BsonDocument.Parse(line));
return documents;
}
async Task<List<byte[]>> Raw(string collection) =>
(await (await _database.GetCollection<RawBsonDocument>(collection).FindAsync(FilterDefinition<RawBsonDocument>.Empty, cancellationToken: Token)).ToListAsync(Token))
.Select(d =>
{
var bytes = new byte[d.Slice.Length];
d.Slice.GetBytes(0, bytes, 0, bytes.Length);
return bytes;
})
.ToList();
[Fact]
public async Task Every_document_comes_back_byte_for_byte_and_the_salt_never_leaves()
{
var odd = new BsonDocument
{
{ "_id", ObjectId.GenerateNewId() },
{ "Guid", new BsonBinaryData(Guid.NewGuid(), GuidRepresentation.Standard) },
{ "OldGuid", new BsonBinaryData(new byte[16], BsonBinarySubType.UuidLegacy) },
{ "Money", new BsonDecimal128(Decimal128.Parse("12345.678900")) },
{ "Long", new BsonInt64(long.MaxValue) },
{ "Int", 7 },
{ "Double", -0.0 },
{ "NaN", double.NaN },
{ "At", new BsonDateTime(new DateTime(2026, 10, 7, 3, 30, 0, 123, DateTimeKind.Utc)) },
{ "Stamp", new BsonTimestamp(1, 2) },
{ "Null", BsonNull.Value },
{ "Regex", new BsonRegularExpression("^a.b$", "i") },
{ "Nested", new BsonDocument { { "z", 1 }, { "a", new BsonArray { 1, "two", new BsonDocument("three", 3) } } } },
{ "Unicode", "città 🌍 \u0000 end" }
};
// an ObjectRecord as big as a remote's long post gets
var big = new BsonDocument { { "_id", ObjectId.GenerateNewId() }, { "Json", new string('x', 10 * 1024 * 1024) } };
await _database.GetCollection<BsonDocument>("Odd").InsertOneAsync(odd, cancellationToken: Token);
await _database.GetCollection<BsonDocument>("ObjectRecord").InsertOneAsync(big, cancellationToken: Token);
await _database.GetCollection<BsonDocument>("Odd").Indexes.CreateOneAsync(
new CreateIndexModel<BsonDocument>(Builders<BsonDocument>.IndexKeys.Ascending("At"), new CreateIndexOptions { Name = "at", Unique = true }), cancellationToken: Token);
await _database.GetCollection<BsonDocument>("InteractionSalt").InsertOneAsync(new BsonDocument("Salt", "never in a backup"), cancellationToken: Token);
await _database.GetCollection<BsonDocument>("Job").InsertOneAsync(new BsonDocument("Kind", "Deliver"), cancellationToken: Token);
await _database.GetCollection<BsonDocument>("_migration_history_").InsertManyAsync([
new BsonDocument { { "Number", 15 }, { "Name", "older" } }, new BsonDocument { { "Number", 16 }, { "Name", "focal points are finite" } }], cancellationToken: Token);
var (backup, error) = await ServerBackup.Create(Context(), "manual", dbOnly: false, Token);
Assert.Null(error);
var directory = Path.Combine(Backups, backup.Id);
Assert.False(Directory.Exists(directory + ServerBackup.PartialSuffix));
Assert.Equal(await Raw("Odd"), Read(Path.Combine(directory, "db", "Odd.jsonl.gz")).Select(d => d.ToBson()).ToList());
Assert.Equal(await Raw("ObjectRecord"), Read(Path.Combine(directory, "db", "ObjectRecord.jsonl.gz")).Select(d => d.ToBson()).ToList());
Assert.False(File.Exists(Path.Combine(directory, "db", "InteractionSalt.jsonl.gz")));
Assert.False(File.Exists(Path.Combine(directory, "db", "Job.jsonl.gz")));
foreach (var file in Directory.EnumerateFiles(directory, "*", SearchOption.AllDirectories))
Assert.DoesNotContain("never in a backup", await ReadAll(file));
Assert.Contains(backup.Manifest.Excluded, e => e.Name == "InteractionSalt");
var odds = backup.Manifest.Collections.Single(c => c.Name == "Odd");
Assert.Equal(1, odds.Count);
Assert.Contains(odds.Indexes, i => BsonDocument.Parse(i)["name"] == "at" && BsonDocument.Parse(i)["unique"].ToBoolean());
Assert.Equal((16, "focal points are finite"), (backup.Manifest.MigrationNumber, backup.Manifest.NewestMigration));
Assert.Equal("https://privapub.test", backup.Manifest.Host);
Assert.Equal(MongoFixture.Connection.Contains("directConnection=true"), backup.Manifest.Consistent);
Assert.Empty(await ServerBackup.Verify(Backups, backup.Id, Token));
if (!OperatingSystem.IsWindows())
{
Assert.Equal(UnixFileMode.UserRead | UnixFileMode.UserWrite | UnixFileMode.GroupRead, File.GetUnixFileMode(Path.Combine(directory, "db", "Odd.jsonl.gz")));
Assert.False(File.GetUnixFileMode(directory).HasFlag(UnixFileMode.OtherRead));
}
}
static async Task<string> ReadAll(string file)
{
if (!file.EndsWith(".gz", StringComparison.Ordinal))
return await File.ReadAllTextAsync(file, Token);
await using var gzip = new GZipStream(File.OpenRead(file), CompressionMode.Decompress);
using var reader = new StreamReader(gzip);
return await reader.ReadToEndAsync(Token);
}
[Fact]
public async Task Media_rows_files_are_linked_from_the_live_directory_or_the_trash()
{
Directory.CreateDirectory(Path.Combine(Media, "2026", "10"));
await File.WriteAllTextAsync(Path.Combine(Media, "2026", "10", "live.jpg"), "live", Token);
await File.WriteAllTextAsync(Path.Combine(Media, "2026", "10", "live-preview.jpg"), "preview", Token);
Directory.CreateDirectory(Path.Combine(Trash, "2026", "10"));
await File.WriteAllTextAsync(Path.Combine(Trash, "2026", "10", "moving.jpg"), "moving", Token);
await File.WriteAllTextAsync(Path.Combine(Media, "2026", "10", "trashed.jpg"), "trashed", Token);
await _database.GetCollection<BsonDocument>("MediaAttachment").InsertManyAsync([
new BsonDocument { { "FilePath", "2026/10/live.jpg" }, { "PreviewPath", "2026/10/live-preview.jpg" }, { "TrashedAt", BsonNull.Value } },
new BsonDocument { { "FilePath", "2026/10/moving.jpg" } },//a file the janitor moved while its row was being trashed
new BsonDocument { { "FilePath", "2026/10/gone.jpg" } },
new BsonDocument { { "FilePath", "2026/10/trashed.jpg" }, { "TrashedAt", DateTime.UtcNow } },
new BsonDocument { { "FilePath", "../../etc/passwd" } }], cancellationToken: Token);
var (backup, _) = await ServerBackup.Create(Context(), "manual", dbOnly: false, Token);
var media = Path.Combine(Backups, backup.Id, "media");
Assert.Equal("live", await File.ReadAllTextAsync(Path.Combine(media, "2026", "10", "live.jpg"), Token));
Assert.Equal("preview", await File.ReadAllTextAsync(Path.Combine(media, "2026", "10", "live-preview.jpg"), Token));
Assert.Equal("moving", await File.ReadAllTextAsync(Path.Combine(media, "2026", "10", "moving.jpg"), Token));
Assert.False(File.Exists(Path.Combine(media, "2026", "10", "trashed.jpg")));
Assert.Equal(["2026/10/live-preview.jpg", "2026/10/live.jpg", "2026/10/moving.jpg"], backup.Manifest.Media.List);
Assert.Equal((3, 1), (backup.Manifest.Media.Files, backup.Manifest.Media.Missing));
// a link, not a copy: the live file deleted, the backup's stays
File.Delete(Path.Combine(Media, "2026", "10", "live.jpg"));
Assert.Equal("live", await File.ReadAllTextAsync(Path.Combine(media, "2026", "10", "live.jpg"), Token));
// db-only lists them and links none
var (listed, _) = await ServerBackup.Create(Context(), "pre-deploy", dbOnly: true, Token);
Assert.False(Directory.Exists(Path.Combine(Backups, listed.Id, "media")));
Assert.Equal(["2026/10/live-preview.jpg", "2026/10/moving.jpg"], listed.Manifest.Media.List);
Assert.Empty(await ServerBackup.Verify(Backups, listed.Id, Token));
Assert.Empty(await ServerBackup.Verify(Backups, backup.Id, Token));
}
[Fact]
public async Task Verify_finds_an_altered_or_missing_file()
{
await _database.GetCollection<BsonDocument>("Post").InsertOneAsync(new BsonDocument("Content", "hello"), cancellationToken: Token);
await _database.GetCollection<BsonDocument>("Avatar").InsertOneAsync(new BsonDocument("UserName", "someone"), cancellationToken: Token);
var (backup, _) = await ServerBackup.Create(Context(), "manual", dbOnly: false, Token);
var db = Path.Combine(Backups, backup.Id, "db");
await File.AppendAllTextAsync(Path.Combine(db, "Post.jsonl.gz"), "tampered", Token);
File.Delete(Path.Combine(db, "Avatar.jsonl.gz"));
Assert.Equal(["Avatar: missing", "Post: altered"], (await ServerBackup.Verify(Backups, backup.Id, Token)).Order());
Assert.Equal(["no such backup, or no manifest"], await ServerBackup.Verify(Backups, "../" + backup.Id, Token));
}
[Fact]
public async Task One_backup_at_a_time()
{
await using (var held = await MaintenanceLock.Take("restore", Token))
{
Assert.NotNull(held);
var (backup, error) = await ServerBackup.Create(Context(), "manual", dbOnly: true, Token);
Assert.Null(backup);
Assert.Contains("already running", error);
Assert.Equal("restore", (await MaintenanceLock.Current(Token)).What);
}
Assert.Null(await MaintenanceLock.Current(Token));
Assert.NotNull((await ServerBackup.Create(Context(), "manual", dbOnly: true, Token)).Backup);
}
[Fact]
public async Task A_holder_that_died_is_taken_over()
{
await using var dead = await MaintenanceLock.Take("backup", Token);
await DB.Default.Update<MaintenanceLock>().Match(l => l.ID == MaintenanceLock.Name)
.Modify(l => l.Heartbeat, DateTime.UtcNow.AddMinutes(-3)).ExecuteAsync(Token);
Assert.Null(await MaintenanceLock.Current(Token));
await using var alive = await MaintenanceLock.Take("restore", Token);
Assert.NotNull(alive);
Assert.Equal("restore", (await MaintenanceLock.Current(Token)).What);
}
// the newest 7 nightly, the newest of each of the 4 weeks before, the newest 3 pre-deploy; manual ones kept
[Fact]
public void Rotation_keeps_days_weeks_and_the_last_deploys()
{
var today = new DateTime(2026, 10, 7, 3, 30, 0, DateTimeKind.Utc);
for (var day = 0; day < 60; day++)
Fake("nightly", today.AddDays(-day));
for (var deploy = 0; deploy < 5; deploy++)
Fake("pre-deploy", today.AddDays(-deploy).AddHours(5));
Fake("manual", today.AddYears(-1));
Directory.CreateDirectory(Path.Combine(Backups, "20260901-000000-nightly" + ServerBackup.PartialSuffix));
Directory.SetLastWriteTimeUtc(Path.Combine(Backups, "20260901-000000-nightly" + ServerBackup.PartialSuffix), today.AddDays(-30));
ServerBackup.Retain(Backups, new BackupOptions());
var kept = ServerBackup.List(Backups);
var nightly = kept.Where(b => b.Kind == "nightly").Select(b => b.CreatedAt).ToList();
Assert.Equal(11, nightly.Count);
Assert.Equal(Enumerable.Range(0, 7).Select(d => today.AddDays(-d)), nightly.Take(7));
Assert.Equal(4, nightly.Skip(7).Select(d => System.Globalization.ISOWeek.GetWeekOfYear(d)).Distinct().Count());
Assert.Equal(3, kept.Count(b => b.Kind == "pre-deploy"));
Assert.Single(kept, b => b.Kind == "manual");
Assert.Empty(Directory.EnumerateDirectories(Backups, "*" + ServerBackup.PartialSuffix));
}
void Fake(string kind, DateTime at)
{
var directory = Path.Combine(Backups, $"{at:yyyyMMdd-HHmmss}-{kind}");
Directory.CreateDirectory(Path.Combine(directory, "db"));
new ArchiveManifest { Kind = kind, CreatedAt = at }.Write(directory);
}
[Theory]
[InlineData("2026-10-07T03:29:00", "2026-10-06T03:31:00", false)]//yesterday's is the last one due
[InlineData("2026-10-07T03:31:00", "2026-10-06T03:31:00", true)]
[InlineData("2026-10-07T03:31:00", "2026-10-07T03:30:30", false)]
[InlineData("2026-10-07T01:00:00", "2026-10-05T03:31:00", true)]//missed last night: at once
public void A_nightly_backup_is_due_once_a_night(string now, string last, bool due) =>
Assert.Equal(due, BackupScheduler.IsDue(DateTime.Parse(now, null, System.Globalization.DateTimeStyles.AdjustToUniversal),
new TimeOnly(3, 30), [DateTime.Parse(last, null, System.Globalization.DateTimeStyles.AdjustToUniversal)]));
// the deploy's: the server's own database, through the configuration, without media links
[Fact]
public async Task The_deploy_backs_up_from_the_command_line()
{
var host = await PrivaPubHost.Shared();
var output = new StringWriter();
Assert.Equal(0, await AdminCommands.Run(["backup", "--kind", "pre-deploy", "--db-only"], host.Services, output: output));
var id = output.ToString().Split(':')[0];
Assert.EndsWith("-pre-deploy", id);
Assert.Equal(2, await AdminCommands.Run(["backup", "--kind", "nightly"], host.Services, output: TextWriter.Null));
output = new StringWriter();
Assert.Equal(0, await AdminCommands.Run(["backups"], host.Services, output: output));
Assert.StartsWith(id + "\tpre-deploy", output.ToString());
output = new StringWriter();
Assert.Equal(0, await AdminCommands.Run(["backup", "verify", id], host.Services, output: output));
Assert.Contains("whole", output.ToString());
var backups = host.Get<Backups>();
var manifest = backups.Find(id).Manifest;
Assert.Contains(manifest.Collections, c => c.Name == "Avatar");
Assert.DoesNotContain(manifest.Collections, c => c.Name is "InteractionSalt" or "Job" or "MaintenanceLock" or "openiddict.tokens" or "AppConfiguration");
Assert.Equal(ServerBackup.CodeMigration(), manifest.MigrationNumber);
Assert.True(backups.Delete(id));
}
}
}
@@ -1,3 +1,5 @@
using Microsoft.Extensions.Logging.Abstractions;
using MongoDB.Entities;
using PrivaPub.Infrastructure.Jobs;
@@ -74,6 +76,8 @@ namespace PrivaPub.Tests.Infrastructure
Assert.Equal(JobState.Pending, after.State);
Assert.True(after.RunAt > DateTime.UtcNow.AddSeconds(10));
// leased again for its third attempt (a finish only counts for the lease it was given)
await DB.Default.Update<Job>().MatchID(leased.ID).Modify(j => j.State, JobState.Running).Modify(j => j.LeaseOwner, leased.LeaseOwner).ExecuteAsync(token);
leased.Attempts = 3;
await queue.Finish(leased, JobOutcome.Retry("boom"), maxAttempts: 3, token);
Assert.Equal(JobState.Dead, (await DB.Default.Find<Job>().OneAsync(leased.ID, token)).State);
@@ -90,5 +94,100 @@ namespace PrivaPub.Tests.Infrastructure
Assert.Equal(JobState.Pending, (await DB.Default.Find<Job>().OneAsync(job.ID, token)).State);
}
// a handler running three times its lease keeps it: the reaper finds nothing to take back and it runs once
[Fact]
public async Task A_long_job_keeps_its_lease_and_runs_once()
{
var token = TestContext.Current.CancellationToken;
var kind = (JobKind)(2000 + Random.Shared.Next(100000));
var queue = new JobQueue(TimeSpan.FromSeconds(1.5), j => j.Kind == kind);
var handler = new SlowHandler(kind, TimeSpan.FromSeconds(4.5));
await queue.Enqueue(kind, "{}", default, default, token);
using var worker = new JobWorker(queue, new IJobHandler[] { handler }, NullLogger<JobWorker>.Instance);
await worker.StartAsync(token);
try
{
var deadline = DateTime.UtcNow.AddSeconds(20);
while (!await DB.Default.Find<Job>().Match(j => j.Kind == kind && j.State == JobState.Done).ExecuteAnyAsync(token))
{
Assert.True(DateTime.UtcNow < deadline, "the job never finished");
Assert.Equal(0, await queue.Reap(token));
await Task.Delay(250, token);
}
}
finally
{
await worker.StopAsync(CancellationToken.None);
}
Assert.Equal(1, handler.Started);
}
// a lease taken by another worker (reaped, then leased again) stops the handler, and its outcome is not recorded
[Fact]
public async Task A_lost_lease_stops_the_handler_and_drops_its_outcome()
{
var token = TestContext.Current.CancellationToken;
var kind = (JobKind)(2000 + Random.Shared.Next(100000));
var queue = new JobQueue(TimeSpan.FromSeconds(1.5), j => j.Kind == kind);
var handler = new SlowHandler(kind, TimeSpan.FromSeconds(30));
await queue.Enqueue(kind, "{}", default, default, token);
using var worker = new JobWorker(queue, new IJobHandler[] { handler }, NullLogger<JobWorker>.Instance);
await worker.StartAsync(token);
try
{
var deadline = DateTime.UtcNow.AddSeconds(20);
while (handler.Started == 0)
{
Assert.True(DateTime.UtcNow < deadline, "the job never started");
await Task.Delay(100, token);
}
await DB.Default.Update<Job>().Match(j => j.Kind == kind).Modify(j => j.LeaseOwner, "another worker").ExecuteAsync(token);
while (!handler.Cancelled)
{
Assert.True(DateTime.UtcNow < deadline, "the handler was never stopped");
await Task.Delay(100, token);
}
await Task.Delay(300, token);
}
finally
{
await worker.StopAsync(CancellationToken.None);
}
var job = await DB.Default.Find<Job>().Match(j => j.Kind == kind).ExecuteFirstAsync(token);
Assert.Equal(JobState.Running, job.State);
Assert.Equal("another worker", job.LeaseOwner);
}
sealed class SlowHandler(JobKind kind, TimeSpan takes) : IJobHandler
{
int _started;
public int Started => _started;
public bool Cancelled { get; private set; }
public JobKind Kind => kind;
public int Concurrency => 1;
public int MaxAttempts => 3;
public int PerHostLimit => 1;
public async Task<JobOutcome> Handle(Job job, CancellationToken token)
{
Interlocked.Increment(ref _started);
try
{
await Task.Delay(takes, token);
}
catch (OperationCanceledException)
{
Cancelled = true;
throw;
}
return JobOutcome.Done;
}
}
}
}
@@ -1,3 +1,4 @@
using PrivaPub.Models.Media;
using MongoDB.Bson;
using MongoDB.Driver;
using MongoDB.Entities;
@@ -22,6 +23,31 @@ namespace PrivaPub.Tests.Infrastructure
[Trait("Category", "Integration")]
public class MigrationTests
{
[Fact]
public async Task A_focal_point_that_is_not_a_number_is_removed()
{
Assert.SkipUnless(MongoFixture.Enabled, MongoFixture.Skip);
var token = TestContext.Current.CancellationToken;
var broken = new MediaAttachment { OwnerAvatarId = "a1", Focus = [float.NaN, 0f] };
var sound = new MediaAttachment { OwnerAvatarId = "a1", Focus = [0.5f, -0.5f] };
var post = new Post
{
ObjectURI = $"https://privapub.test/{Guid.NewGuid():N}",
Media = [new PostMedia { URL = "https://privapub.test/media/files/a.jpg", Focus = [float.PositiveInfinity, 0f] }, new PostMedia { URL = "https://privapub.test/media/files/b.jpg", Focus = [0.1f, 0.2f] }]
};
await DB.Default.SaveAsync(broken, token);
await DB.Default.SaveAsync(sound, token);
await DB.Default.SaveAsync(post, token);
await new _016_focal_points_are_finite().UpgradeAsync();
Assert.Null((await DB.Default.Find<MediaAttachment>().OneAsync(broken.ID, token)).Focus);
Assert.Equal(new[] { 0.5f, -0.5f }, (await DB.Default.Find<MediaAttachment>().OneAsync(sound.ID, token)).Focus);
var migrated = await DB.Default.Find<Post>().OneAsync(post.ID, token);
Assert.Null(migrated.Media[0].Focus);
Assert.Equal(new[] { 0.1f, 0.2f }, migrated.Media[1].Focus);
}
[Fact]
public async Task Stored_remote_content_is_sanitized_and_local_content_rendered()
{
@@ -0,0 +1,316 @@
using Microsoft.Extensions.Logging.Abstractions;
using MongoDB.Bson;
using MongoDB.Driver;
using MongoDB.Entities;
using PrivaPub.Infrastructure.Backup;
using PrivaPub.Infrastructure.Cli;
using PrivaPub.Tests.Support;
using PrivaPub.Tests.Support.Host;
namespace PrivaPub.Tests.Infrastructure
{
// A backup restored at boot: what was made since is lost, what protects is not (owner decision 2026-10-07). A server
// in a database of its own is backed up, changed, and restored; alone, since the maintenance lock is the server's one.
[Trait("Category", "Integration")]
[Xunit.Collection(nameof(Exclusive))]
public sealed class RestoreTests : IAsyncLifetime
{
const string Host = "https://privapub.test";
readonly string _scratch = Path.Combine(Path.GetTempPath(), $"privapub-restore-tests-{Guid.NewGuid():N}");
IMongoDatabase _database;
string Backups => Path.Combine(_scratch, "backups");
string Media => Path.Combine(_scratch, "media");
string Trash => Path.Combine(_scratch, "media-trash");
static CancellationToken Token => TestContext.Current.CancellationToken;
public async ValueTask InitializeAsync()
{
Assert.SkipUnless(MongoFixture.Enabled, MongoFixture.Skip);
await PrivaPubHost.Shared();
_database = DB.Default.Database().Client.GetDatabase($"PrivaPubRestore_{Guid.NewGuid():N}");
Directory.CreateDirectory(Media);
Directory.CreateDirectory(Trash);
}
public async ValueTask DisposeAsync()
{
if (_database != default)
await _database.Client.DropDatabaseAsync(_database.DatabaseNamespace.DatabaseName);
if (Directory.Exists(_scratch))
Directory.Delete(_scratch, recursive: true);
}
BackupContext Context(string host = Host) => new(_database, Backups, Media, Trash, host, new BackupOptions());
IMongoCollection<BsonDocument> C(string name) => _database.GetCollection<BsonDocument>(name);
async Task<BsonDocument> One(string collection, BsonDocument filter) =>
await (await C(collection).FindAsync(filter, cancellationToken: Token)).FirstOrDefaultAsync(Token);
async Task<List<BsonDocument>> All(string collection) =>
await (await C(collection).FindAsync(FilterDefinition<BsonDocument>.Empty, cancellationToken: Token)).ToListAsync(Token);
static BsonDocument Id(ObjectId id) => new("_id", id);
static Task Set(IMongoCollection<BsonDocument> collection, ObjectId id, BsonDocument fields) =>
collection.UpdateOneAsync(Id(id), new BsonDocument("$set", fields), cancellationToken: Token);
// the ids of the town this test builds
readonly ObjectId _alice = ObjectId.GenerateNewId(), _alicePersona = ObjectId.GenerateNewId();
readonly ObjectId _deletedPost = ObjectId.GenerateNewId(), _editedPost = ObjectId.GenerateNewId(), _laterPost = ObjectId.GenerateNewId();
readonly ObjectId _deletedMedia = ObjectId.GenerateNewId(), _laterMedia = ObjectId.GenerateNewId();
readonly ObjectId _bob = ObjectId.GenerateNewId(), _bobPersona = ObjectId.GenerateNewId();
static string Uri(ObjectId post) => $"{Host}/peasants/alice/posts/{post}";
async Task<BackupInfo> TownBackedUp()
{
await File.WriteAllTextAsync(Path.Combine(Media, "deleted.jpg"), "deleted", Token);
await C("RootUser").InsertOneAsync(new BsonDocument { { "_id", _alice }, { "UserName", "alice" }, { "HashedPassword", "old" }, { "SessionStamp", "s1" }, { "Policies", new BsonArray { "IsUser" } }, { "DeletedAt", BsonNull.Value } }, cancellationToken: Token);
await C("Avatar").InsertOneAsync(new BsonDocument { { "_id", _alicePersona }, { "UserName", "alice" }, { "SuspendedAt", BsonNull.Value }, { "DeletionAt", BsonNull.Value } }, cancellationToken: Token);
await C("RootToAvatar").InsertOneAsync(new BsonDocument { { "RootId", _alice.ToString() }, { "AvatarId", _alicePersona.ToString() } }, cancellationToken: Token);
await C("ReservedName").InsertOneAsync(new BsonDocument("Name", "alice"), cancellationToken: Token);
await C("Post").InsertManyAsync([
new BsonDocument { { "_id", _deletedPost }, { "GroupUserId", _alicePersona.ToString() }, { "IsFederatedCopy", false }, { "ObjectURI", Uri(_deletedPost) }, { "Text", "regret" },
{ "Media", new BsonArray { new BsonDocument("AttachmentId", _deletedMedia.ToString()) } }, { "DeletedAt", BsonNull.Value } },
new BsonDocument { { "_id", _editedPost }, { "GroupUserId", _alicePersona.ToString() }, { "IsFederatedCopy", false }, { "ObjectURI", Uri(_editedPost) }, { "Text", "first words" }, { "DeletedAt", BsonNull.Value } }
], cancellationToken: Token);
await C("TimelineEntry").InsertOneAsync(new BsonDocument { { "PostId", _deletedPost.ToString() }, { "ReblogOfPostId", BsonNull.Value } }, cancellationToken: Token);
await C("MediaAttachment").InsertOneAsync(new BsonDocument { { "_id", _deletedMedia }, { "PostId", _deletedPost.ToString() }, { "FilePath", "deleted.jpg" }, { "TrashedAt", BsonNull.Value } }, cancellationToken: Token);
await C("Follower").InsertOneAsync(new BsonDocument { { "LocalActorId", _alicePersona.ToString() }, { "ActorURI", "https://elsewhere.example/users/left" } }, cancellationToken: Token);
await C("RemoteInstance").InsertOneAsync(new BsonDocument { { "Host", "down.example" }, { "ConsecutiveFailures", 9 }, { "UnavailableUntil", DateTime.UtcNow.AddDays(1) } }, cancellationToken: Token);
await C("_migration_history_").InsertOneAsync(new BsonDocument { { "Number", 16 }, { "Name", "focal points are finite" } }, cancellationToken: Token);
await C("InteractionSalt").InsertOneAsync(new BsonDocument("Salt", "the backup's day"), cancellationToken: Token);
var (backup, error) = await ServerBackup.Create(Context(), "manual", dbOnly: false, Token);
Assert.Null(error);
return backup;
}
// what happens after the backup: protective acts, and things merely made or changed
async Task LifeGoesOn()
{
// alice regrets a post: deleted, its media trashed and moved out of what is served
await Set(C("Post"), _deletedPost, new BsonDocument { { "DeletedAt", DateTime.UtcNow }, { "Text", BsonNull.Value }, { "Media", new BsonArray() } });
await Set(C("MediaAttachment"), _deletedMedia, new BsonDocument { { "TrashedAt", DateTime.UtcNow }, { "TrashReason", "deleted" } });
File.Move(Path.Combine(Media, "deleted.jpg"), Path.Combine(Trash, "deleted.jpg"));
await C("TimelineEntry").DeleteManyAsync(new BsonDocument("PostId", _deletedPost.ToString()), Token);
// an edit, a new post with a picture: not protective
await Set(C("Post"), _editedPost, new BsonDocument("Text", "second words"));
await File.WriteAllTextAsync(Path.Combine(Media, "later.jpg"), "later", Token);
await C("Post").InsertOneAsync(new BsonDocument { { "_id", _laterPost }, { "GroupUserId", _alicePersona.ToString() }, { "IsFederatedCopy", false }, { "ObjectURI", Uri(_laterPost) }, { "Text", "later" }, { "DeletedAt", BsonNull.Value } }, cancellationToken: Token);
await C("MediaAttachment").InsertOneAsync(new BsonDocument { { "_id", _laterMedia }, { "PostId", _laterPost.ToString() }, { "FilePath", "later.jpg" }, { "TrashedAt", BsonNull.Value } }, cancellationToken: Token);
// a follower leaves, another comes
await C("Follower").DeleteManyAsync(FilterDefinition<BsonDocument>.Empty, Token);
await C("Follower").InsertOneAsync(new BsonDocument { { "LocalActorId", _alicePersona.ToString() }, { "ActorURI", "https://elsewhere.example/users/came" } }, cancellationToken: Token);
// alice blocks someone, the admin blocks a server, someone is reported, alice's persona is suspended, she changes
// her password
await C("Block").InsertOneAsync(new BsonDocument { { "AvatarId", _alicePersona.ToString() }, { "TargetActorURI", "https://elsewhere.example/users/troll" } }, cancellationToken: Token);
await C("DomainBlock").InsertOneAsync(new BsonDocument { { "Domain", "evil.example" }, { "Severity", "Suspend" } }, cancellationToken: Token);
await C("Report").InsertOneAsync(new BsonDocument("Comment", "harassment"), cancellationToken: Token);
await Set(C("Avatar"), _alicePersona, new BsonDocument("SuspendedAt", DateTime.UtcNow));
await Set(C("RootUser"), _alice, new BsonDocument("HashedPassword", "new"));
// bob arrives
await C("RootUser").InsertOneAsync(new BsonDocument { { "_id", _bob }, { "UserName", "bob" }, { "HashedPassword", "bob's" }, { "Policies", new BsonArray { "IsUser" } }, { "DeletedAt", BsonNull.Value } }, cancellationToken: Token);
await C("Avatar").InsertOneAsync(new BsonDocument { { "_id", _bobPersona }, { "UserName", "bob" }, { "DeletionAt", BsonNull.Value } }, cancellationToken: Token);
await C("RootToAvatar").InsertOneAsync(new BsonDocument { { "RootId", _bob.ToString() }, { "AvatarId", _bobPersona.ToString() } }, cancellationToken: Token);
await C("ReservedName").InsertOneAsync(new BsonDocument("Name", "bob"), cancellationToken: Token);
// sessions, a list made, today's salt, work queued
await C("openiddict.tokens").InsertOneAsync(new BsonDocument("subject", "alice"), cancellationToken: Token);
await C("PersonaList").InsertOneAsync(new BsonDocument { { "AvatarId", _alicePersona.ToString() }, { "Title", "friends" } }, cancellationToken: Token);
await C("InteractionSalt").DeleteManyAsync(FilterDefinition<BsonDocument>.Empty, Token);
await C("InteractionSalt").InsertOneAsync(new BsonDocument("Salt", "today's"), cancellationToken: Token);
await C("Job").InsertOneAsync(new BsonDocument("Kind", "Deliver"), cancellationToken: Token);
}
async Task AssertRestoredAndProtected(BackupInfo backup)
{
// lost: what was merely made or changed since
Assert.Equal("first words", (await One("Post", Id(_editedPost)))["Text"].AsString);
var later = await One("Post", Id(_laterPost));
Assert.False(later["DeletedAt"].IsBsonNull);
Assert.True(later["Text"].IsBsonNull);
Assert.Empty(await All("PersonaList"));
Assert.Equal(16, (await One("_migration_history_", new BsonDocument()))["Number"].ToInt32());
// kept: every protective act
var deleted = await One("Post", Id(_deletedPost));
Assert.False(deleted["DeletedAt"].IsBsonNull);
Assert.True(deleted["Text"].IsBsonNull);
Assert.Null(await One("TimelineEntry", new BsonDocument("PostId", _deletedPost.ToString())));
Assert.False((await One("MediaAttachment", Id(_deletedMedia)))["TrashedAt"].IsBsonNull);
Assert.Equal("restore: made after the backup", (await One("MediaAttachment", Id(_laterMedia)))["TrashReason"].AsString);
Assert.Equal(["https://elsewhere.example/users/came"], (await All("Follower")).Select(f => f["ActorURI"].AsString));
Assert.NotNull(await One("Block", new BsonDocument("TargetActorURI", "https://elsewhere.example/users/troll")));
Assert.NotNull(await One("DomainBlock", new BsonDocument("Domain", "evil.example")));
Assert.NotNull(await One("Report", new BsonDocument("Comment", "harassment")));
Assert.False((await One("Avatar", Id(_alicePersona)))["SuspendedAt"].IsBsonNull);
var alice = await One("RootUser", Id(_alice));
Assert.Equal("new", alice["HashedPassword"].AsString);
Assert.NotEqual("s1", alice["SessionStamp"].AsString);
// bob, made since: deleted, his name kept
var bob = await One("RootUser", Id(_bob));
Assert.Equal($"deleted-{_bob}", bob["UserName"].AsString);
Assert.True(bob["HashedPassword"].IsBsonNull);
Assert.False(bob["DeletedAt"].IsBsonNull);
Assert.False((await One("Avatar", Id(_bobPersona)))["DeletionAt"].IsBsonNull);
Assert.NotNull(await One("ReservedName", new BsonDocument("Name", "bob")));
// what a backup never holds is as it was; sessions end; circuits close
Assert.Equal("today's", (await One("InteractionSalt", new BsonDocument()))["Salt"].AsString);
Assert.Single(await All("Job"));
Assert.Empty(await All("openiddict.tokens"));
Assert.Equal(0, (await One("RemoteInstance", new BsonDocument("Host", "down.example")))["ConsecutiveFailures"].ToInt32());
// the picture of the deleted post came back from the backup, to be trashed again by the janitor
Assert.True(File.Exists(Path.Combine(Media, "deleted.jpg")));
Assert.Null(RestoreMarker.Read(Backups));
var record = await (await _database.GetCollection<RestoreRecord>(nameof(RestoreRecord)).FindAsync(r => r.Backup == backup.Id, cancellationToken: Token)).FirstAsync(Token);
Assert.Equal(["bob"], record.Report.RootsTombstoned);
Assert.Equal(["bob"], record.Report.PersonasTombstoned);
Assert.Equal(1, record.Report.PostsGone);
Assert.Equal(1, record.Report.PostsDeleted);
}
[Fact]
public async Task A_restore_loses_what_was_made_since_but_never_a_protective_act()
{
var backup = await TownBackedUp();
await LifeGoesOn();
Assert.Null(await ServerRestore.Request(Context(), backup.Id, "admin", Token));
Assert.Contains("already pending", await ServerRestore.Request(Context(), backup.Id, "admin", Token));
Assert.Equal(RestoreOutcome.Restored, await ServerRestore.ApplyPending(Context(), NullLogger.Instance, Token));
await AssertRestoredAndProtected(backup);
var before = ServerBackup.List(Backups).Single(b => b.Kind == "pre-restore");
Assert.Equal(before.Id, (await (await _database.GetCollection<RestoreRecord>(nameof(RestoreRecord)).FindAsync(FilterDefinition<RestoreRecord>.Empty, cancellationToken: Token)).FirstAsync(Token)).PreRestore);
}
// an attempt that died midway: the next one starts again from the backup, with the same pre-restore backup, never
// one of the half-restored database
[Fact]
public async Task A_restore_that_died_midway_converges_on_the_next_boot()
{
var backup = await TownBackedUp();
await LifeGoesOn();
Assert.Null(await ServerRestore.Request(Context(), backup.Id, "admin", Token));
Assert.Equal(RestoreOutcome.Restored, await ServerRestore.ApplyPending(Context(), NullLogger.Instance, Token));
var before = ServerBackup.List(Backups).Single(b => b.Kind == "pre-restore");
// as if the boot died after importing a collection, with the marker saying so
await C("Post").DeleteManyAsync(FilterDefinition<BsonDocument>.Empty, Token);
await C("Block").DeleteManyAsync(FilterDefinition<BsonDocument>.Empty, Token);
new RestoreMarker { Backup = backup.Id, PreRestore = before.Id, State = "running", Attempts = 1, RequestedBy = "admin" }.Write(Backups);
Assert.Equal(RestoreOutcome.Restored, await ServerRestore.ApplyPending(Context(), NullLogger.Instance, Token));
Assert.Single(ServerBackup.List(Backups), b => b.Kind == "pre-restore");
await AssertRestoredAndProtected(backup);
}
[Fact]
public async Task A_backup_from_elsewhere_from_a_newer_build_or_altered_is_refused()
{
var backup = await TownBackedUp();
Assert.Contains("not https://other.test", await ServerRestore.Request(Context("https://other.test"), backup.Id, "admin", Token));
Assert.Contains("no such backup", await ServerRestore.Request(Context(), "../" + backup.Id, "admin", Token));
var manifest = ArchiveManifest.Read(Path.Combine(Backups, backup.Id));
manifest.MigrationNumber = ServerBackup.CodeMigration() + 1;
manifest.Write(Path.Combine(Backups, backup.Id));
Assert.Contains("newer build", await ServerRestore.Request(Context(), backup.Id, "admin", Token));
manifest.MigrationNumber = 16;
manifest.Write(Path.Combine(Backups, backup.Id));
await File.AppendAllTextAsync(Path.Combine(Backups, backup.Id, "db", "Post.jsonl.gz"), "x", Token);
Assert.Contains("Post: altered", await ServerRestore.Request(Context(), backup.Id, "admin", Token));
Assert.Null(RestoreMarker.Read(Backups));
}
// refused at boot before anything changed (the backup went meanwhile): recorded, and the server boots as it was
[Fact]
public async Task A_restore_that_cannot_start_is_abandoned_and_changes_nothing()
{
var backup = await TownBackedUp();
await LifeGoesOn();
Assert.Null(await ServerRestore.Request(Context(), backup.Id, "admin", Token));
ServerBackup.Delete(Backups, backup.Id);
Assert.Equal(RestoreOutcome.Abandoned, await ServerRestore.ApplyPending(Context(), NullLogger.Instance, Token));
Assert.Equal("second words", (await One("Post", Id(_editedPost)))["Text"].AsString);
Assert.Null(RestoreMarker.Read(Backups));
Assert.DoesNotContain(ServerBackup.List(Backups), b => b.Kind == "pre-restore");
var record = await (await _database.GetCollection<RestoreRecord>(nameof(RestoreRecord)).FindAsync(FilterDefinition<RestoreRecord>.Empty, cancellationToken: Token)).FirstAsync(Token);
Assert.True(record.Abandoned);
Assert.Contains("no such backup", record.Error);
}
[Fact]
public async Task After_three_failures_it_waits_for_someone_to_look()
{
var backup = await TownBackedUp();
await LifeGoesOn();
new RestoreMarker { Backup = backup.Id, PreRestore = "gone", State = "running", Attempts = RestoreMarker.MaxAttempts, Error = "disk full" }.Write(Backups);
Assert.Equal(RestoreOutcome.GaveUp, await ServerRestore.ApplyPending(Context(), NullLogger.Instance, Token));
Assert.Equal("second words", (await One("Post", Id(_editedPost)))["Text"].AsString);
Assert.NotNull(RestoreMarker.Read(Backups));
await File.WriteAllTextAsync(RestoreMarker.PathIn(Backups), "{ not json", Token);
Assert.Equal(RestoreOutcome.GaveUp, await ServerRestore.ApplyPending(Context(), NullLogger.Instance, Token));
}
// a command waits while a restore does; asking how it goes doesn't
[Fact]
public async Task Commands_wait_for_a_restore_asked_for()
{
var host = await PrivaPubHost.Shared();
var backups = host.Get<Backups>();
new RestoreMarker { Backup = "20260101-000000-manual", RequestedBy = "cli" }.Write(backups.Root);
try
{
Assert.Equal(MaintenanceGate.TryLater, await MaintenanceGate.Enter(host.Services, ["promote", "someone"], Token));
Assert.Null(await MaintenanceGate.Enter(host.Services, ["restore", "--status"], Token));
var output = new StringWriter();
Assert.Equal(0, await AdminCommands.Run(["restore", "--status"], host.Services, output: output));
Assert.Contains("waiting: 20260101-000000-manual, pending", output.ToString());
Assert.Equal(1, await AdminCommands.Run(["restore", "20260101-000000-manual"], host.Services, output: TextWriter.Null));
}
finally
{
RestoreMarker.Clear(backups.Root);
}
}
// FEP-8fcf: for two weeks after a restore, no digests, and no follow undone for them
[Fact]
public async Task Followers_digests_rest_for_two_weeks_after_a_restore()
{
RestoreRecord.Forget();
Assert.False(await RestoreRecord.InFollowersGrace(Token));
var record = new RestoreRecord { Backup = "test", RestoredAt = DateTime.UtcNow.AddDays(-13) };
await DB.Default.SaveAsync(record, Token);
try
{
RestoreRecord.Forget();
Assert.True(await RestoreRecord.InFollowersGrace(Token));
await DB.Default.Update<RestoreRecord>().MatchID(record.ID).Modify(r => r.RestoredAt, DateTime.UtcNow.AddDays(-15)).ExecuteAsync(Token);
RestoreRecord.Forget();
Assert.False(await RestoreRecord.InFollowersGrace(Token));
}
finally
{
await DB.Default.DeleteAsync<RestoreRecord>(record.ID);
RestoreRecord.Forget();
}
}
}
}
@@ -0,0 +1,21 @@
using MongoDB.Entities;
using PrivaPub.Infrastructure.Data;
using PrivaPub.Tests.Support;
namespace PrivaPub.Tests.Infrastructure
{
[Trait("Category", "Integration")]
public sealed class TopologyTests
{
// tools/ci/with-test-mongod.sh starts a one-member replica set, as production's mongod is: the backup's snapshot reads
// are what the tests exercise
[Fact]
public async Task The_test_mongod_is_a_replica_set_like_productions()
{
Assert.SkipUnless(MongoFixture.Enabled, MongoFixture.Skip);
Assert.SkipUnless(MongoFixture.Connection.Contains("directConnection=true"), "a mongod not started by tools/ci/with-test-mongod.sh");
Assert.True(await MongoTopology.IsReplicaSet(DB.Default.Database(), TestContext.Current.CancellationToken));
}
}
}
+2
View File
@@ -71,6 +71,8 @@ namespace PrivaPub.Tests.Support
new LikeHandler(Db, Reactions),
new EmojiReactHandler(Db, Reactions),
new QuoteRequestHandler(Quotes),
new WarnHandler(Db, Local, Remote),
new ResolveHandler(Db, Local),
new DislikeHandler(Db),
new JoinHandler(Db, Local, Delivery),
new AnnounceHandler(Db, Local, RemotePosts, Fanout, Remote, Records, Quotes, relays: Relays),
+11 -3
View File
@@ -24,7 +24,8 @@ namespace PrivaPub.Tests.Support.Host
static readonly SemaphoreSlim Boot = new(1, 1);
static readonly Type[] Unwanted = { typeof(JobWorker), typeof(MediaJanitor), typeof(OAuthPruner), typeof(StatisticsSchedule),
typeof(PrivaPub.Domain.Social.FollowResender) };
typeof(PrivaPub.Domain.Social.FollowResender), typeof(PrivaPub.Infrastructure.Backup.BackupScheduler),
typeof(PrivaPub.Infrastructure.Backup.RestoreWatcher) };
static PrivaPubHost _shared;
readonly string _mediaRoot = Path.Combine(Path.GetTempPath(), $"privapub-tests-{Guid.NewGuid():N}");
@@ -78,6 +79,10 @@ namespace PrivaPub.Tests.Support.Host
["Statistics:Geo:AutoUpdate"] = "false",
["Statistics:Cdn:AutoUpdate"] = "false",
["Media:Root"] = _mediaRoot,
["Backups:Root"] = _mediaRoot + "-backups",
["Backups:Nightly"] = "false",
["RateLimits:UploadsBurst"] = "1000",
["RateLimits:ProxyBurst"] = "100000",
["Logging:LogLevel:Default"] = "Warning",
["Serilog:MinimumLevel:Default"] = Environment.GetEnvironmentVariable("PRIVAPUB_TEST_LOGS") == "1" ? "Information" : "Fatal"
};
@@ -115,8 +120,11 @@ namespace PrivaPub.Tests.Support.Host
protected override void Dispose(bool disposing)
{
base.Dispose(disposing);
if (disposing && Directory.Exists(_mediaRoot))
Directory.Delete(_mediaRoot, recursive: true);
if (!disposing)
return;
foreach (var directory in new[] { _mediaRoot, _mediaRoot + "-backups", _mediaRoot + "-trash", _mediaRoot + "-incoming", _mediaRoot + "-proxy" })
if (Directory.Exists(directory))
Directory.Delete(directory, recursive: true);
}
sealed class ClientAddressFilter : IStartupFilter
+17
View File
@@ -21,6 +21,7 @@ namespace PrivaPub.Tests.Support
readonly ConcurrentDictionary<string, (string Text, string ContentType)> _documents = new();
readonly ConcurrentDictionary<string, (int Status, TimeSpan Delay)> _answers = new();
readonly ConcurrentDictionary<string, (byte[] Bytes, string ContentType)> _files = new();
readonly ConcurrentDictionary<string, string> _etags = new();
public int Port { get; }
public string A => $"http://127.0.0.1:{Port}";
@@ -64,6 +65,15 @@ namespace PrivaPub.Tests.Support
context.Response.StatusCode = StatusCodes.Status404NotFound;
return;
}
if (peer._etags.TryGetValue(key, out var etag))
{
if (context.Request.Headers.IfNoneMatch.ToString() == etag)
{
context.Response.StatusCode = StatusCodes.Status304NotModified;
return;
}
context.Response.Headers.ETag = etag;
}
context.Response.ContentType = document.ContentType;
await context.Response.WriteAsync(document.Text.Replace("{A}", peer.A).Replace("{B}", peer.B));
});
@@ -74,6 +84,13 @@ namespace PrivaPub.Tests.Support
public void Serve(string path, string json) => _documents[path] = (json, "application/activity+json");
// a document with an ETag: answered 304 to a request that names it in If-None-Match
public void ServeTagged(string path, string json, string etag)
{
_documents[path] = (json, "application/activity+json");
_etags[path] = etag;
}
public void ServeText(string path, string text, string contentType) => _documents[path] = (text, contentType);
public void ServeFile(string path, byte[] bytes, string contentType) => _files[path] = (bytes, contentType);
@@ -1,3 +1,4 @@
using Microsoft.AspNetCore.RateLimiting;
using Microsoft.AspNetCore.Mvc;
using MongoDB.Entities;
@@ -16,6 +17,7 @@ using PrivaPub.Models.Post;
using PrivaPub.Models.Social;
using PrivaPub.Models.User;
using PrivaPub.StaticServices;
using PrivaPub.Infrastructure;
using PostEntity = PrivaPub.Models.Post.Post;
@@ -30,10 +32,12 @@ namespace PrivaPub.Api.Mastodon.Controllers
readonly IFollowService _follows;
readonly IOutboxPublisher _outbox;
readonly IRelationshipService _relationships;
readonly IFeedFollows _feeds;
public AccountsController(MastodonMapper mapper, DbEntities dbEntities, ILocalActorService localActors, IRemoteActorService remoteActors,
IFollowService follows, IOutboxPublisher outbox, IRelationshipService relationships)
IFollowService follows, IOutboxPublisher outbox, IRelationshipService relationships, IFeedFollows feeds)
{
_feeds = feeds;
_relationships = relationships;
_mapper = mapper;
_dbEntities = dbEntities;
@@ -46,18 +50,33 @@ namespace PrivaPub.Api.Mastodon.Controllers
[HttpGet("/api/v1/accounts/verify_credentials"), Scope("read:accounts")]
public async Task<IActionResult> VerifyCredentials(CancellationToken token) => Json(await _mapper.Local(Me, withSource: true, token));
[HttpPatch("/api/v1/accounts/update_credentials"), Scope("write:accounts"), RequestSizeLimit(20 * 1024 * 1024),
[HttpPatch("/api/v1/accounts/update_credentials"), Scope("write:accounts"), EnableRateLimiting(RateLimiting.Uploads), RequestSizeLimit(20 * 1024 * 1024),
RequestFormLimits(MultipartBodyLengthLimit = 20 * 1024 * 1024)]
public async Task<IActionResult> UpdateCredentials([FromServices] IMediaService media, CancellationToken token)
{
var avatar = await _dbEntities.Avatars.MatchID(Me.Id).ExecuteFirstAsync(token);
// a new picture is a row of its own; the one it replaces goes to the trash once the profile is saved
var replaced = new List<Models.Media.MediaAttachment>();
if (Request.HasFormContentType)
{
var form = await Request.ReadFormAsync(token);
if (form.Files["avatar"] is { } picture && await media.ProfileImage(picture, 400, 400, token) is { } pictureUrl)
avatar.PictureURL = pictureUrl;
if (form.Files["header"] is { } header && await media.ProfileImage(header, 1500, 500, token) is { } headerUrl)
avatar.ThumbnailURL = headerUrl;
foreach (var (field, kind, width, height) in new[] { ("avatar", "avatar", 400, 400), ("header", "header", 1500, 500) })
{
if (form.Files[field] is not { } file)
continue;
var outcome = await media.ProfileImage(avatar.ID, kind, file, width, height, token);
if (!outcome.Ok)
{
foreach (var made in replaced)
await media.Trash(m => m.ID == made.ID, "profile not saved", token);
return Error(outcome.Status, outcome.Error);
}
replaced.Add(outcome.Attachment);
if (kind == "avatar")
avatar.PictureURL = media.Url(outcome.Attachment.FilePath);
else
avatar.ThumbnailURL = media.Url(outcome.Attachment.FilePath);
}
}
if (Params.Has("display_name"))
avatar.Name = Params.Get("display_name")?.Trim();
@@ -98,12 +117,36 @@ namespace PrivaPub.Api.Mastodon.Controllers
avatar.Fields = fields;
avatar.UpdatedAt = DateTime.UtcNow;
await DB.Default.SaveAsync(avatar, token);
foreach (var made in replaced)
await media.Trash(m => m.ProfileOfAvatarId == avatar.ID && m.Kind == made.Kind && m.ID != made.ID, "replaced", token);
var actor = _localActors.FromAvatar(avatar);
await _outbox.PublishProfile(actor, token);
return Json(await _mapper.Local(actor, withSource: true, token));
}
// Mastodon's: the picture goes (to the trash), the profile shows the default again, and followers are told
[HttpDelete("/api/v1/profile/avatar"), Scope("write:accounts")]
public Task<IActionResult> DeleteAvatar([FromServices] IMediaService media, CancellationToken token) => DeletePicture(media, "avatar", token);
[HttpDelete("/api/v1/profile/header"), Scope("write:accounts")]
public Task<IActionResult> DeleteHeader([FromServices] IMediaService media, CancellationToken token) => DeletePicture(media, "header", token);
async Task<IActionResult> DeletePicture(IMediaService media, string kind, CancellationToken token)
{
var avatar = await _dbEntities.Avatars.MatchID(Me.Id).ExecuteFirstAsync(token);
if (kind == "avatar")
avatar.PictureURL = default;
else
avatar.ThumbnailURL = default;
avatar.UpdatedAt = DateTime.UtcNow;
await DB.Default.SaveAsync(avatar, token);
await media.Trash(m => m.ProfileOfAvatarId == avatar.ID && m.Kind == kind, "removed", token);
var actor = _localActors.FromAvatar(avatar);
await _outbox.PublishProfile(actor, token);
return Json(await _mapper.Local(actor, withSource: true, token));
}
[HttpGet("/api/v1/accounts/lookup"), Scope("read:accounts", requiresUser: false), Microsoft.AspNetCore.Authorization.AllowAnonymous]
public async Task<IActionResult> Lookup(CancellationToken token)
{
@@ -212,7 +255,10 @@ namespace PrivaPub.Api.Mastodon.Controllers
[HttpGet("/api/v1/accounts/{id}/following"), Scope("read:accounts", requiresUser: false), Microsoft.AspNetCore.Authorization.AllowAnonymous]
public async Task<IActionResult> Following(string id, CancellationToken token)
{
var (local, _) = await Find(id, token);
var (local, remote) = await Find(id, token);
// a feed's following is what it is: its communities, as its server lists them
if (remote is { AvatarType: AvatarType.Feed })
return Json(await AccountsFor(remote.FeedCommunities.Take(Limit(40, 80)), token));
if (local == default || local.Id != MyId)
return Json(Array.Empty<Account>());
var following = await Page.From(Params, Limit(40, 80))
@@ -222,12 +268,27 @@ namespace PrivaPub.Api.Mastodon.Controllers
return Json(following.Select(f => accounts.GetValueOrDefault(f.TargetAccountId)).Where(a => a != default).ToList());
}
// PrivaPub's own: the feeds the persona reads (a feed's timeline is /api/v1/timelines/feed/:id)
[HttpGet("/api/v1/feeds"), Scope("read:follows")]
public async Task<IActionResult> Feeds(CancellationToken token)
{
var read = await DB.Default.Find<FeedSubscription>().Match(s => s.AvatarId == MyId).Sort(s => s.ID, MongoDB.Entities.Order.Descending).ExecuteAsync(token);
var accounts = await _mapper.Accounts(read.Select(s => s.FeedAccountId), token);
return Json(read.Select(s => accounts.GetValueOrDefault(s.FeedAccountId)).Where(a => a != default).ToList());
}
[HttpPost("/api/v1/accounts/{id}/follow"), Scope("write:follows")]
public async Task<IActionResult> Follow(string id, CancellationToken token)
{
var (local, remote) = await Find(id, token);
if (local == default && remote == default)
return NotFoundError();
// a feed is read through the server, whose reader follows its communities; the persona follows nothing
if (remote is { AvatarType: AvatarType.Feed })
{
await _feeds.Subscribe(Me, remote, true, token);
return Json(await Relationship(id, token));
}
var following = await _follows.FollowAs(Me, local?.Uri ?? remote.ActorURI, Params.Bool("reblogs") != false, token);
return following == default ? Error(StatusCodes.Status403Forbidden, "This action is not allowed") : Json(await Relationship(id, token));
}
@@ -238,6 +299,9 @@ namespace PrivaPub.Api.Mastodon.Controllers
var (local, remote) = await Find(id, token);
if (local == default && remote == default)
return NotFoundError();
if (remote is { AvatarType: AvatarType.Feed })
await _feeds.Subscribe(Me, remote, false, token);
else
await _follows.UnfollowAs(Me, local?.Uri ?? remote.ActorURI, token);
return Json(await Relationship(id, token));
}
@@ -386,6 +450,7 @@ namespace PrivaPub.Api.Mastodon.Controllers
var (local, remote) = await Find(id, token);
var uri = local?.Uri ?? remote?.ActorURI;
var following = uri == default ? default : await _dbEntities.Followings.Match(f => f.AvatarId == MyId && f.TargetActorURI == uri).ExecuteFirstAsync(token);
var reading = remote is { AvatarType: AvatarType.Feed } && await _feeds.IsSubscribed(MyId, remote.ActorURI, token);
var followedBy = uri == default ? default : await _dbEntities.Followers.Match(f => f.LocalActorId == MyId && f.ActorURI == uri).ExecuteFirstAsync(token);
var blocking = uri != default && await DB.Default.Find<Block>().Match(b => b.AvatarId == MyId && b.TargetActorURI == uri).ExecuteAnyAsync(token);
var mute = uri == default ? default : await DB.Default.Find<Mute>().Match(m => m.AvatarId == MyId && m.TargetActorURI == uri).ExecuteFirstAsync(token);
@@ -402,7 +467,7 @@ namespace PrivaPub.Api.Mastodon.Controllers
Muting = mute != default && (mute.ExpiresAt == default || mute.ExpiresAt > DateTime.UtcNow),
MutingNotifications = mute?.HideNotifications == true,
DomainBlocking = domainBlocked,
Following = following?.State == FollowState.Accepted,
Following = following?.State == FollowState.Accepted || reading,
Requested = following?.State == FollowState.Requested,
ShowingReblogs = following?.ShowReblogs ?? false,
FollowedBy = followedBy?.IsAccepted == true,
@@ -56,10 +56,10 @@ namespace PrivaPub.Api.Mastodon.Controllers
{
supported_mime_types = PrivaPub.Domain.Media.MediaService.SupportedTypes,
image_size_limit = media.MaxImageBytes,
image_matrix_limit = media.MaxImageSide * media.MaxImageSide,
image_matrix_limit = media.MaxPixels,
video_size_limit = media.MaxVideoBytes,
video_frame_rate_limit = 60,
video_matrix_limit = 2_304_000
video_frame_rate_limit = (int)media.MaxFrameRate,
video_matrix_limit = media.MaxVideoPixels
};
}
}
@@ -1,3 +1,4 @@
using Microsoft.AspNetCore.RateLimiting;
using PrivaPub.Infrastructure.Statistics;
using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Mvc;
@@ -7,6 +8,8 @@ using MongoDB.Entities;
using PrivaPub.Api.Mastodon.Infrastructure;
using PrivaPub.Domain.Media;
using PrivaPub.Models.Media;
using PrivaPub.Infrastructure;
using PrivaPub.Infrastructure.Jobs;
using System.Globalization;
@@ -18,78 +21,113 @@ namespace PrivaPub.Api.Mastodon.Controllers
readonly IMediaService _media;
readonly IMediaProxy _proxy;
readonly IJobQueue _jobs;
readonly IInteractionLedger _ledger;
public MediaController(IMediaService media, IMediaProxy proxy, IInteractionLedger ledger = default)
public MediaController(IMediaService media, IMediaProxy proxy, IJobQueue jobs, IInteractionLedger ledger = default)
{
_media = media;
_proxy = proxy;
_jobs = jobs;
_ledger = ledger;
}
[HttpPost("/api/v1/media"), HttpPost("/api/v2/media"), Scope("write:media"), RequestSizeLimit(UploadLimit),
[HttpPost("/api/v1/media"), HttpPost("/api/v2/media"), Scope("write:media"), EnableRateLimiting(RateLimiting.Uploads), RequestSizeLimit(UploadLimit),
RequestFormLimits(MultipartBodyLengthLimit = UploadLimit)]
public async Task<IActionResult> Upload(CancellationToken token)
{
if (!Request.HasFormContentType)
return Error(StatusCodes.Status422UnprocessableEntity, "Validation failed: File can't be blank");
var form = await Request.ReadFormAsync(token);
var outcome = await _media.Upload(Me, form.Files["file"], form["description"], form["focus"], token);
return outcome.Ok ? Json(View(outcome.Attachment)) : Error(outcome.Status, outcome.Error);
// v2 may answer before audio or video is processed (202, its url null until then), as Mastodon does; v1 waits
var later = Request.Path.StartsWithSegments("/api/v2/media");
var outcome = await _media.Upload(Me, form.Files["file"], form["description"], form["focus"], later, token);
if (!outcome.Ok)
return Error(outcome.Status, outcome.Error);
if (outcome.Attachment.ProcessingState != "pending")
return Json(View(outcome.Attachment));
await _jobs.EnqueueMany(new[] { ProcessMediaJob.JobFor(outcome.Attachment, new Uri(Me.BaseAddress).Host) }, token);
return new JsonResult(View(outcome.Attachment)) { StatusCode = StatusCodes.Status202Accepted };
}
[HttpGet("/api/v1/media/{id}"), Scope("write:media")]
public async Task<IActionResult> Get(string id, CancellationToken token)
{
var attachment = await DB.Default.Find<MediaAttachment>().Match(m => m.ID == id && m.OwnerAvatarId == MyId).ExecuteFirstAsync(token);
return attachment == default ? NotFoundError() : Json(View(attachment));
var attachment = await DB.Default.Find<MediaAttachment>().Match(m => m.ID == id && m.OwnerAvatarId == MyId && m.TrashedAt == null && m.ProfileOfAvatarId == null).ExecuteFirstAsync(token);
return attachment?.ProcessingState switch
{
null when attachment == default => NotFoundError(),
"pending" => new JsonResult(View(attachment)) { StatusCode = StatusCodes.Status206PartialContent },
"failed" => Error(StatusCodes.Status422UnprocessableEntity, attachment.ProcessingError ?? "Validation failed: The file could not be processed"),
_ => Json(View(attachment))
};
}
// as on Mastodon, only media not posted yet: a posted one's description and focus change by editing its post (media_attributes)
[HttpPut("/api/v1/media/{id}"), Scope("write:media")]
public async Task<IActionResult> Update(string id, CancellationToken token)
{
var attachment = await DB.Default.Find<MediaAttachment>().Match(m => m.ID == id && m.OwnerAvatarId == MyId).ExecuteFirstAsync(token);
var attachment = await DB.Default.Find<MediaAttachment>().Match(m => m.ID == id && m.OwnerAvatarId == MyId && m.TrashedAt == null && m.ProfileOfAvatarId == null && m.PostId == null).ExecuteFirstAsync(token);
if (attachment == default)
return NotFoundError();
if (Params.Has("description"))
attachment.Description = Params.Get("description")?.Trim() is { Length: > 0 } description ? description[..Math.Min(description.Length, 1500)] : default;
if (Params.Get("focus")?.Split(',') is [var x, var y]
&& float.TryParse(x, NumberStyles.Float, CultureInfo.InvariantCulture, out var fx) && float.TryParse(y, NumberStyles.Float, CultureInfo.InvariantCulture, out var fy))
attachment.Focus = new[] { Math.Clamp(fx, -1, 1), Math.Clamp(fy, -1, 1) };
if (FocalPoint.Parse(Params.Get("focus")) is { } focus)
attachment.Focus = focus;
await DB.Default.SaveAsync(attachment, token);
return Json(View(attachment));
}
[HttpGet("/media/proxy/{signature}/{encoded}"), AllowAnonymous, ApiExplorerSettings(IgnoreApi = true)]
[HttpGet("/media/proxy/{signature}/{encoded}"), AllowAnonymous, EnableRateLimiting(RateLimiting.Proxy), ApiExplorerSettings(IgnoreApi = true)]
public async Task<IActionResult> Proxy(string signature, string encoded, CancellationToken token)
{
var url = _proxy.Verified(signature, encoded);
if (url == default)
if (url == default || _proxy.Refuses(url))
return NotFound();
Response.Headers["X-Content-Type-Options"] = "nosniff";
Response.Headers["Content-Security-Policy"] = "default-src 'none'; sandbox";
Response.Headers["Cache-Control"] = "public, max-age=604800";
if (_proxy.Cached(url) is { Path: not null } cached)
if (_proxy.Cached(url) is { Path: not null } cached && Serve(cached.Path, cached.ContentType) is { } hit)
{
_ledger?.Count(Interactions.HostOf(url), "media:hit");
return PhysicalFile(cached.Path, cached.ContentType, enableRangeProcessing: true);
return hit;
}
if (Request.Headers.Range.Count == 0)
{
var (path, contentType) = await _proxy.Fetch(signature, encoded, token);
if (path != default)
return PhysicalFile(path, contentType, enableRangeProcessing: true);
var (outcome, path, contentType) = await _proxy.Download(url, token);
if (outcome == ProxyOutcome.Cached && Serve(path, contentType) is { } fetched)
return fetched;
if (outcome == ProxyOutcome.Failed)
return NotFound();
}
return await Stream(url, token);
}
// a cached file, opened before it is answered: trimmed meanwhile, what is open still reads; cached by browsers only
// once there is something to cache
IActionResult Serve(string path, string contentType)
{
FileStream file;
try
{
file = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite | FileShare.Delete, 64 * 1024, useAsync: true);
}
catch (IOException)
{
return default;
}
Response.Headers["Cache-Control"] = "public, max-age=604800";
return File(file, contentType, enableRangeProcessing: true);
}
async Task<IActionResult> Stream(string url, CancellationToken token)
{
var range = System.Net.Http.Headers.RangeHeaderValue.TryParse(Request.Headers.Range.ToString(), out var asked) ? asked : default;
using var upstream = await _proxy.Open(url, range, token);
if (upstream == default)
return NotFound();
if (upstream.IsSuccessStatusCode)
Response.Headers["Cache-Control"] = "public, max-age=604800";
Response.StatusCode = (int)upstream.StatusCode;
Response.ContentType = upstream.Content.Headers.ContentType?.ToString() ?? "application/octet-stream";
if (upstream.Content.Headers.ContentLength is { } length)
@@ -96,7 +96,8 @@ namespace PrivaPub.Api.Mastodon.Controllers
if (await Mine(id, token) is not { } scheduled)
return NotFoundError();
await DB.Default.DeleteAsync<ScheduledStatus>(scheduled.ID);
await ScheduledStatuses.Release(scheduled.ID, token);
// its media were the scheduled post's alone, never posted
await _media.Trash(m => m.ScheduledStatusId == scheduled.ID, "scheduled post dropped", token);
return Json(new { });
}
}
@@ -105,15 +105,30 @@ namespace PrivaPub.Api.Mastodon.Controllers
mine.UnionWith((await _dbEntities.Posts.Match(p => p.AuthorAccountId == MyId && ids.Contains(p.ReblogOfPostId) && !p.DeletedAt.HasValue).ExecuteAsync(token))
.Select(p => p.ReblogOfPostId));
var authors = candidates.Select(p => p.AuthorAccountId ?? p.GroupUserId).Where(a => a != default).Distinct().ToList();
var indexable = (await _dbEntities.Avatars.Match(a => authors.Contains(a.ID) && a.Settings.IsIndexable).ExecuteAsync(token)).Select(a => a.ID)
.Concat((await _dbEntities.ForeignAvatars.Match(f => authors.Contains(f.ID) && f.IsIndexable).ExecuteAsync(token)).Select(f => f.ID))
.ToHashSet();
var indexable = (await _dbEntities.Avatars.Match(a => authors.Contains(a.ID) && a.Settings.IsIndexable).ExecuteAsync(token)).Select(a => a.ID).ToHashSet();
// a remote author's searchableBy (FEP-268d) outranks its indexable, and a post's own outranks both
var remoteAuthors = (await _dbEntities.ForeignAvatars.Match(f => authors.Contains(f.ID)).ExecuteAsync(token)).ToDictionary(f => f.ID);
var stating = remoteAuthors.Values.Where(f => f.SearchableBy != default).Select(f => f.ActorURI)
.Concat(candidates.Where(p => p.SearchableBy != default).Select(p => p.ActorURI))
.Where(uri => uri != default).Distinct().ToList();
var followed = stating.Count == 0
? new HashSet<string>()
: (await _dbEntities.Followings.Match(f => f.AvatarId == MyId && stating.Contains(f.TargetActorURI) && f.State == Models.Social.FollowState.Accepted)
.ExecuteAsync(token)).Select(f => f.TargetActorURI).ToHashSet();
bool Searchable(PostEntity post, string author)
{
var remote = remoteAuthors.GetValueOrDefault(author);
var rule = post.SearchableBy ?? remote?.SearchableBy;
if (rule != default)
return Federation.Objects.ObjectShapes.Searchable(rule, remote?.FollowersURL, followed.Contains(post.ActorURI ?? string.Empty));
return remote?.IsIndexable == true || indexable.Contains(author);
}
var found = new List<PostEntity>();
foreach (var post in candidates)
{
var author = post.AuthorAccountId ?? post.GroupUserId;
var reachable = author == MyId && !post.IsFederatedCopy || mine.Contains(post.ID) || post.Mentions.Any(m => m.AccountId == MyId)
|| post.Visibility == PostVisibility.Public && indexable.Contains(author);
|| post.Visibility == PostVisibility.Public && Searchable(post, author);
if (reachable && await VisibilityPolicy.CanSee(post, MyId, token))
found.Add(post);
if (found.Count >= offset + limit)
@@ -111,13 +111,26 @@ namespace PrivaPub.Api.Mastodon.Controllers
return Error(StatusCodes.Status422UnprocessableEntity, "Validation failed: Text can't be blank");
var mediaIds = asked.MediaIds.Distinct().ToList();
if (mediaIds.Count > 4 || mediaIds.Count > 0 && await DB.Default.CountAsync<Models.Media.MediaAttachment>(
m => mediaIds.Contains(m.ID) && m.OwnerAvatarId == MyId && m.PostId == null && m.ScheduledStatusId == null, token) != mediaIds.Count)
m => mediaIds.Contains(m.ID) && m.OwnerAvatarId == MyId && m.PostId == null && m.ScheduledStatusId == null
&& m.TrashedAt == null && m.ProfileOfAvatarId == null, token) != mediaIds.Count)
return Error(StatusCodes.Status422UnprocessableEntity, "Validation failed: Media attachments are not yours, already posted or too many");
var scheduled = new ScheduledStatus { AvatarId = MyId, ScheduledAt = at, Params = asked };
await DB.Default.SaveAsync(scheduled, token);
scheduled.ID = (string)scheduled.GenerateNewID();
// reserved in one conditional update: a post or another scheduled post taking one meanwhile wins
if (mediaIds.Count > 0)
await DB.Default.Update<Models.Media.MediaAttachment>().Match(m => mediaIds.Contains(m.ID)).Modify(m => m.ScheduledStatusId, scheduled.ID).ExecuteAsync(token);
{
var reserved = await DB.Default.Update<Models.Media.MediaAttachment>()
.Match(m => mediaIds.Contains(m.ID) && m.PostId == null && m.ScheduledStatusId == null && m.TrashedAt == null)
.Modify(m => m.ScheduledStatusId, scheduled.ID).ExecuteAsync(token);
if (reserved.MatchedCount != mediaIds.Count)
{
await DB.Default.Update<Models.Media.MediaAttachment>().Match(m => m.ScheduledStatusId == scheduled.ID)
.Modify(m => m.ScheduledStatusId, null).ExecuteAsync(token);
return Error(StatusCodes.Status422UnprocessableEntity, "Validation failed: Media attachments are not yours, already posted or too many");
}
}
await DB.Default.SaveAsync(scheduled, token);
await _jobs.EnqueueMany(new[] { ScheduledStatuses.JobFor(scheduled, new Uri(Me.BaseAddress).Host) }, token);
if (cacheKey != default)
_cache.Set(cacheKey + ":scheduled", scheduled.ID, TimeSpan.FromHours(1));
@@ -154,11 +167,41 @@ namespace PrivaPub.Api.Mastodon.Controllers
SpoilerText = Params.Get("spoiler_text"),
Sensitive = Params.Bool("sensitive") ?? false,
Language = Params.Get("language"),
MediaIds = Params.Has("media_ids") ? Params.List("media_ids") : default
MediaIds = Params.Has("media_ids") ? Params.List("media_ids") : default,
MediaChanges = MediaAttributes()
}, token);
return outcome.Ok ? Json(await _mapper.Status(outcome.Post, MyId, token)) : Error(outcome.Status, outcome.Error);
}
// Mastodon's media_attributes: a JSON body's array of objects, or a form's media_attributes[i][field] or
// media_attributes[][field] (each field then listed in the same order)
IReadOnlyList<MediaChange> MediaAttributes()
{
var changes = new List<MediaChange>();
if (Params.Json is { ValueKind: System.Text.Json.JsonValueKind.Object } json && json.TryGetProperty("media_attributes", out var array)
&& array.ValueKind == System.Text.Json.JsonValueKind.Array)
{
foreach (var item in array.EnumerateArray().Where(i => i.ValueKind == System.Text.Json.JsonValueKind.Object).Take(4))
{
var hasDescription = item.TryGetProperty("description", out var description);
changes.Add(new MediaChange(Text(item, "id"), hasDescription, hasDescription ? Text(item, "description") : default, Text(item, "focus")));
}
return changes;
}
for (var i = 0; i < 4 && Params.Get($"media_attributes[{i}][id]") is { } id; i++)
changes.Add(new MediaChange(id, Params.Has($"media_attributes[{i}][description]"), Params.Get($"media_attributes[{i}][description]"), Params.Get($"media_attributes[{i}][focus]")));
var ids = Params.List("media_attributes[][id]");
var descriptions = Params.List("media_attributes[][description]");
var foci = Params.List("media_attributes[][focus]");
for (var i = 0; i < Math.Min(ids.Count, 4); i++)
changes.Add(new MediaChange(ids[i], i < descriptions.Count, i < descriptions.Count ? descriptions[i] : default, i < foci.Count ? foci[i] : default));
return changes;
static string Text(System.Text.Json.JsonElement item, string name) => item.TryGetProperty(name, out var value)
? value.ValueKind == System.Text.Json.JsonValueKind.String ? value.GetString() : value.ValueKind is System.Text.Json.JsonValueKind.Null ? default : value.GetRawText()
: default;
}
[HttpDelete("/api/v1/statuses/{id}"), Scope("write:statuses")]
public async Task<IActionResult> Delete(string id, CancellationToken token)
{
@@ -290,6 +333,13 @@ namespace PrivaPub.Api.Mastodon.Controllers
[HttpPost("/api/v1/statuses/{id}/unfavourite"), Scope("write:favourites")]
public Task<IActionResult> Unfavourite(string id, CancellationToken token) => Toggle(_statuses.Favourite(Me, id, false, token), id, token);
// PrivaPub's own, for the threadiverse's downvotes (a favourite is the upvote)
[HttpPost("/api/v1/statuses/{id}/downvote"), Scope("write:favourites")]
public Task<IActionResult> Downvote(string id, CancellationToken token) => Toggle(_statuses.Downvote(Me, id, true, token), id, token);
[HttpPost("/api/v1/statuses/{id}/undownvote"), Scope("write:favourites")]
public Task<IActionResult> Undownvote(string id, CancellationToken token) => Toggle(_statuses.Downvote(Me, id, false, token), id, token);
[HttpPost("/api/v1/statuses/{id}/reblog"), Scope("write:statuses")]
public async Task<IActionResult> Reblog(string id, CancellationToken token)
{
@@ -64,6 +64,26 @@ namespace PrivaPub.Api.Mastodon.Controllers
return await Respond(query, "/api/v1/timelines/public", token);
}
// PrivaPub's own: a feed the persona reads (Lemmy's multi-community, PieFed's feed), its communities' threads,
// newest first, as the server's reader of feeds brings them in (owner decision 2026-10-07)
[HttpGet("/api/v1/timelines/feed/{id}"), Scope("read:statuses")]
public async Task<IActionResult> Feed(string id, CancellationToken token)
{
var feed = await _dbEntities.ForeignAvatars.MatchID(id).ExecuteFirstAsync(token);
if (feed is not { AvatarType: Models.User.AvatarType.Feed }
|| !await DB.Default.Find<FeedSubscription>().Match(s => s.AvatarId == MyId && s.FeedActorURI == feed.ActorURI).ExecuteAnyAsync(token))
return NotFoundError();
var communities = feed.FeedCommunities;
var ours = Me.BaseAddress + "/peasants/";
var names = communities.Where(c => c.StartsWith(ours, StringComparison.Ordinal)).Select(c => c[ours.Length..]).ToList();
var groups = names.Count == 0
? new List<string>()
: (await _dbEntities.Groups.Match(g => names.Contains(g.UserName) && !g.DeletionAt.HasValue).ExecuteAsync(token)).Select(g => g.ID).ToList();
var query = _dbEntities.Posts.Match(p => (communities.Contains(p.AudienceURI) || groups.Contains(p.GroupId))
&& p.ReblogOfPostId == null && p.InReplyToURI == null).Match(VisibilityPolicy.IsPublic);
return await Respond(query, $"/api/v1/timelines/feed/{id}", token);
}
[HttpGet("/api/v1/timelines/tag/{hashtag}"), Scope("read:statuses", requiresUser: false), Microsoft.AspNetCore.Authorization.AllowAnonymous]
public async Task<IActionResult> Tag(string hashtag, CancellationToken token)
{
@@ -217,7 +237,8 @@ namespace PrivaPub.Api.Mastodon.Controllers
[NotificationType.Update] = "update",
[NotificationType.Poll] = "poll",
[NotificationType.Reaction] = "pleroma:emoji_reaction",
[NotificationType.Quote] = "quote"
[NotificationType.Quote] = "quote",
[NotificationType.ModerationWarning] = "moderation_warning"
};
readonly MastodonMapper _mapper;
@@ -317,7 +338,8 @@ namespace PrivaPub.Api.Mastodon.Controllers
sample_account_ids = group.Select(n => mapped[n.ID].Account.Id).Distinct().Take(8).ToList(),
status_id = newest.Status?.Id,
emoji = newest.Emoji,
emoji_url = newest.EmojiUrl
emoji_url = newest.EmojiUrl,
moderation_warning = newest.ModerationWarning
});
}
return new
@@ -79,6 +79,7 @@
public PrivaPubPlace Place { get; set; }//a remote post's own place (Pixelfed), shown only
public Account Wall { get; set; }//whose wall it was written on (FEP-400e), when not its author's
public PrivaPubVotes Votes { get; set; } = new();
public List<PrivaPubFlair> Flairs { get; set; } = new();
}
public class PrivaPubSource
@@ -163,10 +164,23 @@
public string Url { get; set; }
}
public class PrivaPubFlair
{
public string Id { get; set; }
public string Name { get; set; }
public string Slug { get; set; }
public string Description { get; set; }
public string Color { get; set; }
public string TextColor { get; set; }
public string BackgroundColor { get; set; }
public bool BlurImages { get; set; }
}
public class PrivaPubVotes
{
public int Up { get; set; }
public int Down { get; set; }
public bool? Downvoted { get; set; }//the viewer's own downvote; null with no viewer
}
public class QuoteEntity
@@ -405,6 +419,18 @@
public Status Status { get; set; }
public string Emoji { get; set; }
public string EmojiUrl { get; set; }
public AccountWarning ModerationWarning { get; set; }
}
public class AccountWarning
{
public string Id { get; set; }
public string Action { get; set; } = "none";
public string Text { get; set; } = string.Empty;
public List<string> StatusIds { get; set; } = new();
public Account TargetAccount { get; set; }
public object Appeal { get; set; }
public string CreatedAt { get; set; }
}
public class Context
@@ -89,6 +89,9 @@ namespace PrivaPub.Api.Mastodon.Infrastructure
Flatten(values, prefix == default ? property.Name : $"{prefix}[{property.Name}]", property.Value);
break;
case JsonValueKind.Array:
// an empty array is still said: media_ids: [] on an edit takes every attachment away
if (prefix != default && element.GetArrayLength() == 0)
values.TryAdd(prefix, new List<string>());
foreach (var item in element.EnumerateArray())
{
if (item.ValueKind is JsonValueKind.Object or JsonValueKind.Array)
@@ -49,6 +49,7 @@ namespace PrivaPub.Api.Mastodon.Mappers
? new List<PostEntity>()
: await _dbEntities.Posts.Match(p => postIds.Contains(p.ID)).Match(VisibilityPolicy.IsShown).ExecuteAsync(token);
var statuses = (await Statuses(posts, viewerId, token)).ToDictionary(s => s.Id);
var warned = notifications.Any(n => n.Type == NotificationType.ModerationWarning) && viewerId != default ? await Account(viewerId, token) : default;
return notifications
.Where(n => accounts.ContainsKey(n.FromAccountId ?? string.Empty) && (n.PostId == default || statuses.ContainsKey(n.PostId)))
.Select(n => new Entities.Notification
@@ -59,9 +60,17 @@ namespace PrivaPub.Api.Mastodon.Mappers
//the group it belongs to by default, so a notification told alone (v1, a stream) joins its group in a client
GroupKey = Controllers.NotificationsController.GroupKey(n, Controllers.NotificationsController.DefaultGrouped),
Account = accounts[n.FromAccountId],
Status = n.PostId == default ? default : statuses[n.PostId],
Status = n.PostId == default || n.Type == NotificationType.ModerationWarning ? default : statuses[n.PostId],
Emoji = n.Emoji == default ? default : n.EmojiURL == default ? n.Emoji : $":{n.Emoji}:",
EmojiUrl = Proxied(n.EmojiURL)
EmojiUrl = Proxied(n.EmojiURL),
ModerationWarning = n.Type != NotificationType.ModerationWarning ? default : new AccountWarning
{
Id = n.ID,
Text = n.Text ?? string.Empty,
StatusIds = n.PostId == default ? new List<string>() : new List<string> { n.PostId },
TargetAccount = warned,
CreatedAt = MastodonJson.Time(n.CreatedAt)
}
})
.ToList();
}
@@ -209,6 +218,9 @@ namespace PrivaPub.Api.Mastodon.Mappers
var favourited = viewerId == default
? new HashSet<string>()
: (await _dbEntities.Favourites.Match(f => f.AccountId == viewerId && ids.Contains(f.PostId)).ExecuteAsync(token)).Select(f => f.PostId).ToHashSet();
var downvoted = viewerId == default
? new HashSet<string>()
: (await DB.Default.Find<Downvote>().Match(d => d.AccountId == viewerId && ids.Contains(d.PostId)).ExecuteAsync(token)).Select(d => d.PostId).ToHashSet();
var bookmarked = viewerId == default
? new HashSet<string>()
: (await DB.Default.Find<Bookmark>().Match(b => b.AvatarId == viewerId && ids.Contains(b.PostId)).ExecuteAsync(token)).Select(b => b.PostId).ToHashSet();
@@ -277,7 +289,7 @@ namespace PrivaPub.Api.Mastodon.Mappers
Card = Card(post),
Poll = Poll(post, viewerId, ownVotes.GetValueOrDefault(post.ID)),
Emojis = Emojis(post.Emojis),
Privapub = Extension(post, accounts),
Privapub = Extension(post, accounts, viewerId == default ? default(bool?) : downvoted.Contains(post.ID)),
EmojiReactions = reactions.GetValueOrDefault(post.ID) ?? new(),
Pleroma = new PleromaStatus { EmojiReactions = reactions.GetValueOrDefault(post.ID) ?? new() },
Mentions = post.Mentions.Where(m => !m.Silent).Select(m => Mention(m, mentionAccounts)).Where(m => m != default).ToList(),
@@ -480,7 +492,7 @@ namespace PrivaPub.Api.Mastodon.Mappers
List<CustomEmojiEntity> Emojis(IEnumerable<CustomEmoji> emojis) =>
emojis?.Select(e => new CustomEmojiEntity { Shortcode = e.Shortcode, Url = Proxied(e.URL), StaticUrl = Proxied(e.URL) }).ToList() ?? new();
PrivaPubStatus Extension(PostEntity post, Dictionary<string, Account> accounts) => new()
PrivaPubStatus Extension(PostEntity post, Dictionary<string, Account> accounts, bool? downvoted) => new()
{
Wall = post.WallOwnerAccountId == default || post.WallOwnerAccountId == AuthorOf(post) ? default : accounts.GetValueOrDefault(post.WallOwnerAccountId),
ObjectType = post.ObjectType,
@@ -534,7 +546,12 @@ namespace PrivaPub.Api.Mastodon.Mappers
{
Name = post.Place.Name, Latitude = post.Place.Latitude, Longitude = post.Place.Longitude, Country = post.Place.Country
},
Votes = new PrivaPubVotes { Up = post.FavouritesCount, Down = post.DownvotesCount }
Votes = new PrivaPubVotes { Up = post.FavouritesCount, Down = post.DownvotesCount, Downvoted = downvoted },
Flairs = post.Flairs?.Select(f => new PrivaPubFlair
{
Id = f.Id, Name = f.Name, Slug = f.Slug, Description = f.Description, Color = f.Color, TextColor = f.TextColor,
BackgroundColor = f.BackgroundColor, BlurImages = f.BlurImages
}).ToList() ?? new()
};
PrivaPubVariant Variant(MediaVariant variant) => new()
@@ -642,7 +659,7 @@ namespace PrivaPub.Api.Mastodon.Mappers
meta["original"] = original;
if (media.DurationSeconds is { } length)
meta["length"] = TimeSpan.FromSeconds(length).ToString(@"h\:mm\:ss\.ff", CultureInfo.InvariantCulture);
if (media.Focus is { Length: 2 })
if (FocalPoint.IsSound(media.Focus))
meta["focus"] = new { x = media.Focus[0], y = media.Focus[1] };
return meta;
}
@@ -650,9 +667,9 @@ namespace PrivaPub.Api.Mastodon.Mappers
MediaAttachment Media(PostMedia media) => new()
{
Id = media.AttachmentId ?? media.Id.ToString("N"),
Type = media.ContentType switch
// a GIF is a gifv only once it is an mp4 (ours): a GIF itself is an image, which a gifv player can't play
Type = media.Kind == "gifv" ? "gifv" : media.ContentType switch
{
{ } type when type.StartsWith("image/gif") => "gifv",
{ } type when type.StartsWith("image/") => "image",
{ } type when type.StartsWith("video/") => "video",
{ } type when type.StartsWith("audio/") => "audio",
@@ -0,0 +1,280 @@
using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Http.Features;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Localization;
using Microsoft.Net.Http.Headers;
using MongoDB.Entities;
using PrivaPub.ClientModels;
using PrivaPub.ClientModels.Admin;
using PrivaPub.Extensions;
using PrivaPub.Infrastructure.Backup;
using PrivaPub.Models.User;
using PrivaPub.Resources;
using PrivaPub.StaticServices;
namespace PrivaPub.Controllers.ClientToServer
{
// The server's backups from the administrator's page (owner decision 2026-10-07: it backs up and restores too). Only
// an administrator, checked again against the database; a download and a restore ask for the password again, and a
// restore for the server's host typed out. A restore asked for stops the service within seconds, and the next start
// carries it out (ServerRestore).
[ApiController,
Route("clientapi/admin/backups"),
Authorize(Policy = Policies.IsAdmin)]
public class BackupController(Backups backups, TransferStore transfers, IPasswordHasher hasher, IStringLocalizer<GenericRes> localizer,
ILogger<BackupController> logger) : ControllerBase
{
[HttpGet, Route("")]
public async Task<IActionResult> List(CancellationToken token)
{
if (await Administrator(token) == default)
return Forbid();
var deleted = await DB.Default.Find<RootUser>().Match(u => u.DeletedAt != null).Project(u => u.Include(x => x.DeletedAt)).ExecuteAsync(token);
var code = ServerBackup.CodeMigration();
var waiting = RestoreMarker.Read(backups.Root);
var last = await DB.Default.Find<RestoreRecord>().Sort(r => r.RestoredAt, Order.Descending).ExecuteFirstAsync(token);
ServerBackup.MakeDirectory(backups.Root);
return Ok(new ViewBackups
{
Backups = [.. backups.List().Select(b => View(b, deleted.Count(u => u.DeletedAt > b.CreatedAt), code))],
Running = (await MaintenanceLock.Current(token))?.What,
Waiting = waiting == default ? default : new ViewRestoreWaiting
{
Backup = waiting.Backup,
State = waiting.State,
Attempts = waiting.Attempts,
RequestedBy = waiting.RequestedBy,
RequestedAt = waiting.RequestedAt,
Error = waiting.Error
},
LastRestore = last == default ? default : View(last),
Host = HostName,
FreeBytes = new DriveInfo(backups.Root).AvailableFreeSpace
});
}
// a backup made now, while the page polls the list
[HttpPost, Route("")]
public async Task<IActionResult> Create(CancellationToken token)
{
var admin = await Administrator(token);
if (admin == default)
return Forbid();
if (await MaintenanceLock.Current(token) is { } held)
return Conflict(new WebResult().Invalidate(localizer["A {0} is running.", held.What]));
_ = Task.Run(async () =>
{
try
{
var (made, error) = await backups.Create("manual", dbOnly: false, CancellationToken.None);
if (made == default)
logger.LogWarning("The backup {Admin} asked for was not made: {Error}", admin.UserName, error);
else
logger.LogInformation("Backup {Id} made for {Admin}", made.Id, admin.UserName);
}
catch (Exception ex)
{
logger.LogError(ex, "The backup {Admin} asked for failed", admin.UserName);
}
}, CancellationToken.None);
return Accepted();
}
[HttpDelete, Route("{id}")]
public async Task<IActionResult> Delete(string id, CancellationToken token)
{
if (await Administrator(token) == default)
return Forbid();
if (RestoreMarker.Read(backups.Root) is { } waiting && (waiting.Backup == id || waiting.PreRestore == id))
return Conflict(new WebResult().Invalidate(localizer["A restore waits for this backup."]));
return backups.Delete(id) ? Ok() : NotFound();
}
// a link to download it with, for the password
[HttpPost, Route("{id}/ticket")]
public async Task<IActionResult> Ticket(string id, BackupPasswordForm form, CancellationToken token)
{
var admin = await Administrator(token);
if (admin == default)
return Forbid();
if (!PasswordHolds(admin, form.Password))
return UnprocessableEntity(new WebResult().Invalidate(localizer["Wrong password."]));
if (backups.Find(id) == default)
return NotFound();
var (ticket, expires) = transfers.Ticket(id);
return Ok(new ViewBackupTicket { Url = $"/clientapi/admin/backups/download/{ticket}", ExpiresAt = expires, Bytes = BackupTar.Of(backups.Root, id).Length });
}
// the backup as one tar, for a ticket: a plain link, so the browser downloads it to disk and resumes it
[HttpGet, Route("download/{ticket}"), AllowAnonymous]
public async Task Download(string ticket, CancellationToken token)
{
var id = transfers.Redeem(ticket);
if (id == default || backups.Find(id) == default)
{
Response.StatusCode = StatusCodes.Status404NotFound;
return;
}
HttpContext.Features.Get<IHttpResponseBodyFeature>()?.DisableBuffering();
var tar = BackupTar.Of(backups.Root, id);
var (from, to) = (0L, tar.Length - 1);
Response.Headers.AcceptRanges = "bytes";
Response.Headers.CacheControl = "no-store";
Response.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment") { FileName = $"privapub-{id}.tar" }.ToString();
var range = Request.GetTypedHeaders().Range;
if (range is { Unit.Value: "bytes", Ranges.Count: 1 })
{
var asked = range.Ranges.First();
var start = asked.From ?? tar.Length - asked.To.GetValueOrDefault();
var end = asked.From.HasValue ? Math.Min(asked.To ?? tar.Length - 1, tar.Length - 1) : tar.Length - 1;
if (start < 0 || start > end)
{
Response.StatusCode = StatusCodes.Status416RangeNotSatisfiable;
Response.Headers.ContentRange = $"bytes */{tar.Length}";
return;
}
(from, to) = (start, end);
Response.StatusCode = StatusCodes.Status206PartialContent;
Response.Headers.ContentRange = $"bytes {from}-{to}/{tar.Length}";
}
Response.ContentType = "application/x-tar";
Response.ContentLength = to - from + 1;
await tar.Write(Response.Body, from, to, token);
}
[HttpPost, Route("uploads")]
public async Task<IActionResult> StartUpload(CancellationToken token)
{
var admin = await Administrator(token);
if (admin == default)
return Forbid();
return Ok(new ViewBackupUpload { Id = transfers.Start(admin.UserName), ChunkBytes = TransferStore.ChunkBytes });
}
[HttpGet, Route("uploads/{id}")]
public async Task<IActionResult> Upload(string id, CancellationToken token)
{
if (await Administrator(token) == default)
return Forbid();
var received = transfers.Received(id);
return received < 0 ? NotFound() : Ok(new ViewBackupUpload { Id = id, Received = received, ChunkBytes = TransferStore.ChunkBytes });
}
// a piece, at the offset already received: 409 with what was received when it isn't
[HttpPut, Route("uploads/{id}"), RequestSizeLimit(TransferStore.ChunkBytes + 1024 * 1024)]
public async Task<IActionResult> Append(string id, [FromQuery] long offset, CancellationToken token)
{
if (await Administrator(token) == default)
return Forbid();
var (received, taken) = await transfers.Append(id, offset, Request.Body, token);
if (received < 0)
return NotFound();
var view = new ViewBackupUpload { Id = id, Received = received, ChunkBytes = TransferStore.ChunkBytes };
return taken ? Ok(view) : Conflict(view);
}
[HttpPost, Route("uploads/{id}/finish")]
public async Task<IActionResult> FinishUpload(string id, CancellationToken token)
{
if (await Administrator(token) == default)
return Forbid();
if (await MaintenanceLock.Current(token) is { What: "restore" })
return Conflict(new WebResult().Invalidate(localizer["A {0} is running.", "restore"]));
var (backup, error) = await transfers.Finish(id, token);
if (backup == default)
return UnprocessableEntity(new WebResult().Invalidate(error));
return Ok(View(backup, 0, ServerBackup.CodeMigration()));
}
[HttpDelete, Route("uploads/{id}")]
public async Task<IActionResult> CancelUpload(string id, CancellationToken token)
{
if (await Administrator(token) == default)
return Forbid();
return transfers.Cancel(id) ? Ok() : NotFound();
}
// asked for with the password and the host typed out: the service stops within seconds and restores it as it starts
[HttpPost, Route("{id}/restore")]
public async Task<IActionResult> Restore(string id, RestoreBackupForm form, CancellationToken token)
{
var admin = await Administrator(token);
if (admin == default)
return Forbid();
if (!PasswordHolds(admin, form.Password))
return UnprocessableEntity(new WebResult().Invalidate(localizer["Wrong password."]));
if (!string.Equals(form.Host?.Trim(), HostName, StringComparison.OrdinalIgnoreCase))
return UnprocessableEntity(new WebResult().Invalidate(localizer["Type this server's host to restore it."]));
var refused = await ServerRestore.Request(backups.Context(), id, admin.UserName, token);
if (refused != default)
return UnprocessableEntity(new WebResult().Invalidate(refused));
logger.LogWarning("{Admin} asked to restore {Backup}", admin.UserName, id);
return Accepted();
}
string HostName => Uri.TryCreate(backups.Host, UriKind.Absolute, out var host) ? host.Authority : backups.Host;
// the token says administrator; the database has the last word
async Task<RootUser> Administrator(CancellationToken token)
{
var root = await DB.Default.Find<RootUser>().MatchID(User.GetUserId()).ExecuteFirstAsync(token);
return root is { IsBanned: false, DeletedAt: null } && root.Policies.Contains(Policies.IsAdmin) ? root : default;
}
bool PasswordHolds(RootUser root, string password) =>
!string.IsNullOrEmpty(password) && !string.IsNullOrEmpty(root.HashedPassword) && hasher.Check(root.HashedPassword, password).verified;
static ViewBackup View(BackupInfo backup, int rootsDeletedSince, int code)
{
var manifest = backup.Manifest;
return new ViewBackup
{
Id = backup.Id,
Kind = backup.Kind,
CreatedAt = backup.CreatedAt,
Bytes = backup.Bytes,
MediaBytes = manifest.Media.Bytes,
MediaFiles = manifest.DbOnly ? manifest.Media.List.Count : manifest.Media.Files,
MediaMissing = manifest.Media.Missing,
DbOnly = manifest.DbOnly,
Consistent = manifest.Consistent,
Collections = manifest.Collections.Count,
Documents = manifest.Collections.Sum(c => c.Count),
AppCommit = manifest.AppCommit,
MigrationNumber = manifest.MigrationNumber,
RootsDeletedSince = rootsDeletedSince,
NotRestorable = manifest.Format != ArchiveManifest.CurrentFormat ? "format"
: manifest.MigrationNumber > code ? "newer"
: default
};
}
static ViewRestore View(RestoreRecord record) => new()
{
Backup = record.Backup,
BackupCreatedAt = record.BackupCreatedAt,
PreRestore = record.PreRestore,
RequestedBy = record.RequestedBy,
RestoredAt = record.RestoredAt,
Abandoned = record.Abandoned,
Error = record.Error,
Documents = record.Report.Documents,
Collections = record.Report.Collections,
MediaRestored = record.Report.MediaRestored,
MediaMissing = record.Report.MediaMissing,
RootsDeleted = record.Report.RootsDeleted,
PersonasDeleted = record.Report.PersonasDeleted,
GroupsDeleted = record.Report.GroupsDeleted,
PostsDeleted = record.Report.PostsDeleted,
RootsTombstoned = record.Report.RootsTombstoned,
PersonasTombstoned = record.Report.PersonasTombstoned,
GroupsTombstoned = record.Report.GroupsTombstoned,
PostsGone = record.Report.PostsGone,
MediaTrashed = record.Report.MediaTrashed,
ProtectiveKept = record.Report.ProtectiveKept,
SessionsEnded = record.Report.SessionsEnded
};
}
}
@@ -19,11 +19,13 @@ namespace PrivaPub.Controllers.ClientToServer
{
readonly IDomainBlocks _domainBlocks;
readonly IStringLocalizer _localizer;
readonly Domain.Media.IMediaProxy _proxy;
public DomainBlockController(IDomainBlocks domainBlocks, IStringLocalizer<GenericRes> localizer)
public DomainBlockController(IDomainBlocks domainBlocks, IStringLocalizer<GenericRes> localizer, Domain.Media.IMediaProxy proxy)
{
_domainBlocks = domainBlocks;
_localizer = localizer;
_proxy = proxy;
}
[HttpGet, Route("/clientapi/admin/domainblocks/list")]
@@ -48,6 +50,9 @@ namespace PrivaPub.Controllers.ClientToServer
.Option(o => o.IsUpsert = true)
.ExecuteAsync(token);
await _domainBlocks.Reload(token);
// what the media proxy holds of it goes with the block
if (block.Severity == DomainBlockSeverity.Suspend || block.RejectMedia)
_proxy.Purge(domain);
return Ok(ToView(block));
}
@@ -31,7 +31,9 @@ namespace PrivaPub.Controllers.ClientToServer
Forwarded = r.Forwarded,
IsResolved = r.IsResolved,
CreatedAt = r.CreatedAt,
ResolvedAt = r.ResolvedAt
ResolvedAt = r.ResolvedAt,
RemoteResolvedAt = r.RemoteResolvedAt,
RemoteResolvedBy = r.RemoteResolvedBy
}));
[HttpPost, Route("/clientapi/moderator/reports/{id}/resolve")]
@@ -0,0 +1,165 @@
using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Mvc;
using MongoDB.Entities;
using PrivaPub.ClientModels;
using PrivaPub.ClientModels.User;
using PrivaPub.Domain.Portability;
using PrivaPub.Extensions;
using PrivaPub.Infrastructure.Backup;
using PrivaPub.Infrastructure.Jobs;
using PrivaPub.Models.User;
namespace PrivaPub.Controllers.ClientToServer
{
// A persona's archive for its root (owner decision 2026-10-07): made in a job and downloaded through a link good for
// ten minutes; another archive (its own, or Mastodon's) uploaded in pieces and imported into it in a job, the parts the
// root picks, with progress, until it asks to stop. Only the persona's own root, never a banned one.
[ApiController,
Route("clientapi/persona"),
Authorize(Policy = Policies.IsUser)]
public class PersonaArchiveController(Backups backups, TransferStore transfers, IJobQueue jobs, Federation.Actors.ILocalActorService localActors)
: ControllerBase
{
[HttpGet, Route("{avatarId}/archive")]
public async Task<IActionResult> Status(string avatarId, CancellationToken token)
{
if (!await Owns(avatarId, token))
return NotFound();
await PersonaArchives.Forget(backups.Root, token);
return Ok(PersonaArchives.View(await PersonaArchives.Of(avatarId, token)));
}
[HttpPost, Route("{avatarId}/archive")]
public async Task<IActionResult> Export(string avatarId, CancellationToken token)
{
if (!await Owns(avatarId, token))
return NotFound();
var state = await PersonaArchives.Of(avatarId, token);
if (state.ExportState is "queued" or "running")
return Conflict(new WebResult().Invalidate("The archive is being made."));
state.ExportState = "queued";
state.ExportAskedAt = DateTime.UtcNow;
state.ExportError = default;
await DB.Default.SaveAsync(state, token);
await jobs.EnqueueMany([ExportArchiveJob.JobFor(avatarId, Host)], token);
return Accepted(PersonaArchives.View(state));
}
[HttpPost, Route("{avatarId}/archive/ticket")]
public async Task<IActionResult> Ticket(string avatarId, CancellationToken token)
{
if (!await Owns(avatarId, token))
return NotFound();
var state = await PersonaArchives.Of(avatarId, token);
var file = state.ExportFile == default ? default : Path.Combine(PersonaArchives.Folder(backups.Root, avatarId), state.ExportFile);
if (state.ExportState != "ready" || file == default || !System.IO.File.Exists(file))
return NotFound();
var (ticket, expires) = transfers.Ticket($"persona/{avatarId}/{state.ExportFile}");
return Ok(new ViewArchiveTicket { Url = $"/clientapi/persona/archive/download/{ticket}", ExpiresAt = expires, Bytes = new FileInfo(file).Length });
}
// the archive itself, for a ticket: a plain link, so the browser saves it to disk (and resumes it)
[HttpGet, Route("archive/download/{ticket}"), AllowAnonymous]
public IActionResult Download(string ticket)
{
if (transfers.Redeem(ticket) is not { } what || what.Split('/') is not ["persona", var avatarId, var name] || name.Contains(".."))
return NotFound();
var file = Path.Combine(PersonaArchives.Folder(backups.Root, avatarId), name);
if (!System.IO.File.Exists(file))
return NotFound();
Response.Headers.CacheControl = "no-store";
return PhysicalFile(file, "application/zip", name, enableRangeProcessing: true);
}
[HttpPost, Route("archive/uploads")]
public IActionResult StartUpload() =>
Ok(new ViewArchiveUpload { Id = transfers.Start(User.GetUserId()), ChunkBytes = TransferStore.ChunkBytes });
[HttpGet, Route("archive/uploads/{id}")]
public IActionResult Upload(string id)
{
if (transfers.Owner(id) != User.GetUserId())
return NotFound();
return Ok(new ViewArchiveUpload { Id = id, Received = transfers.Received(id), ChunkBytes = TransferStore.ChunkBytes });
}
[HttpPut, Route("archive/uploads/{id}"), RequestSizeLimit(TransferStore.ChunkBytes + 1024 * 1024)]
public async Task<IActionResult> Append(string id, [FromQuery] long offset, CancellationToken token)
{
if (transfers.Owner(id) != User.GetUserId())
return NotFound();
var (received, taken) = await transfers.Append(id, offset, Request.Body, token);
var view = new ViewArchiveUpload { Id = id, Received = received, ChunkBytes = TransferStore.ChunkBytes };
return taken ? Ok(view) : Conflict(view);
}
[HttpDelete, Route("archive/uploads/{id}")]
public IActionResult CancelUpload(string id)
{
if (transfers.Owner(id) != User.GetUserId())
return NotFound();
transfers.Cancel(id);
return Ok();
}
// an uploaded archive imported into the persona: refused at once when it is not one, or holds what it must not
[HttpPost, Route("{avatarId}/archive/import")]
public async Task<IActionResult> Import(string avatarId, ImportPersonaArchiveForm form, CancellationToken token)
{
if (!await Owns(avatarId, token))
return NotFound();
var state = await PersonaArchives.Of(avatarId, token);
if (state.ImportState is "queued" or "running")
return Conflict(new WebResult().Invalidate("An import is running."));
if (form.Parts.Count == 0 || form.Parts.Any(p => !PersonaArchives.Parts.Contains(p)))
return UnprocessableEntity(new WebResult().Invalidate("Choose what to import."));
if (transfers.Owner(form.UploadId) != User.GetUserId())
return NotFound();
var path = transfers.Take(form.UploadId, PersonaArchives.Folder(backups.Root, avatarId), $"import-{form.UploadId}.zip");
if (path == default)
return NotFound();
var (archive, refused) = SafeArchive.Open(path);
using (archive)
if (archive?.Has("actor.json") != true)
{
System.IO.File.Delete(path);
return UnprocessableEntity(new WebResult().Invalidate(refused ?? "the archive has no actor.json"));
}
state.ImportState = "queued";
state.ImportAskedAt = DateTime.UtcNow;
state.ImportEndedAt = default;
state.ImportParts = [.. form.Parts.Distinct()];
state.ImportFile = path;
state.ImportCounts = [];
(state.ImportDone, state.ImportTotal, state.ImportError, state.StopAsked) = (0, 0, default, false);
await DB.Default.SaveAsync(state, token);
await jobs.EnqueueMany([ImportArchiveJob.JobFor(avatarId, Host)], token);
return Accepted(PersonaArchives.View(state));
}
[HttpPost, Route("{avatarId}/archive/import/stop")]
public async Task<IActionResult> Stop(string avatarId, CancellationToken token)
{
if (!await Owns(avatarId, token))
return NotFound();
await DB.Default.Update<PersonaArchive>().MatchID(avatarId).Match(a => a.ImportState == "queued" || a.ImportState == "running")
.Modify(a => a.StopAsked, true).ExecuteAsync(token);
return Ok();
}
string Host => new Uri(localActors.BaseAddress).Host;
// the root's own persona, and a root that may still act
async Task<bool> Owns(string avatarId, CancellationToken token)
{
var rootId = User.GetUserId();
if (string.IsNullOrEmpty(avatarId) || !await DB.Default.Find<RootToAvatar>().Match(r => r.RootId == rootId && r.AvatarId == avatarId).ExecuteAnyAsync(token))
return false;
var root = await DB.Default.Find<RootUser>().MatchID(rootId).ExecuteFirstAsync(token);
return root is { IsBanned: false, DeletedAt: null }
&& await DB.Default.Find<Avatar>().MatchID(avatarId).Match(a => !a.DeletionAt.HasValue).ExecuteAnyAsync(token);
}
}
}
@@ -328,6 +328,15 @@ namespace PrivaPub.Controllers.ClientToServer
// }
//}
// what the login's media take, every persona's, and its quota when the server sets one
[HttpGet, Route("/clientapi/user/storage"), Authorize(Policy = Policies.IsUser)]
public async Task<IActionResult> GetStorage([FromServices] Microsoft.Extensions.Options.IOptionsMonitor<Domain.Media.MediaOptions> media, CancellationToken token)
{
var (bytes, files) = await Domain.Privacy.Counted.MediaOfRoot(User.GetUserId(), token);
var quota = media.CurrentValue.QuotaBytesPerRoot;
return Ok(new PrivaPub.ClientModels.User.ViewStorage { MediaBytes = bytes, MediaFiles = files, QuotaBytes = quota > 0 ? quota : null });
}
[HttpGet, Route("/clientapi/user/settings"), Authorize(Policy = Policies.IsUser)]
public async Task<IActionResult> GetUserSettings()
{
@@ -32,8 +32,16 @@ namespace PrivaPub.Controllers.ClientToServer
Ok(await _queries.Crawler(_options.CurrentValue.Crawler.Enabled, _options.CurrentValue.Crawler.RevisitDays, token));
[HttpGet, Route("/clientapi/admin/statistics/overview")]
public async Task<IActionResult> Overview([FromQuery] int days = 30, CancellationToken token = default) =>
Ok(await _queries.Overview(days, token));
public async Task<IActionResult> Overview([FromServices] Domain.Media.IMediaService media, [FromQuery] int days = 30, CancellationToken token = default)
{
var overview = await _queries.Overview(days, token);
overview.Media.ProxyCacheBytes = Bytes(media.ProxyRoot);
overview.Media.TrashBytes = Bytes(media.TrashRoot);
return Ok(overview);
}
static long Bytes(string root) =>
Directory.Exists(root) ? new DirectoryInfo(root).EnumerateFiles("*", SearchOption.AllDirectories).Sum(f => f.Length) : 0;
[HttpGet, Route("/clientapi/admin/statistics/hosts")]
public async Task<IActionResult> Hosts([FromQuery] int days = 30, [FromQuery] string sort = default, [FromQuery] string software = default,
+21
View File
@@ -0,0 +1,21 @@
using System.Globalization;
namespace PrivaPub.Domain.Media
{
// A focal point as Mastodon's API takes it ("x,y", each within -1..1): finite numbers only, clamped to the frame. A NaN
// once stored here made every status, timeline and Note holding the post fail to serialise, for every viewer.
public static class FocalPoint
{
public static float[] Parse(string value)
{
if (value?.Split(',') is not [var x, var y]
|| !float.TryParse(x, NumberStyles.Float, CultureInfo.InvariantCulture, out var fx) || !float.IsFinite(fx)
|| !float.TryParse(y, NumberStyles.Float, CultureInfo.InvariantCulture, out var fy) || !float.IsFinite(fy))
return default;
return [Math.Clamp(fx, -1, 1), Math.Clamp(fy, -1, 1)];
}
/// <summary>Whether a stored focal point can be written out: two finite numbers.</summary>
public static bool IsSound(float[] focus) => focus is [var x, var y] && float.IsFinite(x) && float.IsFinite(y);
}
}
+101
View File
@@ -0,0 +1,101 @@
using MongoDB.Entities;
using PrivaPub.Models.Media;
using PrivaPub.Models.User;
using PostEntity = PrivaPub.Models.Post.Post;
namespace PrivaPub.Domain.Media
{
// What the media directory holds against what the database says holds it, and, with fix, the two made to agree:
// - a persona's current avatar or header with no row (they had none before) becomes a row of its own;
// - media of a deleted post or of a deleted persona go to the trash;
// - rows whose files are gone go to the trash;
// - files nothing holds (deleted posts' media and replaced pictures, from before media were trashed) get a trashed row
// each.
// Trashed files leave what /media/files serves at once, and the janitor deletes them after its grace. Run it without
// fix first; with fix it must run as the user that owns the media (www-data).
public static class MediaAudit
{
public sealed record Report(int Files, int Held, int Adopted, int OfDeleted, int MissingFiles, int Unheld, IReadOnlyList<string> Examples, bool Fixed);
public static async Task<Report> Run(IMediaService media, bool fix, CancellationToken token)
{
var files = Served(media.Root);
var rows = await DB.Default.Find<MediaAttachment>().Match(m => m.TrashedAt == null).ExecuteAsync(token);
var held = rows.SelectMany(r => new[] { r.FilePath, r.PreviewPath }).Where(p => !string.IsNullOrEmpty(p)).ToHashSet();
var examples = new List<string>();
// pictures personas show today, kept as rows from now on
var prefix = media.Url(string.Empty);
var adopted = 0;
foreach (var avatar in await DB.Default.Find<Avatar>().Match(a => !a.DeletionAt.HasValue && (a.PictureURL != null || a.ThumbnailURL != null)).ExecuteAsync(token))
foreach (var (url, kind) in new[] { (avatar.PictureURL, "avatar"), (avatar.ThumbnailURL, "header") })
{
if (url?.StartsWith(prefix, StringComparison.Ordinal) != true)
continue;
var relative = url[prefix.Length..];
if (held.Contains(relative) || !files.Contains(relative))
continue;
adopted++;
held.Add(relative);
if (fix)
await DB.Default.SaveAsync(new MediaAttachment
{
OwnerAvatarId = avatar.ID,
ProfileOfAvatarId = avatar.ID,
Kind = kind,
ContentType = "image/jpeg",
FilePath = relative,
Size = new FileInfo(Path.Combine(media.Root, relative)).Length,
AttachedAt = DateTime.UtcNow
}, token);
}
// held by something that is gone: a deleted post, a deleted persona
var postIds = rows.Where(r => r.PostId != null).Select(r => r.PostId).Distinct().ToList();
var livePosts = (await DB.Default.Find<PostEntity>().Match(p => postIds.Contains(p.ID) && !p.DeletedAt.HasValue).Project(p => p.Include(x => x.ID)).ExecuteAsync(token))
.Select(p => p.ID).ToHashSet();
var ownerIds = rows.Select(r => r.OwnerAvatarId).Where(id => id != null).Distinct().ToList();
var liveOwners = (await DB.Default.Find<Avatar>().Match(a => ownerIds.Contains(a.ID) && !a.DeletionAt.HasValue).Project(a => a.Include(x => x.ID)).ExecuteAsync(token))
.Select(a => a.ID).ToHashSet();
var ofDeleted = rows.Where(r => (r.PostId != null && !livePosts.Contains(r.PostId)) || (r.OwnerAvatarId != null && !liveOwners.Contains(r.OwnerAvatarId))).ToList();
var missing = rows.Except(ofDeleted).Where(r => !string.IsNullOrEmpty(r.FilePath) && !files.Contains(r.FilePath)).ToList();
if (fix)
foreach (var row in ofDeleted.Concat(missing))
await media.Trash(m => m.ID == row.ID, ofDeleted.Contains(row) ? "audit: its holder is deleted" : "audit: its file is missing", token);
foreach (var row in ofDeleted)
held.Remove(row.FilePath);
// files nothing holds
var unheld = files.Where(f => !held.Contains(f)).OrderBy(f => f, StringComparer.Ordinal).ToList();
examples.AddRange(unheld.Take(10));
if (fix)
foreach (var relative in unheld)
{
var orphan = new MediaAttachment
{
Kind = "orphan",
FilePath = relative,
Size = new FileInfo(Path.Combine(media.Root, relative)).Length,
TrashedAt = DateTime.UtcNow,
TrashReason = "audit: nothing holds it"
};
await DB.Default.SaveAsync(orphan, token);
media.Hide(orphan);
}
return new Report(files.Count, rows.Count, adopted, ofDeleted.Count, missing.Count, unheld.Count, examples, fix);
}
// every file /media/files serves: everything under the root
static HashSet<string> Served(string root)
{
if (!Directory.Exists(root))
return new HashSet<string>();
return Directory.EnumerateFiles(root, "*", SearchOption.AllDirectories)
.Select(f => Path.GetRelativePath(root, f).Replace('\\', '/'))
.ToHashSet();
}
}
}
+8
View File
@@ -9,5 +9,13 @@ namespace PrivaPub.Domain.Media
public long ProxyCacheBytes { get; set; } = 5L * 1024 * 1024 * 1024;
public int MaxImageSide { get; set; } = 4096;
public int PreviewSide { get; set; } = 640;
public long MaxPixels { get; set; } = 40_000_000;//an image (or a GIF's frames together) may not decode to more
public int MaxFrames { get; set; } = 500;
public int MaxGifSide { get; set; } = 1280;//a GIF becomes an mp4 at most this wide
public int Concurrency { get; set; } = 2;//uploads processed at once
public long MaxVideoPixels { get; set; } = 2_304_000;//a larger video is made smaller (as Mastodon's video_matrix_limit)
public double MaxFrameRate { get; set; } = 60;
public int MaxSeconds { get; set; } = 3600;//audio or video may not last longer
public long QuotaBytesPerRoot { get; set; }//what a login's media may take (0: no quota)
}
}
+256 -49
View File
@@ -1,3 +1,5 @@
using PrivaPub.Models.Federation;
using System.Collections.Concurrent;
using Microsoft.Extensions.Options;
using MongoDB.Entities;
@@ -11,32 +13,77 @@ using System.Text;
namespace PrivaPub.Domain.Media
{
public enum ProxyOutcome
{
Cached,
TooBig,
Failed
}
public interface IMediaProxy
{
string Wrap(string remoteUrl);
Task<(string Path, string ContentType)> Fetch(string signature, string encodedUrl, CancellationToken token);
string Verified(string signature, string encodedUrl);
/// <summary>Whether a remote file's server is suspended or has its media rejected: nothing of it is proxied.</summary>
bool Refuses(string url);
(string Path, string ContentType) Cached(string url);
/// <summary>Downloads a remote file into the cache, once however many ask at the same time.</summary>
Task<(ProxyOutcome Outcome, string Path, string ContentType)> Download(string url, CancellationToken token);
Task<(string Path, string ContentType)> Fetch(string signature, string encodedUrl, CancellationToken token);
Task<HttpResponseMessage> Open(string url, System.Net.Http.Headers.RangeHeaderValue range, CancellationToken token);
/// <summary>Deletes what the cache holds of a domain and its subdomains; how many files.</summary>
int Purge(string domain);
/// <summary>Trims the cache to its size, oldest first.</summary>
void Trim();
}
// Remote media, fetched for clients so that they never contact a remote server (CLAUDE.md, media invariants):
// - URLs are HMAC-signed, so only what PrivaPub showed is fetched;
// - a download is shared by everyone asking for the same URL at once, streamed into a temporary file beside its place
// and renamed there (a reader never sees half a file), and never held in memory;
// - what is too big to cache is remembered for an hour (it is streamed instead), what failed for five minutes;
// - the cache's size is counted as it grows, and trimmed as soon as it passes the cap;
// - nothing of a suspended server, or of one whose media are rejected, is proxied, and blocking one purges its files.
public class MediaProxy : IMediaProxy
{
static readonly TimeSpan TooBigFor = TimeSpan.FromHours(1);
static readonly TimeSpan FailedFor = TimeSpan.FromMinutes(5);
const int Downloads = 8;//remote files fetched at once, however many clients ask
readonly ILocalActorService _localActors;
readonly IFederationHttp _http;
readonly IMediaService _media;
readonly IOptionsMonitor<MediaOptions> _options;
readonly Federation.Moderation.IDomainBlocks _domainBlocks;
// Lazy: GetOrAdd may run its factory more than once when asked at the same time, but only one Lazy is ever evaluated
readonly ConcurrentDictionary<string, Lazy<Task<(ProxyOutcome, string, string)>>> _inFlight = new();
readonly ConcurrentDictionary<string, (ProxyOutcome Outcome, DateTime Until)> _refused = new();
readonly SemaphoreSlim _downloads = new(Downloads);
readonly object _keyLock = new();
byte[] _key;
long _bytes = -1;//what the cache holds, counted once from the disk and then as it changes
int _trimming;
public MediaProxy(ILocalActorService localActors, IFederationHttp http, IMediaService media, IOptionsMonitor<MediaOptions> options)
public MediaProxy(ILocalActorService localActors, IFederationHttp http, IMediaService media, IOptionsMonitor<MediaOptions> options,
Federation.Moderation.IDomainBlocks domainBlocks = default)
{
_localActors = localActors;
_http = http;
_media = media;
_options = options;
_domainBlocks = domainBlocks;
}
byte[] Key => _key ??= LoadKey();
byte[] Key
{
get
{
if (_key != default)
return _key;
lock (_keyLock)
return _key ??= LoadKey();
}
}
public string Wrap(string remoteUrl)
{
@@ -60,18 +107,186 @@ namespace PrivaPub.Domain.Media
return CryptographicOperations.FixedTimeEquals(Encoding.ASCII.GetBytes(signature ?? string.Empty), Encoding.ASCII.GetBytes(Sign(url))) ? url : default;
}
public bool Refuses(string url)
{
if (_domainBlocks == default || !Uri.TryCreate(url, UriKind.Absolute, out var target))
return false;
var block = _domainBlocks.Find(target.Host);
return block is { Severity: DomainBlockSeverity.Suspend } or { RejectMedia: true };
}
public (string Path, string ContentType) Cached(string url)
{
var (path, typePath) = CachePaths(url);
try
{
if (!File.Exists(path) || !File.Exists(typePath))
return default;
File.SetLastWriteTimeUtc(path, DateTime.UtcNow);
return (path, File.ReadAllText(typePath));
return (path, File.ReadLines(typePath).FirstOrDefault());
}
catch (IOException)
{
return default;//trimmed meanwhile
}
}
public Task<HttpResponseMessage> Open(string url, System.Net.Http.Headers.RangeHeaderValue range, CancellationToken token) =>
_http.OpenMedia(url, range, token);
public async Task<(string Path, string ContentType)> Fetch(string signature, string encodedUrl, CancellationToken token)
{
var url = Verified(signature, encodedUrl);
if (url == default || Refuses(url))
return default;
var (outcome, path, contentType) = await Download(url, token);
return outcome == ProxyOutcome.Cached ? (path, contentType) : default;
}
public async Task<(ProxyOutcome Outcome, string Path, string ContentType)> Download(string url, CancellationToken token)
{
if (Cached(url) is { Path: not null } cached)
return (ProxyOutcome.Cached, cached.Path, cached.ContentType);
if (_refused.TryGetValue(url, out var refused))
{
if (refused.Until > DateTime.UtcNow)
return (refused.Outcome, default, default);
_refused.TryRemove(url, out _);
}
// shared by everyone asking for it now; the download itself isn't cancelled when one of them leaves
var shared = _inFlight.GetOrAdd(url, key => new Lazy<Task<(ProxyOutcome, string, string)>>(() => DownloadOnce(key)));
try
{
return await shared.Value.WaitAsync(token);
}
finally
{
if (shared.Value.IsCompleted)
_inFlight.TryRemove(new KeyValuePair<string, Lazy<Task<(ProxyOutcome, string, string)>>>(url, shared));
}
}
async Task<(ProxyOutcome, string, string)> DownloadOnce(string url)
{
await Task.Yield();
if (!await _downloads.WaitAsync(TimeSpan.FromSeconds(30)))
return (ProxyOutcome.Failed, default, default);//too busy: the client tries again
// cached by a download that ended just before this one began
if (Cached(url) is { Path: not null } done)
{
_downloads.Release();
return (ProxyOutcome.Cached, done.Path, done.ContentType);
}
var (path, typePath) = CachePaths(url);
var part = $"{path}.{Guid.NewGuid():N}.part";
try
{
Directory.CreateDirectory(System.IO.Path.GetDirectoryName(path)!);
string contentType, refusal;
await using (var file = new FileStream(part, FileMode.CreateNew, FileAccess.Write, FileShare.None, 64 * 1024, useAsync: true))
(contentType, refusal) = await _http.DownloadMedia(url, _options.CurrentValue.MaxProxiedBytes, file, CancellationToken.None);
if (contentType == default)
{
var outcome = refusal == "too-large" ? ProxyOutcome.TooBig : ProxyOutcome.Failed;
_refused[url] = (outcome, DateTime.UtcNow + (outcome == ProxyOutcome.TooBig ? TooBigFor : FailedFor));
return (outcome, default, default);
}
await File.WriteAllTextAsync(typePath, contentType + "\n" + new Uri(url).Host);
var size = new FileInfo(part).Length;
File.Move(part, path, overwrite: true);
Grew(size);
return (ProxyOutcome.Cached, path, contentType);
}
finally
{
_downloads.Release();
if (File.Exists(part))
File.Delete(part);
}
}
void Grew(long bytes)
{
if (Interlocked.Read(ref _bytes) < 0)
Interlocked.CompareExchange(ref _bytes, Measure(), -1);
if (Interlocked.Add(ref _bytes, bytes) > _options.CurrentValue.ProxyCacheBytes)
_ = Task.Run(Trim);
}
public void Trim()
{
if (Interlocked.Exchange(ref _trimming, 1) == 1)
return;
try
{
Interlocked.Exchange(ref _bytes, TrimDirectory(_media.ProxyRoot, _options.CurrentValue.ProxyCacheBytes));
}
finally
{
Interlocked.Exchange(ref _trimming, 0);
}
}
public int Purge(string domain)
{
var directory = new DirectoryInfo(_media.ProxyRoot);
if (string.IsNullOrEmpty(domain) || !directory.Exists)
return 0;
var purged = 0;
foreach (var type in directory.EnumerateFiles("*.type", SearchOption.AllDirectories))
{
try
{
var host = File.ReadLines(type.FullName).Skip(1).FirstOrDefault();
if (host == default || !(host.Equals(domain, StringComparison.OrdinalIgnoreCase) || host.EndsWith("." + domain, StringComparison.OrdinalIgnoreCase)))
continue;
var file = new FileInfo(type.FullName[..^".type".Length]);
if (file.Exists)
file.Delete();
type.Delete();
purged++;
}
catch (IOException)
{
}
}
Interlocked.Exchange(ref _bytes, -1);
return purged;
}
long Measure()
{
var directory = new DirectoryInfo(_media.ProxyRoot);
return directory.Exists ? directory.EnumerateFiles("*", SearchOption.AllDirectories).Where(f => f.Extension is not (".type" or ".part")).Sum(f => f.Length) : 0;
}
// deletes the oldest files until the cache is within its cap; what it holds afterwards
public static long TrimDirectory(string root, long cap)
{
var directory = new DirectoryInfo(root);
if (!directory.Exists)
return 0;
var files = directory.EnumerateFiles("*", SearchOption.AllDirectories).Where(f => f.Extension is not (".type" or ".part")).OrderBy(f => f.LastWriteTimeUtc).ToList();
var total = files.Sum(f => f.Length);
foreach (var file in files)
{
if (total <= cap)
break;
try
{
total -= file.Length;
file.Delete();
var type = new FileInfo(file.FullName + ".type");
if (type.Exists)
type.Delete();
}
catch (IOException)
{
}
}
return total;
}
(string Path, string TypePath) CachePaths(string url)
{
var name = Convert.ToHexStringLower(SHA256.HashData(Encoding.UTF8.GetBytes(url)));
@@ -79,35 +294,16 @@ namespace PrivaPub.Domain.Media
return (path, path + ".type");
}
public async Task<(string Path, string ContentType)> Fetch(string signature, string encodedUrl, CancellationToken token)
{
var url = Verified(signature, encodedUrl);
if (url == default)
return default;
if (Cached(url) is { Path: not null } cached)
return cached;
var (path, typePath) = CachePaths(url);
var directory = System.IO.Path.GetDirectoryName(path);
var (bytes, contentType) = await _http.GetMedia(url, _options.CurrentValue.MaxProxiedBytes, token);
if (bytes == default)
return default;
Directory.CreateDirectory(directory);
await File.WriteAllBytesAsync(path, bytes, token);
await File.WriteAllTextAsync(typePath, contentType, token);
return (path, contentType);
}
string Sign(string url) => Base64Url(HMACSHA256.HashData(Key, Encoding.UTF8.GetBytes(url))[..16]);
// the oldest key, so that two first uses at once agree on one
static byte[] LoadKey()
{
var secret = DB.Default.Find<MediaSecret>().ExecuteFirstAsync().GetAwaiter().GetResult();
var secret = DB.Default.Find<MediaSecret>().Sort(m => m.ID, Order.Ascending).ExecuteFirstAsync().GetAwaiter().GetResult();
if (secret == default)
{
secret = new MediaSecret { Key = Convert.ToBase64String(RandomNumberGenerator.GetBytes(32)) };
DB.Default.SaveAsync(secret).GetAwaiter().GetResult();
secret = DB.Default.Find<MediaSecret>().ExecuteFirstAsync().GetAwaiter().GetResult();
DB.Default.SaveAsync(new MediaSecret { Key = Convert.ToBase64String(RandomNumberGenerator.GetBytes(32)) }).GetAwaiter().GetResult();
secret = DB.Default.Find<MediaSecret>().Sort(m => m.ID, Order.Ascending).ExecuteFirstAsync().GetAwaiter().GetResult();
}
return Convert.FromBase64String(secret.Key);
}
@@ -123,8 +319,11 @@ namespace PrivaPub.Domain.Media
public class MediaJanitor : BackgroundService
{
static readonly TimeSpan FirstPass = TimeSpan.FromMinutes(5);
static readonly TimeSpan Interval = TimeSpan.FromHours(1);
static readonly TimeSpan UnattachedLifetime = TimeSpan.FromDays(1);
// a trashed file waits this long before it is deleted, already out of what /media/files serves
public static readonly TimeSpan TrashGrace = TimeSpan.FromDays(1);
readonly IMediaService _media;
readonly IOptionsMonitor<MediaOptions> _options;
@@ -139,11 +338,13 @@ namespace PrivaPub.Domain.Media
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
var wait = FirstPass;
while (!stoppingToken.IsCancellationRequested)
{
try
{
await Task.Delay(Interval, stoppingToken);
await Task.Delay(wait, stoppingToken);
wait = Interval;
await Sweep(stoppingToken);
}
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
@@ -157,32 +358,38 @@ namespace PrivaPub.Domain.Media
}
}
//one pass: uploads left unattached for a day go (unless a scheduled post waits for them), then the proxy cache is trimmed to its size, oldest first
// one pass:
// - uploads never posted for a day (and not waiting for a scheduled post, nor a profile picture) go to the trash;
// - trashed files still served (a crash between the mark and the move) are moved out;
// - trashed files past their grace are deleted, with their rows, unless a backup or a restore is running (one may be
// reading the trash);
// - the proxy cache is trimmed to its size, oldest first
public async Task Sweep(CancellationToken token)
{
var cutoff = DateTime.UtcNow - UnattachedLifetime;
foreach (var stale in await DB.Default.Find<MediaAttachment>().Match(m => m.PostId == null && m.ScheduledStatusId == null && m.CreatedAt < cutoff).Limit(500).ExecuteAsync(token))
await _media.Delete(stale);
var unattached = await _media.Trash(m => m.PostId == null && m.ScheduledStatusId == null && m.ProfileOfAvatarId == null && m.CreatedAt < cutoff,
"never posted", token);
var trashed = await DB.Default.Find<MediaAttachment>().Match(m => m.TrashedAt != null).Limit(2000).ExecuteAsync(token);
var purgeBefore = DateTime.UtcNow - TrashGrace;
var maintenance = await Infrastructure.Backup.MaintenanceLock.Current(token) != default;
var purged = 0;
foreach (var row in trashed)
{
if (row.TrashedAt < purgeBefore && !maintenance)
{
await _media.Purge(row, token);
purged++;
}
else
_media.Hide(row);
}
if (unattached > 0 || purged > 0)
_logger.LogInformation("{Service}: {Unattached} uploads never posted trashed, {Purged} trashed files deleted",
nameof(MediaJanitor), unattached, purged);
TrimProxyCache();
}
void TrimProxyCache()
{
var directory = new DirectoryInfo(_media.ProxyRoot);
if (!directory.Exists)
return;
var files = directory.EnumerateFiles("*", SearchOption.AllDirectories).Where(f => f.Extension != ".type").OrderBy(f => f.LastWriteTimeUtc).ToList();
var total = files.Sum(f => f.Length);
foreach (var file in files)
{
if (total <= _options.CurrentValue.ProxyCacheBytes)
break;
total -= file.Length;
file.Delete();
var type = new FileInfo(file.FullName + ".type");
if (type.Exists)
type.Delete();
}
}
void TrimProxyCache() => MediaProxy.TrimDirectory(_media.ProxyRoot, _options.CurrentValue.ProxyCacheBytes);
}
}
+380 -96
View File
@@ -22,19 +22,46 @@ namespace PrivaPub.Domain.Media
public sealed record ProcessedImage(byte[] Bytes, string Extension, string ContentType, int Width, int Height, byte[] Preview, string Blurhash);
// what libvips makes of an upload's header alone, before anything is decoded
public sealed record ImageHeader(string Loader, int Width, int PageHeight, int Pages)
{
public bool Animated => Pages > 1 && Loader?.StartsWith("gifload", StringComparison.Ordinal) == true;
public long Pixels => (long)Width * PageHeight * (Animated ? Pages : 1);
}
public interface IMediaService
{
string Root { get; }
string ProxyRoot { get; }
string Url(string relativePath);
Task<MediaOutcome> Upload(LocalActor owner, IFormFile file, string description, string focus, CancellationToken token);
Task<string> ProfileImage(IFormFile file, int width, int height, CancellationToken token);
Task Delete(MediaAttachment attachment);
/// <summary>Where trashed files wait for the janitor: the sibling of Root, on the same disk (a move is a rename) and
/// outside what /media/files serves. (Not a dot-prefixed folder inside Root: the file provider only hides a file whose
/// own name starts with a dot.)</summary>
string TrashRoot { get; }
/// <summary>Audio and video waiting for their processing: beside the media root, never served.</summary>
string IncomingRoot { get; }
/// <summary>An upload; with <paramref name="later"/>, audio and video are only stored, "pending" for ProcessMedia.</summary>
Task<MediaOutcome> Upload(LocalActor owner, IFormFile file, string description, string focus, bool later, CancellationToken token);
/// <summary>Processes a pending upload's audio or video; false when it can't be (the row then says why).</summary>
Task<bool> ProcessPending(MediaAttachment pending, CancellationToken token);
/// <summary>A persona's new avatar or header ("avatar" or "header"), cropped to its size, as a row of its own.</summary>
Task<MediaOutcome> ProfileImage(string avatarId, string kind, IFormFile file, int width, int height, CancellationToken token);
/// <summary>Trashes the media <paramref name="which"/> matches (and that isn't trashed yet): each row is marked in one
/// conditional update, so a row attached meanwhile is left alone, and its files stop being served at once.</summary>
Task<long> Trash(System.Linq.Expressions.Expression<Func<MediaAttachment, bool>> which, string reason, CancellationToken token);
/// <summary>Moves a trashed row's files out of what is served, if they are still there (a crash between the mark and the move).</summary>
void Hide(MediaAttachment trashed);
/// <summary>Deletes a trashed row's files and the row.</summary>
Task Purge(MediaAttachment trashed, CancellationToken token);
}
public class MediaService : IMediaService
{
static readonly string[] ImageTypes = { "image/jpeg", "image/png", "image/gif", "image/webp", "image/heic", "image/heif", "image/avif" };
// HEIC and HEIF are not among them: the libvips bundled here decodes AVIF but has no HEVC decoder
static readonly string[] ImageTypes = { "image/jpeg", "image/png", "image/gif", "image/webp", "image/avif" };
// the only loaders an upload ever reaches: everything else libvips could read (SVG, PDF, TIFF, ImageMagick, ...) is
// blocked, whatever type the upload claims
static readonly string[] Loaders = { "VipsForeignLoadJpeg", "VipsForeignLoadPng", "VipsForeignLoadNsgif", "VipsForeignLoadWebp", "VipsForeignLoadHeif" };
static readonly Dictionary<string, string> AvTypes = new()
{
["video/mp4"] = "mp4", ["video/quicktime"] = "mp4", ["video/webm"] = "webm",
@@ -42,11 +69,27 @@ namespace PrivaPub.Domain.Media
["audio/mp4"] = "m4a", ["audio/x-m4a"] = "m4a", ["audio/flac"] = "flac", ["audio/webm"] = "webm"
};
// what /media/files says each file is: ASP.NET's map, which has no entry for FLAC
public static Microsoft.AspNetCore.StaticFiles.FileExtensionContentTypeProvider ContentTypes { get; } = new()
{
Mappings = { [".flac"] = "audio/flac" }
};
// what an upload may be, as the instance API advertises it
public static IReadOnlyList<string> SupportedTypes => ImageTypes.Concat(AvTypes.Keys).ToList();
static MediaService()
{
NetVips.NetVips.BlockUntrusted = true;
Operation.Block("VipsForeignLoad", true);
foreach (var loader in Loaders)
Operation.Block(loader, false);
Cache.Max = 0;//an upload is decoded once: nothing to keep between operations
}
readonly IOptionsMonitor<MediaOptions> _options;
readonly ILocalActorService _localActors;
readonly SemaphoreSlim _processing;
readonly IWebHostEnvironment _environment;
readonly ILogger<MediaService> _logger;
@@ -54,6 +97,7 @@ namespace PrivaPub.Domain.Media
{
_options = options;
_localActors = localActors;
_processing = new SemaphoreSlim(Math.Max(1, options.CurrentValue.Concurrency));
_environment = environment;
_logger = logger;
}
@@ -62,39 +106,50 @@ namespace PrivaPub.Domain.Media
public string ProxyRoot => Root.TrimEnd(Path.DirectorySeparatorChar) + "-proxy";
public string TrashRoot => Root.TrimEnd(Path.DirectorySeparatorChar) + "-trash";
public string IncomingRoot => Root.TrimEnd(Path.DirectorySeparatorChar) + "-incoming";
string Incoming(string id) => Path.Combine(IncomingRoot, id + ".in");
public string Url(string relativePath) => relativePath == default ? default : $"{_localActors.BaseAddress}/media/files/{relativePath.Replace('\\', '/')}";
public async Task<MediaOutcome> Upload(LocalActor owner, IFormFile file, string description, string focus, CancellationToken token)
public async Task<MediaOutcome> Upload(LocalActor owner, IFormFile file, string description, string focus, bool later, CancellationToken token)
{
if (file == default || file.Length == 0)
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: File can't be blank");
var options = _options.CurrentValue;
var contentType = file.ContentType?.Split(';')[0].Trim().ToLowerInvariant();
if (await OverQuota(owner.Id, file.Length, token) is { } full)
return full;
var attachment = new MediaAttachment
{
OwnerAvatarId = owner.Id,
Description = Clean(description, 1500),
Focus = Focus(focus)
Focus = FocalPoint.Parse(focus)
};
if (ImageTypes.Contains(contentType))
{
if (file.Length > options.MaxImageBytes)
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: File is too big");
await using var stream = file.OpenReadStream();
using var buffer = new MemoryStream();
await stream.CopyToAsync(buffer, token);
ProcessedImage processed;
var bytes = await Read(file, token);
if (Refusal(bytes, options) is { } refused)
return refused;
await _processing.WaitAsync(token);
try
{
processed = ProcessImage(buffer.ToArray(), options.MaxImageSide, options.PreviewSide, contentType == "image/gif");
}
catch (VipsException ex)
var header = Inspect(bytes);
if (header.Animated)
{
_logger.LogInformation("Refused an image upload: {Error}", ex.Message);
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: The file is not a readable image");
var animated = await ProcessAnimation(bytes, header, attachment, token);
if (!animated.Ok)
return animated;
}
attachment.Kind = contentType == "image/gif" ? "gifv" : "image";
else
{
var processed = ProcessImage(bytes, options.MaxImageSide, options.PreviewSide);
attachment.Kind = "image";
attachment.ContentType = processed.ContentType;
attachment.FilePath = await Save(processed.Bytes, processed.Extension, token);
attachment.PreviewPath = await Save(processed.Preview, "jpg", token);
@@ -103,14 +158,37 @@ namespace PrivaPub.Domain.Media
attachment.Blurhash = processed.Blurhash;
attachment.Size = processed.Bytes.Length;
}
else if (contentType != default && AvTypes.TryGetValue(contentType, out var extension))
}
catch (VipsException ex)
{
_logger.LogInformation("Refused an image upload: {Error}", ex.Message);
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: The file is not a readable image");
}
finally
{
_processing.Release();
}
}
else if (contentType != default && AvTypes.ContainsKey(contentType))
{
if (file.Length > options.MaxVideoBytes)
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: File is too big");
var outcome = await ProcessAv(file, extension, contentType, attachment, token);
// stored as it came, outside what is served, then processed: by a job when asked later (v2), here otherwise (v1)
attachment.ID = (string)attachment.GenerateNewID();
attachment.Kind = contentType.StartsWith("video/", StringComparison.Ordinal) ? "video" : "audio";
attachment.ContentType = contentType;
Directory.CreateDirectory(IncomingRoot);
await using (var stored = File.Create(Incoming(attachment.ID)))
await file.CopyToAsync(stored, token);
if (later)
attachment.ProcessingState = "pending";
else
{
var outcome = await ProcessIncoming(attachment, token);
if (!outcome.Ok)
return outcome;
}
}
else
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: File type is not supported");
@@ -118,68 +196,205 @@ namespace PrivaPub.Domain.Media
return new MediaOutcome(attachment);
}
public async Task<string> ProfileImage(IFormFile file, int width, int height, CancellationToken token)
public async Task<MediaOutcome> ProfileImage(string avatarId, string kind, IFormFile file, int width, int height, CancellationToken token)
{
if (file == default || file.Length == 0 || file.Length > _options.CurrentValue.MaxImageBytes)
return default;
await using var stream = file.OpenReadStream();
using var buffer = new MemoryStream();
await stream.CopyToAsync(buffer, token);
var contentType = file?.ContentType?.Split(';')[0].Trim().ToLowerInvariant();
if (file == default || file.Length == 0)
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: File can't be blank");
if (!ImageTypes.Contains(contentType))
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: File type is not supported");
if (file.Length > _options.CurrentValue.MaxImageBytes)
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: File is too big");
if (await OverQuota(avatarId, file.Length, token) is { } full)
return full;
var input = await Read(file, token);
if (Refusal(input, _options.CurrentValue) is { } refused)
return refused;
byte[] bytes;
int outWidth, outHeight;
await _processing.WaitAsync(token);
try
{
using var image = Image.ThumbnailBuffer(buffer.ToArray(), width, height: height, crop: Enums.Interesting.Centre, size: Enums.Size.Down);
using var image = Image.ThumbnailBuffer(input, width, height: height, crop: Enums.Interesting.Centre, size: Enums.Size.Down);
using var flat = Flatten(image);
return Url(await Save(flat.WriteToBuffer(".jpg[Q=85,keep=none]"), "jpg", token));
bytes = flat.WriteToBuffer(".jpg[Q=85,keep=none]");
(outWidth, outHeight) = (flat.Width, flat.Height);
}
catch (VipsException ex)
{
_logger.LogInformation("Refused a profile picture: {Error}", ex.Message);
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: The file is not a readable image");
}
finally
{
_processing.Release();
}
var attachment = new MediaAttachment
{
OwnerAvatarId = avatarId,
ProfileOfAvatarId = avatarId,
Kind = kind,
ContentType = "image/jpeg",
FilePath = await Save(bytes, "jpg", token),
Size = bytes.Length,
Width = outWidth,
Height = outHeight,
AttachedAt = DateTime.UtcNow
};
await DB.Default.SaveAsync(attachment, token);
return new MediaOutcome(attachment);
}
// a login over its quota (Media:QuotaBytesPerRoot; none when 0) uploads nothing more
async Task<MediaOutcome> OverQuota(string avatarId, long incoming, CancellationToken token)
{
var quota = _options.CurrentValue.QuotaBytesPerRoot;
if (quota <= 0)
return default;
var rootId = (await DB.Default.Find<Models.User.RootToAvatar>().Match(r => r.AvatarId == avatarId).ExecuteFirstAsync(token))?.RootId;
if (rootId == default)
return default;
var (bytes, _) = await Privacy.Counted.MediaOfRoot(rootId, token);
return bytes + incoming > quota ? MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: Your storage is full") : default;
}
public async Task<long> Trash(System.Linq.Expressions.Expression<Func<MediaAttachment, bool>> which, string reason, CancellationToken token)
{
var trashed = 0L;
foreach (var candidate in await DB.Default.Find<MediaAttachment>().Match(which).Match(m => m.TrashedAt == null).ExecuteAsync(token))
{
var marked = await DB.Default.Update<MediaAttachment>()
.Match(which).Match(m => m.ID == candidate.ID && m.TrashedAt == null)
.Modify(m => m.TrashedAt, DateTime.UtcNow)
.Modify(m => m.TrashReason, reason)
.ExecuteAsync(token);
if (marked.ModifiedCount == 0)
continue;
Hide(candidate);
trashed++;
}
return trashed;
}
public void Hide(MediaAttachment trashed)
{
foreach (var relative in new[] { trashed.FilePath, trashed.PreviewPath }.Where(p => !string.IsNullOrEmpty(p)))
{
var served = Path.Combine(Root, relative);
if (!File.Exists(served))
continue;
var hidden = Path.Combine(TrashRoot, relative);
Directory.CreateDirectory(Path.GetDirectoryName(hidden)!);
File.Move(served, hidden, overwrite: true);
}
}
public async Task Purge(MediaAttachment trashed, CancellationToken token)
{
foreach (var relative in new[] { trashed.FilePath, trashed.PreviewPath }.Where(p => !string.IsNullOrEmpty(p)))
foreach (var full in new[] { Path.Combine(TrashRoot, relative), Path.Combine(Root, relative) })
if (File.Exists(full))
File.Delete(full);
if (File.Exists(Incoming(trashed.ID)))
File.Delete(Incoming(trashed.ID));
await DB.Default.DeleteAsync<MediaAttachment>(m => m.ID == trashed.ID && m.TrashedAt != null);
}
// reads only the header: which loader takes the bytes and how big they would decode (throws VipsException when none)
public static ImageHeader Inspect(byte[] input)
{
using var image = Image.NewFromBuffer(input, access: Enums.Access.Sequential);
var pages = image.Contains("n-pages") ? Math.Max(1, (int)image.Get("n-pages")) : 1;
var pageHeight = image.Contains("page-height") ? (int)image.Get("page-height") : image.Height;
return new ImageHeader(image.Contains("vips-loader") ? (string)image.Get("vips-loader") : default, image.Width, pageHeight, pages);
}
// an upload refused before anything is decoded: no loader takes it, or it would decode to too many pixels or frames
static MediaOutcome Refusal(byte[] input, MediaOptions options)
{
ImageHeader header;
try
{
header = Inspect(input);
}
catch (VipsException)
{
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: The file is not a readable image");
}
if (header.Pixels > options.MaxPixels || header.Pages > options.MaxFrames)
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: The image is too large");
return default;
}
}
public Task Delete(MediaAttachment attachment)
// a still image: shrunk on load to the largest side allowed, turned by its orientation, its colours brought into sRGB
// (thumbnail does all three), then written without its metadata, with a preview and a blurhash
public static ProcessedImage ProcessImage(byte[] input, int maxSide, int previewSide)
{
foreach (var path in new[] { attachment.FilePath, attachment.PreviewPath }.Where(p => p != default))
{
var full = Path.Combine(Root, path);
if (File.Exists(full))
File.Delete(full);
}
return DB.Default.DeleteAsync<MediaAttachment>(attachment.ID);
}
public static ProcessedImage ProcessImage(byte[] input, int maxSide, int previewSide, bool animated)
{
using var loaded = animated ? Image.NewFromBuffer(input, kwargs: new VOption { { "n", -1 } }) : Image.NewFromBuffer(input);
using var rotated = animated ? loaded.Copy() : loaded.Autorot();
var pageHeight = animated && rotated.Contains("page-height") ? (int)rotated.Get("page-height") : rotated.Height;
var longest = Math.Max(rotated.Width, pageHeight);
byte[] bytes;
string extension, contentType;
int width, height;
if (animated)
{
bytes = rotated.WriteToBuffer(".gif");
(extension, contentType, width, height) = ("gif", "image/gif", rotated.Width, pageHeight);
}
else
{
using var resized = longest > maxSide ? rotated.ThumbnailImage(maxSide, height: maxSide, size: Enums.Size.Down) : rotated.Copy();
// rendered into memory once (it is read three times below, and a shrunk-on-load image can be read only in order)
using var shrunk = Image.ThumbnailBuffer(input, maxSide, height: maxSide, size: Enums.Size.Down);
using var resized = shrunk.CopyMemory();
var keepsAlpha = resized.HasAlpha();
bytes = keepsAlpha ? resized.WriteToBuffer(".png[keep=none]") : resized.WriteToBuffer(".jpg[Q=88,keep=none]");
(extension, contentType, width, height) = keepsAlpha ? ("png", "image/png", resized.Width, resized.Height) : ("jpg", "image/jpeg", resized.Width, resized.Height);
}
var bytes = keepsAlpha ? resized.WriteToBuffer(".png[keep=none]") : resized.WriteToBuffer(".jpg[Q=88,keep=none]");
var (extension, contentType) = keepsAlpha ? ("png", "image/png") : ("jpg", "image/jpeg");
using var firstFrame = animated ? rotated.Crop(0, 0, rotated.Width, pageHeight) : rotated.Copy();
using var preview = firstFrame.ThumbnailImage(previewSide, height: previewSide, size: Enums.Size.Down);
using var preview = resized.ThumbnailImage(previewSide, height: previewSide, size: Enums.Size.Down);
using var previewFlat = Flatten(preview);
var previewBytes = previewFlat.WriteToBuffer(".jpg[Q=80,keep=none]");
using var tiny = firstFrame.ThumbnailImage(32, height: 32);
using var tiny = resized.ThumbnailImage(32, height: 32);
using var tinyFlat = Flatten(tiny);
var pixels = tinyFlat.WriteToMemory();
return new ProcessedImage(bytes, extension, contentType, width, height, previewBytes, Blurhash.Encode(pixels, tinyFlat.Width, tinyFlat.Height));
return new ProcessedImage(bytes, extension, contentType, resized.Width, resized.Height, previewBytes, Blurhash.Encode(pixels, tinyFlat.Width, tinyFlat.Height));
}
// an animated GIF becomes what Mastodon makes of one: a silent, looping H.264 mp4 (a gifv) at most MaxGifSide wide,
// its first frame the poster
async Task<MediaOutcome> ProcessAnimation(byte[] input, ImageHeader header, MediaAttachment attachment, CancellationToken token)
{
var options = _options.CurrentValue;
var temp = Path.Combine(Path.GetTempPath(), $"privapub-{Guid.NewGuid():N}");
var gif = temp + ".gif";
var mp4 = temp + ".mp4";
try
{
await File.WriteAllBytesAsync(gif, input, token);
var width = Math.Min(header.Width, options.MaxGifSide) / 2 * 2;
var height = Math.Max(2, (int)Math.Round(header.PageHeight * (double)width / header.Width / 2) * 2);
await FFMpegArguments.FromFileInput(gif, true, o => o.ForceFormat("gif").WithCustomArgument("-protocol_whitelist file"))
.OutputToFile(mp4, true, o => o.WithCustomArgument(
$"-an -map_metadata -1 -map_chapters -1 -c:v libx264 -preset veryfast -crf 23 -pix_fmt yuv420p -movflags +faststart -vf scale={width}:{height}"))
.CancellableThrough(token)
.ProcessAsynchronously();
var poster = ProcessImage(input, options.PreviewSide, options.PreviewSide);
attachment.Kind = "gifv";
attachment.ContentType = "video/mp4";
attachment.Width = width;
attachment.Height = height;
attachment.Size = new FileInfo(mp4).Length;
attachment.Blurhash = poster.Blurhash;
attachment.PreviewPath = await Save(poster.Preview, "jpg", token);
attachment.FilePath = SaveFile(mp4, "mp4");
return new MediaOutcome(attachment);
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
_logger.LogInformation(ex, "Refused a GIF upload");
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: The file could not be processed");
}
finally
{
foreach (var path in new[] { gif, mp4 })
if (File.Exists(path))
File.Delete(path);
}
}
static async Task<byte[]> Read(IFormFile file, CancellationToken token)
{
await using var stream = file.OpenReadStream();
using var buffer = new MemoryStream((int)Math.Min(file.Length, int.MaxValue));
await stream.CopyToAsync(buffer, token);
return buffer.ToArray();
}
static Image Flatten(Image image)
@@ -193,37 +408,102 @@ namespace PrivaPub.Domain.Media
return bands.Format == Enums.BandFormat.Uchar ? bands.Copy() : bands.Cast(Enums.BandFormat.Uchar);
}
async Task<MediaOutcome> ProcessAv(IFormFile file, string extension, string contentType, MediaAttachment attachment, CancellationToken token)
public async Task<bool> ProcessPending(MediaAttachment pending, CancellationToken token)
{
var outcome = await ProcessIncoming(pending, token);
if (outcome.Ok)
{
await DB.Default.Update<MediaAttachment>().Match(m => m.ID == pending.ID && m.TrashedAt == null)
.Modify(m => m.Kind, pending.Kind)
.Modify(m => m.ContentType, pending.ContentType)
.Modify(m => m.FilePath, pending.FilePath)
.Modify(m => m.PreviewPath, pending.PreviewPath)
.Modify(m => m.Size, pending.Size)
.Modify(m => m.Width, pending.Width)
.Modify(m => m.Height, pending.Height)
.Modify(m => m.Blurhash, pending.Blurhash)
.Modify(m => m.DurationSeconds, pending.DurationSeconds)
.Modify(m => m.ProcessingState, null)
.ExecuteAsync(token);
return true;
}
await DB.Default.Update<MediaAttachment>().MatchID(pending.ID)
.Modify(m => m.ProcessingState, "failed").Modify(m => m.ProcessingError, outcome.Error).ExecuteAsync(token);
return false;
}
// the stored upload, made playable: probed, then remuxed without its metadata when browsers play it as it is (H.264,
// VP8, VP9 or AV1 within MaxVideoPixels and MaxFrameRate), or else transcoded to H.264 that fits; a frame becomes the
// poster. ffmpeg only ever reads the local file (no protocol but file, its format forced from the probe). Nothing is
// saved until everything succeeded, and every temporary file goes, the stored upload too.
async Task<MediaOutcome> ProcessIncoming(MediaAttachment attachment, CancellationToken token)
{
var options = _options.CurrentValue;
var input = Incoming(attachment.ID);
var temp = Path.Combine(Path.GetTempPath(), $"privapub-{Guid.NewGuid():N}");
var input = temp + ".in";
var output = temp + "." + extension;
var frame = temp + ".png";
var output = default(string);
await _processing.WaitAsync(token);
try
{
await using (var target = File.Create(input))
await file.CopyToAsync(target, token);
var probe = await FFProbe.AnalyseAsync(input, cancellationToken: token);
var isVideo = contentType.StartsWith("video/") && probe.PrimaryVideoStream != default;
await FFMpegArguments.FromFileInput(input)
.OutputToFile(output, true, o => o.WithCustomArgument("-map 0 -map_metadata -1 -map_chapters -1 -c copy" + (extension is "mp4" or "m4a" ? " -movflags +faststart" : string.Empty)))
var probe = await FFProbe.AnalyseAsync(input, default, token, "-protocol_whitelist file");
if (probe.Duration > TimeSpan.FromSeconds(options.MaxSeconds))
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: The file is too long");
var video = attachment.ContentType.StartsWith("video/", StringComparison.Ordinal) ? probe.PrimaryVideoStream : default;
var audio = probe.PrimaryAudioStream;
if (video == default && audio == default)
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: The file could not be processed");
var extension = AvTypes[attachment.ContentType];
var format = probe.Format.FormatName.Split(',')[0];
string arguments;
if (video == default)
arguments = "-map 0:a:0 -vn -dn -sn -map_metadata -1 -map_chapters -1 -c copy" + (extension == "m4a" ? " -movflags +faststart" : string.Empty);
else if (video.CodecName is "h264" or "vp8" or "vp9" or "av1" && (long)video.Width * video.Height <= options.MaxVideoPixels
&& video.FrameRate <= options.MaxFrameRate + 0.5)
arguments = "-map 0:v:0 -map 0:a:0? -dn -sn -map_metadata -1 -map_chapters -1 -c copy" + (extension == "mp4" ? " -movflags +faststart" : string.Empty);
else
{
var scale = Math.Min(1, Math.Sqrt(options.MaxVideoPixels / (double)((long)video.Width * video.Height)));
var width = Math.Max(2, (int)(video.Width * scale) / 2 * 2);
var height = Math.Max(2, (int)(video.Height * scale) / 2 * 2);
extension = "mp4";
arguments = $"-map 0:v:0 -map 0:a:0? -dn -sn -map_metadata -1 -map_chapters -1 -c:v libx264 -preset veryfast -crf 23 -pix_fmt yuv420p "
+ $"-vf scale={width}:{height} -fpsmax {options.MaxFrameRate.ToString(CultureInfo.InvariantCulture)} -c:a aac -b:a 128k -movflags +faststart";
}
output = temp + "." + extension;
await FFMpegArguments.FromFileInput(input, true, o => o.ForceFormat(format).WithCustomArgument("-protocol_whitelist file"))
.OutputToFile(output, true, o => o.WithCustomArgument(arguments))
.CancellableThrough(token)
.ProcessAsynchronously();
attachment.Kind = isVideo ? "video" : "audio";
attachment.ContentType = extension switch { "mp4" => "video/mp4", "webm" => isVideo ? "video/webm" : "audio/webm", "m4a" => "audio/mp4", "mp3" => "audio/mpeg", _ => contentType };
attachment.FilePath = await Save(await File.ReadAllBytesAsync(output, token), extension, token);
attachment.Size = new FileInfo(output).Length;
if (isVideo)
var made = await FFProbe.AnalyseAsync(output, default, token, "-protocol_whitelist file");
byte[] poster = default;
if (video != default)
{
attachment.Width = probe.PrimaryVideoStream.Width;
attachment.Height = probe.PrimaryVideoStream.Height;
var frame = temp + ".png";
await FFMpeg.SnapshotAsync(input, frame, captureTime: TimeSpan.FromSeconds(Math.Min(1, probe.Duration.TotalSeconds / 2)));
var still = ProcessImage(await File.ReadAllBytesAsync(frame, token), 640, _options.CurrentValue.PreviewSide, false);
attachment.PreviewPath = await Save(still.Preview, "jpg", token);
attachment.Width = made.PrimaryVideoStream?.Width;
attachment.Height = made.PrimaryVideoStream?.Height;
await FFMpeg.SnapshotAsync(output, frame, captureTime: TimeSpan.FromSeconds(Math.Min(1, made.Duration.TotalSeconds / 2)));
var still = ProcessImage(await File.ReadAllBytesAsync(frame, token), 640, options.PreviewSide);
poster = still.Preview;
attachment.Blurhash = still.Blurhash;
File.Delete(frame);
}
attachment.Kind = video != default ? "video" : "audio";
attachment.ContentType = extension switch
{
"mp4" => "video/mp4",
"webm" => video != default ? "video/webm" : "audio/webm",
"m4a" => "audio/mp4",
"mp3" => "audio/mpeg",
"ogg" => "audio/ogg",
"wav" => "audio/wav",
"flac" => "audio/flac",
_ => attachment.ContentType
};
attachment.DurationSeconds = Math.Round(made.Duration.TotalSeconds, 2);
attachment.Size = new FileInfo(output).Length;
attachment.FilePath = SaveFile(output, extension);
if (poster != default)
attachment.PreviewPath = await Save(poster, "jpg", token);
return new MediaOutcome(attachment);
}
catch (Exception ex) when (ex is not OperationCanceledException)
@@ -233,34 +513,38 @@ namespace PrivaPub.Domain.Media
}
finally
{
foreach (var path in new[] { input, output })
_processing.Release();
foreach (var path in new[] { input, output, frame }.Where(p => p != default))
if (File.Exists(path))
File.Delete(path);
}
}
async Task<string> Save(byte[] bytes, string extension, CancellationToken token)
string SaveFile(string source, string extension)
{
var (relative, full) = NewPath(extension);
File.Move(source, full);
return relative;
}
(string Relative, string Full) NewPath(string extension)
{
var now = DateTime.UtcNow;
var relative = Path.Combine(now.ToString("yyyy", CultureInfo.InvariantCulture), now.ToString("MM", CultureInfo.InvariantCulture),
$"{Convert.ToHexStringLower(RandomNumberGenerator.GetBytes(16))}.{extension}");
var full = Path.Combine(Root, relative);
Directory.CreateDirectory(Path.GetDirectoryName(full)!);
return (relative.Replace('\\', '/'), full);
}
async Task<string> Save(byte[] bytes, string extension, CancellationToken token)
{
var (relative, full) = NewPath(extension);
await File.WriteAllBytesAsync(full, bytes, token);
return relative.Replace('\\', '/');
return relative;
}
static string Clean(string value, int max) =>
string.IsNullOrWhiteSpace(value) ? default : value.Trim().Length <= max ? value.Trim() : value.Trim()[..max];
static float[] Focus(string focus)
{
var parts = focus?.Split(',');
if (parts is not { Length: 2 }
|| !float.TryParse(parts[0], NumberStyles.Float, CultureInfo.InvariantCulture, out var x)
|| !float.TryParse(parts[1], NumberStyles.Float, CultureInfo.InvariantCulture, out var y))
return default;
return new[] { Math.Clamp(x, -1, 1), Math.Clamp(y, -1, 1) };
}
}
}
+45
View File
@@ -0,0 +1,45 @@
using MongoDB.Entities;
using PrivaPub.Infrastructure.Jobs;
using PrivaPub.Models.Jobs;
using PrivaPub.Models.Media;
namespace PrivaPub.Domain.Media
{
// Audio and video sent to /api/v2/media, processed off the request (its lease renewed while ffmpeg runs): playable, or
// marked failed with why. One at a time, since a transcode takes the machine's cores.
public class ProcessMediaJob : IJobHandler
{
readonly IMediaService _media;
readonly ILogger<ProcessMediaJob> _logger;
public ProcessMediaJob(IMediaService media, ILogger<ProcessMediaJob> logger)
{
_media = media;
_logger = logger;
}
public JobKind Kind => JobKind.ProcessMedia;
public int Concurrency => 1;
public int MaxAttempts => 3;
public int PerHostLimit => 1;
public static Job JobFor(MediaAttachment pending, string host) => new()
{
Kind = JobKind.ProcessMedia,
Payload = pending.ID,
Host = host,
DedupeKey = $"media|{pending.ID}"
};
public async Task<JobOutcome> Handle(Job job, CancellationToken token)
{
var pending = await DB.Default.Find<MediaAttachment>().Match(m => m.ID == job.Payload && m.ProcessingState == "pending" && m.TrashedAt == null).ExecuteFirstAsync(token);
if (pending == default)
return JobOutcome.Done;//processed already, or trashed meanwhile
if (!await _media.ProcessPending(pending, token))
_logger.LogInformation("Upload {Id} could not be processed", pending.ID);
return JobOutcome.Done;
}
}
}
@@ -0,0 +1,399 @@
using MongoDB.Entities;
using PrivaPub.Domain.Media;
using PrivaPub.Domain.Social;
using PrivaPub.Federation.Actors;
using PrivaPub.Federation.Rendering;
using PrivaPub.Infrastructure.Backup;
using PrivaPub.Infrastructure.Jobs;
using PrivaPub.Models.Federation;
using PrivaPub.Models.Jobs;
using PrivaPub.Models.Media;
using PrivaPub.Models.Post;
using PrivaPub.Models.Social;
using PrivaPub.Models.User;
using System.IO.Compression;
using System.Text;
using System.Text.Encodings.Web;
using System.Text.Json;
using System.Text.Json.Nodes;
using PostEntity = PrivaPub.Models.Post.Post;
namespace PrivaPub.Domain.Portability
{
// A persona's archive, in Mastodon's account-archive layout so other servers' importers read it:
// - actor.json (as other servers see the persona: its public key, never the private one), avatar and header;
// - outbox.json: every post of its own as a Create (any visibility but circles' and located posts, addressed as it was
// delivered) and its boosts as Announces; the media its posts hold under media_attachments/files/, which the
// attachments' urls name;
// - likes.json, bookmarks.json, and the CSV files Mastodon exports: follows, lists, blocks, mutes, blocked servers,
// bookmarks;
// - privapub/: what only PrivaPub has: filters, followed hashtags, the notification policy, pinned posts, scheduled and
// located posts, and archive.json saying whose it is.
// Nothing of its root, its siblings, its keys or anyone's tokens. Written whole to a .partial file, then renamed; kept a
// week (PersonaArchives.Forget).
public class ExportArchiveJob(ILocalActorService localActors, IMediaService media, Backups backups, ILogger<ExportArchiveJob> logger) : IJobHandler
{
public const int Format = 1;
static readonly JsonSerializerOptions Options = new() { WriteIndented = true, Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping };
static readonly JsonWriterOptions Writer = new() { Indented = true, Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping };
const string MediaFolder = "media_attachments/files/";
public JobKind Kind => JobKind.ExportArchive;
public int Concurrency => 1;
public int MaxAttempts => 2;
public int PerHostLimit => 1;
public static Job JobFor(string avatarId, string host) => new()
{
Kind = JobKind.ExportArchive,
Payload = avatarId,
Host = host,
DedupeKey = $"export|{avatarId}|{DateTime.UtcNow.Ticks}"
};
public async Task<JobOutcome> Handle(Job job, CancellationToken token)
{
var avatarId = job.Payload;
var actor = await localActors.FindById(LocalActorKind.Person, avatarId, token);
if (actor == default)
return JobOutcome.Done;
await DB.Default.Update<PersonaArchive>().MatchID(avatarId).Modify(a => a.ExportState, "running").Modify(a => a.ExportError, null).ExecuteAsync(token);
var folder = PersonaArchives.Folder(backups.Root, avatarId);
ServerBackup.MakeDirectory(Path.GetDirectoryName(folder)!);
ServerBackup.MakeDirectory(folder);
var name = $"{actor.UserName}-{DateTime.UtcNow:yyyyMMdd-HHmmss}.zip";
var partial = Path.Combine(folder, name + ServerBackup.PartialSuffix);
try
{
await using (var file = new FileStream(partial, FileMode.Create, FileAccess.Write))
using (var zip = new ZipArchive(file, ZipArchiveMode.Create))
await Write(zip, actor, token);
var final = Path.Combine(folder, name);
File.Move(partial, final);
foreach (var old in Directory.EnumerateFiles(folder, "*.zip").Where(f => f != final))
File.Delete(old);
await DB.Default.Update<PersonaArchive>().MatchID(avatarId)
.Modify(a => a.ExportState, "ready").Modify(a => a.ExportReadyAt, DateTime.UtcNow)
.Modify(a => a.ExportFile, name).Modify(a => a.ExportBytes, new FileInfo(final).Length).ExecuteAsync(token);
return JobOutcome.Done;
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
logger.LogError(ex, "The archive of {Persona} could not be made", actor.Handle);
if (File.Exists(partial))
File.Delete(partial);
await DB.Default.Update<PersonaArchive>().MatchID(avatarId).Modify(a => a.ExportState, "failed")
.Modify(a => a.ExportError, ex.Message).ExecuteAsync(CancellationToken.None);
return JobOutcome.Done;
}
}
async Task Write(ZipArchive zip, LocalActor actor, CancellationToken token)
{
var avatar = await DB.Default.Find<Avatar>().MatchID(actor.Id).ExecuteFirstAsync(token);
var files = new SortedSet<string>(StringComparer.Ordinal);
var prefix = media.Url(string.Empty);
var document = ActivityPubRenderer.Actor(actor);
foreach (var (field, kind) in new[] { ("icon", "avatar"), ("image", "header") })
if (document[field] is JsonObject image && image["url"]?.GetValue<string>() is { } url && url.StartsWith(prefix, StringComparison.Ordinal))
{
var relative = url[prefix.Length..];
var local = Path.Combine(media.Root, relative);
if (!File.Exists(local))
continue;
var entry = kind + Path.GetExtension(relative);
await Copy(zip, local, entry, token);
image["url"] = entry;
}
await Text(zip, "actor.json", document.ToJsonString(Options), token);
var counts = await Outbox(zip, actor, prefix, files, token);
foreach (var relative in files)
{
var local = Path.Combine(media.Root, relative);
if (File.Exists(local))
await Copy(zip, local, MediaFolder + relative, token);
}
var handles = new Handles(localActors);
await Collection(zip, "likes.json", await Uris(DB.Default.Find<Favourite, string>().Match(f => f.AccountId == actor.Id).Project(f => f.PostId), token), token);
var bookmarks = await Uris(DB.Default.Find<Bookmark, string>().Match(b => b.AvatarId == actor.Id).Project(b => b.PostId), token);
await Collection(zip, "bookmarks.json", bookmarks, token);
await Text(zip, "bookmarks.csv", string.Concat(bookmarks.Select(b => b + "\n")), token);
var follows = await DB.Default.Find<Following>().Match(f => f.AvatarId == actor.Id).ExecuteAsync(token);
var csv = new StringBuilder("Account address,Show boosts,Notify on new posts,Languages\n");
foreach (var follow in follows)
csv.Append($"{Csv(await handles.Of(follow.TargetActorURI, token))},{(follow.ShowReblogs ? "true" : "false")},false,\n");
await Text(zip, "following_accounts.csv", csv.ToString(), token);
csv.Clear();
var lists = await DB.Default.Find<PersonaList>().Match(l => l.AvatarId == actor.Id).ExecuteAsync(token);
foreach (var list in lists)
foreach (var member in await DB.Default.Find<PersonaListMember>().Match(m => m.ListId == list.ID).ExecuteAsync(token))
if (follows.FirstOrDefault(f => f.TargetAccountId == member.AccountId) is { } followed)
csv.Append($"{Csv(list.Title)},{Csv(await handles.Of(followed.TargetActorURI, token))}\n");
await Text(zip, "lists.csv", csv.ToString(), token);
csv.Clear();
foreach (var block in await DB.Default.Find<Block>().Match(b => b.AvatarId == actor.Id).ExecuteAsync(token))
csv.Append(Csv(await handles.Of(block.TargetActorURI, token)) + "\n");
await Text(zip, "blocked_accounts.csv", csv.ToString(), token);
csv.Clear().Append("Account address,Hide notifications\n");
foreach (var mute in await DB.Default.Find<Mute>().Match(m => m.AvatarId == actor.Id).ExecuteAsync(token))
csv.Append($"{Csv(await handles.Of(mute.TargetActorURI, token))},{(mute.HideNotifications ? "true" : "false")}\n");
await Text(zip, "muted_accounts.csv", csv.ToString(), token);
csv.Clear();
foreach (var domain in await DB.Default.Find<AccountDomainBlock, string>().Match(d => d.AvatarId == actor.Id).Project(d => d.Domain).ExecuteAsync(token))
csv.Append(domain + "\n");
await Text(zip, "blocked_domains.csv", csv.ToString(), token);
await PrivaPubParts(zip, actor, avatar, prefix, counts, token);
}
// the outbox, a post at a time, newest first as Mastodon writes it: what was published and boosted
async Task<(int Posts, int Boosts, int Located)> Outbox(ZipArchive zip, LocalActor actor, string prefix, ISet<string> files, CancellationToken token)
{
var entry = zip.CreateEntry("outbox.json", CompressionLevel.Optimal);
await using var stream = entry.Open();
await using var writer = new Utf8JsonWriter(stream, Writer);
writer.WriteStartObject();
writer.WriteString("@context", "https://www.w3.org/ns/activitystreams");
writer.WriteString("id", "outbox.json");
writer.WriteString("type", "OrderedCollection");
writer.WritePropertyName("orderedItems");
writer.WriteStartArray();
var (posts, boosts) = (0, 0);
var located = 0;
var before = default(string);
while (true)
{
var find = DB.Default.Find<PostEntity>().Match(p => p.GroupUserId == actor.Id && !p.IsFederatedCopy && !p.DeletedAt.HasValue);
if (before != default)
find = find.Match(f => f.Lt(p => p.ID, before));
var batch = await find.Sort(p => p.ID, Order.Descending).Limit(200).ExecuteAsync(token);
if (batch.Count == 0)
break;
before = batch[^1].ID;
var originals = (await DB.Default.Find<PostEntity>().Match(p => batch.Select(b => b.ReblogOfPostId).Contains(p.ID)).ExecuteAsync(token))
.ToDictionary(p => p.ID);
foreach (var post in batch)
{
if (post.Visibility == PostVisibility.LocalGeo)
{
located++;
continue;
}
if (post.Visibility == PostVisibility.Circle)
continue;
if (post.ReblogOfPostId != default)
{
if (!originals.TryGetValue(post.ReblogOfPostId, out var original) || string.IsNullOrEmpty(original.ObjectURI))
continue;
new JsonObject
{
["id"] = actor.ActivityUri($"announce-{post.ID}"),
["type"] = "Announce",
["actor"] = actor.Uri,
["published"] = ActivityPubRenderer.Timestamp(post.CreationDate),
["to"] = new JsonArray("https://www.w3.org/ns/activitystreams#Public"),
["cc"] = new JsonArray(actor.Followers),
["object"] = original.ObjectURI
}.WriteTo(writer);
boosts++;
continue;
}
var group = post.GroupId == default ? default : await localActors.FindById(LocalActorKind.Group, post.GroupId, token);
var note = ActivityPubRenderer.Note(post, actor, group, post.InReplyToURI);
if (post.Visibility is PostVisibility.Direct or PostVisibility.FollowersOnly && (post.To.Count > 0 || post.Cc.Count > 0))
{
note["to"] = new JsonArray(post.To.Select(t => (JsonNode)t).ToArray());
note["cc"] = new JsonArray(post.Cc.Select(c => (JsonNode)c).ToArray());
}
if (note["attachment"] is JsonArray attachments)
foreach (var attachment in attachments.OfType<JsonObject>())
if (attachment["url"]?.GetValue<string>() is { } url && url.StartsWith(prefix, StringComparison.Ordinal))
{
var relative = url[prefix.Length..];
files.Add(relative);
attachment["url"] = "/" + MediaFolder + relative;
}
var create = ActivityPubRenderer.Create(actor, note, $"create-{post.ID}");
create["to"] = note["to"]?.DeepClone();
create["cc"] = note["cc"]?.DeepClone();
create.WriteTo(writer);
posts++;
}
await writer.FlushAsync(token);
}
writer.WriteEndArray();
writer.WriteNumber("totalItems", posts + boosts);
writer.WriteEndObject();
return (posts, boosts, located);
}
async Task PrivaPubParts(ZipArchive zip, LocalActor actor, Avatar avatar, string prefix, (int Posts, int Boosts, int Located) counts, CancellationToken token)
{
var filters = await DB.Default.Find<PersonaFilter>().Match(f => f.AvatarId == actor.Id).ExecuteAsync(token);
var filtered = await UriMap(filters.SelectMany(f => f.Statuses.Select(s => s.PostId)), token);
await Json(zip, "privapub/filters.json", filters.Select(f => new
{
title = f.Title,
context = f.Context,
expiresAt = f.ExpiresAt,
action = f.Action,
keywords = f.Keywords.Select(k => new { keyword = k.Keyword, wholeWord = k.WholeWord }),
statuses = f.Statuses.Select(s => filtered.GetValueOrDefault(s.PostId)).Where(u => u != default)
}), token);
await Json(zip, "privapub/followed_tags.json",
await DB.Default.Find<FollowedTag, string>().Match(t => t.AvatarId == actor.Id).Project(t => t.Name).ExecuteAsync(token), token);
var policy = await NotificationPolicies.Of(actor.Id, token);
await Json(zip, "privapub/notification_policy.json", new
{
forNotFollowing = policy.ForNotFollowing.ToString(),
forNotFollowers = policy.ForNotFollowers.ToString(),
forNewAccounts = policy.ForNewAccounts.ToString(),
forPrivateMentions = policy.ForPrivateMentions.ToString(),
forLimitedAccounts = policy.ForLimitedAccounts.ToString()
}, token);
await Json(zip, "privapub/pins.json",
await Uris(DB.Default.Find<Pin, string>().Match(p => p.AvatarId == actor.Id).Project(p => p.PostId), token), token);
var scheduled = await DB.Default.Find<ScheduledStatus>().Match(s => s.AvatarId == actor.Id).ExecuteAsync(token);
var scheduledMedia = await MediaFiles(scheduled.SelectMany(s => s.Params.MediaIds), token);
foreach (var file in scheduledMedia.Values)
await CopyMedia(zip, file.Path, token);
await Json(zip, "privapub/scheduled.json", scheduled.Select(s => new
{
scheduledAt = s.ScheduledAt,
text = s.Params.Text,
spoilerText = s.Params.SpoilerText,
sensitive = s.Params.Sensitive,
visibility = s.Params.Visibility,
language = s.Params.Language,
poll = s.Params.Poll,
media = s.Params.MediaIds.Where(scheduledMedia.ContainsKey).Select(id => new { url = MediaFolder + scheduledMedia[id].Path, description = scheduledMedia[id].Description })
}), token);
var located = await DB.Default.Find<PostEntity>().Match(p => p.GroupUserId == actor.Id && !p.IsFederatedCopy && !p.DeletedAt.HasValue
&& p.Visibility == PostVisibility.LocalGeo).ExecuteAsync(token);
foreach (var post in located)
foreach (var item in post.Media.Where(m => m.URL?.StartsWith(prefix, StringComparison.Ordinal) == true))
await CopyMedia(zip, item.URL[prefix.Length..], token);
await Json(zip, "privapub/located.json", located.Select(p => new
{
id = p.ObjectURI,
published = p.CreationDate,
title = p.Title,
text = p.Text,
spoilerText = p.SpoilerText,
latitude = p.Geo?.Coordinates[1],
longitude = p.Geo?.Coordinates[0],
rangeKm = p.RangeKm,
media = p.Media.Where(m => m.URL?.StartsWith(prefix, StringComparison.Ordinal) == true)
.Select(m => new { url = MediaFolder + m.URL[prefix.Length..], description = m.Description })
}), token);
await Json(zip, "privapub/archive.json", new
{
format = Format,
software = "privapub",
actor = actor.Uri,
username = avatar?.UserName ?? actor.UserName,
exportedAt = DateTime.UtcNow,
posts = counts.Posts,
boosts = counts.Boosts,
located = counts.Located,
scheduled = scheduled.Count
}, token);
}
async Task CopyMedia(ZipArchive zip, string relative, CancellationToken token)
{
var local = Path.Combine(media.Root, relative);
if (File.Exists(local) && zip.GetEntry(MediaFolder + relative) == default)
await Copy(zip, local, MediaFolder + relative, token);
}
static async Task<Dictionary<string, (string Path, string Description)>> MediaFiles(IEnumerable<string> ids, CancellationToken token)
{
var wanted = ids.Distinct().ToList();
if (wanted.Count == 0)
return [];
return (await DB.Default.Find<MediaAttachment>().Match(m => wanted.Contains(m.ID) && m.TrashedAt == null).ExecuteAsync(token))
.ToDictionary(m => m.ID, m => (m.FilePath, m.Description));
}
static async Task<Dictionary<string, string>> UriMap(IEnumerable<string> postIds, CancellationToken token)
{
var ids = postIds.Where(id => id != default).Distinct().ToList();
if (ids.Count == 0)
return [];
return (await DB.Default.Find<PostEntity>().Match(p => ids.Contains(p.ID)).Project(p => new PostEntity { ID = p.ID, ObjectURI = p.ObjectURI }).ExecuteAsync(token))
.Where(p => !string.IsNullOrEmpty(p.ObjectURI)).ToDictionary(p => p.ID, p => p.ObjectURI);
}
static async Task<List<string>> Uris<T>(Find<T, string> postIds, CancellationToken token) where T : IEntity
{
var ids = await postIds.ExecuteAsync(token);
var map = await UriMap(ids, token);
return ids.Where(map.ContainsKey).Select(id => map[id]).ToList();
}
static async Task Collection(ZipArchive zip, string name, List<string> items, CancellationToken token) =>
await Text(zip, name, new JsonObject
{
["@context"] = "https://www.w3.org/ns/activitystreams",
["id"] = name,
["type"] = "OrderedCollection",
["totalItems"] = items.Count,
["orderedItems"] = new JsonArray(items.Select(i => (JsonNode)i).ToArray())
}.ToJsonString(Options), token);
static Task Json<T>(ZipArchive zip, string name, T value, CancellationToken token) =>
Text(zip, name, JsonSerializer.Serialize(value, Options), token);
static async Task Text(ZipArchive zip, string name, string text, CancellationToken token)
{
await using var stream = zip.CreateEntry(name, CompressionLevel.Optimal).Open();
await stream.WriteAsync(Encoding.UTF8.GetBytes(text), token);
}
static async Task Copy(ZipArchive zip, string path, string name, CancellationToken token)
{
await using var source = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.Read, 81920, FileOptions.SequentialScan);
await using var target = zip.CreateEntry(name, CompressionLevel.NoCompression).Open();
await source.CopyToAsync(target, token);
}
// a CSV cell, quoted when it must be
static string Csv(string value) =>
value.IndexOfAny([',', '"', '\n', '\r']) < 0 ? value : "\"" + value.Replace("\"", "\"\"") + "\"";
// an account as Mastodon's CSV files name it: user@host, or its address when the handle is not known
sealed class Handles(ILocalActorService localActors)
{
readonly Dictionary<string, string> _known = new(StringComparer.Ordinal);
public async Task<string> Of(string actorUri, CancellationToken token)
{
if (_known.TryGetValue(actorUri, out var handle))
return handle;
if (await localActors.FindByUri(actorUri, token) is { } local)
handle = local.Handle;
else if (await DB.Default.Find<ForeignAvatar>().Match(a => a.ActorURI == actorUri).ExecuteFirstAsync(token) is { UserName.Length: > 0 } remote
&& Uri.TryCreate(actorUri, UriKind.Absolute, out var parsed))
handle = $"{remote.UserName}@{(string.IsNullOrEmpty(remote.Domain) ? parsed.Authority : remote.Domain)}";
else
handle = actorUri;
return _known[actorUri] = handle;
}
}
}
}
@@ -0,0 +1,943 @@
using Microsoft.AspNetCore.Http;
using MongoDB.Driver;
using MongoDB.Entities;
using PrivaPub.Api.Mastodon.Controllers;
using PrivaPub.Domain.Media;
using PrivaPub.Domain.Relationships;
using PrivaPub.Domain.Social;
using PrivaPub.Domain.Statuses;
using PrivaPub.Federation.Actors;
using PrivaPub.Federation.Inbox;
using PrivaPub.Federation.Moderation;
using PrivaPub.Federation.Objects;
using PrivaPub.Federation.Outbox;
using PrivaPub.Federation.Rendering;
using PrivaPub.Infrastructure.Backup;
using PrivaPub.Infrastructure.Ids;
using PrivaPub.Infrastructure.Jobs;
using PrivaPub.Models.Federation;
using PrivaPub.Models.Jobs;
using PrivaPub.Models.Media;
using PrivaPub.Models.Post;
using PrivaPub.Models.Social;
using PrivaPub.Models.User;
using System.Net;
using System.Text.Json.Nodes;
using System.Text.RegularExpressions;
using PostEntity = PrivaPub.Models.Post.Post;
namespace PrivaPub.Domain.Portability
{
// An archive restored into one of a root's personas (owner decision 2026-10-07): PrivaPub's own, or Mastodon's account
// archive. Only the parts asked for, one at a time, with progress, until asked to stop.
// Posts: only the archive actor's own Creates (boosts, direct and circle posts are counted, not imported); delivered to
// no one, put in no home, notifying nobody, they show on the profile, the outbox, hashtags and search. From the same
// actor (a persona's own archive, back home) a post keeps its id and address, and one already here (even deleted) or
// whose address is a tombstone is left alone; from another actor it gets an id of its date and its address here, with
// the original in ImportedFromURI, so importing twice changes nothing. Replies and self-quotes inside the archive point
// at their copies; polls are closed, with no votes; mentions stay links; the HTML is sanitized as any remote post's;
// media go through MediaService, its checks and the root's quota.
// The rest: follows asked again (not on a blocked server), blocks, mutes, blocked servers, lists (with only accounts
// followed), bookmarks (read by signed fetch), filters, followed hashtags, the notification policy, pins (not
// announced), the profile; and only when asked, located and scheduled posts (PrivaPub's archives) and likes, which tell
// their authors. Followers are never imported: they choose.
public partial class ImportArchiveJob(IServiceScopeFactory scopes, ILocalActorService localActors, IMediaService media, IDomainBlocks domainBlocks,
IJobQueue jobs, ILogger<ImportArchiveJob> logger) : IJobHandler
{
static readonly HashSet<string> Public = ["https://www.w3.org/ns/activitystreams#Public", "as:Public", "Public"];
static readonly HashSet<string> Postable = ["Note", "Article", "Page", "Question"];
const int MaxMedia = 4;
public JobKind Kind => JobKind.ImportArchive;
public int Concurrency => 1;
public int MaxAttempts => 1;
public int PerHostLimit => 1;
public static Job JobFor(string avatarId, string host) => new()
{
Kind = JobKind.ImportArchive,
Payload = avatarId,
Host = host,
DedupeKey = $"import|{avatarId}|{DateTime.UtcNow.Ticks}"
};
sealed class Run(PersonaArchive state, LocalActor me, SafeArchive archive, string archiveActor, string archiveFollowers, bool sameActor,
IServiceProvider services)
{
public PersonaArchive State { get; } = state;
public string ArchiveFollowers { get; } = archiveFollowers;
public LocalActor Me { get; } = me;
public SafeArchive Archive { get; } = archive;
public string ArchiveActor { get; } = archiveActor;
public bool SameActor { get; } = sameActor;
public IServiceProvider Services { get; } = services;
public Dictionary<string, (string Id, string Uri)> Posts { get; } = new(StringComparer.Ordinal);//archive object id → here
public Dictionary<string, string> Parents { get; } = new(StringComparer.Ordinal);//archive object id → what it answers
public DateTime LastSaved { get; set; }
public void Count(string what, int by = 1) => State.ImportCounts[what] = State.ImportCounts.GetValueOrDefault(what) + by;
}
public async Task<JobOutcome> Handle(Job job, CancellationToken token)
{
var state = await DB.Default.Find<PersonaArchive>().MatchID(job.Payload).ExecuteFirstAsync(token);
if (state is not { ImportState: "queued" or "running" })
return JobOutcome.Done;
var me = await localActors.FindById(LocalActorKind.Person, job.Payload, token);
var path = state.ImportFile;
try
{
if (me == default || !await RootMayImport(me.Id, token))
{
await End(state, "failed", "the persona or its login can no longer import", token);
return JobOutcome.Done;
}
var (archive, refused) = path == default ? (default, "no archive") : SafeArchive.Open(path);
if (archive == default)
{
await End(state, "failed", refused, token);
return JobOutcome.Done;
}
using (archive)
{
var actor = archive.Json("actor.json") as JsonObject;
var actorId = Text(actor?["id"]);
if (actorId == default)
{
await End(state, "failed", "the archive has no actor.json", token);
return JobOutcome.Done;
}
using var scope = scopes.CreateScope();
var run = new Run(state, me, archive, actorId, Text(actor["followers"]) ?? actorId.TrimEnd('/') + "/followers", actorId == me.Uri,
scope.ServiceProvider);
state.ImportState = "running";
state.ImportFrom = actorId;
state.ImportSameActor = run.SameActor;
state.ImportCounts = [];
await Save(run, token, force: true);
var parts = state.ImportParts.ToHashSet(StringComparer.Ordinal);
var privapub = Text((archive.Json("privapub/archive.json") as JsonObject)?["software"]) == "privapub";
if (parts.Contains("profile") && !await Stopped(run, token))
await Profile(run, actor, token);
if (parts.Contains("posts") && !await Stopped(run, token))
await Posts(run, token);
else
await KnownPosts(run, token);
if (parts.Contains("located") && privapub && !await Stopped(run, token))
await Located(run, token);
if (parts.Contains("scheduled") && privapub && !await Stopped(run, token))
await Scheduled(run, token);
if (parts.Contains("tags") && !await Stopped(run, token))
await Tags(run, token);
if (parts.Contains("notificationpolicy") && !await Stopped(run, token))
await Policy(run, token);
if (parts.Contains("filters") && !await Stopped(run, token))
await Filters(run, token);
if (parts.Contains("pins") && !await Stopped(run, token))
await Pins(run, token);
if (parts.Contains("domainblocks") && !await Stopped(run, token))
await DomainBlocks(run, token);
if (parts.Contains("blocks") && !await Stopped(run, token))
await Accounts(run, "blocked_accounts.csv", "blocks", Block, token);
if (parts.Contains("mutes") && !await Stopped(run, token))
await Accounts(run, "muted_accounts.csv", "mutes", Mute, token);
if (parts.Contains("follows") && !await Stopped(run, token))
await Accounts(run, "following_accounts.csv", "follows", Follow, token);
if (parts.Contains("lists") && !await Stopped(run, token))
await Lists(run, token);
if (parts.Contains("bookmarks") && !await Stopped(run, token))
await Bookmarks(run, token);
if (parts.Contains("likes") && !await Stopped(run, token))
await Likes(run, token);
await End(state, await Stopped(run, token) ? "stopped" : "done", default, token);
}
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
logger.LogError(ex, "The import into {Persona} failed", me?.Handle ?? job.Payload);
await End(state, "failed", ex.Message, CancellationToken.None);
}
finally
{
if (path != default && File.Exists(path))
File.Delete(path);
}
return JobOutcome.Done;
}
static async Task<bool> RootMayImport(string avatarId, CancellationToken token)
{
var link = await DB.Default.Find<RootToAvatar>().Match(r => r.AvatarId == avatarId).ExecuteFirstAsync(token);
var root = link == default ? default : await DB.Default.Find<RootUser>().MatchID(link.RootId).ExecuteFirstAsync(token);
return root is { IsBanned: false, DeletedAt: null };
}
static async Task End(PersonaArchive state, string how, string error, CancellationToken token) =>
await DB.Default.Update<PersonaArchive>().MatchID(state.ID)
.Modify(a => a.ImportState, how).Modify(a => a.ImportError, error).Modify(a => a.ImportEndedAt, DateTime.UtcNow)
.Modify(a => a.ImportCounts, state.ImportCounts).Modify(a => a.ImportDone, state.ImportDone).Modify(a => a.ImportTotal, state.ImportTotal)
.Modify(a => a.ImportFile, null).Modify(a => a.StopAsked, false)
.ExecuteAsync(token);
// progress written at most every two seconds; the answer to whether the root asked to stop
static async Task Save(Run run, CancellationToken token, bool force = false)
{
if (!force && DateTime.UtcNow - run.LastSaved < TimeSpan.FromSeconds(2))
return;
run.LastSaved = DateTime.UtcNow;
await DB.Default.Update<PersonaArchive>().MatchID(run.State.ID)
.Modify(a => a.ImportState, run.State.ImportState).Modify(a => a.ImportFrom, run.State.ImportFrom)
.Modify(a => a.ImportSameActor, run.State.ImportSameActor).Modify(a => a.ImportCounts, run.State.ImportCounts)
.Modify(a => a.ImportDone, run.State.ImportDone).Modify(a => a.ImportTotal, run.State.ImportTotal)
.ExecuteAsync(token);
}
static async Task<bool> Stopped(Run run, CancellationToken token)
{
await Save(run, token);
return await DB.Default.Find<PersonaArchive>().MatchID(run.State.ID).Match(a => a.StopAsked).ExecuteAnyAsync(token);
}
// ---- posts
// what an outbox item is to this import: the object to import, or why not
(JsonObject Note, string Skipped) Importable(Run run, JsonObject item)
{
var type = Text(item["type"]);
if (type == "Announce")
return (default, "boosts skipped");
if (type != "Create" || item["object"] is not JsonObject note || !Postable.Contains(Text(note["type"]) ?? string.Empty))
return (default, "not a post");
if (Text(item["actor"]) != run.ArchiveActor || Id(note["attributedTo"]) is { } author && author != run.ArchiveActor)
return (default, "not the archive's own");
if (Text(note["id"]) == default)
return (default, "not a post");
return Visibility(run, note) == default ? (default, "direct or circle posts skipped") : (note, default);
}
PostVisibility? Visibility(Run run, JsonObject note)
{
var to = Strings(note["to"]);
var cc = Strings(note["cc"]);
if (to.Any(Public.Contains))
return PostVisibility.Public;
if (cc.Any(Public.Contains))
return PostVisibility.Unlisted;
if (to.Concat(cc).Contains(run.ArchiveFollowers) && Text(note["audience"]) == default)
return PostVisibility.FollowersOnly;
return default;
}
// what is already here of the archive's posts, for pins, filters and bookmarks when posts are not imported again
async Task KnownPosts(Run run, CancellationToken token)
{
await foreach (var item in run.Archive.Outbox(token))
{
if (Importable(run, item).Note is not { } note)
continue;
var original = Text(note["id"]);
var here = run.SameActor
? await DB.Default.Find<PostEntity>().Match(p => p.ObjectURI == original && p.GroupUserId == run.Me.Id).ExecuteFirstAsync(token)
: await DB.Default.Find<PostEntity>().Match(p => p.GroupUserId == run.Me.Id && p.ImportedFromURI == original).ExecuteFirstAsync(token);
if (here is { DeletedAt: null })
run.Posts[original] = (here.ID, here.ObjectURI);
}
}
async Task Posts(Run run, CancellationToken token)
{
// first the ids each post will have here, so a reply or a quote anywhere in the archive finds its copy
var planned = new Dictionary<string, (string Id, string Uri, bool New)>(StringComparer.Ordinal);
var total = 0L;
await foreach (var item in run.Archive.Outbox(token))
{
total++;
if (Importable(run, item).Note is not { } note)
continue;
var original = Text(note["id"]);
var published = Date(note["published"]) ?? DateTime.UtcNow;
if (Id(note["inReplyTo"]) is { } answers)
run.Parents[original] = answers;
if (run.SameActor)
{
var id = original.TrimEnd('/').Split('/')[^1];
if (!IsObjectId(id) || original != run.Me.PostUri(id))
id = PrivacyIds.At(Clamp(published));
var here = await DB.Default.Find<PostEntity>().Match(p => p.ID == id || p.ObjectURI == original).ExecuteFirstAsync(token);
var tombstone = await DB.Default.Find<DeletedObject>().Match(d => d.ObjectURI == original).ExecuteAnyAsync(token);
if (here != default || tombstone)
{
if (here is { DeletedAt: null, GroupUserId: var owner } && owner == run.Me.Id)
run.Posts[original] = (here.ID, here.ObjectURI);
run.Count(here is { DeletedAt: not null } || tombstone ? "posts deleted here" : "posts already here");
continue;
}
planned[original] = (id, original, true);
}
else
{
var here = await DB.Default.Find<PostEntity>().Match(p => p.GroupUserId == run.Me.Id && p.ImportedFromURI == original).ExecuteFirstAsync(token);
if (here != default)
{
if (here.DeletedAt == default)
run.Posts[original] = (here.ID, here.ObjectURI);
run.Count("posts already here");
continue;
}
var id = PrivacyIds.At(Clamp(published));
planned[original] = (id, run.Me.PostUri(id), true);
}
}
foreach (var (original, (id, uri, _)) in planned)
run.Posts[original] = (id, uri);
run.State.ImportTotal = total;
run.State.ImportDone = 0;
await foreach (var item in run.Archive.Outbox(token))
{
run.State.ImportDone++;
if (run.State.ImportDone % 50 == 0 && await Stopped(run, token))
return;
var (note, skipped) = Importable(run, item);
if (note == default)
{
run.Count(skipped);
continue;
}
var original = Text(note["id"]);
if (!planned.TryGetValue(original, out var target))
continue;
await Post(run, note, original, target.Id, target.Uri, Visibility(run, note).Value, token);
run.Count("posts");
}
}
async Task Post(Run run, JsonObject note, string original, string id, string uri, PostVisibility visibility, CancellationToken token)
{
var me = run.Me;
var summary = Clean(Text(note["summary"]));
var content = Text(note["content"]) ?? (note["contentMap"] as JsonObject)?.Select(c => Text(c.Value)).FirstOrDefault();
var source = note["source"] as JsonObject;
var post = new PostEntity
{
ID = id,
GroupUserId = me.Id,
AuthorAccountId = me.Id,
ActorURI = me.Uri,
Visibility = visibility,
Title = Clean(Text(note["name"])),
SpoilerText = summary,
HasContentWarning = summary != default || note["sensitive"]?.GetValueKind() == System.Text.Json.JsonValueKind.True,
ContentHtml = content == default ? default : ContentSanitizer.Html(content),
Text = Text(source?["mediaType"]) is "text/plain" or "text/markdown" ? Text(source["content"]) : default,
ContentFormat = ContentFormat.Html,
Language = (note["contentMap"] as JsonObject)?.Select(c => c.Key).FirstOrDefault(),
Tags = (note["tag"] as JsonArray ?? [])
.OfType<JsonObject>().Where(t => Text(t["type"]) == "Hashtag").Select(t => TagsController.Normalise(Text(t["name"])))
.Where(t => t.Length > 0).Distinct().ToList(),
ObjectURI = uri,
Url = me.PostHtmlUrl(id),
ActivityURI = me.ActivityUri("create-" + id),
ImportedFromURI = run.SameActor ? default : original,
CreationDate = Clamp(Date(note["published"]) ?? DateTime.UtcNow),
LocalQuotePolicy = me.Settings.QuotePolicy ?? QuotePolicies.Public
};
if (Id(note["inReplyTo"]) is { } inReplyTo)
{
if (run.Posts.TryGetValue(inReplyTo, out var parent))
(post.AnsweringToPostId, post.InReplyToURI, post.InReplyToAccountId) = (parent.Id, parent.Uri, me.Id);
else
{
post.InReplyToURI = inReplyTo;
if (await DB.Default.Find<PostEntity>().Match(p => p.ObjectURI == inReplyTo).ExecuteFirstAsync(token) is { } held)
(post.AnsweringToPostId, post.InReplyToAccountId) = (held.ID, held.AuthorAccountId ?? held.GroupUserId);
}
}
foreach (var key in new[] { "quote", "quoteUrl", "quoteUri", "_misskey_quote" })
if (Text(note[key]) is { } quoted && run.Posts.TryGetValue(quoted, out var copy))
{
(post.QuotedPostId, post.QuoteURI, post.QuoteState, post.QuoteByConsent) = (copy.Id, copy.Uri, QuoteState.Accepted, true);
break;
}
if (Text(note["type"]) == "Question")
post.Poll = Poll(note);
// the conversation (FEP-7888) of its thread's first post, when the thread starts in the archive
if (visibility is PostVisibility.Public or PostVisibility.Unlisted && Root(run, original) is { } root && run.Posts.TryGetValue(root, out var first))
post.ContextURI = first.Uri + "/context";
post.Media = await Media(run, note["attachment"] as JsonArray, id, token);
var rendered = ActivityPubRenderer.Note(post, me, default, post.InReplyToURI);
post.To = Strings(rendered["to"]);
post.Cc = Strings(rendered["cc"]);
await DB.Default.SaveAsync(post, token);
if (post.AnsweringToPostId != default)
await DB.Default.Update<PostEntity>().MatchID(post.AnsweringToPostId).Modify(b => b.Inc(p => p.RepliesCount, 1)).ExecuteAsync(token);
}
// the first post of an archive post's thread, when the thread starts in the archive
static string Root(Run run, string original)
{
var seen = new HashSet<string>(StringComparer.Ordinal);
while (run.Parents.TryGetValue(original, out var parent) && seen.Add(original))
{
if (!run.Posts.ContainsKey(parent))
return default;
original = parent;
}
return original;
}
static PostPoll Poll(JsonObject note)
{
var multiple = note["anyOf"] is JsonArray;
var options = (note["anyOf"] ?? note["oneOf"]) as JsonArray ?? [];
var ends = Date(note["endTime"]) ?? Date(note["closed"]) ?? DateTime.UtcNow;
var now = DateTime.UtcNow;
return new PostPoll
{
Options = options.OfType<JsonObject>().Select(o => new PollOption { Title = Text(o["name"]) }).Where(o => o.Title != default).ToList(),
Multiple = multiple,
ExpiresAt = ends < now ? ends : now,
ClosedAt = ends < now ? ends : now,
VotersCount = 0
};
}
// the attachments the archive holds, uploaded as the persona's own (checked, processed, counted against its quota)
async Task<List<PostMedia>> Media(Run run, JsonArray attachments, string postId, CancellationToken token)
{
var made = new List<PostMedia>();
foreach (var attachment in (attachments ?? []).OfType<JsonObject>().Take(MaxMedia))
{
var url = Text(attachment["url"]) ?? Text((attachment["url"] as JsonArray)?.OfType<JsonObject>().FirstOrDefault()?["href"]);
var file = url?.TrimStart('/');
if (file == default || file.Contains("://") || !run.Archive.Has(file))
{
run.Count("media not in the archive");
continue;
}
var focus = attachment["focalPoint"] is JsonArray { Count: 2 } point ? $"{point[0]},{point[1]}" : default;
var outcome = await Upload(run, file, Text(attachment["mediaType"]), Clean(Text(attachment["name"])), focus, token);
if (outcome?.Ok != true)
{
run.Count("media refused");
continue;
}
var row = outcome.Attachment;
await DB.Default.Update<MediaAttachment>().MatchID(row.ID).Modify(m => m.PostId, postId).Modify(m => m.AttachedAt, DateTime.UtcNow).ExecuteAsync(token);
made.Add(new PostMedia
{
AttachmentId = row.ID,
ContentType = row.ContentType,
Kind = row.Kind,
DurationSeconds = row.DurationSeconds,
URL = media.Url(row.FilePath),
PreviewURL = media.Url(row.PreviewPath ?? row.FilePath),
Description = row.Description,
Blurhash = row.Blurhash,
Width = row.Width,
Height = row.Height,
Focus = row.Focus
});
run.Count("media");
}
return made;
}
// a file of the archive given to MediaService as an upload would be
async Task<MediaOutcome> Upload(Run run, string file, string contentType, string description, string focus, CancellationToken token)
{
var temporary = Path.Combine(Path.GetTempPath(), $"privapub-import-{Guid.NewGuid():N}{Path.GetExtension(file)}");
try
{
await using (var source = run.Archive.Read(file))
await using (var target = new FileStream(temporary, FileMode.CreateNew, FileAccess.Write))
await source.CopyToAsync(target, token);
await using var stream = new FileStream(temporary, FileMode.Open, FileAccess.Read);
var form = new FormFile(stream, 0, stream.Length, "file", Path.GetFileName(file))
{
Headers = new HeaderDictionary(),
ContentType = contentType ?? (MediaService.ContentTypes.TryGetContentType(file, out var guessed) ? guessed : "application/octet-stream")
};
return await media.Upload(run.Me, form, description, focus, later: false, token);
}
finally
{
File.Delete(temporary);
}
}
// ---- PrivaPub's own parts
async Task Located(Run run, CancellationToken token)
{
foreach (var located in (run.Archive.Json("privapub/located.json") as JsonArray ?? []).OfType<JsonObject>())
{
var original = Text(located["id"]);
if (original != default && await DB.Default.Find<PostEntity>().Match(p => p.GroupUserId == run.Me.Id
&& (p.ObjectURI == original || p.ImportedFromURI == original)).ExecuteAnyAsync(token))
{
run.Count("located posts already here");
continue;
}
if (located["latitude"]?.GetValue<double>() is not { } latitude || located["longitude"]?.GetValue<double>() is not { } longitude)
continue;
var id = PrivacyIds.At(Clamp(Date(located["published"]) ?? DateTime.UtcNow));
var post = new PostEntity
{
ID = id,
GroupUserId = run.Me.Id,
AuthorAccountId = run.Me.Id,
ActorURI = run.Me.Uri,
Visibility = PostVisibility.LocalGeo,
IsLocalOnly = true,
Title = Clean(Text(located["title"])),
SpoilerText = Clean(Text(located["spoilerText"])),
HasContentWarning = Clean(Text(located["spoilerText"])) != default,
Text = Text(located["text"]),
ContentHtml = ActivityPubRenderer.Html(Text(located["text"]) ?? string.Empty),
ContentFormat = ContentFormat.Markdown,
Geo = new MongoDB.Entities.Coordinates2D(Math.Round(longitude, 2), Math.Round(latitude, 2)),
RangeKm = (float)Math.Clamp(located["rangeKm"]?.GetValue<double>() ?? 5, 1, 50),
ObjectURI = run.Me.PostUri(id),
Url = run.Me.PostHtmlUrl(id),
ImportedFromURI = original,
CreationDate = Clamp(Date(located["published"]) ?? DateTime.UtcNow)
};
var files = (located["media"] as JsonArray ?? []).OfType<JsonObject>()
.Select(m => new JsonObject { ["url"] = Text(m["url"]), ["name"] = Text(m["description"]) }).ToArray();
post.Media = await Media(run, new JsonArray(files), id, token);
await DB.Default.SaveAsync(post, token);
run.Count("located posts");
}
}
async Task Scheduled(Run run, CancellationToken token)
{
var made = new List<ScheduledStatus>();
foreach (var scheduled in (run.Archive.Json("privapub/scheduled.json") as JsonArray ?? []).OfType<JsonObject>())
{
var at = Date(scheduled["scheduledAt"]);
if (at is not { } when || when < DateTime.UtcNow.AddMinutes(5) || await ScheduledStatuses.Refusal(run.Me.Id, when, default, token) != default)
{
run.Count("scheduled posts past or refused");
continue;
}
var status = new ScheduledStatus
{
AvatarId = run.Me.Id,
ScheduledAt = when,
Params = new ScheduledParams
{
Text = Text(scheduled["text"]),
SpoilerText = Text(scheduled["spoilerText"]),
Sensitive = scheduled["sensitive"]?.GetValueKind() == System.Text.Json.JsonValueKind.True,
Visibility = Text(scheduled["visibility"]) is { } visibility && visibility is "public" or "unlisted" or "private" or "direct" ? visibility : "public",
Language = Text(scheduled["language"])
}
};
status.ID = (string)status.GenerateNewID();
foreach (var file in (scheduled["media"] as JsonArray ?? []).OfType<JsonObject>().Take(MaxMedia))
{
var path = Text(file["url"])?.TrimStart('/');
if (path == default || !run.Archive.Has(path))
continue;
var outcome = await Upload(run, path, default, Clean(Text(file["description"])), default, token);
if (outcome?.Ok != true)
continue;
await DB.Default.Update<MediaAttachment>().MatchID(outcome.Attachment.ID).Modify(m => m.ScheduledStatusId, status.ID).ExecuteAsync(token);
status.Params.MediaIds.Add(outcome.Attachment.ID);
}
await DB.Default.SaveAsync(status, token);
made.Add(status);
run.Count("scheduled posts");
}
if (made.Count > 0)
await jobs.EnqueueMany(made.Select(s => ScheduledStatuses.JobFor(s, new Uri(localActors.BaseAddress).Host)), token);
}
async Task Tags(Run run, CancellationToken token)
{
var mine = (await DB.Default.Find<FollowedTag, string>().Match(t => t.AvatarId == run.Me.Id).Project(t => t.Name).ExecuteAsync(token)).ToHashSet();
foreach (var name in (run.Archive.Json("privapub/followed_tags.json") as JsonArray ?? []).Select(t => TagsController.Normalise(Text(t))))
{
if (name.Length == 0 || mine.Contains(name) || mine.Count >= 500)
continue;
await DB.Default.SaveAsync(new FollowedTag { AvatarId = run.Me.Id, Name = name }, token);
mine.Add(name);
run.Count("followed hashtags");
}
}
async Task Policy(Run run, CancellationToken token)
{
if (run.Archive.Json("privapub/notification_policy.json") is not JsonObject json)
return;
var policy = await NotificationPolicies.Of(run.Me.Id, token);
NotificationAction Action(string field, NotificationAction kept) => Enum.TryParse<NotificationAction>(Text(json[field]), out var action) ? action : kept;
policy.ForNotFollowing = Action("forNotFollowing", policy.ForNotFollowing);
policy.ForNotFollowers = Action("forNotFollowers", policy.ForNotFollowers);
policy.ForNewAccounts = Action("forNewAccounts", policy.ForNewAccounts);
policy.ForPrivateMentions = Action("forPrivateMentions", policy.ForPrivateMentions);
policy.ForLimitedAccounts = Action("forLimitedAccounts", policy.ForLimitedAccounts);
await NotificationPolicies.Save(policy, token);
run.Count("notification policy");
}
async Task Filters(Run run, CancellationToken token)
{
var titles = (await DB.Default.Find<PersonaFilter, string>().Match(f => f.AvatarId == run.Me.Id).Project(f => f.Title).ExecuteAsync(token)).ToHashSet();
foreach (var filter in (run.Archive.Json("privapub/filters.json") as JsonArray ?? []).OfType<JsonObject>())
{
var title = Clean(Text(filter["title"]));
if (title == default || titles.Contains(title))
continue;
var statuses = new List<FilterStatus>();
foreach (var uri in Strings(filter["statuses"]))
if (await Held(run, uri, token) is { } post)
statuses.Add(new FilterStatus { Id = MongoDB.Bson.ObjectId.GenerateNewId().ToString(), PostId = post.ID });
await DB.Default.SaveAsync(new PersonaFilter
{
AvatarId = run.Me.Id,
Title = title,
Context = Strings(filter["context"]).Where(c => c is "home" or "notifications" or "public" or "thread" or "account").ToList(),
ExpiresAt = Date(filter["expiresAt"]),
Action = Text(filter["action"]) is { } action && action is "warn" or "hide" or "blur" ? action : "warn",
Keywords = (filter["keywords"] as JsonArray ?? []).OfType<JsonObject>().Where(k => Clean(Text(k["keyword"])) != default)
.Select(k => new FilterKeyword { Id = MongoDB.Bson.ObjectId.GenerateNewId().ToString(), Keyword = Text(k["keyword"]).Trim(),
WholeWord = k["wholeWord"]?.GetValueKind() == System.Text.Json.JsonValueKind.True }).ToList(),
Statuses = statuses
}, token);
titles.Add(title);
run.Count("filters");
}
}
// pinned again without announcing it (an Add would tell every follower's server)
async Task Pins(Run run, CancellationToken token)
{
var pinned = (await DB.Default.Find<Pin, string>().Match(p => p.AvatarId == run.Me.Id).Project(p => p.PostId).ExecuteAsync(token)).ToHashSet();
foreach (var uri in Strings(run.Archive.Json("privapub/pins.json")))
{
if (pinned.Count >= 5)
break;
if (!run.Posts.TryGetValue(uri, out var copy) || pinned.Contains(copy.Id))
continue;
await DB.Default.SaveAsync(new Pin { AvatarId = run.Me.Id, PostId = copy.Id }, token);
pinned.Add(copy.Id);
run.Count("pins");
}
}
// ---- relationships
async Task DomainBlocks(Run run, CancellationToken token)
{
var relationships = run.Services.GetRequiredService<IRelationshipService>();
var mine = (await DB.Default.Find<AccountDomainBlock, string>().Match(d => d.AvatarId == run.Me.Id).Project(d => d.Domain).ExecuteAsync(token)).ToHashSet();
foreach (var line in run.Archive.Lines("blocked_domains.csv"))
{
var domain = Cells(line).FirstOrDefault()?.Trim().ToLowerInvariant();
if (string.IsNullOrEmpty(domain) || domain == "#domain" || Uri.CheckHostName(domain) != UriHostNameType.Dns || mine.Contains(domain))
continue;
await relationships.BlockDomain(run.Me, domain, token);
mine.Add(domain);
run.Count("blocked servers");
}
}
// one CSV of accounts, each resolved (WebFinger or its address) and acted on; a header line is skipped
async Task Accounts(Run run, string file, string what, Func<Run, (LocalActor Local, ForeignAvatar Remote), string[], CancellationToken, Task<bool>> act,
CancellationToken token)
{
var follows = run.Services.GetRequiredService<IFollowService>();
var lines = run.Archive.Lines(file);
foreach (var line in lines)
{
if (await Stopped(run, token))
return;
var cells = Cells(line);
var address = cells.FirstOrDefault()?.Trim();
if (string.IsNullOrEmpty(address) || address.Equals("Account address", StringComparison.OrdinalIgnoreCase))
continue;
var target = await follows.Resolve(address, token);
if (target.Local == default && target.Remote == default)
{
run.Count($"{what} not found");
continue;
}
if (target.Local?.Id == run.Me.Id)
continue;
run.Count(await act(run, target, cells, token) ? what : $"{what} skipped");
}
}
static async Task<bool> Block(Run run, (LocalActor Local, ForeignAvatar Remote) target, string[] cells, CancellationToken token)
{
var (uri, id) = target.Local != default ? (target.Local.Uri, target.Local.Id) : (target.Remote.ActorURI, target.Remote.ID);
if (await DB.Default.Find<Block>().Match(b => b.AvatarId == run.Me.Id && b.TargetActorURI == uri).ExecuteAnyAsync(token))
return false;
await run.Services.GetRequiredService<IRelationshipService>().Block(run.Me, uri, id, token);
return true;
}
static async Task<bool> Mute(Run run, (LocalActor Local, ForeignAvatar Remote) target, string[] cells, CancellationToken token)
{
var (uri, id) = target.Local != default ? (target.Local.Uri, target.Local.Id) : (target.Remote.ActorURI, target.Remote.ID);
var hide = cells.Length < 2 || !cells[1].Trim().Equals("false", StringComparison.OrdinalIgnoreCase);
await run.Services.GetRequiredService<IRelationshipService>().Mute(run.Me, uri, id, hide, default, token);
return true;
}
async Task<bool> Follow(Run run, (LocalActor Local, ForeignAvatar Remote) target, string[] cells, CancellationToken token)
{
var uri = target.Local?.Uri ?? target.Remote.ActorURI;
var host = new Uri(uri).Host;
if (domainBlocks.IsSuspended(host)
|| await DB.Default.Find<AccountDomainBlock>().Match(d => d.AvatarId == run.Me.Id && d.Domain == host).ExecuteAnyAsync(token)
|| await DB.Default.Find<Following>().Match(f => f.AvatarId == run.Me.Id && f.TargetActorURI == uri).ExecuteAnyAsync(token))
return false;
var boosts = cells.Length < 2 || !cells[1].Trim().Equals("false", StringComparison.OrdinalIgnoreCase);
return await run.Services.GetRequiredService<IFollowService>().FollowAs(run.Me, uri, boosts, token) != default;
}
// lists by title, with only the accounts the persona follows
async Task Lists(Run run, CancellationToken token)
{
var follows = run.Services.GetRequiredService<IFollowService>();
var lists = (await DB.Default.Find<PersonaList>().Match(l => l.AvatarId == run.Me.Id).ExecuteAsync(token)).ToDictionary(l => l.Title, StringComparer.Ordinal);
foreach (var line in run.Archive.Lines("lists.csv"))
{
var cells = Cells(line);
if (cells.Length < 2 || Clean(cells[0]) is not { } title)
continue;
if (!lists.TryGetValue(title, out var list))
{
if (lists.Count >= 50)
continue;
list = new PersonaList { AvatarId = run.Me.Id, Title = title, RepliesPolicy = "list" };
await DB.Default.SaveAsync(list, token);
lists[title] = list;
run.Count("lists");
}
var target = await follows.Resolve(cells[1].Trim(), token);
var uri = target.Local?.Uri ?? target.Remote?.ActorURI;
var following = uri == default ? default
: await DB.Default.Find<Following>().Match(f => f.AvatarId == run.Me.Id && f.TargetActorURI == uri && f.State == FollowState.Accepted).ExecuteFirstAsync(token);
if (following == default)
{
run.Count("list members not followed");
continue;
}
if (await DB.Default.Find<PersonaListMember>().Match(m => m.ListId == list.ID && m.AccountId == following.TargetAccountId).ExecuteAnyAsync(token))
continue;
await DB.Default.SaveAsync(new PersonaListMember { ListId = list.ID, AvatarId = run.Me.Id, AccountId = following.TargetAccountId }, token);
run.Count("list members");
}
}
async Task Bookmarks(Run run, CancellationToken token)
{
var uris = Strings(run.Archive.Json("bookmarks.json")?["orderedItems"]);
if (uris.Count == 0)
uris = run.Archive.Lines("bookmarks.csv").Select(l => Cells(l).FirstOrDefault()?.Trim()).Where(u => u?.StartsWith("http") == true).ToList();
foreach (var uri in uris.Distinct())
{
if (await Stopped(run, token))
return;
var post = await Fetched(run, uri, token);
if (post == default)
{
run.Count("bookmarks not found");
continue;
}
if (await DB.Default.Find<Bookmark>().Match(b => b.AvatarId == run.Me.Id && b.PostId == post.ID).ExecuteAnyAsync(token))
continue;
await DB.Default.SaveAsync(new Bookmark { AvatarId = run.Me.Id, PostId = post.ID }, token);
run.Count("bookmarks");
}
}
// asked for on purpose: each like tells the post's author
async Task Likes(Run run, CancellationToken token)
{
var statuses = run.Services.GetRequiredService<IStatusService>();
foreach (var uri in Strings(run.Archive.Json("likes.json")?["orderedItems"]).Distinct())
{
if (await Stopped(run, token))
return;
var post = await Fetched(run, uri, token);
if (post == default)
{
run.Count("likes not found");
continue;
}
run.Count((await statuses.Favourite(run.Me, post.ID, true, token)).Ok ? "likes" : "likes not found");
}
}
// a post here by its address, or read from its server (signed) and kept
static async Task<PostEntity> Fetched(Run run, string uri, CancellationToken token)
{
if (await Held(run, uri, token) is { } held)
return held;
return await run.Services.GetRequiredService<IRemotePosts>().StoreContext(uri, 0, token);
}
static async Task<PostEntity> Held(Run run, string uri, CancellationToken token)
{
if (uri == default)
return default;
if (run.Posts.TryGetValue(uri, out var copy))
return await DB.Default.Find<PostEntity>().MatchID(copy.Id).ExecuteFirstAsync(token);
return await DB.Default.Find<PostEntity>().Match(p => (p.ObjectURI == uri || p.Url == uri) && !p.DeletedAt.HasValue).ExecuteFirstAsync(token);
}
// ---- the profile: name, words, fields, flags and pictures, saved as Settings saves them (and announced the same way)
async Task Profile(Run run, JsonObject actor, CancellationToken token)
{
var avatar = await DB.Default.Find<Avatar>().MatchID(run.Me.Id).ExecuteFirstAsync(token);
if (Clean(Text(actor["name"])) is { } name)
avatar.Name = name;
if (Text(actor["summary"]) is { } summary)
avatar.Biography = Plain(summary);
var fields = (actor["attachment"] as JsonArray ?? []).OfType<JsonObject>().Where(f => Text(f["type"]) == "PropertyValue" && Clean(Text(f["name"])) != default)
.Take(4).ToDictionary(f => Text(f["name"]).Trim(), f => Plain(Text(f["value"]) ?? string.Empty));
if (fields.Count > 0)
avatar.Fields = fields;
if (actor["manuallyApprovesFollowers"]?.GetValueKind() is { } locked and (System.Text.Json.JsonValueKind.True or System.Text.Json.JsonValueKind.False))
avatar.Settings.IsLocked = locked == System.Text.Json.JsonValueKind.True;
if (actor["discoverable"]?.GetValueKind() is { } discoverable and (System.Text.Json.JsonValueKind.True or System.Text.Json.JsonValueKind.False))
avatar.Settings.IsDiscoverable = discoverable == System.Text.Json.JsonValueKind.True;
if (actor["indexable"]?.GetValueKind() is { } indexable and (System.Text.Json.JsonValueKind.True or System.Text.Json.JsonValueKind.False))
avatar.Settings.IsIndexable = indexable == System.Text.Json.JsonValueKind.True;
foreach (var (field, kind, width, height) in new[] { ("icon", "avatar", 400, 400), ("image", "header", 1500, 500) })
{
var file = (Text((actor[field] as JsonObject)?["url"]) ?? Text(actor[field]))?.TrimStart('/');
if (file == default || file.Contains("://") || !run.Archive.Has(file))
continue;
var outcome = await Picture(run, file, kind, width, height, token);
if (outcome?.Ok != true)
{
run.Count("pictures refused");
continue;
}
if (kind == "avatar")
avatar.PictureURL = media.Url(outcome.Attachment.FilePath);
else
avatar.ThumbnailURL = media.Url(outcome.Attachment.FilePath);
await media.Trash(m => m.ProfileOfAvatarId == avatar.ID && m.Kind == kind && m.ID != outcome.Attachment.ID, "replaced", token);
}
avatar.UpdatedAt = DateTime.UtcNow;
await DB.Default.SaveAsync(avatar, token);
await run.Services.GetRequiredService<IOutboxPublisher>().PublishProfile(localActors.FromAvatar(avatar), token);
run.Count("profile");
}
async Task<MediaOutcome> Picture(Run run, string file, string kind, int width, int height, CancellationToken token)
{
var temporary = Path.Combine(Path.GetTempPath(), $"privapub-import-{Guid.NewGuid():N}{Path.GetExtension(file)}");
try
{
await using (var source = run.Archive.Read(file))
await using (var target = new FileStream(temporary, FileMode.CreateNew, FileAccess.Write))
await source.CopyToAsync(target, token);
await using var stream = new FileStream(temporary, FileMode.Open, FileAccess.Read);
var form = new FormFile(stream, 0, stream.Length, kind, Path.GetFileName(file))
{
Headers = new HeaderDictionary(),
ContentType = MediaService.ContentTypes.TryGetContentType(file, out var guessed) ? guessed : "application/octet-stream"
};
return await media.ProfileImage(run.Me.Id, kind, form, width, height, token);
}
finally
{
File.Delete(temporary);
}
}
// ---- helpers
static string Text(JsonNode node) => node is JsonValue value && value.TryGetValue<string>(out var text) ? text : default;
static string Id(JsonNode node) => node switch
{
JsonValue => Text(node),
JsonObject json => Text(json["id"]),
JsonArray list => list.Select(Id).FirstOrDefault(i => i != default),
_ => default
};
static List<string> Strings(JsonNode node) => node switch
{
JsonArray list => list.Select(Id).Where(i => i != default).ToList(),
JsonValue => Text(node) is { } one ? [one] : [],
JsonObject json when Text(json["type"]) is "OrderedCollection" or "Collection" => Strings(json["orderedItems"] ?? json["items"]),
_ => []
};
static DateTime? Date(JsonNode node) =>
DateTime.TryParse(Text(node), System.Globalization.CultureInfo.InvariantCulture,
System.Globalization.DateTimeStyles.AdjustToUniversal | System.Globalization.DateTimeStyles.AssumeUniversal, out var date) ? date : default;
static DateTime Clamp(DateTime when) => when > DateTime.UtcNow ? DateTime.UtcNow : when;
static string Clean(string text) => string.IsNullOrWhiteSpace(text) ? default : text.Trim();
static bool IsObjectId(string id) => id is { Length: 24 } && id.All(char.IsAsciiHexDigitLower);
// a CSV line's cells (quotes as Mastodon writes them)
static string[] Cells(string line)
{
var cells = new List<string>();
var cell = new System.Text.StringBuilder();
var quoted = false;
for (var i = 0; i < line.Length; i++)
{
var c = line[i];
if (quoted)
{
if (c == '"' && i + 1 < line.Length && line[i + 1] == '"')
{
cell.Append('"');
i++;
}
else if (c == '"')
quoted = false;
else
cell.Append(c);
}
else if (c == '"')
quoted = true;
else if (c == ',')
{
cells.Add(cell.ToString());
cell.Clear();
}
else
cell.Append(c);
}
cells.Add(cell.ToString());
return [.. cells];
}
// HTML to the plain words a persona writes in its profile
static string Plain(string html)
{
var text = Breaks().Replace(html, "\n");
text = Tags().Replace(text, string.Empty);
return WebUtility.HtmlDecode(text).Trim();
}
[GeneratedRegex(@"<br\s*/?>|</p>\s*<p[^>]*>", RegexOptions.IgnoreCase)]
private static partial Regex Breaks();
[GeneratedRegex(@"<[^>]+>")]
private static partial Regex Tags();
}
}
@@ -0,0 +1,79 @@
using MongoDB.Entities;
using PrivaPub.ClientModels.User;
namespace PrivaPub.Domain.Portability
{
// A persona's archive, made and restored in jobs (owner decision 2026-10-07): where each stands, one row per persona
// (its ID is the persona's). The files live in <backups>/.personas/<avatarId>/: the export, kept a week, and the upload
// being imported.
public class PersonaArchive : Entity
{
public string ExportState { get; set; } = "none";
public DateTime? ExportAskedAt { get; set; }
public DateTime? ExportReadyAt { get; set; }
public string ExportFile { get; set; }
public long ExportBytes { get; set; }
public string ExportError { get; set; }
public string ImportState { get; set; } = "none";
public DateTime? ImportAskedAt { get; set; }
public DateTime? ImportEndedAt { get; set; }
public string ImportFile { get; set; }
public List<string> ImportParts { get; set; } = [];
public long ImportDone { get; set; }
public long ImportTotal { get; set; }
public Dictionary<string, int> ImportCounts { get; set; } = [];
public string ImportError { get; set; }
public string ImportFrom { get; set; }
public bool ImportSameActor { get; set; }
public bool StopAsked { get; set; }
}
public static class PersonaArchives
{
public static readonly TimeSpan Kept = TimeSpan.FromDays(7);
/// <summary>Every part an import can take; the last three only when asked (they act, or tell others).</summary>
public static readonly string[] Parts = ["posts", "profile", "follows", "blocks", "mutes", "domainblocks", "lists", "bookmarks", "filters", "tags",
"notificationpolicy", "pins", "located", "scheduled", "likes"];
public static string Folder(string backupsRoot, string avatarId) => Path.Combine(backupsRoot, ".personas", avatarId);
public static async Task<PersonaArchive> Of(string avatarId, CancellationToken token) =>
await DB.Default.Find<PersonaArchive>().MatchID(avatarId).ExecuteFirstAsync(token) ?? new PersonaArchive { ID = avatarId };
public static ViewPersonaArchive View(PersonaArchive archive) => new()
{
ExportState = archive.ExportState,
ExportAskedAt = archive.ExportAskedAt,
ExportReadyAt = archive.ExportReadyAt,
ExportExpiresAt = archive.ExportReadyAt + Kept,
ExportBytes = archive.ExportBytes,
ExportError = archive.ExportError,
ImportState = archive.ImportState,
ImportAskedAt = archive.ImportAskedAt,
ImportEndedAt = archive.ImportEndedAt,
ImportParts = archive.ImportParts,
ImportDone = archive.ImportDone,
ImportTotal = archive.ImportTotal,
ImportCounts = archive.ImportCounts,
ImportError = archive.ImportError,
ImportFrom = archive.ImportFrom,
ImportSameActor = archive.ImportSameActor
};
/// <summary>Exports older than a week go, as do imports' files left behind.</summary>
public static async Task Forget(string backupsRoot, CancellationToken token)
{
var folder = Path.Combine(backupsRoot, ".personas");
if (!Directory.Exists(folder))
return;
var before = DateTime.UtcNow - Kept;
foreach (var file in Directory.EnumerateFiles(folder, "*", SearchOption.AllDirectories).Where(f => File.GetLastWriteTimeUtc(f) < before))
File.Delete(file);
await DB.Default.Update<PersonaArchive>().Match(a => a.ExportState == "ready" && a.ExportReadyAt < before)
.Modify(a => a.ExportState, "none").Modify(a => a.ExportFile, null).ExecuteAsync(token);
}
}
}
+270
View File
@@ -0,0 +1,270 @@
using System.IO.Compression;
using System.Text.Json;
using System.Text.Json.Nodes;
namespace PrivaPub.Domain.Portability
{
// An archive someone uploads, opened only if nothing in it can harm the server: no link, no path leaving it (.. or
// absolute), no name twice, at most MaxEntries files and MaxBytes in all (and what the disk has), and no file
// compressed more than MaxRatio to one. Every file is read through a stream that stops at its stated size, and the
// outbox one item at a time, none larger than MaxItemBytes, so a large archive never sits in memory.
public sealed class SafeArchive : IDisposable
{
public const int MaxEntries = 200_000;
public const long MaxBytes = 64L * 1024 * 1024 * 1024;
public const int MaxRatio = 100;
public const int MaxItemBytes = 1024 * 1024;
public const int MaxJsonBytes = 64 * 1024 * 1024;
readonly ZipArchive _zip;
readonly Dictionary<string, ZipArchiveEntry> _entries;
readonly string _root;//the folder the archive's files sit in, when it wraps them in one
SafeArchive(ZipArchive zip, Dictionary<string, ZipArchiveEntry> entries, string root)
{
_zip = zip;
_entries = entries;
_root = root;
}
public static (SafeArchive Archive, string Error) Open(string path)
{
ZipArchive zip;
try
{
zip = ZipFile.OpenRead(path);
}
catch (InvalidDataException)
{
return (default, "not a zip archive");
}
var entries = new Dictionary<string, ZipArchiveEntry>(StringComparer.Ordinal);
var total = 0L;
var free = new DriveInfo(Path.GetFullPath(path)).AvailableFreeSpace;
string Refuse(string why)
{
zip.Dispose();
return why;
}
if (zip.Entries.Count > MaxEntries)
return (default, Refuse("too many files"));
foreach (var entry in zip.Entries)
{
var name = entry.FullName;
if (name.Contains('\\') || name.StartsWith('/') || name.Contains(':') || name.Split('/').Any(part => part is ".." or "."))
return (default, Refuse($"{name}: a path that leaves the archive"));
if (((entry.ExternalAttributes >> 16) & 0xF000) == 0xA000)
return (default, Refuse($"{name}: a link"));
if (name.EndsWith('/'))
continue;
if (!entries.TryAdd(name, entry))
return (default, Refuse($"{name}: twice"));
total += entry.Length;
if (total > MaxBytes || total > free)
return (default, Refuse("larger than the server can take"));
if (entry.Length > 1024 * 1024 && entry.Length > (long)MaxRatio * Math.Max(1, entry.CompressedLength))
return (default, Refuse($"{name}: compressed more than {MaxRatio} to one"));
}
// some archivers wrap everything in one folder
var root = string.Empty;
if (!entries.ContainsKey("actor.json") && entries.Keys.FirstOrDefault(k => k.EndsWith("/actor.json", StringComparison.Ordinal)) is { } nested
&& nested.Count(c => c == '/') == 1)
root = nested[..(nested.IndexOf('/') + 1)];
return (new SafeArchive(zip, entries, root), default);
}
public bool Has(string name) => _entries.ContainsKey(_root + name);
public long Length(string name) => _entries.TryGetValue(_root + name, out var entry) ? entry.Length : -1;
/// <summary>A file's contents, never more than its stated size.</summary>
public Stream Read(string name) =>
_entries.TryGetValue(_root + name, out var entry) ? new Bounded(entry.Open(), entry.Length) : default;
/// <summary>A JSON file, or null when it is missing, too large or not JSON.</summary>
public JsonNode Json(string name)
{
if (!_entries.TryGetValue(_root + name, out var entry) || entry.Length > MaxJsonBytes)
return default;
try
{
using var stream = Read(name);
return JsonNode.Parse(stream);
}
catch (JsonException)
{
return default;
}
}
/// <summary>The lines of a text file (a CSV), none when it is missing or too large.</summary>
public List<string> Lines(string name)
{
if (!_entries.TryGetValue(_root + name, out var entry) || entry.Length > MaxJsonBytes)
return [];
using var reader = new StreamReader(Read(name));
var lines = new List<string>();
while (reader.ReadLine() is { } line)
if (line.Length > 0)
lines.Add(line);
return lines;
}
/// <summary>outbox.json's orderedItems, one object at a time; an item larger than MaxItemBytes stops it.</summary>
public async IAsyncEnumerable<JsonObject> Outbox([System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken token)
{
await using var stream = Read("outbox.json");
if (stream == default)
yield break;
var buffer = new byte[64 * 1024];
var filled = 0;
var final = false;
var state = new JsonReaderState();
var phase = Phase.Seeking;
while (phase != Phase.Done)
{
if (!final)
{
if (filled == buffer.Length)
{
if (buffer.Length > MaxItemBytes + 64 * 1024)
throw new InvalidDataException($"an outbox item larger than {MaxItemBytes / 1024} KiB");
Array.Resize(ref buffer, buffer.Length * 2);
}
var read = await stream.ReadAsync(buffer.AsMemory(filled), token);
filled += read;
final = read == 0;
}
var (items, consumed, next, after) = Scan(buffer.AsSpan(0, filled), final, state, phase);
foreach (var item in items)
yield return item;
(state, phase) = (next, after);
Buffer.BlockCopy(buffer, consumed, buffer, 0, filled - consumed);
filled -= consumed;
if (final && consumed == 0 && items.Count == 0)
break;
}
}
enum Phase { Seeking, Array, Done }
static (List<JsonObject> Items, int Consumed, JsonReaderState State, Phase Phase) Scan(ReadOnlySpan<byte> data, bool final, JsonReaderState state, Phase phase)
{
var items = new List<JsonObject>();
var reader = new Utf8JsonReader(data, final, state);
while (phase != Phase.Done)
{
var mark = reader;
if (!reader.Read())
{
reader = mark;
break;
}
if (phase == Phase.Seeking)
{
if (reader.TokenType == JsonTokenType.PropertyName && reader.CurrentDepth == 1 && reader.ValueTextEquals("orderedItems"))
{
var before = reader;
if (!reader.Read())
{
reader = mark;
break;
}
if (reader.TokenType == JsonTokenType.StartArray)
phase = Phase.Array;
else
{
reader = before;
if (!reader.TrySkip())
{
reader = mark;
break;
}
}
continue;
}
if (reader.TokenType is JsonTokenType.StartObject or JsonTokenType.StartArray && reader.CurrentDepth >= 1 && !reader.TrySkip())
{
reader = mark;
break;
}
continue;
}
if (reader.TokenType == JsonTokenType.EndArray)
{
phase = Phase.Done;
break;
}
if (reader.TokenType != JsonTokenType.StartObject)
{
if (reader.TokenType == JsonTokenType.StartArray && !reader.TrySkip())
{
reader = mark;
break;
}
continue;
}
var start = (int)reader.TokenStartIndex;
if (!reader.TrySkip())
{
reader = mark;
break;
}
var length = (int)reader.BytesConsumed - start;
if (length > MaxItemBytes)
throw new InvalidDataException($"an outbox item larger than {MaxItemBytes / 1024} KiB");
if (JsonNode.Parse(data.Slice(start, length)) is JsonObject item)
items.Add(item);
}
return (items, (int)reader.BytesConsumed, reader.CurrentState, phase);
}
public void Dispose() => _zip.Dispose();
// a stream that ends at the size the archive states, whatever the compressed data says
sealed class Bounded(Stream inner, long length) : Stream
{
long _left = length;
public override int Read(byte[] buffer, int offset, int count) => Read(buffer.AsSpan(offset, count));
public override int Read(Span<byte> buffer)
{
if (_left <= 0)
return 0;
var read = inner.Read(buffer[..(int)Math.Min(buffer.Length, _left)]);
_left -= read;
return read;
}
public override async ValueTask<int> ReadAsync(Memory<byte> buffer, CancellationToken token = default)
{
if (_left <= 0)
return 0;
var read = await inner.ReadAsync(buffer[..(int)Math.Min(buffer.Length, _left)], token);
_left -= read;
return read;
}
public override Task<int> ReadAsync(byte[] buffer, int offset, int count, CancellationToken token) =>
ReadAsync(buffer.AsMemory(offset, count), token).AsTask();
public override bool CanRead => true;
public override bool CanSeek => false;
public override bool CanWrite => false;
public override long Length => length;
public override long Position { get => length - _left; set => throw new NotSupportedException(); }
public override void Flush() { }
public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException();
public override void SetLength(long value) => throw new NotSupportedException();
public override void Write(byte[] buffer, int offset, int count) => throw new NotSupportedException();
protected override void Dispose(bool disposing)
{
if (disposing)
inner.Dispose();
base.Dispose(disposing);
}
}
}
}
+19
View File
@@ -1,3 +1,4 @@
using MongoDB.Driver;
using MongoDB.Entities;
using PrivaPub.Federation.Actors;
@@ -64,6 +65,24 @@ namespace PrivaPub.Domain.Privacy
return distinct.Count == 0 ? 0 : await DB.Default.CountAsync<Avatar>(a => distinct.Contains(a.ID) && !a.DeletionAt.HasValue, token);
}
// the media kept (not trashed) that which holds: their bytes and how many
public static async Task<(long Bytes, long Files)> Media(Expression<Func<Models.Media.MediaAttachment, bool>> which, CancellationToken token)
{
var totals = await DB.Default.Fluent<Models.Media.MediaAttachment>()
.Match(m => m.TrashedAt == null)
.Match(which)
.Group(m => true, g => new { Bytes = g.Sum(m => m.Size), Files = g.LongCount() })
.FirstOrDefaultAsync(token);
return totals == default ? (0, 0) : (totals.Bytes, totals.Files);
}
// a login's media: every persona's uploads and pictures
public static async Task<(long Bytes, long Files)> MediaOfRoot(string rootId, CancellationToken token)
{
var personas = (await DB.Default.Find<Models.User.RootToAvatar>().Match(r => r.RootId == rootId).ExecuteAsync(token)).Select(r => r.AvatarId).ToList();
return await Media(m => personas.Contains(m.OwnerAvatarId), token);
}
public static async Task<long> LocalPosts(CancellationToken token)
{
var barred = await BarredPersonas(token);
+134
View File
@@ -0,0 +1,134 @@
using MongoDB.Driver;
using MongoDB.Entities;
using PrivaPub.Federation.Actors;
using PrivaPub.Infrastructure.Jobs;
using PrivaPub.Models.Jobs;
using PrivaPub.Models.Social;
using PrivaPub.Models.User;
using PrivaPub.StaticServices;
namespace PrivaPub.Domain.Social
{
public interface IFeedFollows
{
Task<bool> Subscribe(LocalActor me, ForeignAvatar feed, bool on, CancellationToken token);
Task<bool> IsSubscribed(string avatarId, string feedActorUri, CancellationToken token);
Task Reconcile(CancellationToken token);
}
// Feeds read through the server (owner decision 2026-10-07): a persona following a feed is kept here alone, and the
// server's reader (LocalActorService.ReaderUserName) follows every community in the feeds anyone reads, as Lemmy's own
// follower of multi-communities does, and stops following one no feed holds any more. What it follows arrives as any
// community's posts do; nobody's home shows them, the feed's timeline does.
public class FeedFollows : IFeedFollows
{
public const int MaxCommunities = 500;
static readonly TimeSpan Stale = TimeSpan.FromDays(1);
readonly DbEntities _dbEntities;
readonly ILocalActorService _localActors;
readonly IRemoteActorService _remoteActors;
readonly IFollowService _follows;
readonly IJobQueue _jobs;
public FeedFollows(DbEntities dbEntities, ILocalActorService localActors, IRemoteActorService remoteActors, IFollowService follows, IJobQueue jobs)
{
_dbEntities = dbEntities;
_localActors = localActors;
_remoteActors = remoteActors;
_follows = follows;
_jobs = jobs;
}
public static Task<bool> Sync(IJobQueue jobs, CancellationToken token) =>
jobs.Enqueue(JobKind.SyncFeeds, string.Empty, default, $"feeds|{Guid.NewGuid():N}", token);
public async Task<bool> Subscribe(LocalActor me, ForeignAvatar feed, bool on, CancellationToken token)
{
if (feed is not { AvatarType: AvatarType.Feed })
return false;
if (on)
try
{
await DB.Default.SaveAsync(new FeedSubscription { AvatarId = me.Id, FeedActorURI = feed.ActorURI, FeedAccountId = feed.ID }, token);
}
catch (MongoWriteException ex) when (ex.WriteError?.Category == ServerErrorCategory.DuplicateKey)
{
return true;
}
else if ((await DB.Default.DeleteAsync<FeedSubscription>(s => s.AvatarId == me.Id && s.FeedActorURI == feed.ActorURI)).DeletedCount == 0)
return true;
await Sync(_jobs, token);
return true;
}
public Task<bool> IsSubscribed(string avatarId, string feedActorUri, CancellationToken token) =>
DB.Default.Find<FeedSubscription>().Match(s => s.AvatarId == avatarId && s.FeedActorURI == feedActorUri).ExecuteAnyAsync(token);
public async Task Reconcile(CancellationToken token)
{
var read = (await DB.Default.Find<FeedSubscription, string>().Project(s => s.FeedActorURI).ExecuteAsync(token)).Distinct().ToList();
var feeds = await _dbEntities.ForeignAvatars.Match(a => read.Contains(a.ActorURI) && a.AvatarType == AvatarType.Feed && !a.DeletionAt.HasValue)
.ExecuteAsync(token);
// a feed's communities are as its server listed them a day ago at most (AccountCountsJob reads them, then asks
// for this again)
foreach (var stale in feeds.Where(f => !(DateTime.UtcNow - f.CountedAt < Stale)))
if (Uri.TryCreate(stale.ActorURI, UriKind.Absolute, out var uri))
await _jobs.Enqueue(JobKind.CountAccount, stale.ActorURI, uri.Host.ToLowerInvariant(), AccountCountsJob.DedupeKey(stale.ActorURI, DateTime.UtcNow), token);
var ours = _localActors.BaseAddress + "/";
var wanted = feeds.SelectMany(f => f.FeedCommunities).Where(c => !c.StartsWith(ours, StringComparison.Ordinal))
.Distinct(StringComparer.Ordinal).Take(MaxCommunities).ToHashSet(StringComparer.Ordinal);
var reader = await _localActors.GetReaderActor(token);
var following = await _dbEntities.Followings.Match(f => f.AvatarId == reader.Id).ExecuteAsync(token);
foreach (var gone in following.Where(f => !wanted.Contains(f.TargetActorURI)))
await _follows.UnfollowAs(reader, gone.TargetActorURI, token);
var followed = following.Select(f => f.TargetActorURI).ToHashSet(StringComparer.Ordinal);
foreach (var community in wanted.Where(c => !followed.Contains(c)))
// only a community: a feed lists nothing else, and the reader follows nothing else
if (await _remoteActors.GetActor(community, refresh: false, token) is { AvatarType: AvatarType.Group })
await _follows.FollowAs(reader, community, showReblogs: false, token);
}
}
public class FeedSyncJob(IServiceScopeFactory scopes) : IJobHandler
{
public JobKind Kind => JobKind.SyncFeeds;
public int Concurrency => 1;
public int MaxAttempts => 2;
public int PerHostLimit => 1;
public async Task<JobOutcome> Handle(Job job, CancellationToken token)
{
using var scope = scopes.CreateScope();
await scope.ServiceProvider.GetRequiredService<IFeedFollows>().Reconcile(token);
return JobOutcome.Done;
}
}
// brings the reader's follows up to date with the feeds read here, a minute after start and then every six hours
public sealed class FeedReader(IServiceScopeFactory scopes, ILogger<FeedReader> logger) : BackgroundService
{
static readonly TimeSpan Every = TimeSpan.FromHours(6);
protected override async Task ExecuteAsync(CancellationToken token)
{
await Task.Delay(TimeSpan.FromMinutes(1), token);
using var timer = new PeriodicTimer(Every);
do
{
try
{
using var scope = scopes.CreateScope();
await FeedFollows.Sync(scope.ServiceProvider.GetRequiredService<IJobQueue>(), token);
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
logger.LogWarning(ex, "Feeds could not be brought up to date");
}
}
while (await timer.WaitForNextTickAsync(token));
}
}
}
+6 -1
View File
@@ -30,6 +30,8 @@ namespace PrivaPub.Domain.Social
Task UnfollowAs(LocalActor follower, string target, CancellationToken token);
Task<bool> Decide(LocalActor me, string followerAccountId, bool accept, CancellationToken token);
Task<int> ResendPending(DateTime now, CancellationToken token);
/// <summary>An account named by its address or its handle (user@host): a persona here, or one elsewhere.</summary>
Task<(LocalActor Local, ForeignAvatar Remote)> Resolve(string target, CancellationToken token);
}
public class FollowService : IFollowService
@@ -117,6 +119,7 @@ namespace PrivaPub.Domain.Social
var following = new Following
{
AvatarId = follower.Id,
FollowerKind = follower.Kind == LocalActorKind.Person ? default(LocalActorKind?) : follower.Kind,
TargetActorURI = targetUri,
TargetAccountId = local?.Id ?? remote.ID,
TargetIsLocal = local != default,
@@ -257,6 +260,8 @@ namespace PrivaPub.Domain.Social
follower.Id, follower.Uri, default, token);
}
public Task<(LocalActor Local, ForeignAvatar Remote)> Resolve(string target, CancellationToken token) => ResolveTarget(target, token);
async Task<(LocalActor Local, ForeignAvatar Remote)> ResolveTarget(string target, CancellationToken token)
{
target = target?.Trim();
@@ -294,7 +299,7 @@ namespace PrivaPub.Domain.Social
.ExecuteAsync(token);
foreach (var following in pending.Where(f => (f.ResentAt ?? f.CreatedAt) + ResendAfter[f.Resent] <= now))
{
var follower = await _localActors.FindById(LocalActorKind.Person, following.AvatarId, token);
var follower = await _localActors.FindById(following.FollowerKind ?? LocalActorKind.Person, following.AvatarId, token);
var inbox = following.TargetInboxURL ?? (await _remoteActors.GetActor(following.TargetActorURI, refresh: false, token))?.InboxURL;
if (follower != default && !string.IsNullOrEmpty(inbox))
{
+2 -1
View File
@@ -8,7 +8,7 @@ namespace PrivaPub.Domain.Social
public static class Notifications
{
public static async Task Add(string avatarId, NotificationType type, string fromAccountId, string fromActorUri, string postId,
CancellationToken token, string emoji = default, string emojiUrl = default)
CancellationToken token, string emoji = default, string emojiUrl = default, string text = default)
{
if (string.IsNullOrEmpty(avatarId) || avatarId == fromAccountId && type != NotificationType.Poll)
return;
@@ -28,6 +28,7 @@ namespace PrivaPub.Domain.Social
PostId = postId,
Emoji = emoji,
EmojiURL = emojiUrl,
Text = text,
DedupeKey = emoji == default ? $"{type}|{avatarId}|{fromActorUri}|{postId}" : $"{type}|{avatarId}|{fromActorUri}|{postId}|{emoji}",
Filtered = action == NotificationAction.Filter
};
@@ -74,10 +74,18 @@ namespace PrivaPub.Domain.Statistics
Written = server.Sum(d => d.LedgerWritten),
Dropped = server.Sum(d => d.LedgerDropped),
Failed = server.Sum(d => d.LedgerFailed)
}
},
Media = await MediaTotals(token)
};
}
// the media rows kept; the disk's caches are added by whoever knows where they are
static async Task<ViewMediaTotals> MediaTotals(CancellationToken token)
{
var (bytes, files) = await Domain.Privacy.Counted.Media(m => true, token);
return new ViewMediaTotals { MediaBytes = bytes, MediaFiles = files };
}
public async Task<ViewHostsPage> Hosts(int days, string sort, string software, int page, int limit, CancellationToken token)
{
var hostDays = await Days(days, default, token);
+21 -2
View File
@@ -7,6 +7,8 @@ using PrivaPub.Models.Jobs;
using PrivaPub.Models.Media;
using PrivaPub.Models.Post;
using PrivaPub.Models.Social;
using PrivaPub.Models.User;
using PrivaPub.Domain.Media;
namespace PrivaPub.Domain.Statuses
{
@@ -19,8 +21,9 @@ namespace PrivaPub.Domain.Statuses
public const int MaxWaiting = 300;
public const int MaxPerDay = 25;
public static StatusDraft Draft(ScheduledParams p) => new()
public static StatusDraft Draft(ScheduledParams p, string scheduledId = default) => new()
{
ScheduledStatusId = scheduledId,
Text = p.Text,
PlainText = true,
SpoilerText = p.SpoilerText,
@@ -66,6 +69,15 @@ namespace PrivaPub.Domain.Statuses
RunAt = scheduled.ScheduledAt
};
// whether a persona may still publish: neither deleted nor its root deleted or banned
public static async Task<bool> MayPublish(string avatarId, CancellationToken token)
{
if (!await DB.Default.Find<Avatar>().Match(a => a.ID == avatarId && !a.DeletionAt.HasValue).ExecuteAnyAsync(token))
return false;
var rootIds = (await DB.Default.Find<RootToAvatar>().Match(r => r.AvatarId == avatarId).ExecuteAsync(token)).Select(r => r.RootId).ToList();
return await DB.Default.Find<RootUser>().Match(u => rootIds.Contains(u.ID) && u.DeletedAt == null && !u.IsBanned).ExecuteAnyAsync(token);
}
// the post's media go back to being plain uploads (the janitor's after a day), or are attached to the post by now
public static Task Release(string scheduledId, CancellationToken token) =>
DB.Default.Update<MediaAttachment>().Match(m => m.ScheduledStatusId == scheduledId).Modify(m => m.ScheduledStatusId, null).ExecuteAsync(token);
@@ -93,10 +105,17 @@ namespace PrivaPub.Domain.Statuses
if (scheduled == default || scheduled.ScheduledAt > DateTime.UtcNow.AddSeconds(30))
return JobOutcome.Done;//dropped, or moved to a later time that has its own job
using var scope = _scopes.CreateScope();
// a persona deleted or banned since it was scheduled publishes nothing, and what it held is trashed
if (!await ScheduledStatuses.MayPublish(scheduled.AvatarId, token))
{
await scope.ServiceProvider.GetRequiredService<IMediaService>().Trash(m => m.ScheduledStatusId == scheduled.ID, "its author is gone", token);
await DB.Default.DeleteAsync<ScheduledStatus>(scheduled.ID);
return JobOutcome.Done;
}
var author = await scope.ServiceProvider.GetRequiredService<ILocalActorService>().FindById(LocalActorKind.Person, scheduled.AvatarId, token);
if (author != default)
{
var outcome = await scope.ServiceProvider.GetRequiredService<IStatusService>().Publish(author, ScheduledStatuses.Draft(scheduled.Params), token);
var outcome = await scope.ServiceProvider.GetRequiredService<IStatusService>().Publish(author, ScheduledStatuses.Draft(scheduled.Params, scheduled.ID), token);
if (!outcome.Ok)
_logger.LogWarning("Scheduled status {Id} of {Avatar} could not be published: {Error}", scheduled.ID, scheduled.AvatarId, outcome.Error);
}
+146 -11
View File
@@ -24,6 +24,9 @@ using PostEntity = PrivaPub.Models.Post.Post;
namespace PrivaPub.Domain.Statuses
{
// an attachment's description or focal point changed by an edit (Mastodon's media_attributes)
public sealed record MediaChange(string Id, bool HasDescription, string Description, string Focus);
public sealed class StatusDraft
{
public string Text { get; init; }
@@ -38,6 +41,9 @@ namespace PrivaPub.Domain.Statuses
public string ConversationId { get; init; }
public IReadOnlyList<string> Recipients { get; init; } = Array.Empty<string>();
public IReadOnlyList<string> MediaIds { get; init; }
public IReadOnlyList<MediaChange> MediaChanges { get; init; } = Array.Empty<MediaChange>();
// the scheduled post being published, whose media are reserved for it alone
public string ScheduledStatusId { get; init; }
public double? Latitude { get; init; }
public double? Longitude { get; init; }
public double? RangeKm { get; init; }
@@ -58,6 +64,7 @@ namespace PrivaPub.Domain.Statuses
Task<StatusOutcome> Edit(LocalActor author, string postId, StatusDraft draft, CancellationToken token);
Task<StatusOutcome> Remove(LocalActor author, string postId, CancellationToken token);
Task<StatusOutcome> Favourite(LocalActor me, string postId, bool on, CancellationToken token);
Task<StatusOutcome> Downvote(LocalActor me, string postId, bool on, CancellationToken token);
Task<StatusOutcome> Reblog(LocalActor me, string postId, bool on, PostVisibility visibility, CancellationToken token);
}
@@ -109,7 +116,7 @@ namespace PrivaPub.Domain.Statuses
{
if (TooLong(draft) is { } tooLong)
return tooLong;
var media = await Media(author, draft.MediaIds, default, token);
var media = await Media(author, draft.MediaIds, default, draft.ScheduledStatusId, token);
if (media == default)
return StatusOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: Media attachments are invalid");
if (string.IsNullOrWhiteSpace(draft.Text) && media.Count == 0 && draft.Poll == default)
@@ -225,6 +232,9 @@ namespace PrivaPub.Domain.Statuses
post.ID = (string)post.GenerateNewID();
post.ObjectURI = author.PostUri(post.ID);
post.Url = author.PostHtmlUrl(post.ID);
// the media are this post's before anything is written: another post claiming them meanwhile wins, and this one is refused
if (!await Claim(media, post.ID, draft.ScheduledStatusId, token))
return StatusOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: Media attachments are invalid");
post.LocalQuotePolicy = QuotePolicies.IsKnown(draft.QuotePolicy) ? draft.QuotePolicy : author.Settings.QuotePolicy ?? QuotePolicies.Public;
if (quoted is { IsFederatedCopy: false } && post.QuoteState == QuoteState.Accepted
&& await _localActors.FindById(LocalActorKind.Person, quoted.GroupUserId, token) is { } quotedAuthor)
@@ -270,7 +280,6 @@ namespace PrivaPub.Domain.Statuses
post.ActivityURI = create?["id"]?.GetValue<string>();
await DB.Default.SaveAsync(post, token);
await Attach(media, post.ID, token);
if (parent != default && Counted.Reply(post))
await DB.Default.Update<PostEntity>().MatchID(parent.ID).Modify(b => b.Inc(p => p.RepliesCount, 1)).ExecuteAsync(token);
await _fanout.Distribute(post, token);
@@ -297,7 +306,7 @@ namespace PrivaPub.Domain.Statuses
return StatusOutcome.Fail(StatusCodes.Status404NotFound, "Record not found");
if (TooLong(draft) is { } tooLong)
return tooLong;
var media = draft.MediaIds == default ? default : await Media(author, draft.MediaIds, post.ID, token);
var media = draft.MediaIds == default ? default : await Media(author, draft.MediaIds, post.ID, default, token);
if (draft.MediaIds != default && media == default)
return StatusOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: Media attachments are invalid");
if (string.IsNullOrWhiteSpace(draft.Text) && (media ?? new List<MediaAttachment>()).Count == 0 && post.Media.Count == 0)
@@ -316,12 +325,34 @@ namespace PrivaPub.Domain.Statuses
var plain = draft.PlainText || post.ContentFormat == ContentFormat.Plain;
var rendered = plain ? await _content.PlainText(draft.Text ?? string.Empty, token) : await _content.Markdown(draft.Text ?? string.Empty, token);
// descriptions and focal points changed by the edit, on the media it keeps or those the post already has
var described = media ?? (draft.MediaChanges.Count > 0
? await DB.Default.Find<MediaAttachment>().Match(m => m.PostId == post.ID && m.TrashedAt == null).ExecuteAsync(token)
: default);
foreach (var change in draft.MediaChanges)
{
if (described?.FirstOrDefault(m => m.ID == change.Id) is not { } attachment)
continue;
if (change.HasDescription)
attachment.Description = string.IsNullOrWhiteSpace(change.Description) ? default : change.Description.Trim()[..Math.Min(change.Description.Trim().Length, 1500)];
if (Domain.Media.FocalPoint.Parse(change.Focus) is { } focus)
attachment.Focus = focus;
await DB.Default.Update<MediaAttachment>().MatchID(attachment.ID)
.Modify(m => m.Description, attachment.Description).Modify(m => m.Focus, attachment.Focus).ExecuteAsync(token);
}
if (media == default && described != default)
post.Media = post.Media.Select(copy => described.FirstOrDefault(m => m.ID == copy.AttachmentId) is { } changed ? ToPostMedia(changed) : copy).ToList();
if (media != default)
{
if (!await Claim(media, post.ID, default, token))
return StatusOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: Media attachments are invalid");
post.Media = media.Select(ToPostMedia).ToList();
await Attach(media, post.ID, token);
// what the edit left out is no longer held by anything
var kept = media.Select(m => m.ID).ToList();
await _media.Trash(m => m.PostId == post.ID && !kept.Contains(m.ID), "edited out", token);
}
post.Title = draft.Title == default ? post.Title : Clean(draft.Title);
// as on Mastodon, an edit sends the whole post: no warning sent is no warning (the language alone falls back)
post.SpoilerText = Clean(draft.SpoilerText);
post.HasContentWarning = draft.Sensitive || post.SpoilerText != default;
post.Text = draft.Text;
@@ -380,6 +411,7 @@ namespace PrivaPub.Domain.Statuses
.Modify(p => p.Revisions, new List<PostRevision>())
.ExecuteAsync(token);
await Fanout.Deleting(post, token);
await _media.Trash(m => m.PostId == post.ID, "post deleted", token);
await DB.Default.DeleteAsync<TimelineEntry>(e => e.PostId == post.ID || e.ReblogOfPostId == post.ID);
// boosts of it end with it, as on Mastodon, so no count keeps them
await DB.Default.Update<PostEntity>().Match(p => p.ReblogOfPostId == post.ID && !p.DeletedAt.HasValue)
@@ -407,6 +439,9 @@ namespace PrivaPub.Domain.Statuses
var likePermission = on ? await Permission(post, me, InteractionKind.Like, token) : QuotePermission.Granted;
if (likePermission == QuotePermission.Denied)
return StatusOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: Its author does not take likes from you");
// one vote a post: an upvote replaces the downvote, here and, by its Like alone, there
if (on)
await Forget<Downvote>(d => d.PostId == post.ID && d.ActorURI == me.Uri, post, p => p.DownvotesCount, token);
// each favourite is a Like of its own, as Mastodon's are: a favourite after an unfavourite is a new Like, which no
// server (and no delivery queue) takes for the one already undone; an unfavourite undoes the Like it ends
@@ -441,7 +476,7 @@ namespace PrivaPub.Domain.Statuses
post.FavouritesCount--;
}
if (post.IsFederatedCopy && await AuthorInbox(post, token) is { } inbox)
if (post.IsFederatedCopy && await VoteInboxes(post, token) is { Count: > 0 } inboxes)
{
var like = new JsonObject
{
@@ -454,11 +489,95 @@ namespace PrivaPub.Domain.Statuses
if (on && favourite.Approval == ApprovalState.Pending)
await _approvals.Ask(me, post, InteractionKind.Like, like, favourite.ID, token);
else
await _delivery.Enqueue(me, new[] { inbox }, on ? like : Undo(me, like, $"undo-like-{favourite.ID}"), token);
await _delivery.Enqueue(me, inboxes, on ? like : Undo(me, like, $"undo-like-{favourite.ID}"), token);
}
return new StatusOutcome(post);
}
// A downvote, as the threadiverse counts them (Lemmy, PieFed): a `Dislike` to the post's author and its community,
// `Undo{Dislike}` to take it back. One vote a post: a downvote replaces the upvote by its Dislike alone, as Lemmy
// sends a changed vote, since an Undo{Like} sent beside it could arrive after it and take the downvote away (Mbin,
// which keeps a favourite apart, keeps it). Public, as a favourite is (owner decision: the client tells each
// persona once).
public async Task<StatusOutcome> Downvote(LocalActor me, string postId, bool on, CancellationToken token)
{
var post = await Visible(me, postId, token);
if (post?.ReblogOfPostId != default)
post = await Visible(me, post.ReblogOfPostId, token);
if (post == default)
return StatusOutcome.Fail(StatusCodes.Status404NotFound, "Record not found");
Downvote downvote;
if (on)
{
await Forget<Favourite>(f => f.AccountId == me.Id && f.PostId == post.ID, post, p => p.FavouritesCount, token);
downvote = new Downvote { ID = ObjectId.GenerateNewId().ToString(), AccountId = me.Id, ActorURI = me.Uri, PostId = post.ID };
downvote.ActivityURI = me.ActivityUri($"dislike-{downvote.ID}");
try
{
await DB.Default.SaveAsync(downvote, token);
}
catch (MongoWriteException ex) when (ex.WriteError?.Category == ServerErrorCategory.DuplicateKey)
{
return new StatusOutcome(post);
}
await DB.Default.Update<PostEntity>().MatchID(post.ID).Modify(b => b.Inc(p => p.DownvotesCount, 1)).ExecuteAsync(token);
post.DownvotesCount++;
}
else
{
downvote = await DB.Default.Find<Downvote>().Match(d => d.PostId == post.ID && d.ActorURI == me.Uri).ExecuteFirstAsync(token);
if (downvote == default || (await DB.Default.DeleteAsync<Downvote>(downvote.ID)).DeletedCount == 0)
return new StatusOutcome(post);
await DB.Default.Update<PostEntity>().MatchID(post.ID).Modify(b => b.Inc(p => p.DownvotesCount, -1)).ExecuteAsync(token);
post.DownvotesCount--;
}
if (post.IsFederatedCopy && await VoteInboxes(post, token) is { Count: > 0 } inboxes)
{
var dislike = new JsonObject
{
["@context"] = ActivityPubRenderer.ActivityStreams,
["id"] = downvote.ActivityURI,
["type"] = "Dislike",
["actor"] = me.Uri,
["object"] = post.ObjectURI
};
await _delivery.Enqueue(me, inboxes, on ? dislike : Undo(me, dislike, $"undo-dislike-{downvote.ID}"), token);
}
return new StatusOutcome(post);
}
// the persona's other vote on the post, dropped here alone: the vote that replaces it says so there
async Task Forget<TVote>(System.Linq.Expressions.Expression<Func<TVote, bool>> mine, PostEntity post,
System.Linq.Expressions.Expression<Func<PostEntity, int>> count, CancellationToken token) where TVote : Entity
{
var vote = await DB.Default.Find<TVote>().Match(mine).ExecuteFirstAsync(token);
if (vote == default || (await DB.Default.DeleteAsync<TVote>(vote.ID)).DeletedCount == 0)
return;
await DB.Default.Update<PostEntity>().MatchID(post.ID).Modify(b => b.Inc(count, -1)).ExecuteAsync(token);
if (typeof(TVote) == typeof(Favourite))
post.FavouritesCount--;
else
post.DownvotesCount--;
}
// where a vote on a remote post goes: the community it was made in, which counts the votes and passes them on, as
// Lemmy sends one; and its author when on another server (a Mastodon account's post in a Lemmy community). One copy a
// server: PieFed drops the second it is sent within a minute, though its first may yet fail.
async Task<List<string>> VoteInboxes(PostEntity post, CancellationToken token)
{
var inboxes = new List<string>();
if (await Federation.Inbox.Communities.Of(post, _dbEntities, token) is { } communityUri
&& await _dbEntities.ForeignAvatars.Match(f => f.ActorURI == communityUri && f.AvatarType == AvatarType.Group).ExecuteFirstAsync(token) is { } community
&& !string.IsNullOrEmpty(community.InboxURL))
inboxes.Add(community.InboxURL);
if (await AuthorInbox(post, token) is { } author
&& !inboxes.Any(i => string.Equals(new Uri(i).Authority, new Uri(author).Authority, StringComparison.OrdinalIgnoreCase)))
inboxes.Add(author);
return inboxes;
}
// how a remote post's interaction policy takes an interaction of ours: at once, once asked, or not at all
async Task<QuotePermission> Permission(PostEntity target, LocalActor actor, InteractionKind kind, CancellationToken token) =>
target is { IsFederatedCopy: true } && _approvals != default ? await _approvals.Judge(target, actor, kind, token) : QuotePermission.Granted;
@@ -684,7 +803,8 @@ namespace PrivaPub.Domain.Statuses
};
}
async Task<List<MediaAttachment>> Media(LocalActor author, IReadOnlyList<string> ids, string postId, CancellationToken token)
// the author's own uploads, not yet another post's (nor held by another scheduled post), nor trashed, nor pictures
async Task<List<MediaAttachment>> Media(LocalActor author, IReadOnlyList<string> ids, string postId, string scheduledId, CancellationToken token)
{
if (ids == default || ids.Count == 0)
return new List<MediaAttachment>();
@@ -692,23 +812,38 @@ namespace PrivaPub.Domain.Statuses
return default;
var wanted = ids.Distinct().ToList();
var found = await DB.Default.Find<MediaAttachment>()
.Match(m => wanted.Contains(m.ID) && m.OwnerAvatarId == author.Id && (m.PostId == null || m.PostId == postId))
.Match(m => wanted.Contains(m.ID) && m.OwnerAvatarId == author.Id && (m.PostId == null || m.PostId == postId)
&& (m.ScheduledStatusId == null || m.ScheduledStatusId == scheduledId) && m.TrashedAt == null && m.ProfileOfAvatarId == null
&& m.ProcessingState == null)
.ExecuteAsync(token);
return found.Count == wanted.Count ? wanted.Select(id => found.First(m => m.ID == id)).ToList() : default;
}
static async Task Attach(IEnumerable<MediaAttachment> media, string postId, CancellationToken token)
// attaches the media in one conditional update, true when all of them are the post's; when another post took one
// meanwhile, the ones this call took go back as they were
static async Task<bool> Claim(IReadOnlyList<MediaAttachment> media, string postId, string scheduledId, CancellationToken token)
{
var ids = media.Select(m => m.ID).ToList();
if (ids.Count > 0)
await DB.Default.Update<MediaAttachment>().Match(m => ids.Contains(m.ID))
if (ids.Count == 0)
return true;
var claimed = await DB.Default.Update<MediaAttachment>()
.Match(m => ids.Contains(m.ID) && (m.PostId == null || m.PostId == postId) && m.TrashedAt == null
&& (m.ScheduledStatusId == null || m.ScheduledStatusId == scheduledId))
.Modify(m => m.PostId, postId).Modify(m => m.AttachedAt, DateTime.UtcNow).ExecuteAsync(token);
if (claimed.MatchedCount == ids.Count)
return true;
var taken = media.Where(m => m.PostId == null).Select(m => m.ID).ToList();
await DB.Default.Update<MediaAttachment>().Match(m => taken.Contains(m.ID) && m.PostId == postId)
.Modify(m => m.PostId, null).Modify(m => m.AttachedAt, null).ExecuteAsync(token);
return false;
}
PostMedia ToPostMedia(MediaAttachment attachment) => new()
{
AttachmentId = attachment.ID,
ContentType = attachment.ContentType,
Kind = attachment.Kind,
DurationSeconds = attachment.DurationSeconds,
URL = _media.Url(attachment.FilePath),
PreviewURL = _media.Url(attachment.PreviewPath ?? attachment.FilePath),
Description = attachment.Description,
+3 -2
View File
@@ -40,13 +40,14 @@ namespace PrivaPub.Domain.Timelines
{
case PostVisibility.Public or PostVisibility.Unlisted or PostVisibility.FollowersOnly:
var followers = post.IsFederatedCopy
? await _dbEntities.Followings.Match(f => f.TargetActorURI == post.ActorURI && f.State == FollowState.Accepted).ExecuteAsync(token)
? await _dbEntities.Followings.Match(f => f.TargetActorURI == post.ActorURI && f.State == FollowState.Accepted && f.FollowerKind == null).ExecuteAsync(token)
: await _dbEntities.Followings.Match(f => f.TargetAccountId == post.GroupUserId && f.TargetIsLocal && f.State == FollowState.Accepted).ExecuteAsync(token);
foreach (var following in followers)
if (await Shows(following, post, token))
recipients.Add(following.AvatarId);
if (!string.IsNullOrEmpty(post.AudienceURI))
foreach (var member in await _dbEntities.Followings.Match(f => f.TargetActorURI == post.AudienceURI && f.State == FollowState.Accepted).ExecuteAsync(token))
// (the server's reader of feeds follows communities for nobody's home)
foreach (var member in await _dbEntities.Followings.Match(f => f.TargetActorURI == post.AudienceURI && f.State == FollowState.Accepted && f.FollowerKind == null).ExecuteAsync(token))
recipients.Add(member.AvatarId);
// a post on a wall (FEP-400e): its owner, when a persona, and those following the owner
if (!string.IsNullOrEmpty(post.WallOwnerURI))
+53 -15
View File
@@ -12,18 +12,25 @@ namespace PrivaPub.Federation.Actors
// A remote account's follower, following and post counts, as its own server publishes them (the totalItems of its
// followers, following and outbox collections), and the posts it features. Read when the account is fetched, at most
// once a day, never when someone looks at it; a hidden or missing collection leaves its count unknown. Mastodon and
// GoToSocial do the same.
// GoToSocial do the same. A feed's following is its communities: they are kept, and read in turn, so its list shows.
public class AccountCountsJob : IJobHandler
{
readonly IRemoteActorService _remoteActors;
readonly DbEntities _dbEntities;
readonly IFeaturedPosts _featured;
readonly ILocalActorService _localActors;
readonly IJobQueue _jobs;
public AccountCountsJob(IRemoteActorService remoteActors, DbEntities dbEntities, IFeaturedPosts featured = default)
public const int MaxFeedCommunities = 50;
public AccountCountsJob(IRemoteActorService remoteActors, DbEntities dbEntities, IFeaturedPosts featured = default,
ILocalActorService localActors = default, IJobQueue jobs = default)
{
_remoteActors = remoteActors;
_dbEntities = dbEntities;
_featured = featured;
_localActors = localActors;
_jobs = jobs;
}
public JobKind Kind => JobKind.CountAccount;
@@ -38,28 +45,59 @@ namespace PrivaPub.Federation.Actors
var actor = await _dbEntities.ForeignAvatars.Match(a => a.ActorURI == job.Payload && !a.DeletionAt.HasValue).ExecuteFirstAsync(token);
if (actor == default)
return JobOutcome.Done;
await DB.Default.Update<ForeignAvatar>().MatchID(actor.ID)
.Modify(a => a.FollowersCount, await Total(actor, actor.FollowersURL, token))
.Modify(a => a.FollowingCount, await Total(actor, actor.FollowingURL, token))
.Modify(a => a.StatusesCount, await Total(actor, actor.OutboxURL, token))
.Modify(a => a.CountedAt, DateTime.UtcNow)
.ExecuteAsync(token);
var feed = actor.AvatarType == AvatarType.Feed;
var (following, communities, listed) = await Read(actor, actor.FollowingURL, feed, token);
var update = DB.Default.Update<ForeignAvatar>().MatchID(actor.ID)
.Modify(a => a.FollowersCount, (await Read(actor, actor.FollowersURL, false, token)).Total)
.Modify(a => a.FollowingCount, following)
.Modify(a => a.StatusesCount, (await Read(actor, actor.OutboxURL, false, token)).Total)
.Modify(a => a.CountedAt, DateTime.UtcNow);
// (a feed whose list could not be read keeps the one it had: the reader of feeds would unfollow them all)
if (feed && listed)
update = update.Modify(a => a.FeedCommunities, communities);
await update.ExecuteAsync(token);
if (feed && listed)
{
foreach (var community in communities)
if (_localActors == default || !community.StartsWith(_localActors.BaseAddress + "/", StringComparison.Ordinal))
await _remoteActors.GetActor(community, refresh: false, token);
if (_jobs != default && await DB.Default.Find<Models.Social.FeedSubscription>().Match(s => s.FeedActorURI == actor.ActorURI).ExecuteAnyAsync(token))
await Domain.Social.FeedFollows.Sync(_jobs, token);
}
if (_featured != default)
await _featured.Sync(actor, token);
return JobOutcome.Done;
}
// only from the actor's own server, as everything we believe about it
async Task<int?> Total(ForeignAvatar actor, string collection, CancellationToken token)
// only from the actor's own server, as everything we believe about it: its total, and the ids it lists when asked
// (inline, or on its first page)
async Task<(int? Total, List<string> Items, bool Listed)> Read(ForeignAvatar actor, string collection, bool items, CancellationToken token)
{
var listed = new List<string>();
if (string.IsNullOrEmpty(collection) || !Origin.Same(collection, actor.ActorURI))
return default;
return (default, listed, false);
using var scope = HttpScope.For("collection");
using var fetched = await _remoteActors.FetchObject(collection, token);
return fetched != default && fetched.Root.ValueKind == System.Text.Json.JsonValueKind.Object
&& fetched.Root.TryGetProperty("totalItems", out var total) && total.TryGetInt32(out var count) && count >= 0
? count
: default;
if (fetched == default || fetched.Root.ValueKind != System.Text.Json.JsonValueKind.Object)
return (default, listed, false);
int? total = fetched.Root.TryGetProperty("totalItems", out var counted) && counted.TryGetInt32(out var count) && count >= 0 ? count : default;
if (!items)
return (total, listed, true);
var document = System.Text.Json.Nodes.JsonNode.Parse(fetched.Root.GetRawText());
var page = document["orderedItems"] ?? document["items"] ?? document["first"]?["orderedItems"] ?? document["first"]?["items"];
if (page == default && ActivityJson.Id(document["first"]) is { } first && Origin.Same(first, actor.ActorURI))
{
using var next = await _remoteActors.FetchObject(first, token);
if (next == default)
return (total, listed, false);
var nextPage = System.Text.Json.Nodes.JsonNode.Parse(next.Root.GetRawText());
page = nextPage?["orderedItems"] ?? nextPage?["items"];
}
if (page is System.Text.Json.Nodes.JsonArray array)
listed = array.Select(ActivityJson.Id)
.Where(id => Uri.TryCreate(id, UriKind.Absolute, out var uri) && uri.Scheme is "https" or "http")
.Distinct(StringComparer.Ordinal).Take(MaxFeedCommunities).ToList();
return (total, listed, true);
}
}
}
+10 -2
View File
@@ -11,7 +11,7 @@ namespace PrivaPub.Federation.Actors
public sealed class ActorDocument
{
static readonly string[] ActorTypes = { "Person", "Service", "Application", "Group", "Organization" };
static readonly string[] ActorTypes = { "Person", "Service", "Application", "Group", "Organization", "Feed" };
public string Id { get; init; }
public string Type { get; init; }
@@ -36,6 +36,9 @@ namespace PrivaPub.Federation.Actors
public string IconDescription { get; init; }
public string HeaderDescription { get; init; }
public string MovedTo { get; init; }
public string Moderators { get; init; }
public List<PostFlair> Flairs { get; init; } = new();
public List<string> SearchableBy { get; init; }
public List<string> AlsoKnownAs { get; init; } = new();//the accounts it says it also is (a Move's target names the moved one)
public DateTime? Published { get; init; }
public List<CustomEmoji> Emojis { get; init; } = new();
@@ -71,7 +74,8 @@ namespace PrivaPub.Federation.Actors
Features = ActorFeatures.Detect(root),
PreferredUsername = RemoteActorService.Text(root, "preferredUsername"),
Name = RemoteActorService.Text(root, "name"),
Summary = RemoteActorService.Text(root, "summary"),
// (Lemmy's feed keeps its text in `description`)
Summary = RemoteActorService.Text(root, "summary") ?? (type == "Feed" ? RemoteActorService.Text(root, "description") : default),
Url = RemoteActorService.Text(root, "url") ?? id,
Inbox = Origin.Same(inbox, id) ? inbox : default,
Outbox = RemoteActorService.Text(root, "outbox"),
@@ -92,6 +96,10 @@ namespace PrivaPub.Federation.Actors
IconDescription = root.TryGetProperty("icon", out var described) ? Alt(described) : default,
HeaderDescription = root.TryGetProperty("image", out var headerImage) ? Alt(headerImage) : default,
MovedTo = RemoteActorService.Text(root, "movedTo"),
Flairs = type != "Group" ? new() : ObjectShapes.Flairs(root.TryGetProperty("tag", out var labels) ? JsonNode.Parse(labels.GetRawText()) : default,
root.TryGetProperty("lemmy:tagsForPosts", out var older) ? JsonNode.Parse(older.GetRawText()) : default),
SearchableBy = root.TryGetProperty("searchableBy", out var searchable) ? ObjectShapes.SearchableBy(JsonNode.Parse(searchable.GetRawText())) : default,
Moderators = type == "Group" ? RemoteActorService.Text(root, "attributedTo") ?? RemoteActorService.Text(root, "moderators") : default,
AlsoKnownAs = root.TryGetProperty("alsoKnownAs", out var aliases) ? Uris(aliases) : new(),
Published = DateTimeOffset.TryParse(RemoteActorService.Text(root, "published"), CultureInfo.InvariantCulture, DateTimeStyles.AssumeUniversal, out var published)
? published.UtcDateTime
@@ -21,7 +21,8 @@ namespace PrivaPub.Federation.Actors
// FEP-8fcf, received: an account elsewhere tells, with its delivery, a digest of its followers here. When the personas
// following it here digest otherwise, its partial list (read signed by the instance actor) says who they are: a follow
// only PrivaPub remembers ends, a request it answered without our knowing is accepted, and a follow only it remembers
// is undone by the persona, as Mastodon does. A follow ends only when the list is the one the digest describes.
// is undone by the persona, as Mastodon does, or kept within RestoreRecord.FollowersGrace of a restore, which may have
// lost it. A follow ends only when the list is the one the digest describes.
public class FollowingSynchronization : IJobHandler
{
const int MaxPages = 10;
@@ -87,6 +88,19 @@ namespace PrivaPub.Federation.Actors
}
if (await _localActors.FindByUri(uri, token) is not { Kind: LocalActorKind.Person } stranger)
continue;
// a follow the restore lost: kept, not undone
if (await Infrastructure.Backup.RestoreRecord.InFollowersGrace(token))
{
await DB.Default.SaveAsync(new Following
{
AvatarId = stranger.Id,
TargetActorURI = actor.ActorURI,
TargetAccountId = actor.ID,
TargetInboxURL = string.IsNullOrEmpty(actor.InboxURL) ? actor.SharedInboxURL : actor.InboxURL,
State = FollowState.Accepted
}, token);
continue;
}
// (it has no id for a follow PrivaPub never made; it is undone by actor and object)
var undo = new JsonObject
{
@@ -51,7 +51,7 @@ namespace PrivaPub.Federation.Actors
public string Wall => $"{Uri}/graffiti";
public bool HasWall => Kind == LocalActorKind.Person && !IsCircle && IsFederated;
// the server's own actors (the instance actor, the reporter): nobody follows, mentions or looks them up as accounts
public bool IsServerActor => Kind is LocalActorKind.Application or LocalActorKind.Reporter;
public bool IsServerActor => Kind is LocalActorKind.Application or LocalActorKind.Reporter or LocalActorKind.Reader;
public string Flock => $"{Uri}/flock";
public string Wardens => $"{Uri}/wardens";
public string SharedInbox => $"{BaseAddress}/human-centipede";
@@ -74,6 +74,7 @@ namespace PrivaPub.Federation.Actors
Task<LocalActor> FindByAddress(string address, CancellationToken token);
Task<LocalActor> GetInstanceActor(CancellationToken token);
Task<LocalActor> GetReporterActor(CancellationToken token);
Task<LocalActor> GetReaderActor(CancellationToken token);
Task<bool> IsUserNameTaken(string userName, CancellationToken token);
Task<bool> TryReserveUserName(string userName, LocalActorKind kind, string ownerId, CancellationToken token);
LocalActor FromAvatar(Avatar avatar);
@@ -89,13 +90,17 @@ namespace PrivaPub.Federation.Actors
// the anonymous Service that carries this server's reports to Lemmy, which takes no report from an Application
// (owner decision 2026-10-06): one actor for the server, never one per persona
public const string ReporterUserName = "privapub_reports";
// the anonymous Service that follows the communities of the feeds personas read, so no community learns which persona
// reads it (owner decision 2026-10-07); a Service, since Lemmy takes a follow from no Application
public const string ReaderUserName = "privapub_feeds";
public static bool IsServerActorName(string userName) =>
string.Equals(userName, InstanceUserName, StringComparison.OrdinalIgnoreCase) || string.Equals(userName, ReporterUserName, StringComparison.OrdinalIgnoreCase);
string.Equals(userName, InstanceUserName, StringComparison.OrdinalIgnoreCase) || string.Equals(userName, ReporterUserName, StringComparison.OrdinalIgnoreCase)
|| string.Equals(userName, ReaderUserName, StringComparison.OrdinalIgnoreCase);
static readonly HashSet<string> ReservedByInstance = new(StringComparer.Ordinal)
{
InstanceUserName, ReporterUserName, "admin", "administrator", "root", "system", "support", "help", "moderator", "mod",
InstanceUserName, ReporterUserName, ReaderUserName, "admin", "administrator", "root", "system", "support", "help", "moderator", "mod",
"abuse", "postmaster", "webmaster", "hostmaster", "security", "noreply", "no_reply", "null", "undefined"
};
@@ -103,6 +108,7 @@ namespace PrivaPub.Federation.Actors
readonly IOptionsMonitor<AppConfiguration> _appConfiguration;
InstanceActor _instanceActor;
ReporterActor _reporterActor;
ReaderActor _readerActor;
public LocalActorService(DbEntities dbEntities, IOptionsMonitor<AppConfiguration> appConfiguration)
{
@@ -121,6 +127,8 @@ namespace PrivaPub.Federation.Actors
return await GetInstanceActor(token);
if (userName == ReporterUserName)
return await GetReporterActor(token);
if (userName == ReaderUserName)
return await GetReaderActor(token);
var avatar = await _dbEntities.Avatars
.Match(a => a.UserName == userName && !a.DeletionAt.HasValue)
@@ -158,6 +166,8 @@ namespace PrivaPub.Federation.Actors
return group == default ? default : FromGroup(group);
case LocalActorKind.Reporter:
return await GetReporterActor(token);
case LocalActorKind.Reader:
return await GetReaderActor(token);
default:
return await GetInstanceActor(token);
}
@@ -221,6 +231,36 @@ namespace PrivaPub.Federation.Actors
};
}
public async Task<LocalActor> GetReaderActor(CancellationToken token)
{
var reader = _readerActor ??= await LoadReaderActor(token);
return new LocalActor
{
Id = reader.ID,
Kind = LocalActorKind.Reader,
UserName = ReaderUserName,
Name = $"Feeds read on {new Uri(BaseAddress).Host}",
Summary = "It follows the communities in the feeds this PrivaPub server's people read; it never names who reads them.",
PrivateKeyPem = reader.PrivateKey,
PublicKeyPem = reader.PublicKey,
Discoverable = false,
Published = reader.CreationDate,
BaseAddress = BaseAddress
};
}
async Task<ReaderActor> LoadReaderActor(CancellationToken token)
{
var reader = await _dbEntities.ReaderActors.Sort(r => r.CreationDate, Order.Ascending).ExecuteFirstAsync(token);
if (reader != default)
return reader;
var (privateKey, publicKey) = Keys.NewKeyPair();
reader = new ReaderActor { PrivateKey = privateKey, PublicKey = publicKey };
await DB.Default.SaveAsync(reader, token);
return reader;
}
async Task<ReporterActor> LoadReporterActor(CancellationToken token)
{
var reporter = await _dbEntities.ReporterActors.Sort(r => r.CreationDate, Order.Ascending).ExecuteFirstAsync(token);
@@ -17,6 +17,8 @@ namespace PrivaPub.Federation.Actors
public interface IRemoteActorService
{
Task<FetchedJson> FetchObject(string uri, CancellationToken token);
// the same, or NotModified while the version `etag` names is still current
Task<FetchedJson> FetchObject(string uri, string etag, CancellationToken token) => FetchObject(uri, token);
Task<ForeignAvatar> GetActor(string actorUri, bool refresh, CancellationToken token);
Task<ForeignAvatar> GetActorByKeyId(string keyId, bool refresh, CancellationToken token);
bool KeyTemporarilyUnavailable(string keyId) => false;
@@ -50,13 +52,17 @@ namespace PrivaPub.Federation.Actors
_options = options;
}
public async Task<FetchedJson> FetchObject(string uri, CancellationToken token)
public Task<FetchedJson> FetchObject(string uri, CancellationToken token) => FetchObject(uri, default, token);
public async Task<FetchedJson> FetchObject(string uri, string etag, CancellationToken token)
{
using var scope = HttpScope.Default("object");
var signer = await _localActors.GetInstanceActor(token);
var fetched = await Get(uri, signer, token);
if (fetched == default)
return default;
var fetched = string.IsNullOrEmpty(etag)
? await Get(uri, signer, token)
: await _http.GetJsonIfChanged(uri, Accept, etag, request => HttpSignatures.Sign(request, signer, body: null), token);
if (fetched == default || fetched.NotModified)
return fetched;
var id = Text(fetched.Root, "id");
if (Origin.IsDocumentAt(id, fetched.FinalUri))
@@ -234,6 +240,9 @@ namespace PrivaPub.Federation.Actors
.Modify(a => a.IsLocked, actor.Locked)
.Modify(a => a.IsMemorial, actor.Memorial)
.Modify(a => a.MovedToURL, Origin.Of(actor.MovedTo) == default ? default : actor.MovedTo)
.Modify(a => a.ModeratorsURL, Origin.Same(actor.Moderators, actor.Id) ? actor.Moderators : default)
.Modify(a => a.Flairs, actor.Flairs)
.Modify(a => a.SearchableBy, actor.SearchableBy)
.Modify(a => a.AlsoKnownAs, actor.AlsoKnownAs)
.Modify(a => a.Published, actor.Published)
.Modify(a => a.Emojis, actor.Emojis)
@@ -532,8 +532,8 @@ namespace PrivaPub.Federation.Controllers
if (HttpMethods.IsGet(Request.Method))
Response.Headers.Vary = "Accept";
// SecureMode asks every reader of ActivityPub documents for a signature, except for the server's own actors (the
// instance actor, the reporter), whose keys peers need first, and except for browsers, which only get redirected to
// the public pages
// instance actor, the reporter, the reader of feeds), whose keys peers need first, and except for browsers, which
// only get redirected to the public pages
if (_federation.CurrentValue.SecureMode && HttpMethods.IsGet(Request.Method) && !WantsHtml() && Request.Path.Value != "/"
&& !LocalActorService.IsServerActorName(context.RouteData.Values["actor"] as string)
&& await _fetches.Requester(Request, HttpContext.RequestAborted) == default)
+1
View File
@@ -67,6 +67,7 @@ namespace PrivaPub.Federation.Inbox
LocalActorKind.Group when !local.IsCircle => "group",
LocalActorKind.Application => "application",
LocalActorKind.Reporter => "reporter",
LocalActorKind.Reader => "reader",
_ => default
};
}
+25
View File
@@ -1,7 +1,14 @@
using MongoDB.Entities;
using PrivaPub.Federation.Actors;
using PrivaPub.Federation.Objects;
using PrivaPub.Models.User;
using PrivaPub.StaticServices;
using System.Text.Json.Nodes;
using static PrivaPub.Federation.Objects.ActivityJson;
using PostEntity = PrivaPub.Models.Post.Post;
namespace PrivaPub.Federation.Inbox
@@ -20,5 +27,23 @@ namespace PrivaPub.Federation.Inbox
}
return default;
}
// whether an account may act for a community's moderators: anyone on the community's own server (its admins and
// moderators, as Lemmy's verify_mod_or_admin_action allows), or an account its moderators collection lists
public static async Task<bool> Moderates(ForeignAvatar community, string actorUri, IRemoteActorService remoteActors, CancellationToken token)
{
if (actorUri == default || community == default)
return false;
if (Origin.Same(actorUri, community.ActorURI))
return true;
if (string.IsNullOrEmpty(community.ModeratorsURL))
return false;
using var fetched = await remoteActors.FetchObject(community.ModeratorsURL, token);
if (fetched == default)
return false;
var collection = JsonNode.Parse(fetched.Root.GetRawText());
var items = collection?["orderedItems"] ?? collection?["items"] ?? collection?["first"]?["orderedItems"] ?? collection?["first"]?["items"];
return items is JsonArray listed && listed.Any(item => Id(item) == actorUri);
}
}
}
@@ -59,7 +59,7 @@ namespace PrivaPub.Federation.Inbox.Handlers
public async Task Handle(JsonNode activity, ForeignAvatar actor, CancellationToken token)
{
var inner = activity["object"];
if (inner is JsonObject && Value(inner, "type") is "Create" or "Update" or "Delete" or "Like" or "Dislike" or "Undo" or "Add" or "Remove" or "Block" or "Lock")
if (inner is JsonObject && Value(inner, "type") is "Create" or "Update" or "Delete" or "Like" or "Dislike" or "Undo" or "Add" or "Remove" or "Block" or "Lock" or "Move")
{
await GroupActivity(inner, actor, token);
return;
@@ -242,6 +242,9 @@ namespace PrivaPub.Federation.Inbox.Handlers
case "Lock" when objectUri != default:
await Lock(objectUri, group, true, token);
break;
case "Move" when objectUri != default && Id(inner["origin"]) == group.ActorURI:
await Move(objectUri, Id(inner["target"]), group, token);
break;
case "Undo" when Value(inner["object"], "type") == "Lock":
await Lock(Id(inner["object"]?["object"]), group, false, token);
break;
@@ -305,6 +308,42 @@ namespace PrivaPub.Federation.Inbox.Handlers
Arrival.Accept(locked ? "locked" : "unlocked");
}
// A post moved to another community (PieFed's Move{object: post, origin, target}), relayed by the community it leaves,
// which vouches for what is done to its own posts as for a lock; its thread goes with it. A move into a community we
// host is not taken: it would be ours to announce.
async Task Move(string objectUri, string targetUri, ForeignAvatar group, CancellationToken token)
{
var post = await _dbEntities.Posts.Match(p => p.ObjectURI == objectUri && p.AudienceURI == group.ActorURI && !p.DeletedAt.HasValue)
.ExecuteFirstAsync(token);
if (post == default)
{
Arrival.Drop("unknown-object");
return;
}
Arrival.About(visibility: post.Visibility, created: post.CreationDate);
var target = targetUri == default || targetUri == group.ActorURI || await _localActors.FindByUri(targetUri, token) != default
? default
: await _remoteActors.GetActor(targetUri, refresh: false, token);
if (target is not { AvatarType: AvatarType.Group })
{
Arrival.Drop("unknown-target");
return;
}
var thread = new List<string> { post.ID };
var level = thread.ToList();
while (level.Count > 0 && thread.Count < MaxMoved)
{
var parents = level;
level = (await _dbEntities.Posts.Match(p => parents.Contains(p.AnsweringToPostId) && p.AudienceURI == group.ActorURI)
.Project(p => new PostEntity { ID = p.ID }).ExecuteAsync(token)).Select(p => p.ID).ToList();
thread.AddRange(level);
}
await DB.Default.Update<PostEntity>().Match(p => thread.Contains(p.ID)).Modify(p => p.AudienceURI, target.ActorURI).ExecuteAsync(token);
Arrival.Accept("moved");
}
const int MaxMoved = 1000;
// A community bans one of our personas (Lemmy's Block with the community as its target): kept as the community
// blocking the persona, so its relationship says blocked_by and nothing of the persona's is posted there until the
// Undo. A ban of someone from another server is no business of ours.
@@ -0,0 +1,66 @@
using MongoDB.Entities;
using PrivaPub.Federation.Actors;
using PrivaPub.Federation.Objects;
using PrivaPub.Models.Social;
using PrivaPub.Models.User;
using PrivaPub.StaticServices;
using System.Text.Json.Nodes;
using static PrivaPub.Federation.Objects.ActivityJson;
namespace PrivaPub.Federation.Inbox.Handlers
{
// Lemmy 1.0 tells a report's sender that it was dealt with: Resolve{Flag}, the Flag being one we sent
// (`/grunts/flag-<report id>`). It is believed from the server holding what was reported, or the community it was
// reported to, and kept on the report beside our own moderators' resolution, which it never replaces.
public class ResolveHandler : IActivityHandler
{
const string FlagMarker = "/grunts/flag-";
readonly DbEntities _dbEntities;
readonly ILocalActorService _localActors;
public ResolveHandler(DbEntities dbEntities, ILocalActorService localActors)
{
_dbEntities = dbEntities;
_localActors = localActors;
}
public string Type => "Resolve";
public async Task Handle(JsonNode activity, ForeignAvatar actor, CancellationToken token)
{
var flag = activity["object"];
var flagId = Id(flag);
var marker = flagId?.IndexOf(FlagMarker, StringComparison.Ordinal) ?? -1;
if (flag is JsonObject && Value(flag, "type") != "Flag" || marker < 0
|| !flagId.StartsWith(_localActors.BaseAddress + "/", StringComparison.OrdinalIgnoreCase))
{
Arrival.Drop("not-ours");
return;
}
var reportId = flagId[(marker + FlagMarker.Length)..].Split('-')[0];
var report = await DB.Default.Find<Report>().MatchID(reportId).ExecuteFirstAsync(token);
if (report == default)
{
Arrival.Drop("unknown-object");
return;
}
var concerned = new List<string>(report.ObjectURIs) { report.TargetActorURI };
foreach (var post in report.PostIds.Count == 0 ? new List<Models.Post.Post>() : await _dbEntities.Posts.Match(p => report.PostIds.Contains(p.ID)).ExecuteAsync(token))
concerned.Add(await Communities.Of(post, _dbEntities, token));
if (!concerned.Any(uri => Origin.Same(uri, actor.ActorURI)))
{
Arrival.Drop("misattributed");
return;
}
await DB.Default.Update<Report>().MatchID(report.ID)
.Modify(r => r.RemoteResolvedAt, DateTime.UtcNow)
.Modify(r => r.RemoteResolvedBy, actor.ActorURI)
.ExecuteAsync(token);
Arrival.Accept("resolved");
}
}
}
@@ -0,0 +1,86 @@
using MongoDB.Entities;
using PrivaPub.Domain.Social;
using PrivaPub.Federation.Actors;
using PrivaPub.Models.Federation;
using PrivaPub.Models.Social;
using PrivaPub.Models.User;
using PrivaPub.StaticServices;
using System.Text.Json.Nodes;
using static PrivaPub.Federation.Objects.ActivityJson;
namespace PrivaPub.Federation.Inbox.Handlers
{
// Lemmy 1.0's warning: a community's moderator tells one of our personas, about one of its posts there, why (Warn{to:
// [persona], object: post, summary: reason, audience: community}), sent to the persona's inbox alone. It becomes the
// persona's moderation_warning notification, once the sender is shown to moderate that community and the post is the
// persona's own, in it.
public class WarnHandler : IActivityHandler
{
const int MaxReason = 1000;
readonly DbEntities _dbEntities;
readonly ILocalActorService _localActors;
readonly IRemoteActorService _remoteActors;
public WarnHandler(DbEntities dbEntities, ILocalActorService localActors, IRemoteActorService remoteActors)
{
_dbEntities = dbEntities;
_localActors = localActors;
_remoteActors = remoteActors;
}
public string Type => "Warn";
public async Task Handle(JsonNode activity, ForeignAvatar actor, CancellationToken token)
{
LocalActor persona = default;
foreach (var uri in Recipients(activity["to"]))
if (await _localActors.FindByUri(uri, token) is { Kind: LocalActorKind.Person } found)
{
persona = found;
break;
}
if (persona == default)
{
Arrival.Drop("not-ours");
return;
}
var communityUri = Id(activity["audience"]);
var community = communityUri == default ? default : await _remoteActors.GetActor(communityUri, refresh: false, token);
if (community is not { AvatarType: AvatarType.Group })
{
Arrival.Drop("unknown-community");
return;
}
var objectUri = Id(activity["object"]);
var post = objectUri == default
? default
: await _dbEntities.Posts.Match(p => p.ObjectURI == objectUri && p.GroupUserId == persona.Id && !p.IsFederatedCopy).ExecuteFirstAsync(token);
if (post == default || await Communities.Of(post, _dbEntities, token) != community.ActorURI)
{
Arrival.Drop("unknown-object");
return;
}
Arrival.About(visibility: post.Visibility, created: post.CreationDate, local: persona);
if (!await Communities.Moderates(community, actor.ActorURI, _remoteActors, token))
{
Arrival.Drop("not-moderator");
return;
}
var reason = Value(activity, "summary") ?? Value(activity, "content");
await Notifications.Add(persona.Id, NotificationType.ModerationWarning, community.ID, community.ActorURI, post.ID, token,
text: string.IsNullOrWhiteSpace(reason) ? default : reason.Trim().Length <= MaxReason ? reason.Trim() : reason.Trim()[..MaxReason]);
Arrival.Accept("warned");
}
static IEnumerable<string> Recipients(JsonNode node) => node switch
{
JsonArray array => array.Select(Id).Where(id => id != default),
JsonNode single when Id(single) is { } id => new[] { id },
_ => Enumerable.Empty<string>()
};
}
}
+2
View File
@@ -38,6 +38,8 @@ namespace PrivaPub.Federation.Inbox
post.Audio = note.Audio ?? post.Audio;
post.Event = note.Event ?? post.Event;
post.Place = note.Place;
post.Flairs = await RemotePosts.Flairs(note, post.AudienceURI, token);
post.SearchableBy = note.SearchableBy;
if (note.CommentsEnabled is { } enabled)
post.LockedAt = enabled ? null : post.LockedAt ?? DateTime.UtcNow;
if (!IsEdit(note, post))
+11
View File
@@ -106,6 +106,7 @@ namespace PrivaPub.Federation.Inbox
Excerpt = note.Excerpt,
Source = note.Source,
Poll = note.Poll,
SearchableBy = note.SearchableBy,
QuotePolicy = note.QuotePolicy,
ReplyPolicy = note.ReplyPolicy,
LikePolicy = note.LikePolicy,
@@ -137,10 +138,20 @@ namespace PrivaPub.Federation.Inbox
EditedAt = note.Updated,
Revisions = note.FormerRepresentations.ToList()
};
post.Flairs = await Flairs(note, post.AudienceURI, token);
await Episode(post, note, author, token);
return post;
}
// a post's flairs, filled in from its community's list
public static async Task<List<PostFlair>> Flairs(NoteDocument note, string communityUri, CancellationToken token)
{
if (note.Flairs.Count == 0)
return default;
var community = communityUri == default ? default : await DB.Default.Find<ForeignAvatar>().Match(f => f.ActorURI == communityUri).ExecuteFirstAsync(token);
return ObjectShapes.Described(note.Flairs, community?.Flairs);
}
static readonly System.Text.RegularExpressions.Regex Href = new(@"href=""([^""]+)""", System.Text.RegularExpressions.RegexOptions.Compiled);
// a post that is only a link to a podcast episode of its own author (Castopod announces each episode so): the
+74 -20
View File
@@ -20,9 +20,11 @@ namespace PrivaPub.Federation.Inbox
public sealed record RepliesPayload(string PostId, int Depth);
// What was said under a remote post on other servers, read when a persona opens its thread, so the thread shows more
// than what happened to reach us. The thread's own collection comes first (FEP-7888's `context`, which Mastodon 4.5+
// serves with every reply at any depth); otherwise the post's `replies` (PeerTube's `comments`), and the replies'
// own, two levels down. Read with the instance actor's signature, never a persona's, a few pages and at most
// than what happened to reach us. The thread's own collections come first: its conversation container (FEP-171b's
// `contextHistory`, every activity its owner added to the thread), then its posts (FEP-7888's `context`, which
// Mastodon 4.5+, Mitra and NodeBB serve with every reply at any depth), sent the ETag of the last whole read so an
// unchanged thread costs one 304; otherwise the post's `replies` (PeerTube's `comments`, Forte's whole thread), and the
// replies' own, two levels down. Read with the instance actor's signature, never a persona's, a few pages and at most
// MaxItems posts a job; each one is fetched from its own server and stored like any post fetched for a thread
// (RemotePosts.StoreContext: public and unlisted only, its author checked against its origin).
public class RepliesJobHandler : IJobHandler
@@ -74,14 +76,35 @@ namespace PrivaPub.Federation.Inbox
if (await Document(post, token) is not { } document)
return JobOutcome.Done;
if (await Collection(document["context"], token) is { } thread)
foreach (var name in new[] { "contextHistory", "context" })
{
await StoreAll(thread, token);
var uri = Id(document[name]);
// Forte's `context` is the thread's first post
if (uri != default && await _dbEntities.Posts.Match(p => p.ObjectURI == uri).ExecuteAnyAsync(token))
continue;
var known = uri == default ? default : await DB.Default.Find<ThreadCollection>().Match(c => c.URI == uri).ExecuteFirstAsync(token);
var (thread, etag, unchanged) = await Collection(document[name], known?.ETag, token);
if (unchanged)
{
await DB.Default.Update<ThreadCollection>().MatchID(known.ID).Modify(c => c.ReadAt, DateTime.UtcNow).ExecuteAsync(token);
return JobOutcome.Done;
}
if (await Collection(document["replies"] ?? document["comments"], token) is not { } replies)
if (thread == default)
continue;
var (_, read) = await StoreAll(thread, name == "contextHistory", token);
var whole = read && Counted(thread);
if (uri != default && (known != default || whole && etag != default))
await DB.Default.Update<ThreadCollection>()
.Match(c => c.URI == uri)
.Modify(c => c.ETag, whole ? etag : default)
.Modify(c => c.ReadAt, DateTime.UtcNow)
.Option(o => o.IsUpsert = true)
.ExecuteAsync(token);
return JobOutcome.Done;
var answers = await StoreAll(replies, token);
}
if ((await Collection(document["replies"] ?? document["comments"], default, token)).Collection is not { } replies)
return JobOutcome.Done;
var (answers, _) = await StoreAll(replies, false, token);
if (payload.Depth < MaxDepth)
await _queue.EnqueueMany(answers.Where(a => a.IsFederatedCopy && a.AnsweringToPostId == post.ID).Take(MaxBranches)
.Select(a => For(a, payload.Depth + 1)), token);
@@ -100,35 +123,41 @@ namespace PrivaPub.Federation.Inbox
return fetched == default ? default : JsonNode.Parse(fetched.Root.GetRawText()) as JsonObject;
}
// a collection as its server serves it now: what a post carries is read again by its id, since an embedded first
// page is as old as the post; a value that is no web address (Pleroma's `context` is a tag: URI), or no collection,
// gives nothing
async Task<JsonObject> Collection(JsonNode node, CancellationToken token)
// a collection as its server serves it now, and its ETag: what a post carries is read again by its id, since an
// embedded first page is as old as the post; a value that is no web address (Pleroma's `context` is a tag: URI), or no
// collection, gives nothing, and one whose server still has the version `etag` names is unchanged
async Task<(JsonObject Collection, string ETag, bool Unchanged)> Collection(JsonNode node, string etag, CancellationToken token)
{
var uri = Id(node);
string served = default;
if (uri != default && Uri.TryCreate(uri, UriKind.Absolute, out var parsed) && (parsed.Scheme == Uri.UriSchemeHttps || parsed.Scheme == Uri.UriSchemeHttp))
{
using var scope = HttpScope.For("replies");
using var fetched = await _remoteActors.FetchObject(uri, token);
using var fetched = await _remoteActors.FetchObject(uri, etag, token);
if (fetched?.NotModified == true)
return (default, etag, true);
node = fetched == default ? default : JsonNode.Parse(fetched.Root.GetRawText());
served = fetched?.ETag;
}
return node is JsonObject collection && Value(collection, "type") is { } type && CollectionTypes.Contains(type) ? collection : default;
return (node is JsonObject collection && Value(collection, "type") is { } type && CollectionTypes.Contains(type) ? collection : default, served, false);
}
// every post the collection lists, stored when we lack it: its pages in order, at most MaxPages and MaxItems
async Task<List<PostEntity>> StoreAll(JsonObject collection, CancellationToken token)
// every post the collection lists, stored when we lack it: its pages in order, at most MaxPages and MaxItems; whether
// it was read to its end. A conversation container's items are its owner's Adds of the thread's activities.
async Task<(List<PostEntity> Posts, bool Whole)> StoreAll(JsonObject collection, bool activities, CancellationToken token)
{
var posts = new List<PostEntity>();
var seen = 0;
var page = collection["orderedItems"] is JsonArray || collection["items"] is JsonArray ? collection : await Page(collection["first"], token);
for (var pages = 0; page != default && pages < MaxPages && seen < MaxItems; pages++)
for (var pages = 0; page != default; pages++)
{
if (pages == MaxPages)
return (posts, false);
foreach (var item in Items(page))
{
if (++seen > MaxItems)
break;
// a context may list the activities that made the thread (FEP-f228) rather than its posts
var id = Value(item, "type") is "Create" or "Update" ? Id(item["object"]) : Id(item);
return (posts, false);
var id = activities ? await Added(item, token) : PostId(item);
if (id == default)
continue;
var post = await _dbEntities.Posts.Match(p => p.ObjectURI == id && !p.DeletedAt.HasValue).ExecuteFirstAsync(token)
@@ -138,7 +167,32 @@ namespace PrivaPub.Federation.Inbox
}
page = page["next"] is { } next ? await Page(next, token) : default;
}
return posts;
return (posts, true);
}
// whether the document itself changes with every post added to the thread, so its ETag is the thread's: it counts
// them (NodeBB), or holds every one with no further page (a short Mastodon thread); a long Mastodon thread's first
// page stays as it was while its last one grows
static bool Counted(JsonObject collection)
{
if (collection["totalItems"] is JsonValue)
return true;
var page = collection["orderedItems"] is JsonArray || collection["items"] is JsonArray ? collection : collection["first"] as JsonObject;
return page != default && (page["orderedItems"] is JsonArray || page["items"] is JsonArray) && page["next"] == default;
}
// a context may list the activities that made the thread (FEP-f228) rather than its posts
static string PostId(JsonNode item) => Value(item, "type") is "Create" or "Update" ? Id(item["object"]) : Id(item);
// the post an activity in a conversation container brought into the thread, added by its owner or (Forte's) not: a
// Create's or an Update's object; a like, a removal or anything else adds no post. Only its id is taken, and the post
// is read from its own server.
async Task<string> Added(JsonNode item, CancellationToken token)
{
var activity = await Page(item, token);
if (activity != default && Value(activity, "type") == "Add")
activity = await Page(activity["object"], token);
return activity != default && Value(activity, "type") is "Create" or "Update" ? Id(activity["object"]) : default;
}
async Task<JsonObject> Page(JsonNode node, CancellationToken token)
@@ -40,6 +40,8 @@ namespace PrivaPub.Federation.Objects
public IReadOnlyList<PostMedia> Attachments { get; init; } = Array.Empty<PostMedia>();
public PostSource Source { get; init; }
public PostPoll Poll { get; init; }
public List<PostFlair> Flairs { get; init; } = new();
public List<string> SearchableBy { get; init; }
public InteractionRule QuotePolicy { get; init; }
public InteractionRule ReplyPolicy { get; init; }
public InteractionRule LikePolicy { get; init; }
@@ -119,6 +121,8 @@ namespace PrivaPub.Federation.Objects
Attachments = Attachments(note),
Source = ObjectShapes.Source(note),
Poll = ObjectShapes.Poll(note),
Flairs = ObjectShapes.Flairs(note["tag"]),
SearchableBy = ObjectShapes.SearchableBy(note["searchableBy"]),
QuotePolicy = ObjectShapes.QuotePolicy(note),
ReplyPolicy = ObjectShapes.Policy(note, "canReply"),
LikePolicy = ObjectShapes.Policy(note, "canLike"),
@@ -98,6 +98,60 @@ namespace PrivaPub.Federation.Objects
.Take(MaxEmojis)
.ToList();
const int MaxFlairs = 20;
// FEP-268d: who may find this in a search (Public, the author's followers, nobody else); null when not stated
public static List<string> SearchableBy(JsonNode node) => node switch
{
null => default,
JsonArray array => array.Select(Id).Where(id => id != default).Take(MaxAudience).ToList(),
_ when Id(node) is { } single => new List<string> { single },
_ => new List<string>()
};
public static bool Searchable(List<string> searchableBy, string followers, bool viewerFollows) =>
searchableBy.Any(Addressing.IsPublic) || viewerFollows && searchableBy.Any(w => w == followers || w.EndsWith("/followers", StringComparison.Ordinal));
// a post's flairs, or a community's (its `tag`, and PieFed's older `lemmy:tagsForPosts`), in both dialects
public static List<PostFlair> Flairs(params JsonNode[] lists) =>
lists.SelectMany(Objects)
.Where(t => Value(t, "type") is "CommunityPostTag" or "lemmy:CommunityTag" or "CommunityTag")
.Select(t => new PostFlair
{
Id = Id(t) is { } id && Origin.Of(id) != default ? id : default,
Name = Text(Value(t, "name") ?? Value(t, "display_name") ?? Value(t, "preferredUsername"), 100),
Slug = Text(Value(t, "preferredUsername") ?? Value(t, "display_name"), 100),
Description = Text(Value(t, "content"), 500),
Color = Value(t, "color") is { } slot && FlairSlot().IsMatch(slot) ? slot : default,
TextColor = Hex(Value(t, "textColor") ?? Value(t, "text_color")),
BackgroundColor = Hex(Value(t, "backgroundColor") ?? Value(t, "background_color")),
BlurImages = Bool(t, "blurImages") ?? Bool(t, "blur_images") ?? false
})
.Where(f => f.Name != default)
.DistinctBy(f => f.Id ?? f.Slug)
.Take(MaxFlairs)
.ToList();
// a post's flairs as its community describes them: Lemmy names only the id and slug on the post
public static List<PostFlair> Described(List<PostFlair> onPost, List<PostFlair> ofCommunity)
{
if (onPost == default || onPost.Count == 0)
return default;
foreach (var flair in onPost)
if (ofCommunity?.FirstOrDefault(c => c.Id != default && c.Id == flair.Id || c.Slug == flair.Slug) is { } known)
{
flair.Name = flair.Name == flair.Slug ? known.Name ?? flair.Name : flair.Name;
flair.Description ??= known.Description;
flair.Color ??= known.Color;
flair.TextColor ??= known.TextColor;
flair.BackgroundColor ??= known.BackgroundColor;
flair.BlurImages |= known.BlurImages;
}
return onPost;
}
static string Hex(string value) => value != default && HexColour().IsMatch(value) ? value : default;
public static InteractionRule QuotePolicy(JsonObject note) => Policy(note, "canQuote");
// one rule of a post's interactionPolicy (GoToSocial, FEP-044f): who may, automatically or once asked; the old names
@@ -399,6 +453,12 @@ namespace PrivaPub.Federation.Objects
[GeneratedRegex(@"^[\w+\-.@]+$")]
private static partial Regex Shortcode();
[GeneratedRegex(@"^color\d{2}$")]
private static partial Regex FlairSlot();
[GeneratedRegex(@"^#[0-9a-fA-F]{3,8}$")]
private static partial Regex HexColour();
[GeneratedRegex("<[^>]*>")]
private static partial Regex Tags();
@@ -182,16 +182,18 @@ namespace PrivaPub.Federation.Outbox
{ Kind: LocalActorKind.Person } => "person",
{ Kind: LocalActorKind.Group } => "group",
{ Kind: LocalActorKind.Reporter } => "reporter",
{ Kind: LocalActorKind.Reader } => "reader",
_ => "application"
},
Signature = "cavage:rsa-sha256"
});
}
// FEP-8fcf: what a delivery to the persona's followers tells the receiving server of its followers there
// FEP-8fcf: what a delivery to the persona's followers tells the receiving server of its followers there; nothing for
// a while after a restore, whose losses are not the other server's to mend
static async Task<(string, string)[]> Synchronization(LocalActor signer, string body, Uri inbox, CancellationToken token)
{
if (!FollowersSynchronization.Synchronizes(signer) || JsonNode.Parse(body) is not JsonObject activity || !FollowersSynchronization.ForFollowers(activity, signer))
if (!FollowersSynchronization.Synchronizes(signer) || await Infrastructure.Backup.RestoreRecord.InFollowersGrace(token) || JsonNode.Parse(body) is not JsonObject activity || !FollowersSynchronization.ForFollowers(activity, signer))
return [];
var there = await FollowersSynchronization.On(signer, FollowersSynchronization.Origin(inbox.AbsoluteUri), token);
return [(FollowersSynchronization.Header, FollowersSynchronization.HeaderValue(signer, FollowersSynchronization.Digest(there)))];
@@ -4,6 +4,7 @@ using PrivaPub.Federation.Actors;
using PrivaPub.Federation.Objects;
using PrivaPub.Models.Federation;
using PrivaPub.Models.Post;
using PrivaPub.Domain.Media;
using System.Globalization;
using System.Net;
@@ -125,7 +126,7 @@ namespace PrivaPub.Federation.Rendering
{
LocalActorKind.Group => "Group",
LocalActorKind.Application => "Application",
LocalActorKind.Reporter => "Service",
LocalActorKind.Reporter or LocalActorKind.Reader => "Service",
_ when actor.IsBot => "Service",
_ => "Person"
},
@@ -386,7 +387,7 @@ namespace PrivaPub.Federation.Rendering
};
if (!string.IsNullOrEmpty(m.Blurhash))
attachment["blurhash"] = m.Blurhash;
if (m.Focus is { Length: 2 })
if (FocalPoint.IsSound(m.Focus))
attachment["focalPoint"] = new JsonArray(m.Focus[0], m.Focus[1]);
if (m.Width.HasValue && m.Height.HasValue)
{
@@ -0,0 +1,65 @@
using System.Text.Json;
using System.Text.Json.Serialization;
namespace PrivaPub.Infrastructure.Backup
{
// What a backup holds (manifest.json at its root): enough to check it is whole, to restore it on this host only, and to
// rebuild what Mongo had (each collection's indexes).
public sealed class ArchiveManifest
{
public const int CurrentFormat = 1;
public const string FileName = "manifest.json";
public int Format { get; set; } = CurrentFormat;
public string Kind { get; set; }//nightly, manual, pre-deploy, pre-restore, uploaded
public DateTime CreatedAt { get; set; } = DateTime.UtcNow;
public string Host { get; set; }//BackendBaseAddress: URIs and media URLs are stored whole, so only this host can restore it
public string AppCommit { get; set; }
public string AppRef { get; set; }
public string NewestMigration { get; set; }
public int MigrationNumber { get; set; }
public bool Consistent { get; set; }//read at one instant (a snapshot session on a replica set)
public bool DbOnly { get; set; }//no media files, only their list (the deploy's, which can't link www-data's files)
public List<CollectionEntry> Collections { get; set; } = [];
public List<ExcludedEntry> Excluded { get; set; } = [];
public MediaEntry Media { get; set; } = new();
public sealed class CollectionEntry
{
public string Name { get; set; }
public long Count { get; set; }
public long Bytes { get; set; }
public string Sha256 { get; set; }
public List<string> Indexes { get; set; } = [];//each as canonical Extended JSON
}
public sealed class ExcludedEntry
{
public string Name { get; set; }
public string Why { get; set; }
}
public sealed class MediaEntry
{
public int Files { get; set; }
public long Bytes { get; set; }
public int Missing { get; set; }
public List<string> List { get; set; } = [];
}
static readonly JsonSerializerOptions Json = new() { WriteIndented = true, PropertyNamingPolicy = JsonNamingPolicy.CamelCase, DefaultIgnoreCondition = JsonIgnoreCondition.Never };
public static ArchiveManifest Read(string directory)
{
var path = Path.Combine(directory, FileName);
return File.Exists(path) ? JsonSerializer.Deserialize<ArchiveManifest>(File.ReadAllText(path), Json) : default;
}
public void Write(string directory)
{
using var file = new FileStream(Path.Combine(directory, FileName), FileMode.Create, FileAccess.Write);
JsonSerializer.Serialize(file, this, Json);
file.Flush(flushToDisk: true);
}
}
}
@@ -0,0 +1,13 @@
namespace PrivaPub.Infrastructure.Backup
{
public class BackupOptions
{
public string Root { get; set; }//where backups are kept (/var/lib/privapub/backups); <media root>-backups by default
public string NightlyAt { get; set; } = "03:30";//UTC
public int KeepDaily { get; set; } = 7;
public int KeepWeekly { get; set; } = 4;
public int KeepPreDeploy { get; set; } = 3;
public int KeepPreRestore { get; set; } = 3;
public bool Nightly { get; set; } = true;
}
}
+224
View File
@@ -0,0 +1,224 @@
using System.Formats.Tar;
using System.Text;
using System.Text.RegularExpressions;
namespace PrivaPub.Infrastructure.Backup
{
// A backup as one tar file, for the administrator to download and to upload again: every file under <id>/, its length
// known before the first byte (so a download has a Content-Length and resumes with Range), written straight from the
// backup's files, never held in memory.
public sealed partial class BackupTar
{
const int Block = 512;
readonly List<Entry> _entries = [];
public long Length { get; private set; }
sealed record Entry(string Name, string Path, long Size, DateTime ModifiedAt, long Offset);
public static BackupTar Of(string backupsRoot, string id)
{
var directory = Path.Combine(backupsRoot, id);
var tar = new BackupTar();
var offset = 0L;
foreach (var file in Directory.EnumerateFiles(directory, "*", SearchOption.AllDirectories).OrderBy(f => f, StringComparer.Ordinal))
{
var info = new FileInfo(file);
var name = id + "/" + Path.GetRelativePath(directory, file).Replace('\\', '/');
tar._entries.Add(new Entry(name, file, info.Length, info.LastWriteTimeUtc, offset));
offset += Block + Padded(info.Length);
}
tar.Length = offset + 2 * Block;
return tar;
}
static long Padded(long size) => (size + Block - 1) / Block * Block;
/// <summary>Writes bytes from to to (inclusive) of the tar.</summary>
public async Task Write(Stream output, long from, long to, CancellationToken token)
{
var buffer = new byte[81920];
foreach (var entry in _entries)
{
var headerEnd = entry.Offset + Block;
var dataEnd = headerEnd + entry.Size;
var entryEnd = headerEnd + Padded(entry.Size);
if (entryEnd <= from)
continue;
if (entry.Offset > to)
return;
if (from < headerEnd)
{
var header = Header(entry);
var start = (int)Math.Max(0, from - entry.Offset);
var end = (int)Math.Min(Block, to - entry.Offset + 1);
await output.WriteAsync(header.AsMemory(start, end - start), token);
}
if (from < dataEnd && to >= headerEnd && entry.Size > 0)
{
var start = Math.Max(0, from - headerEnd);
var end = Math.Min(entry.Size, to - headerEnd + 1);
await using var file = new FileStream(entry.Path, FileMode.Open, FileAccess.Read, FileShare.Read, 1, FileOptions.SequentialScan);
file.Seek(start, SeekOrigin.Begin);
var left = end - start;
while (left > 0)
{
var read = await file.ReadAsync(buffer.AsMemory(0, (int)Math.Min(buffer.Length, left)), token);
if (read == 0)
throw new IOException($"{entry.Name} shrank while it was sent");
await output.WriteAsync(buffer.AsMemory(0, read), token);
left -= read;
}
}
if (to >= dataEnd && from < entryEnd)
{
var start = Math.Max(from, dataEnd);
var end = Math.Min(entryEnd - 1, to);
if (end >= start)
await output.WriteAsync(new byte[end - start + 1], token);
}
}
var trailer = Length - 2 * Block;
if (to >= trailer)
await output.WriteAsync(new byte[to - Math.Max(from, trailer) + 1], token);
}
// a ustar header (name split into prefix and name past 100 bytes; sizes past 8 GiB in base 256)
static byte[] Header(Entry entry)
{
var header = new byte[Block];
var name = entry.Name;
var prefix = string.Empty;
if (Encoding.UTF8.GetByteCount(name) > 100)
{
var cut = name.LastIndexOf('/', Math.Min(name.Length - 1, 155));
while (cut > 0 && Encoding.UTF8.GetByteCount(name[(cut + 1)..]) > 100)
cut = name.IndexOf('/', cut + 1);
if (cut <= 0 || Encoding.UTF8.GetByteCount(name[..cut]) > 155)
throw new InvalidOperationException($"{name} is too long for a tar header");
prefix = name[..cut];
name = name[(cut + 1)..];
}
Text(header, 0, 100, name);
Octal(header, 100, 8, Convert.ToInt32("640", 8));
Octal(header, 108, 8, 0);
Octal(header, 116, 8, 0);
if (entry.Size < 1L << 33)
Octal(header, 124, 12, entry.Size);
else
{
header[124] = 0x80;
for (var i = 0; i < 8; i++)
header[135 - i] = (byte)(entry.Size >> (8 * i));
}
Octal(header, 136, 12, new DateTimeOffset(entry.ModifiedAt).ToUnixTimeSeconds());
header[156] = (byte)'0';
Text(header, 257, 6, "ustar");
Text(header, 263, 2, "00");
Text(header, 265, 32, "privapub");
Text(header, 297, 32, "privapub");
Text(header, 345, 155, prefix);
for (var i = 148; i < 156; i++)
header[i] = (byte)' ';
var sum = header.Sum(b => (long)b);
Text(header, 148, 7, Convert.ToString(sum, 8).PadLeft(6, '0'));
header[155] = (byte)' ';
return header;
}
static void Text(byte[] header, int at, int length, string value)
{
var bytes = Encoding.UTF8.GetBytes(value);
Array.Copy(bytes, 0, header, at, Math.Min(bytes.Length, length));
}
static void Octal(byte[] header, int at, int length, long value) =>
Text(header, at, length - 1, Convert.ToString(value, 8).PadLeft(length - 1, '0'));
[GeneratedRegex(@"^\d{8}-\d{6}-[a-z-]{1,20}$")]
public static partial Regex BackupId();
/// <summary>
/// A backup read from an uploaded tar into the backups' root: refused when it holds anything but plain files and
/// folders under one backup's folder, a path leaving it, or more than the disk can take. The backup, or why not.
/// </summary>
public static async Task<(BackupInfo Backup, string Error)> Import(Stream tar, string backupsRoot, CancellationToken token)
{
const int MaxEntries = 2_000_000;
var id = default(string);
var partial = default(string);
var entries = 0;
var free = new DriveInfo(Path.GetFullPath(backupsRoot)).AvailableFreeSpace;
var written = 0L;
try
{
await using var reader = new TarReader(tar, leaveOpen: true);
while (await reader.GetNextEntryAsync(copyData: false, token) is { } entry)
{
if (++entries > MaxEntries)
return (default, "too many files");
var name = entry.Name.TrimEnd('/');
var slash = name.IndexOf('/');
var top = slash < 0 ? name : name[..slash];
if (id == default)
{
if (!BackupId().IsMatch(top))
return (default, $"{top} is not a backup's folder");
id = top;
if (Directory.Exists(Path.Combine(backupsRoot, id)))
return (default, $"{id} is already here");
partial = Path.Combine(backupsRoot, id + ServerBackup.PartialSuffix);
if (Directory.Exists(partial))
Directory.Delete(partial, recursive: true);
ServerBackup.MakeDirectory(partial);
}
if (top != id)
return (default, $"{name} is outside {id}");
if (entry.EntryType is TarEntryType.Directory)
continue;
if (entry.EntryType is not (TarEntryType.RegularFile or TarEntryType.V7RegularFile))
return (default, $"{name} is not a plain file");
if (slash < 0)
return (default, $"{name} is not inside a folder");
written += entry.Length;
if (written > free - 512L * 1024 * 1024)
return (default, "not enough free disk");
var path = ServerBackup.Inside(partial, name[(slash + 1)..]);
Directory.CreateDirectory(Path.GetDirectoryName(path)!);
await using var file = OperatingSystem.IsWindows()
? new FileStream(path, FileMode.CreateNew, FileAccess.Write)
: new FileStream(path, new FileStreamOptions { Mode = FileMode.CreateNew, Access = FileAccess.Write,
UnixCreateMode = UnixFileMode.UserRead | UnixFileMode.UserWrite | UnixFileMode.GroupRead });
if (entry.DataStream != default)
await entry.DataStream.CopyToAsync(file, token);
}
if (id == default)
return (default, "the file holds nothing");
var manifest = ArchiveManifest.Read(partial);
if (manifest == default)
return (default, "it has no manifest");
manifest.Kind = "uploaded";
manifest.Write(partial);
Directory.Move(partial, Path.Combine(backupsRoot, id));
partial = default;
var problems = await ServerBackup.Verify(backupsRoot, id, token);
if (problems.Count > 0)
{
ServerBackup.Delete(backupsRoot, id);
return (default, string.Join("; ", problems.Take(5)));
}
return (ServerBackup.Find(backupsRoot, id), default);
}
catch (Exception ex) when (ex is FormatException or InvalidDataException or InvalidOperationException or EndOfStreamException or System.Text.Json.JsonException)
{
return (default, $"not a backup: {ex.Message}");
}
finally
{
if (partial != default && Directory.Exists(partial))
Directory.Delete(partial, recursive: true);
}
}
}
}
+90
View File
@@ -0,0 +1,90 @@
using Microsoft.Extensions.Options;
using MongoDB.Entities;
using PrivaPub.Domain.Media;
using PrivaPub.Models;
namespace PrivaPub.Infrastructure.Backup
{
// The server's backups, wherever they are started from: the CLI, the nightly schedule, the administrator's page.
public class Backups(IOptionsMonitor<BackupOptions> options, IOptionsMonitor<AppConfiguration> app, IMediaService media)
{
/// <summary>Where backups are kept: Backups:Root (production's /var/lib/privapub/backups), else beside the media root.</summary>
public string Root => Path.GetFullPath(options.CurrentValue.Root ?? media.Root.TrimEnd(Path.DirectorySeparatorChar) + "-backups");
public BackupOptions Options => options.CurrentValue;
public string Host => app.CurrentValue.BackendBaseAddress?.TrimEnd('/');
public BackupContext Context() => new(DB.Default.Database(), Root, media.Root, media.TrashRoot, Host, options.CurrentValue);
public Task<(BackupInfo Backup, string Error)> Create(string kind, bool dbOnly, CancellationToken token) =>
ServerBackup.Create(Context(), kind, dbOnly, token);
public List<BackupInfo> List() => ServerBackup.List(Root);
public BackupInfo Find(string id) => ServerBackup.Find(Root, id);
public Task<List<string>> Verify(string id, CancellationToken token) => ServerBackup.Verify(Root, id, token);
public bool Delete(string id) => ServerBackup.Delete(Root, id);
}
// A nightly backup at Backups:NightlyAt (UTC), or as soon as the service is up after missing it; the old ones rotated
// out as each is made. Personas' archives a week old go too.
public class BackupScheduler(Backups backups, ILogger<BackupScheduler> logger) : BackgroundService
{
static readonly TimeSpan Interval = TimeSpan.FromMinutes(10);
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
try
{
await Task.Delay(Interval, stoppingToken);
}
catch (OperationCanceledException)
{
return;
}
try
{
await RunIfDue(DateTime.UtcNow, stoppingToken);
await Domain.Portability.PersonaArchives.Forget(backups.Root, stoppingToken);
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
logger.LogError(ex, "The nightly backup failed");
}
}
}
/// <summary>Backs up when the day's time has come and there is no nightly backup since: whether it did.</summary>
public async Task<bool> RunIfDue(DateTime now, CancellationToken token)
{
var options = backups.Options;
if (!options.Nightly || !TimeOnly.TryParse(options.NightlyAt, System.Globalization.CultureInfo.InvariantCulture, out var at)
|| !IsDue(now, at, backups.List().Where(b => b.Kind == "nightly").Select(b => b.CreatedAt)))
return false;
var (made, error) = await backups.Create("nightly", dbOnly: false, token);
if (made == default)
{
logger.LogWarning("The nightly backup was not made: {Error}", error);
return false;
}
logger.LogInformation("Nightly backup {Id}: {Collections} collections, {Files} media files", made.Id, made.Manifest.Collections.Count, made.Manifest.Media.Files);
return true;
}
/// <summary>Whether the last time of day for a nightly backup has passed with no nightly backup made since.</summary>
public static bool IsDue(DateTime now, TimeOnly at, IEnumerable<DateTime> nightlies)
{
var due = now.Date + at.ToTimeSpan();
if (now < due)
due = due.AddDays(-1);
return !nightlies.Any(made => made >= due);
}
}
}
@@ -0,0 +1,35 @@
using System.Runtime.InteropServices;
namespace PrivaPub.Infrastructure.Backup
{
// A second name for a file (.NET has no API for it): a backup links the live media, which costs no disk, since names are
// random and files never change; deleting the live one leaves the backup's.
static class HardLink
{
// strings go to libc as UTF-8 on Unix
[DllImport("libc", EntryPoint = "link", SetLastError = true, CharSet = CharSet.Ansi)]
static extern int Link(string existing, string created);
/// <summary>Links, or copies where a link can't be made (another disk, a filesystem refusing it).</summary>
public static bool LinkOrCopy(string existing, string created)
{
Directory.CreateDirectory(Path.GetDirectoryName(created)!);
if (OperatingSystem.IsLinux())
{
try
{
if (Link(existing, created) == 0)
return true;
}
catch (DllNotFoundException)
{
}
catch (EntryPointNotFoundException)
{
}
}
File.Copy(existing, created, overwrite: true);
return false;
}
}
}
@@ -0,0 +1,59 @@
namespace PrivaPub.Infrastructure.Backup
{
// What Program asks before migrations: whether a restore waits, and what this process does about it. The service carries
// it out; a command waits for it (exit 75, EX_TEMPFAIL), except asking how it went.
public static class MaintenanceGate
{
/// <summary>The exit code that stops systemd restarting the service (RestartPreventExitStatus=75): a restore gave up.</summary>
public const int TryLater = 75;
/// <summary>Null to go on booting; else the code to exit with.</summary>
public static async Task<int?> Enter(IServiceProvider services, string[] command, CancellationToken token)
{
var backups = services.GetRequiredService<Backups>();
var logger = services.GetRequiredService<ILoggerFactory>().CreateLogger(nameof(MaintenanceGate));
if (command != default)
{
var waiting = RestoreMarker.Read(backups.Root);
if (waiting == default || command is ["restore", ..])
return default;
Console.Error.WriteLine($"the restore of {waiting.Backup} is {waiting.State}{(waiting.Error == default ? string.Empty : $" ({waiting.Error})")}: try again once the service has carried it out (PrivaPub admin restore --status)");
return TryLater;
}
return await ServerRestore.ApplyPending(backups.Context(), logger, token) switch
{
RestoreOutcome.Failed => 1,//systemd starts it again, and the next attempt redoes everything
RestoreOutcome.GaveUp => TryLater,
_ => (int?)null
};
}
}
// A restore asked for while the service runs (from the administrator's page or the CLI): the service stops within
// seconds, systemd starts it again, and the restore runs before anything else.
public class RestoreWatcher(Backups backups, IHostApplicationLifetime lifetime, ILogger<RestoreWatcher> logger) : BackgroundService
{
static readonly TimeSpan Interval = TimeSpan.FromSeconds(3);
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
try
{
await Task.Delay(Interval, stoppingToken);
}
catch (OperationCanceledException)
{
return;
}
if (RestoreMarker.Read(backups.Root) is { State: "pending" } marker)
{
logger.LogWarning("A restore of {Backup} was asked for by {RequestedBy}: stopping, to restore it at the next start", marker.Backup, marker.RequestedBy);
lifetime.StopApplication();
return;
}
}
}
}
}
@@ -0,0 +1,75 @@
using MongoDB.Driver;
using MongoDB.Entities;
namespace PrivaPub.Infrastructure.Backup
{
// One backup or restore at a time, whoever starts it (the CLI, the nightly schedule, the administrator's page): a
// document held with a heartbeat. A holder that died (no heartbeat for two minutes) is taken over. The collection is
// never backed up, nor dropped by a restore.
public class MaintenanceLock : Entity
{
public const string Name = "maintenance";
static readonly TimeSpan Stale = TimeSpan.FromMinutes(2);
static readonly TimeSpan Beat = TimeSpan.FromSeconds(30);
public string Owner { get; set; }
public string What { get; set; }
public DateTime Since { get; set; }
public DateTime Heartbeat { get; set; }
/// <summary>Takes the lock for what is said, or null when someone else holds it; disposing the result frees it.</summary>
public static async Task<Held> Take(string what, CancellationToken token)
{
var owner = $"{Environment.MachineName}:{Environment.ProcessId}:{Guid.NewGuid():N}";
var now = DateTime.UtcNow;
var stale = now - Stale;
try
{
var taken = await DB.Default.UpdateAndGet<MaintenanceLock>()
.Match(l => l.ID == Name && (l.Heartbeat < stale || l.Owner == null))
.Modify(l => l.Owner, owner)
.Modify(l => l.What, what)
.Modify(l => l.Since, now)
.Modify(l => l.Heartbeat, now)
.Option(o => o.IsUpsert = true)
.ExecuteAsync(token);
return taken?.Owner == owner ? new Held(owner) : default;
}
catch (MongoCommandException ex) when (ex.Code == 11000)
{
return default;//held: the upsert met the live lock's id
}
catch (MongoWriteException ex) when (ex.WriteError?.Category == ServerErrorCategory.DuplicateKey)
{
return default;
}
}
/// <summary>Who holds it now and for what, or null.</summary>
public static async Task<MaintenanceLock> Current(CancellationToken token)
{
var held = await DB.Default.Find<MaintenanceLock>().Match(l => l.ID == Name).ExecuteFirstAsync(token);
return held?.Owner != null && held.Heartbeat >= DateTime.UtcNow - Stale ? held : default;
}
public sealed class Held : IAsyncDisposable
{
readonly string _owner;
readonly Timer _heartbeat;
public Held(string owner)
{
_owner = owner;
_heartbeat = new Timer(_ => _ = DB.Default.Update<MaintenanceLock>().Match(l => l.ID == Name && l.Owner == owner)
.Modify(l => l.Heartbeat, DateTime.UtcNow).ExecuteAsync(), default, Beat, Beat);
}
public async ValueTask DisposeAsync()
{
await _heartbeat.DisposeAsync();
await DB.Default.Update<MaintenanceLock>().Match(l => l.ID == Name && l.Owner == _owner)
.Modify(l => l.Owner, null).Modify(l => l.Heartbeat, DateTime.MinValue).ExecuteAsync();
}
}
}
}
@@ -0,0 +1,271 @@
using MongoDB.Bson;
using MongoDB.Driver;
namespace PrivaPub.Infrastructure.Backup
{
// A restore may lose what was made since the backup, but it never undoes a protective act (owner decision 2026-10-07).
// From the pre-restore backup P, after the backup is imported:
// - who follows whom is P's: a follower that left stays gone, so no followers-only post reaches it, and one gained is
// kept; the persona's follows are P's too;
// - blocks, mutes, domain blocks (the server's and the personas'), being blocked, reserved names, deletion tombstones,
// reports, filters and OAuth applications are the union of both, P's row winning;
// - deletions win: a root, persona, group, post or remote account deleted since is deleted again (P's row, as its
// deletion left it), with what a deletion takes away; P's moderation of a persona (silenced, suspended, banned) stays;
// - a root keeps P's password, e-mail, ban and policies;
// - roots, personas and groups made since become tombstones: deleted, their names kept; local posts made since answer
// 410; media made since go to the trash, as do media trashed since (and they leave the posts restored with them);
// - every session ends, and every server's circuit is closed again.
// Run on the imported database, it reads P's files only, so it can run again after a failed attempt.
public static class ProtectiveMerge
{
static readonly (string Collection, string[] Key)[] Unions =
[
("ReservedName", ["Name"]),
("DomainBlock", ["Domain"]),
("Block", ["AvatarId", "TargetActorURI"]),
("Mute", ["AvatarId", "TargetActorURI"]),
("AccountDomainBlock", ["AvatarId", "Domain"]),
("BlockedBy", ["AvatarId", "ActorURI"]),
("DeletedObject", ["ObjectURI"]),
("Report", ["_id"]),
("PersonaFilter", ["_id"]),
("openiddict.applications", ["client_id"])
];
const string Since = "restore: made after the backup";
public static async Task Apply(IMongoDatabase database, string previous, RestoreReport report, CancellationToken token)
{
IEnumerable<BsonDocument> P(string collection) => ServerRestore.Read(Path.Combine(previous, collection + ".jsonl.gz"));
var now = DateTime.UtcNow;
// who follows whom: P's
foreach (var collection in new[] { "Follower", "Following" })
{
await database.DropCollectionAsync(collection, token);
await ServerRestore.Insert(database.GetCollection<BsonDocument>(collection), P(collection), token);
}
foreach (var (collection, key) in Unions)
report.ProtectiveKept += await Union(database.GetCollection<BsonDocument>(collection), P(collection), key, token);
var roots = database.GetCollection<BsonDocument>("RootUser");
foreach (var root in P("RootUser"))
{
var restored = await ById(roots, root["_id"], token);
if (restored == default)
{
// made since: deleted, as RootRemoval leaves a root
report.RootsTombstoned.Add(root.GetValue("UserName", "").ToString());
root["UserName"] = $"deleted-{root["_id"]}";
root["Email"] = BsonNull.Value;
root["HashedPassword"] = BsonNull.Value;
root["Policies"] = new BsonArray();
root["IsBanned"] = false;
root["IsEmailValidated"] = false;
root["DeletedAt"] = Deleted(root, "DeletedAt", now);
await roots.InsertOneAsync(root, cancellationToken: token);
continue;
}
if (IsSet(root, "DeletedAt"))
{
if (!IsSet(restored, "DeletedAt"))
report.RootsDeleted++;
await roots.ReplaceOneAsync(Id(root["_id"]), root, cancellationToken: token);
continue;
}
await roots.UpdateOneAsync(Id(root["_id"]), Copy(root, "HashedPassword", "Email", "IsEmailValidated", "IsBanned", "Policies",
"CredentialsChangedAt", "ResetPasswordToken", "ResetPasswordTokenSentAt"), cancellationToken: token);
}
var avatars = database.GetCollection<BsonDocument>("Avatar");
foreach (var avatar in P("Avatar"))
{
var restored = await ById(avatars, avatar["_id"], token);
if (restored == default)
{
report.PersonasTombstoned.Add(avatar.GetValue("UserName", "").ToString());
avatar["DeletionAt"] = Deleted(avatar, "DeletionAt", now);
await avatars.InsertOneAsync(avatar, cancellationToken: token);
continue;
}
if (IsSet(avatar, "DeletionAt"))
{
if (!IsSet(restored, "DeletionAt"))
report.PersonasDeleted++;
await avatars.ReplaceOneAsync(Id(avatar["_id"]), avatar, cancellationToken: token);
// what RootRemoval takes with a persona, which the backup still had
foreach (var collection in new[] { "PersonaListMember", "PersonaList", "PersonaFilter", "FollowedTag", "ScheduledStatus" })
await database.GetCollection<BsonDocument>(collection).DeleteManyAsync(new BsonDocument("AvatarId", avatar["_id"].ToString()), token);
continue;
}
var moderated = new[] { "SilencedAt", "SuspendedAt", "BannedAt" }.Where(field => IsSet(avatar, field)).ToArray();
if (moderated.Length > 0)
await avatars.UpdateOneAsync(Id(avatar["_id"]), Copy(avatar, moderated), cancellationToken: token);
}
var links = database.GetCollection<BsonDocument>("RootToAvatar");
var linked = (await (await links.FindAsync(FilterDefinition<BsonDocument>.Empty, cancellationToken: token)).ToListAsync(token))
.Select(l => l.GetValue("AvatarId", BsonNull.Value).ToString()).ToHashSet(StringComparer.Ordinal);
await ServerRestore.Insert(links, P("RootToAvatar").Where(l => !linked.Contains(l.GetValue("AvatarId", BsonNull.Value).ToString())), token);
var groups = database.GetCollection<BsonDocument>("Group");
foreach (var group in P("Group"))
{
var restored = await ById(groups, group["_id"], token);
if (restored == default)
{
report.GroupsTombstoned.Add(group.GetValue("UserName", "").ToString());
group["DeletionAt"] = Deleted(group, "DeletionAt", now);
await groups.InsertOneAsync(group, cancellationToken: token);
continue;
}
if (!IsSet(group, "DeletionAt"))
continue;
if (!IsSet(restored, "DeletionAt"))
report.GroupsDeleted++;
await groups.ReplaceOneAsync(Id(group["_id"]), group, cancellationToken: token);
}
// remote accounts deleted or gone since: as P has them
var foreign = database.GetCollection<BsonDocument>("ForeignAvatar");
foreach (var account in P("ForeignAvatar").Where(a => IsSet(a, "DeletionAt") || IsSet(a, "ForgottenAt")))
await foreign.ReplaceOneAsync(Id(account["_id"]), account, cancellationToken: token);
await Posts(database, P("Post"), report, token);
await Media(database, P("MediaAttachment"), report, token);
// every session ends: a new stamp for each root's tokens, and the personas' OAuth tokens go
foreach (var root in await (await roots.FindAsync(FilterDefinition<BsonDocument>.Empty, cancellationToken: token)).ToListAsync(token))
await roots.UpdateOneAsync(Id(root["_id"]), Builders<BsonDocument>.Update
.Set("SessionStamp", Guid.NewGuid().ToString("N"))
.Set("CredentialsChangedAt", now), cancellationToken: token);
foreach (var collection in new[] { "openiddict.tokens", "openiddict.authorizations" })
report.SessionsEnded += (int)(await database.GetCollection<BsonDocument>(collection).DeleteManyAsync(FilterDefinition<BsonDocument>.Empty, token)).DeletedCount;
// what the backup remembered of servers being down is long past
await database.GetCollection<BsonDocument>("RemoteInstance").UpdateManyAsync(FilterDefinition<BsonDocument>.Empty,
Builders<BsonDocument>.Update.Set("ConsecutiveFailures", 0).Set("UnavailableUntil", BsonNull.Value), cancellationToken: token);
}
// posts deleted since: deleted again, out of timelines and pins; local posts made since: there as deleted, as a
// deletion leaves them, so they answer 410 and their ids and addresses are never given again
static async Task Posts(IMongoDatabase database, IEnumerable<BsonDocument> previous, RestoreReport report, CancellationToken token)
{
var posts = database.GetCollection<BsonDocument>("Post");
var now = DateTime.UtcNow;
foreach (var batch in previous.Where(p => IsSet(p, "DeletedAt") || IsLocal(p)).Chunk(500))
{
var ids = new BsonArray(batch.Select(p => p["_id"]));
var restored = (await (await posts.FindAsync(new BsonDocument("_id", new BsonDocument("$in", ids)), cancellationToken: token)).ToListAsync(token))
.ToDictionary(p => p["_id"]);
foreach (var post in batch)
{
if (!restored.TryGetValue(post["_id"], out var mine))
{
if (!IsLocal(post))
continue;
if (!IsSet(post, "DeletedAt"))
{
post["DeletedAt"] = now;
foreach (var field in new[] { "Text", "ContentHtml", "Title", "SpoilerText" })
post[field] = BsonNull.Value;
post["Media"] = new BsonArray();
post["Revisions"] = new BsonArray();
}
await posts.InsertOneAsync(post, cancellationToken: token);
report.PostsGone++;
continue;
}
if (!IsSet(post, "DeletedAt") || IsSet(mine, "DeletedAt"))
continue;
await posts.ReplaceOneAsync(Id(post["_id"]), post, cancellationToken: token);
var id = post["_id"].ToString();
await database.GetCollection<BsonDocument>("TimelineEntry").DeleteManyAsync(
new BsonDocument("$or", new BsonArray { new BsonDocument("PostId", id), new BsonDocument("ReblogOfPostId", id) }), token);
await database.GetCollection<BsonDocument>("Pin").DeleteManyAsync(new BsonDocument("PostId", id), token);
report.PostsDeleted++;
}
}
}
// media trashed since, and media made since: in the trash (the janitor deletes their files after its grace), and out
// of the posts restored; media of posts deleted now: in the trash too
static async Task Media(IMongoDatabase database, IEnumerable<BsonDocument> previous, RestoreReport report, CancellationToken token)
{
var media = database.GetCollection<BsonDocument>("MediaAttachment");
var posts = database.GetCollection<BsonDocument>("Post");
var now = DateTime.UtcNow;
foreach (var batch in previous.Chunk(1000))
{
var ids = new BsonArray(batch.Select(m => m["_id"]));
var restored = (await (await media.FindAsync(new BsonDocument("_id", new BsonDocument("$in", ids)), cancellationToken: token)).ToListAsync(token))
.ToDictionary(m => m["_id"]);
var made = new List<BsonDocument>();
foreach (var row in batch)
{
if (!restored.TryGetValue(row["_id"], out var mine))
{
if (!IsSet(row, "TrashedAt"))
{
row["TrashedAt"] = now;
row["TrashReason"] = Since;
}
made.Add(row);
continue;
}
if (!IsSet(row, "TrashedAt") || IsSet(mine, "TrashedAt"))
continue;
await media.UpdateOneAsync(Id(row["_id"]), Copy(row, "TrashedAt", "TrashReason"), cancellationToken: token);
if (mine.GetValue("PostId", BsonNull.Value) is BsonString postId)
await posts.UpdateOneAsync(Id(ObjectId.TryParse(postId.AsString, out var post) ? post : postId), Builders<BsonDocument>.Update.PullFilter<BsonDocument>("Media",
new BsonDocument("AttachmentId", row["_id"].ToString())), cancellationToken: token);
report.MediaTrashed++;
}
await ServerRestore.Insert(media, made, token);
report.MediaTrashed += made.Count(m => m["TrashReason"] == Since);
}
var deleted = await (await posts.FindAsync(new BsonDocument("DeletedAt", new BsonDocument("$ne", BsonNull.Value)),
new FindOptions<BsonDocument> { Projection = new BsonDocument("_id", 1) }, token)).ToListAsync(token);
foreach (var batch in deleted.Select(p => (BsonValue)p["_id"].ToString()).Chunk(1000))
report.MediaTrashed += (int)(await media.UpdateManyAsync(
new BsonDocument { { "PostId", new BsonDocument("$in", new BsonArray(batch)) }, { "TrashedAt", BsonNull.Value } },
Builders<BsonDocument>.Update.Set("TrashedAt", now).Set("TrashReason", "restore: its post was deleted"), cancellationToken: token)).ModifiedCount;
}
// P's rows added, each replacing the restored row with the same key: how many P had
static async Task<int> Union(IMongoCollection<BsonDocument> collection, IEnumerable<BsonDocument> previous, string[] key, CancellationToken token)
{
var count = 0;
foreach (var batch in previous.Chunk(500))
{
var writes = new List<WriteModel<BsonDocument>>();
foreach (var row in batch)
{
var match = new BsonDocument(key.Select(field => new BsonElement(field, row.GetValue(field, BsonNull.Value))));
writes.Add(new DeleteManyModel<BsonDocument>(match));
writes.Add(new InsertOneModel<BsonDocument>(row));
}
await collection.BulkWriteAsync(writes, new BulkWriteOptions { IsOrdered = true, BypassDocumentValidation = true }, token);
count += batch.Length;
}
return count;
}
static bool IsLocal(BsonDocument post) =>
post.GetValue("GroupUserId", BsonNull.Value) is BsonString && !post.GetValue("IsFederatedCopy", false).ToBoolean();
static bool IsSet(BsonDocument row, string field) => row.TryGetValue(field, out var value) && !value.IsBsonNull;
static BsonValue Deleted(BsonDocument row, string field, DateTime now) => IsSet(row, field) ? row[field] : now;
static FilterDefinition<BsonDocument> Id(BsonValue id) => new BsonDocument("_id", id);
static async Task<BsonDocument> ById(IMongoCollection<BsonDocument> collection, BsonValue id, CancellationToken token) =>
await (await collection.FindAsync(Id(id), cancellationToken: token)).FirstOrDefaultAsync(token);
static UpdateDefinition<BsonDocument> Copy(BsonDocument from, params string[] fields) =>
Builders<BsonDocument>.Update.Combine(fields.Select(field => Builders<BsonDocument>.Update.Set(field, from.GetValue(field, BsonNull.Value))));
}
}
@@ -0,0 +1,58 @@
using System.Text.Json;
namespace PrivaPub.Infrastructure.Backup
{
// A restore asked for (by the administrator's page or the CLI) and carried out at the next boot, before anything
// else (Program, MaintenanceGate): restore.json in the backups' directory, written whole or not at all.
public sealed class RestoreMarker
{
public const string FileName = "restore.json";
public const int MaxAttempts = 3;
public string Backup { get; set; }//the backup restored
public string PreRestore { get; set; }//the backup taken just before, what the protective merge reads
public string State { get; set; } = "pending";//pending, then running
public string RequestedBy { get; set; }//who asked: a root's name, or "cli"
public int Attempts { get; set; }
public DateTime RequestedAt { get; set; } = DateTime.UtcNow;
public string Error { get; set; }
public static string PathIn(string backupsRoot) => Path.Combine(backupsRoot, FileName);
public static RestoreMarker Read(string backupsRoot)
{
var path = PathIn(backupsRoot);
if (!File.Exists(path))
return default;
try
{
return JsonSerializer.Deserialize<RestoreMarker>(File.ReadAllText(path));
}
catch (JsonException)
{
//never written half (Write is atomic): someone's hand; the server won't guess what was meant
return new RestoreMarker { State = "unreadable", Attempts = MaxAttempts, Error = $"{FileName} can't be read" };
}
}
public void Write(string backupsRoot)
{
Directory.CreateDirectory(backupsRoot);
var path = PathIn(backupsRoot);
var part = path + ".part";
using (var file = new FileStream(part, FileMode.Create, FileAccess.Write))
{
JsonSerializer.Serialize(file, this);
file.Flush(flushToDisk: true);
}
File.Move(part, path, overwrite: true);
}
public static void Clear(string backupsRoot)
{
var path = PathIn(backupsRoot);
if (File.Exists(path))
File.Delete(path);
}
}
}
@@ -0,0 +1,62 @@
using MongoDB.Entities;
namespace PrivaPub.Infrastructure.Backup
{
// What a restore did (or why it was abandoned), kept for the administrator's page and for the followers' grace: for
// FollowersGrace after a restore the followers' digests (FEP-8fcf) are neither sent nor acted on to undo a follow, since
// what the restore lost of either side is not the other's fault.
public class RestoreRecord : Entity
{
public static readonly TimeSpan FollowersGrace = TimeSpan.FromDays(14);
public string Backup { get; set; }
public DateTime BackupCreatedAt { get; set; }
public string PreRestore { get; set; }
public string RequestedBy { get; set; }
public DateTime RequestedAt { get; set; }
public DateTime RestoredAt { get; set; } = DateTime.UtcNow;
public int Attempts { get; set; }
public bool Abandoned { get; set; }//nothing was changed: the server booted as it was
public string Error { get; set; }
public RestoreReport Report { get; set; } = new();
static (DateTime Until, DateTime Read) _grace;
/// <summary>Whether a restore ended less than FollowersGrace ago (read at most once a minute).</summary>
public static async Task<bool> InFollowersGrace(CancellationToken token)
{
var now = DateTime.UtcNow;
var cached = _grace;
if (now - cached.Read > TimeSpan.FromMinutes(1))
{
var last = await DB.Default.Find<RestoreRecord>().Match(r => !r.Abandoned).Sort(r => r.RestoredAt, Order.Descending).ExecuteFirstAsync(token);
cached = (last == default ? DateTime.MinValue : last.RestoredAt + FollowersGrace, now);
_grace = cached;
}
return now < cached.Until;
}
/// <summary>Forgets what was read, so a restore made in this process counts at once.</summary>
public static void Forget() => _grace = default;
}
public class RestoreReport
{
public int Collections { get; set; }
public long Documents { get; set; }
public int CollectionsDropped { get; set; }//live collections the backup had no documents in
public int MediaRestored { get; set; }//files the live directory lacked, brought back
public int MediaMissing { get; set; }
public int RootsDeleted { get; set; }//deleted since the backup, deleted again
public int PersonasDeleted { get; set; }
public int GroupsDeleted { get; set; }
public int PostsDeleted { get; set; }
public List<string> RootsTombstoned { get; set; } = [];//made after the backup: deleted, their names kept
public List<string> PersonasTombstoned { get; set; } = [];
public List<string> GroupsTombstoned { get; set; } = [];
public int PostsGone { get; set; }//local posts made after the backup: 410 from now on
public int MediaTrashed { get; set; }
public int ProtectiveKept { get; set; }//blocks, mutes, domain blocks, reserved names, reports, filters, deletions kept from since
public int SessionsEnded { get; set; }
}
}
@@ -0,0 +1,343 @@
using MongoDB.Bson;
using MongoDB.Bson.IO;
using MongoDB.Driver;
using PrivaPub.Infrastructure.Data;
using System.Globalization;
using System.IO.Compression;
using System.Security.Cryptography;
using System.Text;
namespace PrivaPub.Infrastructure.Backup
{
// What a backup needs to know: where things are, and whose host it is.
public sealed record BackupContext(IMongoDatabase Database, string BackupsRoot, string MediaRoot, string TrashRoot, string Host, BackupOptions Options);
public sealed record BackupInfo(string Id, string Kind, DateTime CreatedAt, long Bytes, ArchiveManifest Manifest);
// The whole server, backed up as files (owner decision 2026-10-07: a whole-server backup, plain, protected by file
// permissions). Each backup is a directory <stamp>-<kind> in the backups' root:
// - manifest.json: what it holds (ArchiveManifest);
// - db/<collection>.jsonl.gz: every document of each collection, one per line, in canonical Extended JSON (every BSON
// type kept as it was), all read at one instant when Mongo is a replica set (a snapshot session);
// - media/<path>: the media files rows hold, as hard links to the live ones (no disk spent; a deleted file stays here
// until the backup is rotated out), copied where a link can't be made; a db-only backup lists them instead.
// It is written as <id>.partial and renamed once whole. Left out: what belongs to the moment (Excluded). Everything is
// readable by the service's user and group only: it holds every persona's private key and private posts.
public static class ServerBackup
{
public const string PartialSuffix = ".partial";
public static readonly IReadOnlyDictionary<string, string> Excluded = new Dictionary<string, string>
{
["InteractionSalt"] = "the day's statistics salt never outlives its day (owner rule)",
["Job"] = "work in flight: deliveries and inbox processing belong to the moment",
["Delivery"] = "work in flight, from before jobs",
["EmailRecovery"] = "recovery codes live an hour",
["openiddict.tokens"] = "sessions: a restore ends every one",
["openiddict.authorizations"] = "sessions: a restore ends every one",
[nameof(MaintenanceLock)] = "who is backing up or restoring now",
["AppConfiguration"] = "the configuration's copy holds the SMTP password, and is made again from appsettings at boot",
[nameof(RestoreRecord)] = "what restores did outlives what they restore",
["PersonaArchive"] = "personas' archives being made or imported belong to the moment"
};
static readonly JsonWriterSettings Json = new() { OutputMode = JsonOutputMode.CanonicalExtendedJson, Indent = false };
// 2770: the service (www-data) and the deploy (in www-data's group) each read and rotate what the other wrote, and new
// files take the directory's group
const UnixFileMode DirectoryMode = UnixFileMode.UserRead | UnixFileMode.UserWrite | UnixFileMode.UserExecute
| UnixFileMode.GroupRead | UnixFileMode.GroupWrite | UnixFileMode.GroupExecute | UnixFileMode.SetGroup;
const UnixFileMode FileMode0640 = UnixFileMode.UserRead | UnixFileMode.UserWrite | UnixFileMode.GroupRead;
/// <summary>Takes the maintenance lock and backs the server up: the backup, or why not.</summary>
public static async Task<(BackupInfo Backup, string Error)> Create(BackupContext context, string kind, bool dbOnly, CancellationToken token)
{
await using var held = await MaintenanceLock.Take("backup", token);
if (held == default)
return (default, "a backup or a restore is already running");
return await CreateHeld(context, kind, dbOnly, token);
}
/// <summary>Backs the server up with the maintenance lock already held (a restore's own backup, taken first).</summary>
public static async Task<(BackupInfo Backup, string Error)> CreateHeld(BackupContext context, string kind, bool dbOnly, CancellationToken token)
{
var database = context.Database;
var names = (await (await database.ListCollectionNamesAsync(cancellationToken: token)).ToListAsync(token))
.Where(n => !n.StartsWith("system.", StringComparison.Ordinal))
.OrderBy(n => n, StringComparer.Ordinal)
.ToList();
MakeDirectory(context.BackupsRoot);
var stats = await database.RunCommandAsync<BsonDocument>(new BsonDocument("dbStats", 1), cancellationToken: token);
var needed = (long)(stats.GetValue("dataSize", 0).ToDouble() * 1.1);
if (new DriveInfo(Path.GetFullPath(context.BackupsRoot)).AvailableFreeSpace < needed)
return (default, $"not enough free disk for a backup: about {needed / 1024 / 1024} MB needed");
var id = $"{DateTime.UtcNow.ToString("yyyyMMdd-HHmmss", CultureInfo.InvariantCulture)}-{kind}";
var partial = Path.Combine(context.BackupsRoot, id + PartialSuffix);
MakeDirectory(partial);
MakeDirectory(Path.Combine(partial, "db"));
try
{
var manifest = new ArchiveManifest
{
Kind = kind,
Host = context.Host,
AppCommit = BuildInfo.Commit,
AppRef = BuildInfo.Ref,
DbOnly = dbOnly
};
var replica = await MongoTopology.IsReplicaSet(database, token);
var media = new SortedSet<string>(StringComparer.Ordinal);
using var session = replica ? await database.Client.StartSessionAsync(new ClientSessionOptions { Snapshot = true }, token) : default;
var window = replica ? await RaiseSnapshotWindow(database, token) : default(int?);
try
{
foreach (var name in names)
{
if (Excluded.TryGetValue(name, out var why))
{
manifest.Excluded.Add(new ArchiveManifest.ExcludedEntry { Name = name, Why = why });
continue;
}
manifest.Collections.Add(await Dump(database, session, name, partial, name == "MediaAttachment" ? media : default, token));
}
(manifest.NewestMigration, manifest.MigrationNumber) = await NewestMigration(database, session, token);
}
finally
{
if (window is { } previous)
await SetSnapshotWindow(database, previous, CancellationToken.None);
}
manifest.Consistent = replica;
// the files it holds (or, db-only, lists); a row whose file was already gone is only counted
foreach (var relative in media)
{
var source = new[] { context.MediaRoot, context.TrashRoot }.Select(root => Inside(root, relative)).FirstOrDefault(File.Exists);
if (source == default)
{
manifest.Media.Missing++;
continue;
}
if (!dbOnly)
HardLink.LinkOrCopy(source, Inside(Path.Combine(partial, "media"), relative));
manifest.Media.List.Add(relative);
manifest.Media.Files++;
manifest.Media.Bytes += new FileInfo(source).Length;
}
manifest.Write(partial);
Directory.Move(partial, Path.Combine(context.BackupsRoot, id));
Retain(context.BackupsRoot, context.Options);
return (Info(context.BackupsRoot, id), default);
}
catch
{
if (Directory.Exists(partial))
Directory.Delete(partial, recursive: true);
throw;
}
}
// a collection read as raw BSON in batches, each document a line of canonical Extended JSON, gzipped; the media rows'
// files collected on the way
static async Task<ArchiveManifest.CollectionEntry> Dump(IMongoDatabase database, IClientSessionHandle session, string name, string directory,
ISet<string> media, CancellationToken token)
{
var collection = database.GetCollection<RawBsonDocument>(name);
var path = Path.Combine(directory, "db", name + ".jsonl.gz");
var count = 0L;
await using (var file = NewFile(path))
await using (var gzip = new GZipStream(file, CompressionLevel.Fastest))
await using (var writer = new StreamWriter(gzip, new UTF8Encoding(false)))
{
var options = new FindOptions<RawBsonDocument> { BatchSize = 1000 };
using var cursor = session == default
? await collection.FindAsync(FilterDefinition<RawBsonDocument>.Empty, options, token)
: await collection.FindAsync(session, FilterDefinition<RawBsonDocument>.Empty, options, token);
while (await cursor.MoveNextAsync(token))
foreach (var document in cursor.Current)
using (document)
{
await writer.WriteLineAsync(document.ToJson(Json));
count++;
if (media != default)
Collect(document, media);
}
}
var indexes = await (await collection.Indexes.ListAsync(token)).ToListAsync(token);
return new ArchiveManifest.CollectionEntry
{
Name = name,
Count = count,
Bytes = new FileInfo(path).Length,
Sha256 = await Hash(path, token),
Indexes = [.. indexes.Select(i => i.ToJson(Json))]
};
}
// a media row's files, unless it is trashed
static void Collect(RawBsonDocument row, ISet<string> media)
{
if (row.TryGetValue("TrashedAt", out var trashed) && !trashed.IsBsonNull)
return;
foreach (var field in new[] { "FilePath", "PreviewPath" })
if (row.TryGetValue(field, out var value) && value.IsString && Safe(value.AsString))
media.Add(value.AsString);
}
// MongoDB.Entities records each migration run with its class's number (_016_... is 16) and its name in words
static async Task<(string Name, int Number)> NewestMigration(IMongoDatabase database, IClientSessionHandle session, CancellationToken token)
{
var history = database.GetCollection<BsonDocument>("_migration_history_");
var options = new FindOptions<BsonDocument> { Sort = new BsonDocument("Number", -1), Limit = 1 };
var newest = session == default
? await (await history.FindAsync(FilterDefinition<BsonDocument>.Empty, options, token)).FirstOrDefaultAsync(token)
: await (await history.FindAsync(session, FilterDefinition<BsonDocument>.Empty, options, token)).FirstOrDefaultAsync(token);
return newest == default ? default : (newest.GetValue("Name", BsonNull.Value).ToString(), newest.GetValue("Number", 0).ToInt32());
}
/// <summary>The newest migration this build has: a backup made by a newer one can't be restored by it.</summary>
public static int CodeMigration() => typeof(ServerBackup).Assembly.GetTypes()
.Where(t => typeof(MongoDB.Entities.IMigration).IsAssignableFrom(t) && !t.IsAbstract)
.Select(t => int.TryParse(new string(t.Name.TrimStart('_').TakeWhile(char.IsDigit).ToArray()), out var number) ? number : 0)
.DefaultIfEmpty(0)
.Max();
// how long a snapshot stays readable: raised only while a backup reads (a longer window keeps old versions in the
// small cache all day); the value it had, to set back
static async Task<int?> RaiseSnapshotWindow(IMongoDatabase database, CancellationToken token)
{
var admin = database.Client.GetDatabase("admin");
try
{
var current = await admin.RunCommandAsync<BsonDocument>(new BsonDocument { { "getParameter", 1 }, { "minSnapshotHistoryWindowInSeconds", 1 } }, cancellationToken: token);
var previous = current.GetValue("minSnapshotHistoryWindowInSeconds", 300).ToInt32();
await SetSnapshotWindow(database, Math.Max(previous, 3600), token);
return previous;
}
catch (MongoCommandException)
{
return default;
}
}
static async Task SetSnapshotWindow(IMongoDatabase database, int seconds, CancellationToken token) =>
await database.Client.GetDatabase("admin").RunCommandAsync<BsonDocument>(
new BsonDocument { { "setParameter", 1 }, { "minSnapshotHistoryWindowInSeconds", seconds } }, cancellationToken: token);
/// <summary>The backups kept, newest first (not one still being written).</summary>
public static List<BackupInfo> List(string backupsRoot)
{
if (!Directory.Exists(backupsRoot))
return [];
return Directory.EnumerateDirectories(backupsRoot)
.Select(Path.GetFileName)
.Where(name => !name.EndsWith(PartialSuffix, StringComparison.Ordinal) && File.Exists(Path.Combine(backupsRoot, name, ArchiveManifest.FileName)))
.Select(name => Info(backupsRoot, name))
.OrderByDescending(b => b.CreatedAt)
.ToList();
}
public static BackupInfo Find(string backupsRoot, string id) =>
Safe(id) && !id.Contains('/') && Directory.Exists(Path.Combine(backupsRoot, id)) && !id.EndsWith(PartialSuffix, StringComparison.Ordinal)
? Info(backupsRoot, id)
: default;
static BackupInfo Info(string backupsRoot, string id)
{
var directory = Path.Combine(backupsRoot, id);
var manifest = ArchiveManifest.Read(directory);
var bytes = Directory.EnumerateFiles(Path.Combine(directory, "db")).Sum(f => new FileInfo(f).Length);
return new BackupInfo(id, manifest?.Kind, manifest?.CreatedAt ?? Directory.GetCreationTimeUtc(directory), bytes, manifest);
}
/// <summary>Whether every file of a backup is what its manifest says: the problems found (none when whole).</summary>
public static async Task<List<string>> Verify(string backupsRoot, string id, CancellationToken token)
{
var problems = new List<string>();
var backup = Find(backupsRoot, id);
if (backup?.Manifest == default)
return ["no such backup, or no manifest"];
var directory = Path.Combine(backupsRoot, id);
foreach (var collection in backup.Manifest.Collections)
{
var path = Path.Combine(directory, "db", collection.Name + ".jsonl.gz");
if (!File.Exists(path))
problems.Add($"{collection.Name}: missing");
else if (await Hash(path, token) != collection.Sha256)
problems.Add($"{collection.Name}: altered");
}
if (!backup.Manifest.DbOnly)
foreach (var relative in backup.Manifest.Media.List.Where(r => !File.Exists(Inside(Path.Combine(directory, "media"), r))))
problems.Add($"media {relative}: missing");
return problems;
}
/// <summary>Deletes a backup (its media links go; the live files stay).</summary>
public static bool Delete(string backupsRoot, string id)
{
if (Find(backupsRoot, id) == default)
return false;
Directory.Delete(Path.Combine(backupsRoot, id), recursive: true);
return true;
}
// what is kept: the newest nightly ones for KeepDaily days and the newest of each of KeepWeekly weeks before, the
// newest pre-deploy and pre-restore ones; manual and uploaded ones until deleted; a backup that died writing goes
public static void Retain(string backupsRoot, BackupOptions options)
{
var all = List(backupsRoot);
var nightly = all.Where(b => b.Kind == "nightly").OrderByDescending(b => b.CreatedAt).ToList();
var kept = nightly.Take(options.KeepDaily).ToHashSet();
foreach (var week in nightly.Skip(options.KeepDaily).GroupBy(b => (ISOWeek.GetYear(b.CreatedAt), ISOWeek.GetWeekOfYear(b.CreatedAt))).Take(options.KeepWeekly))
kept.Add(week.First());
var doomed = nightly.Where(b => !kept.Contains(b))
.Concat(all.Where(b => b.Kind == "pre-deploy").OrderByDescending(b => b.CreatedAt).Skip(options.KeepPreDeploy))
.Concat(all.Where(b => b.Kind == "pre-restore").OrderByDescending(b => b.CreatedAt).Skip(options.KeepPreRestore));
var stale = Directory.EnumerateDirectories(backupsRoot, "*" + PartialSuffix).Where(p => Directory.GetLastWriteTimeUtc(p) < DateTime.UtcNow.AddDays(-1));
foreach (var directory in doomed.Select(b => Path.Combine(backupsRoot, b.Id)).Concat(stale).ToList())
try
{
Directory.Delete(directory, recursive: true);
}
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
{
//another user's backup this one may not delete: the next rotation tries again
}
}
// a relative path from a database row or a manifest, never outside its root
static bool Safe(string relative) =>
!string.IsNullOrEmpty(relative) && !Path.IsPathRooted(relative) && !relative.Split('/', '\\').Any(part => part is ".." or ".");
public static string Inside(string root, string relative)
{
var full = Path.GetFullPath(Path.Combine(root, relative));
if (!Safe(relative) || !full.StartsWith(Path.GetFullPath(root) + Path.DirectorySeparatorChar, StringComparison.Ordinal))
throw new InvalidOperationException($"{relative} is not inside {root}");
return full;
}
static async Task<string> Hash(string path, CancellationToken token)
{
await using var file = File.OpenRead(path);
return Convert.ToHexStringLower(await SHA256.HashDataAsync(file, token));
}
static FileStream NewFile(string path) => OperatingSystem.IsWindows()
? new FileStream(path, FileMode.CreateNew, FileAccess.Write)
: new FileStream(path, new FileStreamOptions { Mode = FileMode.CreateNew, Access = FileAccess.Write, UnixCreateMode = FileMode0640 });
// set after creating, since the umask would take the group's write away
public static void MakeDirectory(string path)
{
var existed = Directory.Exists(path);
Directory.CreateDirectory(path);
if (!existed && !OperatingSystem.IsWindows())
File.SetUnixFileMode(path, DirectoryMode);
}
}
}
@@ -0,0 +1,247 @@
using MongoDB.Bson;
using MongoDB.Driver;
using System.IO.Compression;
namespace PrivaPub.Infrastructure.Backup
{
public enum RestoreOutcome
{
None,//nothing was asked
Restored,
Abandoned,//refused before anything changed: the server boots as it was
Failed,//failed midway: the next boot tries again from the start
GaveUp//failed MaxAttempts times: the server stays down until someone looks
}
// A backup restored (owner decision 2026-10-07: from the CLI or the administrator's page). Asking writes restore.json
// and the running service stops at once (RestoreWatcher); the restore itself runs at the next boot, before migrations,
// indexes and every hosted service, with nothing else touching the database:
// 1. a pre-restore backup P is taken, once (a retry reuses it);
// 2. every collection the backup holds is dropped and imported raw, with its indexes; every other one is dropped,
// except what a backup never holds (ServerBackup.Excluded), which stays as it is;
// 3. media files the live directory lacks come back from the backup (or the trash); a restore deletes no file;
// 4. ProtectiveMerge brings back from P every protective act made since the backup;
// 5. every session ends, every server's circuit closes, and a RestoreRecord tells what happened.
// Each attempt redoes everything, so one that dies midway converges on the next.
public static class ServerRestore
{
static readonly TimeSpan LockWait = TimeSpan.FromMinutes(10);
/// <summary>Why a backup can't be restored here: none when it can.</summary>
public static async Task<List<string>> Check(BackupContext context, string id, CancellationToken token)
{
var backup = ServerBackup.Find(context.BackupsRoot, id);
if (backup?.Manifest == default)
return ["no such backup"];
var manifest = backup.Manifest;
var problems = new List<string>();
if (manifest.Format != ArchiveManifest.CurrentFormat)
problems.Add($"its format is {manifest.Format}, this build reads {ArchiveManifest.CurrentFormat}");
if (!string.Equals(manifest.Host?.TrimEnd('/'), context.Host, StringComparison.OrdinalIgnoreCase))
problems.Add($"it was made by {manifest.Host}, not {context.Host}: every id and address in it names its host");
var code = ServerBackup.CodeMigration();
if (manifest.MigrationNumber > code)
problems.Add($"it was made by a newer build (migration {manifest.MigrationNumber}; this one knows {code})");
problems.AddRange(await ServerBackup.Verify(context.BackupsRoot, id, token));
return problems;
}
/// <summary>Asks for a backup to be restored at the next boot: why not, or null once asked.</summary>
public static async Task<string> Request(BackupContext context, string id, string requestedBy, CancellationToken token)
{
var problems = await Check(context, id, token);
if (problems.Count > 0)
return string.Join("; ", problems.Take(5)) + (problems.Count > 5 ? $"; and {problems.Count - 5} more" : string.Empty);
var waiting = RestoreMarker.Read(context.BackupsRoot);
if (waiting != default && waiting.Attempts < RestoreMarker.MaxAttempts)
return $"the restore of {waiting.Backup} is already {waiting.State}";
if (await MaintenanceLock.Current(token) is { } held)
return $"a {held.What} is running";
new RestoreMarker { Backup = id, RequestedBy = requestedBy }.Write(context.BackupsRoot);
return default;
}
/// <summary>The restore asked for, carried out (at boot, before migrations): what came of it.</summary>
public static async Task<RestoreOutcome> ApplyPending(BackupContext context, ILogger logger, CancellationToken token)
{
var marker = RestoreMarker.Read(context.BackupsRoot);
if (marker == default)
return RestoreOutcome.None;
if (marker.Attempts >= RestoreMarker.MaxAttempts)
{
logger.LogCritical("The restore of {Backup} failed {Attempts} times ({Error}). The database may be half restored; {PreRestore} holds it as it was before. Restore that backup or another (PrivaPub admin restore <id>), or delete {Marker} to boot as it is",
marker.Backup, marker.Attempts, marker.Error, marker.PreRestore ?? "no backup", RestoreMarker.PathIn(context.BackupsRoot));
return RestoreOutcome.GaveUp;
}
await using var held = await TakeLock(token);
if (held == default)
return await Abandon(context, marker, "a backup kept the maintenance lock for ten minutes", logger, token);
var problems = await Check(context, marker.Backup, token);
if (problems.Count > 0 && marker.PreRestore == default)
return await Abandon(context, marker, string.Join("; ", problems), logger, token);
if (marker.PreRestore == default)
{
var (taken, error) = await ServerBackup.CreateHeld(context, "pre-restore", dbOnly: false, token);
if (taken == default)
return await Abandon(context, marker, $"the pre-restore backup could not be made: {error}", logger, token);
marker.PreRestore = taken.Id;
}
marker.State = "running";
marker.Attempts++;
marker.Error = default;
marker.Write(context.BackupsRoot);
logger.LogWarning("Restoring {Backup} (attempt {Attempt}); the server as it was is {PreRestore}", marker.Backup, marker.Attempts, marker.PreRestore);
try
{
if (problems.Count > 0)
throw new InvalidOperationException(string.Join("; ", problems));
var backup = ServerBackup.Find(context.BackupsRoot, marker.Backup);
var report = new RestoreReport();
await Import(context, backup, report, token);
Media(context, backup, report);
await ProtectiveMerge.Apply(context.Database, Path.Combine(context.BackupsRoot, marker.PreRestore, "db"), report, token);
await Record(context, new RestoreRecord
{
Backup = backup.Id,
BackupCreatedAt = backup.CreatedAt,
PreRestore = marker.PreRestore,
RequestedBy = marker.RequestedBy,
RequestedAt = marker.RequestedAt,
Attempts = marker.Attempts,
Report = report
}, token);
RestoreRecord.Forget();
RestoreMarker.Clear(context.BackupsRoot);
logger.LogWarning("Restored {Backup}: {Documents} documents in {Collections} collections, {Media} media files brought back, {Tombstoned} accounts made since deleted",
backup.Id, report.Documents, report.Collections, report.MediaRestored, report.RootsTombstoned.Count + report.PersonasTombstoned.Count + report.GroupsTombstoned.Count);
return RestoreOutcome.Restored;
}
catch (Exception ex) when (ex is not OperationCanceledException)
{
marker.Error = ex.Message;
marker.Write(context.BackupsRoot);
logger.LogError(ex, "The restore of {Backup} failed (attempt {Attempt} of {Max})", marker.Backup, marker.Attempts, RestoreMarker.MaxAttempts);
return marker.Attempts >= RestoreMarker.MaxAttempts ? RestoreOutcome.GaveUp : RestoreOutcome.Failed;
}
}
static async Task<MaintenanceLock.Held> TakeLock(CancellationToken token)
{
var until = DateTime.UtcNow + LockWait;
while (true)
{
var held = await MaintenanceLock.Take("restore", token);
if (held != default || DateTime.UtcNow > until)
return held;
await Task.Delay(TimeSpan.FromSeconds(5), token);
}
}
// refused before anything changed: recorded for the administrator's page, and the server boots as it was
static async Task<RestoreOutcome> Abandon(BackupContext context, RestoreMarker marker, string why, ILogger logger, CancellationToken token)
{
logger.LogError("The restore of {Backup} was abandoned, nothing changed: {Why}", marker.Backup, why);
await Record(context, new RestoreRecord
{
Backup = marker.Backup,
RequestedBy = marker.RequestedBy,
RequestedAt = marker.RequestedAt,
Attempts = marker.Attempts,
Abandoned = true,
Error = why
}, token);
RestoreMarker.Clear(context.BackupsRoot);
return RestoreOutcome.Abandoned;
}
// in the database restored (a backup never holds these records: they outlive what they restore)
static async Task Record(BackupContext context, RestoreRecord record, CancellationToken token)
{
record.ID = ObjectId.GenerateNewId().ToString();
await context.Database.GetCollection<RestoreRecord>(nameof(RestoreRecord)).InsertOneAsync(record, cancellationToken: token);
}
// each collection dropped and imported as the backup holds it, its indexes made again; the others dropped
static async Task Import(BackupContext context, BackupInfo backup, RestoreReport report, CancellationToken token)
{
var database = context.Database;
var directory = Path.Combine(context.BackupsRoot, backup.Id, "db");
foreach (var entry in backup.Manifest.Collections)
{
await database.DropCollectionAsync(entry.Name, token);
await database.CreateCollectionAsync(entry.Name, cancellationToken: token);
report.Documents += await Insert(database.GetCollection<BsonDocument>(entry.Name), Read(Path.Combine(directory, entry.Name + ".jsonl.gz")), token);
var indexes = entry.Indexes.Select(BsonDocument.Parse).Where(i => i.GetValue("name", "").AsString != "_id_").ToList();
foreach (var index in indexes)
index.Remove("ns");
if (indexes.Count > 0)
await database.RunCommandAsync<BsonDocument>(new BsonDocument { { "createIndexes", entry.Name }, { "indexes", new BsonArray(indexes) } }, cancellationToken: token);
report.Collections++;
}
var held = backup.Manifest.Collections.Select(c => c.Name).ToHashSet(StringComparer.Ordinal);
foreach (var name in await (await database.ListCollectionNamesAsync(cancellationToken: token)).ToListAsync(token))
{
if (held.Contains(name) || ServerBackup.Excluded.ContainsKey(name) || name.StartsWith("system.", StringComparison.Ordinal))
continue;
await database.DropCollectionAsync(name, token);
report.CollectionsDropped++;
}
}
/// <summary>Documents inserted in batches, unordered and unvalidated, as they were: how many.</summary>
public static async Task<long> Insert(IMongoCollection<BsonDocument> collection, IEnumerable<BsonDocument> documents, CancellationToken token)
{
var count = 0L;
foreach (var batch in documents.Chunk(1000))
{
await collection.InsertManyAsync(batch, new InsertManyOptions { IsOrdered = false, BypassDocumentValidation = true }, token);
count += batch.Length;
}
return count;
}
/// <summary>A collection's documents in a backup, one by one; none when it has no file.</summary>
public static IEnumerable<BsonDocument> Read(string file)
{
if (!File.Exists(file))
yield break;
using var gzip = new GZipStream(File.OpenRead(file), CompressionMode.Decompress);
using var reader = new StreamReader(gzip);
while (reader.ReadLine() is { } line)
if (line.Length > 0)
yield return BsonDocument.Parse(line);
}
// the backup's media files the live directory lacks: linked from the backup, or moved back from the trash
static void Media(BackupContext context, BackupInfo backup, RestoreReport report)
{
var kept = Path.Combine(context.BackupsRoot, backup.Id, "media");
foreach (var relative in backup.Manifest.Media.List)
{
var live = ServerBackup.Inside(context.MediaRoot, relative);
if (File.Exists(live))
continue;
var fromBackup = ServerBackup.Inside(kept, relative);
var fromTrash = ServerBackup.Inside(context.TrashRoot, relative);
if (File.Exists(fromBackup))
HardLink.LinkOrCopy(fromBackup, live);
else if (File.Exists(fromTrash))
{
Directory.CreateDirectory(Path.GetDirectoryName(live)!);
File.Move(fromTrash, live);
}
else
{
report.MediaMissing++;
continue;
}
report.MediaRestored++;
}
}
}
}
@@ -0,0 +1,164 @@
using System.Collections.Concurrent;
using System.Security.Cryptography;
using System.Text.Json;
namespace PrivaPub.Infrastructure.Backup
{
// Backups leaving and arriving through the administrator's page, never through the client's memory:
// - a download is a link with a ticket in its path, good for ten minutes for one backup (a browser downloads it, and
// resumes it with Range), given for the administrator's password;
// - an upload arrives in pieces appended at the offset already received (a piece that would leave a gap or overlap is
// refused with what was received, so a broken upload resumes), into <backups>/.uploads, and is read into a backup
// once whole. One left for a day is deleted.
public class TransferStore(Backups backups)
{
public static readonly TimeSpan TicketLifetime = TimeSpan.FromMinutes(10);
public const long ChunkBytes = 32L * 1024 * 1024;
const string Uploads = ".uploads";
readonly ConcurrentDictionary<string, (string Backup, DateTime Expires)> _tickets = new();
readonly ConcurrentDictionary<string, SemaphoreSlim> _appending = new();
public (string Ticket, DateTime Expires) Ticket(string backup)
{
foreach (var (ticket, stale) in _tickets.Where(t => t.Value.Expires < DateTime.UtcNow).ToList())
_tickets.TryRemove(ticket, out _);
var expires = DateTime.UtcNow + TicketLifetime;
var token = Convert.ToHexStringLower(RandomNumberGenerator.GetBytes(32));
_tickets[token] = (backup, expires);
return (token, expires);
}
/// <summary>The backup a ticket is for, while it is good.</summary>
public string Redeem(string ticket) =>
ticket != default && _tickets.TryGetValue(ticket, out var held) && held.Expires > DateTime.UtcNow ? held.Backup : default;
string Folder => Path.Combine(backups.Root, Uploads);
string File(string id) => Path.Combine(Folder, id + ".tar");
static bool IsId(string id) => id is { Length: 32 } && id.All(char.IsAsciiHexDigitLower);
public string Start(string by)
{
ServerBackup.MakeDirectory(backups.Root);
ServerBackup.MakeDirectory(Folder);
Forget(DateTime.UtcNow.AddDays(-1));
var id = Convert.ToHexStringLower(RandomNumberGenerator.GetBytes(16));
using (OperatingSystem.IsWindows()
? new FileStream(File(id), FileMode.CreateNew, FileAccess.Write)
: new FileStream(File(id), new FileStreamOptions { Mode = FileMode.CreateNew, Access = FileAccess.Write,
UnixCreateMode = UnixFileMode.UserRead | UnixFileMode.UserWrite | UnixFileMode.GroupRead }))
{
}
System.IO.File.WriteAllText(Path.Combine(Folder, id + ".json"), JsonSerializer.Serialize(new { by, at = DateTime.UtcNow }));
return id;
}
/// <summary>Who started an upload (an administrator's name, or the root a persona's archive belongs to).</summary>
public string Owner(string id)
{
if (!IsId(id) || !System.IO.File.Exists(Path.Combine(Folder, id + ".json")))
return default;
try
{
return JsonDocument.Parse(System.IO.File.ReadAllText(Path.Combine(Folder, id + ".json"))).RootElement.GetProperty("by").GetString();
}
catch (JsonException)
{
return default;
}
}
/// <summary>A whole upload moved where it is read from: its new path, or null when there is no such upload.</summary>
public string Take(string id, string folder, string name)
{
if (!IsId(id) || !System.IO.File.Exists(File(id)))
return default;
ServerBackup.MakeDirectory(folder);
var path = Path.Combine(folder, name);
System.IO.File.Move(File(id), path, overwrite: true);
System.IO.File.Delete(Path.Combine(Folder, id + ".json"));
_appending.TryRemove(id, out _);
return path;
}
/// <summary>How much of an upload has arrived; -1 when there is no such upload.</summary>
public long Received(string id) => IsId(id) && System.IO.File.Exists(File(id)) ? new FileInfo(File(id)).Length : -1;
/// <summary>A piece appended where the upload stands: what has arrived since, and whether it was taken.</summary>
public async Task<(long Received, bool Taken)> Append(string id, long offset, Stream piece, CancellationToken token)
{
if (!IsId(id) || !System.IO.File.Exists(File(id)))
return (-1, false);
var gate = _appending.GetOrAdd(id, _ => new SemaphoreSlim(1, 1));
await gate.WaitAsync(token);
try
{
await using var file = new FileStream(File(id), FileMode.Open, FileAccess.Write);
if (file.Length != offset)
return (file.Length, false);
file.Seek(offset, SeekOrigin.Begin);
var buffer = new byte[81920];
var total = 0L;
int read;
while ((read = await piece.ReadAsync(buffer, token)) > 0)
{
total += read;
if (total > ChunkBytes)
{
file.SetLength(offset);
return (offset, false);
}
await file.WriteAsync(buffer.AsMemory(0, read), token);
}
await file.FlushAsync(token);
return (file.Length, true);
}
catch (IOException)
{
//the piece broke off: what arrived of it is dropped, so the next one starts where this one did
await using var file = new FileStream(File(id), FileMode.Open, FileAccess.Write);
file.SetLength(offset);
return (offset, false);
}
finally
{
gate.Release();
}
}
/// <summary>The whole upload read into a backup: the backup, or why not. The upload goes either way.</summary>
public async Task<(BackupInfo Backup, string Error)> Finish(string id, CancellationToken token)
{
if (!IsId(id) || !System.IO.File.Exists(File(id)))
return (default, "no such upload");
try
{
await using var tar = new FileStream(File(id), FileMode.Open, FileAccess.Read, FileShare.Read, 81920, FileOptions.SequentialScan);
return await BackupTar.Import(tar, backups.Root, token);
}
finally
{
Cancel(id);
}
}
public bool Cancel(string id)
{
if (!IsId(id) || !System.IO.File.Exists(File(id)))
return false;
System.IO.File.Delete(File(id));
System.IO.File.Delete(Path.Combine(Folder, id + ".json"));
_appending.TryRemove(id, out _);
return true;
}
// uploads left behind
void Forget(DateTime before)
{
foreach (var file in Directory.EnumerateFiles(Folder).Where(f => System.IO.File.GetLastWriteTimeUtc(f) < before))
System.IO.File.Delete(file);
}
}
}
@@ -3,6 +3,7 @@ using MongoDB.Entities;
using PrivaPub.ClientModels;
using PrivaPub.ClientModels.User;
using PrivaPub.ClientModels.User.Avatar;
using PrivaPub.Infrastructure.Backup;
using PrivaPub.Models.User;
using PrivaPub.Services;
using PrivaPub.Services.ClientToServer.Private;
@@ -22,8 +23,23 @@ namespace PrivaPub.Infrastructure.Cli
usage: PrivaPub admin promote|demote <root>
PrivaPub admin create-root <login> [--admin] the password is read from standard input
PrivaPub admin smoke <persona> prints "<login> <password>" for the deploy's signed-in check
PrivaPub admin media audit [--fix] media files against what holds them; --fix (as www-data) trashes
what nothing holds and gives today's pictures their rows
PrivaPub admin backup [--kind manual|pre-deploy] [--db-only]
backs the server up; --db-only lists the media without
linking them (the deploy's, which can't link www-data's files)
PrivaPub admin backups the backups kept, newest first
PrivaPub admin backup verify <id> whether a backup's files are what its manifest says
PrivaPub admin restore <id> restores a backup: the service stops within seconds and
restores it when systemd starts it again; everyone signs
in again, and what was made since is lost, except what
protects (blocks, deletions, ...)
PrivaPub admin restore --status the restore waiting, and the last one done
""";
/// <summary>Whether a command runs before migrations: backups are of the database as it was.</summary>
public static bool BeforeMigrations(string[] args) => args is ["backup", ..] or ["backups"] or ["restore", ..];
public static async Task<int> Run(string[] args, IServiceProvider services, TextReader input = default, TextWriter output = default)
{
input ??= Console.In;
@@ -36,12 +52,138 @@ namespace PrivaPub.Infrastructure.Cli
return await CreateRoot(services, login, input.ReadLine(), flags.Contains("--admin"), output);
case ["smoke", var persona]:
return await Smoke(services, persona, output);
case ["media", "audit", .. var flags] when flags.All(f => f == "--fix"):
return await AuditMedia(services, flags.Contains("--fix"), output);
case ["backup", "verify", var id]:
return await VerifyBackup(services, id, output);
case ["backup", .. var flags] when BackupFlags(flags, out var kind, out var dbOnly):
return await BackUp(services, kind, dbOnly, output);
case ["backups"]:
return ListBackups(services, output);
case ["restore", "--status"]:
return await RestoreStatus(services, output);
case ["restore", var id] when !id.StartsWith('-'):
return await Restore(services, id, output);
default:
Console.Error.WriteLine(Usage);
return 2;
}
}
static readonly string[] CommandKinds = ["manual", "pre-deploy"];
static bool BackupFlags(string[] flags, out string kind, out bool dbOnly)
{
kind = "manual";
dbOnly = false;
for (var i = 0; i < flags.Length; i++)
switch (flags[i])
{
case "--db-only":
dbOnly = true;
break;
case "--kind" when i + 1 < flags.Length && CommandKinds.Contains(flags[i + 1]):
kind = flags[++i];
break;
default:
return false;
}
return true;
}
static async Task<int> BackUp(IServiceProvider services, string kind, bool dbOnly, TextWriter output)
{
var (made, error) = await services.GetRequiredService<Backups>().Create(kind, dbOnly, CancellationToken.None);
if (made == default)
{
Console.Error.WriteLine(error);
return 1;
}
var manifest = made.Manifest;
output.WriteLine($"{made.Id}: {manifest.Collections.Count} collections, {manifest.Collections.Sum(c => c.Count)} documents, {made.Bytes / 1024} KiB");
output.WriteLine(dbOnly
? $"{manifest.Media.List.Count} media files listed, not linked"
: $"{manifest.Media.Files} media files, {manifest.Media.Bytes / 1024 / 1024} MiB, {manifest.Media.Missing} missing");
if (!manifest.Consistent)
output.WriteLine("not read at one instant: Mongo is not a replica set");
return 0;
}
static int ListBackups(IServiceProvider services, TextWriter output)
{
foreach (var backup in services.GetRequiredService<Backups>().List())
output.WriteLine($"{backup.Id}\t{backup.Kind}\t{backup.CreatedAt:u}\t{backup.Bytes / 1024} KiB\t{backup.Manifest?.Media.Files ?? 0} media{(backup.Manifest?.DbOnly == true ? " (listed)" : string.Empty)}");
return 0;
}
static async Task<int> VerifyBackup(IServiceProvider services, string id, TextWriter output)
{
var problems = await services.GetRequiredService<Backups>().Verify(id, CancellationToken.None);
foreach (var problem in problems)
output.WriteLine(problem);
if (problems.Count > 0)
return 1;
output.WriteLine($"{id}: whole");
return 0;
}
static async Task<int> Restore(IServiceProvider services, string id, TextWriter output)
{
var backups = services.GetRequiredService<Backups>();
var refused = await ServerRestore.Request(backups.Context(), id, "cli", CancellationToken.None);
if (refused != default)
{
Console.Error.WriteLine(refused);
return 1;
}
output.WriteLine($"{id} will be restored: the running service stops within seconds and restores it when it starts again");
output.WriteLine("(systemd starts it again; follow it with journalctl -u privapub, then PrivaPub admin restore --status)");
return 0;
}
static async Task<int> RestoreStatus(IServiceProvider services, TextWriter output)
{
var waiting = RestoreMarker.Read(services.GetRequiredService<Backups>().Root);
if (waiting != default)
output.WriteLine($"waiting: {waiting.Backup}, {waiting.State}, attempt {waiting.Attempts} of {RestoreMarker.MaxAttempts}, asked by {waiting.RequestedBy} at {waiting.RequestedAt:u}{(waiting.Error == default ? string.Empty : $": {waiting.Error}")}");
var last = await DB.Default.Find<RestoreRecord>().Sort(r => r.RestoredAt, Order.Descending).ExecuteFirstAsync();
if (last == default)
{
if (waiting == default)
output.WriteLine("no restore asked for, none done");
return 0;
}
if (last.Abandoned)
{
output.WriteLine($"last: {last.Backup} abandoned at {last.RestoredAt:u}, nothing changed: {last.Error}");
return 0;
}
var report = last.Report;
output.WriteLine($"last: {last.Backup} (made {last.BackupCreatedAt:u}) restored at {last.RestoredAt:u}, asked by {last.RequestedBy}; before it: {last.PreRestore}");
output.WriteLine($" {report.Documents} documents in {report.Collections} collections, {report.MediaRestored} media files brought back, {report.MediaMissing} missing");
output.WriteLine($" deleted again: {report.RootsDeleted} roots, {report.PersonasDeleted} personas, {report.GroupsDeleted} groups, {report.PostsDeleted} posts");
output.WriteLine($" made since, now deleted: roots {Names(report.RootsTombstoned)}, personas {Names(report.PersonasTombstoned)}, groups {Names(report.GroupsTombstoned)}; {report.PostsGone} posts gone");
output.WriteLine($" {report.ProtectiveKept} protective rows kept, {report.MediaTrashed} media trashed, {report.SessionsEnded} sessions ended");
return 0;
}
static string Names(List<string> names) => names.Count == 0 ? "none" : string.Join(", ", names);
static async Task<int> AuditMedia(IServiceProvider services, bool fix, TextWriter output)
{
var report = await Domain.Media.MediaAudit.Run(services.GetRequiredService<Domain.Media.IMediaService>(), fix, CancellationToken.None);
output.WriteLine($"{report.Files} files served, {report.Held} rows holding media");
output.WriteLine($"{report.Adopted} pictures personas show without a row{(fix ? ": given one" : string.Empty)}");
output.WriteLine($"{report.OfDeleted} media of deleted posts or personas{(fix ? ": trashed" : string.Empty)}");
output.WriteLine($"{report.MissingFiles} rows whose files are missing{(fix ? ": trashed" : string.Empty)}");
output.WriteLine($"{report.Unheld} files nothing holds{(fix ? ": trashed" : string.Empty)}");
foreach (var example in report.Examples)
output.WriteLine($" {example}");
if (!fix && report.Adopted + report.OfDeleted + report.MissingFiles + report.Unheld > 0)
output.WriteLine("run again with --fix, as www-data, to apply this; trashed files are deleted after a day");
return 0;
}
static async Task<int> Promote(bool promote, string userName, TextWriter output)
{
userName = userName.ToLowerInvariant();
@@ -4,7 +4,22 @@ namespace PrivaPub.Infrastructure.Data
{
public static class EntityMaps
{
static readonly Lock Gate = new();
static bool _warm;
// once per process, one map at a time: test hosts boot side by side
public static void Warm()
{
lock (Gate)
{
if (_warm)
return;
Map();
_warm = true;
}
}
static void Map()
{
var collection = typeof(DB).GetMethods()
.Single(m => m.Name == nameof(DB.Collection) && m.IsGenericMethodDefinition && m.GetParameters().Length == 0);
Loaded 100 of 141 files, more files were not shown because too many files have changed in this diff. Show more