Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4eda3da75d | ||
|
|
163f7e4e2b | ||
|
|
44a5895b0c | ||
|
|
3d7d406834 | ||
|
|
bdc8be4508 | ||
|
|
9ad96f560d | ||
|
|
bc5f2beaf7 | ||
|
|
fba57318fa | ||
|
|
12bb75809f | ||
|
|
37b12c5fee | ||
|
|
830385ea2b | ||
|
|
dc63fa57b8 | ||
|
|
5252b8f320 | ||
|
|
a4ced664cf | ||
|
|
8fecf7de2d | ||
|
|
3ca29603ed | ||
|
|
8e59145826 | ||
|
|
e4f9d0b61e | ||
|
|
65938bb2a0 | ||
|
|
bb680e8cb8 | ||
|
|
52d201eee9 | ||
|
|
402f9e0d75 | ||
|
|
8492f24064 |
No files matched your search
@@ -12,8 +12,7 @@ env:
|
||||
BACKUPS: /var/backups/privapub.thepra.dev
|
||||
LOCAL_URL: http://127.0.0.1:6970
|
||||
PUBLIC_URL: https://privapub.thepra.dev
|
||||
MONGO_URI: mongodb://127.0.0.1:27022
|
||||
MONGO_DB: PrivaPub
|
||||
SERVER_BACKUPS: /var/lib/privapub/backups
|
||||
|
||||
jobs:
|
||||
site:
|
||||
@@ -57,12 +56,24 @@ jobs:
|
||||
echo "SNAPSHOT=$BACKUPS/site-$STAMP" >> "$GITHUB_ENV"
|
||||
ls -1dt "$BACKUPS"/site-* 2>/dev/null | tail -n +4 | xargs -r rm -rf || true
|
||||
|
||||
- name: Dump the database
|
||||
# The server's own backup (PrivaPub admin backup, owner decision 2026-10-07) of the database as the live build left it:
|
||||
# the new build's command runs before its migrations. The media are listed, not linked (the runner may not link
|
||||
# www-data's files; the nightly backups hold them). Never the statistics salt, which must not outlive its day (owner
|
||||
# rule), nor the configuration's copy with its secrets. No deploy while a restore waits for its boot.
|
||||
- name: Back up the database
|
||||
run: |
|
||||
DUMP="$BACKUPS/mongo-$(date +%Y%m%d-%H%M%S).archive.gz"
|
||||
mongodump --quiet --uri "$MONGO_URI" --db "$MONGO_DB" --gzip --archive="$DUMP"
|
||||
echo "::notice::database dumped to $DUMP ($(du -h "$DUMP" | cut -f1))"
|
||||
ls -1t "$BACKUPS"/mongo-*.archive.gz 2>/dev/null | tail -n +8 | xargs -r rm -f || true
|
||||
[ ! -e "$SERVER_BACKUPS/restore.json" ] || { echo "::error::a restore is pending or running ($SERVER_BACKUPS/restore.json): deploy after it"; exit 1; }
|
||||
out=$(cd "$GITHUB_WORKSPACE/publish" && ASPNETCORE_ENVIRONMENT=Production ./PrivaPub admin backup --kind pre-deploy --db-only)
|
||||
echo "$out"
|
||||
id=$(echo "$out" | grep -oE '^[0-9]{8}-[0-9]{6}-pre-deploy' | tail -1)
|
||||
[ -d "$SERVER_BACKUPS/$id" ] || { echo "::error::the backup was not made"; exit 1; }
|
||||
(cd "$GITHUB_WORKSPACE/publish" && ASPNETCORE_ENVIRONMENT=Production ./PrivaPub admin backup verify "$id")
|
||||
if [ -e "$SERVER_BACKUPS/$id/db/InteractionSalt.jsonl.gz" ]; then
|
||||
echo "::error::the backup holds the statistics salt"
|
||||
exit 1
|
||||
fi
|
||||
echo "BACKUP_ID=$id" >> "$GITHUB_ENV"
|
||||
echo "::notice::database backed up as $id"
|
||||
|
||||
- name: Stop, sync, start
|
||||
run: |
|
||||
|
||||
@@ -257,6 +257,7 @@ Generated_Code/
|
||||
# because we have git ;-)
|
||||
_UpgradeReport_Files/
|
||||
Backup*/
|
||||
!PrivaPub/Infrastructure/Backup/
|
||||
UpgradeLog*.XML
|
||||
UpgradeLog*.htm
|
||||
ServiceFabricBackup/
|
||||
@@ -401,6 +402,9 @@ FodyWeavers.xsd
|
||||
|
||||
PrivaPub/media-store/
|
||||
PrivaPub/media-store-proxy/
|
||||
PrivaPub/media-store-trash/
|
||||
PrivaPub/media-store-incoming/
|
||||
PrivaPub/media-store-backups/
|
||||
tools/pasture/.publish/
|
||||
tools/pasture/.flood/
|
||||
.claude/worktrees/
|
||||
|
||||
@@ -226,7 +226,9 @@ group www-data and reaches the private mongod; `sudo -u www-data` works too.
|
||||
marks (`Domain/Statuses/ConversationStates.cs`; writing in a conversation reads it).
|
||||
10. **Nothing slow happens inside a request.** Deliveries and inbox processing are `Job`s (`Infrastructure/Jobs`):
|
||||
leased, retried on Mastodon's curve, at most two per host, paused per host by `RemoteInstance`. The inbox answers
|
||||
202 once it has verified and queued; a handler must be idempotent (unique `ObjectURI`, job `DedupeKey`).
|
||||
202 once it has verified and queued; a handler must be idempotent (unique `ObjectURI`, job `DedupeKey`). A lease
|
||||
(2 minutes, stamped with its own owner) is renewed every third of its length while the handler runs, so a long job
|
||||
runs once; a lease found taken cancels the handler, and `Finish` only counts for the lease it was given.
|
||||
11. **Every number about posts and users goes through `Domain/Privacy/Counted`** (owner decision 2026-10-04: one
|
||||
answer everywhere): a persona's `statuses_count`, its outbox `totalItems`, NodeInfo `localPosts` and the instance
|
||||
`status_count` all count Mastodon's way (not deleted, not a DM, boosts included, circle and located posts too);
|
||||
@@ -294,18 +296,72 @@ group www-data and reaches the private mongod; `sudo -u www-data` works too.
|
||||
|
||||
1. **No upload keeps its metadata.** Images are re-encoded by libvips with `keep=none`; audio and video are remuxed with
|
||||
`-map_metadata -1`. `MediaProcessingTests` checks EXIF and XMP are gone.
|
||||
2. Files live under `Media:Root` (`/var/lib/privapub/media`), never in the published directory; the proxy cache is the
|
||||
sibling `media-proxy`, which `/media/files` does not serve.
|
||||
3. **A client never contacts a remote server for media:** every remote URL the API returns goes through
|
||||
2. **An image is checked before it is decoded.**
|
||||
- **Loaders:** only libvips' JPEG, PNG, GIF, WebP and HEIF loaders ever run on an upload (`MediaService`'s static
|
||||
constructor blocks every other loader), so an SVG, PDF or TIFF claiming another type never loads.
|
||||
- **Size:** the header alone tells width × height (× frames for a GIF), refused above `Media:MaxPixels` (40 MP) or
|
||||
`Media:MaxFrames`.
|
||||
- **Then:** the image is shrunk on load to `MaxImageSide`, turned by its orientation, and its colours brought into
|
||||
sRGB (`thumbnail`).
|
||||
- **GIFs:** an animated GIF becomes a looping silent H.264 mp4 typed `gifv`, as on Mastodon (ffmpeg with libx264;
|
||||
`PostMedia.Kind` says so), and a still GIF is an image.
|
||||
- **Types:** HEIC and HEIF are not accepted, since the bundled libvips has no HEVC decoder.
|
||||
- **Load:** processing runs `Media:Concurrency` at a time, and uploads have their own rate limit (`uploads`, per
|
||||
credential).
|
||||
3. **Audio and video are processed off the request when the client allows it.**
|
||||
- **v2:** `POST /api/v2/media` stores them as sent in `media-incoming` (beside the root, never served) and answers
|
||||
202 with no url, while a `ProcessMedia` job (one at a time, its lease renewed) does the work. Until then
|
||||
`GET /api/v1/media/:id` answers 206, and 422 with the reason if it failed. Media still processing can't be posted.
|
||||
- **v1:** processes before answering.
|
||||
- **ffmpeg** reads only the stored file: `-protocol_whitelist file`, and the input format is forced from the probe.
|
||||
Data and subtitle tracks are dropped (`-dn -sn`, which iPhone MOVs need).
|
||||
- **Remux or transcode:** a video browsers play as it is (H.264, VP8, VP9 or AV1, within `Media:MaxVideoPixels` and
|
||||
`MaxFrameRate`) is remuxed; anything else is transcoded to H.264 that fits, as Mastodon does.
|
||||
- **Limits:** longer than `Media:MaxSeconds` is refused.
|
||||
- **Saving:** nothing is saved until everything succeeded; outputs move into place rather than being read into
|
||||
memory, and every temporary file goes. FLAC is served as `audio/flac`, and the unit runs with `PrivateTmp`.
|
||||
4. Files live under `Media:Root` (`/var/lib/privapub/media`), never in the published directory; the proxy cache is the
|
||||
sibling `media-proxy` and the trash the sibling `media-trash`, neither of which `/media/files` serves.
|
||||
5. **A file lives exactly as long as something holds it.** Every upload is a `MediaAttachment` row, profile pictures
|
||||
too (`Kind` avatar or header, `ProfileOfAvatarId`). Deleting a post, an edit leaving media out, a replaced picture,
|
||||
a dropped scheduled post, a removed root, and an upload never posted for a day each trash theirs
|
||||
(`IMediaService.Trash`):
|
||||
- the row gets `TrashedAt` in one conditional update, so a row attached meanwhile is left alone;
|
||||
- its files move into `media-trash` at once, so `/media/files` stops serving them;
|
||||
- `MediaJanitor` deletes them a day later (`TrashGrace`).
|
||||
|
||||
What a login's media take (every persona's, trashed ones no longer) is `Counted.MediaOfRoot`, shown to the login at
|
||||
`/clientapi/user/storage`; `Media:QuotaBytesPerRoot` (0, none, by default) refuses an upload over it with 422; the
|
||||
administrator's statistics overview has the totals, the proxy cache and the trash.
|
||||
|
||||
Nothing is deleted because it looks unused. `PrivaPub admin media audit [--fix]` compares disk and database: with
|
||||
`--fix` (as www-data) it gives pictures shown from before their rows a row, and trashes media of deleted posts or
|
||||
personas, rows whose files are missing, and files nothing holds.
|
||||
6. **A client never contacts a remote server for media:** every remote URL the API returns goes through
|
||||
`IMediaProxy.Wrap`, an HMAC-signed `/media/proxy/` URL fetched by `IFederationHttp.GetMedia`.
|
||||
4. **The proxy serves three ways:**
|
||||
- **Cached:** a file already cached is served from disk, ranges included.
|
||||
- **Downloaded:** a request without a `Range` is downloaded whole, up to `Media:MaxProxiedBytes`, then cached.
|
||||
7. **The proxy serves three ways:**
|
||||
- **Cached:** a file already cached is served from disk, ranges included. It is opened before the answer, so a trim
|
||||
that deletes it meanwhile breaks nothing.
|
||||
- **Downloaded:** a request without a `Range` is downloaded whole, up to `Media:MaxProxiedBytes`, then cached:
|
||||
- once for everyone asking for it at the same time;
|
||||
- streamed into a `.part` file beside its place and renamed there, never held in memory;
|
||||
- with its host in its `.type` sidecar, so that blocking a server can purge it.
|
||||
- **Streamed:** a ranged request, or anything too big to cache, is streamed from the origin with the range passed on,
|
||||
and never cached. That is how remote video plays.
|
||||
and never cached. That is how remote video plays. A file found too big is remembered for an hour, so it isn't
|
||||
fetched twice; one that failed is remembered for five minutes.
|
||||
|
||||
Also:
|
||||
- The cache's size is counted as it grows and trimmed as soon as it passes `ProxyCacheBytes`; the janitor
|
||||
recounts hourly.
|
||||
- Browsers may cache only a success.
|
||||
- Nothing of a suspended server, or of one whose media are rejected, is proxied (avatars, emoji, covers and link
|
||||
cards included, since they all go through it), and blocking one purges its cache.
|
||||
- The `proxy` rate limit counts per client address.
|
||||
|
||||
nginx has a `/media/proxy/` location with `proxy_buffering off` and a 600 s read timeout for those streams.
|
||||
5. **Remote video and audio become one playable attachment** in the Mastodon API (`MastodonMapper.Playable`): the best MP4
|
||||
8. **A focal point is two finite numbers** within -1..1 (`FocalPoint.Parse`); anything else is ignored. A stored NaN made
|
||||
every status, timeline and Note holding its post fail to serialise; migration `_016` removed the ones stored before.
|
||||
9. **Remote video and audio become one playable attachment** in the Mastodon API (`MastodonMapper.Playable`): the best MP4
|
||||
up to 720p that carries both sound and picture, including PeerTube's fragmented files inside an HLS entry. HLS
|
||||
playlists themselves are not rewritten.
|
||||
|
||||
@@ -424,7 +480,89 @@ group www-data and reaches the private mongod; `sudo -u www-data` works too.
|
||||
`_NNN_` order, each runs once), then `Indexes.Create()`. A new entity needs nothing; a new unique index needs a
|
||||
dedupe migration before it.
|
||||
- Production runs its **own mongod** on 127.0.0.1:27022 (unit `privapub-mongod`, no auth, data in
|
||||
`/var/lib/privapub/mongo`). The box's shared mongod needs credentials nobody here has.
|
||||
`/var/lib/privapub/mongo`). The box's shared mongod needs credentials nobody here has. It is a one-member replica set
|
||||
(`--replSet rs0`, a 990 MB oplog; owner decision 2026-10-07), so a backup reads every collection at one instant;
|
||||
`setup.sh` converts it once (PrivaPub stopped, mongod restarted, `rs.initiate`), and every connection string says
|
||||
`directConnection=true`, which also works against a standalone. The pasture's mongo is one too.
|
||||
- **Backups** (`Infrastructure/Backup/`; owner decisions 2026-10-07: the whole server, plain files protected by their
|
||||
permissions, run from the CLI, nightly and from the administrator's page). Each backup is a directory
|
||||
`<yyyyMMdd-HHmmss>-<kind>` under `Backups:Root` (`/var/lib/privapub/backups`, www-data 2770, files 0640), written as
|
||||
`.partial` and renamed once whole:
|
||||
- `manifest.json` (`ArchiveManifest`): host, build, newest migration, whether it was read at one instant, each
|
||||
collection's count, size, sha256 and indexes, what was left out and why, and the media list;
|
||||
- `db/<collection>.jsonl.gz`: each document as one line of canonical Extended JSON, read raw, so every BSON type comes
|
||||
back byte for byte; on a replica set every collection is read in one snapshot session
|
||||
(`minSnapshotHistoryWindowInSeconds` is raised to an hour only while it reads);
|
||||
- `media/`: hard links to the files of untrashed `MediaAttachment` rows (no disk spent; a deleted file stays until the
|
||||
backup rotates out), copies where a link can't be made. `--db-only` lists them instead.
|
||||
|
||||
**Never in a backup** (`ServerBackup.Excluded`): `InteractionSalt` (owner rule), `Job` and `Delivery` (work in flight),
|
||||
`EmailRecovery`, `openiddict.tokens` and `.authorizations` (sessions), `MaintenanceLock`, and `AppConfiguration`
|
||||
(its SMTP password; it is made again from appsettings at boot). One backup or restore runs at a time
|
||||
(`MaintenanceLock`, a heartbeat document; a holder silent for 2 minutes is taken over), and the media janitor purges
|
||||
nothing meanwhile. `BackupScheduler` backs up at `Backups:NightlyAt` (03:30 UTC), or at once when it missed the night;
|
||||
rotation keeps 7 daily and 4 weekly nightly backups, 3 pre-deploy, 3 pre-restore, and manual and uploaded ones until
|
||||
deleted. CLI: `PrivaPub admin backup [--kind manual|pre-deploy] [--db-only]`, `admin backups`, `admin backup verify
|
||||
<id>`; these run before migrations, so the deploy's backup is of the database as the live build left it.
|
||||
- **Restores** (`ServerRestore`, `ProtectiveMerge`; owner decision 2026-10-07). Asking (`PrivaPub admin restore <id>`, or
|
||||
the administrator's page) checks the backup (same host, a format and newest migration this build reads, every hash)
|
||||
and writes `restore.json` in the backups' root; the running service sees it within seconds (`RestoreWatcher`) and
|
||||
stops, and systemd starts it again. `MaintenanceGate` (in `Program`, right after the build, before migrations, indexes
|
||||
and hosted services) carries it out:
|
||||
1. a pre-restore backup P, taken once (a retry reuses it, never a half-restored database);
|
||||
2. every collection the backup holds dropped and imported raw with its indexes, every other one dropped, except what a
|
||||
backup never holds, which stays as it is;
|
||||
3. media files the live directory lacks linked back from the backup (or moved from the trash); a restore deletes no file;
|
||||
4. **the protective merge from P: a restore never undoes a protective act.** Followers and follows are P's; blocks,
|
||||
mutes, domain blocks, being blocked, reserved names, `DeletedObject`, reports, filters and OAuth applications are
|
||||
the union, P's row winning; roots, personas, groups, posts and remote accounts deleted since are deleted again, with
|
||||
what their deletion takes away; P's moderation of a persona and a root's password, e-mail, ban and policies stay;
|
||||
roots, personas and groups made since become tombstones (deleted, names kept), local posts made since answer 410,
|
||||
media made or trashed since go to the trash;
|
||||
5. every session ends (a new `SessionStamp` for each root, persona tokens deleted), circuits close, and a
|
||||
`RestoreRecord` (never in a backup) tells what happened (`admin restore --status`).
|
||||
|
||||
Each attempt redoes everything. Refused before anything changed, it is abandoned and recorded, and the server boots as
|
||||
it was; failed midway, the process exits 1 and the next start tries again; after 3 failures it exits 75, which the
|
||||
unit's `RestartPreventExitStatus` leaves down for someone to look. While `restore.json` exists, commands exit 75 (but
|
||||
`admin restore --status`) and the deploy refuses to run. For `RestoreRecord.FollowersGrace` (14 days) after a restore,
|
||||
FEP-8fcf rests: no `Collection-Synchronization` header goes out, and a follow only the remote remembers is adopted,
|
||||
not undone.
|
||||
- **The administrator's page** (`BackupController`, `/clientapi/admin/backups`; owner decision 2026-10-07, which approved
|
||||
these endpoints in production): the list (with what runs, the restore waiting, the last restore's report, logins
|
||||
deleted since each backup), back up now (202; the page polls), delete, and:
|
||||
- **download** for the password: a ticket good for ten minutes in the link's path (`/download/{ticket}`, anonymous, so a
|
||||
browser saves it to disk), the backup as one tar (`BackupTar`: its exact length known first, written from the
|
||||
files, `Range` honoured);
|
||||
- **upload** in pieces of at most 32 MB (`TransferStore`): each `PUT ?offset=` must start where the upload stands, else
|
||||
409 with what was received, so a broken upload resumes; `finish` reads it into a backup (`kind` uploaded), refusing
|
||||
anything but plain files under one backup's folder, a path leaving it, or more than the disk holds;
|
||||
- **restore** for the password and the server's host typed out.
|
||||
|
||||
Every call checks the administrator against the database, not only the token. nginx streams downloads for an hour and
|
||||
takes upload pieces unbuffered (`deploy/nginx`, applied by `setup.sh`).
|
||||
- **A persona's archive** (`Domain/Portability/`, `PersonaArchiveController` at `/clientapi/persona/{avatarId}/archive`;
|
||||
owner decision 2026-10-07), for the persona's own root only, never a banned one. State per persona in `PersonaArchive`
|
||||
(never in a backup), files in `<backups>/.personas/<avatarId>/`, a week.
|
||||
- **Export** (`ExportArchiveJob`): Mastodon's account-archive layout, so other servers' importers read it: `actor.json`
|
||||
(its public key only), `outbox.json` (its own posts as Creates, addressed as delivered, but circles' and located
|
||||
posts; its boosts as Announces), `media_attachments/files/` (what the attachments' urls name), avatar and header,
|
||||
`likes.json`, `bookmarks.json` and Mastodon's CSV files; then `privapub/`: filters, followed hashtags, the
|
||||
notification policy, pins, scheduled and located posts, `archive.json`. Nothing of its root, its siblings, its keys,
|
||||
anyone's token. Downloaded through a ten-minute ticket in the link's path.
|
||||
- **Import** (`ImportArchiveJob`, the parts the root picks, with progress, stoppable): uploaded in pieces like a backup,
|
||||
read through `SafeArchive`, which refuses links, paths leaving it, names twice, too many or too large files, a file
|
||||
compressed over 100:1, and streams the outbox an item at a time (none over 1 MB). Only the archive actor's own
|
||||
Creates become posts; boosts, direct and circle posts are counted. **Imported posts are delivered to no one**, put in
|
||||
no home and notify nobody (an exception next to located posts), yet show on the profile, the outbox, hashtags and
|
||||
search. Back home (same actor) a post keeps its id and address, and one already here, even deleted, or tombstoned,
|
||||
stays as it is; from another actor it gets an id of its date, its address here and `ImportedFromURI` (unique per
|
||||
persona), so importing twice changes nothing. Replies and self-quotes inside the archive point at their copies; polls
|
||||
come closed without votes; mentions stay links; HTML is sanitized; media go through `MediaService` and the quota.
|
||||
The other parts go through the existing services: follows asked again (never on a blocked server), blocks, mutes,
|
||||
blocked servers, lists (members only if followed), bookmarks (by signed fetch), filters, followed hashtags, the
|
||||
notification policy, pins (no `Add` delivered), the profile (announced as Settings would). Located and scheduled
|
||||
posts (PrivaPub's archives) and likes (each tells its author) only when asked. Followers are never imported.
|
||||
|
||||
## Code style
|
||||
|
||||
@@ -449,6 +587,8 @@ group www-data and reaches the private mongod; `sudo -u www-data` works too.
|
||||
`PrivaPub.Tests` (xUnit v3). Unit tests need nothing; tests marked `Category=Integration` need a mongod and skip
|
||||
without `PRIVAPUB_TEST_MONGOD=1`. CI (`build.yml` and `deploy.yml`) runs all of them through
|
||||
`tools/ci/with-test-mongod.sh`, which starts a throwaway mongod on a random localhost port and deletes it afterwards.
|
||||
Like production's, it is a one-member replica set (`rs0`, reached with `directConnection=true`), so backups' snapshot
|
||||
reads are what the tests exercise (`TopologyTests`).
|
||||
The box's own mongods are production, so `MongoFixture` refuses port 27022, a data directory under `/var/lib/privapub`,
|
||||
and in CI anything but the wrapper's mongod. With `PRIVAPUB_TEST_REQUIRE_MONGOD=1`, which the wrapper sets, a missing
|
||||
mongod fails the run instead of silently skipping half the tests.
|
||||
@@ -779,7 +919,9 @@ the owner's GoToSocial account.**
|
||||
## Deploy
|
||||
|
||||
- **CI/CD:** push to `master` runs `build.yml` (build + tests) on the instance-wide `build` runner. A `v*` tag runs
|
||||
`deploy.yml`: tests, self-contained linux-x64 publish, snapshot and `mongodump` to `/var/backups/privapub.thepra.dev`,
|
||||
`deploy.yml`: tests, self-contained linux-x64 publish, a snapshot of the site to `/var/backups/privapub.thepra.dev`,
|
||||
the server's own pre-deploy backup (`PrivaPub admin backup --kind pre-deploy --db-only`, run from the new build before
|
||||
its migrations, verified, and checked to hold no statistics salt; no deploy while `restore.json` waits),
|
||||
stop → rsync → start, a `127.0.0.1:6970/build.json` health loop with rollback, then public checks (actor, NodeInfo,
|
||||
Swagger 404, inbox junk 400, unsigned 401) and `tools/smoke/mastodon-api.sh` (app registration, client credentials,
|
||||
discovery, instance, public timeline). Then it checks that what production should be running is running:
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
using PrivaPub.ClientModels.Resources;
|
||||
|
||||
using System.ComponentModel.DataAnnotations;
|
||||
|
||||
namespace PrivaPub.ClientModels.Admin
|
||||
{
|
||||
// the server's backups as the administrator's page shows them, with what is running and the last restore
|
||||
public class ViewBackups
|
||||
{
|
||||
public List<ViewBackup> Backups { get; set; } = [];
|
||||
public string Running { get; set; }//"backup" or "restore" while one runs
|
||||
public ViewRestoreWaiting Waiting { get; set; }
|
||||
public ViewRestore LastRestore { get; set; }
|
||||
public string Host { get; set; }//what a restore asks to be typed
|
||||
public long FreeBytes { get; set; }
|
||||
}
|
||||
|
||||
public class ViewBackup
|
||||
{
|
||||
public string Id { get; set; }
|
||||
public string Kind { get; set; }//nightly, manual, pre-deploy, pre-restore, uploaded
|
||||
public DateTime CreatedAt { get; set; }
|
||||
public long Bytes { get; set; }//the database's files
|
||||
public long MediaBytes { get; set; }
|
||||
public int MediaFiles { get; set; }
|
||||
public int MediaMissing { get; set; }
|
||||
public bool DbOnly { get; set; }
|
||||
public bool Consistent { get; set; }
|
||||
public int Collections { get; set; }
|
||||
public long Documents { get; set; }
|
||||
public string AppCommit { get; set; }
|
||||
public int MigrationNumber { get; set; }
|
||||
public int RootsDeletedSince { get; set; }//logins deleted after it, which a restore deletes again
|
||||
public string NotRestorable { get; set; }//why not, from its manifest; hashes are checked when asked
|
||||
}
|
||||
|
||||
public class ViewRestoreWaiting
|
||||
{
|
||||
public string Backup { get; set; }
|
||||
public string State { get; set; }
|
||||
public int Attempts { get; set; }
|
||||
public string RequestedBy { get; set; }
|
||||
public DateTime RequestedAt { get; set; }
|
||||
public string Error { get; set; }
|
||||
}
|
||||
|
||||
public class ViewRestore
|
||||
{
|
||||
public string Backup { get; set; }
|
||||
public DateTime BackupCreatedAt { get; set; }
|
||||
public string PreRestore { get; set; }
|
||||
public string RequestedBy { get; set; }
|
||||
public DateTime RestoredAt { get; set; }
|
||||
public bool Abandoned { get; set; }
|
||||
public string Error { get; set; }
|
||||
public long Documents { get; set; }
|
||||
public int Collections { get; set; }
|
||||
public int MediaRestored { get; set; }
|
||||
public int MediaMissing { get; set; }
|
||||
public int RootsDeleted { get; set; }
|
||||
public int PersonasDeleted { get; set; }
|
||||
public int GroupsDeleted { get; set; }
|
||||
public int PostsDeleted { get; set; }
|
||||
public List<string> RootsTombstoned { get; set; } = [];
|
||||
public List<string> PersonasTombstoned { get; set; } = [];
|
||||
public List<string> GroupsTombstoned { get; set; } = [];
|
||||
public int PostsGone { get; set; }
|
||||
public int MediaTrashed { get; set; }
|
||||
public int ProtectiveKept { get; set; }
|
||||
public int SessionsEnded { get; set; }
|
||||
}
|
||||
|
||||
// a backup downloaded or restored: the administrator's password again, and for a restore the server's host typed out
|
||||
public class BackupPasswordForm
|
||||
{
|
||||
[Required(ErrorMessageResourceName = "Required", ErrorMessageResourceType = typeof(ErrorsResource)),
|
||||
StringLength(200, ErrorMessageResourceName = "MaxLengthString", ErrorMessageResourceType = typeof(ErrorsResource))]
|
||||
public string Password { get; set; }
|
||||
}
|
||||
|
||||
public class RestoreBackupForm : BackupPasswordForm
|
||||
{
|
||||
[Required(ErrorMessageResourceName = "Required", ErrorMessageResourceType = typeof(ErrorsResource)),
|
||||
StringLength(253, ErrorMessageResourceName = "MaxLengthString", ErrorMessageResourceType = typeof(ErrorsResource))]
|
||||
public string Host { get; set; }
|
||||
}
|
||||
|
||||
// a download link good for a while, for one backup
|
||||
public class ViewBackupTicket
|
||||
{
|
||||
public string Url { get; set; }
|
||||
public DateTime ExpiresAt { get; set; }
|
||||
public long Bytes { get; set; }
|
||||
}
|
||||
|
||||
// a backup being uploaded in pieces: how much has arrived
|
||||
public class ViewBackupUpload
|
||||
{
|
||||
public string Id { get; set; }
|
||||
public long Received { get; set; }
|
||||
public long ChunkBytes { get; set; }//the largest piece accepted
|
||||
}
|
||||
}
|
||||
@@ -12,6 +12,16 @@ namespace PrivaPub.ClientModels.Admin
|
||||
public ViewHostCounts Hosts { get; set; } = new();
|
||||
public int Accounts { get; set; }
|
||||
public ViewLedgerCounts Ledger { get; set; } = new();
|
||||
public ViewMediaTotals Media { get; set; } = new();
|
||||
}
|
||||
|
||||
// what media take on the server: the uploads and pictures kept, the remote media cache, the trash waiting its grace
|
||||
public class ViewMediaTotals
|
||||
{
|
||||
public long MediaBytes { get; set; }
|
||||
public long MediaFiles { get; set; }
|
||||
public long ProxyCacheBytes { get; set; }
|
||||
public long TrashBytes { get; set; }
|
||||
}
|
||||
|
||||
public class ViewLatency
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
namespace PrivaPub.ClientModels.User
|
||||
{
|
||||
// A persona's archive (owner decision 2026-10-07): made for it to take away, in Mastodon's account-archive layout, and
|
||||
// restored into it. States: none, queued, running, ready (export) or done (import), failed, stopped.
|
||||
public class ViewPersonaArchive
|
||||
{
|
||||
public string ExportState { get; set; }
|
||||
public DateTime? ExportAskedAt { get; set; }
|
||||
public DateTime? ExportReadyAt { get; set; }
|
||||
public DateTime? ExportExpiresAt { get; set; }
|
||||
public long ExportBytes { get; set; }
|
||||
public string ExportError { get; set; }
|
||||
|
||||
public string ImportState { get; set; }
|
||||
public DateTime? ImportAskedAt { get; set; }
|
||||
public DateTime? ImportEndedAt { get; set; }
|
||||
public List<string> ImportParts { get; set; } = [];
|
||||
public long ImportDone { get; set; }
|
||||
public long ImportTotal { get; set; }
|
||||
public Dictionary<string, int> ImportCounts { get; set; } = [];//what was imported and what skipped, by why
|
||||
public string ImportError { get; set; }
|
||||
public string ImportFrom { get; set; }//the archive's actor
|
||||
public bool ImportSameActor { get; set; }
|
||||
}
|
||||
|
||||
// the parts of an archive to import: posts (with their media), profile, follows, blocks, mutes, domainblocks, lists,
|
||||
// bookmarks, filters, tags, notificationpolicy, pins; and only when asked, located, scheduled, likes (which tell their
|
||||
// authors)
|
||||
public class ImportPersonaArchiveForm
|
||||
{
|
||||
public string UploadId { get; set; }
|
||||
public List<string> Parts { get; set; } = [];
|
||||
}
|
||||
|
||||
public class ViewArchiveTicket
|
||||
{
|
||||
public string Url { get; set; }
|
||||
public DateTime ExpiresAt { get; set; }
|
||||
public long Bytes { get; set; }
|
||||
}
|
||||
|
||||
public class ViewArchiveUpload
|
||||
{
|
||||
public string Id { get; set; }
|
||||
public long Received { get; set; }
|
||||
public long ChunkBytes { get; set; }
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
namespace PrivaPub.ClientModels.User
|
||||
{
|
||||
// /clientapi/user/storage: what the login's media take (every persona's uploads and pictures), and its quota when the
|
||||
// server sets one
|
||||
public class ViewStorage
|
||||
{
|
||||
public long MediaBytes { get; set; }
|
||||
public long MediaFiles { get; set; }
|
||||
public long? QuotaBytes { get; set; }
|
||||
}
|
||||
}
|
||||
@@ -1,3 +1,4 @@
|
||||
using PrivaPub.Models.Federation;
|
||||
using Microsoft.AspNetCore.Http;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
|
||||
@@ -46,7 +47,7 @@ namespace PrivaPub.Tests.Domain
|
||||
var (_, alice) = await _harness.Persona("alice");
|
||||
var (_, mallory) = await _harness.Persona("mallory");
|
||||
|
||||
var upload = await _harness.Media.Upload(alice, Upload(Png(300, 200), "image/png"), "a blue square", "0.25,-0.5", token);
|
||||
var upload = await _harness.Media.Upload(alice, Upload(Png(300, 200), "image/png"), "a blue square", "0.25,-0.5", false, token);
|
||||
Assert.True(upload.Ok);
|
||||
Assert.True(File.Exists(Path.Combine(_harness.Media.Root, upload.Attachment.FilePath)));
|
||||
|
||||
@@ -73,7 +74,7 @@ namespace PrivaPub.Tests.Domain
|
||||
{
|
||||
var (_, alice) = await _harness.Persona("alice");
|
||||
|
||||
var outcome = await _harness.Media.Upload(alice, Upload(System.Text.Encoding.UTF8.GetBytes("<svg/>"), "image/svg+xml"), default, default, TestContext.Current.CancellationToken);
|
||||
var outcome = await _harness.Media.Upload(alice, Upload(System.Text.Encoding.UTF8.GetBytes("<svg/>"), "image/svg+xml"), default, default, false, TestContext.Current.CancellationToken);
|
||||
|
||||
Assert.False(outcome.Ok);
|
||||
Assert.Equal(422, outcome.Status);
|
||||
@@ -98,5 +99,62 @@ namespace PrivaPub.Tests.Domain
|
||||
Assert.StartsWith(_harness.Media.ProxyRoot, file);
|
||||
Assert.Null(tampered);
|
||||
}
|
||||
|
||||
// a login over its quota uploads nothing more, whichever persona tries
|
||||
[Fact]
|
||||
public async Task A_login_over_its_quota_uploads_nothing_more()
|
||||
{
|
||||
var token = TestContext.Current.CancellationToken;
|
||||
var (root, alice) = await _harness.Persona("alice");
|
||||
var (_, sibling) = await _harness.Persona("sibling", root);
|
||||
var quota = new MediaService(new StaticOptions<MediaOptions>(new MediaOptions { Root = _harness.Media.Root, QuotaBytesPerRoot = 3000 }),
|
||||
_harness.Local, default, Microsoft.Extensions.Logging.Abstractions.NullLogger<MediaService>.Instance);
|
||||
// noise: its PNG is about as big as its pixels, so two of them are over the quota and one is not
|
||||
static byte[] Noise()
|
||||
{
|
||||
var pixels = new byte[30 * 20 * 3];
|
||||
Random.Shared.NextBytes(pixels);
|
||||
using var image = NetVips.Image.NewFromMemory(pixels, 30, 20, 3, NetVips.Enums.BandFormat.Uchar);
|
||||
return image.WriteToBuffer(".png");
|
||||
}
|
||||
|
||||
Assert.True((await quota.Upload(alice, Upload(Noise(), "image/png"), default, default, false, token)).Ok);
|
||||
var full = await quota.Upload(sibling, Upload(Noise(), "image/png"), default, default, false, token);
|
||||
|
||||
Assert.False(full.Ok);
|
||||
Assert.Contains("storage is full", full.Error);
|
||||
}
|
||||
|
||||
// nothing of a suspended server, or of one whose media are rejected, is proxied; blocking one purges what was cached
|
||||
[Fact]
|
||||
public async Task A_blocked_servers_media_are_not_proxied_and_their_cache_goes()
|
||||
{
|
||||
var token = TestContext.Current.CancellationToken;
|
||||
var path = $"/files/{Guid.NewGuid():N}.png";
|
||||
_harness.Peer.ServeFile(path, Png(10, 10), "image/png");
|
||||
var remote = _harness.Peer.A + path;
|
||||
var host = new Uri(remote).Host;
|
||||
var blocks = new Blocks();
|
||||
var proxy = new MediaProxy(_harness.Local, Peer.Http(), _harness.Media, new StaticOptions<MediaOptions>(new MediaOptions()), blocks);
|
||||
Assert.Equal(ProxyOutcome.Cached, (await proxy.Download(remote, token)).Outcome);
|
||||
|
||||
blocks.Blocked[host] = new DomainBlock { Domain = host, Severity = DomainBlockSeverity.Silence, RejectMedia = true };
|
||||
Assert.True(proxy.Refuses(remote));
|
||||
Assert.True(proxy.Purge(host) >= 1);
|
||||
Assert.Equal(default, proxy.Cached(remote));
|
||||
|
||||
blocks.Blocked[host] = new DomainBlock { Domain = host, Severity = DomainBlockSeverity.Silence };
|
||||
Assert.False(proxy.Refuses(remote));
|
||||
blocks.Blocked[host] = new DomainBlock { Domain = host, Severity = DomainBlockSeverity.Suspend };
|
||||
Assert.True(proxy.Refuses(remote));
|
||||
}
|
||||
|
||||
sealed class Blocks : PrivaPub.Federation.Moderation.IDomainBlocks
|
||||
{
|
||||
public Dictionary<string, DomainBlock> Blocked { get; } = new();
|
||||
public DomainBlock Find(string host) => Blocked.GetValueOrDefault(host);
|
||||
public bool IsSuspended(string host) => Find(host)?.Severity == DomainBlockSeverity.Suspend;
|
||||
public Task Reload(CancellationToken token) => Task.CompletedTask;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -19,6 +19,37 @@ namespace PrivaPub.Tests.Domain
|
||||
return tagged.WriteToBuffer(".jpg");
|
||||
}
|
||||
|
||||
// SVG, PDF and the other formats libvips could read never load from an upload, whatever it claims to be
|
||||
[Fact]
|
||||
public void Only_the_upload_formats_are_ever_loaded()
|
||||
{
|
||||
var svg = "<svg xmlns='http://www.w3.org/2000/svg' width='20000' height='20000'><rect width='10' height='10'/></svg>"u8.ToArray();
|
||||
Assert.Throws<VipsException>(() => MediaService.Inspect(svg));
|
||||
Assert.Throws<VipsException>(() => MediaService.Inspect("%PDF-1.4\n1 0 obj<<>>endobj\ntrailer<<>>\n%%EOF"u8.ToArray()));
|
||||
using var colour = (Image.Black(8, 8, bands: 3) + 100).Cast(Enums.BandFormat.Uchar);
|
||||
foreach (var format in new[] { ".jpg", ".png", ".gif", ".webp" })
|
||||
Assert.Equal(8, MediaService.Inspect(colour.WriteToBuffer(format)).Width);
|
||||
Assert.Throws<VipsException>(() => MediaService.Inspect(colour.WriteToBuffer(".tif")));
|
||||
}
|
||||
|
||||
// the header tells how big an image would decode before anything is decoded
|
||||
[Fact]
|
||||
public void The_header_says_how_big_an_image_would_decode()
|
||||
{
|
||||
using var huge = Image.Black(8000, 6000);
|
||||
var header = MediaService.Inspect(huge.WriteToBuffer(".png"));
|
||||
Assert.Equal(48_000_000, header.Pixels);
|
||||
Assert.False(header.Animated);
|
||||
|
||||
using var frame = (Image.Black(40, 30, bands: 3) + 60).Cast(Enums.BandFormat.Uchar);
|
||||
using var frames = Image.Arrayjoin(new[] { frame, frame + 80, frame + 160 }, across: 1).Cast(Enums.BandFormat.Uchar);
|
||||
using var paged = frames.Mutate(m => m.Set(GValue.GIntType, "page-height", 30));
|
||||
var gif = MediaService.Inspect(paged.WriteToBuffer(".gif"));
|
||||
Assert.True(gif.Animated);
|
||||
Assert.Equal(3, gif.Pages);
|
||||
Assert.Equal(40 * 30 * 3, gif.Pixels);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Uploaded_images_lose_every_kind_of_metadata()
|
||||
{
|
||||
@@ -26,7 +57,7 @@ namespace PrivaPub.Tests.Domain
|
||||
using (var original = Image.NewFromBuffer(input))
|
||||
Assert.Contains("exif-data", original.GetFields());
|
||||
|
||||
var processed = MediaService.ProcessImage(input, 4096, 640, animated: false);
|
||||
var processed = MediaService.ProcessImage(input, 4096, 640);
|
||||
|
||||
using var output = Image.NewFromBuffer(processed.Bytes);
|
||||
var fields = output.GetFields();
|
||||
@@ -44,7 +75,7 @@ namespace PrivaPub.Tests.Domain
|
||||
[Fact]
|
||||
public void Large_images_are_capped()
|
||||
{
|
||||
var processed = MediaService.ProcessImage(JpegWithMetadata(5000, 2500), 4096, 640, animated: false);
|
||||
var processed = MediaService.ProcessImage(JpegWithMetadata(5000, 2500), 4096, 640);
|
||||
|
||||
Assert.Equal((4096, 2048), (processed.Width, processed.Height));
|
||||
}
|
||||
@@ -52,7 +83,7 @@ namespace PrivaPub.Tests.Domain
|
||||
[Fact]
|
||||
public void A_blurhash_is_well_formed()
|
||||
{
|
||||
var processed = MediaService.ProcessImage(JpegWithMetadata(64, 64), 4096, 640, animated: false);
|
||||
var processed = MediaService.ProcessImage(JpegWithMetadata(64, 64), 4096, 640);
|
||||
|
||||
Assert.Equal(28, processed.Blurhash.Length);
|
||||
Assert.Equal('L', processed.Blurhash[0]);
|
||||
|
||||
@@ -489,9 +489,17 @@ namespace PrivaPub.Tests.Federation
|
||||
|
||||
await new MediaJanitor(media, options, NullLogger<MediaJanitor>.Instance).Sweep(token);
|
||||
|
||||
Assert.False(await DB.Default.Find<MediaAttachment>().Match(m => m.ID == stale.ID).ExecuteAnyAsync(token));
|
||||
// never posted for a day: trashed, its files out of what /media/files serves at once
|
||||
Assert.NotNull((await DB.Default.Find<MediaAttachment>().OneAsync(stale.ID, token)).TrashedAt);
|
||||
Assert.False(File.Exists(Path.Combine(root, stale.FilePath)));
|
||||
Assert.False(File.Exists(Path.Combine(root, stale.PreviewPath)));
|
||||
Assert.True(File.Exists(Path.Combine(media.TrashRoot, stale.FilePath)));
|
||||
// and once its grace has passed, deleted with its row
|
||||
await DB.Default.Update<MediaAttachment>().MatchID(stale.ID).Modify(m => m.TrashedAt, DateTime.UtcNow - MediaJanitor.TrashGrace - TimeSpan.FromMinutes(1)).ExecuteAsync(token);
|
||||
await new MediaJanitor(media, options, NullLogger<MediaJanitor>.Instance).Sweep(token);
|
||||
Assert.False(await DB.Default.Find<MediaAttachment>().Match(m => m.ID == stale.ID).ExecuteAnyAsync(token));
|
||||
Assert.False(File.Exists(Path.Combine(media.TrashRoot, stale.FilePath)));
|
||||
Assert.False(File.Exists(Path.Combine(media.TrashRoot, stale.PreviewPath)));
|
||||
Assert.True(await DB.Default.Find<MediaAttachment>().Match(m => m.ID == fresh.ID).ExecuteAnyAsync(token));
|
||||
Assert.True(await DB.Default.Find<MediaAttachment>().Match(m => m.ID == attached.ID).ExecuteAnyAsync(token));
|
||||
Assert.True(File.Exists(Path.Combine(root, fresh.FilePath)));
|
||||
|
||||
@@ -0,0 +1,200 @@
|
||||
using MongoDB.Entities;
|
||||
|
||||
using PrivaPub.ClientModels;
|
||||
using PrivaPub.Infrastructure.Backup;
|
||||
using PrivaPub.Models.User;
|
||||
using PrivaPub.Tests.Support;
|
||||
using PrivaPub.Tests.Support.Host;
|
||||
|
||||
using System.Formats.Tar;
|
||||
using System.Net;
|
||||
using System.Net.Http.Headers;
|
||||
using System.Net.Http.Json;
|
||||
using System.Text.Json.Nodes;
|
||||
|
||||
namespace PrivaPub.Tests.Http
|
||||
{
|
||||
// The administrator's page backs up, downloads, uploads and restores (owner decision 2026-10-07): only an
|
||||
// administrator the database still says is one; a download and a restore ask for the password again, a restore for
|
||||
// the host typed out; an upload resumes where it stopped, and one that would write outside its backup is refused.
|
||||
[Trait("Category", "Integration")]
|
||||
[Xunit.Collection(nameof(Exclusive))]
|
||||
public sealed class BackupEndpointTests : IAsyncLifetime
|
||||
{
|
||||
const string Base = "/clientapi/admin/backups";
|
||||
PrivaPubHost _host;
|
||||
|
||||
static CancellationToken Token => TestContext.Current.CancellationToken;
|
||||
|
||||
public async ValueTask InitializeAsync()
|
||||
{
|
||||
Assert.SkipUnless(MongoFixture.Enabled, MongoFixture.Skip);
|
||||
_host = await PrivaPubHost.Shared();
|
||||
}
|
||||
|
||||
public ValueTask DisposeAsync()
|
||||
{
|
||||
RestoreMarker.Clear(_host?.Get<Backups>().Root ?? Path.GetTempPath());
|
||||
return ValueTask.CompletedTask;
|
||||
}
|
||||
|
||||
async Task<string> BackedUp(HttpClient admin)
|
||||
{
|
||||
var before = _host.Get<Backups>().List().Select(b => b.Id).ToHashSet();
|
||||
Assert.Equal(HttpStatusCode.Accepted, (await admin.PostAsync(Base, null, Token)).StatusCode);
|
||||
var made = default(string);
|
||||
Assert.True(await Wait(async () =>
|
||||
{
|
||||
var list = (await admin.GetFromJsonAsync<JsonObject>(Base, Token))!;
|
||||
made = list["backups"]!.AsArray().Select(b => b!["id"]!.GetValue<string>()).FirstOrDefault(id => !before.Contains(id));
|
||||
return made != default && list["running"] is null;
|
||||
}), "the backup was never made");
|
||||
return made;
|
||||
}
|
||||
|
||||
static async Task<bool> Wait(Func<Task<bool>> done)
|
||||
{
|
||||
for (var i = 0; i < 120; i++)
|
||||
{
|
||||
if (await done())
|
||||
return true;
|
||||
await Task.Delay(250, Token);
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Only_an_administrator_the_database_still_knows()
|
||||
{
|
||||
var root = await _host.SignUp("plain");
|
||||
using (var plain = _host.As(root.Jwt))
|
||||
Assert.Equal(HttpStatusCode.Forbidden, (await plain.GetAsync(Base, Token)).StatusCode);
|
||||
|
||||
var admin = await _host.Admin();
|
||||
using var client = _host.As(admin.Jwt);
|
||||
Assert.Equal(HttpStatusCode.OK, (await client.GetAsync(Base, Token)).StatusCode);
|
||||
await DB.Default.Update<RootUser>().MatchID(admin.Id).Modify(u => u.Policies, [Policies.IsUser]).ExecuteAsync(Token);
|
||||
Assert.Equal(HttpStatusCode.Forbidden, (await client.GetAsync(Base, Token)).StatusCode);
|
||||
Assert.Equal(HttpStatusCode.Forbidden, (await client.PostAsync(Base, null, Token)).StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_backup_is_made_downloaded_whole_or_in_part_and_uploaded_again()
|
||||
{
|
||||
var admin = await _host.Admin();
|
||||
using var client = _host.As(admin.Jwt);
|
||||
var id = await BackedUp(client);
|
||||
var listed = (await client.GetFromJsonAsync<JsonObject>(Base, Token))!;
|
||||
Assert.Equal("privapub.test", listed["host"]!.GetValue<string>());
|
||||
Assert.Equal("manual", listed["backups"]!.AsArray().Single(b => b!["id"]!.GetValue<string>() == id)!["kind"]!.GetValue<string>());
|
||||
|
||||
Assert.Equal(HttpStatusCode.UnprocessableEntity, (await client.PostAsJsonAsync($"{Base}/{id}/ticket", new { password = "wrong" }, Token)).StatusCode);
|
||||
var ticket = (await (await client.PostAsJsonAsync($"{Base}/{id}/ticket", new { password = admin.Password }, Token)).Content.ReadFromJsonAsync<JsonObject>(Token))!;
|
||||
var url = ticket["url"]!.GetValue<string>();
|
||||
|
||||
// a plain link: no token needed, the ticket is the credential
|
||||
using var anonymous = _host.Client();
|
||||
var whole = await anonymous.GetAsync(url, Token);
|
||||
Assert.Equal(HttpStatusCode.OK, whole.StatusCode);
|
||||
var bytes = await whole.Content.ReadAsByteArrayAsync(Token);
|
||||
Assert.Equal(ticket["bytes"]!.GetValue<long>(), bytes.Length);
|
||||
Assert.Equal(bytes.Length, whole.Content.Headers.ContentLength);
|
||||
var names = new List<string>();
|
||||
await using (var reader = new TarReader(new MemoryStream(bytes)))
|
||||
while (await reader.GetNextEntryAsync(cancellationToken: Token) is { } entry)
|
||||
names.Add(entry.Name);
|
||||
Assert.Contains($"{id}/manifest.json", names);
|
||||
Assert.Contains($"{id}/db/Avatar.jsonl.gz", names);
|
||||
|
||||
var part = new HttpRequestMessage(HttpMethod.Get, url);
|
||||
part.Headers.Range = new RangeHeaderValue(700, 2747);
|
||||
var partial = await anonymous.SendAsync(part, Token);
|
||||
Assert.Equal(HttpStatusCode.PartialContent, partial.StatusCode);
|
||||
Assert.Equal($"bytes 700-2747/{bytes.Length}", partial.Content.Headers.ContentRange!.ToString());
|
||||
Assert.Equal(bytes.AsSpan(700, 2048).ToArray(), await partial.Content.ReadAsByteArrayAsync(Token));
|
||||
Assert.Equal(HttpStatusCode.NotFound, (await anonymous.GetAsync($"{Base}/download/{new string('0', 64)}", Token)).StatusCode);
|
||||
|
||||
// gone, then uploaded again in three pieces, one sent at the wrong place and refused with where to go on
|
||||
Assert.Equal(HttpStatusCode.OK, (await client.DeleteAsync($"{Base}/{id}", Token)).StatusCode);
|
||||
Assert.Equal(HttpStatusCode.NotFound, (await client.DeleteAsync($"{Base}/{id}", Token)).StatusCode);
|
||||
var upload = (await (await client.PostAsync($"{Base}/uploads", null, Token)).Content.ReadFromJsonAsync<JsonObject>(Token))!["id"]!.GetValue<string>();
|
||||
var third = bytes.Length / 3;
|
||||
Assert.Equal(HttpStatusCode.OK, (await Piece(client, upload, 0, bytes[..third])).StatusCode);
|
||||
var gap = await Piece(client, upload, third + 10, bytes[(third + 10)..]);
|
||||
Assert.Equal(HttpStatusCode.Conflict, gap.StatusCode);
|
||||
Assert.Equal(third, (await gap.Content.ReadFromJsonAsync<JsonObject>(Token))!["received"]!.GetValue<long>());
|
||||
Assert.Equal(HttpStatusCode.OK, (await Piece(client, upload, third, bytes[third..(2 * third)])).StatusCode);
|
||||
Assert.Equal(HttpStatusCode.OK, (await Piece(client, upload, 2 * third, bytes[(2 * third)..])).StatusCode);
|
||||
var finished = await client.PostAsync($"{Base}/uploads/{upload}/finish", null, Token);
|
||||
Assert.Equal(HttpStatusCode.OK, finished.StatusCode);
|
||||
Assert.Equal("uploaded", (await finished.Content.ReadFromJsonAsync<JsonObject>(Token))!["kind"]!.GetValue<string>());
|
||||
Assert.Empty(await _host.Get<Backups>().Verify(id, Token));
|
||||
Assert.Equal(HttpStatusCode.UnprocessableEntity, (await client.PostAsync($"{Base}/uploads/{upload}/finish", null, Token)).StatusCode);
|
||||
_host.Get<Backups>().Delete(id);
|
||||
}
|
||||
|
||||
Task<HttpResponseMessage> Piece(HttpClient client, string upload, long offset, byte[] bytes) =>
|
||||
client.PutAsync($"{Base}/uploads/{upload}?offset={offset}", new ByteArrayContent(bytes), Token);
|
||||
|
||||
[Theory]
|
||||
[InlineData("escape")]
|
||||
[InlineData("link")]
|
||||
[InlineData("elsewhere")]
|
||||
[InlineData("junk")]
|
||||
public async Task An_upload_that_is_not_a_plain_backup_is_refused_and_leaves_nothing(string how)
|
||||
{
|
||||
var admin = await _host.Admin();
|
||||
using var client = _host.As(admin.Jwt);
|
||||
const string id = "20260101-000000-manual";
|
||||
var tar = new MemoryStream();
|
||||
if (how == "junk")
|
||||
tar.Write("not a tar at all, not even close"u8);
|
||||
else
|
||||
await using (var writer = new TarWriter(tar, TarEntryFormat.Pax, leaveOpen: true))
|
||||
{
|
||||
await writer.WriteEntryAsync(new PaxTarEntry(TarEntryType.RegularFile, $"{id}/manifest.json") { DataStream = new MemoryStream("{}"u8.ToArray()) }, Token);
|
||||
await writer.WriteEntryAsync(how switch
|
||||
{
|
||||
"escape" => new PaxTarEntry(TarEntryType.RegularFile, $"{id}/../../evil") { DataStream = new MemoryStream("x"u8.ToArray()) },
|
||||
"link" => new PaxTarEntry(TarEntryType.SymbolicLink, $"{id}/media/x.jpg") { LinkName = "/etc/passwd" },
|
||||
_ => new PaxTarEntry(TarEntryType.RegularFile, "20260101-000000-nightly/db/Post.jsonl.gz") { DataStream = new MemoryStream("x"u8.ToArray()) }
|
||||
}, Token);
|
||||
}
|
||||
var upload = (await (await client.PostAsync($"{Base}/uploads", null, Token)).Content.ReadFromJsonAsync<JsonObject>(Token))!["id"]!.GetValue<string>();
|
||||
Assert.Equal(HttpStatusCode.OK, (await Piece(client, upload, 0, tar.ToArray())).StatusCode);
|
||||
|
||||
Assert.Equal(HttpStatusCode.UnprocessableEntity, (await client.PostAsync($"{Base}/uploads/{upload}/finish", null, Token)).StatusCode);
|
||||
var root = _host.Get<Backups>().Root;
|
||||
Assert.False(Directory.Exists(Path.Combine(root, id)));
|
||||
Assert.False(Directory.Exists(Path.Combine(root, id + ServerBackup.PartialSuffix)));
|
||||
Assert.False(File.Exists(Path.Combine(root, "..", "evil")));
|
||||
}
|
||||
|
||||
// asked for: written for the next start (the test host has no watcher to stop it, and the marker is taken away)
|
||||
[Fact]
|
||||
public async Task A_restore_asks_for_the_password_and_the_host_typed_out()
|
||||
{
|
||||
var admin = await _host.Admin();
|
||||
using var client = _host.As(admin.Jwt);
|
||||
var id = await BackedUp(client);
|
||||
var root = _host.Get<Backups>().Root;
|
||||
try
|
||||
{
|
||||
Assert.Equal(HttpStatusCode.UnprocessableEntity, (await client.PostAsJsonAsync($"{Base}/{id}/restore", new { password = "wrong", host = "privapub.test" }, Token)).StatusCode);
|
||||
Assert.Equal(HttpStatusCode.UnprocessableEntity, (await client.PostAsJsonAsync($"{Base}/{id}/restore", new { password = admin.Password, host = "elsewhere.test" }, Token)).StatusCode);
|
||||
Assert.Null(RestoreMarker.Read(root));
|
||||
|
||||
Assert.Equal(HttpStatusCode.Accepted, (await client.PostAsJsonAsync($"{Base}/{id}/restore", new { password = admin.Password, host = "PrivaPub.test" }, Token)).StatusCode);
|
||||
var marker = RestoreMarker.Read(root);
|
||||
Assert.Equal((id, "pending", admin.UserName), (marker.Backup, marker.State, marker.RequestedBy));
|
||||
Assert.Equal(id, (await client.GetFromJsonAsync<JsonObject>(Base, Token))!["waiting"]!["backup"]!.GetValue<string>());
|
||||
Assert.Equal(HttpStatusCode.Conflict, (await client.DeleteAsync($"{Base}/{id}", Token)).StatusCode);
|
||||
}
|
||||
finally
|
||||
{
|
||||
RestoreMarker.Clear(root);
|
||||
_host.Get<Backups>().Delete(id);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -45,6 +45,8 @@ namespace PrivaPub.Tests.Http
|
||||
Assert.Equal($"wss://{PrivaPubHost.Host}", v1.Body["urls"]!.Text("streaming_api"));
|
||||
Assert.Equal(PrivaPub.Api.Mastodon.Controllers.StatusesController.MaxPins, v2.Body["configuration"]!["accounts"].Number("max_pinned_statuses"));
|
||||
Assert.Contains("image/avif", v2.Body["configuration"]!["media_attachments"]!["supported_mime_types"]!.AsArray().Select(t => t!.GetValue<string>()));
|
||||
// what the bundled libvips can't decode is not offered
|
||||
Assert.DoesNotContain("image/heic", v2.Body["configuration"]!["media_attachments"]!["supported_mime_types"]!.AsArray().Select(t => t!.GetValue<string>()));
|
||||
Assert.True(v2.Body["registrations"].Flag("enabled"));
|
||||
Assert.True((await anonymous.Get("/nodeinfo/2.1")).Ok().Body.Flag("openRegistrations"));
|
||||
Assert.False(v2.Body["configuration"]!["translation"].Flag("enabled"));
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
using PrivaPub.Models.Jobs;
|
||||
using PrivaPub.Domain.Media;
|
||||
using PrivaPub.Tests.Support;
|
||||
using PrivaPub.Tests.Support.Host;
|
||||
@@ -31,6 +32,14 @@ namespace PrivaPub.Tests.Http
|
||||
|
||||
static CancellationToken Token => TestContext.Current.CancellationToken;
|
||||
|
||||
// what GET /api/v1/media/:id answers once the job processing an upload sent to v2 has run
|
||||
async Task<ApiAnswer> Processed(Mastodon account, ApiAnswer accepted)
|
||||
{
|
||||
var id = accepted.Body.Text("id");
|
||||
await _host.Run(j => j.Kind == JobKind.ProcessMedia && j.Payload == id, Token);
|
||||
return await account.Client.Get($"/api/v1/media/{id}");
|
||||
}
|
||||
|
||||
static Task<ApiAnswer> Upload(Mastodon account, string path, byte[] bytes, string contentType, string fileName, params (string Key, string Value)[] fields)
|
||||
{
|
||||
var form = MastodonHelpers.Multipart(("file", bytes, contentType, fileName));
|
||||
@@ -136,6 +145,25 @@ namespace PrivaPub.Tests.Http
|
||||
Assert.Null((await alice.Client.Put($"/api/v1/media/{id}", ("description", ""))).Ok().Body.Text("description"));
|
||||
}
|
||||
|
||||
// a focal point is two finite numbers: NaN or Infinity is ignored, never stored (it broke every timeline holding the post)
|
||||
[Fact]
|
||||
public async Task A_focal_point_that_is_not_a_number_is_ignored()
|
||||
{
|
||||
var alice = await _host.Mastodon("alice");
|
||||
var uploaded = (await Upload(alice, "/api/v2/media", MastodonHelpers.JpegWithMetadata(32, 32), "image/jpeg", "a.jpg", ("focus", "NaN,NaN"))).Ok();
|
||||
Assert.Null(uploaded.Body["meta"]?["focus"]);
|
||||
var id = uploaded.Body.Text("id");
|
||||
|
||||
(await alice.Client.Put($"/api/v1/media/{id}", ("focus", "0.5,0.25"))).Ok();
|
||||
foreach (var unsound in new[] { "NaN,0", "Infinity,1", "0,-Infinity" })
|
||||
{
|
||||
var focus = (await alice.Client.Put($"/api/v1/media/{id}", ("focus", unsound))).Ok().Body["meta"]!["focus"]!;
|
||||
Assert.Equal((0.5, 0.25), (focus["x"]!.GetValue<double>(), focus["y"]!.GetValue<double>()));
|
||||
}
|
||||
var status = (await alice.Client.Post("/api/v1/statuses", ("status", "focused"), ("media_ids[]", id))).Ok();
|
||||
Assert.Equal(0.5, status.Body["media_attachments"]![0]!["meta"]!["focus"]!["x"]!.GetValue<double>());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Unsupported_unreadable_and_missing_files_are_refused_with_422()
|
||||
{
|
||||
@@ -145,12 +173,50 @@ namespace PrivaPub.Tests.Http
|
||||
Assert.Equal(HttpStatusCode.UnprocessableEntity, text.Status);
|
||||
Assert.Contains("not supported", text.Body.Text("error"));
|
||||
Assert.Equal(HttpStatusCode.UnprocessableEntity, (await Upload(alice, "/api/v2/media", Encoding.UTF8.GetBytes("not a jpeg"), "image/jpeg", "a.jpg")).Status);
|
||||
Assert.Equal(HttpStatusCode.UnprocessableEntity, (await Upload(alice, "/api/v2/media", Encoding.UTF8.GetBytes("not a video"), "video/mp4", "a.mp4")).Status);
|
||||
Assert.Equal(HttpStatusCode.UnprocessableEntity, (await Upload(alice, "/api/v1/media", Encoding.UTF8.GetBytes("not a video"), "video/mp4", "a.mp4")).Status);
|
||||
// sent to v2, it is taken, then refused once processed
|
||||
var later = await Upload(alice, "/api/v2/media", Encoding.UTF8.GetBytes("not a video"), "video/mp4", "a.mp4");
|
||||
Assert.Equal(HttpStatusCode.Accepted, later.Status);
|
||||
Assert.Equal(HttpStatusCode.UnprocessableEntity, (await Processed(alice, later)).Status);
|
||||
var empty = new MultipartFormDataContent { { new StringContent("no file"), "description" } };
|
||||
Assert.Equal(HttpStatusCode.UnprocessableEntity, (await alice.Client.Exchange(new HttpRequestMessage(HttpMethod.Post, "/api/v2/media") { Content = empty })).Status);
|
||||
Assert.Equal(HttpStatusCode.UnprocessableEntity, (await alice.Client.Post("/api/v1/media")).Status);
|
||||
}
|
||||
|
||||
// refused before anything is decoded: an SVG claiming to be a PNG, and an image that would decode to too many pixels
|
||||
[Fact]
|
||||
public async Task An_image_that_is_not_one_or_too_large_is_refused_before_decoding()
|
||||
{
|
||||
var alice = await _host.Mastodon("alice");
|
||||
var svg = "<svg xmlns='http://www.w3.org/2000/svg' width='30000' height='30000'><rect width='1' height='1'/></svg>"u8.ToArray();
|
||||
Assert.Equal(HttpStatusCode.UnprocessableEntity, (await Upload(alice, "/api/v2/media", svg, "image/png", "a.png")).Status);
|
||||
|
||||
using var huge = NetVips.Image.Black(8000, 6000);
|
||||
var tooLarge = await Upload(alice, "/api/v2/media", huge.WriteToBuffer(".png"), "image/png", "huge.png");
|
||||
Assert.Equal(HttpStatusCode.UnprocessableEntity, tooLarge.Status);
|
||||
Assert.Contains("too large", tooLarge.Body.Text("error"));
|
||||
}
|
||||
|
||||
// an animated GIF becomes what Mastodon makes of one, a looping mp4 typed gifv; a still GIF stays an image
|
||||
[Fact]
|
||||
public async Task An_animated_gif_becomes_a_gifv_and_a_still_one_an_image()
|
||||
{
|
||||
var alice = await _host.Mastodon("alice");
|
||||
using var frame = (NetVips.Image.Black(64, 48, bands: 3) + 60).Cast(NetVips.Enums.BandFormat.Uchar);
|
||||
using var frames = NetVips.Image.Arrayjoin(new[] { frame, frame + 80, frame + 160 }, across: 1).Cast(NetVips.Enums.BandFormat.Uchar);
|
||||
using var animated = frames.Mutate(m => m.Set(NetVips.GValue.GIntType, "page-height", 48));
|
||||
|
||||
var gifv = (await Upload(alice, "/api/v2/media", animated.WriteToBuffer(".gif"), "image/gif", "dance.gif")).Ok();
|
||||
Assert.Equal("gifv", gifv.Body.Text("type"));
|
||||
Assert.EndsWith(".mp4", gifv.Body.Text("url"));
|
||||
Assert.Equal(64, gifv.Body["meta"]!["original"]!["width"]!.GetValue<int>());
|
||||
var status = (await alice.Client.Post("/api/v1/statuses", ("status", "dancing"), ("media_ids[]", gifv.Body.Text("id")))).Ok();
|
||||
Assert.Equal("gifv", status.Body["media_attachments"]![0]!.Text("type"));
|
||||
|
||||
var still = (await Upload(alice, "/api/v2/media", frame.WriteToBuffer(".gif"), "image/gif", "still.gif")).Ok();
|
||||
Assert.Equal("image", still.Body.Text("type"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Video_is_remuxed_without_its_metadata()
|
||||
{
|
||||
@@ -160,7 +226,13 @@ namespace PrivaPub.Tests.Http
|
||||
"-c:v", "mpeg4", "-c:a", "aac", "-shortest");
|
||||
Assert.Contains("secret title", Encoding.Latin1.GetString(video));
|
||||
|
||||
var uploaded = (await Upload(alice, "/api/v2/media", video, "video/mp4", "clip.mp4")).Ok();
|
||||
// v2 answers before it is processed: 202, no url yet, and it can't be posted until it is
|
||||
var accepted = await Upload(alice, "/api/v2/media", video, "video/mp4", "clip.mp4");
|
||||
Assert.Equal(HttpStatusCode.Accepted, accepted.Status);
|
||||
Assert.Null(accepted.Body["url"]);
|
||||
Assert.Equal(HttpStatusCode.PartialContent, (await alice.Client.Get($"/api/v1/media/{accepted.Body.Text("id")}")).Status);
|
||||
Assert.Equal(HttpStatusCode.UnprocessableEntity, (await alice.Client.Post("/api/v1/statuses", ("status", "too soon"), ("media_ids[]", accepted.Body.Text("id")))).Status);
|
||||
var uploaded = (await Processed(alice, accepted)).Ok();
|
||||
|
||||
Assert.Equal("video", uploaded.Body.Text("type"));
|
||||
Assert.Equal(320, uploaded.Body["meta"]!["original"].Number("width"));
|
||||
@@ -168,6 +240,8 @@ namespace PrivaPub.Tests.Http
|
||||
Assert.EndsWith(".jpg", uploaded.Body.Text("preview_url"));
|
||||
var probe = await Probe(uploaded.Body.Text("url"));
|
||||
Assert.Contains("\"codec_type\": \"video\"", probe);
|
||||
// MPEG-4 Part 2, which browsers don't play, made H.264
|
||||
Assert.Contains("\"codec_name\": \"h264\"", probe);
|
||||
Assert.DoesNotContain("secret title", probe);
|
||||
Assert.DoesNotContain("filmed at home", probe);
|
||||
Assert.DoesNotContain("hidden handler", probe);
|
||||
@@ -180,7 +254,8 @@ namespace PrivaPub.Tests.Http
|
||||
var audio = await Made("m4a", "-f", "lavfi", "-i", "sine=frequency=330:duration=1", "-metadata", "title=secret song", "-metadata", "artist=Alice Smith",
|
||||
"-c:a", "aac");
|
||||
|
||||
var uploaded = (await Upload(alice, "/api/v2/media", audio, "audio/mp4", "song.m4a")).Ok();
|
||||
// v1 waits for it
|
||||
var uploaded = (await Upload(alice, "/api/v1/media", audio, "audio/mp4", "song.m4a")).Ok();
|
||||
|
||||
Assert.Equal("audio", uploaded.Body.Text("type"));
|
||||
var probe = await Probe(uploaded.Body.Text("url"));
|
||||
@@ -189,6 +264,35 @@ namespace PrivaPub.Tests.Http
|
||||
Assert.DoesNotContain("Alice Smith", probe);
|
||||
}
|
||||
|
||||
// FLAC is served as what it is (ASP.NET's map has no entry for it, so it was a 404)
|
||||
[Fact]
|
||||
public async Task Flac_is_served()
|
||||
{
|
||||
var alice = await _host.Mastodon("alice");
|
||||
var flac = await Made("flac", "-f", "lavfi", "-i", "sine=frequency=500:duration=1", "-c:a", "flac");
|
||||
|
||||
var uploaded = (await Upload(alice, "/api/v1/media", flac, "audio/flac", "song.flac")).Ok();
|
||||
|
||||
var served = await _host.Client().GetAsync(new Uri(uploaded.Body.Text("url")).PathAndQuery, Token);
|
||||
Assert.Equal(HttpStatusCode.OK, served.StatusCode);
|
||||
Assert.Equal("audio/flac", served.Content.Headers.ContentType?.MediaType);
|
||||
}
|
||||
|
||||
// a video larger than video_matrix_limit is made smaller, its shape kept
|
||||
[Fact]
|
||||
public async Task A_video_larger_than_the_limit_is_made_smaller()
|
||||
{
|
||||
var alice = await _host.Mastodon("alice");
|
||||
var video = await Made("mp4", "-f", "lavfi", "-i", "testsrc=duration=1:size=2000x1500:rate=5", "-c:v", "libx264", "-preset", "ultrafast", "-pix_fmt", "yuv420p");
|
||||
|
||||
var uploaded = (await Upload(alice, "/api/v1/media", video, "video/mp4", "big.mp4")).Ok();
|
||||
|
||||
var width = uploaded.Body["meta"]!["original"].Number("width");
|
||||
var height = uploaded.Body["meta"]!["original"].Number("height");
|
||||
Assert.True((long)width * height <= 2_304_000, $"{width}x{height}");
|
||||
Assert.InRange((double)width / height, 1.3, 1.37);
|
||||
}
|
||||
|
||||
static byte[] Bytes(int length)
|
||||
{
|
||||
var bytes = new byte[length];
|
||||
@@ -311,6 +415,36 @@ namespace PrivaPub.Tests.Http
|
||||
Assert.Single(_peer.Requests);
|
||||
}
|
||||
|
||||
// clients asking at once for a file not cached yet share one download
|
||||
[Fact]
|
||||
public async Task Clients_asking_at_once_share_one_download()
|
||||
{
|
||||
var proxy = _host.Get<IMediaProxy>();
|
||||
var bytes = Bytes(200_000);
|
||||
var remote = Served(bytes, "image/png");
|
||||
var path = new Uri(remote).AbsolutePath;
|
||||
using var client = _host.Client();
|
||||
|
||||
var answers = await Task.WhenAll(Enumerable.Range(0, 5).Select(_ => client.GetAsync(proxy.Wrap(remote), Token)));
|
||||
|
||||
foreach (var answer in answers)
|
||||
Assert.Equal(bytes, await answer.Content.ReadAsByteArrayAsync(Token));
|
||||
Assert.Single(_peer.Requests, r => r.Path == path);
|
||||
}
|
||||
|
||||
// a remote file that can't be had is a 404 no browser keeps
|
||||
[Fact]
|
||||
public async Task A_remote_file_that_fails_is_not_cached_by_browsers()
|
||||
{
|
||||
var proxy = _host.Get<IMediaProxy>();
|
||||
using var client = _host.Client();
|
||||
|
||||
using var missing = await client.GetAsync(proxy.Wrap($"{_peer.A}/media/{Guid.NewGuid():N}.png"), Token);
|
||||
|
||||
Assert.Equal(HttpStatusCode.NotFound, missing.StatusCode);
|
||||
Assert.Null(missing.Headers.CacheControl);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Anything_over_the_proxy_limit_is_streamed_and_never_cached()
|
||||
{
|
||||
@@ -328,7 +462,8 @@ namespace PrivaPub.Tests.Http
|
||||
Assert.Equal(default, proxy.Cached(remote));
|
||||
using var again = await client.GetAsync(proxy.Wrap(remote), Token);
|
||||
Assert.Equal(bytes, await again.Content.ReadAsByteArrayAsync(Token));
|
||||
Assert.Equal(4, _peer.Requests.Count);
|
||||
// the first time, an attempt to cache it and the stream; then it is known to be too big, and only streamed
|
||||
Assert.Equal(3, _peer.Requests.Count);
|
||||
var small = Served(Bytes(SmallProxyHost.Limit / 2));
|
||||
using (var fits = await client.GetAsync(proxy.Wrap(small), Token))
|
||||
Assert.Equal(HttpStatusCode.OK, fits.StatusCode);
|
||||
|
||||
@@ -99,5 +99,60 @@ namespace PrivaPub.Tests.Http
|
||||
Assert.Equal(published.Text("id"), media.PostId);
|
||||
Assert.Null(media.ScheduledStatusId);
|
||||
}
|
||||
|
||||
async Task<string> Picture(Mastodon account)
|
||||
{
|
||||
var upload = MastodonHelpers.Multipart(("file", MastodonHelpers.JpegWithMetadata(16, 16), "image/jpeg", "a.jpg"));
|
||||
return (await account.Client.Exchange(new HttpRequestMessage(HttpMethod.Post, "/api/v2/media") { Content = upload })).Ok().Body.Text("id");
|
||||
}
|
||||
|
||||
// a scheduled post's media are its alone: a post now, or an edit, can't take them
|
||||
[Fact]
|
||||
public async Task A_scheduled_posts_media_cannot_be_taken_by_a_post_now()
|
||||
{
|
||||
var alice = await _host.Mastodon("alice");
|
||||
var mediaId = await Picture(alice);
|
||||
(await alice.Client.Post("/api/v1/statuses", ("status", "later"), ("media_ids[]", mediaId), ("scheduled_at", In(TimeSpan.FromMinutes(10))))).Ok();
|
||||
|
||||
Assert.Equal(HttpStatusCode.UnprocessableEntity, (await alice.Client.Post("/api/v1/statuses", ("status", "now"), ("media_ids[]", mediaId))).Status);
|
||||
var plain = (await alice.Client.Post("/api/v1/statuses", ("status", "plain"))).Ok().Body.Text("id");
|
||||
Assert.Equal(HttpStatusCode.UnprocessableEntity, (await alice.Client.Put($"/api/v1/statuses/{plain}", ("status", "edited"), ("media_ids[]", mediaId))).Status);
|
||||
var media = await DB.Default.Find<MediaAttachment>().OneAsync(mediaId, Token);
|
||||
Assert.Null(media.PostId);
|
||||
Assert.NotNull(media.ScheduledStatusId);
|
||||
}
|
||||
|
||||
// two posts asking for the same upload at once: exactly one gets it
|
||||
[Fact]
|
||||
public async Task Two_posts_racing_for_one_upload_never_both_get_it()
|
||||
{
|
||||
var alice = await _host.Mastodon("alice");
|
||||
for (var round = 0; round < 5; round++)
|
||||
{
|
||||
var mediaId = await Picture(alice);
|
||||
var answers = await Task.WhenAll(Enumerable.Range(0, 2).Select(i =>
|
||||
alice.Client.Post("/api/v1/statuses", ("status", $"race {round} {i}"), ("media_ids[]", mediaId))));
|
||||
Assert.Single(answers, a => a.Status == HttpStatusCode.OK);
|
||||
Assert.Single(answers, a => a.Status == HttpStatusCode.UnprocessableEntity);
|
||||
}
|
||||
}
|
||||
|
||||
// a persona deleted since it scheduled a post publishes nothing, and what the post held is trashed
|
||||
[Fact]
|
||||
public async Task A_gone_personas_scheduled_post_never_publishes()
|
||||
{
|
||||
var gone = await _host.Mastodon($"gone{Guid.NewGuid():N}"[..12]);
|
||||
var mediaId = await Picture(gone);
|
||||
var id = (await gone.Client.Post("/api/v1/statuses", ("status", "from beyond"), ("media_ids[]", mediaId), ("scheduled_at", In(TimeSpan.FromMinutes(10))))).Ok().Object.Text("id");
|
||||
await DB.Default.Update<Models.User.Avatar>().MatchID(gone.Persona.Id).Modify(a => a.DeletionAt, DateTime.UtcNow).ExecuteAsync(Token);
|
||||
|
||||
await DB.Default.Update<ScheduledStatus>().MatchID(id).Modify(s => s.ScheduledAt, DateTime.UtcNow.AddSeconds(-1)).ExecuteAsync(Token);
|
||||
await DB.Default.Update<Job>().Match(j => j.Kind == JobKind.PublishScheduled && j.Payload == id).Modify(j => j.RunAt, DateTime.UtcNow).ExecuteAsync(Token);
|
||||
await _host.Run(j => j.Kind == JobKind.PublishScheduled && j.Payload == id, Token);
|
||||
|
||||
Assert.False(await DB.Default.Find<Models.Post.Post>().Match(p => p.GroupUserId == gone.Persona.Id && p.Text == "from beyond").ExecuteAnyAsync(Token));
|
||||
Assert.False(await DB.Default.Find<ScheduledStatus>().Match(s => s.ID == id).ExecuteAnyAsync(Token));
|
||||
Assert.NotNull((await DB.Default.Find<MediaAttachment>().OneAsync(mediaId, Token)).TrashedAt);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,234 @@
|
||||
using MongoDB.Entities;
|
||||
|
||||
using PrivaPub.Domain.Media;
|
||||
using PrivaPub.Models.Media;
|
||||
using PrivaPub.Services;
|
||||
using PrivaPub.Tests.Support;
|
||||
using PrivaPub.Tests.Support.Host;
|
||||
|
||||
using System.Net;
|
||||
|
||||
namespace PrivaPub.Tests.Http
|
||||
{
|
||||
// A file lives exactly as long as something holds it: a deleted post, an edit leaving media out, a replaced picture and
|
||||
// a removed root each trash theirs, which stops /media/files serving them at once; the janitor deletes them later.
|
||||
[Trait("Category", "Integration")]
|
||||
public sealed class MediaLifecycleTests : IAsyncLifetime
|
||||
{
|
||||
PrivaPubHost _host;
|
||||
|
||||
public async ValueTask InitializeAsync()
|
||||
{
|
||||
Assert.SkipUnless(MongoFixture.Enabled, MongoFixture.Skip);
|
||||
_host = await PrivaPubHost.Shared();
|
||||
}
|
||||
|
||||
public ValueTask DisposeAsync() => ValueTask.CompletedTask;
|
||||
|
||||
static CancellationToken Token => TestContext.Current.CancellationToken;
|
||||
|
||||
async Task<string> Upload(Mastodon account, string name)
|
||||
{
|
||||
var form = MastodonHelpers.Multipart(("file", MastodonHelpers.JpegWithMetadata(64, 48), "image/jpeg", name));
|
||||
return (await account.Client.Exchange(new HttpRequestMessage(HttpMethod.Post, "/api/v2/media") { Content = form })).Ok().Body.Text("id");
|
||||
}
|
||||
|
||||
async Task<HttpStatusCode> Served(string url) =>
|
||||
(await _host.Client().GetAsync(new Uri(url).PathAndQuery, Token)).StatusCode;
|
||||
|
||||
[Fact]
|
||||
public async Task A_deleted_posts_media_stop_being_served()
|
||||
{
|
||||
var alice = await _host.Mastodon("alice");
|
||||
var id = await Upload(alice, "a.jpg");
|
||||
var status = (await alice.Client.Post("/api/v1/statuses", ("status", "with a picture"), ("media_ids[]", id))).Ok();
|
||||
var url = status.Body["media_attachments"]![0]!["url"]!.GetValue<string>();
|
||||
var preview = status.Body["media_attachments"]![0]!["preview_url"]!.GetValue<string>();
|
||||
Assert.Equal(HttpStatusCode.OK, await Served(url));
|
||||
|
||||
(await alice.Client.Delete($"/api/v1/statuses/{status.Body.Text("id")}")).Ok();
|
||||
|
||||
Assert.Equal(HttpStatusCode.NotFound, await Served(url));
|
||||
Assert.Equal(HttpStatusCode.NotFound, await Served(preview));
|
||||
Assert.Equal("post deleted", (await DB.Default.Find<MediaAttachment>().OneAsync(id, Token)).TrashReason);
|
||||
// the file waits in the trash, beside what /media/files serves
|
||||
var relative = url[(url.IndexOf("/media/files/", StringComparison.Ordinal) + "/media/files/".Length)..];
|
||||
Assert.True(File.Exists(Path.Combine(_host.Get<IMediaService>().TrashRoot, relative)));
|
||||
Assert.Equal(HttpStatusCode.NotFound, await Served(url.Replace("/media/files/", "/media/files/.trash/")));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task An_edit_that_leaves_media_out_trashes_them()
|
||||
{
|
||||
var alice = await _host.Mastodon("alice");
|
||||
var kept = await Upload(alice, "kept.jpg");
|
||||
var dropped = await Upload(alice, "dropped.jpg");
|
||||
var status = (await alice.Client.Post("/api/v1/statuses", ("status", "two pictures"), ("media_ids[]", kept), ("media_ids[]", dropped))).Ok();
|
||||
var urls = status.Body["media_attachments"]!.AsArray().ToDictionary(m => m!["id"]!.GetValue<string>(), m => m!["url"]!.GetValue<string>());
|
||||
|
||||
(await alice.Client.Put($"/api/v1/statuses/{status.Body.Text("id")}", ("status", "one picture"), ("media_ids[]", kept))).Ok();
|
||||
|
||||
Assert.Equal(HttpStatusCode.OK, await Served(urls[kept]));
|
||||
Assert.Equal(HttpStatusCode.NotFound, await Served(urls[dropped]));
|
||||
Assert.Null((await DB.Default.Find<MediaAttachment>().OneAsync(kept, Token)).TrashedAt);
|
||||
Assert.Equal("edited out", (await DB.Default.Find<MediaAttachment>().OneAsync(dropped, Token)).TrashReason);
|
||||
}
|
||||
|
||||
// a posted picture's description and focus change by editing its post (media_attributes), not by PUT, as on Mastodon
|
||||
[Fact]
|
||||
public async Task An_edit_describes_its_media_again_and_keeps_what_it_is_not_sent()
|
||||
{
|
||||
var alice = await _host.Mastodon("alice");
|
||||
var id = await Upload(alice, "a.jpg");
|
||||
var status = (await alice.Client.Post("/api/v1/statuses", ("status", "described later"), ("media_ids[]", id),
|
||||
("spoiler_text", "a warning"), ("sensitive", "true"), ("language", "it"))).Ok();
|
||||
var statusId = status.Body.Text("id");
|
||||
Assert.Equal(HttpStatusCode.NotFound, (await alice.Client.Put($"/api/v1/media/{id}", ("description", "too late"))).Status);
|
||||
|
||||
var edited = (await alice.Client.Json(HttpMethod.Put, $"/api/v1/statuses/{statusId}", new System.Text.Json.Nodes.JsonObject
|
||||
{
|
||||
["status"] = "described now",
|
||||
["spoiler_text"] = "a warning",
|
||||
["sensitive"] = true,
|
||||
["media_ids"] = new System.Text.Json.Nodes.JsonArray(id),
|
||||
["media_attributes"] = new System.Text.Json.Nodes.JsonArray(new System.Text.Json.Nodes.JsonObject { ["id"] = id, ["description"] = "a red square", ["focus"] = "0.5,0.5" })
|
||||
})).Ok();
|
||||
|
||||
Assert.Equal("a red square", edited.Body["media_attachments"]![0]!.Text("description"));
|
||||
Assert.Equal(0.5, edited.Body["media_attachments"]![0]!["meta"]!["focus"]!["x"]!.GetValue<double>());
|
||||
Assert.Equal("a warning", edited.Body.Text("spoiler_text"));
|
||||
Assert.True(edited.Body["sensitive"]!.GetValue<bool>());
|
||||
Assert.Equal("it", edited.Body.Text("language"));
|
||||
Assert.Equal("a red square", (await DB.Default.Find<MediaAttachment>().OneAsync(id, Token)).Description);
|
||||
|
||||
// an edit that doesn't send media keeps them, described as they were; an empty media_ids takes them away
|
||||
var kept = (await alice.Client.Put($"/api/v1/statuses/{statusId}", ("status", "media kept"))).Ok();
|
||||
Assert.Equal("a red square", kept.Body["media_attachments"]![0]!.Text("description"));
|
||||
var bare = (await alice.Client.Json(HttpMethod.Put, $"/api/v1/statuses/{statusId}", new System.Text.Json.Nodes.JsonObject
|
||||
{
|
||||
["status"] = "no picture now",
|
||||
["media_ids"] = new System.Text.Json.Nodes.JsonArray()
|
||||
})).Ok();
|
||||
Assert.Empty(bare.Body["media_attachments"]!.AsArray());
|
||||
Assert.Equal("edited out", (await DB.Default.Find<MediaAttachment>().OneAsync(id, Token)).TrashReason);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_replaced_avatar_is_trashed_and_a_picture_that_is_not_one_is_refused()
|
||||
{
|
||||
var alice = await _host.Mastodon("alice");
|
||||
async Task<ApiAnswer> Picture(byte[] bytes, string type, string name) =>
|
||||
await alice.Client.Exchange(new HttpRequestMessage(HttpMethod.Patch, "/api/v1/accounts/update_credentials")
|
||||
{
|
||||
Content = MastodonHelpers.Multipart(("avatar", bytes, type, name))
|
||||
});
|
||||
var first = (await Picture(MastodonHelpers.JpegWithMetadata(300, 300), "image/jpeg", "one.jpg")).Ok().Body.Text("avatar");
|
||||
var second = (await Picture(MastodonHelpers.JpegWithMetadata(320, 300), "image/jpeg", "two.jpg")).Ok().Body.Text("avatar");
|
||||
|
||||
Assert.NotEqual(first, second);
|
||||
Assert.Equal(HttpStatusCode.NotFound, await Served(first));
|
||||
Assert.Equal(HttpStatusCode.OK, await Served(second));
|
||||
var row = await DB.Default.Find<MediaAttachment>().Match(m => m.ProfileOfAvatarId == alice.Persona.Id && m.Kind == "avatar" && m.TrashedAt == null).ExecuteSingleAsync(Token);
|
||||
Assert.EndsWith(row.FilePath, second);
|
||||
|
||||
// not an image: 422, and the avatar stays as it was
|
||||
var refused = await Picture("<svg xmlns='http://www.w3.org/2000/svg'/>"u8.ToArray(), "image/svg+xml", "x.svg");
|
||||
Assert.Equal(HttpStatusCode.UnprocessableEntity, refused.Status);
|
||||
Assert.Equal(second, (await alice.Client.Get("/api/v1/accounts/verify_credentials")).Ok().Body.Text("avatar"));
|
||||
}
|
||||
|
||||
// Mastodon's DELETE /api/v1/profile/avatar and /header: the picture goes, the default shows again
|
||||
[Fact]
|
||||
public async Task A_picture_is_removed_and_the_default_shows_again()
|
||||
{
|
||||
var alice = await _host.Mastodon($"pic{Guid.NewGuid():N}"[..12]);
|
||||
var form = MastodonHelpers.Multipart(("avatar", MastodonHelpers.JpegWithMetadata(200, 200), "image/jpeg", "me.jpg"),
|
||||
("header", MastodonHelpers.JpegWithMetadata(900, 300), "image/jpeg", "top.jpg"));
|
||||
var set = (await alice.Client.Exchange(new HttpRequestMessage(HttpMethod.Patch, "/api/v1/accounts/update_credentials") { Content = form })).Ok();
|
||||
var avatar = set.Body.Text("avatar");
|
||||
var header = set.Body.Text("header");
|
||||
|
||||
var removed = (await alice.Client.Delete("/api/v1/profile/avatar")).Ok();
|
||||
|
||||
Assert.Contains("missing", removed.Body.Text("avatar"));
|
||||
Assert.Equal(header, removed.Body.Text("header"));
|
||||
Assert.Equal(HttpStatusCode.NotFound, await Served(avatar));
|
||||
Assert.Equal(HttpStatusCode.OK, await Served(header));
|
||||
|
||||
Assert.Contains("missing", (await alice.Client.Delete("/api/v1/profile/header")).Ok().Body.Text("header"));
|
||||
Assert.Equal(HttpStatusCode.NotFound, await Served(header));
|
||||
Assert.False(await DB.Default.Find<MediaAttachment>().Match(m => m.ProfileOfAvatarId == alice.Persona.Id && m.TrashedAt == null).ExecuteAnyAsync(Token));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_removed_roots_media_and_pictures_are_trashed_and_its_scheduled_posts_dropped()
|
||||
{
|
||||
var gone = await _host.Mastodon($"gone{Guid.NewGuid():N}"[..12]);
|
||||
var posted = await Upload(gone, "posted.jpg");
|
||||
var url = (await gone.Client.Post("/api/v1/statuses", ("status", "soon gone"), ("media_ids[]", posted))).Ok()
|
||||
.Body["media_attachments"]![0]!["url"]!.GetValue<string>();
|
||||
var held = await Upload(gone, "held.jpg");
|
||||
(await gone.Client.Post("/api/v1/statuses", ("status", "later"), ("media_ids[]", held),
|
||||
("scheduled_at", DateTime.UtcNow.AddHours(2).ToString("O")))).Ok();
|
||||
var avatar = (await gone.Client.Exchange(new HttpRequestMessage(HttpMethod.Patch, "/api/v1/accounts/update_credentials")
|
||||
{
|
||||
Content = MastodonHelpers.Multipart(("avatar", MastodonHelpers.JpegWithMetadata(200, 200), "image/jpeg", "me.jpg"))
|
||||
})).Ok().Body.Text("avatar");
|
||||
|
||||
Assert.True(await _host.Get<IRootRemoval>().Remove(gone.Persona.Root.Id, Token));
|
||||
|
||||
Assert.Equal(HttpStatusCode.NotFound, await Served(url));
|
||||
Assert.Equal(HttpStatusCode.NotFound, await Served(avatar));
|
||||
Assert.False(await DB.Default.Find<MediaAttachment>().Match(m => m.OwnerAvatarId == gone.Persona.Id && m.TrashedAt == null).ExecuteAnyAsync(Token));
|
||||
Assert.False(await DB.Default.Find<Models.Social.ScheduledStatus>().Match(s => s.AvatarId == gone.Persona.Id).ExecuteAnyAsync(Token));
|
||||
}
|
||||
|
||||
// what a login's media take: every persona's uploads and pictures, trashed ones no longer
|
||||
[Fact]
|
||||
public async Task A_logins_storage_counts_what_its_personas_keep()
|
||||
{
|
||||
var alice = await _host.Mastodon($"store{Guid.NewGuid():N}"[..12]);
|
||||
using var root = _host.As(alice.Persona.Root.Jwt);
|
||||
async Task<System.Text.Json.Nodes.JsonObject> Storage() => await (await root.GetAsync("/clientapi/user/storage", Token)).JsonBody();
|
||||
Assert.Equal(0, Storage().Result["mediaBytes"]!.GetValue<long>());
|
||||
|
||||
var id = await Upload(alice, "a.jpg");
|
||||
var stored = await Storage();
|
||||
Assert.True(stored["mediaBytes"]!.GetValue<long>() > 0);
|
||||
Assert.Equal(1, stored["mediaFiles"]!.GetValue<long>());
|
||||
Assert.Null(stored["quotaBytes"]);
|
||||
|
||||
await _host.Get<IMediaService>().Trash(m => m.ID == id, "test", Token);
|
||||
Assert.Equal(0, (await Storage())["mediaFiles"]!.GetValue<long>());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task The_audit_adopts_todays_pictures_and_trashes_what_nothing_holds()
|
||||
{
|
||||
var media = _host.Get<IMediaService>();
|
||||
var bob = await _host.Mastodon($"bob{Guid.NewGuid():N}"[..12]);
|
||||
// a picture shown from before pictures had rows, and a leftover nothing holds
|
||||
var picture = $"2020/01/{Guid.NewGuid():N}.jpg";
|
||||
var leftover = $"2020/01/{Guid.NewGuid():N}.jpg";
|
||||
foreach (var relative in new[] { picture, leftover })
|
||||
{
|
||||
Directory.CreateDirectory(Path.GetDirectoryName(Path.Combine(media.Root, relative))!);
|
||||
await File.WriteAllBytesAsync(Path.Combine(media.Root, relative), new byte[] { 1, 2, 3 }, Token);
|
||||
}
|
||||
await DB.Default.Update<Models.User.Avatar>().MatchID(bob.Persona.Id).Modify(a => a.PictureURL, media.Url(picture)).ExecuteAsync(Token);
|
||||
|
||||
var looked = await MediaAudit.Run(media, fix: false, Token);
|
||||
Assert.True(looked.Adopted >= 1);
|
||||
Assert.True(looked.Unheld >= 1);
|
||||
Assert.True(File.Exists(Path.Combine(media.Root, leftover)));
|
||||
|
||||
await MediaAudit.Run(media, fix: true, Token);
|
||||
|
||||
Assert.True(await DB.Default.Find<MediaAttachment>().Match(m => m.ProfileOfAvatarId == bob.Persona.Id && m.FilePath == picture && m.TrashedAt == null).ExecuteAnyAsync(Token));
|
||||
Assert.Equal(HttpStatusCode.OK, await Served(media.Url(picture)));
|
||||
Assert.Equal(HttpStatusCode.NotFound, await Served(media.Url(leftover)));
|
||||
Assert.True(File.Exists(Path.Combine(media.TrashRoot, leftover)));
|
||||
Assert.True(await DB.Default.Find<MediaAttachment>().Match(m => m.FilePath == leftover && m.TrashedAt != null).ExecuteAnyAsync(Token));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,346 @@
|
||||
using MongoDB.Entities;
|
||||
|
||||
using PrivaPub.Models.Jobs;
|
||||
using PrivaPub.Models.Media;
|
||||
using PrivaPub.Models.Social;
|
||||
using PrivaPub.Models.User;
|
||||
using PrivaPub.Tests.Support;
|
||||
using PrivaPub.Tests.Support.Host;
|
||||
|
||||
using System.IO.Compression;
|
||||
using System.Net;
|
||||
using System.Net.Http.Json;
|
||||
using System.Text;
|
||||
using System.Text.Json.Nodes;
|
||||
|
||||
using PostEntity = PrivaPub.Models.Post.Post;
|
||||
|
||||
namespace PrivaPub.Tests.Http
|
||||
{
|
||||
// A persona's archive (owner decision 2026-10-07): exported in Mastodon's layout with nothing of its root, its siblings
|
||||
// or its keys; imported back home without bringing anything back, or into another persona as posts nobody is told of,
|
||||
// once; Mastodon's own archives read too; and hostile ones refused.
|
||||
[Trait("Category", "Integration")]
|
||||
public sealed class PersonaArchiveTests : IAsyncLifetime
|
||||
{
|
||||
PrivaPubHost _host;
|
||||
|
||||
static CancellationToken Token => TestContext.Current.CancellationToken;
|
||||
|
||||
public async ValueTask InitializeAsync()
|
||||
{
|
||||
Assert.SkipUnless(MongoFixture.Enabled, MongoFixture.Skip);
|
||||
_host = await PrivaPubHost.Shared();
|
||||
}
|
||||
|
||||
public ValueTask DisposeAsync() => ValueTask.CompletedTask;
|
||||
|
||||
static string Base(Persona persona) => $"/clientapi/persona/{persona.Id}/archive";
|
||||
|
||||
async Task<JsonObject> Status(Persona persona)
|
||||
{
|
||||
using var client = _host.As(persona.Root.Jwt);
|
||||
return (await client.GetFromJsonAsync<JsonObject>(Base(persona), Token))!;
|
||||
}
|
||||
|
||||
async Task<byte[]> Exported(Persona persona)
|
||||
{
|
||||
using var client = _host.As(persona.Root.Jwt);
|
||||
Assert.Equal(HttpStatusCode.Accepted, (await client.PostAsync(Base(persona), null, Token)).StatusCode);
|
||||
await _host.Run(j => j.Kind == JobKind.ExportArchive && j.Payload == persona.Id, Token);
|
||||
Assert.Equal("ready", (await Status(persona))["exportState"]!.GetValue<string>());
|
||||
var ticket = (await (await client.PostAsync(Base(persona) + "/ticket", null, Token)).Content.ReadFromJsonAsync<JsonObject>(Token))!;
|
||||
using var anonymous = _host.Client();
|
||||
var response = await anonymous.GetAsync(ticket["url"]!.GetValue<string>(), Token);
|
||||
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
|
||||
return await response.Content.ReadAsByteArrayAsync(Token);
|
||||
}
|
||||
|
||||
// uploaded in two pieces, then imported with the parts asked for, the job run: the import's state
|
||||
async Task<(HttpStatusCode Status, JsonObject State)> Imported(Persona persona, byte[] archive, params string[] parts)
|
||||
{
|
||||
using var client = _host.As(persona.Root.Jwt);
|
||||
var upload = (await (await client.PostAsync("/clientapi/persona/archive/uploads", null, Token)).Content.ReadFromJsonAsync<JsonObject>(Token))!["id"]!.GetValue<string>();
|
||||
var half = archive.Length / 2;
|
||||
Assert.Equal(HttpStatusCode.OK, (await client.PutAsync($"/clientapi/persona/archive/uploads/{upload}?offset=0", new ByteArrayContent(archive[..half]), Token)).StatusCode);
|
||||
Assert.Equal(HttpStatusCode.OK, (await client.PutAsync($"/clientapi/persona/archive/uploads/{upload}?offset={half}", new ByteArrayContent(archive[half..]), Token)).StatusCode);
|
||||
var asked = await client.PostAsJsonAsync(Base(persona) + "/import", new { uploadId = upload, parts }, Token);
|
||||
if (asked.StatusCode != HttpStatusCode.Accepted)
|
||||
return (asked.StatusCode, default);
|
||||
await _host.Run(j => j.Kind == JobKind.ImportArchive && j.Payload == persona.Id, Token);
|
||||
return (asked.StatusCode, await Status(persona));
|
||||
}
|
||||
|
||||
static int Count(JsonObject state, string what) => state["importCounts"]?[what]?.GetValue<int>() ?? 0;
|
||||
|
||||
static Dictionary<string, string> Texts(byte[] zip)
|
||||
{
|
||||
using var archive = new ZipArchive(new MemoryStream(zip));
|
||||
return archive.Entries.Where(e => e.Name.EndsWith(".json") || e.Name.EndsWith(".csv"))
|
||||
.ToDictionary(e => e.FullName, e => new StreamReader(e.Open()).ReadToEnd());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task An_archive_holds_the_persona_and_nothing_of_its_root_its_siblings_or_its_keys()
|
||||
{
|
||||
var alice = await _host.Mastodon("alice");
|
||||
var sibling = await _host.Persona(alice.Persona.Root, "sibling");
|
||||
var bob = await _host.Mastodon("bob");
|
||||
var picture = (await alice.Client.Exchange(new HttpRequestMessage(HttpMethod.Post, "/api/v2/media")
|
||||
{
|
||||
Content = MastodonHelpers.Multipart(("file", MastodonHelpers.JpegWithMetadata(64, 48), "image/jpeg", "dot.jpg"))
|
||||
})).Ok().Body.Text("id");
|
||||
var first = await alice.Status("a public post with a picture #archives", ("media_ids[]", picture));
|
||||
await alice.Status("for my followers only", ("visibility", "private"));
|
||||
await alice.Status("a reply to myself", ("in_reply_to_id", first.Text("id")));
|
||||
await alice.Client.Post($"/api/v1/statuses/{first.Text("id")}/bookmark");
|
||||
await alice.Client.Post($"/api/v1/statuses/{first.Text("id")}/pin");
|
||||
await alice.Client.Post($"/api/v1/accounts/{bob.Persona.Id}/block");
|
||||
await alice.Client.Post("/api/v1/domain_blocks", ("domain", "spam.example"));
|
||||
await alice.Client.Post("/api/v1/tags/archives/follow");
|
||||
|
||||
var zip = await Exported(alice.Persona);
|
||||
var texts = Texts(zip);
|
||||
|
||||
foreach (var file in new[] { "actor.json", "outbox.json", "likes.json", "bookmarks.json", "following_accounts.csv", "blocked_accounts.csv",
|
||||
"muted_accounts.csv", "blocked_domains.csv", "lists.csv", "bookmarks.csv", "privapub/archive.json", "privapub/pins.json",
|
||||
"privapub/followed_tags.json", "privapub/filters.json", "privapub/notification_policy.json" })
|
||||
Assert.True(texts.ContainsKey(file), $"{file} is missing");
|
||||
using (var archive = new ZipArchive(new MemoryStream(zip)))
|
||||
Assert.Contains(archive.Entries, e => e.FullName.StartsWith("media_attachments/files/", StringComparison.Ordinal));
|
||||
var outbox = JsonNode.Parse(texts["outbox.json"])!["orderedItems"]!.AsArray();
|
||||
Assert.Equal(3, outbox.Count);
|
||||
Assert.Contains(outbox, i => i!["object"]!["attachment"]?[0]?["url"]?.GetValue<string>().StartsWith("/media_attachments/files/") == true);
|
||||
Assert.Contains(bob.Persona.UserName, texts["blocked_accounts.csv"]);
|
||||
Assert.Contains("spam.example", texts["blocked_domains.csv"]);
|
||||
Assert.Contains(first.Text("uri"), texts["privapub/pins.json"]);
|
||||
|
||||
var avatar = await DB.Default.Find<Avatar>().MatchID(alice.Persona.Id).ExecuteFirstAsync(Token);
|
||||
var root = await DB.Default.Find<RootUser>().MatchID(alice.Persona.Root.Id).ExecuteFirstAsync(Token);
|
||||
var all = string.Concat(texts.Values);
|
||||
Assert.DoesNotContain(avatar.PrivateKey.Split('\n')[1], all);
|
||||
Assert.DoesNotContain(avatar.SigningKey, all);
|
||||
Assert.DoesNotContain(root.UserName, all);
|
||||
Assert.DoesNotContain(root.HashedPassword, all);
|
||||
Assert.DoesNotContain(sibling.UserName, all);
|
||||
Assert.DoesNotContain(alice.Token, all);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Back_home_an_archive_brings_nothing_back()
|
||||
{
|
||||
var alice = await _host.Mastodon("alice");
|
||||
await alice.Status("kept");
|
||||
var regretted = await alice.Status("regretted");
|
||||
var zip = await Exported(alice.Persona);
|
||||
await alice.Client.Delete($"/api/v1/statuses/{regretted.Text("id")}");
|
||||
var before = await DB.Default.CountAsync<PostEntity>(p => p.GroupUserId == alice.Persona.Id, Token);
|
||||
|
||||
var (status, state) = await Imported(alice.Persona, zip, "posts", "pins");
|
||||
|
||||
Assert.Equal(HttpStatusCode.Accepted, status);
|
||||
Assert.Equal("done", state["importState"]!.GetValue<string>());
|
||||
Assert.True(state["importSameActor"]!.GetValue<bool>());
|
||||
Assert.Equal((0, 1, 1), (Count(state, "posts"), Count(state, "posts already here"), Count(state, "posts deleted here")));
|
||||
Assert.Equal(before, await DB.Default.CountAsync<PostEntity>(p => p.GroupUserId == alice.Persona.Id, Token));
|
||||
Assert.NotNull((await DB.Default.Find<PostEntity>().MatchID(regretted.Text("id")).ExecuteFirstAsync(Token)).DeletedAt);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Into_another_persona_posts_arrive_once_and_nobody_is_told()
|
||||
{
|
||||
var alice = await _host.Mastodon("alice");
|
||||
var follower = await _host.Mastodon("follower");
|
||||
var picture = (await alice.Client.Exchange(new HttpRequestMessage(HttpMethod.Post, "/api/v2/media")
|
||||
{
|
||||
Content = MastodonHelpers.Multipart(("file", MastodonHelpers.JpegWithMetadata(64, 48), "image/jpeg", "dot.jpg"))
|
||||
})).Ok().Body.Text("id");
|
||||
var first = await alice.Status("the first words #moving", ("media_ids[]", picture));
|
||||
await alice.Status("and a reply to them", ("in_reply_to_id", first.Text("id")));
|
||||
await _host.Located(alice, "here and nowhere else");
|
||||
var zip = await Exported(alice.Persona);
|
||||
|
||||
var carol = await _host.Mastodon("carol");
|
||||
await follower.Client.Post($"/api/v1/accounts/{carol.Persona.Id}/follow");
|
||||
var actor = $"/peasants/{carol.Persona.UserName}";
|
||||
var jobs = await DB.Default.CountAsync<Job>(j => j.Kind == JobKind.Deliver && j.Payload.Contains(actor), Token);
|
||||
var (_, state) = await Imported(carol.Persona, zip, "posts", "pins", "tags", "located");
|
||||
|
||||
Assert.Equal("done", state["importState"]!.GetValue<string>());
|
||||
Assert.False(state["importSameActor"]!.GetValue<bool>());
|
||||
Assert.Equal(2, Count(state, "posts"));
|
||||
Assert.Equal(1, Count(state, "media"));
|
||||
Assert.Equal(1, Count(state, "located posts"));
|
||||
var copies = await DB.Default.Find<PostEntity>().Match(p => p.GroupUserId == carol.Persona.Id && p.Visibility != Models.Post.PostVisibility.LocalGeo).ExecuteAsync(Token);
|
||||
Assert.Equal(2, copies.Count);
|
||||
var copy = copies.Single(p => p.ImportedFromURI == first.Text("uri"));
|
||||
Assert.StartsWith($"{PrivaPubHost.Base}/peasants/{carol.Persona.UserName}/scribbles/", copy.ObjectURI);
|
||||
Assert.Equal(["moving"], copy.Tags);
|
||||
Assert.Single(copy.Media);
|
||||
Assert.Equal(carol.Persona.Id, (await DB.Default.Find<MediaAttachment>().MatchID(copy.Media[0].AttachmentId).ExecuteFirstAsync(Token)).OwnerAvatarId);
|
||||
var reply = copies.Single(p => p.ID != copy.ID);
|
||||
Assert.Equal(copy.ID, reply.AnsweringToPostId);
|
||||
Assert.Equal(copy.ObjectURI + "/context", reply.ContextURI);
|
||||
|
||||
// nobody told: no delivery, no home, no notification; on the profile all the same
|
||||
Assert.Equal(jobs, await DB.Default.CountAsync<Job>(j => j.Kind == JobKind.Deliver && j.Payload.Contains(actor), Token));
|
||||
var ids = copies.Select(p => p.ID).ToList();
|
||||
Assert.False(await DB.Default.Find<TimelineEntry>().Match(e => ids.Contains(e.PostId)).ExecuteAnyAsync(Token));
|
||||
Assert.Contains((await carol.Client.Get($"/api/v1/accounts/{carol.Persona.Id}/statuses")).Ok().Body!.AsArray(), s => s!.Text("id") == copy.ID);
|
||||
|
||||
// twice changes nothing
|
||||
var (_, again) = await Imported(carol.Persona, zip, "posts", "located");
|
||||
Assert.Equal((0, 2, 1), (Count(again, "posts"), Count(again, "posts already here"), Count(again, "located posts already here")));
|
||||
Assert.Equal(3, await DB.Default.CountAsync<PostEntity>(p => p.GroupUserId == carol.Persona.Id, Token));
|
||||
}
|
||||
|
||||
// Mastodon's own archive: its posts, not its boosts nor its direct messages
|
||||
[Fact]
|
||||
public async Task A_mastodon_archive_is_read()
|
||||
{
|
||||
var dave = await _host.Mastodon("dave");
|
||||
const string actor = "https://mastodon.example/users/dave";
|
||||
const string Public = "https://www.w3.org/ns/activitystreams#Public";
|
||||
var outbox = new JsonObject
|
||||
{
|
||||
["@context"] = "https://www.w3.org/ns/activitystreams",
|
||||
["id"] = "outbox.json",
|
||||
["type"] = "OrderedCollection",
|
||||
["orderedItems"] = new JsonArray(
|
||||
Create(actor, "1", "<p>hello from <b>Mastodon</b><script>alert(1)</script></p>", [Public], [actor + "/followers"], "2023-04-01T10:00:00Z"),
|
||||
Create(actor, "2", "<p>only for one</p>", ["https://elsewhere.example/users/x"], [], "2023-04-02T10:00:00Z"),
|
||||
new JsonObject { ["type"] = "Announce", ["actor"] = actor, ["object"] = "https://elsewhere.example/notes/9" })
|
||||
};
|
||||
var zip = Zip(("actor.json", new JsonObject { ["id"] = actor, ["type"] = "Person", ["followers"] = actor + "/followers" }.ToJsonString()),
|
||||
("outbox.json", outbox.ToJsonString()));
|
||||
|
||||
var (_, state) = await Imported(dave.Persona, zip, "posts");
|
||||
|
||||
Assert.Equal("done", state["importState"]!.GetValue<string>());
|
||||
Assert.Equal((1, 1, 1), (Count(state, "posts"), Count(state, "boosts skipped"), Count(state, "direct or circle posts skipped")));
|
||||
var post = await DB.Default.Find<PostEntity>().Match(p => p.GroupUserId == dave.Persona.Id).ExecuteFirstAsync(Token);
|
||||
Assert.Equal(actor + "/statuses/1", post.ImportedFromURI);
|
||||
Assert.Equal(new DateTime(2023, 4, 1, 10, 0, 0, DateTimeKind.Utc), post.CreationDate);
|
||||
Assert.DoesNotContain("<script", post.ContentHtml);
|
||||
Assert.Contains("Mastodon", post.ContentHtml);
|
||||
}
|
||||
|
||||
static JsonObject Create(string actor, string id, string content, string[] to, string[] cc, string published) => new()
|
||||
{
|
||||
["type"] = "Create",
|
||||
["actor"] = actor,
|
||||
["object"] = new JsonObject
|
||||
{
|
||||
["id"] = $"{actor}/statuses/{id}",
|
||||
["type"] = "Note",
|
||||
["attributedTo"] = actor,
|
||||
["content"] = content,
|
||||
["published"] = published,
|
||||
["to"] = new JsonArray(to.Select(t => (JsonNode)t).ToArray()),
|
||||
["cc"] = new JsonArray(cc.Select(c => (JsonNode)c).ToArray())
|
||||
}
|
||||
};
|
||||
|
||||
static byte[] Zip(params (string Name, string Text)[] files)
|
||||
{
|
||||
var stream = new MemoryStream();
|
||||
using (var archive = new ZipArchive(stream, ZipArchiveMode.Create, leaveOpen: true))
|
||||
foreach (var (name, text) in files)
|
||||
{
|
||||
using var entry = archive.CreateEntry(name).Open();
|
||||
entry.Write(Encoding.UTF8.GetBytes(text));
|
||||
}
|
||||
return stream.ToArray();
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("escape")]
|
||||
[InlineData("twice")]
|
||||
[InlineData("bomb")]
|
||||
[InlineData("link")]
|
||||
[InlineData("junk")]
|
||||
public async Task A_hostile_archive_is_refused(string how)
|
||||
{
|
||||
var erin = await _host.Mastodon("erin");
|
||||
var actorJson = new JsonObject { ["id"] = "https://elsewhere.example/users/erin" }.ToJsonString();
|
||||
var zip = how switch
|
||||
{
|
||||
"escape" => Zip(("actor.json", actorJson), ("../escape.txt", "x")),
|
||||
"twice" => Zip(("actor.json", actorJson), ("outbox.json", "{}"), ("outbox.json", "{}")),
|
||||
"bomb" => Zip(("actor.json", actorJson), ("outbox.json", new string(' ', 30 * 1024 * 1024))),
|
||||
"link" => Link(actorJson),
|
||||
_ => "not a zip at all"u8.ToArray()
|
||||
};
|
||||
|
||||
var (status, _) = await Imported(erin.Persona, zip, "posts");
|
||||
|
||||
Assert.Equal(HttpStatusCode.UnprocessableEntity, status);
|
||||
Assert.False(await DB.Default.Find<PostEntity>().Match(p => p.GroupUserId == erin.Persona.Id).ExecuteAnyAsync(Token));
|
||||
}
|
||||
|
||||
// a symbolic link, as zip on Unix stores one
|
||||
static byte[] Link(string actorJson)
|
||||
{
|
||||
var stream = new MemoryStream();
|
||||
using (var archive = new ZipArchive(stream, ZipArchiveMode.Create, leaveOpen: true))
|
||||
{
|
||||
using (var actor = archive.CreateEntry("actor.json").Open())
|
||||
actor.Write(Encoding.UTF8.GetBytes(actorJson));
|
||||
var link = archive.CreateEntry("media_attachments/files/passwd");
|
||||
link.ExternalAttributes = unchecked((int)(0xA1FFu << 16));
|
||||
using var target = link.Open();
|
||||
target.Write("/etc/passwd"u8);
|
||||
}
|
||||
return stream.ToArray();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task An_outbox_item_too_large_stops_the_import()
|
||||
{
|
||||
var frank = await _host.Mastodon("frank");
|
||||
const string actor = "https://elsewhere.example/users/frank";
|
||||
// (random, so it is no compression bomb: that is refused before the import starts)
|
||||
var words = Convert.ToBase64String(System.Security.Cryptography.RandomNumberGenerator.GetBytes(1536 * 1024));
|
||||
var huge = Create(actor, "1", words, ["https://www.w3.org/ns/activitystreams#Public"], [], "2024-01-01T00:00:00Z");
|
||||
var zip = Zip(("actor.json", new JsonObject { ["id"] = actor }.ToJsonString()),
|
||||
("outbox.json", new JsonObject { ["orderedItems"] = new JsonArray(huge) }.ToJsonString()));
|
||||
|
||||
var (_, state) = await Imported(frank.Persona, zip, "posts");
|
||||
|
||||
Assert.Equal("failed", state["importState"]!.GetValue<string>());
|
||||
Assert.Contains("larger than", state["importError"]!.GetValue<string>());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Only_the_personas_own_root_and_never_a_banned_one()
|
||||
{
|
||||
var grace = await _host.Mastodon("grace");
|
||||
var stranger = await _host.SignUp("stranger");
|
||||
using (var other = _host.As(stranger.Jwt))
|
||||
Assert.Equal(HttpStatusCode.NotFound, (await other.GetAsync(Base(grace.Persona), Token)).StatusCode);
|
||||
Assert.Equal(HttpStatusCode.OK, (await _host.As(grace.Persona.Root.Jwt).GetAsync(Base(grace.Persona), Token)).StatusCode);
|
||||
await DB.Default.Update<RootUser>().MatchID(grace.Persona.Root.Id).Modify(u => u.IsBanned, true).ExecuteAsync(Token);
|
||||
Assert.NotEqual(HttpStatusCode.OK, (await _host.As(grace.Persona.Root.Jwt).GetAsync(Base(grace.Persona), Token)).StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task An_import_asked_to_stop_stops()
|
||||
{
|
||||
var henry = await _host.Mastodon("henry");
|
||||
const string actor = "https://elsewhere.example/users/henry";
|
||||
var zip = Zip(("actor.json", new JsonObject { ["id"] = actor }.ToJsonString()),
|
||||
("outbox.json", new JsonObject { ["orderedItems"] = new JsonArray(Create(actor, "1", "one", ["https://www.w3.org/ns/activitystreams#Public"], [], "2024-01-01T00:00:00Z")) }.ToJsonString()));
|
||||
using var client = _host.As(henry.Persona.Root.Jwt);
|
||||
var upload = (await (await client.PostAsync("/clientapi/persona/archive/uploads", null, Token)).Content.ReadFromJsonAsync<JsonObject>(Token))!["id"]!.GetValue<string>();
|
||||
await client.PutAsync($"/clientapi/persona/archive/uploads/{upload}?offset=0", new ByteArrayContent(zip), Token);
|
||||
Assert.Equal(HttpStatusCode.Accepted, (await client.PostAsJsonAsync(Base(henry.Persona) + "/import", new { uploadId = upload, parts = new[] { "posts" } }, Token)).StatusCode);
|
||||
Assert.Equal(HttpStatusCode.Conflict, (await client.PostAsJsonAsync(Base(henry.Persona) + "/import", new { uploadId = upload, parts = new[] { "posts" } }, Token)).StatusCode);
|
||||
Assert.Equal(HttpStatusCode.OK, (await client.PostAsync(Base(henry.Persona) + "/import/stop", null, Token)).StatusCode);
|
||||
|
||||
await _host.Run(j => j.Kind == JobKind.ImportArchive && j.Payload == henry.Persona.Id, Token);
|
||||
|
||||
Assert.Equal("stopped", (await Status(henry.Persona))["importState"]!.GetValue<string>());
|
||||
Assert.False(await DB.Default.Find<PostEntity>().Match(p => p.GroupUserId == henry.Persona.Id).ExecuteAnyAsync(Token));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,287 @@
|
||||
using MongoDB.Bson;
|
||||
using MongoDB.Bson.IO;
|
||||
using MongoDB.Driver;
|
||||
using MongoDB.Entities;
|
||||
|
||||
using PrivaPub.Infrastructure.Backup;
|
||||
using PrivaPub.Infrastructure.Cli;
|
||||
using PrivaPub.Tests.Support;
|
||||
using PrivaPub.Tests.Support.Host;
|
||||
|
||||
using System.IO.Compression;
|
||||
|
||||
namespace PrivaPub.Tests.Infrastructure
|
||||
{
|
||||
// The server's backup: every collection as it was, byte for byte, read at one instant on a replica set, without what
|
||||
// belongs to the moment (the statistics salt above all), with the media rows' files linked, checkable and rotated.
|
||||
// Each test backs up a database of its own; alone, since the maintenance lock is the server's one.
|
||||
[Trait("Category", "Integration")]
|
||||
[Xunit.Collection(nameof(Exclusive))]
|
||||
public sealed class BackupTests : IAsyncLifetime
|
||||
{
|
||||
readonly string _scratch = Path.Combine(Path.GetTempPath(), $"privapub-backup-tests-{Guid.NewGuid():N}");
|
||||
IMongoDatabase _database;
|
||||
|
||||
string Backups => Path.Combine(_scratch, "backups");
|
||||
string Media => Path.Combine(_scratch, "media");
|
||||
string Trash => Path.Combine(_scratch, "media-trash");
|
||||
|
||||
public async ValueTask InitializeAsync()
|
||||
{
|
||||
Assert.SkipUnless(MongoFixture.Enabled, MongoFixture.Skip);
|
||||
await PrivaPubHost.Shared();
|
||||
_database = DB.Default.Database().Client.GetDatabase($"PrivaPubBackup_{Guid.NewGuid():N}");
|
||||
Directory.CreateDirectory(Media);
|
||||
Directory.CreateDirectory(Trash);
|
||||
}
|
||||
|
||||
public async ValueTask DisposeAsync()
|
||||
{
|
||||
if (_database != default)
|
||||
await _database.Client.DropDatabaseAsync(_database.DatabaseNamespace.DatabaseName);
|
||||
if (Directory.Exists(_scratch))
|
||||
Directory.Delete(_scratch, recursive: true);
|
||||
}
|
||||
|
||||
BackupContext Context(BackupOptions options = default) => new(_database, Backups, Media, Trash, "https://privapub.test", options ?? new BackupOptions());
|
||||
|
||||
static CancellationToken Token => TestContext.Current.CancellationToken;
|
||||
|
||||
static List<BsonDocument> Read(string file)
|
||||
{
|
||||
using var gzip = new GZipStream(File.OpenRead(file), CompressionMode.Decompress);
|
||||
using var reader = new StreamReader(gzip);
|
||||
var documents = new List<BsonDocument>();
|
||||
while (reader.ReadLine() is { } line)
|
||||
documents.Add(BsonDocument.Parse(line));
|
||||
return documents;
|
||||
}
|
||||
|
||||
async Task<List<byte[]>> Raw(string collection) =>
|
||||
(await (await _database.GetCollection<RawBsonDocument>(collection).FindAsync(FilterDefinition<RawBsonDocument>.Empty, cancellationToken: Token)).ToListAsync(Token))
|
||||
.Select(d =>
|
||||
{
|
||||
var bytes = new byte[d.Slice.Length];
|
||||
d.Slice.GetBytes(0, bytes, 0, bytes.Length);
|
||||
return bytes;
|
||||
})
|
||||
.ToList();
|
||||
|
||||
[Fact]
|
||||
public async Task Every_document_comes_back_byte_for_byte_and_the_salt_never_leaves()
|
||||
{
|
||||
var odd = new BsonDocument
|
||||
{
|
||||
{ "_id", ObjectId.GenerateNewId() },
|
||||
{ "Guid", new BsonBinaryData(Guid.NewGuid(), GuidRepresentation.Standard) },
|
||||
{ "OldGuid", new BsonBinaryData(new byte[16], BsonBinarySubType.UuidLegacy) },
|
||||
{ "Money", new BsonDecimal128(Decimal128.Parse("12345.678900")) },
|
||||
{ "Long", new BsonInt64(long.MaxValue) },
|
||||
{ "Int", 7 },
|
||||
{ "Double", -0.0 },
|
||||
{ "NaN", double.NaN },
|
||||
{ "At", new BsonDateTime(new DateTime(2026, 10, 7, 3, 30, 0, 123, DateTimeKind.Utc)) },
|
||||
{ "Stamp", new BsonTimestamp(1, 2) },
|
||||
{ "Null", BsonNull.Value },
|
||||
{ "Regex", new BsonRegularExpression("^a.b$", "i") },
|
||||
{ "Nested", new BsonDocument { { "z", 1 }, { "a", new BsonArray { 1, "two", new BsonDocument("three", 3) } } } },
|
||||
{ "Unicode", "città 🌍 \u0000 end" }
|
||||
};
|
||||
// an ObjectRecord as big as a remote's long post gets
|
||||
var big = new BsonDocument { { "_id", ObjectId.GenerateNewId() }, { "Json", new string('x', 10 * 1024 * 1024) } };
|
||||
await _database.GetCollection<BsonDocument>("Odd").InsertOneAsync(odd, cancellationToken: Token);
|
||||
await _database.GetCollection<BsonDocument>("ObjectRecord").InsertOneAsync(big, cancellationToken: Token);
|
||||
await _database.GetCollection<BsonDocument>("Odd").Indexes.CreateOneAsync(
|
||||
new CreateIndexModel<BsonDocument>(Builders<BsonDocument>.IndexKeys.Ascending("At"), new CreateIndexOptions { Name = "at", Unique = true }), cancellationToken: Token);
|
||||
await _database.GetCollection<BsonDocument>("InteractionSalt").InsertOneAsync(new BsonDocument("Salt", "never in a backup"), cancellationToken: Token);
|
||||
await _database.GetCollection<BsonDocument>("Job").InsertOneAsync(new BsonDocument("Kind", "Deliver"), cancellationToken: Token);
|
||||
await _database.GetCollection<BsonDocument>("_migration_history_").InsertManyAsync([
|
||||
new BsonDocument { { "Number", 15 }, { "Name", "older" } }, new BsonDocument { { "Number", 16 }, { "Name", "focal points are finite" } }], cancellationToken: Token);
|
||||
|
||||
var (backup, error) = await ServerBackup.Create(Context(), "manual", dbOnly: false, Token);
|
||||
|
||||
Assert.Null(error);
|
||||
var directory = Path.Combine(Backups, backup.Id);
|
||||
Assert.False(Directory.Exists(directory + ServerBackup.PartialSuffix));
|
||||
Assert.Equal(await Raw("Odd"), Read(Path.Combine(directory, "db", "Odd.jsonl.gz")).Select(d => d.ToBson()).ToList());
|
||||
Assert.Equal(await Raw("ObjectRecord"), Read(Path.Combine(directory, "db", "ObjectRecord.jsonl.gz")).Select(d => d.ToBson()).ToList());
|
||||
|
||||
Assert.False(File.Exists(Path.Combine(directory, "db", "InteractionSalt.jsonl.gz")));
|
||||
Assert.False(File.Exists(Path.Combine(directory, "db", "Job.jsonl.gz")));
|
||||
foreach (var file in Directory.EnumerateFiles(directory, "*", SearchOption.AllDirectories))
|
||||
Assert.DoesNotContain("never in a backup", await ReadAll(file));
|
||||
Assert.Contains(backup.Manifest.Excluded, e => e.Name == "InteractionSalt");
|
||||
|
||||
var odds = backup.Manifest.Collections.Single(c => c.Name == "Odd");
|
||||
Assert.Equal(1, odds.Count);
|
||||
Assert.Contains(odds.Indexes, i => BsonDocument.Parse(i)["name"] == "at" && BsonDocument.Parse(i)["unique"].ToBoolean());
|
||||
Assert.Equal((16, "focal points are finite"), (backup.Manifest.MigrationNumber, backup.Manifest.NewestMigration));
|
||||
Assert.Equal("https://privapub.test", backup.Manifest.Host);
|
||||
Assert.Equal(MongoFixture.Connection.Contains("directConnection=true"), backup.Manifest.Consistent);
|
||||
Assert.Empty(await ServerBackup.Verify(Backups, backup.Id, Token));
|
||||
if (!OperatingSystem.IsWindows())
|
||||
{
|
||||
Assert.Equal(UnixFileMode.UserRead | UnixFileMode.UserWrite | UnixFileMode.GroupRead, File.GetUnixFileMode(Path.Combine(directory, "db", "Odd.jsonl.gz")));
|
||||
Assert.False(File.GetUnixFileMode(directory).HasFlag(UnixFileMode.OtherRead));
|
||||
}
|
||||
}
|
||||
|
||||
static async Task<string> ReadAll(string file)
|
||||
{
|
||||
if (!file.EndsWith(".gz", StringComparison.Ordinal))
|
||||
return await File.ReadAllTextAsync(file, Token);
|
||||
await using var gzip = new GZipStream(File.OpenRead(file), CompressionMode.Decompress);
|
||||
using var reader = new StreamReader(gzip);
|
||||
return await reader.ReadToEndAsync(Token);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Media_rows_files_are_linked_from_the_live_directory_or_the_trash()
|
||||
{
|
||||
Directory.CreateDirectory(Path.Combine(Media, "2026", "10"));
|
||||
await File.WriteAllTextAsync(Path.Combine(Media, "2026", "10", "live.jpg"), "live", Token);
|
||||
await File.WriteAllTextAsync(Path.Combine(Media, "2026", "10", "live-preview.jpg"), "preview", Token);
|
||||
Directory.CreateDirectory(Path.Combine(Trash, "2026", "10"));
|
||||
await File.WriteAllTextAsync(Path.Combine(Trash, "2026", "10", "moving.jpg"), "moving", Token);
|
||||
await File.WriteAllTextAsync(Path.Combine(Media, "2026", "10", "trashed.jpg"), "trashed", Token);
|
||||
await _database.GetCollection<BsonDocument>("MediaAttachment").InsertManyAsync([
|
||||
new BsonDocument { { "FilePath", "2026/10/live.jpg" }, { "PreviewPath", "2026/10/live-preview.jpg" }, { "TrashedAt", BsonNull.Value } },
|
||||
new BsonDocument { { "FilePath", "2026/10/moving.jpg" } },//a file the janitor moved while its row was being trashed
|
||||
new BsonDocument { { "FilePath", "2026/10/gone.jpg" } },
|
||||
new BsonDocument { { "FilePath", "2026/10/trashed.jpg" }, { "TrashedAt", DateTime.UtcNow } },
|
||||
new BsonDocument { { "FilePath", "../../etc/passwd" } }], cancellationToken: Token);
|
||||
|
||||
var (backup, _) = await ServerBackup.Create(Context(), "manual", dbOnly: false, Token);
|
||||
|
||||
var media = Path.Combine(Backups, backup.Id, "media");
|
||||
Assert.Equal("live", await File.ReadAllTextAsync(Path.Combine(media, "2026", "10", "live.jpg"), Token));
|
||||
Assert.Equal("preview", await File.ReadAllTextAsync(Path.Combine(media, "2026", "10", "live-preview.jpg"), Token));
|
||||
Assert.Equal("moving", await File.ReadAllTextAsync(Path.Combine(media, "2026", "10", "moving.jpg"), Token));
|
||||
Assert.False(File.Exists(Path.Combine(media, "2026", "10", "trashed.jpg")));
|
||||
Assert.Equal(["2026/10/live-preview.jpg", "2026/10/live.jpg", "2026/10/moving.jpg"], backup.Manifest.Media.List);
|
||||
Assert.Equal((3, 1), (backup.Manifest.Media.Files, backup.Manifest.Media.Missing));
|
||||
|
||||
// a link, not a copy: the live file deleted, the backup's stays
|
||||
File.Delete(Path.Combine(Media, "2026", "10", "live.jpg"));
|
||||
Assert.Equal("live", await File.ReadAllTextAsync(Path.Combine(media, "2026", "10", "live.jpg"), Token));
|
||||
|
||||
// db-only lists them and links none
|
||||
var (listed, _) = await ServerBackup.Create(Context(), "pre-deploy", dbOnly: true, Token);
|
||||
Assert.False(Directory.Exists(Path.Combine(Backups, listed.Id, "media")));
|
||||
Assert.Equal(["2026/10/live-preview.jpg", "2026/10/moving.jpg"], listed.Manifest.Media.List);
|
||||
Assert.Empty(await ServerBackup.Verify(Backups, listed.Id, Token));
|
||||
Assert.Empty(await ServerBackup.Verify(Backups, backup.Id, Token));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Verify_finds_an_altered_or_missing_file()
|
||||
{
|
||||
await _database.GetCollection<BsonDocument>("Post").InsertOneAsync(new BsonDocument("Content", "hello"), cancellationToken: Token);
|
||||
await _database.GetCollection<BsonDocument>("Avatar").InsertOneAsync(new BsonDocument("UserName", "someone"), cancellationToken: Token);
|
||||
var (backup, _) = await ServerBackup.Create(Context(), "manual", dbOnly: false, Token);
|
||||
var db = Path.Combine(Backups, backup.Id, "db");
|
||||
|
||||
await File.AppendAllTextAsync(Path.Combine(db, "Post.jsonl.gz"), "tampered", Token);
|
||||
File.Delete(Path.Combine(db, "Avatar.jsonl.gz"));
|
||||
|
||||
Assert.Equal(["Avatar: missing", "Post: altered"], (await ServerBackup.Verify(Backups, backup.Id, Token)).Order());
|
||||
Assert.Equal(["no such backup, or no manifest"], await ServerBackup.Verify(Backups, "../" + backup.Id, Token));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task One_backup_at_a_time()
|
||||
{
|
||||
await using (var held = await MaintenanceLock.Take("restore", Token))
|
||||
{
|
||||
Assert.NotNull(held);
|
||||
var (backup, error) = await ServerBackup.Create(Context(), "manual", dbOnly: true, Token);
|
||||
Assert.Null(backup);
|
||||
Assert.Contains("already running", error);
|
||||
Assert.Equal("restore", (await MaintenanceLock.Current(Token)).What);
|
||||
}
|
||||
Assert.Null(await MaintenanceLock.Current(Token));
|
||||
Assert.NotNull((await ServerBackup.Create(Context(), "manual", dbOnly: true, Token)).Backup);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_holder_that_died_is_taken_over()
|
||||
{
|
||||
await using var dead = await MaintenanceLock.Take("backup", Token);
|
||||
await DB.Default.Update<MaintenanceLock>().Match(l => l.ID == MaintenanceLock.Name)
|
||||
.Modify(l => l.Heartbeat, DateTime.UtcNow.AddMinutes(-3)).ExecuteAsync(Token);
|
||||
|
||||
Assert.Null(await MaintenanceLock.Current(Token));
|
||||
await using var alive = await MaintenanceLock.Take("restore", Token);
|
||||
Assert.NotNull(alive);
|
||||
Assert.Equal("restore", (await MaintenanceLock.Current(Token)).What);
|
||||
}
|
||||
|
||||
// the newest 7 nightly, the newest of each of the 4 weeks before, the newest 3 pre-deploy; manual ones kept
|
||||
[Fact]
|
||||
public void Rotation_keeps_days_weeks_and_the_last_deploys()
|
||||
{
|
||||
var today = new DateTime(2026, 10, 7, 3, 30, 0, DateTimeKind.Utc);
|
||||
for (var day = 0; day < 60; day++)
|
||||
Fake("nightly", today.AddDays(-day));
|
||||
for (var deploy = 0; deploy < 5; deploy++)
|
||||
Fake("pre-deploy", today.AddDays(-deploy).AddHours(5));
|
||||
Fake("manual", today.AddYears(-1));
|
||||
Directory.CreateDirectory(Path.Combine(Backups, "20260901-000000-nightly" + ServerBackup.PartialSuffix));
|
||||
Directory.SetLastWriteTimeUtc(Path.Combine(Backups, "20260901-000000-nightly" + ServerBackup.PartialSuffix), today.AddDays(-30));
|
||||
|
||||
ServerBackup.Retain(Backups, new BackupOptions());
|
||||
|
||||
var kept = ServerBackup.List(Backups);
|
||||
var nightly = kept.Where(b => b.Kind == "nightly").Select(b => b.CreatedAt).ToList();
|
||||
Assert.Equal(11, nightly.Count);
|
||||
Assert.Equal(Enumerable.Range(0, 7).Select(d => today.AddDays(-d)), nightly.Take(7));
|
||||
Assert.Equal(4, nightly.Skip(7).Select(d => System.Globalization.ISOWeek.GetWeekOfYear(d)).Distinct().Count());
|
||||
Assert.Equal(3, kept.Count(b => b.Kind == "pre-deploy"));
|
||||
Assert.Single(kept, b => b.Kind == "manual");
|
||||
Assert.Empty(Directory.EnumerateDirectories(Backups, "*" + ServerBackup.PartialSuffix));
|
||||
}
|
||||
|
||||
void Fake(string kind, DateTime at)
|
||||
{
|
||||
var directory = Path.Combine(Backups, $"{at:yyyyMMdd-HHmmss}-{kind}");
|
||||
Directory.CreateDirectory(Path.Combine(directory, "db"));
|
||||
new ArchiveManifest { Kind = kind, CreatedAt = at }.Write(directory);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("2026-10-07T03:29:00", "2026-10-06T03:31:00", false)]//yesterday's is the last one due
|
||||
[InlineData("2026-10-07T03:31:00", "2026-10-06T03:31:00", true)]
|
||||
[InlineData("2026-10-07T03:31:00", "2026-10-07T03:30:30", false)]
|
||||
[InlineData("2026-10-07T01:00:00", "2026-10-05T03:31:00", true)]//missed last night: at once
|
||||
public void A_nightly_backup_is_due_once_a_night(string now, string last, bool due) =>
|
||||
Assert.Equal(due, BackupScheduler.IsDue(DateTime.Parse(now, null, System.Globalization.DateTimeStyles.AdjustToUniversal),
|
||||
new TimeOnly(3, 30), [DateTime.Parse(last, null, System.Globalization.DateTimeStyles.AdjustToUniversal)]));
|
||||
|
||||
// the deploy's: the server's own database, through the configuration, without media links
|
||||
[Fact]
|
||||
public async Task The_deploy_backs_up_from_the_command_line()
|
||||
{
|
||||
var host = await PrivaPubHost.Shared();
|
||||
var output = new StringWriter();
|
||||
|
||||
Assert.Equal(0, await AdminCommands.Run(["backup", "--kind", "pre-deploy", "--db-only"], host.Services, output: output));
|
||||
var id = output.ToString().Split(':')[0];
|
||||
Assert.EndsWith("-pre-deploy", id);
|
||||
Assert.Equal(2, await AdminCommands.Run(["backup", "--kind", "nightly"], host.Services, output: TextWriter.Null));
|
||||
|
||||
output = new StringWriter();
|
||||
Assert.Equal(0, await AdminCommands.Run(["backups"], host.Services, output: output));
|
||||
Assert.StartsWith(id + "\tpre-deploy", output.ToString());
|
||||
output = new StringWriter();
|
||||
Assert.Equal(0, await AdminCommands.Run(["backup", "verify", id], host.Services, output: output));
|
||||
Assert.Contains("whole", output.ToString());
|
||||
|
||||
var backups = host.Get<Backups>();
|
||||
var manifest = backups.Find(id).Manifest;
|
||||
Assert.Contains(manifest.Collections, c => c.Name == "Avatar");
|
||||
Assert.DoesNotContain(manifest.Collections, c => c.Name is "InteractionSalt" or "Job" or "MaintenanceLock" or "openiddict.tokens" or "AppConfiguration");
|
||||
Assert.Equal(ServerBackup.CodeMigration(), manifest.MigrationNumber);
|
||||
Assert.True(backups.Delete(id));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,3 +1,5 @@
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
|
||||
using MongoDB.Entities;
|
||||
|
||||
using PrivaPub.Infrastructure.Jobs;
|
||||
@@ -74,6 +76,8 @@ namespace PrivaPub.Tests.Infrastructure
|
||||
Assert.Equal(JobState.Pending, after.State);
|
||||
Assert.True(after.RunAt > DateTime.UtcNow.AddSeconds(10));
|
||||
|
||||
// leased again for its third attempt (a finish only counts for the lease it was given)
|
||||
await DB.Default.Update<Job>().MatchID(leased.ID).Modify(j => j.State, JobState.Running).Modify(j => j.LeaseOwner, leased.LeaseOwner).ExecuteAsync(token);
|
||||
leased.Attempts = 3;
|
||||
await queue.Finish(leased, JobOutcome.Retry("boom"), maxAttempts: 3, token);
|
||||
Assert.Equal(JobState.Dead, (await DB.Default.Find<Job>().OneAsync(leased.ID, token)).State);
|
||||
@@ -90,5 +94,100 @@ namespace PrivaPub.Tests.Infrastructure
|
||||
|
||||
Assert.Equal(JobState.Pending, (await DB.Default.Find<Job>().OneAsync(job.ID, token)).State);
|
||||
}
|
||||
|
||||
// a handler running three times its lease keeps it: the reaper finds nothing to take back and it runs once
|
||||
[Fact]
|
||||
public async Task A_long_job_keeps_its_lease_and_runs_once()
|
||||
{
|
||||
var token = TestContext.Current.CancellationToken;
|
||||
var kind = (JobKind)(2000 + Random.Shared.Next(100000));
|
||||
var queue = new JobQueue(TimeSpan.FromSeconds(1.5), j => j.Kind == kind);
|
||||
var handler = new SlowHandler(kind, TimeSpan.FromSeconds(4.5));
|
||||
await queue.Enqueue(kind, "{}", default, default, token);
|
||||
|
||||
using var worker = new JobWorker(queue, new IJobHandler[] { handler }, NullLogger<JobWorker>.Instance);
|
||||
await worker.StartAsync(token);
|
||||
try
|
||||
{
|
||||
var deadline = DateTime.UtcNow.AddSeconds(20);
|
||||
while (!await DB.Default.Find<Job>().Match(j => j.Kind == kind && j.State == JobState.Done).ExecuteAnyAsync(token))
|
||||
{
|
||||
Assert.True(DateTime.UtcNow < deadline, "the job never finished");
|
||||
Assert.Equal(0, await queue.Reap(token));
|
||||
await Task.Delay(250, token);
|
||||
}
|
||||
}
|
||||
finally
|
||||
{
|
||||
await worker.StopAsync(CancellationToken.None);
|
||||
}
|
||||
|
||||
Assert.Equal(1, handler.Started);
|
||||
}
|
||||
|
||||
// a lease taken by another worker (reaped, then leased again) stops the handler, and its outcome is not recorded
|
||||
[Fact]
|
||||
public async Task A_lost_lease_stops_the_handler_and_drops_its_outcome()
|
||||
{
|
||||
var token = TestContext.Current.CancellationToken;
|
||||
var kind = (JobKind)(2000 + Random.Shared.Next(100000));
|
||||
var queue = new JobQueue(TimeSpan.FromSeconds(1.5), j => j.Kind == kind);
|
||||
var handler = new SlowHandler(kind, TimeSpan.FromSeconds(30));
|
||||
await queue.Enqueue(kind, "{}", default, default, token);
|
||||
|
||||
using var worker = new JobWorker(queue, new IJobHandler[] { handler }, NullLogger<JobWorker>.Instance);
|
||||
await worker.StartAsync(token);
|
||||
try
|
||||
{
|
||||
var deadline = DateTime.UtcNow.AddSeconds(20);
|
||||
while (handler.Started == 0)
|
||||
{
|
||||
Assert.True(DateTime.UtcNow < deadline, "the job never started");
|
||||
await Task.Delay(100, token);
|
||||
}
|
||||
await DB.Default.Update<Job>().Match(j => j.Kind == kind).Modify(j => j.LeaseOwner, "another worker").ExecuteAsync(token);
|
||||
while (!handler.Cancelled)
|
||||
{
|
||||
Assert.True(DateTime.UtcNow < deadline, "the handler was never stopped");
|
||||
await Task.Delay(100, token);
|
||||
}
|
||||
await Task.Delay(300, token);
|
||||
}
|
||||
finally
|
||||
{
|
||||
await worker.StopAsync(CancellationToken.None);
|
||||
}
|
||||
|
||||
var job = await DB.Default.Find<Job>().Match(j => j.Kind == kind).ExecuteFirstAsync(token);
|
||||
Assert.Equal(JobState.Running, job.State);
|
||||
Assert.Equal("another worker", job.LeaseOwner);
|
||||
}
|
||||
|
||||
sealed class SlowHandler(JobKind kind, TimeSpan takes) : IJobHandler
|
||||
{
|
||||
int _started;
|
||||
|
||||
public int Started => _started;
|
||||
public bool Cancelled { get; private set; }
|
||||
public JobKind Kind => kind;
|
||||
public int Concurrency => 1;
|
||||
public int MaxAttempts => 3;
|
||||
public int PerHostLimit => 1;
|
||||
|
||||
public async Task<JobOutcome> Handle(Job job, CancellationToken token)
|
||||
{
|
||||
Interlocked.Increment(ref _started);
|
||||
try
|
||||
{
|
||||
await Task.Delay(takes, token);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
Cancelled = true;
|
||||
throw;
|
||||
}
|
||||
return JobOutcome.Done;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,3 +1,4 @@
|
||||
using PrivaPub.Models.Media;
|
||||
using MongoDB.Bson;
|
||||
using MongoDB.Driver;
|
||||
using MongoDB.Entities;
|
||||
@@ -22,6 +23,31 @@ namespace PrivaPub.Tests.Infrastructure
|
||||
[Trait("Category", "Integration")]
|
||||
public class MigrationTests
|
||||
{
|
||||
[Fact]
|
||||
public async Task A_focal_point_that_is_not_a_number_is_removed()
|
||||
{
|
||||
Assert.SkipUnless(MongoFixture.Enabled, MongoFixture.Skip);
|
||||
var token = TestContext.Current.CancellationToken;
|
||||
var broken = new MediaAttachment { OwnerAvatarId = "a1", Focus = [float.NaN, 0f] };
|
||||
var sound = new MediaAttachment { OwnerAvatarId = "a1", Focus = [0.5f, -0.5f] };
|
||||
var post = new Post
|
||||
{
|
||||
ObjectURI = $"https://privapub.test/{Guid.NewGuid():N}",
|
||||
Media = [new PostMedia { URL = "https://privapub.test/media/files/a.jpg", Focus = [float.PositiveInfinity, 0f] }, new PostMedia { URL = "https://privapub.test/media/files/b.jpg", Focus = [0.1f, 0.2f] }]
|
||||
};
|
||||
await DB.Default.SaveAsync(broken, token);
|
||||
await DB.Default.SaveAsync(sound, token);
|
||||
await DB.Default.SaveAsync(post, token);
|
||||
|
||||
await new _016_focal_points_are_finite().UpgradeAsync();
|
||||
|
||||
Assert.Null((await DB.Default.Find<MediaAttachment>().OneAsync(broken.ID, token)).Focus);
|
||||
Assert.Equal(new[] { 0.5f, -0.5f }, (await DB.Default.Find<MediaAttachment>().OneAsync(sound.ID, token)).Focus);
|
||||
var migrated = await DB.Default.Find<Post>().OneAsync(post.ID, token);
|
||||
Assert.Null(migrated.Media[0].Focus);
|
||||
Assert.Equal(new[] { 0.1f, 0.2f }, migrated.Media[1].Focus);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Stored_remote_content_is_sanitized_and_local_content_rendered()
|
||||
{
|
||||
|
||||
@@ -0,0 +1,316 @@
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
|
||||
using MongoDB.Bson;
|
||||
using MongoDB.Driver;
|
||||
using MongoDB.Entities;
|
||||
|
||||
using PrivaPub.Infrastructure.Backup;
|
||||
using PrivaPub.Infrastructure.Cli;
|
||||
using PrivaPub.Tests.Support;
|
||||
using PrivaPub.Tests.Support.Host;
|
||||
|
||||
namespace PrivaPub.Tests.Infrastructure
|
||||
{
|
||||
// A backup restored at boot: what was made since is lost, what protects is not (owner decision 2026-10-07). A server
|
||||
// in a database of its own is backed up, changed, and restored; alone, since the maintenance lock is the server's one.
|
||||
[Trait("Category", "Integration")]
|
||||
[Xunit.Collection(nameof(Exclusive))]
|
||||
public sealed class RestoreTests : IAsyncLifetime
|
||||
{
|
||||
const string Host = "https://privapub.test";
|
||||
|
||||
readonly string _scratch = Path.Combine(Path.GetTempPath(), $"privapub-restore-tests-{Guid.NewGuid():N}");
|
||||
IMongoDatabase _database;
|
||||
|
||||
string Backups => Path.Combine(_scratch, "backups");
|
||||
string Media => Path.Combine(_scratch, "media");
|
||||
string Trash => Path.Combine(_scratch, "media-trash");
|
||||
|
||||
static CancellationToken Token => TestContext.Current.CancellationToken;
|
||||
|
||||
public async ValueTask InitializeAsync()
|
||||
{
|
||||
Assert.SkipUnless(MongoFixture.Enabled, MongoFixture.Skip);
|
||||
await PrivaPubHost.Shared();
|
||||
_database = DB.Default.Database().Client.GetDatabase($"PrivaPubRestore_{Guid.NewGuid():N}");
|
||||
Directory.CreateDirectory(Media);
|
||||
Directory.CreateDirectory(Trash);
|
||||
}
|
||||
|
||||
public async ValueTask DisposeAsync()
|
||||
{
|
||||
if (_database != default)
|
||||
await _database.Client.DropDatabaseAsync(_database.DatabaseNamespace.DatabaseName);
|
||||
if (Directory.Exists(_scratch))
|
||||
Directory.Delete(_scratch, recursive: true);
|
||||
}
|
||||
|
||||
BackupContext Context(string host = Host) => new(_database, Backups, Media, Trash, host, new BackupOptions());
|
||||
|
||||
IMongoCollection<BsonDocument> C(string name) => _database.GetCollection<BsonDocument>(name);
|
||||
|
||||
async Task<BsonDocument> One(string collection, BsonDocument filter) =>
|
||||
await (await C(collection).FindAsync(filter, cancellationToken: Token)).FirstOrDefaultAsync(Token);
|
||||
|
||||
async Task<List<BsonDocument>> All(string collection) =>
|
||||
await (await C(collection).FindAsync(FilterDefinition<BsonDocument>.Empty, cancellationToken: Token)).ToListAsync(Token);
|
||||
|
||||
static BsonDocument Id(ObjectId id) => new("_id", id);
|
||||
|
||||
static Task Set(IMongoCollection<BsonDocument> collection, ObjectId id, BsonDocument fields) =>
|
||||
collection.UpdateOneAsync(Id(id), new BsonDocument("$set", fields), cancellationToken: Token);
|
||||
|
||||
// the ids of the town this test builds
|
||||
readonly ObjectId _alice = ObjectId.GenerateNewId(), _alicePersona = ObjectId.GenerateNewId();
|
||||
readonly ObjectId _deletedPost = ObjectId.GenerateNewId(), _editedPost = ObjectId.GenerateNewId(), _laterPost = ObjectId.GenerateNewId();
|
||||
readonly ObjectId _deletedMedia = ObjectId.GenerateNewId(), _laterMedia = ObjectId.GenerateNewId();
|
||||
readonly ObjectId _bob = ObjectId.GenerateNewId(), _bobPersona = ObjectId.GenerateNewId();
|
||||
|
||||
static string Uri(ObjectId post) => $"{Host}/peasants/alice/posts/{post}";
|
||||
|
||||
async Task<BackupInfo> TownBackedUp()
|
||||
{
|
||||
await File.WriteAllTextAsync(Path.Combine(Media, "deleted.jpg"), "deleted", Token);
|
||||
await C("RootUser").InsertOneAsync(new BsonDocument { { "_id", _alice }, { "UserName", "alice" }, { "HashedPassword", "old" }, { "SessionStamp", "s1" }, { "Policies", new BsonArray { "IsUser" } }, { "DeletedAt", BsonNull.Value } }, cancellationToken: Token);
|
||||
await C("Avatar").InsertOneAsync(new BsonDocument { { "_id", _alicePersona }, { "UserName", "alice" }, { "SuspendedAt", BsonNull.Value }, { "DeletionAt", BsonNull.Value } }, cancellationToken: Token);
|
||||
await C("RootToAvatar").InsertOneAsync(new BsonDocument { { "RootId", _alice.ToString() }, { "AvatarId", _alicePersona.ToString() } }, cancellationToken: Token);
|
||||
await C("ReservedName").InsertOneAsync(new BsonDocument("Name", "alice"), cancellationToken: Token);
|
||||
await C("Post").InsertManyAsync([
|
||||
new BsonDocument { { "_id", _deletedPost }, { "GroupUserId", _alicePersona.ToString() }, { "IsFederatedCopy", false }, { "ObjectURI", Uri(_deletedPost) }, { "Text", "regret" },
|
||||
{ "Media", new BsonArray { new BsonDocument("AttachmentId", _deletedMedia.ToString()) } }, { "DeletedAt", BsonNull.Value } },
|
||||
new BsonDocument { { "_id", _editedPost }, { "GroupUserId", _alicePersona.ToString() }, { "IsFederatedCopy", false }, { "ObjectURI", Uri(_editedPost) }, { "Text", "first words" }, { "DeletedAt", BsonNull.Value } }
|
||||
], cancellationToken: Token);
|
||||
await C("TimelineEntry").InsertOneAsync(new BsonDocument { { "PostId", _deletedPost.ToString() }, { "ReblogOfPostId", BsonNull.Value } }, cancellationToken: Token);
|
||||
await C("MediaAttachment").InsertOneAsync(new BsonDocument { { "_id", _deletedMedia }, { "PostId", _deletedPost.ToString() }, { "FilePath", "deleted.jpg" }, { "TrashedAt", BsonNull.Value } }, cancellationToken: Token);
|
||||
await C("Follower").InsertOneAsync(new BsonDocument { { "LocalActorId", _alicePersona.ToString() }, { "ActorURI", "https://elsewhere.example/users/left" } }, cancellationToken: Token);
|
||||
await C("RemoteInstance").InsertOneAsync(new BsonDocument { { "Host", "down.example" }, { "ConsecutiveFailures", 9 }, { "UnavailableUntil", DateTime.UtcNow.AddDays(1) } }, cancellationToken: Token);
|
||||
await C("_migration_history_").InsertOneAsync(new BsonDocument { { "Number", 16 }, { "Name", "focal points are finite" } }, cancellationToken: Token);
|
||||
await C("InteractionSalt").InsertOneAsync(new BsonDocument("Salt", "the backup's day"), cancellationToken: Token);
|
||||
|
||||
var (backup, error) = await ServerBackup.Create(Context(), "manual", dbOnly: false, Token);
|
||||
Assert.Null(error);
|
||||
return backup;
|
||||
}
|
||||
|
||||
// what happens after the backup: protective acts, and things merely made or changed
|
||||
async Task LifeGoesOn()
|
||||
{
|
||||
// alice regrets a post: deleted, its media trashed and moved out of what is served
|
||||
await Set(C("Post"), _deletedPost, new BsonDocument { { "DeletedAt", DateTime.UtcNow }, { "Text", BsonNull.Value }, { "Media", new BsonArray() } });
|
||||
await Set(C("MediaAttachment"), _deletedMedia, new BsonDocument { { "TrashedAt", DateTime.UtcNow }, { "TrashReason", "deleted" } });
|
||||
File.Move(Path.Combine(Media, "deleted.jpg"), Path.Combine(Trash, "deleted.jpg"));
|
||||
await C("TimelineEntry").DeleteManyAsync(new BsonDocument("PostId", _deletedPost.ToString()), Token);
|
||||
// an edit, a new post with a picture: not protective
|
||||
await Set(C("Post"), _editedPost, new BsonDocument("Text", "second words"));
|
||||
await File.WriteAllTextAsync(Path.Combine(Media, "later.jpg"), "later", Token);
|
||||
await C("Post").InsertOneAsync(new BsonDocument { { "_id", _laterPost }, { "GroupUserId", _alicePersona.ToString() }, { "IsFederatedCopy", false }, { "ObjectURI", Uri(_laterPost) }, { "Text", "later" }, { "DeletedAt", BsonNull.Value } }, cancellationToken: Token);
|
||||
await C("MediaAttachment").InsertOneAsync(new BsonDocument { { "_id", _laterMedia }, { "PostId", _laterPost.ToString() }, { "FilePath", "later.jpg" }, { "TrashedAt", BsonNull.Value } }, cancellationToken: Token);
|
||||
// a follower leaves, another comes
|
||||
await C("Follower").DeleteManyAsync(FilterDefinition<BsonDocument>.Empty, Token);
|
||||
await C("Follower").InsertOneAsync(new BsonDocument { { "LocalActorId", _alicePersona.ToString() }, { "ActorURI", "https://elsewhere.example/users/came" } }, cancellationToken: Token);
|
||||
// alice blocks someone, the admin blocks a server, someone is reported, alice's persona is suspended, she changes
|
||||
// her password
|
||||
await C("Block").InsertOneAsync(new BsonDocument { { "AvatarId", _alicePersona.ToString() }, { "TargetActorURI", "https://elsewhere.example/users/troll" } }, cancellationToken: Token);
|
||||
await C("DomainBlock").InsertOneAsync(new BsonDocument { { "Domain", "evil.example" }, { "Severity", "Suspend" } }, cancellationToken: Token);
|
||||
await C("Report").InsertOneAsync(new BsonDocument("Comment", "harassment"), cancellationToken: Token);
|
||||
await Set(C("Avatar"), _alicePersona, new BsonDocument("SuspendedAt", DateTime.UtcNow));
|
||||
await Set(C("RootUser"), _alice, new BsonDocument("HashedPassword", "new"));
|
||||
// bob arrives
|
||||
await C("RootUser").InsertOneAsync(new BsonDocument { { "_id", _bob }, { "UserName", "bob" }, { "HashedPassword", "bob's" }, { "Policies", new BsonArray { "IsUser" } }, { "DeletedAt", BsonNull.Value } }, cancellationToken: Token);
|
||||
await C("Avatar").InsertOneAsync(new BsonDocument { { "_id", _bobPersona }, { "UserName", "bob" }, { "DeletionAt", BsonNull.Value } }, cancellationToken: Token);
|
||||
await C("RootToAvatar").InsertOneAsync(new BsonDocument { { "RootId", _bob.ToString() }, { "AvatarId", _bobPersona.ToString() } }, cancellationToken: Token);
|
||||
await C("ReservedName").InsertOneAsync(new BsonDocument("Name", "bob"), cancellationToken: Token);
|
||||
// sessions, a list made, today's salt, work queued
|
||||
await C("openiddict.tokens").InsertOneAsync(new BsonDocument("subject", "alice"), cancellationToken: Token);
|
||||
await C("PersonaList").InsertOneAsync(new BsonDocument { { "AvatarId", _alicePersona.ToString() }, { "Title", "friends" } }, cancellationToken: Token);
|
||||
await C("InteractionSalt").DeleteManyAsync(FilterDefinition<BsonDocument>.Empty, Token);
|
||||
await C("InteractionSalt").InsertOneAsync(new BsonDocument("Salt", "today's"), cancellationToken: Token);
|
||||
await C("Job").InsertOneAsync(new BsonDocument("Kind", "Deliver"), cancellationToken: Token);
|
||||
}
|
||||
|
||||
async Task AssertRestoredAndProtected(BackupInfo backup)
|
||||
{
|
||||
// lost: what was merely made or changed since
|
||||
Assert.Equal("first words", (await One("Post", Id(_editedPost)))["Text"].AsString);
|
||||
var later = await One("Post", Id(_laterPost));
|
||||
Assert.False(later["DeletedAt"].IsBsonNull);
|
||||
Assert.True(later["Text"].IsBsonNull);
|
||||
Assert.Empty(await All("PersonaList"));
|
||||
Assert.Equal(16, (await One("_migration_history_", new BsonDocument()))["Number"].ToInt32());
|
||||
|
||||
// kept: every protective act
|
||||
var deleted = await One("Post", Id(_deletedPost));
|
||||
Assert.False(deleted["DeletedAt"].IsBsonNull);
|
||||
Assert.True(deleted["Text"].IsBsonNull);
|
||||
Assert.Null(await One("TimelineEntry", new BsonDocument("PostId", _deletedPost.ToString())));
|
||||
Assert.False((await One("MediaAttachment", Id(_deletedMedia)))["TrashedAt"].IsBsonNull);
|
||||
Assert.Equal("restore: made after the backup", (await One("MediaAttachment", Id(_laterMedia)))["TrashReason"].AsString);
|
||||
Assert.Equal(["https://elsewhere.example/users/came"], (await All("Follower")).Select(f => f["ActorURI"].AsString));
|
||||
Assert.NotNull(await One("Block", new BsonDocument("TargetActorURI", "https://elsewhere.example/users/troll")));
|
||||
Assert.NotNull(await One("DomainBlock", new BsonDocument("Domain", "evil.example")));
|
||||
Assert.NotNull(await One("Report", new BsonDocument("Comment", "harassment")));
|
||||
Assert.False((await One("Avatar", Id(_alicePersona)))["SuspendedAt"].IsBsonNull);
|
||||
var alice = await One("RootUser", Id(_alice));
|
||||
Assert.Equal("new", alice["HashedPassword"].AsString);
|
||||
Assert.NotEqual("s1", alice["SessionStamp"].AsString);
|
||||
|
||||
// bob, made since: deleted, his name kept
|
||||
var bob = await One("RootUser", Id(_bob));
|
||||
Assert.Equal($"deleted-{_bob}", bob["UserName"].AsString);
|
||||
Assert.True(bob["HashedPassword"].IsBsonNull);
|
||||
Assert.False(bob["DeletedAt"].IsBsonNull);
|
||||
Assert.False((await One("Avatar", Id(_bobPersona)))["DeletionAt"].IsBsonNull);
|
||||
Assert.NotNull(await One("ReservedName", new BsonDocument("Name", "bob")));
|
||||
|
||||
// what a backup never holds is as it was; sessions end; circuits close
|
||||
Assert.Equal("today's", (await One("InteractionSalt", new BsonDocument()))["Salt"].AsString);
|
||||
Assert.Single(await All("Job"));
|
||||
Assert.Empty(await All("openiddict.tokens"));
|
||||
Assert.Equal(0, (await One("RemoteInstance", new BsonDocument("Host", "down.example")))["ConsecutiveFailures"].ToInt32());
|
||||
|
||||
// the picture of the deleted post came back from the backup, to be trashed again by the janitor
|
||||
Assert.True(File.Exists(Path.Combine(Media, "deleted.jpg")));
|
||||
Assert.Null(RestoreMarker.Read(Backups));
|
||||
var record = await (await _database.GetCollection<RestoreRecord>(nameof(RestoreRecord)).FindAsync(r => r.Backup == backup.Id, cancellationToken: Token)).FirstAsync(Token);
|
||||
Assert.Equal(["bob"], record.Report.RootsTombstoned);
|
||||
Assert.Equal(["bob"], record.Report.PersonasTombstoned);
|
||||
Assert.Equal(1, record.Report.PostsGone);
|
||||
Assert.Equal(1, record.Report.PostsDeleted);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_restore_loses_what_was_made_since_but_never_a_protective_act()
|
||||
{
|
||||
var backup = await TownBackedUp();
|
||||
await LifeGoesOn();
|
||||
|
||||
Assert.Null(await ServerRestore.Request(Context(), backup.Id, "admin", Token));
|
||||
Assert.Contains("already pending", await ServerRestore.Request(Context(), backup.Id, "admin", Token));
|
||||
Assert.Equal(RestoreOutcome.Restored, await ServerRestore.ApplyPending(Context(), NullLogger.Instance, Token));
|
||||
|
||||
await AssertRestoredAndProtected(backup);
|
||||
var before = ServerBackup.List(Backups).Single(b => b.Kind == "pre-restore");
|
||||
Assert.Equal(before.Id, (await (await _database.GetCollection<RestoreRecord>(nameof(RestoreRecord)).FindAsync(FilterDefinition<RestoreRecord>.Empty, cancellationToken: Token)).FirstAsync(Token)).PreRestore);
|
||||
}
|
||||
|
||||
// an attempt that died midway: the next one starts again from the backup, with the same pre-restore backup, never
|
||||
// one of the half-restored database
|
||||
[Fact]
|
||||
public async Task A_restore_that_died_midway_converges_on_the_next_boot()
|
||||
{
|
||||
var backup = await TownBackedUp();
|
||||
await LifeGoesOn();
|
||||
Assert.Null(await ServerRestore.Request(Context(), backup.Id, "admin", Token));
|
||||
Assert.Equal(RestoreOutcome.Restored, await ServerRestore.ApplyPending(Context(), NullLogger.Instance, Token));
|
||||
var before = ServerBackup.List(Backups).Single(b => b.Kind == "pre-restore");
|
||||
|
||||
// as if the boot died after importing a collection, with the marker saying so
|
||||
await C("Post").DeleteManyAsync(FilterDefinition<BsonDocument>.Empty, Token);
|
||||
await C("Block").DeleteManyAsync(FilterDefinition<BsonDocument>.Empty, Token);
|
||||
new RestoreMarker { Backup = backup.Id, PreRestore = before.Id, State = "running", Attempts = 1, RequestedBy = "admin" }.Write(Backups);
|
||||
|
||||
Assert.Equal(RestoreOutcome.Restored, await ServerRestore.ApplyPending(Context(), NullLogger.Instance, Token));
|
||||
Assert.Single(ServerBackup.List(Backups), b => b.Kind == "pre-restore");
|
||||
await AssertRestoredAndProtected(backup);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_backup_from_elsewhere_from_a_newer_build_or_altered_is_refused()
|
||||
{
|
||||
var backup = await TownBackedUp();
|
||||
|
||||
Assert.Contains("not https://other.test", await ServerRestore.Request(Context("https://other.test"), backup.Id, "admin", Token));
|
||||
Assert.Contains("no such backup", await ServerRestore.Request(Context(), "../" + backup.Id, "admin", Token));
|
||||
|
||||
var manifest = ArchiveManifest.Read(Path.Combine(Backups, backup.Id));
|
||||
manifest.MigrationNumber = ServerBackup.CodeMigration() + 1;
|
||||
manifest.Write(Path.Combine(Backups, backup.Id));
|
||||
Assert.Contains("newer build", await ServerRestore.Request(Context(), backup.Id, "admin", Token));
|
||||
|
||||
manifest.MigrationNumber = 16;
|
||||
manifest.Write(Path.Combine(Backups, backup.Id));
|
||||
await File.AppendAllTextAsync(Path.Combine(Backups, backup.Id, "db", "Post.jsonl.gz"), "x", Token);
|
||||
Assert.Contains("Post: altered", await ServerRestore.Request(Context(), backup.Id, "admin", Token));
|
||||
Assert.Null(RestoreMarker.Read(Backups));
|
||||
}
|
||||
|
||||
// refused at boot before anything changed (the backup went meanwhile): recorded, and the server boots as it was
|
||||
[Fact]
|
||||
public async Task A_restore_that_cannot_start_is_abandoned_and_changes_nothing()
|
||||
{
|
||||
var backup = await TownBackedUp();
|
||||
await LifeGoesOn();
|
||||
Assert.Null(await ServerRestore.Request(Context(), backup.Id, "admin", Token));
|
||||
ServerBackup.Delete(Backups, backup.Id);
|
||||
|
||||
Assert.Equal(RestoreOutcome.Abandoned, await ServerRestore.ApplyPending(Context(), NullLogger.Instance, Token));
|
||||
|
||||
Assert.Equal("second words", (await One("Post", Id(_editedPost)))["Text"].AsString);
|
||||
Assert.Null(RestoreMarker.Read(Backups));
|
||||
Assert.DoesNotContain(ServerBackup.List(Backups), b => b.Kind == "pre-restore");
|
||||
var record = await (await _database.GetCollection<RestoreRecord>(nameof(RestoreRecord)).FindAsync(FilterDefinition<RestoreRecord>.Empty, cancellationToken: Token)).FirstAsync(Token);
|
||||
Assert.True(record.Abandoned);
|
||||
Assert.Contains("no such backup", record.Error);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task After_three_failures_it_waits_for_someone_to_look()
|
||||
{
|
||||
var backup = await TownBackedUp();
|
||||
await LifeGoesOn();
|
||||
new RestoreMarker { Backup = backup.Id, PreRestore = "gone", State = "running", Attempts = RestoreMarker.MaxAttempts, Error = "disk full" }.Write(Backups);
|
||||
|
||||
Assert.Equal(RestoreOutcome.GaveUp, await ServerRestore.ApplyPending(Context(), NullLogger.Instance, Token));
|
||||
Assert.Equal("second words", (await One("Post", Id(_editedPost)))["Text"].AsString);
|
||||
Assert.NotNull(RestoreMarker.Read(Backups));
|
||||
|
||||
await File.WriteAllTextAsync(RestoreMarker.PathIn(Backups), "{ not json", Token);
|
||||
Assert.Equal(RestoreOutcome.GaveUp, await ServerRestore.ApplyPending(Context(), NullLogger.Instance, Token));
|
||||
}
|
||||
|
||||
// a command waits while a restore does; asking how it goes doesn't
|
||||
[Fact]
|
||||
public async Task Commands_wait_for_a_restore_asked_for()
|
||||
{
|
||||
var host = await PrivaPubHost.Shared();
|
||||
var backups = host.Get<Backups>();
|
||||
new RestoreMarker { Backup = "20260101-000000-manual", RequestedBy = "cli" }.Write(backups.Root);
|
||||
try
|
||||
{
|
||||
Assert.Equal(MaintenanceGate.TryLater, await MaintenanceGate.Enter(host.Services, ["promote", "someone"], Token));
|
||||
Assert.Null(await MaintenanceGate.Enter(host.Services, ["restore", "--status"], Token));
|
||||
var output = new StringWriter();
|
||||
Assert.Equal(0, await AdminCommands.Run(["restore", "--status"], host.Services, output: output));
|
||||
Assert.Contains("waiting: 20260101-000000-manual, pending", output.ToString());
|
||||
Assert.Equal(1, await AdminCommands.Run(["restore", "20260101-000000-manual"], host.Services, output: TextWriter.Null));
|
||||
}
|
||||
finally
|
||||
{
|
||||
RestoreMarker.Clear(backups.Root);
|
||||
}
|
||||
}
|
||||
|
||||
// FEP-8fcf: for two weeks after a restore, no digests, and no follow undone for them
|
||||
[Fact]
|
||||
public async Task Followers_digests_rest_for_two_weeks_after_a_restore()
|
||||
{
|
||||
RestoreRecord.Forget();
|
||||
Assert.False(await RestoreRecord.InFollowersGrace(Token));
|
||||
var record = new RestoreRecord { Backup = "test", RestoredAt = DateTime.UtcNow.AddDays(-13) };
|
||||
await DB.Default.SaveAsync(record, Token);
|
||||
try
|
||||
{
|
||||
RestoreRecord.Forget();
|
||||
Assert.True(await RestoreRecord.InFollowersGrace(Token));
|
||||
await DB.Default.Update<RestoreRecord>().MatchID(record.ID).Modify(r => r.RestoredAt, DateTime.UtcNow.AddDays(-15)).ExecuteAsync(Token);
|
||||
RestoreRecord.Forget();
|
||||
Assert.False(await RestoreRecord.InFollowersGrace(Token));
|
||||
}
|
||||
finally
|
||||
{
|
||||
await DB.Default.DeleteAsync<RestoreRecord>(record.ID);
|
||||
RestoreRecord.Forget();
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
using MongoDB.Entities;
|
||||
|
||||
using PrivaPub.Infrastructure.Data;
|
||||
using PrivaPub.Tests.Support;
|
||||
|
||||
namespace PrivaPub.Tests.Infrastructure
|
||||
{
|
||||
[Trait("Category", "Integration")]
|
||||
public sealed class TopologyTests
|
||||
{
|
||||
// tools/ci/with-test-mongod.sh starts a one-member replica set, as production's mongod is: the backup's snapshot reads
|
||||
// are what the tests exercise
|
||||
[Fact]
|
||||
public async Task The_test_mongod_is_a_replica_set_like_productions()
|
||||
{
|
||||
Assert.SkipUnless(MongoFixture.Enabled, MongoFixture.Skip);
|
||||
Assert.SkipUnless(MongoFixture.Connection.Contains("directConnection=true"), "a mongod not started by tools/ci/with-test-mongod.sh");
|
||||
Assert.True(await MongoTopology.IsReplicaSet(DB.Default.Database(), TestContext.Current.CancellationToken));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -24,7 +24,8 @@ namespace PrivaPub.Tests.Support.Host
|
||||
|
||||
static readonly SemaphoreSlim Boot = new(1, 1);
|
||||
static readonly Type[] Unwanted = { typeof(JobWorker), typeof(MediaJanitor), typeof(OAuthPruner), typeof(StatisticsSchedule),
|
||||
typeof(PrivaPub.Domain.Social.FollowResender) };
|
||||
typeof(PrivaPub.Domain.Social.FollowResender), typeof(PrivaPub.Infrastructure.Backup.BackupScheduler),
|
||||
typeof(PrivaPub.Infrastructure.Backup.RestoreWatcher) };
|
||||
static PrivaPubHost _shared;
|
||||
|
||||
readonly string _mediaRoot = Path.Combine(Path.GetTempPath(), $"privapub-tests-{Guid.NewGuid():N}");
|
||||
@@ -78,6 +79,10 @@ namespace PrivaPub.Tests.Support.Host
|
||||
["Statistics:Geo:AutoUpdate"] = "false",
|
||||
["Statistics:Cdn:AutoUpdate"] = "false",
|
||||
["Media:Root"] = _mediaRoot,
|
||||
["Backups:Root"] = _mediaRoot + "-backups",
|
||||
["Backups:Nightly"] = "false",
|
||||
["RateLimits:UploadsBurst"] = "1000",
|
||||
["RateLimits:ProxyBurst"] = "100000",
|
||||
["Logging:LogLevel:Default"] = "Warning",
|
||||
["Serilog:MinimumLevel:Default"] = Environment.GetEnvironmentVariable("PRIVAPUB_TEST_LOGS") == "1" ? "Information" : "Fatal"
|
||||
};
|
||||
@@ -115,8 +120,11 @@ namespace PrivaPub.Tests.Support.Host
|
||||
protected override void Dispose(bool disposing)
|
||||
{
|
||||
base.Dispose(disposing);
|
||||
if (disposing && Directory.Exists(_mediaRoot))
|
||||
Directory.Delete(_mediaRoot, recursive: true);
|
||||
if (!disposing)
|
||||
return;
|
||||
foreach (var directory in new[] { _mediaRoot, _mediaRoot + "-backups", _mediaRoot + "-trash", _mediaRoot + "-incoming", _mediaRoot + "-proxy" })
|
||||
if (Directory.Exists(directory))
|
||||
Directory.Delete(directory, recursive: true);
|
||||
}
|
||||
|
||||
sealed class ClientAddressFilter : IStartupFilter
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
using Microsoft.AspNetCore.RateLimiting;
|
||||
using Microsoft.AspNetCore.Mvc;
|
||||
|
||||
using MongoDB.Entities;
|
||||
@@ -16,6 +17,7 @@ using PrivaPub.Models.Post;
|
||||
using PrivaPub.Models.Social;
|
||||
using PrivaPub.Models.User;
|
||||
using PrivaPub.StaticServices;
|
||||
using PrivaPub.Infrastructure;
|
||||
|
||||
using PostEntity = PrivaPub.Models.Post.Post;
|
||||
|
||||
@@ -46,18 +48,33 @@ namespace PrivaPub.Api.Mastodon.Controllers
|
||||
[HttpGet("/api/v1/accounts/verify_credentials"), Scope("read:accounts")]
|
||||
public async Task<IActionResult> VerifyCredentials(CancellationToken token) => Json(await _mapper.Local(Me, withSource: true, token));
|
||||
|
||||
[HttpPatch("/api/v1/accounts/update_credentials"), Scope("write:accounts"), RequestSizeLimit(20 * 1024 * 1024),
|
||||
[HttpPatch("/api/v1/accounts/update_credentials"), Scope("write:accounts"), EnableRateLimiting(RateLimiting.Uploads), RequestSizeLimit(20 * 1024 * 1024),
|
||||
RequestFormLimits(MultipartBodyLengthLimit = 20 * 1024 * 1024)]
|
||||
public async Task<IActionResult> UpdateCredentials([FromServices] IMediaService media, CancellationToken token)
|
||||
{
|
||||
var avatar = await _dbEntities.Avatars.MatchID(Me.Id).ExecuteFirstAsync(token);
|
||||
// a new picture is a row of its own; the one it replaces goes to the trash once the profile is saved
|
||||
var replaced = new List<Models.Media.MediaAttachment>();
|
||||
if (Request.HasFormContentType)
|
||||
{
|
||||
var form = await Request.ReadFormAsync(token);
|
||||
if (form.Files["avatar"] is { } picture && await media.ProfileImage(picture, 400, 400, token) is { } pictureUrl)
|
||||
avatar.PictureURL = pictureUrl;
|
||||
if (form.Files["header"] is { } header && await media.ProfileImage(header, 1500, 500, token) is { } headerUrl)
|
||||
avatar.ThumbnailURL = headerUrl;
|
||||
foreach (var (field, kind, width, height) in new[] { ("avatar", "avatar", 400, 400), ("header", "header", 1500, 500) })
|
||||
{
|
||||
if (form.Files[field] is not { } file)
|
||||
continue;
|
||||
var outcome = await media.ProfileImage(avatar.ID, kind, file, width, height, token);
|
||||
if (!outcome.Ok)
|
||||
{
|
||||
foreach (var made in replaced)
|
||||
await media.Trash(m => m.ID == made.ID, "profile not saved", token);
|
||||
return Error(outcome.Status, outcome.Error);
|
||||
}
|
||||
replaced.Add(outcome.Attachment);
|
||||
if (kind == "avatar")
|
||||
avatar.PictureURL = media.Url(outcome.Attachment.FilePath);
|
||||
else
|
||||
avatar.ThumbnailURL = media.Url(outcome.Attachment.FilePath);
|
||||
}
|
||||
}
|
||||
if (Params.Has("display_name"))
|
||||
avatar.Name = Params.Get("display_name")?.Trim();
|
||||
@@ -98,12 +115,36 @@ namespace PrivaPub.Api.Mastodon.Controllers
|
||||
avatar.Fields = fields;
|
||||
avatar.UpdatedAt = DateTime.UtcNow;
|
||||
await DB.Default.SaveAsync(avatar, token);
|
||||
foreach (var made in replaced)
|
||||
await media.Trash(m => m.ProfileOfAvatarId == avatar.ID && m.Kind == made.Kind && m.ID != made.ID, "replaced", token);
|
||||
|
||||
var actor = _localActors.FromAvatar(avatar);
|
||||
await _outbox.PublishProfile(actor, token);
|
||||
return Json(await _mapper.Local(actor, withSource: true, token));
|
||||
}
|
||||
|
||||
// Mastodon's: the picture goes (to the trash), the profile shows the default again, and followers are told
|
||||
[HttpDelete("/api/v1/profile/avatar"), Scope("write:accounts")]
|
||||
public Task<IActionResult> DeleteAvatar([FromServices] IMediaService media, CancellationToken token) => DeletePicture(media, "avatar", token);
|
||||
|
||||
[HttpDelete("/api/v1/profile/header"), Scope("write:accounts")]
|
||||
public Task<IActionResult> DeleteHeader([FromServices] IMediaService media, CancellationToken token) => DeletePicture(media, "header", token);
|
||||
|
||||
async Task<IActionResult> DeletePicture(IMediaService media, string kind, CancellationToken token)
|
||||
{
|
||||
var avatar = await _dbEntities.Avatars.MatchID(Me.Id).ExecuteFirstAsync(token);
|
||||
if (kind == "avatar")
|
||||
avatar.PictureURL = default;
|
||||
else
|
||||
avatar.ThumbnailURL = default;
|
||||
avatar.UpdatedAt = DateTime.UtcNow;
|
||||
await DB.Default.SaveAsync(avatar, token);
|
||||
await media.Trash(m => m.ProfileOfAvatarId == avatar.ID && m.Kind == kind, "removed", token);
|
||||
var actor = _localActors.FromAvatar(avatar);
|
||||
await _outbox.PublishProfile(actor, token);
|
||||
return Json(await _mapper.Local(actor, withSource: true, token));
|
||||
}
|
||||
|
||||
[HttpGet("/api/v1/accounts/lookup"), Scope("read:accounts", requiresUser: false), Microsoft.AspNetCore.Authorization.AllowAnonymous]
|
||||
public async Task<IActionResult> Lookup(CancellationToken token)
|
||||
{
|
||||
|
||||
@@ -56,10 +56,10 @@ namespace PrivaPub.Api.Mastodon.Controllers
|
||||
{
|
||||
supported_mime_types = PrivaPub.Domain.Media.MediaService.SupportedTypes,
|
||||
image_size_limit = media.MaxImageBytes,
|
||||
image_matrix_limit = media.MaxImageSide * media.MaxImageSide,
|
||||
image_matrix_limit = media.MaxPixels,
|
||||
video_size_limit = media.MaxVideoBytes,
|
||||
video_frame_rate_limit = 60,
|
||||
video_matrix_limit = 2_304_000
|
||||
video_frame_rate_limit = (int)media.MaxFrameRate,
|
||||
video_matrix_limit = media.MaxVideoPixels
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
using Microsoft.AspNetCore.RateLimiting;
|
||||
using PrivaPub.Infrastructure.Statistics;
|
||||
using Microsoft.AspNetCore.Authorization;
|
||||
using Microsoft.AspNetCore.Mvc;
|
||||
@@ -7,6 +8,8 @@ using MongoDB.Entities;
|
||||
using PrivaPub.Api.Mastodon.Infrastructure;
|
||||
using PrivaPub.Domain.Media;
|
||||
using PrivaPub.Models.Media;
|
||||
using PrivaPub.Infrastructure;
|
||||
using PrivaPub.Infrastructure.Jobs;
|
||||
|
||||
using System.Globalization;
|
||||
|
||||
@@ -18,78 +21,113 @@ namespace PrivaPub.Api.Mastodon.Controllers
|
||||
|
||||
readonly IMediaService _media;
|
||||
readonly IMediaProxy _proxy;
|
||||
readonly IJobQueue _jobs;
|
||||
|
||||
readonly IInteractionLedger _ledger;
|
||||
|
||||
public MediaController(IMediaService media, IMediaProxy proxy, IInteractionLedger ledger = default)
|
||||
public MediaController(IMediaService media, IMediaProxy proxy, IJobQueue jobs, IInteractionLedger ledger = default)
|
||||
{
|
||||
_media = media;
|
||||
_proxy = proxy;
|
||||
_jobs = jobs;
|
||||
_ledger = ledger;
|
||||
}
|
||||
|
||||
[HttpPost("/api/v1/media"), HttpPost("/api/v2/media"), Scope("write:media"), RequestSizeLimit(UploadLimit),
|
||||
[HttpPost("/api/v1/media"), HttpPost("/api/v2/media"), Scope("write:media"), EnableRateLimiting(RateLimiting.Uploads), RequestSizeLimit(UploadLimit),
|
||||
RequestFormLimits(MultipartBodyLengthLimit = UploadLimit)]
|
||||
public async Task<IActionResult> Upload(CancellationToken token)
|
||||
{
|
||||
if (!Request.HasFormContentType)
|
||||
return Error(StatusCodes.Status422UnprocessableEntity, "Validation failed: File can't be blank");
|
||||
var form = await Request.ReadFormAsync(token);
|
||||
var outcome = await _media.Upload(Me, form.Files["file"], form["description"], form["focus"], token);
|
||||
return outcome.Ok ? Json(View(outcome.Attachment)) : Error(outcome.Status, outcome.Error);
|
||||
// v2 may answer before audio or video is processed (202, its url null until then), as Mastodon does; v1 waits
|
||||
var later = Request.Path.StartsWithSegments("/api/v2/media");
|
||||
var outcome = await _media.Upload(Me, form.Files["file"], form["description"], form["focus"], later, token);
|
||||
if (!outcome.Ok)
|
||||
return Error(outcome.Status, outcome.Error);
|
||||
if (outcome.Attachment.ProcessingState != "pending")
|
||||
return Json(View(outcome.Attachment));
|
||||
await _jobs.EnqueueMany(new[] { ProcessMediaJob.JobFor(outcome.Attachment, new Uri(Me.BaseAddress).Host) }, token);
|
||||
return new JsonResult(View(outcome.Attachment)) { StatusCode = StatusCodes.Status202Accepted };
|
||||
}
|
||||
|
||||
[HttpGet("/api/v1/media/{id}"), Scope("write:media")]
|
||||
public async Task<IActionResult> Get(string id, CancellationToken token)
|
||||
{
|
||||
var attachment = await DB.Default.Find<MediaAttachment>().Match(m => m.ID == id && m.OwnerAvatarId == MyId).ExecuteFirstAsync(token);
|
||||
return attachment == default ? NotFoundError() : Json(View(attachment));
|
||||
var attachment = await DB.Default.Find<MediaAttachment>().Match(m => m.ID == id && m.OwnerAvatarId == MyId && m.TrashedAt == null && m.ProfileOfAvatarId == null).ExecuteFirstAsync(token);
|
||||
return attachment?.ProcessingState switch
|
||||
{
|
||||
null when attachment == default => NotFoundError(),
|
||||
"pending" => new JsonResult(View(attachment)) { StatusCode = StatusCodes.Status206PartialContent },
|
||||
"failed" => Error(StatusCodes.Status422UnprocessableEntity, attachment.ProcessingError ?? "Validation failed: The file could not be processed"),
|
||||
_ => Json(View(attachment))
|
||||
};
|
||||
}
|
||||
|
||||
// as on Mastodon, only media not posted yet: a posted one's description and focus change by editing its post (media_attributes)
|
||||
[HttpPut("/api/v1/media/{id}"), Scope("write:media")]
|
||||
public async Task<IActionResult> Update(string id, CancellationToken token)
|
||||
{
|
||||
var attachment = await DB.Default.Find<MediaAttachment>().Match(m => m.ID == id && m.OwnerAvatarId == MyId).ExecuteFirstAsync(token);
|
||||
var attachment = await DB.Default.Find<MediaAttachment>().Match(m => m.ID == id && m.OwnerAvatarId == MyId && m.TrashedAt == null && m.ProfileOfAvatarId == null && m.PostId == null).ExecuteFirstAsync(token);
|
||||
if (attachment == default)
|
||||
return NotFoundError();
|
||||
if (Params.Has("description"))
|
||||
attachment.Description = Params.Get("description")?.Trim() is { Length: > 0 } description ? description[..Math.Min(description.Length, 1500)] : default;
|
||||
if (Params.Get("focus")?.Split(',') is [var x, var y]
|
||||
&& float.TryParse(x, NumberStyles.Float, CultureInfo.InvariantCulture, out var fx) && float.TryParse(y, NumberStyles.Float, CultureInfo.InvariantCulture, out var fy))
|
||||
attachment.Focus = new[] { Math.Clamp(fx, -1, 1), Math.Clamp(fy, -1, 1) };
|
||||
if (FocalPoint.Parse(Params.Get("focus")) is { } focus)
|
||||
attachment.Focus = focus;
|
||||
await DB.Default.SaveAsync(attachment, token);
|
||||
return Json(View(attachment));
|
||||
}
|
||||
|
||||
[HttpGet("/media/proxy/{signature}/{encoded}"), AllowAnonymous, ApiExplorerSettings(IgnoreApi = true)]
|
||||
[HttpGet("/media/proxy/{signature}/{encoded}"), AllowAnonymous, EnableRateLimiting(RateLimiting.Proxy), ApiExplorerSettings(IgnoreApi = true)]
|
||||
public async Task<IActionResult> Proxy(string signature, string encoded, CancellationToken token)
|
||||
{
|
||||
var url = _proxy.Verified(signature, encoded);
|
||||
if (url == default)
|
||||
if (url == default || _proxy.Refuses(url))
|
||||
return NotFound();
|
||||
Response.Headers["X-Content-Type-Options"] = "nosniff";
|
||||
Response.Headers["Content-Security-Policy"] = "default-src 'none'; sandbox";
|
||||
Response.Headers["Cache-Control"] = "public, max-age=604800";
|
||||
if (_proxy.Cached(url) is { Path: not null } cached)
|
||||
if (_proxy.Cached(url) is { Path: not null } cached && Serve(cached.Path, cached.ContentType) is { } hit)
|
||||
{
|
||||
_ledger?.Count(Interactions.HostOf(url), "media:hit");
|
||||
return PhysicalFile(cached.Path, cached.ContentType, enableRangeProcessing: true);
|
||||
return hit;
|
||||
}
|
||||
if (Request.Headers.Range.Count == 0)
|
||||
{
|
||||
var (path, contentType) = await _proxy.Fetch(signature, encoded, token);
|
||||
if (path != default)
|
||||
return PhysicalFile(path, contentType, enableRangeProcessing: true);
|
||||
var (outcome, path, contentType) = await _proxy.Download(url, token);
|
||||
if (outcome == ProxyOutcome.Cached && Serve(path, contentType) is { } fetched)
|
||||
return fetched;
|
||||
if (outcome == ProxyOutcome.Failed)
|
||||
return NotFound();
|
||||
}
|
||||
return await Stream(url, token);
|
||||
}
|
||||
|
||||
// a cached file, opened before it is answered: trimmed meanwhile, what is open still reads; cached by browsers only
|
||||
// once there is something to cache
|
||||
IActionResult Serve(string path, string contentType)
|
||||
{
|
||||
FileStream file;
|
||||
try
|
||||
{
|
||||
file = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite | FileShare.Delete, 64 * 1024, useAsync: true);
|
||||
}
|
||||
catch (IOException)
|
||||
{
|
||||
return default;
|
||||
}
|
||||
Response.Headers["Cache-Control"] = "public, max-age=604800";
|
||||
return File(file, contentType, enableRangeProcessing: true);
|
||||
}
|
||||
|
||||
async Task<IActionResult> Stream(string url, CancellationToken token)
|
||||
{
|
||||
var range = System.Net.Http.Headers.RangeHeaderValue.TryParse(Request.Headers.Range.ToString(), out var asked) ? asked : default;
|
||||
using var upstream = await _proxy.Open(url, range, token);
|
||||
if (upstream == default)
|
||||
return NotFound();
|
||||
if (upstream.IsSuccessStatusCode)
|
||||
Response.Headers["Cache-Control"] = "public, max-age=604800";
|
||||
Response.StatusCode = (int)upstream.StatusCode;
|
||||
Response.ContentType = upstream.Content.Headers.ContentType?.ToString() ?? "application/octet-stream";
|
||||
if (upstream.Content.Headers.ContentLength is { } length)
|
||||
|
||||
@@ -96,7 +96,8 @@ namespace PrivaPub.Api.Mastodon.Controllers
|
||||
if (await Mine(id, token) is not { } scheduled)
|
||||
return NotFoundError();
|
||||
await DB.Default.DeleteAsync<ScheduledStatus>(scheduled.ID);
|
||||
await ScheduledStatuses.Release(scheduled.ID, token);
|
||||
// its media were the scheduled post's alone, never posted
|
||||
await _media.Trash(m => m.ScheduledStatusId == scheduled.ID, "scheduled post dropped", token);
|
||||
return Json(new { });
|
||||
}
|
||||
}
|
||||
|
||||
@@ -111,13 +111,26 @@ namespace PrivaPub.Api.Mastodon.Controllers
|
||||
return Error(StatusCodes.Status422UnprocessableEntity, "Validation failed: Text can't be blank");
|
||||
var mediaIds = asked.MediaIds.Distinct().ToList();
|
||||
if (mediaIds.Count > 4 || mediaIds.Count > 0 && await DB.Default.CountAsync<Models.Media.MediaAttachment>(
|
||||
m => mediaIds.Contains(m.ID) && m.OwnerAvatarId == MyId && m.PostId == null && m.ScheduledStatusId == null, token) != mediaIds.Count)
|
||||
m => mediaIds.Contains(m.ID) && m.OwnerAvatarId == MyId && m.PostId == null && m.ScheduledStatusId == null
|
||||
&& m.TrashedAt == null && m.ProfileOfAvatarId == null, token) != mediaIds.Count)
|
||||
return Error(StatusCodes.Status422UnprocessableEntity, "Validation failed: Media attachments are not yours, already posted or too many");
|
||||
|
||||
var scheduled = new ScheduledStatus { AvatarId = MyId, ScheduledAt = at, Params = asked };
|
||||
await DB.Default.SaveAsync(scheduled, token);
|
||||
scheduled.ID = (string)scheduled.GenerateNewID();
|
||||
// reserved in one conditional update: a post or another scheduled post taking one meanwhile wins
|
||||
if (mediaIds.Count > 0)
|
||||
await DB.Default.Update<Models.Media.MediaAttachment>().Match(m => mediaIds.Contains(m.ID)).Modify(m => m.ScheduledStatusId, scheduled.ID).ExecuteAsync(token);
|
||||
{
|
||||
var reserved = await DB.Default.Update<Models.Media.MediaAttachment>()
|
||||
.Match(m => mediaIds.Contains(m.ID) && m.PostId == null && m.ScheduledStatusId == null && m.TrashedAt == null)
|
||||
.Modify(m => m.ScheduledStatusId, scheduled.ID).ExecuteAsync(token);
|
||||
if (reserved.MatchedCount != mediaIds.Count)
|
||||
{
|
||||
await DB.Default.Update<Models.Media.MediaAttachment>().Match(m => m.ScheduledStatusId == scheduled.ID)
|
||||
.Modify(m => m.ScheduledStatusId, null).ExecuteAsync(token);
|
||||
return Error(StatusCodes.Status422UnprocessableEntity, "Validation failed: Media attachments are not yours, already posted or too many");
|
||||
}
|
||||
}
|
||||
await DB.Default.SaveAsync(scheduled, token);
|
||||
await _jobs.EnqueueMany(new[] { ScheduledStatuses.JobFor(scheduled, new Uri(Me.BaseAddress).Host) }, token);
|
||||
if (cacheKey != default)
|
||||
_cache.Set(cacheKey + ":scheduled", scheduled.ID, TimeSpan.FromHours(1));
|
||||
@@ -154,11 +167,41 @@ namespace PrivaPub.Api.Mastodon.Controllers
|
||||
SpoilerText = Params.Get("spoiler_text"),
|
||||
Sensitive = Params.Bool("sensitive") ?? false,
|
||||
Language = Params.Get("language"),
|
||||
MediaIds = Params.Has("media_ids") ? Params.List("media_ids") : default
|
||||
MediaIds = Params.Has("media_ids") ? Params.List("media_ids") : default,
|
||||
MediaChanges = MediaAttributes()
|
||||
}, token);
|
||||
return outcome.Ok ? Json(await _mapper.Status(outcome.Post, MyId, token)) : Error(outcome.Status, outcome.Error);
|
||||
}
|
||||
|
||||
// Mastodon's media_attributes: a JSON body's array of objects, or a form's media_attributes[i][field] or
|
||||
// media_attributes[][field] (each field then listed in the same order)
|
||||
IReadOnlyList<MediaChange> MediaAttributes()
|
||||
{
|
||||
var changes = new List<MediaChange>();
|
||||
if (Params.Json is { ValueKind: System.Text.Json.JsonValueKind.Object } json && json.TryGetProperty("media_attributes", out var array)
|
||||
&& array.ValueKind == System.Text.Json.JsonValueKind.Array)
|
||||
{
|
||||
foreach (var item in array.EnumerateArray().Where(i => i.ValueKind == System.Text.Json.JsonValueKind.Object).Take(4))
|
||||
{
|
||||
var hasDescription = item.TryGetProperty("description", out var description);
|
||||
changes.Add(new MediaChange(Text(item, "id"), hasDescription, hasDescription ? Text(item, "description") : default, Text(item, "focus")));
|
||||
}
|
||||
return changes;
|
||||
}
|
||||
for (var i = 0; i < 4 && Params.Get($"media_attributes[{i}][id]") is { } id; i++)
|
||||
changes.Add(new MediaChange(id, Params.Has($"media_attributes[{i}][description]"), Params.Get($"media_attributes[{i}][description]"), Params.Get($"media_attributes[{i}][focus]")));
|
||||
var ids = Params.List("media_attributes[][id]");
|
||||
var descriptions = Params.List("media_attributes[][description]");
|
||||
var foci = Params.List("media_attributes[][focus]");
|
||||
for (var i = 0; i < Math.Min(ids.Count, 4); i++)
|
||||
changes.Add(new MediaChange(ids[i], i < descriptions.Count, i < descriptions.Count ? descriptions[i] : default, i < foci.Count ? foci[i] : default));
|
||||
return changes;
|
||||
|
||||
static string Text(System.Text.Json.JsonElement item, string name) => item.TryGetProperty(name, out var value)
|
||||
? value.ValueKind == System.Text.Json.JsonValueKind.String ? value.GetString() : value.ValueKind is System.Text.Json.JsonValueKind.Null ? default : value.GetRawText()
|
||||
: default;
|
||||
}
|
||||
|
||||
[HttpDelete("/api/v1/statuses/{id}"), Scope("write:statuses")]
|
||||
public async Task<IActionResult> Delete(string id, CancellationToken token)
|
||||
{
|
||||
|
||||
@@ -89,6 +89,9 @@ namespace PrivaPub.Api.Mastodon.Infrastructure
|
||||
Flatten(values, prefix == default ? property.Name : $"{prefix}[{property.Name}]", property.Value);
|
||||
break;
|
||||
case JsonValueKind.Array:
|
||||
// an empty array is still said: media_ids: [] on an edit takes every attachment away
|
||||
if (prefix != default && element.GetArrayLength() == 0)
|
||||
values.TryAdd(prefix, new List<string>());
|
||||
foreach (var item in element.EnumerateArray())
|
||||
{
|
||||
if (item.ValueKind is JsonValueKind.Object or JsonValueKind.Array)
|
||||
|
||||
@@ -642,7 +642,7 @@ namespace PrivaPub.Api.Mastodon.Mappers
|
||||
meta["original"] = original;
|
||||
if (media.DurationSeconds is { } length)
|
||||
meta["length"] = TimeSpan.FromSeconds(length).ToString(@"h\:mm\:ss\.ff", CultureInfo.InvariantCulture);
|
||||
if (media.Focus is { Length: 2 })
|
||||
if (FocalPoint.IsSound(media.Focus))
|
||||
meta["focus"] = new { x = media.Focus[0], y = media.Focus[1] };
|
||||
return meta;
|
||||
}
|
||||
@@ -650,9 +650,9 @@ namespace PrivaPub.Api.Mastodon.Mappers
|
||||
MediaAttachment Media(PostMedia media) => new()
|
||||
{
|
||||
Id = media.AttachmentId ?? media.Id.ToString("N"),
|
||||
Type = media.ContentType switch
|
||||
// a GIF is a gifv only once it is an mp4 (ours): a GIF itself is an image, which a gifv player can't play
|
||||
Type = media.Kind == "gifv" ? "gifv" : media.ContentType switch
|
||||
{
|
||||
{ } type when type.StartsWith("image/gif") => "gifv",
|
||||
{ } type when type.StartsWith("image/") => "image",
|
||||
{ } type when type.StartsWith("video/") => "video",
|
||||
{ } type when type.StartsWith("audio/") => "audio",
|
||||
|
||||
@@ -0,0 +1,280 @@
|
||||
using Microsoft.AspNetCore.Authorization;
|
||||
using Microsoft.AspNetCore.Http.Features;
|
||||
using Microsoft.AspNetCore.Mvc;
|
||||
using Microsoft.Extensions.Localization;
|
||||
using Microsoft.Net.Http.Headers;
|
||||
|
||||
using MongoDB.Entities;
|
||||
|
||||
using PrivaPub.ClientModels;
|
||||
using PrivaPub.ClientModels.Admin;
|
||||
using PrivaPub.Extensions;
|
||||
using PrivaPub.Infrastructure.Backup;
|
||||
using PrivaPub.Models.User;
|
||||
using PrivaPub.Resources;
|
||||
using PrivaPub.StaticServices;
|
||||
|
||||
namespace PrivaPub.Controllers.ClientToServer
|
||||
{
|
||||
// The server's backups from the administrator's page (owner decision 2026-10-07: it backs up and restores too). Only
|
||||
// an administrator, checked again against the database; a download and a restore ask for the password again, and a
|
||||
// restore for the server's host typed out. A restore asked for stops the service within seconds, and the next start
|
||||
// carries it out (ServerRestore).
|
||||
[ApiController,
|
||||
Route("clientapi/admin/backups"),
|
||||
Authorize(Policy = Policies.IsAdmin)]
|
||||
public class BackupController(Backups backups, TransferStore transfers, IPasswordHasher hasher, IStringLocalizer<GenericRes> localizer,
|
||||
ILogger<BackupController> logger) : ControllerBase
|
||||
{
|
||||
[HttpGet, Route("")]
|
||||
public async Task<IActionResult> List(CancellationToken token)
|
||||
{
|
||||
if (await Administrator(token) == default)
|
||||
return Forbid();
|
||||
var deleted = await DB.Default.Find<RootUser>().Match(u => u.DeletedAt != null).Project(u => u.Include(x => x.DeletedAt)).ExecuteAsync(token);
|
||||
var code = ServerBackup.CodeMigration();
|
||||
var waiting = RestoreMarker.Read(backups.Root);
|
||||
var last = await DB.Default.Find<RestoreRecord>().Sort(r => r.RestoredAt, Order.Descending).ExecuteFirstAsync(token);
|
||||
ServerBackup.MakeDirectory(backups.Root);
|
||||
return Ok(new ViewBackups
|
||||
{
|
||||
Backups = [.. backups.List().Select(b => View(b, deleted.Count(u => u.DeletedAt > b.CreatedAt), code))],
|
||||
Running = (await MaintenanceLock.Current(token))?.What,
|
||||
Waiting = waiting == default ? default : new ViewRestoreWaiting
|
||||
{
|
||||
Backup = waiting.Backup,
|
||||
State = waiting.State,
|
||||
Attempts = waiting.Attempts,
|
||||
RequestedBy = waiting.RequestedBy,
|
||||
RequestedAt = waiting.RequestedAt,
|
||||
Error = waiting.Error
|
||||
},
|
||||
LastRestore = last == default ? default : View(last),
|
||||
Host = HostName,
|
||||
FreeBytes = new DriveInfo(backups.Root).AvailableFreeSpace
|
||||
});
|
||||
}
|
||||
|
||||
// a backup made now, while the page polls the list
|
||||
[HttpPost, Route("")]
|
||||
public async Task<IActionResult> Create(CancellationToken token)
|
||||
{
|
||||
var admin = await Administrator(token);
|
||||
if (admin == default)
|
||||
return Forbid();
|
||||
if (await MaintenanceLock.Current(token) is { } held)
|
||||
return Conflict(new WebResult().Invalidate(localizer["A {0} is running.", held.What]));
|
||||
_ = Task.Run(async () =>
|
||||
{
|
||||
try
|
||||
{
|
||||
var (made, error) = await backups.Create("manual", dbOnly: false, CancellationToken.None);
|
||||
if (made == default)
|
||||
logger.LogWarning("The backup {Admin} asked for was not made: {Error}", admin.UserName, error);
|
||||
else
|
||||
logger.LogInformation("Backup {Id} made for {Admin}", made.Id, admin.UserName);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
logger.LogError(ex, "The backup {Admin} asked for failed", admin.UserName);
|
||||
}
|
||||
}, CancellationToken.None);
|
||||
return Accepted();
|
||||
}
|
||||
|
||||
[HttpDelete, Route("{id}")]
|
||||
public async Task<IActionResult> Delete(string id, CancellationToken token)
|
||||
{
|
||||
if (await Administrator(token) == default)
|
||||
return Forbid();
|
||||
if (RestoreMarker.Read(backups.Root) is { } waiting && (waiting.Backup == id || waiting.PreRestore == id))
|
||||
return Conflict(new WebResult().Invalidate(localizer["A restore waits for this backup."]));
|
||||
return backups.Delete(id) ? Ok() : NotFound();
|
||||
}
|
||||
|
||||
// a link to download it with, for the password
|
||||
[HttpPost, Route("{id}/ticket")]
|
||||
public async Task<IActionResult> Ticket(string id, BackupPasswordForm form, CancellationToken token)
|
||||
{
|
||||
var admin = await Administrator(token);
|
||||
if (admin == default)
|
||||
return Forbid();
|
||||
if (!PasswordHolds(admin, form.Password))
|
||||
return UnprocessableEntity(new WebResult().Invalidate(localizer["Wrong password."]));
|
||||
if (backups.Find(id) == default)
|
||||
return NotFound();
|
||||
var (ticket, expires) = transfers.Ticket(id);
|
||||
return Ok(new ViewBackupTicket { Url = $"/clientapi/admin/backups/download/{ticket}", ExpiresAt = expires, Bytes = BackupTar.Of(backups.Root, id).Length });
|
||||
}
|
||||
|
||||
// the backup as one tar, for a ticket: a plain link, so the browser downloads it to disk and resumes it
|
||||
[HttpGet, Route("download/{ticket}"), AllowAnonymous]
|
||||
public async Task Download(string ticket, CancellationToken token)
|
||||
{
|
||||
var id = transfers.Redeem(ticket);
|
||||
if (id == default || backups.Find(id) == default)
|
||||
{
|
||||
Response.StatusCode = StatusCodes.Status404NotFound;
|
||||
return;
|
||||
}
|
||||
HttpContext.Features.Get<IHttpResponseBodyFeature>()?.DisableBuffering();
|
||||
var tar = BackupTar.Of(backups.Root, id);
|
||||
var (from, to) = (0L, tar.Length - 1);
|
||||
Response.Headers.AcceptRanges = "bytes";
|
||||
Response.Headers.CacheControl = "no-store";
|
||||
Response.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment") { FileName = $"privapub-{id}.tar" }.ToString();
|
||||
var range = Request.GetTypedHeaders().Range;
|
||||
if (range is { Unit.Value: "bytes", Ranges.Count: 1 })
|
||||
{
|
||||
var asked = range.Ranges.First();
|
||||
var start = asked.From ?? tar.Length - asked.To.GetValueOrDefault();
|
||||
var end = asked.From.HasValue ? Math.Min(asked.To ?? tar.Length - 1, tar.Length - 1) : tar.Length - 1;
|
||||
if (start < 0 || start > end)
|
||||
{
|
||||
Response.StatusCode = StatusCodes.Status416RangeNotSatisfiable;
|
||||
Response.Headers.ContentRange = $"bytes */{tar.Length}";
|
||||
return;
|
||||
}
|
||||
(from, to) = (start, end);
|
||||
Response.StatusCode = StatusCodes.Status206PartialContent;
|
||||
Response.Headers.ContentRange = $"bytes {from}-{to}/{tar.Length}";
|
||||
}
|
||||
Response.ContentType = "application/x-tar";
|
||||
Response.ContentLength = to - from + 1;
|
||||
await tar.Write(Response.Body, from, to, token);
|
||||
}
|
||||
|
||||
[HttpPost, Route("uploads")]
|
||||
public async Task<IActionResult> StartUpload(CancellationToken token)
|
||||
{
|
||||
var admin = await Administrator(token);
|
||||
if (admin == default)
|
||||
return Forbid();
|
||||
return Ok(new ViewBackupUpload { Id = transfers.Start(admin.UserName), ChunkBytes = TransferStore.ChunkBytes });
|
||||
}
|
||||
|
||||
[HttpGet, Route("uploads/{id}")]
|
||||
public async Task<IActionResult> Upload(string id, CancellationToken token)
|
||||
{
|
||||
if (await Administrator(token) == default)
|
||||
return Forbid();
|
||||
var received = transfers.Received(id);
|
||||
return received < 0 ? NotFound() : Ok(new ViewBackupUpload { Id = id, Received = received, ChunkBytes = TransferStore.ChunkBytes });
|
||||
}
|
||||
|
||||
// a piece, at the offset already received: 409 with what was received when it isn't
|
||||
[HttpPut, Route("uploads/{id}"), RequestSizeLimit(TransferStore.ChunkBytes + 1024 * 1024)]
|
||||
public async Task<IActionResult> Append(string id, [FromQuery] long offset, CancellationToken token)
|
||||
{
|
||||
if (await Administrator(token) == default)
|
||||
return Forbid();
|
||||
var (received, taken) = await transfers.Append(id, offset, Request.Body, token);
|
||||
if (received < 0)
|
||||
return NotFound();
|
||||
var view = new ViewBackupUpload { Id = id, Received = received, ChunkBytes = TransferStore.ChunkBytes };
|
||||
return taken ? Ok(view) : Conflict(view);
|
||||
}
|
||||
|
||||
[HttpPost, Route("uploads/{id}/finish")]
|
||||
public async Task<IActionResult> FinishUpload(string id, CancellationToken token)
|
||||
{
|
||||
if (await Administrator(token) == default)
|
||||
return Forbid();
|
||||
if (await MaintenanceLock.Current(token) is { What: "restore" })
|
||||
return Conflict(new WebResult().Invalidate(localizer["A {0} is running.", "restore"]));
|
||||
var (backup, error) = await transfers.Finish(id, token);
|
||||
if (backup == default)
|
||||
return UnprocessableEntity(new WebResult().Invalidate(error));
|
||||
return Ok(View(backup, 0, ServerBackup.CodeMigration()));
|
||||
}
|
||||
|
||||
[HttpDelete, Route("uploads/{id}")]
|
||||
public async Task<IActionResult> CancelUpload(string id, CancellationToken token)
|
||||
{
|
||||
if (await Administrator(token) == default)
|
||||
return Forbid();
|
||||
return transfers.Cancel(id) ? Ok() : NotFound();
|
||||
}
|
||||
|
||||
// asked for with the password and the host typed out: the service stops within seconds and restores it as it starts
|
||||
[HttpPost, Route("{id}/restore")]
|
||||
public async Task<IActionResult> Restore(string id, RestoreBackupForm form, CancellationToken token)
|
||||
{
|
||||
var admin = await Administrator(token);
|
||||
if (admin == default)
|
||||
return Forbid();
|
||||
if (!PasswordHolds(admin, form.Password))
|
||||
return UnprocessableEntity(new WebResult().Invalidate(localizer["Wrong password."]));
|
||||
if (!string.Equals(form.Host?.Trim(), HostName, StringComparison.OrdinalIgnoreCase))
|
||||
return UnprocessableEntity(new WebResult().Invalidate(localizer["Type this server's host to restore it."]));
|
||||
var refused = await ServerRestore.Request(backups.Context(), id, admin.UserName, token);
|
||||
if (refused != default)
|
||||
return UnprocessableEntity(new WebResult().Invalidate(refused));
|
||||
logger.LogWarning("{Admin} asked to restore {Backup}", admin.UserName, id);
|
||||
return Accepted();
|
||||
}
|
||||
|
||||
string HostName => Uri.TryCreate(backups.Host, UriKind.Absolute, out var host) ? host.Authority : backups.Host;
|
||||
|
||||
// the token says administrator; the database has the last word
|
||||
async Task<RootUser> Administrator(CancellationToken token)
|
||||
{
|
||||
var root = await DB.Default.Find<RootUser>().MatchID(User.GetUserId()).ExecuteFirstAsync(token);
|
||||
return root is { IsBanned: false, DeletedAt: null } && root.Policies.Contains(Policies.IsAdmin) ? root : default;
|
||||
}
|
||||
|
||||
bool PasswordHolds(RootUser root, string password) =>
|
||||
!string.IsNullOrEmpty(password) && !string.IsNullOrEmpty(root.HashedPassword) && hasher.Check(root.HashedPassword, password).verified;
|
||||
|
||||
static ViewBackup View(BackupInfo backup, int rootsDeletedSince, int code)
|
||||
{
|
||||
var manifest = backup.Manifest;
|
||||
return new ViewBackup
|
||||
{
|
||||
Id = backup.Id,
|
||||
Kind = backup.Kind,
|
||||
CreatedAt = backup.CreatedAt,
|
||||
Bytes = backup.Bytes,
|
||||
MediaBytes = manifest.Media.Bytes,
|
||||
MediaFiles = manifest.DbOnly ? manifest.Media.List.Count : manifest.Media.Files,
|
||||
MediaMissing = manifest.Media.Missing,
|
||||
DbOnly = manifest.DbOnly,
|
||||
Consistent = manifest.Consistent,
|
||||
Collections = manifest.Collections.Count,
|
||||
Documents = manifest.Collections.Sum(c => c.Count),
|
||||
AppCommit = manifest.AppCommit,
|
||||
MigrationNumber = manifest.MigrationNumber,
|
||||
RootsDeletedSince = rootsDeletedSince,
|
||||
NotRestorable = manifest.Format != ArchiveManifest.CurrentFormat ? "format"
|
||||
: manifest.MigrationNumber > code ? "newer"
|
||||
: default
|
||||
};
|
||||
}
|
||||
|
||||
static ViewRestore View(RestoreRecord record) => new()
|
||||
{
|
||||
Backup = record.Backup,
|
||||
BackupCreatedAt = record.BackupCreatedAt,
|
||||
PreRestore = record.PreRestore,
|
||||
RequestedBy = record.RequestedBy,
|
||||
RestoredAt = record.RestoredAt,
|
||||
Abandoned = record.Abandoned,
|
||||
Error = record.Error,
|
||||
Documents = record.Report.Documents,
|
||||
Collections = record.Report.Collections,
|
||||
MediaRestored = record.Report.MediaRestored,
|
||||
MediaMissing = record.Report.MediaMissing,
|
||||
RootsDeleted = record.Report.RootsDeleted,
|
||||
PersonasDeleted = record.Report.PersonasDeleted,
|
||||
GroupsDeleted = record.Report.GroupsDeleted,
|
||||
PostsDeleted = record.Report.PostsDeleted,
|
||||
RootsTombstoned = record.Report.RootsTombstoned,
|
||||
PersonasTombstoned = record.Report.PersonasTombstoned,
|
||||
GroupsTombstoned = record.Report.GroupsTombstoned,
|
||||
PostsGone = record.Report.PostsGone,
|
||||
MediaTrashed = record.Report.MediaTrashed,
|
||||
ProtectiveKept = record.Report.ProtectiveKept,
|
||||
SessionsEnded = record.Report.SessionsEnded
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -19,11 +19,13 @@ namespace PrivaPub.Controllers.ClientToServer
|
||||
{
|
||||
readonly IDomainBlocks _domainBlocks;
|
||||
readonly IStringLocalizer _localizer;
|
||||
readonly Domain.Media.IMediaProxy _proxy;
|
||||
|
||||
public DomainBlockController(IDomainBlocks domainBlocks, IStringLocalizer<GenericRes> localizer)
|
||||
public DomainBlockController(IDomainBlocks domainBlocks, IStringLocalizer<GenericRes> localizer, Domain.Media.IMediaProxy proxy)
|
||||
{
|
||||
_domainBlocks = domainBlocks;
|
||||
_localizer = localizer;
|
||||
_proxy = proxy;
|
||||
}
|
||||
|
||||
[HttpGet, Route("/clientapi/admin/domainblocks/list")]
|
||||
@@ -48,6 +50,9 @@ namespace PrivaPub.Controllers.ClientToServer
|
||||
.Option(o => o.IsUpsert = true)
|
||||
.ExecuteAsync(token);
|
||||
await _domainBlocks.Reload(token);
|
||||
// what the media proxy holds of it goes with the block
|
||||
if (block.Severity == DomainBlockSeverity.Suspend || block.RejectMedia)
|
||||
_proxy.Purge(domain);
|
||||
return Ok(ToView(block));
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,165 @@
|
||||
using Microsoft.AspNetCore.Authorization;
|
||||
using Microsoft.AspNetCore.Mvc;
|
||||
|
||||
using MongoDB.Entities;
|
||||
|
||||
using PrivaPub.ClientModels;
|
||||
using PrivaPub.ClientModels.User;
|
||||
using PrivaPub.Domain.Portability;
|
||||
using PrivaPub.Extensions;
|
||||
using PrivaPub.Infrastructure.Backup;
|
||||
using PrivaPub.Infrastructure.Jobs;
|
||||
using PrivaPub.Models.User;
|
||||
|
||||
namespace PrivaPub.Controllers.ClientToServer
|
||||
{
|
||||
// A persona's archive for its root (owner decision 2026-10-07): made in a job and downloaded through a link good for
|
||||
// ten minutes; another archive (its own, or Mastodon's) uploaded in pieces and imported into it in a job, the parts the
|
||||
// root picks, with progress, until it asks to stop. Only the persona's own root, never a banned one.
|
||||
[ApiController,
|
||||
Route("clientapi/persona"),
|
||||
Authorize(Policy = Policies.IsUser)]
|
||||
public class PersonaArchiveController(Backups backups, TransferStore transfers, IJobQueue jobs, Federation.Actors.ILocalActorService localActors)
|
||||
: ControllerBase
|
||||
{
|
||||
[HttpGet, Route("{avatarId}/archive")]
|
||||
public async Task<IActionResult> Status(string avatarId, CancellationToken token)
|
||||
{
|
||||
if (!await Owns(avatarId, token))
|
||||
return NotFound();
|
||||
await PersonaArchives.Forget(backups.Root, token);
|
||||
return Ok(PersonaArchives.View(await PersonaArchives.Of(avatarId, token)));
|
||||
}
|
||||
|
||||
[HttpPost, Route("{avatarId}/archive")]
|
||||
public async Task<IActionResult> Export(string avatarId, CancellationToken token)
|
||||
{
|
||||
if (!await Owns(avatarId, token))
|
||||
return NotFound();
|
||||
var state = await PersonaArchives.Of(avatarId, token);
|
||||
if (state.ExportState is "queued" or "running")
|
||||
return Conflict(new WebResult().Invalidate("The archive is being made."));
|
||||
state.ExportState = "queued";
|
||||
state.ExportAskedAt = DateTime.UtcNow;
|
||||
state.ExportError = default;
|
||||
await DB.Default.SaveAsync(state, token);
|
||||
await jobs.EnqueueMany([ExportArchiveJob.JobFor(avatarId, Host)], token);
|
||||
return Accepted(PersonaArchives.View(state));
|
||||
}
|
||||
|
||||
[HttpPost, Route("{avatarId}/archive/ticket")]
|
||||
public async Task<IActionResult> Ticket(string avatarId, CancellationToken token)
|
||||
{
|
||||
if (!await Owns(avatarId, token))
|
||||
return NotFound();
|
||||
var state = await PersonaArchives.Of(avatarId, token);
|
||||
var file = state.ExportFile == default ? default : Path.Combine(PersonaArchives.Folder(backups.Root, avatarId), state.ExportFile);
|
||||
if (state.ExportState != "ready" || file == default || !System.IO.File.Exists(file))
|
||||
return NotFound();
|
||||
var (ticket, expires) = transfers.Ticket($"persona/{avatarId}/{state.ExportFile}");
|
||||
return Ok(new ViewArchiveTicket { Url = $"/clientapi/persona/archive/download/{ticket}", ExpiresAt = expires, Bytes = new FileInfo(file).Length });
|
||||
}
|
||||
|
||||
// the archive itself, for a ticket: a plain link, so the browser saves it to disk (and resumes it)
|
||||
[HttpGet, Route("archive/download/{ticket}"), AllowAnonymous]
|
||||
public IActionResult Download(string ticket)
|
||||
{
|
||||
if (transfers.Redeem(ticket) is not { } what || what.Split('/') is not ["persona", var avatarId, var name] || name.Contains(".."))
|
||||
return NotFound();
|
||||
var file = Path.Combine(PersonaArchives.Folder(backups.Root, avatarId), name);
|
||||
if (!System.IO.File.Exists(file))
|
||||
return NotFound();
|
||||
Response.Headers.CacheControl = "no-store";
|
||||
return PhysicalFile(file, "application/zip", name, enableRangeProcessing: true);
|
||||
}
|
||||
|
||||
[HttpPost, Route("archive/uploads")]
|
||||
public IActionResult StartUpload() =>
|
||||
Ok(new ViewArchiveUpload { Id = transfers.Start(User.GetUserId()), ChunkBytes = TransferStore.ChunkBytes });
|
||||
|
||||
[HttpGet, Route("archive/uploads/{id}")]
|
||||
public IActionResult Upload(string id)
|
||||
{
|
||||
if (transfers.Owner(id) != User.GetUserId())
|
||||
return NotFound();
|
||||
return Ok(new ViewArchiveUpload { Id = id, Received = transfers.Received(id), ChunkBytes = TransferStore.ChunkBytes });
|
||||
}
|
||||
|
||||
[HttpPut, Route("archive/uploads/{id}"), RequestSizeLimit(TransferStore.ChunkBytes + 1024 * 1024)]
|
||||
public async Task<IActionResult> Append(string id, [FromQuery] long offset, CancellationToken token)
|
||||
{
|
||||
if (transfers.Owner(id) != User.GetUserId())
|
||||
return NotFound();
|
||||
var (received, taken) = await transfers.Append(id, offset, Request.Body, token);
|
||||
var view = new ViewArchiveUpload { Id = id, Received = received, ChunkBytes = TransferStore.ChunkBytes };
|
||||
return taken ? Ok(view) : Conflict(view);
|
||||
}
|
||||
|
||||
[HttpDelete, Route("archive/uploads/{id}")]
|
||||
public IActionResult CancelUpload(string id)
|
||||
{
|
||||
if (transfers.Owner(id) != User.GetUserId())
|
||||
return NotFound();
|
||||
transfers.Cancel(id);
|
||||
return Ok();
|
||||
}
|
||||
|
||||
// an uploaded archive imported into the persona: refused at once when it is not one, or holds what it must not
|
||||
[HttpPost, Route("{avatarId}/archive/import")]
|
||||
public async Task<IActionResult> Import(string avatarId, ImportPersonaArchiveForm form, CancellationToken token)
|
||||
{
|
||||
if (!await Owns(avatarId, token))
|
||||
return NotFound();
|
||||
var state = await PersonaArchives.Of(avatarId, token);
|
||||
if (state.ImportState is "queued" or "running")
|
||||
return Conflict(new WebResult().Invalidate("An import is running."));
|
||||
if (form.Parts.Count == 0 || form.Parts.Any(p => !PersonaArchives.Parts.Contains(p)))
|
||||
return UnprocessableEntity(new WebResult().Invalidate("Choose what to import."));
|
||||
if (transfers.Owner(form.UploadId) != User.GetUserId())
|
||||
return NotFound();
|
||||
var path = transfers.Take(form.UploadId, PersonaArchives.Folder(backups.Root, avatarId), $"import-{form.UploadId}.zip");
|
||||
if (path == default)
|
||||
return NotFound();
|
||||
var (archive, refused) = SafeArchive.Open(path);
|
||||
using (archive)
|
||||
if (archive?.Has("actor.json") != true)
|
||||
{
|
||||
System.IO.File.Delete(path);
|
||||
return UnprocessableEntity(new WebResult().Invalidate(refused ?? "the archive has no actor.json"));
|
||||
}
|
||||
state.ImportState = "queued";
|
||||
state.ImportAskedAt = DateTime.UtcNow;
|
||||
state.ImportEndedAt = default;
|
||||
state.ImportParts = [.. form.Parts.Distinct()];
|
||||
state.ImportFile = path;
|
||||
state.ImportCounts = [];
|
||||
(state.ImportDone, state.ImportTotal, state.ImportError, state.StopAsked) = (0, 0, default, false);
|
||||
await DB.Default.SaveAsync(state, token);
|
||||
await jobs.EnqueueMany([ImportArchiveJob.JobFor(avatarId, Host)], token);
|
||||
return Accepted(PersonaArchives.View(state));
|
||||
}
|
||||
|
||||
[HttpPost, Route("{avatarId}/archive/import/stop")]
|
||||
public async Task<IActionResult> Stop(string avatarId, CancellationToken token)
|
||||
{
|
||||
if (!await Owns(avatarId, token))
|
||||
return NotFound();
|
||||
await DB.Default.Update<PersonaArchive>().MatchID(avatarId).Match(a => a.ImportState == "queued" || a.ImportState == "running")
|
||||
.Modify(a => a.StopAsked, true).ExecuteAsync(token);
|
||||
return Ok();
|
||||
}
|
||||
|
||||
string Host => new Uri(localActors.BaseAddress).Host;
|
||||
|
||||
// the root's own persona, and a root that may still act
|
||||
async Task<bool> Owns(string avatarId, CancellationToken token)
|
||||
{
|
||||
var rootId = User.GetUserId();
|
||||
if (string.IsNullOrEmpty(avatarId) || !await DB.Default.Find<RootToAvatar>().Match(r => r.RootId == rootId && r.AvatarId == avatarId).ExecuteAnyAsync(token))
|
||||
return false;
|
||||
var root = await DB.Default.Find<RootUser>().MatchID(rootId).ExecuteFirstAsync(token);
|
||||
return root is { IsBanned: false, DeletedAt: null }
|
||||
&& await DB.Default.Find<Avatar>().MatchID(avatarId).Match(a => !a.DeletionAt.HasValue).ExecuteAnyAsync(token);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -328,6 +328,15 @@ namespace PrivaPub.Controllers.ClientToServer
|
||||
// }
|
||||
//}
|
||||
|
||||
// what the login's media take, every persona's, and its quota when the server sets one
|
||||
[HttpGet, Route("/clientapi/user/storage"), Authorize(Policy = Policies.IsUser)]
|
||||
public async Task<IActionResult> GetStorage([FromServices] Microsoft.Extensions.Options.IOptionsMonitor<Domain.Media.MediaOptions> media, CancellationToken token)
|
||||
{
|
||||
var (bytes, files) = await Domain.Privacy.Counted.MediaOfRoot(User.GetUserId(), token);
|
||||
var quota = media.CurrentValue.QuotaBytesPerRoot;
|
||||
return Ok(new PrivaPub.ClientModels.User.ViewStorage { MediaBytes = bytes, MediaFiles = files, QuotaBytes = quota > 0 ? quota : null });
|
||||
}
|
||||
|
||||
[HttpGet, Route("/clientapi/user/settings"), Authorize(Policy = Policies.IsUser)]
|
||||
public async Task<IActionResult> GetUserSettings()
|
||||
{
|
||||
|
||||
@@ -32,8 +32,16 @@ namespace PrivaPub.Controllers.ClientToServer
|
||||
Ok(await _queries.Crawler(_options.CurrentValue.Crawler.Enabled, _options.CurrentValue.Crawler.RevisitDays, token));
|
||||
|
||||
[HttpGet, Route("/clientapi/admin/statistics/overview")]
|
||||
public async Task<IActionResult> Overview([FromQuery] int days = 30, CancellationToken token = default) =>
|
||||
Ok(await _queries.Overview(days, token));
|
||||
public async Task<IActionResult> Overview([FromServices] Domain.Media.IMediaService media, [FromQuery] int days = 30, CancellationToken token = default)
|
||||
{
|
||||
var overview = await _queries.Overview(days, token);
|
||||
overview.Media.ProxyCacheBytes = Bytes(media.ProxyRoot);
|
||||
overview.Media.TrashBytes = Bytes(media.TrashRoot);
|
||||
return Ok(overview);
|
||||
}
|
||||
|
||||
static long Bytes(string root) =>
|
||||
Directory.Exists(root) ? new DirectoryInfo(root).EnumerateFiles("*", SearchOption.AllDirectories).Sum(f => f.Length) : 0;
|
||||
|
||||
[HttpGet, Route("/clientapi/admin/statistics/hosts")]
|
||||
public async Task<IActionResult> Hosts([FromQuery] int days = 30, [FromQuery] string sort = default, [FromQuery] string software = default,
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
using System.Globalization;
|
||||
|
||||
namespace PrivaPub.Domain.Media
|
||||
{
|
||||
// A focal point as Mastodon's API takes it ("x,y", each within -1..1): finite numbers only, clamped to the frame. A NaN
|
||||
// once stored here made every status, timeline and Note holding the post fail to serialise, for every viewer.
|
||||
public static class FocalPoint
|
||||
{
|
||||
public static float[] Parse(string value)
|
||||
{
|
||||
if (value?.Split(',') is not [var x, var y]
|
||||
|| !float.TryParse(x, NumberStyles.Float, CultureInfo.InvariantCulture, out var fx) || !float.IsFinite(fx)
|
||||
|| !float.TryParse(y, NumberStyles.Float, CultureInfo.InvariantCulture, out var fy) || !float.IsFinite(fy))
|
||||
return default;
|
||||
return [Math.Clamp(fx, -1, 1), Math.Clamp(fy, -1, 1)];
|
||||
}
|
||||
|
||||
/// <summary>Whether a stored focal point can be written out: two finite numbers.</summary>
|
||||
public static bool IsSound(float[] focus) => focus is [var x, var y] && float.IsFinite(x) && float.IsFinite(y);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
using MongoDB.Entities;
|
||||
|
||||
using PrivaPub.Models.Media;
|
||||
using PrivaPub.Models.User;
|
||||
|
||||
using PostEntity = PrivaPub.Models.Post.Post;
|
||||
|
||||
namespace PrivaPub.Domain.Media
|
||||
{
|
||||
// What the media directory holds against what the database says holds it, and, with fix, the two made to agree:
|
||||
// - a persona's current avatar or header with no row (they had none before) becomes a row of its own;
|
||||
// - media of a deleted post or of a deleted persona go to the trash;
|
||||
// - rows whose files are gone go to the trash;
|
||||
// - files nothing holds (deleted posts' media and replaced pictures, from before media were trashed) get a trashed row
|
||||
// each.
|
||||
// Trashed files leave what /media/files serves at once, and the janitor deletes them after its grace. Run it without
|
||||
// fix first; with fix it must run as the user that owns the media (www-data).
|
||||
public static class MediaAudit
|
||||
{
|
||||
public sealed record Report(int Files, int Held, int Adopted, int OfDeleted, int MissingFiles, int Unheld, IReadOnlyList<string> Examples, bool Fixed);
|
||||
|
||||
public static async Task<Report> Run(IMediaService media, bool fix, CancellationToken token)
|
||||
{
|
||||
var files = Served(media.Root);
|
||||
var rows = await DB.Default.Find<MediaAttachment>().Match(m => m.TrashedAt == null).ExecuteAsync(token);
|
||||
var held = rows.SelectMany(r => new[] { r.FilePath, r.PreviewPath }).Where(p => !string.IsNullOrEmpty(p)).ToHashSet();
|
||||
var examples = new List<string>();
|
||||
|
||||
// pictures personas show today, kept as rows from now on
|
||||
var prefix = media.Url(string.Empty);
|
||||
var adopted = 0;
|
||||
foreach (var avatar in await DB.Default.Find<Avatar>().Match(a => !a.DeletionAt.HasValue && (a.PictureURL != null || a.ThumbnailURL != null)).ExecuteAsync(token))
|
||||
foreach (var (url, kind) in new[] { (avatar.PictureURL, "avatar"), (avatar.ThumbnailURL, "header") })
|
||||
{
|
||||
if (url?.StartsWith(prefix, StringComparison.Ordinal) != true)
|
||||
continue;
|
||||
var relative = url[prefix.Length..];
|
||||
if (held.Contains(relative) || !files.Contains(relative))
|
||||
continue;
|
||||
adopted++;
|
||||
held.Add(relative);
|
||||
if (fix)
|
||||
await DB.Default.SaveAsync(new MediaAttachment
|
||||
{
|
||||
OwnerAvatarId = avatar.ID,
|
||||
ProfileOfAvatarId = avatar.ID,
|
||||
Kind = kind,
|
||||
ContentType = "image/jpeg",
|
||||
FilePath = relative,
|
||||
Size = new FileInfo(Path.Combine(media.Root, relative)).Length,
|
||||
AttachedAt = DateTime.UtcNow
|
||||
}, token);
|
||||
}
|
||||
|
||||
// held by something that is gone: a deleted post, a deleted persona
|
||||
var postIds = rows.Where(r => r.PostId != null).Select(r => r.PostId).Distinct().ToList();
|
||||
var livePosts = (await DB.Default.Find<PostEntity>().Match(p => postIds.Contains(p.ID) && !p.DeletedAt.HasValue).Project(p => p.Include(x => x.ID)).ExecuteAsync(token))
|
||||
.Select(p => p.ID).ToHashSet();
|
||||
var ownerIds = rows.Select(r => r.OwnerAvatarId).Where(id => id != null).Distinct().ToList();
|
||||
var liveOwners = (await DB.Default.Find<Avatar>().Match(a => ownerIds.Contains(a.ID) && !a.DeletionAt.HasValue).Project(a => a.Include(x => x.ID)).ExecuteAsync(token))
|
||||
.Select(a => a.ID).ToHashSet();
|
||||
var ofDeleted = rows.Where(r => (r.PostId != null && !livePosts.Contains(r.PostId)) || (r.OwnerAvatarId != null && !liveOwners.Contains(r.OwnerAvatarId))).ToList();
|
||||
var missing = rows.Except(ofDeleted).Where(r => !string.IsNullOrEmpty(r.FilePath) && !files.Contains(r.FilePath)).ToList();
|
||||
if (fix)
|
||||
foreach (var row in ofDeleted.Concat(missing))
|
||||
await media.Trash(m => m.ID == row.ID, ofDeleted.Contains(row) ? "audit: its holder is deleted" : "audit: its file is missing", token);
|
||||
foreach (var row in ofDeleted)
|
||||
held.Remove(row.FilePath);
|
||||
|
||||
// files nothing holds
|
||||
var unheld = files.Where(f => !held.Contains(f)).OrderBy(f => f, StringComparer.Ordinal).ToList();
|
||||
examples.AddRange(unheld.Take(10));
|
||||
if (fix)
|
||||
foreach (var relative in unheld)
|
||||
{
|
||||
var orphan = new MediaAttachment
|
||||
{
|
||||
Kind = "orphan",
|
||||
FilePath = relative,
|
||||
Size = new FileInfo(Path.Combine(media.Root, relative)).Length,
|
||||
TrashedAt = DateTime.UtcNow,
|
||||
TrashReason = "audit: nothing holds it"
|
||||
};
|
||||
await DB.Default.SaveAsync(orphan, token);
|
||||
media.Hide(orphan);
|
||||
}
|
||||
|
||||
return new Report(files.Count, rows.Count, adopted, ofDeleted.Count, missing.Count, unheld.Count, examples, fix);
|
||||
}
|
||||
|
||||
// every file /media/files serves: everything under the root
|
||||
static HashSet<string> Served(string root)
|
||||
{
|
||||
if (!Directory.Exists(root))
|
||||
return new HashSet<string>();
|
||||
return Directory.EnumerateFiles(root, "*", SearchOption.AllDirectories)
|
||||
.Select(f => Path.GetRelativePath(root, f).Replace('\\', '/'))
|
||||
.ToHashSet();
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -9,5 +9,13 @@ namespace PrivaPub.Domain.Media
|
||||
public long ProxyCacheBytes { get; set; } = 5L * 1024 * 1024 * 1024;
|
||||
public int MaxImageSide { get; set; } = 4096;
|
||||
public int PreviewSide { get; set; } = 640;
|
||||
public long MaxPixels { get; set; } = 40_000_000;//an image (or a GIF's frames together) may not decode to more
|
||||
public int MaxFrames { get; set; } = 500;
|
||||
public int MaxGifSide { get; set; } = 1280;//a GIF becomes an mp4 at most this wide
|
||||
public int Concurrency { get; set; } = 2;//uploads processed at once
|
||||
public long MaxVideoPixels { get; set; } = 2_304_000;//a larger video is made smaller (as Mastodon's video_matrix_limit)
|
||||
public double MaxFrameRate { get; set; } = 60;
|
||||
public int MaxSeconds { get; set; } = 3600;//audio or video may not last longer
|
||||
public long QuotaBytesPerRoot { get; set; }//what a login's media may take (0: no quota)
|
||||
}
|
||||
}
|
||||
@@ -1,3 +1,5 @@
|
||||
using PrivaPub.Models.Federation;
|
||||
using System.Collections.Concurrent;
|
||||
using Microsoft.Extensions.Options;
|
||||
|
||||
using MongoDB.Entities;
|
||||
@@ -11,32 +13,77 @@ using System.Text;
|
||||
|
||||
namespace PrivaPub.Domain.Media
|
||||
{
|
||||
public enum ProxyOutcome
|
||||
{
|
||||
Cached,
|
||||
TooBig,
|
||||
Failed
|
||||
}
|
||||
|
||||
public interface IMediaProxy
|
||||
{
|
||||
string Wrap(string remoteUrl);
|
||||
Task<(string Path, string ContentType)> Fetch(string signature, string encodedUrl, CancellationToken token);
|
||||
string Verified(string signature, string encodedUrl);
|
||||
/// <summary>Whether a remote file's server is suspended or has its media rejected: nothing of it is proxied.</summary>
|
||||
bool Refuses(string url);
|
||||
(string Path, string ContentType) Cached(string url);
|
||||
/// <summary>Downloads a remote file into the cache, once however many ask at the same time.</summary>
|
||||
Task<(ProxyOutcome Outcome, string Path, string ContentType)> Download(string url, CancellationToken token);
|
||||
Task<(string Path, string ContentType)> Fetch(string signature, string encodedUrl, CancellationToken token);
|
||||
Task<HttpResponseMessage> Open(string url, System.Net.Http.Headers.RangeHeaderValue range, CancellationToken token);
|
||||
/// <summary>Deletes what the cache holds of a domain and its subdomains; how many files.</summary>
|
||||
int Purge(string domain);
|
||||
/// <summary>Trims the cache to its size, oldest first.</summary>
|
||||
void Trim();
|
||||
}
|
||||
|
||||
// Remote media, fetched for clients so that they never contact a remote server (CLAUDE.md, media invariants):
|
||||
// - URLs are HMAC-signed, so only what PrivaPub showed is fetched;
|
||||
// - a download is shared by everyone asking for the same URL at once, streamed into a temporary file beside its place
|
||||
// and renamed there (a reader never sees half a file), and never held in memory;
|
||||
// - what is too big to cache is remembered for an hour (it is streamed instead), what failed for five minutes;
|
||||
// - the cache's size is counted as it grows, and trimmed as soon as it passes the cap;
|
||||
// - nothing of a suspended server, or of one whose media are rejected, is proxied, and blocking one purges its files.
|
||||
public class MediaProxy : IMediaProxy
|
||||
{
|
||||
static readonly TimeSpan TooBigFor = TimeSpan.FromHours(1);
|
||||
static readonly TimeSpan FailedFor = TimeSpan.FromMinutes(5);
|
||||
const int Downloads = 8;//remote files fetched at once, however many clients ask
|
||||
|
||||
readonly ILocalActorService _localActors;
|
||||
readonly IFederationHttp _http;
|
||||
readonly IMediaService _media;
|
||||
readonly IOptionsMonitor<MediaOptions> _options;
|
||||
readonly Federation.Moderation.IDomainBlocks _domainBlocks;
|
||||
// Lazy: GetOrAdd may run its factory more than once when asked at the same time, but only one Lazy is ever evaluated
|
||||
readonly ConcurrentDictionary<string, Lazy<Task<(ProxyOutcome, string, string)>>> _inFlight = new();
|
||||
readonly ConcurrentDictionary<string, (ProxyOutcome Outcome, DateTime Until)> _refused = new();
|
||||
readonly SemaphoreSlim _downloads = new(Downloads);
|
||||
readonly object _keyLock = new();
|
||||
byte[] _key;
|
||||
long _bytes = -1;//what the cache holds, counted once from the disk and then as it changes
|
||||
int _trimming;
|
||||
|
||||
public MediaProxy(ILocalActorService localActors, IFederationHttp http, IMediaService media, IOptionsMonitor<MediaOptions> options)
|
||||
public MediaProxy(ILocalActorService localActors, IFederationHttp http, IMediaService media, IOptionsMonitor<MediaOptions> options,
|
||||
Federation.Moderation.IDomainBlocks domainBlocks = default)
|
||||
{
|
||||
_localActors = localActors;
|
||||
_http = http;
|
||||
_media = media;
|
||||
_options = options;
|
||||
_domainBlocks = domainBlocks;
|
||||
}
|
||||
|
||||
byte[] Key => _key ??= LoadKey();
|
||||
byte[] Key
|
||||
{
|
||||
get
|
||||
{
|
||||
if (_key != default)
|
||||
return _key;
|
||||
lock (_keyLock)
|
||||
return _key ??= LoadKey();
|
||||
}
|
||||
}
|
||||
|
||||
public string Wrap(string remoteUrl)
|
||||
{
|
||||
@@ -60,18 +107,186 @@ namespace PrivaPub.Domain.Media
|
||||
return CryptographicOperations.FixedTimeEquals(Encoding.ASCII.GetBytes(signature ?? string.Empty), Encoding.ASCII.GetBytes(Sign(url))) ? url : default;
|
||||
}
|
||||
|
||||
public bool Refuses(string url)
|
||||
{
|
||||
if (_domainBlocks == default || !Uri.TryCreate(url, UriKind.Absolute, out var target))
|
||||
return false;
|
||||
var block = _domainBlocks.Find(target.Host);
|
||||
return block is { Severity: DomainBlockSeverity.Suspend } or { RejectMedia: true };
|
||||
}
|
||||
|
||||
public (string Path, string ContentType) Cached(string url)
|
||||
{
|
||||
var (path, typePath) = CachePaths(url);
|
||||
try
|
||||
{
|
||||
if (!File.Exists(path) || !File.Exists(typePath))
|
||||
return default;
|
||||
File.SetLastWriteTimeUtc(path, DateTime.UtcNow);
|
||||
return (path, File.ReadAllText(typePath));
|
||||
return (path, File.ReadLines(typePath).FirstOrDefault());
|
||||
}
|
||||
catch (IOException)
|
||||
{
|
||||
return default;//trimmed meanwhile
|
||||
}
|
||||
}
|
||||
|
||||
public Task<HttpResponseMessage> Open(string url, System.Net.Http.Headers.RangeHeaderValue range, CancellationToken token) =>
|
||||
_http.OpenMedia(url, range, token);
|
||||
|
||||
public async Task<(string Path, string ContentType)> Fetch(string signature, string encodedUrl, CancellationToken token)
|
||||
{
|
||||
var url = Verified(signature, encodedUrl);
|
||||
if (url == default || Refuses(url))
|
||||
return default;
|
||||
var (outcome, path, contentType) = await Download(url, token);
|
||||
return outcome == ProxyOutcome.Cached ? (path, contentType) : default;
|
||||
}
|
||||
|
||||
public async Task<(ProxyOutcome Outcome, string Path, string ContentType)> Download(string url, CancellationToken token)
|
||||
{
|
||||
if (Cached(url) is { Path: not null } cached)
|
||||
return (ProxyOutcome.Cached, cached.Path, cached.ContentType);
|
||||
if (_refused.TryGetValue(url, out var refused))
|
||||
{
|
||||
if (refused.Until > DateTime.UtcNow)
|
||||
return (refused.Outcome, default, default);
|
||||
_refused.TryRemove(url, out _);
|
||||
}
|
||||
// shared by everyone asking for it now; the download itself isn't cancelled when one of them leaves
|
||||
var shared = _inFlight.GetOrAdd(url, key => new Lazy<Task<(ProxyOutcome, string, string)>>(() => DownloadOnce(key)));
|
||||
try
|
||||
{
|
||||
return await shared.Value.WaitAsync(token);
|
||||
}
|
||||
finally
|
||||
{
|
||||
if (shared.Value.IsCompleted)
|
||||
_inFlight.TryRemove(new KeyValuePair<string, Lazy<Task<(ProxyOutcome, string, string)>>>(url, shared));
|
||||
}
|
||||
}
|
||||
|
||||
async Task<(ProxyOutcome, string, string)> DownloadOnce(string url)
|
||||
{
|
||||
await Task.Yield();
|
||||
if (!await _downloads.WaitAsync(TimeSpan.FromSeconds(30)))
|
||||
return (ProxyOutcome.Failed, default, default);//too busy: the client tries again
|
||||
// cached by a download that ended just before this one began
|
||||
if (Cached(url) is { Path: not null } done)
|
||||
{
|
||||
_downloads.Release();
|
||||
return (ProxyOutcome.Cached, done.Path, done.ContentType);
|
||||
}
|
||||
var (path, typePath) = CachePaths(url);
|
||||
var part = $"{path}.{Guid.NewGuid():N}.part";
|
||||
try
|
||||
{
|
||||
Directory.CreateDirectory(System.IO.Path.GetDirectoryName(path)!);
|
||||
string contentType, refusal;
|
||||
await using (var file = new FileStream(part, FileMode.CreateNew, FileAccess.Write, FileShare.None, 64 * 1024, useAsync: true))
|
||||
(contentType, refusal) = await _http.DownloadMedia(url, _options.CurrentValue.MaxProxiedBytes, file, CancellationToken.None);
|
||||
if (contentType == default)
|
||||
{
|
||||
var outcome = refusal == "too-large" ? ProxyOutcome.TooBig : ProxyOutcome.Failed;
|
||||
_refused[url] = (outcome, DateTime.UtcNow + (outcome == ProxyOutcome.TooBig ? TooBigFor : FailedFor));
|
||||
return (outcome, default, default);
|
||||
}
|
||||
await File.WriteAllTextAsync(typePath, contentType + "\n" + new Uri(url).Host);
|
||||
var size = new FileInfo(part).Length;
|
||||
File.Move(part, path, overwrite: true);
|
||||
Grew(size);
|
||||
return (ProxyOutcome.Cached, path, contentType);
|
||||
}
|
||||
finally
|
||||
{
|
||||
_downloads.Release();
|
||||
if (File.Exists(part))
|
||||
File.Delete(part);
|
||||
}
|
||||
}
|
||||
|
||||
void Grew(long bytes)
|
||||
{
|
||||
if (Interlocked.Read(ref _bytes) < 0)
|
||||
Interlocked.CompareExchange(ref _bytes, Measure(), -1);
|
||||
if (Interlocked.Add(ref _bytes, bytes) > _options.CurrentValue.ProxyCacheBytes)
|
||||
_ = Task.Run(Trim);
|
||||
}
|
||||
|
||||
public void Trim()
|
||||
{
|
||||
if (Interlocked.Exchange(ref _trimming, 1) == 1)
|
||||
return;
|
||||
try
|
||||
{
|
||||
Interlocked.Exchange(ref _bytes, TrimDirectory(_media.ProxyRoot, _options.CurrentValue.ProxyCacheBytes));
|
||||
}
|
||||
finally
|
||||
{
|
||||
Interlocked.Exchange(ref _trimming, 0);
|
||||
}
|
||||
}
|
||||
|
||||
public int Purge(string domain)
|
||||
{
|
||||
var directory = new DirectoryInfo(_media.ProxyRoot);
|
||||
if (string.IsNullOrEmpty(domain) || !directory.Exists)
|
||||
return 0;
|
||||
var purged = 0;
|
||||
foreach (var type in directory.EnumerateFiles("*.type", SearchOption.AllDirectories))
|
||||
{
|
||||
try
|
||||
{
|
||||
var host = File.ReadLines(type.FullName).Skip(1).FirstOrDefault();
|
||||
if (host == default || !(host.Equals(domain, StringComparison.OrdinalIgnoreCase) || host.EndsWith("." + domain, StringComparison.OrdinalIgnoreCase)))
|
||||
continue;
|
||||
var file = new FileInfo(type.FullName[..^".type".Length]);
|
||||
if (file.Exists)
|
||||
file.Delete();
|
||||
type.Delete();
|
||||
purged++;
|
||||
}
|
||||
catch (IOException)
|
||||
{
|
||||
}
|
||||
}
|
||||
Interlocked.Exchange(ref _bytes, -1);
|
||||
return purged;
|
||||
}
|
||||
|
||||
long Measure()
|
||||
{
|
||||
var directory = new DirectoryInfo(_media.ProxyRoot);
|
||||
return directory.Exists ? directory.EnumerateFiles("*", SearchOption.AllDirectories).Where(f => f.Extension is not (".type" or ".part")).Sum(f => f.Length) : 0;
|
||||
}
|
||||
|
||||
// deletes the oldest files until the cache is within its cap; what it holds afterwards
|
||||
public static long TrimDirectory(string root, long cap)
|
||||
{
|
||||
var directory = new DirectoryInfo(root);
|
||||
if (!directory.Exists)
|
||||
return 0;
|
||||
var files = directory.EnumerateFiles("*", SearchOption.AllDirectories).Where(f => f.Extension is not (".type" or ".part")).OrderBy(f => f.LastWriteTimeUtc).ToList();
|
||||
var total = files.Sum(f => f.Length);
|
||||
foreach (var file in files)
|
||||
{
|
||||
if (total <= cap)
|
||||
break;
|
||||
try
|
||||
{
|
||||
total -= file.Length;
|
||||
file.Delete();
|
||||
var type = new FileInfo(file.FullName + ".type");
|
||||
if (type.Exists)
|
||||
type.Delete();
|
||||
}
|
||||
catch (IOException)
|
||||
{
|
||||
}
|
||||
}
|
||||
return total;
|
||||
}
|
||||
|
||||
(string Path, string TypePath) CachePaths(string url)
|
||||
{
|
||||
var name = Convert.ToHexStringLower(SHA256.HashData(Encoding.UTF8.GetBytes(url)));
|
||||
@@ -79,35 +294,16 @@ namespace PrivaPub.Domain.Media
|
||||
return (path, path + ".type");
|
||||
}
|
||||
|
||||
public async Task<(string Path, string ContentType)> Fetch(string signature, string encodedUrl, CancellationToken token)
|
||||
{
|
||||
var url = Verified(signature, encodedUrl);
|
||||
if (url == default)
|
||||
return default;
|
||||
if (Cached(url) is { Path: not null } cached)
|
||||
return cached;
|
||||
var (path, typePath) = CachePaths(url);
|
||||
var directory = System.IO.Path.GetDirectoryName(path);
|
||||
|
||||
var (bytes, contentType) = await _http.GetMedia(url, _options.CurrentValue.MaxProxiedBytes, token);
|
||||
if (bytes == default)
|
||||
return default;
|
||||
Directory.CreateDirectory(directory);
|
||||
await File.WriteAllBytesAsync(path, bytes, token);
|
||||
await File.WriteAllTextAsync(typePath, contentType, token);
|
||||
return (path, contentType);
|
||||
}
|
||||
|
||||
string Sign(string url) => Base64Url(HMACSHA256.HashData(Key, Encoding.UTF8.GetBytes(url))[..16]);
|
||||
|
||||
// the oldest key, so that two first uses at once agree on one
|
||||
static byte[] LoadKey()
|
||||
{
|
||||
var secret = DB.Default.Find<MediaSecret>().ExecuteFirstAsync().GetAwaiter().GetResult();
|
||||
var secret = DB.Default.Find<MediaSecret>().Sort(m => m.ID, Order.Ascending).ExecuteFirstAsync().GetAwaiter().GetResult();
|
||||
if (secret == default)
|
||||
{
|
||||
secret = new MediaSecret { Key = Convert.ToBase64String(RandomNumberGenerator.GetBytes(32)) };
|
||||
DB.Default.SaveAsync(secret).GetAwaiter().GetResult();
|
||||
secret = DB.Default.Find<MediaSecret>().ExecuteFirstAsync().GetAwaiter().GetResult();
|
||||
DB.Default.SaveAsync(new MediaSecret { Key = Convert.ToBase64String(RandomNumberGenerator.GetBytes(32)) }).GetAwaiter().GetResult();
|
||||
secret = DB.Default.Find<MediaSecret>().Sort(m => m.ID, Order.Ascending).ExecuteFirstAsync().GetAwaiter().GetResult();
|
||||
}
|
||||
return Convert.FromBase64String(secret.Key);
|
||||
}
|
||||
@@ -123,8 +319,11 @@ namespace PrivaPub.Domain.Media
|
||||
|
||||
public class MediaJanitor : BackgroundService
|
||||
{
|
||||
static readonly TimeSpan FirstPass = TimeSpan.FromMinutes(5);
|
||||
static readonly TimeSpan Interval = TimeSpan.FromHours(1);
|
||||
static readonly TimeSpan UnattachedLifetime = TimeSpan.FromDays(1);
|
||||
// a trashed file waits this long before it is deleted, already out of what /media/files serves
|
||||
public static readonly TimeSpan TrashGrace = TimeSpan.FromDays(1);
|
||||
|
||||
readonly IMediaService _media;
|
||||
readonly IOptionsMonitor<MediaOptions> _options;
|
||||
@@ -139,11 +338,13 @@ namespace PrivaPub.Domain.Media
|
||||
|
||||
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
|
||||
{
|
||||
var wait = FirstPass;
|
||||
while (!stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
try
|
||||
{
|
||||
await Task.Delay(Interval, stoppingToken);
|
||||
await Task.Delay(wait, stoppingToken);
|
||||
wait = Interval;
|
||||
await Sweep(stoppingToken);
|
||||
}
|
||||
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
|
||||
@@ -157,32 +358,38 @@ namespace PrivaPub.Domain.Media
|
||||
}
|
||||
}
|
||||
|
||||
//one pass: uploads left unattached for a day go (unless a scheduled post waits for them), then the proxy cache is trimmed to its size, oldest first
|
||||
// one pass:
|
||||
// - uploads never posted for a day (and not waiting for a scheduled post, nor a profile picture) go to the trash;
|
||||
// - trashed files still served (a crash between the mark and the move) are moved out;
|
||||
// - trashed files past their grace are deleted, with their rows, unless a backup or a restore is running (one may be
|
||||
// reading the trash);
|
||||
// - the proxy cache is trimmed to its size, oldest first
|
||||
public async Task Sweep(CancellationToken token)
|
||||
{
|
||||
var cutoff = DateTime.UtcNow - UnattachedLifetime;
|
||||
foreach (var stale in await DB.Default.Find<MediaAttachment>().Match(m => m.PostId == null && m.ScheduledStatusId == null && m.CreatedAt < cutoff).Limit(500).ExecuteAsync(token))
|
||||
await _media.Delete(stale);
|
||||
var unattached = await _media.Trash(m => m.PostId == null && m.ScheduledStatusId == null && m.ProfileOfAvatarId == null && m.CreatedAt < cutoff,
|
||||
"never posted", token);
|
||||
|
||||
var trashed = await DB.Default.Find<MediaAttachment>().Match(m => m.TrashedAt != null).Limit(2000).ExecuteAsync(token);
|
||||
var purgeBefore = DateTime.UtcNow - TrashGrace;
|
||||
var maintenance = await Infrastructure.Backup.MaintenanceLock.Current(token) != default;
|
||||
var purged = 0;
|
||||
foreach (var row in trashed)
|
||||
{
|
||||
if (row.TrashedAt < purgeBefore && !maintenance)
|
||||
{
|
||||
await _media.Purge(row, token);
|
||||
purged++;
|
||||
}
|
||||
else
|
||||
_media.Hide(row);
|
||||
}
|
||||
if (unattached > 0 || purged > 0)
|
||||
_logger.LogInformation("{Service}: {Unattached} uploads never posted trashed, {Purged} trashed files deleted",
|
||||
nameof(MediaJanitor), unattached, purged);
|
||||
TrimProxyCache();
|
||||
}
|
||||
|
||||
void TrimProxyCache()
|
||||
{
|
||||
var directory = new DirectoryInfo(_media.ProxyRoot);
|
||||
if (!directory.Exists)
|
||||
return;
|
||||
var files = directory.EnumerateFiles("*", SearchOption.AllDirectories).Where(f => f.Extension != ".type").OrderBy(f => f.LastWriteTimeUtc).ToList();
|
||||
var total = files.Sum(f => f.Length);
|
||||
foreach (var file in files)
|
||||
{
|
||||
if (total <= _options.CurrentValue.ProxyCacheBytes)
|
||||
break;
|
||||
total -= file.Length;
|
||||
file.Delete();
|
||||
var type = new FileInfo(file.FullName + ".type");
|
||||
if (type.Exists)
|
||||
type.Delete();
|
||||
}
|
||||
}
|
||||
void TrimProxyCache() => MediaProxy.TrimDirectory(_media.ProxyRoot, _options.CurrentValue.ProxyCacheBytes);
|
||||
}
|
||||
}
|
||||
@@ -22,19 +22,46 @@ namespace PrivaPub.Domain.Media
|
||||
|
||||
public sealed record ProcessedImage(byte[] Bytes, string Extension, string ContentType, int Width, int Height, byte[] Preview, string Blurhash);
|
||||
|
||||
// what libvips makes of an upload's header alone, before anything is decoded
|
||||
public sealed record ImageHeader(string Loader, int Width, int PageHeight, int Pages)
|
||||
{
|
||||
public bool Animated => Pages > 1 && Loader?.StartsWith("gifload", StringComparison.Ordinal) == true;
|
||||
public long Pixels => (long)Width * PageHeight * (Animated ? Pages : 1);
|
||||
}
|
||||
|
||||
public interface IMediaService
|
||||
{
|
||||
string Root { get; }
|
||||
string ProxyRoot { get; }
|
||||
string Url(string relativePath);
|
||||
Task<MediaOutcome> Upload(LocalActor owner, IFormFile file, string description, string focus, CancellationToken token);
|
||||
Task<string> ProfileImage(IFormFile file, int width, int height, CancellationToken token);
|
||||
Task Delete(MediaAttachment attachment);
|
||||
/// <summary>Where trashed files wait for the janitor: the sibling of Root, on the same disk (a move is a rename) and
|
||||
/// outside what /media/files serves. (Not a dot-prefixed folder inside Root: the file provider only hides a file whose
|
||||
/// own name starts with a dot.)</summary>
|
||||
string TrashRoot { get; }
|
||||
/// <summary>Audio and video waiting for their processing: beside the media root, never served.</summary>
|
||||
string IncomingRoot { get; }
|
||||
/// <summary>An upload; with <paramref name="later"/>, audio and video are only stored, "pending" for ProcessMedia.</summary>
|
||||
Task<MediaOutcome> Upload(LocalActor owner, IFormFile file, string description, string focus, bool later, CancellationToken token);
|
||||
/// <summary>Processes a pending upload's audio or video; false when it can't be (the row then says why).</summary>
|
||||
Task<bool> ProcessPending(MediaAttachment pending, CancellationToken token);
|
||||
/// <summary>A persona's new avatar or header ("avatar" or "header"), cropped to its size, as a row of its own.</summary>
|
||||
Task<MediaOutcome> ProfileImage(string avatarId, string kind, IFormFile file, int width, int height, CancellationToken token);
|
||||
/// <summary>Trashes the media <paramref name="which"/> matches (and that isn't trashed yet): each row is marked in one
|
||||
/// conditional update, so a row attached meanwhile is left alone, and its files stop being served at once.</summary>
|
||||
Task<long> Trash(System.Linq.Expressions.Expression<Func<MediaAttachment, bool>> which, string reason, CancellationToken token);
|
||||
/// <summary>Moves a trashed row's files out of what is served, if they are still there (a crash between the mark and the move).</summary>
|
||||
void Hide(MediaAttachment trashed);
|
||||
/// <summary>Deletes a trashed row's files and the row.</summary>
|
||||
Task Purge(MediaAttachment trashed, CancellationToken token);
|
||||
}
|
||||
|
||||
public class MediaService : IMediaService
|
||||
{
|
||||
static readonly string[] ImageTypes = { "image/jpeg", "image/png", "image/gif", "image/webp", "image/heic", "image/heif", "image/avif" };
|
||||
// HEIC and HEIF are not among them: the libvips bundled here decodes AVIF but has no HEVC decoder
|
||||
static readonly string[] ImageTypes = { "image/jpeg", "image/png", "image/gif", "image/webp", "image/avif" };
|
||||
// the only loaders an upload ever reaches: everything else libvips could read (SVG, PDF, TIFF, ImageMagick, ...) is
|
||||
// blocked, whatever type the upload claims
|
||||
static readonly string[] Loaders = { "VipsForeignLoadJpeg", "VipsForeignLoadPng", "VipsForeignLoadNsgif", "VipsForeignLoadWebp", "VipsForeignLoadHeif" };
|
||||
static readonly Dictionary<string, string> AvTypes = new()
|
||||
{
|
||||
["video/mp4"] = "mp4", ["video/quicktime"] = "mp4", ["video/webm"] = "webm",
|
||||
@@ -42,11 +69,27 @@ namespace PrivaPub.Domain.Media
|
||||
["audio/mp4"] = "m4a", ["audio/x-m4a"] = "m4a", ["audio/flac"] = "flac", ["audio/webm"] = "webm"
|
||||
};
|
||||
|
||||
// what /media/files says each file is: ASP.NET's map, which has no entry for FLAC
|
||||
public static Microsoft.AspNetCore.StaticFiles.FileExtensionContentTypeProvider ContentTypes { get; } = new()
|
||||
{
|
||||
Mappings = { [".flac"] = "audio/flac" }
|
||||
};
|
||||
|
||||
// what an upload may be, as the instance API advertises it
|
||||
public static IReadOnlyList<string> SupportedTypes => ImageTypes.Concat(AvTypes.Keys).ToList();
|
||||
|
||||
static MediaService()
|
||||
{
|
||||
NetVips.NetVips.BlockUntrusted = true;
|
||||
Operation.Block("VipsForeignLoad", true);
|
||||
foreach (var loader in Loaders)
|
||||
Operation.Block(loader, false);
|
||||
Cache.Max = 0;//an upload is decoded once: nothing to keep between operations
|
||||
}
|
||||
|
||||
readonly IOptionsMonitor<MediaOptions> _options;
|
||||
readonly ILocalActorService _localActors;
|
||||
readonly SemaphoreSlim _processing;
|
||||
readonly IWebHostEnvironment _environment;
|
||||
readonly ILogger<MediaService> _logger;
|
||||
|
||||
@@ -54,6 +97,7 @@ namespace PrivaPub.Domain.Media
|
||||
{
|
||||
_options = options;
|
||||
_localActors = localActors;
|
||||
_processing = new SemaphoreSlim(Math.Max(1, options.CurrentValue.Concurrency));
|
||||
_environment = environment;
|
||||
_logger = logger;
|
||||
}
|
||||
@@ -62,39 +106,50 @@ namespace PrivaPub.Domain.Media
|
||||
|
||||
public string ProxyRoot => Root.TrimEnd(Path.DirectorySeparatorChar) + "-proxy";
|
||||
|
||||
public string TrashRoot => Root.TrimEnd(Path.DirectorySeparatorChar) + "-trash";
|
||||
|
||||
public string IncomingRoot => Root.TrimEnd(Path.DirectorySeparatorChar) + "-incoming";
|
||||
|
||||
string Incoming(string id) => Path.Combine(IncomingRoot, id + ".in");
|
||||
|
||||
public string Url(string relativePath) => relativePath == default ? default : $"{_localActors.BaseAddress}/media/files/{relativePath.Replace('\\', '/')}";
|
||||
|
||||
public async Task<MediaOutcome> Upload(LocalActor owner, IFormFile file, string description, string focus, CancellationToken token)
|
||||
public async Task<MediaOutcome> Upload(LocalActor owner, IFormFile file, string description, string focus, bool later, CancellationToken token)
|
||||
{
|
||||
if (file == default || file.Length == 0)
|
||||
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: File can't be blank");
|
||||
var options = _options.CurrentValue;
|
||||
var contentType = file.ContentType?.Split(';')[0].Trim().ToLowerInvariant();
|
||||
if (await OverQuota(owner.Id, file.Length, token) is { } full)
|
||||
return full;
|
||||
|
||||
var attachment = new MediaAttachment
|
||||
{
|
||||
OwnerAvatarId = owner.Id,
|
||||
Description = Clean(description, 1500),
|
||||
Focus = Focus(focus)
|
||||
Focus = FocalPoint.Parse(focus)
|
||||
};
|
||||
if (ImageTypes.Contains(contentType))
|
||||
{
|
||||
if (file.Length > options.MaxImageBytes)
|
||||
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: File is too big");
|
||||
await using var stream = file.OpenReadStream();
|
||||
using var buffer = new MemoryStream();
|
||||
await stream.CopyToAsync(buffer, token);
|
||||
ProcessedImage processed;
|
||||
var bytes = await Read(file, token);
|
||||
if (Refusal(bytes, options) is { } refused)
|
||||
return refused;
|
||||
await _processing.WaitAsync(token);
|
||||
try
|
||||
{
|
||||
processed = ProcessImage(buffer.ToArray(), options.MaxImageSide, options.PreviewSide, contentType == "image/gif");
|
||||
}
|
||||
catch (VipsException ex)
|
||||
var header = Inspect(bytes);
|
||||
if (header.Animated)
|
||||
{
|
||||
_logger.LogInformation("Refused an image upload: {Error}", ex.Message);
|
||||
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: The file is not a readable image");
|
||||
var animated = await ProcessAnimation(bytes, header, attachment, token);
|
||||
if (!animated.Ok)
|
||||
return animated;
|
||||
}
|
||||
attachment.Kind = contentType == "image/gif" ? "gifv" : "image";
|
||||
else
|
||||
{
|
||||
var processed = ProcessImage(bytes, options.MaxImageSide, options.PreviewSide);
|
||||
attachment.Kind = "image";
|
||||
attachment.ContentType = processed.ContentType;
|
||||
attachment.FilePath = await Save(processed.Bytes, processed.Extension, token);
|
||||
attachment.PreviewPath = await Save(processed.Preview, "jpg", token);
|
||||
@@ -103,14 +158,37 @@ namespace PrivaPub.Domain.Media
|
||||
attachment.Blurhash = processed.Blurhash;
|
||||
attachment.Size = processed.Bytes.Length;
|
||||
}
|
||||
else if (contentType != default && AvTypes.TryGetValue(contentType, out var extension))
|
||||
}
|
||||
catch (VipsException ex)
|
||||
{
|
||||
_logger.LogInformation("Refused an image upload: {Error}", ex.Message);
|
||||
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: The file is not a readable image");
|
||||
}
|
||||
finally
|
||||
{
|
||||
_processing.Release();
|
||||
}
|
||||
}
|
||||
else if (contentType != default && AvTypes.ContainsKey(contentType))
|
||||
{
|
||||
if (file.Length > options.MaxVideoBytes)
|
||||
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: File is too big");
|
||||
var outcome = await ProcessAv(file, extension, contentType, attachment, token);
|
||||
// stored as it came, outside what is served, then processed: by a job when asked later (v2), here otherwise (v1)
|
||||
attachment.ID = (string)attachment.GenerateNewID();
|
||||
attachment.Kind = contentType.StartsWith("video/", StringComparison.Ordinal) ? "video" : "audio";
|
||||
attachment.ContentType = contentType;
|
||||
Directory.CreateDirectory(IncomingRoot);
|
||||
await using (var stored = File.Create(Incoming(attachment.ID)))
|
||||
await file.CopyToAsync(stored, token);
|
||||
if (later)
|
||||
attachment.ProcessingState = "pending";
|
||||
else
|
||||
{
|
||||
var outcome = await ProcessIncoming(attachment, token);
|
||||
if (!outcome.Ok)
|
||||
return outcome;
|
||||
}
|
||||
}
|
||||
else
|
||||
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: File type is not supported");
|
||||
|
||||
@@ -118,68 +196,205 @@ namespace PrivaPub.Domain.Media
|
||||
return new MediaOutcome(attachment);
|
||||
}
|
||||
|
||||
public async Task<string> ProfileImage(IFormFile file, int width, int height, CancellationToken token)
|
||||
public async Task<MediaOutcome> ProfileImage(string avatarId, string kind, IFormFile file, int width, int height, CancellationToken token)
|
||||
{
|
||||
if (file == default || file.Length == 0 || file.Length > _options.CurrentValue.MaxImageBytes)
|
||||
return default;
|
||||
await using var stream = file.OpenReadStream();
|
||||
using var buffer = new MemoryStream();
|
||||
await stream.CopyToAsync(buffer, token);
|
||||
var contentType = file?.ContentType?.Split(';')[0].Trim().ToLowerInvariant();
|
||||
if (file == default || file.Length == 0)
|
||||
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: File can't be blank");
|
||||
if (!ImageTypes.Contains(contentType))
|
||||
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: File type is not supported");
|
||||
if (file.Length > _options.CurrentValue.MaxImageBytes)
|
||||
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: File is too big");
|
||||
if (await OverQuota(avatarId, file.Length, token) is { } full)
|
||||
return full;
|
||||
var input = await Read(file, token);
|
||||
if (Refusal(input, _options.CurrentValue) is { } refused)
|
||||
return refused;
|
||||
byte[] bytes;
|
||||
int outWidth, outHeight;
|
||||
await _processing.WaitAsync(token);
|
||||
try
|
||||
{
|
||||
using var image = Image.ThumbnailBuffer(buffer.ToArray(), width, height: height, crop: Enums.Interesting.Centre, size: Enums.Size.Down);
|
||||
using var image = Image.ThumbnailBuffer(input, width, height: height, crop: Enums.Interesting.Centre, size: Enums.Size.Down);
|
||||
using var flat = Flatten(image);
|
||||
return Url(await Save(flat.WriteToBuffer(".jpg[Q=85,keep=none]"), "jpg", token));
|
||||
bytes = flat.WriteToBuffer(".jpg[Q=85,keep=none]");
|
||||
(outWidth, outHeight) = (flat.Width, flat.Height);
|
||||
}
|
||||
catch (VipsException ex)
|
||||
{
|
||||
_logger.LogInformation("Refused a profile picture: {Error}", ex.Message);
|
||||
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: The file is not a readable image");
|
||||
}
|
||||
finally
|
||||
{
|
||||
_processing.Release();
|
||||
}
|
||||
var attachment = new MediaAttachment
|
||||
{
|
||||
OwnerAvatarId = avatarId,
|
||||
ProfileOfAvatarId = avatarId,
|
||||
Kind = kind,
|
||||
ContentType = "image/jpeg",
|
||||
FilePath = await Save(bytes, "jpg", token),
|
||||
Size = bytes.Length,
|
||||
Width = outWidth,
|
||||
Height = outHeight,
|
||||
AttachedAt = DateTime.UtcNow
|
||||
};
|
||||
await DB.Default.SaveAsync(attachment, token);
|
||||
return new MediaOutcome(attachment);
|
||||
}
|
||||
|
||||
// a login over its quota (Media:QuotaBytesPerRoot; none when 0) uploads nothing more
|
||||
async Task<MediaOutcome> OverQuota(string avatarId, long incoming, CancellationToken token)
|
||||
{
|
||||
var quota = _options.CurrentValue.QuotaBytesPerRoot;
|
||||
if (quota <= 0)
|
||||
return default;
|
||||
var rootId = (await DB.Default.Find<Models.User.RootToAvatar>().Match(r => r.AvatarId == avatarId).ExecuteFirstAsync(token))?.RootId;
|
||||
if (rootId == default)
|
||||
return default;
|
||||
var (bytes, _) = await Privacy.Counted.MediaOfRoot(rootId, token);
|
||||
return bytes + incoming > quota ? MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: Your storage is full") : default;
|
||||
}
|
||||
|
||||
public async Task<long> Trash(System.Linq.Expressions.Expression<Func<MediaAttachment, bool>> which, string reason, CancellationToken token)
|
||||
{
|
||||
var trashed = 0L;
|
||||
foreach (var candidate in await DB.Default.Find<MediaAttachment>().Match(which).Match(m => m.TrashedAt == null).ExecuteAsync(token))
|
||||
{
|
||||
var marked = await DB.Default.Update<MediaAttachment>()
|
||||
.Match(which).Match(m => m.ID == candidate.ID && m.TrashedAt == null)
|
||||
.Modify(m => m.TrashedAt, DateTime.UtcNow)
|
||||
.Modify(m => m.TrashReason, reason)
|
||||
.ExecuteAsync(token);
|
||||
if (marked.ModifiedCount == 0)
|
||||
continue;
|
||||
Hide(candidate);
|
||||
trashed++;
|
||||
}
|
||||
return trashed;
|
||||
}
|
||||
|
||||
public void Hide(MediaAttachment trashed)
|
||||
{
|
||||
foreach (var relative in new[] { trashed.FilePath, trashed.PreviewPath }.Where(p => !string.IsNullOrEmpty(p)))
|
||||
{
|
||||
var served = Path.Combine(Root, relative);
|
||||
if (!File.Exists(served))
|
||||
continue;
|
||||
var hidden = Path.Combine(TrashRoot, relative);
|
||||
Directory.CreateDirectory(Path.GetDirectoryName(hidden)!);
|
||||
File.Move(served, hidden, overwrite: true);
|
||||
}
|
||||
}
|
||||
|
||||
public async Task Purge(MediaAttachment trashed, CancellationToken token)
|
||||
{
|
||||
foreach (var relative in new[] { trashed.FilePath, trashed.PreviewPath }.Where(p => !string.IsNullOrEmpty(p)))
|
||||
foreach (var full in new[] { Path.Combine(TrashRoot, relative), Path.Combine(Root, relative) })
|
||||
if (File.Exists(full))
|
||||
File.Delete(full);
|
||||
if (File.Exists(Incoming(trashed.ID)))
|
||||
File.Delete(Incoming(trashed.ID));
|
||||
await DB.Default.DeleteAsync<MediaAttachment>(m => m.ID == trashed.ID && m.TrashedAt != null);
|
||||
}
|
||||
|
||||
// reads only the header: which loader takes the bytes and how big they would decode (throws VipsException when none)
|
||||
public static ImageHeader Inspect(byte[] input)
|
||||
{
|
||||
using var image = Image.NewFromBuffer(input, access: Enums.Access.Sequential);
|
||||
var pages = image.Contains("n-pages") ? Math.Max(1, (int)image.Get("n-pages")) : 1;
|
||||
var pageHeight = image.Contains("page-height") ? (int)image.Get("page-height") : image.Height;
|
||||
return new ImageHeader(image.Contains("vips-loader") ? (string)image.Get("vips-loader") : default, image.Width, pageHeight, pages);
|
||||
}
|
||||
|
||||
// an upload refused before anything is decoded: no loader takes it, or it would decode to too many pixels or frames
|
||||
static MediaOutcome Refusal(byte[] input, MediaOptions options)
|
||||
{
|
||||
ImageHeader header;
|
||||
try
|
||||
{
|
||||
header = Inspect(input);
|
||||
}
|
||||
catch (VipsException)
|
||||
{
|
||||
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: The file is not a readable image");
|
||||
}
|
||||
if (header.Pixels > options.MaxPixels || header.Pages > options.MaxFrames)
|
||||
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: The image is too large");
|
||||
return default;
|
||||
}
|
||||
}
|
||||
|
||||
public Task Delete(MediaAttachment attachment)
|
||||
// a still image: shrunk on load to the largest side allowed, turned by its orientation, its colours brought into sRGB
|
||||
// (thumbnail does all three), then written without its metadata, with a preview and a blurhash
|
||||
public static ProcessedImage ProcessImage(byte[] input, int maxSide, int previewSide)
|
||||
{
|
||||
foreach (var path in new[] { attachment.FilePath, attachment.PreviewPath }.Where(p => p != default))
|
||||
{
|
||||
var full = Path.Combine(Root, path);
|
||||
if (File.Exists(full))
|
||||
File.Delete(full);
|
||||
}
|
||||
return DB.Default.DeleteAsync<MediaAttachment>(attachment.ID);
|
||||
}
|
||||
|
||||
public static ProcessedImage ProcessImage(byte[] input, int maxSide, int previewSide, bool animated)
|
||||
{
|
||||
using var loaded = animated ? Image.NewFromBuffer(input, kwargs: new VOption { { "n", -1 } }) : Image.NewFromBuffer(input);
|
||||
using var rotated = animated ? loaded.Copy() : loaded.Autorot();
|
||||
var pageHeight = animated && rotated.Contains("page-height") ? (int)rotated.Get("page-height") : rotated.Height;
|
||||
var longest = Math.Max(rotated.Width, pageHeight);
|
||||
|
||||
byte[] bytes;
|
||||
string extension, contentType;
|
||||
int width, height;
|
||||
if (animated)
|
||||
{
|
||||
bytes = rotated.WriteToBuffer(".gif");
|
||||
(extension, contentType, width, height) = ("gif", "image/gif", rotated.Width, pageHeight);
|
||||
}
|
||||
else
|
||||
{
|
||||
using var resized = longest > maxSide ? rotated.ThumbnailImage(maxSide, height: maxSide, size: Enums.Size.Down) : rotated.Copy();
|
||||
// rendered into memory once (it is read three times below, and a shrunk-on-load image can be read only in order)
|
||||
using var shrunk = Image.ThumbnailBuffer(input, maxSide, height: maxSide, size: Enums.Size.Down);
|
||||
using var resized = shrunk.CopyMemory();
|
||||
var keepsAlpha = resized.HasAlpha();
|
||||
bytes = keepsAlpha ? resized.WriteToBuffer(".png[keep=none]") : resized.WriteToBuffer(".jpg[Q=88,keep=none]");
|
||||
(extension, contentType, width, height) = keepsAlpha ? ("png", "image/png", resized.Width, resized.Height) : ("jpg", "image/jpeg", resized.Width, resized.Height);
|
||||
}
|
||||
var bytes = keepsAlpha ? resized.WriteToBuffer(".png[keep=none]") : resized.WriteToBuffer(".jpg[Q=88,keep=none]");
|
||||
var (extension, contentType) = keepsAlpha ? ("png", "image/png") : ("jpg", "image/jpeg");
|
||||
|
||||
using var firstFrame = animated ? rotated.Crop(0, 0, rotated.Width, pageHeight) : rotated.Copy();
|
||||
using var preview = firstFrame.ThumbnailImage(previewSide, height: previewSide, size: Enums.Size.Down);
|
||||
using var preview = resized.ThumbnailImage(previewSide, height: previewSide, size: Enums.Size.Down);
|
||||
using var previewFlat = Flatten(preview);
|
||||
var previewBytes = previewFlat.WriteToBuffer(".jpg[Q=80,keep=none]");
|
||||
|
||||
using var tiny = firstFrame.ThumbnailImage(32, height: 32);
|
||||
using var tiny = resized.ThumbnailImage(32, height: 32);
|
||||
using var tinyFlat = Flatten(tiny);
|
||||
var pixels = tinyFlat.WriteToMemory();
|
||||
return new ProcessedImage(bytes, extension, contentType, width, height, previewBytes, Blurhash.Encode(pixels, tinyFlat.Width, tinyFlat.Height));
|
||||
return new ProcessedImage(bytes, extension, contentType, resized.Width, resized.Height, previewBytes, Blurhash.Encode(pixels, tinyFlat.Width, tinyFlat.Height));
|
||||
}
|
||||
|
||||
// an animated GIF becomes what Mastodon makes of one: a silent, looping H.264 mp4 (a gifv) at most MaxGifSide wide,
|
||||
// its first frame the poster
|
||||
async Task<MediaOutcome> ProcessAnimation(byte[] input, ImageHeader header, MediaAttachment attachment, CancellationToken token)
|
||||
{
|
||||
var options = _options.CurrentValue;
|
||||
var temp = Path.Combine(Path.GetTempPath(), $"privapub-{Guid.NewGuid():N}");
|
||||
var gif = temp + ".gif";
|
||||
var mp4 = temp + ".mp4";
|
||||
try
|
||||
{
|
||||
await File.WriteAllBytesAsync(gif, input, token);
|
||||
var width = Math.Min(header.Width, options.MaxGifSide) / 2 * 2;
|
||||
var height = Math.Max(2, (int)Math.Round(header.PageHeight * (double)width / header.Width / 2) * 2);
|
||||
await FFMpegArguments.FromFileInput(gif, true, o => o.ForceFormat("gif").WithCustomArgument("-protocol_whitelist file"))
|
||||
.OutputToFile(mp4, true, o => o.WithCustomArgument(
|
||||
$"-an -map_metadata -1 -map_chapters -1 -c:v libx264 -preset veryfast -crf 23 -pix_fmt yuv420p -movflags +faststart -vf scale={width}:{height}"))
|
||||
.CancellableThrough(token)
|
||||
.ProcessAsynchronously();
|
||||
var poster = ProcessImage(input, options.PreviewSide, options.PreviewSide);
|
||||
attachment.Kind = "gifv";
|
||||
attachment.ContentType = "video/mp4";
|
||||
attachment.Width = width;
|
||||
attachment.Height = height;
|
||||
attachment.Size = new FileInfo(mp4).Length;
|
||||
attachment.Blurhash = poster.Blurhash;
|
||||
attachment.PreviewPath = await Save(poster.Preview, "jpg", token);
|
||||
attachment.FilePath = SaveFile(mp4, "mp4");
|
||||
return new MediaOutcome(attachment);
|
||||
}
|
||||
catch (Exception ex) when (ex is not OperationCanceledException)
|
||||
{
|
||||
_logger.LogInformation(ex, "Refused a GIF upload");
|
||||
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: The file could not be processed");
|
||||
}
|
||||
finally
|
||||
{
|
||||
foreach (var path in new[] { gif, mp4 })
|
||||
if (File.Exists(path))
|
||||
File.Delete(path);
|
||||
}
|
||||
}
|
||||
|
||||
static async Task<byte[]> Read(IFormFile file, CancellationToken token)
|
||||
{
|
||||
await using var stream = file.OpenReadStream();
|
||||
using var buffer = new MemoryStream((int)Math.Min(file.Length, int.MaxValue));
|
||||
await stream.CopyToAsync(buffer, token);
|
||||
return buffer.ToArray();
|
||||
}
|
||||
|
||||
static Image Flatten(Image image)
|
||||
@@ -193,37 +408,102 @@ namespace PrivaPub.Domain.Media
|
||||
return bands.Format == Enums.BandFormat.Uchar ? bands.Copy() : bands.Cast(Enums.BandFormat.Uchar);
|
||||
}
|
||||
|
||||
async Task<MediaOutcome> ProcessAv(IFormFile file, string extension, string contentType, MediaAttachment attachment, CancellationToken token)
|
||||
public async Task<bool> ProcessPending(MediaAttachment pending, CancellationToken token)
|
||||
{
|
||||
var outcome = await ProcessIncoming(pending, token);
|
||||
if (outcome.Ok)
|
||||
{
|
||||
await DB.Default.Update<MediaAttachment>().Match(m => m.ID == pending.ID && m.TrashedAt == null)
|
||||
.Modify(m => m.Kind, pending.Kind)
|
||||
.Modify(m => m.ContentType, pending.ContentType)
|
||||
.Modify(m => m.FilePath, pending.FilePath)
|
||||
.Modify(m => m.PreviewPath, pending.PreviewPath)
|
||||
.Modify(m => m.Size, pending.Size)
|
||||
.Modify(m => m.Width, pending.Width)
|
||||
.Modify(m => m.Height, pending.Height)
|
||||
.Modify(m => m.Blurhash, pending.Blurhash)
|
||||
.Modify(m => m.DurationSeconds, pending.DurationSeconds)
|
||||
.Modify(m => m.ProcessingState, null)
|
||||
.ExecuteAsync(token);
|
||||
return true;
|
||||
}
|
||||
await DB.Default.Update<MediaAttachment>().MatchID(pending.ID)
|
||||
.Modify(m => m.ProcessingState, "failed").Modify(m => m.ProcessingError, outcome.Error).ExecuteAsync(token);
|
||||
return false;
|
||||
}
|
||||
|
||||
// the stored upload, made playable: probed, then remuxed without its metadata when browsers play it as it is (H.264,
|
||||
// VP8, VP9 or AV1 within MaxVideoPixels and MaxFrameRate), or else transcoded to H.264 that fits; a frame becomes the
|
||||
// poster. ffmpeg only ever reads the local file (no protocol but file, its format forced from the probe). Nothing is
|
||||
// saved until everything succeeded, and every temporary file goes, the stored upload too.
|
||||
async Task<MediaOutcome> ProcessIncoming(MediaAttachment attachment, CancellationToken token)
|
||||
{
|
||||
var options = _options.CurrentValue;
|
||||
var input = Incoming(attachment.ID);
|
||||
var temp = Path.Combine(Path.GetTempPath(), $"privapub-{Guid.NewGuid():N}");
|
||||
var input = temp + ".in";
|
||||
var output = temp + "." + extension;
|
||||
var frame = temp + ".png";
|
||||
var output = default(string);
|
||||
await _processing.WaitAsync(token);
|
||||
try
|
||||
{
|
||||
await using (var target = File.Create(input))
|
||||
await file.CopyToAsync(target, token);
|
||||
var probe = await FFProbe.AnalyseAsync(input, cancellationToken: token);
|
||||
var isVideo = contentType.StartsWith("video/") && probe.PrimaryVideoStream != default;
|
||||
await FFMpegArguments.FromFileInput(input)
|
||||
.OutputToFile(output, true, o => o.WithCustomArgument("-map 0 -map_metadata -1 -map_chapters -1 -c copy" + (extension is "mp4" or "m4a" ? " -movflags +faststart" : string.Empty)))
|
||||
var probe = await FFProbe.AnalyseAsync(input, default, token, "-protocol_whitelist file");
|
||||
if (probe.Duration > TimeSpan.FromSeconds(options.MaxSeconds))
|
||||
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: The file is too long");
|
||||
var video = attachment.ContentType.StartsWith("video/", StringComparison.Ordinal) ? probe.PrimaryVideoStream : default;
|
||||
var audio = probe.PrimaryAudioStream;
|
||||
if (video == default && audio == default)
|
||||
return MediaOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: The file could not be processed");
|
||||
var extension = AvTypes[attachment.ContentType];
|
||||
var format = probe.Format.FormatName.Split(',')[0];
|
||||
string arguments;
|
||||
if (video == default)
|
||||
arguments = "-map 0:a:0 -vn -dn -sn -map_metadata -1 -map_chapters -1 -c copy" + (extension == "m4a" ? " -movflags +faststart" : string.Empty);
|
||||
else if (video.CodecName is "h264" or "vp8" or "vp9" or "av1" && (long)video.Width * video.Height <= options.MaxVideoPixels
|
||||
&& video.FrameRate <= options.MaxFrameRate + 0.5)
|
||||
arguments = "-map 0:v:0 -map 0:a:0? -dn -sn -map_metadata -1 -map_chapters -1 -c copy" + (extension == "mp4" ? " -movflags +faststart" : string.Empty);
|
||||
else
|
||||
{
|
||||
var scale = Math.Min(1, Math.Sqrt(options.MaxVideoPixels / (double)((long)video.Width * video.Height)));
|
||||
var width = Math.Max(2, (int)(video.Width * scale) / 2 * 2);
|
||||
var height = Math.Max(2, (int)(video.Height * scale) / 2 * 2);
|
||||
extension = "mp4";
|
||||
arguments = $"-map 0:v:0 -map 0:a:0? -dn -sn -map_metadata -1 -map_chapters -1 -c:v libx264 -preset veryfast -crf 23 -pix_fmt yuv420p "
|
||||
+ $"-vf scale={width}:{height} -fpsmax {options.MaxFrameRate.ToString(CultureInfo.InvariantCulture)} -c:a aac -b:a 128k -movflags +faststart";
|
||||
}
|
||||
output = temp + "." + extension;
|
||||
await FFMpegArguments.FromFileInput(input, true, o => o.ForceFormat(format).WithCustomArgument("-protocol_whitelist file"))
|
||||
.OutputToFile(output, true, o => o.WithCustomArgument(arguments))
|
||||
.CancellableThrough(token)
|
||||
.ProcessAsynchronously();
|
||||
|
||||
attachment.Kind = isVideo ? "video" : "audio";
|
||||
attachment.ContentType = extension switch { "mp4" => "video/mp4", "webm" => isVideo ? "video/webm" : "audio/webm", "m4a" => "audio/mp4", "mp3" => "audio/mpeg", _ => contentType };
|
||||
attachment.FilePath = await Save(await File.ReadAllBytesAsync(output, token), extension, token);
|
||||
attachment.Size = new FileInfo(output).Length;
|
||||
if (isVideo)
|
||||
var made = await FFProbe.AnalyseAsync(output, default, token, "-protocol_whitelist file");
|
||||
byte[] poster = default;
|
||||
if (video != default)
|
||||
{
|
||||
attachment.Width = probe.PrimaryVideoStream.Width;
|
||||
attachment.Height = probe.PrimaryVideoStream.Height;
|
||||
var frame = temp + ".png";
|
||||
await FFMpeg.SnapshotAsync(input, frame, captureTime: TimeSpan.FromSeconds(Math.Min(1, probe.Duration.TotalSeconds / 2)));
|
||||
var still = ProcessImage(await File.ReadAllBytesAsync(frame, token), 640, _options.CurrentValue.PreviewSide, false);
|
||||
attachment.PreviewPath = await Save(still.Preview, "jpg", token);
|
||||
attachment.Width = made.PrimaryVideoStream?.Width;
|
||||
attachment.Height = made.PrimaryVideoStream?.Height;
|
||||
await FFMpeg.SnapshotAsync(output, frame, captureTime: TimeSpan.FromSeconds(Math.Min(1, made.Duration.TotalSeconds / 2)));
|
||||
var still = ProcessImage(await File.ReadAllBytesAsync(frame, token), 640, options.PreviewSide);
|
||||
poster = still.Preview;
|
||||
attachment.Blurhash = still.Blurhash;
|
||||
File.Delete(frame);
|
||||
}
|
||||
attachment.Kind = video != default ? "video" : "audio";
|
||||
attachment.ContentType = extension switch
|
||||
{
|
||||
"mp4" => "video/mp4",
|
||||
"webm" => video != default ? "video/webm" : "audio/webm",
|
||||
"m4a" => "audio/mp4",
|
||||
"mp3" => "audio/mpeg",
|
||||
"ogg" => "audio/ogg",
|
||||
"wav" => "audio/wav",
|
||||
"flac" => "audio/flac",
|
||||
_ => attachment.ContentType
|
||||
};
|
||||
attachment.DurationSeconds = Math.Round(made.Duration.TotalSeconds, 2);
|
||||
attachment.Size = new FileInfo(output).Length;
|
||||
attachment.FilePath = SaveFile(output, extension);
|
||||
if (poster != default)
|
||||
attachment.PreviewPath = await Save(poster, "jpg", token);
|
||||
return new MediaOutcome(attachment);
|
||||
}
|
||||
catch (Exception ex) when (ex is not OperationCanceledException)
|
||||
@@ -233,34 +513,38 @@ namespace PrivaPub.Domain.Media
|
||||
}
|
||||
finally
|
||||
{
|
||||
foreach (var path in new[] { input, output })
|
||||
_processing.Release();
|
||||
foreach (var path in new[] { input, output, frame }.Where(p => p != default))
|
||||
if (File.Exists(path))
|
||||
File.Delete(path);
|
||||
}
|
||||
}
|
||||
|
||||
async Task<string> Save(byte[] bytes, string extension, CancellationToken token)
|
||||
string SaveFile(string source, string extension)
|
||||
{
|
||||
var (relative, full) = NewPath(extension);
|
||||
File.Move(source, full);
|
||||
return relative;
|
||||
}
|
||||
|
||||
(string Relative, string Full) NewPath(string extension)
|
||||
{
|
||||
var now = DateTime.UtcNow;
|
||||
var relative = Path.Combine(now.ToString("yyyy", CultureInfo.InvariantCulture), now.ToString("MM", CultureInfo.InvariantCulture),
|
||||
$"{Convert.ToHexStringLower(RandomNumberGenerator.GetBytes(16))}.{extension}");
|
||||
var full = Path.Combine(Root, relative);
|
||||
Directory.CreateDirectory(Path.GetDirectoryName(full)!);
|
||||
return (relative.Replace('\\', '/'), full);
|
||||
}
|
||||
|
||||
async Task<string> Save(byte[] bytes, string extension, CancellationToken token)
|
||||
{
|
||||
var (relative, full) = NewPath(extension);
|
||||
await File.WriteAllBytesAsync(full, bytes, token);
|
||||
return relative.Replace('\\', '/');
|
||||
return relative;
|
||||
}
|
||||
|
||||
static string Clean(string value, int max) =>
|
||||
string.IsNullOrWhiteSpace(value) ? default : value.Trim().Length <= max ? value.Trim() : value.Trim()[..max];
|
||||
|
||||
static float[] Focus(string focus)
|
||||
{
|
||||
var parts = focus?.Split(',');
|
||||
if (parts is not { Length: 2 }
|
||||
|| !float.TryParse(parts[0], NumberStyles.Float, CultureInfo.InvariantCulture, out var x)
|
||||
|| !float.TryParse(parts[1], NumberStyles.Float, CultureInfo.InvariantCulture, out var y))
|
||||
return default;
|
||||
return new[] { Math.Clamp(x, -1, 1), Math.Clamp(y, -1, 1) };
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
using MongoDB.Entities;
|
||||
|
||||
using PrivaPub.Infrastructure.Jobs;
|
||||
using PrivaPub.Models.Jobs;
|
||||
using PrivaPub.Models.Media;
|
||||
|
||||
namespace PrivaPub.Domain.Media
|
||||
{
|
||||
// Audio and video sent to /api/v2/media, processed off the request (its lease renewed while ffmpeg runs): playable, or
|
||||
// marked failed with why. One at a time, since a transcode takes the machine's cores.
|
||||
public class ProcessMediaJob : IJobHandler
|
||||
{
|
||||
readonly IMediaService _media;
|
||||
readonly ILogger<ProcessMediaJob> _logger;
|
||||
|
||||
public ProcessMediaJob(IMediaService media, ILogger<ProcessMediaJob> logger)
|
||||
{
|
||||
_media = media;
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
public JobKind Kind => JobKind.ProcessMedia;
|
||||
public int Concurrency => 1;
|
||||
public int MaxAttempts => 3;
|
||||
public int PerHostLimit => 1;
|
||||
|
||||
public static Job JobFor(MediaAttachment pending, string host) => new()
|
||||
{
|
||||
Kind = JobKind.ProcessMedia,
|
||||
Payload = pending.ID,
|
||||
Host = host,
|
||||
DedupeKey = $"media|{pending.ID}"
|
||||
};
|
||||
|
||||
public async Task<JobOutcome> Handle(Job job, CancellationToken token)
|
||||
{
|
||||
var pending = await DB.Default.Find<MediaAttachment>().Match(m => m.ID == job.Payload && m.ProcessingState == "pending" && m.TrashedAt == null).ExecuteFirstAsync(token);
|
||||
if (pending == default)
|
||||
return JobOutcome.Done;//processed already, or trashed meanwhile
|
||||
if (!await _media.ProcessPending(pending, token))
|
||||
_logger.LogInformation("Upload {Id} could not be processed", pending.ID);
|
||||
return JobOutcome.Done;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,399 @@
|
||||
using MongoDB.Entities;
|
||||
|
||||
using PrivaPub.Domain.Media;
|
||||
using PrivaPub.Domain.Social;
|
||||
using PrivaPub.Federation.Actors;
|
||||
using PrivaPub.Federation.Rendering;
|
||||
using PrivaPub.Infrastructure.Backup;
|
||||
using PrivaPub.Infrastructure.Jobs;
|
||||
using PrivaPub.Models.Federation;
|
||||
using PrivaPub.Models.Jobs;
|
||||
using PrivaPub.Models.Media;
|
||||
using PrivaPub.Models.Post;
|
||||
using PrivaPub.Models.Social;
|
||||
using PrivaPub.Models.User;
|
||||
|
||||
using System.IO.Compression;
|
||||
using System.Text;
|
||||
using System.Text.Encodings.Web;
|
||||
using System.Text.Json;
|
||||
using System.Text.Json.Nodes;
|
||||
|
||||
using PostEntity = PrivaPub.Models.Post.Post;
|
||||
|
||||
namespace PrivaPub.Domain.Portability
|
||||
{
|
||||
// A persona's archive, in Mastodon's account-archive layout so other servers' importers read it:
|
||||
// - actor.json (as other servers see the persona: its public key, never the private one), avatar and header;
|
||||
// - outbox.json: every post of its own as a Create (any visibility but circles' and located posts, addressed as it was
|
||||
// delivered) and its boosts as Announces; the media its posts hold under media_attachments/files/, which the
|
||||
// attachments' urls name;
|
||||
// - likes.json, bookmarks.json, and the CSV files Mastodon exports: follows, lists, blocks, mutes, blocked servers,
|
||||
// bookmarks;
|
||||
// - privapub/: what only PrivaPub has: filters, followed hashtags, the notification policy, pinned posts, scheduled and
|
||||
// located posts, and archive.json saying whose it is.
|
||||
// Nothing of its root, its siblings, its keys or anyone's tokens. Written whole to a .partial file, then renamed; kept a
|
||||
// week (PersonaArchives.Forget).
|
||||
public class ExportArchiveJob(ILocalActorService localActors, IMediaService media, Backups backups, ILogger<ExportArchiveJob> logger) : IJobHandler
|
||||
{
|
||||
public const int Format = 1;
|
||||
static readonly JsonSerializerOptions Options = new() { WriteIndented = true, Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping };
|
||||
static readonly JsonWriterOptions Writer = new() { Indented = true, Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping };
|
||||
const string MediaFolder = "media_attachments/files/";
|
||||
|
||||
public JobKind Kind => JobKind.ExportArchive;
|
||||
public int Concurrency => 1;
|
||||
public int MaxAttempts => 2;
|
||||
public int PerHostLimit => 1;
|
||||
|
||||
public static Job JobFor(string avatarId, string host) => new()
|
||||
{
|
||||
Kind = JobKind.ExportArchive,
|
||||
Payload = avatarId,
|
||||
Host = host,
|
||||
DedupeKey = $"export|{avatarId}|{DateTime.UtcNow.Ticks}"
|
||||
};
|
||||
|
||||
public async Task<JobOutcome> Handle(Job job, CancellationToken token)
|
||||
{
|
||||
var avatarId = job.Payload;
|
||||
var actor = await localActors.FindById(LocalActorKind.Person, avatarId, token);
|
||||
if (actor == default)
|
||||
return JobOutcome.Done;
|
||||
await DB.Default.Update<PersonaArchive>().MatchID(avatarId).Modify(a => a.ExportState, "running").Modify(a => a.ExportError, null).ExecuteAsync(token);
|
||||
var folder = PersonaArchives.Folder(backups.Root, avatarId);
|
||||
ServerBackup.MakeDirectory(Path.GetDirectoryName(folder)!);
|
||||
ServerBackup.MakeDirectory(folder);
|
||||
var name = $"{actor.UserName}-{DateTime.UtcNow:yyyyMMdd-HHmmss}.zip";
|
||||
var partial = Path.Combine(folder, name + ServerBackup.PartialSuffix);
|
||||
try
|
||||
{
|
||||
await using (var file = new FileStream(partial, FileMode.Create, FileAccess.Write))
|
||||
using (var zip = new ZipArchive(file, ZipArchiveMode.Create))
|
||||
await Write(zip, actor, token);
|
||||
var final = Path.Combine(folder, name);
|
||||
File.Move(partial, final);
|
||||
foreach (var old in Directory.EnumerateFiles(folder, "*.zip").Where(f => f != final))
|
||||
File.Delete(old);
|
||||
await DB.Default.Update<PersonaArchive>().MatchID(avatarId)
|
||||
.Modify(a => a.ExportState, "ready").Modify(a => a.ExportReadyAt, DateTime.UtcNow)
|
||||
.Modify(a => a.ExportFile, name).Modify(a => a.ExportBytes, new FileInfo(final).Length).ExecuteAsync(token);
|
||||
return JobOutcome.Done;
|
||||
}
|
||||
catch (Exception ex) when (ex is not OperationCanceledException)
|
||||
{
|
||||
logger.LogError(ex, "The archive of {Persona} could not be made", actor.Handle);
|
||||
if (File.Exists(partial))
|
||||
File.Delete(partial);
|
||||
await DB.Default.Update<PersonaArchive>().MatchID(avatarId).Modify(a => a.ExportState, "failed")
|
||||
.Modify(a => a.ExportError, ex.Message).ExecuteAsync(CancellationToken.None);
|
||||
return JobOutcome.Done;
|
||||
}
|
||||
}
|
||||
|
||||
async Task Write(ZipArchive zip, LocalActor actor, CancellationToken token)
|
||||
{
|
||||
var avatar = await DB.Default.Find<Avatar>().MatchID(actor.Id).ExecuteFirstAsync(token);
|
||||
var files = new SortedSet<string>(StringComparer.Ordinal);
|
||||
var prefix = media.Url(string.Empty);
|
||||
|
||||
var document = ActivityPubRenderer.Actor(actor);
|
||||
foreach (var (field, kind) in new[] { ("icon", "avatar"), ("image", "header") })
|
||||
if (document[field] is JsonObject image && image["url"]?.GetValue<string>() is { } url && url.StartsWith(prefix, StringComparison.Ordinal))
|
||||
{
|
||||
var relative = url[prefix.Length..];
|
||||
var local = Path.Combine(media.Root, relative);
|
||||
if (!File.Exists(local))
|
||||
continue;
|
||||
var entry = kind + Path.GetExtension(relative);
|
||||
await Copy(zip, local, entry, token);
|
||||
image["url"] = entry;
|
||||
}
|
||||
await Text(zip, "actor.json", document.ToJsonString(Options), token);
|
||||
|
||||
var counts = await Outbox(zip, actor, prefix, files, token);
|
||||
foreach (var relative in files)
|
||||
{
|
||||
var local = Path.Combine(media.Root, relative);
|
||||
if (File.Exists(local))
|
||||
await Copy(zip, local, MediaFolder + relative, token);
|
||||
}
|
||||
|
||||
var handles = new Handles(localActors);
|
||||
await Collection(zip, "likes.json", await Uris(DB.Default.Find<Favourite, string>().Match(f => f.AccountId == actor.Id).Project(f => f.PostId), token), token);
|
||||
var bookmarks = await Uris(DB.Default.Find<Bookmark, string>().Match(b => b.AvatarId == actor.Id).Project(b => b.PostId), token);
|
||||
await Collection(zip, "bookmarks.json", bookmarks, token);
|
||||
await Text(zip, "bookmarks.csv", string.Concat(bookmarks.Select(b => b + "\n")), token);
|
||||
|
||||
var follows = await DB.Default.Find<Following>().Match(f => f.AvatarId == actor.Id).ExecuteAsync(token);
|
||||
var csv = new StringBuilder("Account address,Show boosts,Notify on new posts,Languages\n");
|
||||
foreach (var follow in follows)
|
||||
csv.Append($"{Csv(await handles.Of(follow.TargetActorURI, token))},{(follow.ShowReblogs ? "true" : "false")},false,\n");
|
||||
await Text(zip, "following_accounts.csv", csv.ToString(), token);
|
||||
|
||||
csv.Clear();
|
||||
var lists = await DB.Default.Find<PersonaList>().Match(l => l.AvatarId == actor.Id).ExecuteAsync(token);
|
||||
foreach (var list in lists)
|
||||
foreach (var member in await DB.Default.Find<PersonaListMember>().Match(m => m.ListId == list.ID).ExecuteAsync(token))
|
||||
if (follows.FirstOrDefault(f => f.TargetAccountId == member.AccountId) is { } followed)
|
||||
csv.Append($"{Csv(list.Title)},{Csv(await handles.Of(followed.TargetActorURI, token))}\n");
|
||||
await Text(zip, "lists.csv", csv.ToString(), token);
|
||||
|
||||
csv.Clear();
|
||||
foreach (var block in await DB.Default.Find<Block>().Match(b => b.AvatarId == actor.Id).ExecuteAsync(token))
|
||||
csv.Append(Csv(await handles.Of(block.TargetActorURI, token)) + "\n");
|
||||
await Text(zip, "blocked_accounts.csv", csv.ToString(), token);
|
||||
|
||||
csv.Clear().Append("Account address,Hide notifications\n");
|
||||
foreach (var mute in await DB.Default.Find<Mute>().Match(m => m.AvatarId == actor.Id).ExecuteAsync(token))
|
||||
csv.Append($"{Csv(await handles.Of(mute.TargetActorURI, token))},{(mute.HideNotifications ? "true" : "false")}\n");
|
||||
await Text(zip, "muted_accounts.csv", csv.ToString(), token);
|
||||
|
||||
csv.Clear();
|
||||
foreach (var domain in await DB.Default.Find<AccountDomainBlock, string>().Match(d => d.AvatarId == actor.Id).Project(d => d.Domain).ExecuteAsync(token))
|
||||
csv.Append(domain + "\n");
|
||||
await Text(zip, "blocked_domains.csv", csv.ToString(), token);
|
||||
|
||||
await PrivaPubParts(zip, actor, avatar, prefix, counts, token);
|
||||
}
|
||||
|
||||
// the outbox, a post at a time, newest first as Mastodon writes it: what was published and boosted
|
||||
async Task<(int Posts, int Boosts, int Located)> Outbox(ZipArchive zip, LocalActor actor, string prefix, ISet<string> files, CancellationToken token)
|
||||
{
|
||||
var entry = zip.CreateEntry("outbox.json", CompressionLevel.Optimal);
|
||||
await using var stream = entry.Open();
|
||||
await using var writer = new Utf8JsonWriter(stream, Writer);
|
||||
writer.WriteStartObject();
|
||||
writer.WriteString("@context", "https://www.w3.org/ns/activitystreams");
|
||||
writer.WriteString("id", "outbox.json");
|
||||
writer.WriteString("type", "OrderedCollection");
|
||||
writer.WritePropertyName("orderedItems");
|
||||
writer.WriteStartArray();
|
||||
var (posts, boosts) = (0, 0);
|
||||
var located = 0;
|
||||
var before = default(string);
|
||||
while (true)
|
||||
{
|
||||
var find = DB.Default.Find<PostEntity>().Match(p => p.GroupUserId == actor.Id && !p.IsFederatedCopy && !p.DeletedAt.HasValue);
|
||||
if (before != default)
|
||||
find = find.Match(f => f.Lt(p => p.ID, before));
|
||||
var batch = await find.Sort(p => p.ID, Order.Descending).Limit(200).ExecuteAsync(token);
|
||||
if (batch.Count == 0)
|
||||
break;
|
||||
before = batch[^1].ID;
|
||||
var originals = (await DB.Default.Find<PostEntity>().Match(p => batch.Select(b => b.ReblogOfPostId).Contains(p.ID)).ExecuteAsync(token))
|
||||
.ToDictionary(p => p.ID);
|
||||
foreach (var post in batch)
|
||||
{
|
||||
if (post.Visibility == PostVisibility.LocalGeo)
|
||||
{
|
||||
located++;
|
||||
continue;
|
||||
}
|
||||
if (post.Visibility == PostVisibility.Circle)
|
||||
continue;
|
||||
if (post.ReblogOfPostId != default)
|
||||
{
|
||||
if (!originals.TryGetValue(post.ReblogOfPostId, out var original) || string.IsNullOrEmpty(original.ObjectURI))
|
||||
continue;
|
||||
new JsonObject
|
||||
{
|
||||
["id"] = actor.ActivityUri($"announce-{post.ID}"),
|
||||
["type"] = "Announce",
|
||||
["actor"] = actor.Uri,
|
||||
["published"] = ActivityPubRenderer.Timestamp(post.CreationDate),
|
||||
["to"] = new JsonArray("https://www.w3.org/ns/activitystreams#Public"),
|
||||
["cc"] = new JsonArray(actor.Followers),
|
||||
["object"] = original.ObjectURI
|
||||
}.WriteTo(writer);
|
||||
boosts++;
|
||||
continue;
|
||||
}
|
||||
var group = post.GroupId == default ? default : await localActors.FindById(LocalActorKind.Group, post.GroupId, token);
|
||||
var note = ActivityPubRenderer.Note(post, actor, group, post.InReplyToURI);
|
||||
if (post.Visibility is PostVisibility.Direct or PostVisibility.FollowersOnly && (post.To.Count > 0 || post.Cc.Count > 0))
|
||||
{
|
||||
note["to"] = new JsonArray(post.To.Select(t => (JsonNode)t).ToArray());
|
||||
note["cc"] = new JsonArray(post.Cc.Select(c => (JsonNode)c).ToArray());
|
||||
}
|
||||
if (note["attachment"] is JsonArray attachments)
|
||||
foreach (var attachment in attachments.OfType<JsonObject>())
|
||||
if (attachment["url"]?.GetValue<string>() is { } url && url.StartsWith(prefix, StringComparison.Ordinal))
|
||||
{
|
||||
var relative = url[prefix.Length..];
|
||||
files.Add(relative);
|
||||
attachment["url"] = "/" + MediaFolder + relative;
|
||||
}
|
||||
var create = ActivityPubRenderer.Create(actor, note, $"create-{post.ID}");
|
||||
create["to"] = note["to"]?.DeepClone();
|
||||
create["cc"] = note["cc"]?.DeepClone();
|
||||
create.WriteTo(writer);
|
||||
posts++;
|
||||
}
|
||||
await writer.FlushAsync(token);
|
||||
}
|
||||
writer.WriteEndArray();
|
||||
writer.WriteNumber("totalItems", posts + boosts);
|
||||
writer.WriteEndObject();
|
||||
return (posts, boosts, located);
|
||||
}
|
||||
|
||||
async Task PrivaPubParts(ZipArchive zip, LocalActor actor, Avatar avatar, string prefix, (int Posts, int Boosts, int Located) counts, CancellationToken token)
|
||||
{
|
||||
var filters = await DB.Default.Find<PersonaFilter>().Match(f => f.AvatarId == actor.Id).ExecuteAsync(token);
|
||||
var filtered = await UriMap(filters.SelectMany(f => f.Statuses.Select(s => s.PostId)), token);
|
||||
await Json(zip, "privapub/filters.json", filters.Select(f => new
|
||||
{
|
||||
title = f.Title,
|
||||
context = f.Context,
|
||||
expiresAt = f.ExpiresAt,
|
||||
action = f.Action,
|
||||
keywords = f.Keywords.Select(k => new { keyword = k.Keyword, wholeWord = k.WholeWord }),
|
||||
statuses = f.Statuses.Select(s => filtered.GetValueOrDefault(s.PostId)).Where(u => u != default)
|
||||
}), token);
|
||||
await Json(zip, "privapub/followed_tags.json",
|
||||
await DB.Default.Find<FollowedTag, string>().Match(t => t.AvatarId == actor.Id).Project(t => t.Name).ExecuteAsync(token), token);
|
||||
var policy = await NotificationPolicies.Of(actor.Id, token);
|
||||
await Json(zip, "privapub/notification_policy.json", new
|
||||
{
|
||||
forNotFollowing = policy.ForNotFollowing.ToString(),
|
||||
forNotFollowers = policy.ForNotFollowers.ToString(),
|
||||
forNewAccounts = policy.ForNewAccounts.ToString(),
|
||||
forPrivateMentions = policy.ForPrivateMentions.ToString(),
|
||||
forLimitedAccounts = policy.ForLimitedAccounts.ToString()
|
||||
}, token);
|
||||
await Json(zip, "privapub/pins.json",
|
||||
await Uris(DB.Default.Find<Pin, string>().Match(p => p.AvatarId == actor.Id).Project(p => p.PostId), token), token);
|
||||
|
||||
var scheduled = await DB.Default.Find<ScheduledStatus>().Match(s => s.AvatarId == actor.Id).ExecuteAsync(token);
|
||||
var scheduledMedia = await MediaFiles(scheduled.SelectMany(s => s.Params.MediaIds), token);
|
||||
foreach (var file in scheduledMedia.Values)
|
||||
await CopyMedia(zip, file.Path, token);
|
||||
await Json(zip, "privapub/scheduled.json", scheduled.Select(s => new
|
||||
{
|
||||
scheduledAt = s.ScheduledAt,
|
||||
text = s.Params.Text,
|
||||
spoilerText = s.Params.SpoilerText,
|
||||
sensitive = s.Params.Sensitive,
|
||||
visibility = s.Params.Visibility,
|
||||
language = s.Params.Language,
|
||||
poll = s.Params.Poll,
|
||||
media = s.Params.MediaIds.Where(scheduledMedia.ContainsKey).Select(id => new { url = MediaFolder + scheduledMedia[id].Path, description = scheduledMedia[id].Description })
|
||||
}), token);
|
||||
|
||||
var located = await DB.Default.Find<PostEntity>().Match(p => p.GroupUserId == actor.Id && !p.IsFederatedCopy && !p.DeletedAt.HasValue
|
||||
&& p.Visibility == PostVisibility.LocalGeo).ExecuteAsync(token);
|
||||
foreach (var post in located)
|
||||
foreach (var item in post.Media.Where(m => m.URL?.StartsWith(prefix, StringComparison.Ordinal) == true))
|
||||
await CopyMedia(zip, item.URL[prefix.Length..], token);
|
||||
await Json(zip, "privapub/located.json", located.Select(p => new
|
||||
{
|
||||
id = p.ObjectURI,
|
||||
published = p.CreationDate,
|
||||
title = p.Title,
|
||||
text = p.Text,
|
||||
spoilerText = p.SpoilerText,
|
||||
latitude = p.Geo?.Coordinates[1],
|
||||
longitude = p.Geo?.Coordinates[0],
|
||||
rangeKm = p.RangeKm,
|
||||
media = p.Media.Where(m => m.URL?.StartsWith(prefix, StringComparison.Ordinal) == true)
|
||||
.Select(m => new { url = MediaFolder + m.URL[prefix.Length..], description = m.Description })
|
||||
}), token);
|
||||
|
||||
await Json(zip, "privapub/archive.json", new
|
||||
{
|
||||
format = Format,
|
||||
software = "privapub",
|
||||
actor = actor.Uri,
|
||||
username = avatar?.UserName ?? actor.UserName,
|
||||
exportedAt = DateTime.UtcNow,
|
||||
posts = counts.Posts,
|
||||
boosts = counts.Boosts,
|
||||
located = counts.Located,
|
||||
scheduled = scheduled.Count
|
||||
}, token);
|
||||
}
|
||||
|
||||
async Task CopyMedia(ZipArchive zip, string relative, CancellationToken token)
|
||||
{
|
||||
var local = Path.Combine(media.Root, relative);
|
||||
if (File.Exists(local) && zip.GetEntry(MediaFolder + relative) == default)
|
||||
await Copy(zip, local, MediaFolder + relative, token);
|
||||
}
|
||||
|
||||
static async Task<Dictionary<string, (string Path, string Description)>> MediaFiles(IEnumerable<string> ids, CancellationToken token)
|
||||
{
|
||||
var wanted = ids.Distinct().ToList();
|
||||
if (wanted.Count == 0)
|
||||
return [];
|
||||
return (await DB.Default.Find<MediaAttachment>().Match(m => wanted.Contains(m.ID) && m.TrashedAt == null).ExecuteAsync(token))
|
||||
.ToDictionary(m => m.ID, m => (m.FilePath, m.Description));
|
||||
}
|
||||
|
||||
static async Task<Dictionary<string, string>> UriMap(IEnumerable<string> postIds, CancellationToken token)
|
||||
{
|
||||
var ids = postIds.Where(id => id != default).Distinct().ToList();
|
||||
if (ids.Count == 0)
|
||||
return [];
|
||||
return (await DB.Default.Find<PostEntity>().Match(p => ids.Contains(p.ID)).Project(p => new PostEntity { ID = p.ID, ObjectURI = p.ObjectURI }).ExecuteAsync(token))
|
||||
.Where(p => !string.IsNullOrEmpty(p.ObjectURI)).ToDictionary(p => p.ID, p => p.ObjectURI);
|
||||
}
|
||||
|
||||
static async Task<List<string>> Uris<T>(Find<T, string> postIds, CancellationToken token) where T : IEntity
|
||||
{
|
||||
var ids = await postIds.ExecuteAsync(token);
|
||||
var map = await UriMap(ids, token);
|
||||
return ids.Where(map.ContainsKey).Select(id => map[id]).ToList();
|
||||
}
|
||||
|
||||
static async Task Collection(ZipArchive zip, string name, List<string> items, CancellationToken token) =>
|
||||
await Text(zip, name, new JsonObject
|
||||
{
|
||||
["@context"] = "https://www.w3.org/ns/activitystreams",
|
||||
["id"] = name,
|
||||
["type"] = "OrderedCollection",
|
||||
["totalItems"] = items.Count,
|
||||
["orderedItems"] = new JsonArray(items.Select(i => (JsonNode)i).ToArray())
|
||||
}.ToJsonString(Options), token);
|
||||
|
||||
static Task Json<T>(ZipArchive zip, string name, T value, CancellationToken token) =>
|
||||
Text(zip, name, JsonSerializer.Serialize(value, Options), token);
|
||||
|
||||
static async Task Text(ZipArchive zip, string name, string text, CancellationToken token)
|
||||
{
|
||||
await using var stream = zip.CreateEntry(name, CompressionLevel.Optimal).Open();
|
||||
await stream.WriteAsync(Encoding.UTF8.GetBytes(text), token);
|
||||
}
|
||||
|
||||
static async Task Copy(ZipArchive zip, string path, string name, CancellationToken token)
|
||||
{
|
||||
await using var source = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.Read, 81920, FileOptions.SequentialScan);
|
||||
await using var target = zip.CreateEntry(name, CompressionLevel.NoCompression).Open();
|
||||
await source.CopyToAsync(target, token);
|
||||
}
|
||||
|
||||
// a CSV cell, quoted when it must be
|
||||
static string Csv(string value) =>
|
||||
value.IndexOfAny([',', '"', '\n', '\r']) < 0 ? value : "\"" + value.Replace("\"", "\"\"") + "\"";
|
||||
|
||||
// an account as Mastodon's CSV files name it: user@host, or its address when the handle is not known
|
||||
sealed class Handles(ILocalActorService localActors)
|
||||
{
|
||||
readonly Dictionary<string, string> _known = new(StringComparer.Ordinal);
|
||||
|
||||
public async Task<string> Of(string actorUri, CancellationToken token)
|
||||
{
|
||||
if (_known.TryGetValue(actorUri, out var handle))
|
||||
return handle;
|
||||
if (await localActors.FindByUri(actorUri, token) is { } local)
|
||||
handle = local.Handle;
|
||||
else if (await DB.Default.Find<ForeignAvatar>().Match(a => a.ActorURI == actorUri).ExecuteFirstAsync(token) is { UserName.Length: > 0 } remote
|
||||
&& Uri.TryCreate(actorUri, UriKind.Absolute, out var parsed))
|
||||
handle = $"{remote.UserName}@{(string.IsNullOrEmpty(remote.Domain) ? parsed.Authority : remote.Domain)}";
|
||||
else
|
||||
handle = actorUri;
|
||||
return _known[actorUri] = handle;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,943 @@
|
||||
using Microsoft.AspNetCore.Http;
|
||||
|
||||
using MongoDB.Driver;
|
||||
using MongoDB.Entities;
|
||||
|
||||
using PrivaPub.Api.Mastodon.Controllers;
|
||||
using PrivaPub.Domain.Media;
|
||||
using PrivaPub.Domain.Relationships;
|
||||
using PrivaPub.Domain.Social;
|
||||
using PrivaPub.Domain.Statuses;
|
||||
using PrivaPub.Federation.Actors;
|
||||
using PrivaPub.Federation.Inbox;
|
||||
using PrivaPub.Federation.Moderation;
|
||||
using PrivaPub.Federation.Objects;
|
||||
using PrivaPub.Federation.Outbox;
|
||||
using PrivaPub.Federation.Rendering;
|
||||
using PrivaPub.Infrastructure.Backup;
|
||||
using PrivaPub.Infrastructure.Ids;
|
||||
using PrivaPub.Infrastructure.Jobs;
|
||||
using PrivaPub.Models.Federation;
|
||||
using PrivaPub.Models.Jobs;
|
||||
using PrivaPub.Models.Media;
|
||||
using PrivaPub.Models.Post;
|
||||
using PrivaPub.Models.Social;
|
||||
using PrivaPub.Models.User;
|
||||
|
||||
using System.Net;
|
||||
using System.Text.Json.Nodes;
|
||||
using System.Text.RegularExpressions;
|
||||
|
||||
using PostEntity = PrivaPub.Models.Post.Post;
|
||||
|
||||
namespace PrivaPub.Domain.Portability
|
||||
{
|
||||
// An archive restored into one of a root's personas (owner decision 2026-10-07): PrivaPub's own, or Mastodon's account
|
||||
// archive. Only the parts asked for, one at a time, with progress, until asked to stop.
|
||||
// Posts: only the archive actor's own Creates (boosts, direct and circle posts are counted, not imported); delivered to
|
||||
// no one, put in no home, notifying nobody, they show on the profile, the outbox, hashtags and search. From the same
|
||||
// actor (a persona's own archive, back home) a post keeps its id and address, and one already here (even deleted) or
|
||||
// whose address is a tombstone is left alone; from another actor it gets an id of its date and its address here, with
|
||||
// the original in ImportedFromURI, so importing twice changes nothing. Replies and self-quotes inside the archive point
|
||||
// at their copies; polls are closed, with no votes; mentions stay links; the HTML is sanitized as any remote post's;
|
||||
// media go through MediaService, its checks and the root's quota.
|
||||
// The rest: follows asked again (not on a blocked server), blocks, mutes, blocked servers, lists (with only accounts
|
||||
// followed), bookmarks (read by signed fetch), filters, followed hashtags, the notification policy, pins (not
|
||||
// announced), the profile; and only when asked, located and scheduled posts (PrivaPub's archives) and likes, which tell
|
||||
// their authors. Followers are never imported: they choose.
|
||||
public partial class ImportArchiveJob(IServiceScopeFactory scopes, ILocalActorService localActors, IMediaService media, IDomainBlocks domainBlocks,
|
||||
IJobQueue jobs, ILogger<ImportArchiveJob> logger) : IJobHandler
|
||||
{
|
||||
static readonly HashSet<string> Public = ["https://www.w3.org/ns/activitystreams#Public", "as:Public", "Public"];
|
||||
static readonly HashSet<string> Postable = ["Note", "Article", "Page", "Question"];
|
||||
const int MaxMedia = 4;
|
||||
|
||||
public JobKind Kind => JobKind.ImportArchive;
|
||||
public int Concurrency => 1;
|
||||
public int MaxAttempts => 1;
|
||||
public int PerHostLimit => 1;
|
||||
|
||||
public static Job JobFor(string avatarId, string host) => new()
|
||||
{
|
||||
Kind = JobKind.ImportArchive,
|
||||
Payload = avatarId,
|
||||
Host = host,
|
||||
DedupeKey = $"import|{avatarId}|{DateTime.UtcNow.Ticks}"
|
||||
};
|
||||
|
||||
sealed class Run(PersonaArchive state, LocalActor me, SafeArchive archive, string archiveActor, string archiveFollowers, bool sameActor,
|
||||
IServiceProvider services)
|
||||
{
|
||||
public PersonaArchive State { get; } = state;
|
||||
public string ArchiveFollowers { get; } = archiveFollowers;
|
||||
public LocalActor Me { get; } = me;
|
||||
public SafeArchive Archive { get; } = archive;
|
||||
public string ArchiveActor { get; } = archiveActor;
|
||||
public bool SameActor { get; } = sameActor;
|
||||
public IServiceProvider Services { get; } = services;
|
||||
public Dictionary<string, (string Id, string Uri)> Posts { get; } = new(StringComparer.Ordinal);//archive object id → here
|
||||
public Dictionary<string, string> Parents { get; } = new(StringComparer.Ordinal);//archive object id → what it answers
|
||||
public DateTime LastSaved { get; set; }
|
||||
|
||||
public void Count(string what, int by = 1) => State.ImportCounts[what] = State.ImportCounts.GetValueOrDefault(what) + by;
|
||||
}
|
||||
|
||||
public async Task<JobOutcome> Handle(Job job, CancellationToken token)
|
||||
{
|
||||
var state = await DB.Default.Find<PersonaArchive>().MatchID(job.Payload).ExecuteFirstAsync(token);
|
||||
if (state is not { ImportState: "queued" or "running" })
|
||||
return JobOutcome.Done;
|
||||
var me = await localActors.FindById(LocalActorKind.Person, job.Payload, token);
|
||||
var path = state.ImportFile;
|
||||
try
|
||||
{
|
||||
if (me == default || !await RootMayImport(me.Id, token))
|
||||
{
|
||||
await End(state, "failed", "the persona or its login can no longer import", token);
|
||||
return JobOutcome.Done;
|
||||
}
|
||||
var (archive, refused) = path == default ? (default, "no archive") : SafeArchive.Open(path);
|
||||
if (archive == default)
|
||||
{
|
||||
await End(state, "failed", refused, token);
|
||||
return JobOutcome.Done;
|
||||
}
|
||||
using (archive)
|
||||
{
|
||||
var actor = archive.Json("actor.json") as JsonObject;
|
||||
var actorId = Text(actor?["id"]);
|
||||
if (actorId == default)
|
||||
{
|
||||
await End(state, "failed", "the archive has no actor.json", token);
|
||||
return JobOutcome.Done;
|
||||
}
|
||||
using var scope = scopes.CreateScope();
|
||||
var run = new Run(state, me, archive, actorId, Text(actor["followers"]) ?? actorId.TrimEnd('/') + "/followers", actorId == me.Uri,
|
||||
scope.ServiceProvider);
|
||||
state.ImportState = "running";
|
||||
state.ImportFrom = actorId;
|
||||
state.ImportSameActor = run.SameActor;
|
||||
state.ImportCounts = [];
|
||||
await Save(run, token, force: true);
|
||||
|
||||
var parts = state.ImportParts.ToHashSet(StringComparer.Ordinal);
|
||||
var privapub = Text((archive.Json("privapub/archive.json") as JsonObject)?["software"]) == "privapub";
|
||||
if (parts.Contains("profile") && !await Stopped(run, token))
|
||||
await Profile(run, actor, token);
|
||||
if (parts.Contains("posts") && !await Stopped(run, token))
|
||||
await Posts(run, token);
|
||||
else
|
||||
await KnownPosts(run, token);
|
||||
if (parts.Contains("located") && privapub && !await Stopped(run, token))
|
||||
await Located(run, token);
|
||||
if (parts.Contains("scheduled") && privapub && !await Stopped(run, token))
|
||||
await Scheduled(run, token);
|
||||
if (parts.Contains("tags") && !await Stopped(run, token))
|
||||
await Tags(run, token);
|
||||
if (parts.Contains("notificationpolicy") && !await Stopped(run, token))
|
||||
await Policy(run, token);
|
||||
if (parts.Contains("filters") && !await Stopped(run, token))
|
||||
await Filters(run, token);
|
||||
if (parts.Contains("pins") && !await Stopped(run, token))
|
||||
await Pins(run, token);
|
||||
if (parts.Contains("domainblocks") && !await Stopped(run, token))
|
||||
await DomainBlocks(run, token);
|
||||
if (parts.Contains("blocks") && !await Stopped(run, token))
|
||||
await Accounts(run, "blocked_accounts.csv", "blocks", Block, token);
|
||||
if (parts.Contains("mutes") && !await Stopped(run, token))
|
||||
await Accounts(run, "muted_accounts.csv", "mutes", Mute, token);
|
||||
if (parts.Contains("follows") && !await Stopped(run, token))
|
||||
await Accounts(run, "following_accounts.csv", "follows", Follow, token);
|
||||
if (parts.Contains("lists") && !await Stopped(run, token))
|
||||
await Lists(run, token);
|
||||
if (parts.Contains("bookmarks") && !await Stopped(run, token))
|
||||
await Bookmarks(run, token);
|
||||
if (parts.Contains("likes") && !await Stopped(run, token))
|
||||
await Likes(run, token);
|
||||
await End(state, await Stopped(run, token) ? "stopped" : "done", default, token);
|
||||
}
|
||||
}
|
||||
catch (Exception ex) when (ex is not OperationCanceledException)
|
||||
{
|
||||
logger.LogError(ex, "The import into {Persona} failed", me?.Handle ?? job.Payload);
|
||||
await End(state, "failed", ex.Message, CancellationToken.None);
|
||||
}
|
||||
finally
|
||||
{
|
||||
if (path != default && File.Exists(path))
|
||||
File.Delete(path);
|
||||
}
|
||||
return JobOutcome.Done;
|
||||
}
|
||||
|
||||
static async Task<bool> RootMayImport(string avatarId, CancellationToken token)
|
||||
{
|
||||
var link = await DB.Default.Find<RootToAvatar>().Match(r => r.AvatarId == avatarId).ExecuteFirstAsync(token);
|
||||
var root = link == default ? default : await DB.Default.Find<RootUser>().MatchID(link.RootId).ExecuteFirstAsync(token);
|
||||
return root is { IsBanned: false, DeletedAt: null };
|
||||
}
|
||||
|
||||
static async Task End(PersonaArchive state, string how, string error, CancellationToken token) =>
|
||||
await DB.Default.Update<PersonaArchive>().MatchID(state.ID)
|
||||
.Modify(a => a.ImportState, how).Modify(a => a.ImportError, error).Modify(a => a.ImportEndedAt, DateTime.UtcNow)
|
||||
.Modify(a => a.ImportCounts, state.ImportCounts).Modify(a => a.ImportDone, state.ImportDone).Modify(a => a.ImportTotal, state.ImportTotal)
|
||||
.Modify(a => a.ImportFile, null).Modify(a => a.StopAsked, false)
|
||||
.ExecuteAsync(token);
|
||||
|
||||
// progress written at most every two seconds; the answer to whether the root asked to stop
|
||||
static async Task Save(Run run, CancellationToken token, bool force = false)
|
||||
{
|
||||
if (!force && DateTime.UtcNow - run.LastSaved < TimeSpan.FromSeconds(2))
|
||||
return;
|
||||
run.LastSaved = DateTime.UtcNow;
|
||||
await DB.Default.Update<PersonaArchive>().MatchID(run.State.ID)
|
||||
.Modify(a => a.ImportState, run.State.ImportState).Modify(a => a.ImportFrom, run.State.ImportFrom)
|
||||
.Modify(a => a.ImportSameActor, run.State.ImportSameActor).Modify(a => a.ImportCounts, run.State.ImportCounts)
|
||||
.Modify(a => a.ImportDone, run.State.ImportDone).Modify(a => a.ImportTotal, run.State.ImportTotal)
|
||||
.ExecuteAsync(token);
|
||||
}
|
||||
|
||||
static async Task<bool> Stopped(Run run, CancellationToken token)
|
||||
{
|
||||
await Save(run, token);
|
||||
return await DB.Default.Find<PersonaArchive>().MatchID(run.State.ID).Match(a => a.StopAsked).ExecuteAnyAsync(token);
|
||||
}
|
||||
|
||||
// ---- posts
|
||||
|
||||
// what an outbox item is to this import: the object to import, or why not
|
||||
(JsonObject Note, string Skipped) Importable(Run run, JsonObject item)
|
||||
{
|
||||
var type = Text(item["type"]);
|
||||
if (type == "Announce")
|
||||
return (default, "boosts skipped");
|
||||
if (type != "Create" || item["object"] is not JsonObject note || !Postable.Contains(Text(note["type"]) ?? string.Empty))
|
||||
return (default, "not a post");
|
||||
if (Text(item["actor"]) != run.ArchiveActor || Id(note["attributedTo"]) is { } author && author != run.ArchiveActor)
|
||||
return (default, "not the archive's own");
|
||||
if (Text(note["id"]) == default)
|
||||
return (default, "not a post");
|
||||
return Visibility(run, note) == default ? (default, "direct or circle posts skipped") : (note, default);
|
||||
}
|
||||
|
||||
PostVisibility? Visibility(Run run, JsonObject note)
|
||||
{
|
||||
var to = Strings(note["to"]);
|
||||
var cc = Strings(note["cc"]);
|
||||
if (to.Any(Public.Contains))
|
||||
return PostVisibility.Public;
|
||||
if (cc.Any(Public.Contains))
|
||||
return PostVisibility.Unlisted;
|
||||
if (to.Concat(cc).Contains(run.ArchiveFollowers) && Text(note["audience"]) == default)
|
||||
return PostVisibility.FollowersOnly;
|
||||
return default;
|
||||
}
|
||||
|
||||
// what is already here of the archive's posts, for pins, filters and bookmarks when posts are not imported again
|
||||
async Task KnownPosts(Run run, CancellationToken token)
|
||||
{
|
||||
await foreach (var item in run.Archive.Outbox(token))
|
||||
{
|
||||
if (Importable(run, item).Note is not { } note)
|
||||
continue;
|
||||
var original = Text(note["id"]);
|
||||
var here = run.SameActor
|
||||
? await DB.Default.Find<PostEntity>().Match(p => p.ObjectURI == original && p.GroupUserId == run.Me.Id).ExecuteFirstAsync(token)
|
||||
: await DB.Default.Find<PostEntity>().Match(p => p.GroupUserId == run.Me.Id && p.ImportedFromURI == original).ExecuteFirstAsync(token);
|
||||
if (here is { DeletedAt: null })
|
||||
run.Posts[original] = (here.ID, here.ObjectURI);
|
||||
}
|
||||
}
|
||||
|
||||
async Task Posts(Run run, CancellationToken token)
|
||||
{
|
||||
// first the ids each post will have here, so a reply or a quote anywhere in the archive finds its copy
|
||||
var planned = new Dictionary<string, (string Id, string Uri, bool New)>(StringComparer.Ordinal);
|
||||
var total = 0L;
|
||||
await foreach (var item in run.Archive.Outbox(token))
|
||||
{
|
||||
total++;
|
||||
if (Importable(run, item).Note is not { } note)
|
||||
continue;
|
||||
var original = Text(note["id"]);
|
||||
var published = Date(note["published"]) ?? DateTime.UtcNow;
|
||||
if (Id(note["inReplyTo"]) is { } answers)
|
||||
run.Parents[original] = answers;
|
||||
if (run.SameActor)
|
||||
{
|
||||
var id = original.TrimEnd('/').Split('/')[^1];
|
||||
if (!IsObjectId(id) || original != run.Me.PostUri(id))
|
||||
id = PrivacyIds.At(Clamp(published));
|
||||
var here = await DB.Default.Find<PostEntity>().Match(p => p.ID == id || p.ObjectURI == original).ExecuteFirstAsync(token);
|
||||
var tombstone = await DB.Default.Find<DeletedObject>().Match(d => d.ObjectURI == original).ExecuteAnyAsync(token);
|
||||
if (here != default || tombstone)
|
||||
{
|
||||
if (here is { DeletedAt: null, GroupUserId: var owner } && owner == run.Me.Id)
|
||||
run.Posts[original] = (here.ID, here.ObjectURI);
|
||||
run.Count(here is { DeletedAt: not null } || tombstone ? "posts deleted here" : "posts already here");
|
||||
continue;
|
||||
}
|
||||
planned[original] = (id, original, true);
|
||||
}
|
||||
else
|
||||
{
|
||||
var here = await DB.Default.Find<PostEntity>().Match(p => p.GroupUserId == run.Me.Id && p.ImportedFromURI == original).ExecuteFirstAsync(token);
|
||||
if (here != default)
|
||||
{
|
||||
if (here.DeletedAt == default)
|
||||
run.Posts[original] = (here.ID, here.ObjectURI);
|
||||
run.Count("posts already here");
|
||||
continue;
|
||||
}
|
||||
var id = PrivacyIds.At(Clamp(published));
|
||||
planned[original] = (id, run.Me.PostUri(id), true);
|
||||
}
|
||||
}
|
||||
foreach (var (original, (id, uri, _)) in planned)
|
||||
run.Posts[original] = (id, uri);
|
||||
run.State.ImportTotal = total;
|
||||
run.State.ImportDone = 0;
|
||||
|
||||
await foreach (var item in run.Archive.Outbox(token))
|
||||
{
|
||||
run.State.ImportDone++;
|
||||
if (run.State.ImportDone % 50 == 0 && await Stopped(run, token))
|
||||
return;
|
||||
var (note, skipped) = Importable(run, item);
|
||||
if (note == default)
|
||||
{
|
||||
run.Count(skipped);
|
||||
continue;
|
||||
}
|
||||
var original = Text(note["id"]);
|
||||
if (!planned.TryGetValue(original, out var target))
|
||||
continue;
|
||||
await Post(run, note, original, target.Id, target.Uri, Visibility(run, note).Value, token);
|
||||
run.Count("posts");
|
||||
}
|
||||
}
|
||||
|
||||
async Task Post(Run run, JsonObject note, string original, string id, string uri, PostVisibility visibility, CancellationToken token)
|
||||
{
|
||||
var me = run.Me;
|
||||
var summary = Clean(Text(note["summary"]));
|
||||
var content = Text(note["content"]) ?? (note["contentMap"] as JsonObject)?.Select(c => Text(c.Value)).FirstOrDefault();
|
||||
var source = note["source"] as JsonObject;
|
||||
var post = new PostEntity
|
||||
{
|
||||
ID = id,
|
||||
GroupUserId = me.Id,
|
||||
AuthorAccountId = me.Id,
|
||||
ActorURI = me.Uri,
|
||||
Visibility = visibility,
|
||||
Title = Clean(Text(note["name"])),
|
||||
SpoilerText = summary,
|
||||
HasContentWarning = summary != default || note["sensitive"]?.GetValueKind() == System.Text.Json.JsonValueKind.True,
|
||||
ContentHtml = content == default ? default : ContentSanitizer.Html(content),
|
||||
Text = Text(source?["mediaType"]) is "text/plain" or "text/markdown" ? Text(source["content"]) : default,
|
||||
ContentFormat = ContentFormat.Html,
|
||||
Language = (note["contentMap"] as JsonObject)?.Select(c => c.Key).FirstOrDefault(),
|
||||
Tags = (note["tag"] as JsonArray ?? [])
|
||||
.OfType<JsonObject>().Where(t => Text(t["type"]) == "Hashtag").Select(t => TagsController.Normalise(Text(t["name"])))
|
||||
.Where(t => t.Length > 0).Distinct().ToList(),
|
||||
ObjectURI = uri,
|
||||
Url = me.PostHtmlUrl(id),
|
||||
ActivityURI = me.ActivityUri("create-" + id),
|
||||
ImportedFromURI = run.SameActor ? default : original,
|
||||
CreationDate = Clamp(Date(note["published"]) ?? DateTime.UtcNow),
|
||||
LocalQuotePolicy = me.Settings.QuotePolicy ?? QuotePolicies.Public
|
||||
};
|
||||
if (Id(note["inReplyTo"]) is { } inReplyTo)
|
||||
{
|
||||
if (run.Posts.TryGetValue(inReplyTo, out var parent))
|
||||
(post.AnsweringToPostId, post.InReplyToURI, post.InReplyToAccountId) = (parent.Id, parent.Uri, me.Id);
|
||||
else
|
||||
{
|
||||
post.InReplyToURI = inReplyTo;
|
||||
if (await DB.Default.Find<PostEntity>().Match(p => p.ObjectURI == inReplyTo).ExecuteFirstAsync(token) is { } held)
|
||||
(post.AnsweringToPostId, post.InReplyToAccountId) = (held.ID, held.AuthorAccountId ?? held.GroupUserId);
|
||||
}
|
||||
}
|
||||
foreach (var key in new[] { "quote", "quoteUrl", "quoteUri", "_misskey_quote" })
|
||||
if (Text(note[key]) is { } quoted && run.Posts.TryGetValue(quoted, out var copy))
|
||||
{
|
||||
(post.QuotedPostId, post.QuoteURI, post.QuoteState, post.QuoteByConsent) = (copy.Id, copy.Uri, QuoteState.Accepted, true);
|
||||
break;
|
||||
}
|
||||
if (Text(note["type"]) == "Question")
|
||||
post.Poll = Poll(note);
|
||||
// the conversation (FEP-7888) of its thread's first post, when the thread starts in the archive
|
||||
if (visibility is PostVisibility.Public or PostVisibility.Unlisted && Root(run, original) is { } root && run.Posts.TryGetValue(root, out var first))
|
||||
post.ContextURI = first.Uri + "/context";
|
||||
post.Media = await Media(run, note["attachment"] as JsonArray, id, token);
|
||||
var rendered = ActivityPubRenderer.Note(post, me, default, post.InReplyToURI);
|
||||
post.To = Strings(rendered["to"]);
|
||||
post.Cc = Strings(rendered["cc"]);
|
||||
await DB.Default.SaveAsync(post, token);
|
||||
if (post.AnsweringToPostId != default)
|
||||
await DB.Default.Update<PostEntity>().MatchID(post.AnsweringToPostId).Modify(b => b.Inc(p => p.RepliesCount, 1)).ExecuteAsync(token);
|
||||
}
|
||||
|
||||
// the first post of an archive post's thread, when the thread starts in the archive
|
||||
static string Root(Run run, string original)
|
||||
{
|
||||
var seen = new HashSet<string>(StringComparer.Ordinal);
|
||||
while (run.Parents.TryGetValue(original, out var parent) && seen.Add(original))
|
||||
{
|
||||
if (!run.Posts.ContainsKey(parent))
|
||||
return default;
|
||||
original = parent;
|
||||
}
|
||||
return original;
|
||||
}
|
||||
|
||||
static PostPoll Poll(JsonObject note)
|
||||
{
|
||||
var multiple = note["anyOf"] is JsonArray;
|
||||
var options = (note["anyOf"] ?? note["oneOf"]) as JsonArray ?? [];
|
||||
var ends = Date(note["endTime"]) ?? Date(note["closed"]) ?? DateTime.UtcNow;
|
||||
var now = DateTime.UtcNow;
|
||||
return new PostPoll
|
||||
{
|
||||
Options = options.OfType<JsonObject>().Select(o => new PollOption { Title = Text(o["name"]) }).Where(o => o.Title != default).ToList(),
|
||||
Multiple = multiple,
|
||||
ExpiresAt = ends < now ? ends : now,
|
||||
ClosedAt = ends < now ? ends : now,
|
||||
VotersCount = 0
|
||||
};
|
||||
}
|
||||
|
||||
// the attachments the archive holds, uploaded as the persona's own (checked, processed, counted against its quota)
|
||||
async Task<List<PostMedia>> Media(Run run, JsonArray attachments, string postId, CancellationToken token)
|
||||
{
|
||||
var made = new List<PostMedia>();
|
||||
foreach (var attachment in (attachments ?? []).OfType<JsonObject>().Take(MaxMedia))
|
||||
{
|
||||
var url = Text(attachment["url"]) ?? Text((attachment["url"] as JsonArray)?.OfType<JsonObject>().FirstOrDefault()?["href"]);
|
||||
var file = url?.TrimStart('/');
|
||||
if (file == default || file.Contains("://") || !run.Archive.Has(file))
|
||||
{
|
||||
run.Count("media not in the archive");
|
||||
continue;
|
||||
}
|
||||
var focus = attachment["focalPoint"] is JsonArray { Count: 2 } point ? $"{point[0]},{point[1]}" : default;
|
||||
var outcome = await Upload(run, file, Text(attachment["mediaType"]), Clean(Text(attachment["name"])), focus, token);
|
||||
if (outcome?.Ok != true)
|
||||
{
|
||||
run.Count("media refused");
|
||||
continue;
|
||||
}
|
||||
var row = outcome.Attachment;
|
||||
await DB.Default.Update<MediaAttachment>().MatchID(row.ID).Modify(m => m.PostId, postId).Modify(m => m.AttachedAt, DateTime.UtcNow).ExecuteAsync(token);
|
||||
made.Add(new PostMedia
|
||||
{
|
||||
AttachmentId = row.ID,
|
||||
ContentType = row.ContentType,
|
||||
Kind = row.Kind,
|
||||
DurationSeconds = row.DurationSeconds,
|
||||
URL = media.Url(row.FilePath),
|
||||
PreviewURL = media.Url(row.PreviewPath ?? row.FilePath),
|
||||
Description = row.Description,
|
||||
Blurhash = row.Blurhash,
|
||||
Width = row.Width,
|
||||
Height = row.Height,
|
||||
Focus = row.Focus
|
||||
});
|
||||
run.Count("media");
|
||||
}
|
||||
return made;
|
||||
}
|
||||
|
||||
// a file of the archive given to MediaService as an upload would be
|
||||
async Task<MediaOutcome> Upload(Run run, string file, string contentType, string description, string focus, CancellationToken token)
|
||||
{
|
||||
var temporary = Path.Combine(Path.GetTempPath(), $"privapub-import-{Guid.NewGuid():N}{Path.GetExtension(file)}");
|
||||
try
|
||||
{
|
||||
await using (var source = run.Archive.Read(file))
|
||||
await using (var target = new FileStream(temporary, FileMode.CreateNew, FileAccess.Write))
|
||||
await source.CopyToAsync(target, token);
|
||||
await using var stream = new FileStream(temporary, FileMode.Open, FileAccess.Read);
|
||||
var form = new FormFile(stream, 0, stream.Length, "file", Path.GetFileName(file))
|
||||
{
|
||||
Headers = new HeaderDictionary(),
|
||||
ContentType = contentType ?? (MediaService.ContentTypes.TryGetContentType(file, out var guessed) ? guessed : "application/octet-stream")
|
||||
};
|
||||
return await media.Upload(run.Me, form, description, focus, later: false, token);
|
||||
}
|
||||
finally
|
||||
{
|
||||
File.Delete(temporary);
|
||||
}
|
||||
}
|
||||
|
||||
// ---- PrivaPub's own parts
|
||||
|
||||
async Task Located(Run run, CancellationToken token)
|
||||
{
|
||||
foreach (var located in (run.Archive.Json("privapub/located.json") as JsonArray ?? []).OfType<JsonObject>())
|
||||
{
|
||||
var original = Text(located["id"]);
|
||||
if (original != default && await DB.Default.Find<PostEntity>().Match(p => p.GroupUserId == run.Me.Id
|
||||
&& (p.ObjectURI == original || p.ImportedFromURI == original)).ExecuteAnyAsync(token))
|
||||
{
|
||||
run.Count("located posts already here");
|
||||
continue;
|
||||
}
|
||||
if (located["latitude"]?.GetValue<double>() is not { } latitude || located["longitude"]?.GetValue<double>() is not { } longitude)
|
||||
continue;
|
||||
var id = PrivacyIds.At(Clamp(Date(located["published"]) ?? DateTime.UtcNow));
|
||||
var post = new PostEntity
|
||||
{
|
||||
ID = id,
|
||||
GroupUserId = run.Me.Id,
|
||||
AuthorAccountId = run.Me.Id,
|
||||
ActorURI = run.Me.Uri,
|
||||
Visibility = PostVisibility.LocalGeo,
|
||||
IsLocalOnly = true,
|
||||
Title = Clean(Text(located["title"])),
|
||||
SpoilerText = Clean(Text(located["spoilerText"])),
|
||||
HasContentWarning = Clean(Text(located["spoilerText"])) != default,
|
||||
Text = Text(located["text"]),
|
||||
ContentHtml = ActivityPubRenderer.Html(Text(located["text"]) ?? string.Empty),
|
||||
ContentFormat = ContentFormat.Markdown,
|
||||
Geo = new MongoDB.Entities.Coordinates2D(Math.Round(longitude, 2), Math.Round(latitude, 2)),
|
||||
RangeKm = (float)Math.Clamp(located["rangeKm"]?.GetValue<double>() ?? 5, 1, 50),
|
||||
ObjectURI = run.Me.PostUri(id),
|
||||
Url = run.Me.PostHtmlUrl(id),
|
||||
ImportedFromURI = original,
|
||||
CreationDate = Clamp(Date(located["published"]) ?? DateTime.UtcNow)
|
||||
};
|
||||
var files = (located["media"] as JsonArray ?? []).OfType<JsonObject>()
|
||||
.Select(m => new JsonObject { ["url"] = Text(m["url"]), ["name"] = Text(m["description"]) }).ToArray();
|
||||
post.Media = await Media(run, new JsonArray(files), id, token);
|
||||
await DB.Default.SaveAsync(post, token);
|
||||
run.Count("located posts");
|
||||
}
|
||||
}
|
||||
|
||||
async Task Scheduled(Run run, CancellationToken token)
|
||||
{
|
||||
var made = new List<ScheduledStatus>();
|
||||
foreach (var scheduled in (run.Archive.Json("privapub/scheduled.json") as JsonArray ?? []).OfType<JsonObject>())
|
||||
{
|
||||
var at = Date(scheduled["scheduledAt"]);
|
||||
if (at is not { } when || when < DateTime.UtcNow.AddMinutes(5) || await ScheduledStatuses.Refusal(run.Me.Id, when, default, token) != default)
|
||||
{
|
||||
run.Count("scheduled posts past or refused");
|
||||
continue;
|
||||
}
|
||||
var status = new ScheduledStatus
|
||||
{
|
||||
AvatarId = run.Me.Id,
|
||||
ScheduledAt = when,
|
||||
Params = new ScheduledParams
|
||||
{
|
||||
Text = Text(scheduled["text"]),
|
||||
SpoilerText = Text(scheduled["spoilerText"]),
|
||||
Sensitive = scheduled["sensitive"]?.GetValueKind() == System.Text.Json.JsonValueKind.True,
|
||||
Visibility = Text(scheduled["visibility"]) is { } visibility && visibility is "public" or "unlisted" or "private" or "direct" ? visibility : "public",
|
||||
Language = Text(scheduled["language"])
|
||||
}
|
||||
};
|
||||
status.ID = (string)status.GenerateNewID();
|
||||
foreach (var file in (scheduled["media"] as JsonArray ?? []).OfType<JsonObject>().Take(MaxMedia))
|
||||
{
|
||||
var path = Text(file["url"])?.TrimStart('/');
|
||||
if (path == default || !run.Archive.Has(path))
|
||||
continue;
|
||||
var outcome = await Upload(run, path, default, Clean(Text(file["description"])), default, token);
|
||||
if (outcome?.Ok != true)
|
||||
continue;
|
||||
await DB.Default.Update<MediaAttachment>().MatchID(outcome.Attachment.ID).Modify(m => m.ScheduledStatusId, status.ID).ExecuteAsync(token);
|
||||
status.Params.MediaIds.Add(outcome.Attachment.ID);
|
||||
}
|
||||
await DB.Default.SaveAsync(status, token);
|
||||
made.Add(status);
|
||||
run.Count("scheduled posts");
|
||||
}
|
||||
if (made.Count > 0)
|
||||
await jobs.EnqueueMany(made.Select(s => ScheduledStatuses.JobFor(s, new Uri(localActors.BaseAddress).Host)), token);
|
||||
}
|
||||
|
||||
async Task Tags(Run run, CancellationToken token)
|
||||
{
|
||||
var mine = (await DB.Default.Find<FollowedTag, string>().Match(t => t.AvatarId == run.Me.Id).Project(t => t.Name).ExecuteAsync(token)).ToHashSet();
|
||||
foreach (var name in (run.Archive.Json("privapub/followed_tags.json") as JsonArray ?? []).Select(t => TagsController.Normalise(Text(t))))
|
||||
{
|
||||
if (name.Length == 0 || mine.Contains(name) || mine.Count >= 500)
|
||||
continue;
|
||||
await DB.Default.SaveAsync(new FollowedTag { AvatarId = run.Me.Id, Name = name }, token);
|
||||
mine.Add(name);
|
||||
run.Count("followed hashtags");
|
||||
}
|
||||
}
|
||||
|
||||
async Task Policy(Run run, CancellationToken token)
|
||||
{
|
||||
if (run.Archive.Json("privapub/notification_policy.json") is not JsonObject json)
|
||||
return;
|
||||
var policy = await NotificationPolicies.Of(run.Me.Id, token);
|
||||
NotificationAction Action(string field, NotificationAction kept) => Enum.TryParse<NotificationAction>(Text(json[field]), out var action) ? action : kept;
|
||||
policy.ForNotFollowing = Action("forNotFollowing", policy.ForNotFollowing);
|
||||
policy.ForNotFollowers = Action("forNotFollowers", policy.ForNotFollowers);
|
||||
policy.ForNewAccounts = Action("forNewAccounts", policy.ForNewAccounts);
|
||||
policy.ForPrivateMentions = Action("forPrivateMentions", policy.ForPrivateMentions);
|
||||
policy.ForLimitedAccounts = Action("forLimitedAccounts", policy.ForLimitedAccounts);
|
||||
await NotificationPolicies.Save(policy, token);
|
||||
run.Count("notification policy");
|
||||
}
|
||||
|
||||
async Task Filters(Run run, CancellationToken token)
|
||||
{
|
||||
var titles = (await DB.Default.Find<PersonaFilter, string>().Match(f => f.AvatarId == run.Me.Id).Project(f => f.Title).ExecuteAsync(token)).ToHashSet();
|
||||
foreach (var filter in (run.Archive.Json("privapub/filters.json") as JsonArray ?? []).OfType<JsonObject>())
|
||||
{
|
||||
var title = Clean(Text(filter["title"]));
|
||||
if (title == default || titles.Contains(title))
|
||||
continue;
|
||||
var statuses = new List<FilterStatus>();
|
||||
foreach (var uri in Strings(filter["statuses"]))
|
||||
if (await Held(run, uri, token) is { } post)
|
||||
statuses.Add(new FilterStatus { Id = MongoDB.Bson.ObjectId.GenerateNewId().ToString(), PostId = post.ID });
|
||||
await DB.Default.SaveAsync(new PersonaFilter
|
||||
{
|
||||
AvatarId = run.Me.Id,
|
||||
Title = title,
|
||||
Context = Strings(filter["context"]).Where(c => c is "home" or "notifications" or "public" or "thread" or "account").ToList(),
|
||||
ExpiresAt = Date(filter["expiresAt"]),
|
||||
Action = Text(filter["action"]) is { } action && action is "warn" or "hide" or "blur" ? action : "warn",
|
||||
Keywords = (filter["keywords"] as JsonArray ?? []).OfType<JsonObject>().Where(k => Clean(Text(k["keyword"])) != default)
|
||||
.Select(k => new FilterKeyword { Id = MongoDB.Bson.ObjectId.GenerateNewId().ToString(), Keyword = Text(k["keyword"]).Trim(),
|
||||
WholeWord = k["wholeWord"]?.GetValueKind() == System.Text.Json.JsonValueKind.True }).ToList(),
|
||||
Statuses = statuses
|
||||
}, token);
|
||||
titles.Add(title);
|
||||
run.Count("filters");
|
||||
}
|
||||
}
|
||||
|
||||
// pinned again without announcing it (an Add would tell every follower's server)
|
||||
async Task Pins(Run run, CancellationToken token)
|
||||
{
|
||||
var pinned = (await DB.Default.Find<Pin, string>().Match(p => p.AvatarId == run.Me.Id).Project(p => p.PostId).ExecuteAsync(token)).ToHashSet();
|
||||
foreach (var uri in Strings(run.Archive.Json("privapub/pins.json")))
|
||||
{
|
||||
if (pinned.Count >= 5)
|
||||
break;
|
||||
if (!run.Posts.TryGetValue(uri, out var copy) || pinned.Contains(copy.Id))
|
||||
continue;
|
||||
await DB.Default.SaveAsync(new Pin { AvatarId = run.Me.Id, PostId = copy.Id }, token);
|
||||
pinned.Add(copy.Id);
|
||||
run.Count("pins");
|
||||
}
|
||||
}
|
||||
|
||||
// ---- relationships
|
||||
|
||||
async Task DomainBlocks(Run run, CancellationToken token)
|
||||
{
|
||||
var relationships = run.Services.GetRequiredService<IRelationshipService>();
|
||||
var mine = (await DB.Default.Find<AccountDomainBlock, string>().Match(d => d.AvatarId == run.Me.Id).Project(d => d.Domain).ExecuteAsync(token)).ToHashSet();
|
||||
foreach (var line in run.Archive.Lines("blocked_domains.csv"))
|
||||
{
|
||||
var domain = Cells(line).FirstOrDefault()?.Trim().ToLowerInvariant();
|
||||
if (string.IsNullOrEmpty(domain) || domain == "#domain" || Uri.CheckHostName(domain) != UriHostNameType.Dns || mine.Contains(domain))
|
||||
continue;
|
||||
await relationships.BlockDomain(run.Me, domain, token);
|
||||
mine.Add(domain);
|
||||
run.Count("blocked servers");
|
||||
}
|
||||
}
|
||||
|
||||
// one CSV of accounts, each resolved (WebFinger or its address) and acted on; a header line is skipped
|
||||
async Task Accounts(Run run, string file, string what, Func<Run, (LocalActor Local, ForeignAvatar Remote), string[], CancellationToken, Task<bool>> act,
|
||||
CancellationToken token)
|
||||
{
|
||||
var follows = run.Services.GetRequiredService<IFollowService>();
|
||||
var lines = run.Archive.Lines(file);
|
||||
foreach (var line in lines)
|
||||
{
|
||||
if (await Stopped(run, token))
|
||||
return;
|
||||
var cells = Cells(line);
|
||||
var address = cells.FirstOrDefault()?.Trim();
|
||||
if (string.IsNullOrEmpty(address) || address.Equals("Account address", StringComparison.OrdinalIgnoreCase))
|
||||
continue;
|
||||
var target = await follows.Resolve(address, token);
|
||||
if (target.Local == default && target.Remote == default)
|
||||
{
|
||||
run.Count($"{what} not found");
|
||||
continue;
|
||||
}
|
||||
if (target.Local?.Id == run.Me.Id)
|
||||
continue;
|
||||
run.Count(await act(run, target, cells, token) ? what : $"{what} skipped");
|
||||
}
|
||||
}
|
||||
|
||||
static async Task<bool> Block(Run run, (LocalActor Local, ForeignAvatar Remote) target, string[] cells, CancellationToken token)
|
||||
{
|
||||
var (uri, id) = target.Local != default ? (target.Local.Uri, target.Local.Id) : (target.Remote.ActorURI, target.Remote.ID);
|
||||
if (await DB.Default.Find<Block>().Match(b => b.AvatarId == run.Me.Id && b.TargetActorURI == uri).ExecuteAnyAsync(token))
|
||||
return false;
|
||||
await run.Services.GetRequiredService<IRelationshipService>().Block(run.Me, uri, id, token);
|
||||
return true;
|
||||
}
|
||||
|
||||
static async Task<bool> Mute(Run run, (LocalActor Local, ForeignAvatar Remote) target, string[] cells, CancellationToken token)
|
||||
{
|
||||
var (uri, id) = target.Local != default ? (target.Local.Uri, target.Local.Id) : (target.Remote.ActorURI, target.Remote.ID);
|
||||
var hide = cells.Length < 2 || !cells[1].Trim().Equals("false", StringComparison.OrdinalIgnoreCase);
|
||||
await run.Services.GetRequiredService<IRelationshipService>().Mute(run.Me, uri, id, hide, default, token);
|
||||
return true;
|
||||
}
|
||||
|
||||
async Task<bool> Follow(Run run, (LocalActor Local, ForeignAvatar Remote) target, string[] cells, CancellationToken token)
|
||||
{
|
||||
var uri = target.Local?.Uri ?? target.Remote.ActorURI;
|
||||
var host = new Uri(uri).Host;
|
||||
if (domainBlocks.IsSuspended(host)
|
||||
|| await DB.Default.Find<AccountDomainBlock>().Match(d => d.AvatarId == run.Me.Id && d.Domain == host).ExecuteAnyAsync(token)
|
||||
|| await DB.Default.Find<Following>().Match(f => f.AvatarId == run.Me.Id && f.TargetActorURI == uri).ExecuteAnyAsync(token))
|
||||
return false;
|
||||
var boosts = cells.Length < 2 || !cells[1].Trim().Equals("false", StringComparison.OrdinalIgnoreCase);
|
||||
return await run.Services.GetRequiredService<IFollowService>().FollowAs(run.Me, uri, boosts, token) != default;
|
||||
}
|
||||
|
||||
// lists by title, with only the accounts the persona follows
|
||||
async Task Lists(Run run, CancellationToken token)
|
||||
{
|
||||
var follows = run.Services.GetRequiredService<IFollowService>();
|
||||
var lists = (await DB.Default.Find<PersonaList>().Match(l => l.AvatarId == run.Me.Id).ExecuteAsync(token)).ToDictionary(l => l.Title, StringComparer.Ordinal);
|
||||
foreach (var line in run.Archive.Lines("lists.csv"))
|
||||
{
|
||||
var cells = Cells(line);
|
||||
if (cells.Length < 2 || Clean(cells[0]) is not { } title)
|
||||
continue;
|
||||
if (!lists.TryGetValue(title, out var list))
|
||||
{
|
||||
if (lists.Count >= 50)
|
||||
continue;
|
||||
list = new PersonaList { AvatarId = run.Me.Id, Title = title, RepliesPolicy = "list" };
|
||||
await DB.Default.SaveAsync(list, token);
|
||||
lists[title] = list;
|
||||
run.Count("lists");
|
||||
}
|
||||
var target = await follows.Resolve(cells[1].Trim(), token);
|
||||
var uri = target.Local?.Uri ?? target.Remote?.ActorURI;
|
||||
var following = uri == default ? default
|
||||
: await DB.Default.Find<Following>().Match(f => f.AvatarId == run.Me.Id && f.TargetActorURI == uri && f.State == FollowState.Accepted).ExecuteFirstAsync(token);
|
||||
if (following == default)
|
||||
{
|
||||
run.Count("list members not followed");
|
||||
continue;
|
||||
}
|
||||
if (await DB.Default.Find<PersonaListMember>().Match(m => m.ListId == list.ID && m.AccountId == following.TargetAccountId).ExecuteAnyAsync(token))
|
||||
continue;
|
||||
await DB.Default.SaveAsync(new PersonaListMember { ListId = list.ID, AvatarId = run.Me.Id, AccountId = following.TargetAccountId }, token);
|
||||
run.Count("list members");
|
||||
}
|
||||
}
|
||||
|
||||
async Task Bookmarks(Run run, CancellationToken token)
|
||||
{
|
||||
var uris = Strings(run.Archive.Json("bookmarks.json")?["orderedItems"]);
|
||||
if (uris.Count == 0)
|
||||
uris = run.Archive.Lines("bookmarks.csv").Select(l => Cells(l).FirstOrDefault()?.Trim()).Where(u => u?.StartsWith("http") == true).ToList();
|
||||
foreach (var uri in uris.Distinct())
|
||||
{
|
||||
if (await Stopped(run, token))
|
||||
return;
|
||||
var post = await Fetched(run, uri, token);
|
||||
if (post == default)
|
||||
{
|
||||
run.Count("bookmarks not found");
|
||||
continue;
|
||||
}
|
||||
if (await DB.Default.Find<Bookmark>().Match(b => b.AvatarId == run.Me.Id && b.PostId == post.ID).ExecuteAnyAsync(token))
|
||||
continue;
|
||||
await DB.Default.SaveAsync(new Bookmark { AvatarId = run.Me.Id, PostId = post.ID }, token);
|
||||
run.Count("bookmarks");
|
||||
}
|
||||
}
|
||||
|
||||
// asked for on purpose: each like tells the post's author
|
||||
async Task Likes(Run run, CancellationToken token)
|
||||
{
|
||||
var statuses = run.Services.GetRequiredService<IStatusService>();
|
||||
foreach (var uri in Strings(run.Archive.Json("likes.json")?["orderedItems"]).Distinct())
|
||||
{
|
||||
if (await Stopped(run, token))
|
||||
return;
|
||||
var post = await Fetched(run, uri, token);
|
||||
if (post == default)
|
||||
{
|
||||
run.Count("likes not found");
|
||||
continue;
|
||||
}
|
||||
run.Count((await statuses.Favourite(run.Me, post.ID, true, token)).Ok ? "likes" : "likes not found");
|
||||
}
|
||||
}
|
||||
|
||||
// a post here by its address, or read from its server (signed) and kept
|
||||
static async Task<PostEntity> Fetched(Run run, string uri, CancellationToken token)
|
||||
{
|
||||
if (await Held(run, uri, token) is { } held)
|
||||
return held;
|
||||
return await run.Services.GetRequiredService<IRemotePosts>().StoreContext(uri, 0, token);
|
||||
}
|
||||
|
||||
static async Task<PostEntity> Held(Run run, string uri, CancellationToken token)
|
||||
{
|
||||
if (uri == default)
|
||||
return default;
|
||||
if (run.Posts.TryGetValue(uri, out var copy))
|
||||
return await DB.Default.Find<PostEntity>().MatchID(copy.Id).ExecuteFirstAsync(token);
|
||||
return await DB.Default.Find<PostEntity>().Match(p => (p.ObjectURI == uri || p.Url == uri) && !p.DeletedAt.HasValue).ExecuteFirstAsync(token);
|
||||
}
|
||||
|
||||
// ---- the profile: name, words, fields, flags and pictures, saved as Settings saves them (and announced the same way)
|
||||
|
||||
async Task Profile(Run run, JsonObject actor, CancellationToken token)
|
||||
{
|
||||
var avatar = await DB.Default.Find<Avatar>().MatchID(run.Me.Id).ExecuteFirstAsync(token);
|
||||
if (Clean(Text(actor["name"])) is { } name)
|
||||
avatar.Name = name;
|
||||
if (Text(actor["summary"]) is { } summary)
|
||||
avatar.Biography = Plain(summary);
|
||||
var fields = (actor["attachment"] as JsonArray ?? []).OfType<JsonObject>().Where(f => Text(f["type"]) == "PropertyValue" && Clean(Text(f["name"])) != default)
|
||||
.Take(4).ToDictionary(f => Text(f["name"]).Trim(), f => Plain(Text(f["value"]) ?? string.Empty));
|
||||
if (fields.Count > 0)
|
||||
avatar.Fields = fields;
|
||||
if (actor["manuallyApprovesFollowers"]?.GetValueKind() is { } locked and (System.Text.Json.JsonValueKind.True or System.Text.Json.JsonValueKind.False))
|
||||
avatar.Settings.IsLocked = locked == System.Text.Json.JsonValueKind.True;
|
||||
if (actor["discoverable"]?.GetValueKind() is { } discoverable and (System.Text.Json.JsonValueKind.True or System.Text.Json.JsonValueKind.False))
|
||||
avatar.Settings.IsDiscoverable = discoverable == System.Text.Json.JsonValueKind.True;
|
||||
if (actor["indexable"]?.GetValueKind() is { } indexable and (System.Text.Json.JsonValueKind.True or System.Text.Json.JsonValueKind.False))
|
||||
avatar.Settings.IsIndexable = indexable == System.Text.Json.JsonValueKind.True;
|
||||
foreach (var (field, kind, width, height) in new[] { ("icon", "avatar", 400, 400), ("image", "header", 1500, 500) })
|
||||
{
|
||||
var file = (Text((actor[field] as JsonObject)?["url"]) ?? Text(actor[field]))?.TrimStart('/');
|
||||
if (file == default || file.Contains("://") || !run.Archive.Has(file))
|
||||
continue;
|
||||
var outcome = await Picture(run, file, kind, width, height, token);
|
||||
if (outcome?.Ok != true)
|
||||
{
|
||||
run.Count("pictures refused");
|
||||
continue;
|
||||
}
|
||||
if (kind == "avatar")
|
||||
avatar.PictureURL = media.Url(outcome.Attachment.FilePath);
|
||||
else
|
||||
avatar.ThumbnailURL = media.Url(outcome.Attachment.FilePath);
|
||||
await media.Trash(m => m.ProfileOfAvatarId == avatar.ID && m.Kind == kind && m.ID != outcome.Attachment.ID, "replaced", token);
|
||||
}
|
||||
avatar.UpdatedAt = DateTime.UtcNow;
|
||||
await DB.Default.SaveAsync(avatar, token);
|
||||
await run.Services.GetRequiredService<IOutboxPublisher>().PublishProfile(localActors.FromAvatar(avatar), token);
|
||||
run.Count("profile");
|
||||
}
|
||||
|
||||
async Task<MediaOutcome> Picture(Run run, string file, string kind, int width, int height, CancellationToken token)
|
||||
{
|
||||
var temporary = Path.Combine(Path.GetTempPath(), $"privapub-import-{Guid.NewGuid():N}{Path.GetExtension(file)}");
|
||||
try
|
||||
{
|
||||
await using (var source = run.Archive.Read(file))
|
||||
await using (var target = new FileStream(temporary, FileMode.CreateNew, FileAccess.Write))
|
||||
await source.CopyToAsync(target, token);
|
||||
await using var stream = new FileStream(temporary, FileMode.Open, FileAccess.Read);
|
||||
var form = new FormFile(stream, 0, stream.Length, kind, Path.GetFileName(file))
|
||||
{
|
||||
Headers = new HeaderDictionary(),
|
||||
ContentType = MediaService.ContentTypes.TryGetContentType(file, out var guessed) ? guessed : "application/octet-stream"
|
||||
};
|
||||
return await media.ProfileImage(run.Me.Id, kind, form, width, height, token);
|
||||
}
|
||||
finally
|
||||
{
|
||||
File.Delete(temporary);
|
||||
}
|
||||
}
|
||||
|
||||
// ---- helpers
|
||||
|
||||
static string Text(JsonNode node) => node is JsonValue value && value.TryGetValue<string>(out var text) ? text : default;
|
||||
|
||||
static string Id(JsonNode node) => node switch
|
||||
{
|
||||
JsonValue => Text(node),
|
||||
JsonObject json => Text(json["id"]),
|
||||
JsonArray list => list.Select(Id).FirstOrDefault(i => i != default),
|
||||
_ => default
|
||||
};
|
||||
|
||||
static List<string> Strings(JsonNode node) => node switch
|
||||
{
|
||||
JsonArray list => list.Select(Id).Where(i => i != default).ToList(),
|
||||
JsonValue => Text(node) is { } one ? [one] : [],
|
||||
JsonObject json when Text(json["type"]) is "OrderedCollection" or "Collection" => Strings(json["orderedItems"] ?? json["items"]),
|
||||
_ => []
|
||||
};
|
||||
|
||||
static DateTime? Date(JsonNode node) =>
|
||||
DateTime.TryParse(Text(node), System.Globalization.CultureInfo.InvariantCulture,
|
||||
System.Globalization.DateTimeStyles.AdjustToUniversal | System.Globalization.DateTimeStyles.AssumeUniversal, out var date) ? date : default;
|
||||
|
||||
static DateTime Clamp(DateTime when) => when > DateTime.UtcNow ? DateTime.UtcNow : when;
|
||||
|
||||
static string Clean(string text) => string.IsNullOrWhiteSpace(text) ? default : text.Trim();
|
||||
|
||||
static bool IsObjectId(string id) => id is { Length: 24 } && id.All(char.IsAsciiHexDigitLower);
|
||||
|
||||
// a CSV line's cells (quotes as Mastodon writes them)
|
||||
static string[] Cells(string line)
|
||||
{
|
||||
var cells = new List<string>();
|
||||
var cell = new System.Text.StringBuilder();
|
||||
var quoted = false;
|
||||
for (var i = 0; i < line.Length; i++)
|
||||
{
|
||||
var c = line[i];
|
||||
if (quoted)
|
||||
{
|
||||
if (c == '"' && i + 1 < line.Length && line[i + 1] == '"')
|
||||
{
|
||||
cell.Append('"');
|
||||
i++;
|
||||
}
|
||||
else if (c == '"')
|
||||
quoted = false;
|
||||
else
|
||||
cell.Append(c);
|
||||
}
|
||||
else if (c == '"')
|
||||
quoted = true;
|
||||
else if (c == ',')
|
||||
{
|
||||
cells.Add(cell.ToString());
|
||||
cell.Clear();
|
||||
}
|
||||
else
|
||||
cell.Append(c);
|
||||
}
|
||||
cells.Add(cell.ToString());
|
||||
return [.. cells];
|
||||
}
|
||||
|
||||
// HTML to the plain words a persona writes in its profile
|
||||
static string Plain(string html)
|
||||
{
|
||||
var text = Breaks().Replace(html, "\n");
|
||||
text = Tags().Replace(text, string.Empty);
|
||||
return WebUtility.HtmlDecode(text).Trim();
|
||||
}
|
||||
|
||||
[GeneratedRegex(@"<br\s*/?>|</p>\s*<p[^>]*>", RegexOptions.IgnoreCase)]
|
||||
private static partial Regex Breaks();
|
||||
|
||||
[GeneratedRegex(@"<[^>]+>")]
|
||||
private static partial Regex Tags();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
using MongoDB.Entities;
|
||||
|
||||
using PrivaPub.ClientModels.User;
|
||||
|
||||
namespace PrivaPub.Domain.Portability
|
||||
{
|
||||
// A persona's archive, made and restored in jobs (owner decision 2026-10-07): where each stands, one row per persona
|
||||
// (its ID is the persona's). The files live in <backups>/.personas/<avatarId>/: the export, kept a week, and the upload
|
||||
// being imported.
|
||||
public class PersonaArchive : Entity
|
||||
{
|
||||
public string ExportState { get; set; } = "none";
|
||||
public DateTime? ExportAskedAt { get; set; }
|
||||
public DateTime? ExportReadyAt { get; set; }
|
||||
public string ExportFile { get; set; }
|
||||
public long ExportBytes { get; set; }
|
||||
public string ExportError { get; set; }
|
||||
|
||||
public string ImportState { get; set; } = "none";
|
||||
public DateTime? ImportAskedAt { get; set; }
|
||||
public DateTime? ImportEndedAt { get; set; }
|
||||
public string ImportFile { get; set; }
|
||||
public List<string> ImportParts { get; set; } = [];
|
||||
public long ImportDone { get; set; }
|
||||
public long ImportTotal { get; set; }
|
||||
public Dictionary<string, int> ImportCounts { get; set; } = [];
|
||||
public string ImportError { get; set; }
|
||||
public string ImportFrom { get; set; }
|
||||
public bool ImportSameActor { get; set; }
|
||||
public bool StopAsked { get; set; }
|
||||
}
|
||||
|
||||
public static class PersonaArchives
|
||||
{
|
||||
public static readonly TimeSpan Kept = TimeSpan.FromDays(7);
|
||||
|
||||
/// <summary>Every part an import can take; the last three only when asked (they act, or tell others).</summary>
|
||||
public static readonly string[] Parts = ["posts", "profile", "follows", "blocks", "mutes", "domainblocks", "lists", "bookmarks", "filters", "tags",
|
||||
"notificationpolicy", "pins", "located", "scheduled", "likes"];
|
||||
|
||||
public static string Folder(string backupsRoot, string avatarId) => Path.Combine(backupsRoot, ".personas", avatarId);
|
||||
|
||||
public static async Task<PersonaArchive> Of(string avatarId, CancellationToken token) =>
|
||||
await DB.Default.Find<PersonaArchive>().MatchID(avatarId).ExecuteFirstAsync(token) ?? new PersonaArchive { ID = avatarId };
|
||||
|
||||
public static ViewPersonaArchive View(PersonaArchive archive) => new()
|
||||
{
|
||||
ExportState = archive.ExportState,
|
||||
ExportAskedAt = archive.ExportAskedAt,
|
||||
ExportReadyAt = archive.ExportReadyAt,
|
||||
ExportExpiresAt = archive.ExportReadyAt + Kept,
|
||||
ExportBytes = archive.ExportBytes,
|
||||
ExportError = archive.ExportError,
|
||||
ImportState = archive.ImportState,
|
||||
ImportAskedAt = archive.ImportAskedAt,
|
||||
ImportEndedAt = archive.ImportEndedAt,
|
||||
ImportParts = archive.ImportParts,
|
||||
ImportDone = archive.ImportDone,
|
||||
ImportTotal = archive.ImportTotal,
|
||||
ImportCounts = archive.ImportCounts,
|
||||
ImportError = archive.ImportError,
|
||||
ImportFrom = archive.ImportFrom,
|
||||
ImportSameActor = archive.ImportSameActor
|
||||
};
|
||||
|
||||
/// <summary>Exports older than a week go, as do imports' files left behind.</summary>
|
||||
public static async Task Forget(string backupsRoot, CancellationToken token)
|
||||
{
|
||||
var folder = Path.Combine(backupsRoot, ".personas");
|
||||
if (!Directory.Exists(folder))
|
||||
return;
|
||||
var before = DateTime.UtcNow - Kept;
|
||||
foreach (var file in Directory.EnumerateFiles(folder, "*", SearchOption.AllDirectories).Where(f => File.GetLastWriteTimeUtc(f) < before))
|
||||
File.Delete(file);
|
||||
await DB.Default.Update<PersonaArchive>().Match(a => a.ExportState == "ready" && a.ExportReadyAt < before)
|
||||
.Modify(a => a.ExportState, "none").Modify(a => a.ExportFile, null).ExecuteAsync(token);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,270 @@
|
||||
using System.IO.Compression;
|
||||
using System.Text.Json;
|
||||
using System.Text.Json.Nodes;
|
||||
|
||||
namespace PrivaPub.Domain.Portability
|
||||
{
|
||||
// An archive someone uploads, opened only if nothing in it can harm the server: no link, no path leaving it (.. or
|
||||
// absolute), no name twice, at most MaxEntries files and MaxBytes in all (and what the disk has), and no file
|
||||
// compressed more than MaxRatio to one. Every file is read through a stream that stops at its stated size, and the
|
||||
// outbox one item at a time, none larger than MaxItemBytes, so a large archive never sits in memory.
|
||||
public sealed class SafeArchive : IDisposable
|
||||
{
|
||||
public const int MaxEntries = 200_000;
|
||||
public const long MaxBytes = 64L * 1024 * 1024 * 1024;
|
||||
public const int MaxRatio = 100;
|
||||
public const int MaxItemBytes = 1024 * 1024;
|
||||
public const int MaxJsonBytes = 64 * 1024 * 1024;
|
||||
|
||||
readonly ZipArchive _zip;
|
||||
readonly Dictionary<string, ZipArchiveEntry> _entries;
|
||||
readonly string _root;//the folder the archive's files sit in, when it wraps them in one
|
||||
|
||||
SafeArchive(ZipArchive zip, Dictionary<string, ZipArchiveEntry> entries, string root)
|
||||
{
|
||||
_zip = zip;
|
||||
_entries = entries;
|
||||
_root = root;
|
||||
}
|
||||
|
||||
public static (SafeArchive Archive, string Error) Open(string path)
|
||||
{
|
||||
ZipArchive zip;
|
||||
try
|
||||
{
|
||||
zip = ZipFile.OpenRead(path);
|
||||
}
|
||||
catch (InvalidDataException)
|
||||
{
|
||||
return (default, "not a zip archive");
|
||||
}
|
||||
var entries = new Dictionary<string, ZipArchiveEntry>(StringComparer.Ordinal);
|
||||
var total = 0L;
|
||||
var free = new DriveInfo(Path.GetFullPath(path)).AvailableFreeSpace;
|
||||
string Refuse(string why)
|
||||
{
|
||||
zip.Dispose();
|
||||
return why;
|
||||
}
|
||||
if (zip.Entries.Count > MaxEntries)
|
||||
return (default, Refuse("too many files"));
|
||||
foreach (var entry in zip.Entries)
|
||||
{
|
||||
var name = entry.FullName;
|
||||
if (name.Contains('\\') || name.StartsWith('/') || name.Contains(':') || name.Split('/').Any(part => part is ".." or "."))
|
||||
return (default, Refuse($"{name}: a path that leaves the archive"));
|
||||
if (((entry.ExternalAttributes >> 16) & 0xF000) == 0xA000)
|
||||
return (default, Refuse($"{name}: a link"));
|
||||
if (name.EndsWith('/'))
|
||||
continue;
|
||||
if (!entries.TryAdd(name, entry))
|
||||
return (default, Refuse($"{name}: twice"));
|
||||
total += entry.Length;
|
||||
if (total > MaxBytes || total > free)
|
||||
return (default, Refuse("larger than the server can take"));
|
||||
if (entry.Length > 1024 * 1024 && entry.Length > (long)MaxRatio * Math.Max(1, entry.CompressedLength))
|
||||
return (default, Refuse($"{name}: compressed more than {MaxRatio} to one"));
|
||||
}
|
||||
// some archivers wrap everything in one folder
|
||||
var root = string.Empty;
|
||||
if (!entries.ContainsKey("actor.json") && entries.Keys.FirstOrDefault(k => k.EndsWith("/actor.json", StringComparison.Ordinal)) is { } nested
|
||||
&& nested.Count(c => c == '/') == 1)
|
||||
root = nested[..(nested.IndexOf('/') + 1)];
|
||||
return (new SafeArchive(zip, entries, root), default);
|
||||
}
|
||||
|
||||
public bool Has(string name) => _entries.ContainsKey(_root + name);
|
||||
|
||||
public long Length(string name) => _entries.TryGetValue(_root + name, out var entry) ? entry.Length : -1;
|
||||
|
||||
/// <summary>A file's contents, never more than its stated size.</summary>
|
||||
public Stream Read(string name) =>
|
||||
_entries.TryGetValue(_root + name, out var entry) ? new Bounded(entry.Open(), entry.Length) : default;
|
||||
|
||||
/// <summary>A JSON file, or null when it is missing, too large or not JSON.</summary>
|
||||
public JsonNode Json(string name)
|
||||
{
|
||||
if (!_entries.TryGetValue(_root + name, out var entry) || entry.Length > MaxJsonBytes)
|
||||
return default;
|
||||
try
|
||||
{
|
||||
using var stream = Read(name);
|
||||
return JsonNode.Parse(stream);
|
||||
}
|
||||
catch (JsonException)
|
||||
{
|
||||
return default;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>The lines of a text file (a CSV), none when it is missing or too large.</summary>
|
||||
public List<string> Lines(string name)
|
||||
{
|
||||
if (!_entries.TryGetValue(_root + name, out var entry) || entry.Length > MaxJsonBytes)
|
||||
return [];
|
||||
using var reader = new StreamReader(Read(name));
|
||||
var lines = new List<string>();
|
||||
while (reader.ReadLine() is { } line)
|
||||
if (line.Length > 0)
|
||||
lines.Add(line);
|
||||
return lines;
|
||||
}
|
||||
|
||||
/// <summary>outbox.json's orderedItems, one object at a time; an item larger than MaxItemBytes stops it.</summary>
|
||||
public async IAsyncEnumerable<JsonObject> Outbox([System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken token)
|
||||
{
|
||||
await using var stream = Read("outbox.json");
|
||||
if (stream == default)
|
||||
yield break;
|
||||
var buffer = new byte[64 * 1024];
|
||||
var filled = 0;
|
||||
var final = false;
|
||||
var state = new JsonReaderState();
|
||||
var phase = Phase.Seeking;
|
||||
while (phase != Phase.Done)
|
||||
{
|
||||
if (!final)
|
||||
{
|
||||
if (filled == buffer.Length)
|
||||
{
|
||||
if (buffer.Length > MaxItemBytes + 64 * 1024)
|
||||
throw new InvalidDataException($"an outbox item larger than {MaxItemBytes / 1024} KiB");
|
||||
Array.Resize(ref buffer, buffer.Length * 2);
|
||||
}
|
||||
var read = await stream.ReadAsync(buffer.AsMemory(filled), token);
|
||||
filled += read;
|
||||
final = read == 0;
|
||||
}
|
||||
var (items, consumed, next, after) = Scan(buffer.AsSpan(0, filled), final, state, phase);
|
||||
foreach (var item in items)
|
||||
yield return item;
|
||||
(state, phase) = (next, after);
|
||||
Buffer.BlockCopy(buffer, consumed, buffer, 0, filled - consumed);
|
||||
filled -= consumed;
|
||||
if (final && consumed == 0 && items.Count == 0)
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
enum Phase { Seeking, Array, Done }
|
||||
|
||||
static (List<JsonObject> Items, int Consumed, JsonReaderState State, Phase Phase) Scan(ReadOnlySpan<byte> data, bool final, JsonReaderState state, Phase phase)
|
||||
{
|
||||
var items = new List<JsonObject>();
|
||||
var reader = new Utf8JsonReader(data, final, state);
|
||||
while (phase != Phase.Done)
|
||||
{
|
||||
var mark = reader;
|
||||
if (!reader.Read())
|
||||
{
|
||||
reader = mark;
|
||||
break;
|
||||
}
|
||||
if (phase == Phase.Seeking)
|
||||
{
|
||||
if (reader.TokenType == JsonTokenType.PropertyName && reader.CurrentDepth == 1 && reader.ValueTextEquals("orderedItems"))
|
||||
{
|
||||
var before = reader;
|
||||
if (!reader.Read())
|
||||
{
|
||||
reader = mark;
|
||||
break;
|
||||
}
|
||||
if (reader.TokenType == JsonTokenType.StartArray)
|
||||
phase = Phase.Array;
|
||||
else
|
||||
{
|
||||
reader = before;
|
||||
if (!reader.TrySkip())
|
||||
{
|
||||
reader = mark;
|
||||
break;
|
||||
}
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (reader.TokenType is JsonTokenType.StartObject or JsonTokenType.StartArray && reader.CurrentDepth >= 1 && !reader.TrySkip())
|
||||
{
|
||||
reader = mark;
|
||||
break;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (reader.TokenType == JsonTokenType.EndArray)
|
||||
{
|
||||
phase = Phase.Done;
|
||||
break;
|
||||
}
|
||||
if (reader.TokenType != JsonTokenType.StartObject)
|
||||
{
|
||||
if (reader.TokenType == JsonTokenType.StartArray && !reader.TrySkip())
|
||||
{
|
||||
reader = mark;
|
||||
break;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
var start = (int)reader.TokenStartIndex;
|
||||
if (!reader.TrySkip())
|
||||
{
|
||||
reader = mark;
|
||||
break;
|
||||
}
|
||||
var length = (int)reader.BytesConsumed - start;
|
||||
if (length > MaxItemBytes)
|
||||
throw new InvalidDataException($"an outbox item larger than {MaxItemBytes / 1024} KiB");
|
||||
if (JsonNode.Parse(data.Slice(start, length)) is JsonObject item)
|
||||
items.Add(item);
|
||||
}
|
||||
return (items, (int)reader.BytesConsumed, reader.CurrentState, phase);
|
||||
}
|
||||
|
||||
public void Dispose() => _zip.Dispose();
|
||||
|
||||
// a stream that ends at the size the archive states, whatever the compressed data says
|
||||
sealed class Bounded(Stream inner, long length) : Stream
|
||||
{
|
||||
long _left = length;
|
||||
|
||||
public override int Read(byte[] buffer, int offset, int count) => Read(buffer.AsSpan(offset, count));
|
||||
|
||||
public override int Read(Span<byte> buffer)
|
||||
{
|
||||
if (_left <= 0)
|
||||
return 0;
|
||||
var read = inner.Read(buffer[..(int)Math.Min(buffer.Length, _left)]);
|
||||
_left -= read;
|
||||
return read;
|
||||
}
|
||||
|
||||
public override async ValueTask<int> ReadAsync(Memory<byte> buffer, CancellationToken token = default)
|
||||
{
|
||||
if (_left <= 0)
|
||||
return 0;
|
||||
var read = await inner.ReadAsync(buffer[..(int)Math.Min(buffer.Length, _left)], token);
|
||||
_left -= read;
|
||||
return read;
|
||||
}
|
||||
|
||||
public override Task<int> ReadAsync(byte[] buffer, int offset, int count, CancellationToken token) =>
|
||||
ReadAsync(buffer.AsMemory(offset, count), token).AsTask();
|
||||
|
||||
public override bool CanRead => true;
|
||||
public override bool CanSeek => false;
|
||||
public override bool CanWrite => false;
|
||||
public override long Length => length;
|
||||
public override long Position { get => length - _left; set => throw new NotSupportedException(); }
|
||||
public override void Flush() { }
|
||||
public override long Seek(long offset, SeekOrigin origin) => throw new NotSupportedException();
|
||||
public override void SetLength(long value) => throw new NotSupportedException();
|
||||
public override void Write(byte[] buffer, int offset, int count) => throw new NotSupportedException();
|
||||
|
||||
protected override void Dispose(bool disposing)
|
||||
{
|
||||
if (disposing)
|
||||
inner.Dispose();
|
||||
base.Dispose(disposing);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,3 +1,4 @@
|
||||
using MongoDB.Driver;
|
||||
using MongoDB.Entities;
|
||||
|
||||
using PrivaPub.Federation.Actors;
|
||||
@@ -64,6 +65,24 @@ namespace PrivaPub.Domain.Privacy
|
||||
return distinct.Count == 0 ? 0 : await DB.Default.CountAsync<Avatar>(a => distinct.Contains(a.ID) && !a.DeletionAt.HasValue, token);
|
||||
}
|
||||
|
||||
// the media kept (not trashed) that which holds: their bytes and how many
|
||||
public static async Task<(long Bytes, long Files)> Media(Expression<Func<Models.Media.MediaAttachment, bool>> which, CancellationToken token)
|
||||
{
|
||||
var totals = await DB.Default.Fluent<Models.Media.MediaAttachment>()
|
||||
.Match(m => m.TrashedAt == null)
|
||||
.Match(which)
|
||||
.Group(m => true, g => new { Bytes = g.Sum(m => m.Size), Files = g.LongCount() })
|
||||
.FirstOrDefaultAsync(token);
|
||||
return totals == default ? (0, 0) : (totals.Bytes, totals.Files);
|
||||
}
|
||||
|
||||
// a login's media: every persona's uploads and pictures
|
||||
public static async Task<(long Bytes, long Files)> MediaOfRoot(string rootId, CancellationToken token)
|
||||
{
|
||||
var personas = (await DB.Default.Find<Models.User.RootToAvatar>().Match(r => r.RootId == rootId).ExecuteAsync(token)).Select(r => r.AvatarId).ToList();
|
||||
return await Media(m => personas.Contains(m.OwnerAvatarId), token);
|
||||
}
|
||||
|
||||
public static async Task<long> LocalPosts(CancellationToken token)
|
||||
{
|
||||
var barred = await BarredPersonas(token);
|
||||
|
||||
@@ -30,6 +30,8 @@ namespace PrivaPub.Domain.Social
|
||||
Task UnfollowAs(LocalActor follower, string target, CancellationToken token);
|
||||
Task<bool> Decide(LocalActor me, string followerAccountId, bool accept, CancellationToken token);
|
||||
Task<int> ResendPending(DateTime now, CancellationToken token);
|
||||
/// <summary>An account named by its address or its handle (user@host): a persona here, or one elsewhere.</summary>
|
||||
Task<(LocalActor Local, ForeignAvatar Remote)> Resolve(string target, CancellationToken token);
|
||||
}
|
||||
|
||||
public class FollowService : IFollowService
|
||||
@@ -257,6 +259,8 @@ namespace PrivaPub.Domain.Social
|
||||
follower.Id, follower.Uri, default, token);
|
||||
}
|
||||
|
||||
public Task<(LocalActor Local, ForeignAvatar Remote)> Resolve(string target, CancellationToken token) => ResolveTarget(target, token);
|
||||
|
||||
async Task<(LocalActor Local, ForeignAvatar Remote)> ResolveTarget(string target, CancellationToken token)
|
||||
{
|
||||
target = target?.Trim();
|
||||
|
||||
@@ -74,10 +74,18 @@ namespace PrivaPub.Domain.Statistics
|
||||
Written = server.Sum(d => d.LedgerWritten),
|
||||
Dropped = server.Sum(d => d.LedgerDropped),
|
||||
Failed = server.Sum(d => d.LedgerFailed)
|
||||
}
|
||||
},
|
||||
Media = await MediaTotals(token)
|
||||
};
|
||||
}
|
||||
|
||||
// the media rows kept; the disk's caches are added by whoever knows where they are
|
||||
static async Task<ViewMediaTotals> MediaTotals(CancellationToken token)
|
||||
{
|
||||
var (bytes, files) = await Domain.Privacy.Counted.Media(m => true, token);
|
||||
return new ViewMediaTotals { MediaBytes = bytes, MediaFiles = files };
|
||||
}
|
||||
|
||||
public async Task<ViewHostsPage> Hosts(int days, string sort, string software, int page, int limit, CancellationToken token)
|
||||
{
|
||||
var hostDays = await Days(days, default, token);
|
||||
|
||||
@@ -7,6 +7,8 @@ using PrivaPub.Models.Jobs;
|
||||
using PrivaPub.Models.Media;
|
||||
using PrivaPub.Models.Post;
|
||||
using PrivaPub.Models.Social;
|
||||
using PrivaPub.Models.User;
|
||||
using PrivaPub.Domain.Media;
|
||||
|
||||
namespace PrivaPub.Domain.Statuses
|
||||
{
|
||||
@@ -19,8 +21,9 @@ namespace PrivaPub.Domain.Statuses
|
||||
public const int MaxWaiting = 300;
|
||||
public const int MaxPerDay = 25;
|
||||
|
||||
public static StatusDraft Draft(ScheduledParams p) => new()
|
||||
public static StatusDraft Draft(ScheduledParams p, string scheduledId = default) => new()
|
||||
{
|
||||
ScheduledStatusId = scheduledId,
|
||||
Text = p.Text,
|
||||
PlainText = true,
|
||||
SpoilerText = p.SpoilerText,
|
||||
@@ -66,6 +69,15 @@ namespace PrivaPub.Domain.Statuses
|
||||
RunAt = scheduled.ScheduledAt
|
||||
};
|
||||
|
||||
// whether a persona may still publish: neither deleted nor its root deleted or banned
|
||||
public static async Task<bool> MayPublish(string avatarId, CancellationToken token)
|
||||
{
|
||||
if (!await DB.Default.Find<Avatar>().Match(a => a.ID == avatarId && !a.DeletionAt.HasValue).ExecuteAnyAsync(token))
|
||||
return false;
|
||||
var rootIds = (await DB.Default.Find<RootToAvatar>().Match(r => r.AvatarId == avatarId).ExecuteAsync(token)).Select(r => r.RootId).ToList();
|
||||
return await DB.Default.Find<RootUser>().Match(u => rootIds.Contains(u.ID) && u.DeletedAt == null && !u.IsBanned).ExecuteAnyAsync(token);
|
||||
}
|
||||
|
||||
// the post's media go back to being plain uploads (the janitor's after a day), or are attached to the post by now
|
||||
public static Task Release(string scheduledId, CancellationToken token) =>
|
||||
DB.Default.Update<MediaAttachment>().Match(m => m.ScheduledStatusId == scheduledId).Modify(m => m.ScheduledStatusId, null).ExecuteAsync(token);
|
||||
@@ -93,10 +105,17 @@ namespace PrivaPub.Domain.Statuses
|
||||
if (scheduled == default || scheduled.ScheduledAt > DateTime.UtcNow.AddSeconds(30))
|
||||
return JobOutcome.Done;//dropped, or moved to a later time that has its own job
|
||||
using var scope = _scopes.CreateScope();
|
||||
// a persona deleted or banned since it was scheduled publishes nothing, and what it held is trashed
|
||||
if (!await ScheduledStatuses.MayPublish(scheduled.AvatarId, token))
|
||||
{
|
||||
await scope.ServiceProvider.GetRequiredService<IMediaService>().Trash(m => m.ScheduledStatusId == scheduled.ID, "its author is gone", token);
|
||||
await DB.Default.DeleteAsync<ScheduledStatus>(scheduled.ID);
|
||||
return JobOutcome.Done;
|
||||
}
|
||||
var author = await scope.ServiceProvider.GetRequiredService<ILocalActorService>().FindById(LocalActorKind.Person, scheduled.AvatarId, token);
|
||||
if (author != default)
|
||||
{
|
||||
var outcome = await scope.ServiceProvider.GetRequiredService<IStatusService>().Publish(author, ScheduledStatuses.Draft(scheduled.Params), token);
|
||||
var outcome = await scope.ServiceProvider.GetRequiredService<IStatusService>().Publish(author, ScheduledStatuses.Draft(scheduled.Params, scheduled.ID), token);
|
||||
if (!outcome.Ok)
|
||||
_logger.LogWarning("Scheduled status {Id} of {Avatar} could not be published: {Error}", scheduled.ID, scheduled.AvatarId, outcome.Error);
|
||||
}
|
||||
|
||||
@@ -24,6 +24,9 @@ using PostEntity = PrivaPub.Models.Post.Post;
|
||||
|
||||
namespace PrivaPub.Domain.Statuses
|
||||
{
|
||||
// an attachment's description or focal point changed by an edit (Mastodon's media_attributes)
|
||||
public sealed record MediaChange(string Id, bool HasDescription, string Description, string Focus);
|
||||
|
||||
public sealed class StatusDraft
|
||||
{
|
||||
public string Text { get; init; }
|
||||
@@ -38,6 +41,9 @@ namespace PrivaPub.Domain.Statuses
|
||||
public string ConversationId { get; init; }
|
||||
public IReadOnlyList<string> Recipients { get; init; } = Array.Empty<string>();
|
||||
public IReadOnlyList<string> MediaIds { get; init; }
|
||||
public IReadOnlyList<MediaChange> MediaChanges { get; init; } = Array.Empty<MediaChange>();
|
||||
// the scheduled post being published, whose media are reserved for it alone
|
||||
public string ScheduledStatusId { get; init; }
|
||||
public double? Latitude { get; init; }
|
||||
public double? Longitude { get; init; }
|
||||
public double? RangeKm { get; init; }
|
||||
@@ -109,7 +115,7 @@ namespace PrivaPub.Domain.Statuses
|
||||
{
|
||||
if (TooLong(draft) is { } tooLong)
|
||||
return tooLong;
|
||||
var media = await Media(author, draft.MediaIds, default, token);
|
||||
var media = await Media(author, draft.MediaIds, default, draft.ScheduledStatusId, token);
|
||||
if (media == default)
|
||||
return StatusOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: Media attachments are invalid");
|
||||
if (string.IsNullOrWhiteSpace(draft.Text) && media.Count == 0 && draft.Poll == default)
|
||||
@@ -225,6 +231,9 @@ namespace PrivaPub.Domain.Statuses
|
||||
post.ID = (string)post.GenerateNewID();
|
||||
post.ObjectURI = author.PostUri(post.ID);
|
||||
post.Url = author.PostHtmlUrl(post.ID);
|
||||
// the media are this post's before anything is written: another post claiming them meanwhile wins, and this one is refused
|
||||
if (!await Claim(media, post.ID, draft.ScheduledStatusId, token))
|
||||
return StatusOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: Media attachments are invalid");
|
||||
post.LocalQuotePolicy = QuotePolicies.IsKnown(draft.QuotePolicy) ? draft.QuotePolicy : author.Settings.QuotePolicy ?? QuotePolicies.Public;
|
||||
if (quoted is { IsFederatedCopy: false } && post.QuoteState == QuoteState.Accepted
|
||||
&& await _localActors.FindById(LocalActorKind.Person, quoted.GroupUserId, token) is { } quotedAuthor)
|
||||
@@ -270,7 +279,6 @@ namespace PrivaPub.Domain.Statuses
|
||||
post.ActivityURI = create?["id"]?.GetValue<string>();
|
||||
|
||||
await DB.Default.SaveAsync(post, token);
|
||||
await Attach(media, post.ID, token);
|
||||
if (parent != default && Counted.Reply(post))
|
||||
await DB.Default.Update<PostEntity>().MatchID(parent.ID).Modify(b => b.Inc(p => p.RepliesCount, 1)).ExecuteAsync(token);
|
||||
await _fanout.Distribute(post, token);
|
||||
@@ -297,7 +305,7 @@ namespace PrivaPub.Domain.Statuses
|
||||
return StatusOutcome.Fail(StatusCodes.Status404NotFound, "Record not found");
|
||||
if (TooLong(draft) is { } tooLong)
|
||||
return tooLong;
|
||||
var media = draft.MediaIds == default ? default : await Media(author, draft.MediaIds, post.ID, token);
|
||||
var media = draft.MediaIds == default ? default : await Media(author, draft.MediaIds, post.ID, default, token);
|
||||
if (draft.MediaIds != default && media == default)
|
||||
return StatusOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: Media attachments are invalid");
|
||||
if (string.IsNullOrWhiteSpace(draft.Text) && (media ?? new List<MediaAttachment>()).Count == 0 && post.Media.Count == 0)
|
||||
@@ -316,12 +324,34 @@ namespace PrivaPub.Domain.Statuses
|
||||
|
||||
var plain = draft.PlainText || post.ContentFormat == ContentFormat.Plain;
|
||||
var rendered = plain ? await _content.PlainText(draft.Text ?? string.Empty, token) : await _content.Markdown(draft.Text ?? string.Empty, token);
|
||||
// descriptions and focal points changed by the edit, on the media it keeps or those the post already has
|
||||
var described = media ?? (draft.MediaChanges.Count > 0
|
||||
? await DB.Default.Find<MediaAttachment>().Match(m => m.PostId == post.ID && m.TrashedAt == null).ExecuteAsync(token)
|
||||
: default);
|
||||
foreach (var change in draft.MediaChanges)
|
||||
{
|
||||
if (described?.FirstOrDefault(m => m.ID == change.Id) is not { } attachment)
|
||||
continue;
|
||||
if (change.HasDescription)
|
||||
attachment.Description = string.IsNullOrWhiteSpace(change.Description) ? default : change.Description.Trim()[..Math.Min(change.Description.Trim().Length, 1500)];
|
||||
if (Domain.Media.FocalPoint.Parse(change.Focus) is { } focus)
|
||||
attachment.Focus = focus;
|
||||
await DB.Default.Update<MediaAttachment>().MatchID(attachment.ID)
|
||||
.Modify(m => m.Description, attachment.Description).Modify(m => m.Focus, attachment.Focus).ExecuteAsync(token);
|
||||
}
|
||||
if (media == default && described != default)
|
||||
post.Media = post.Media.Select(copy => described.FirstOrDefault(m => m.ID == copy.AttachmentId) is { } changed ? ToPostMedia(changed) : copy).ToList();
|
||||
if (media != default)
|
||||
{
|
||||
if (!await Claim(media, post.ID, default, token))
|
||||
return StatusOutcome.Fail(StatusCodes.Status422UnprocessableEntity, "Validation failed: Media attachments are invalid");
|
||||
post.Media = media.Select(ToPostMedia).ToList();
|
||||
await Attach(media, post.ID, token);
|
||||
// what the edit left out is no longer held by anything
|
||||
var kept = media.Select(m => m.ID).ToList();
|
||||
await _media.Trash(m => m.PostId == post.ID && !kept.Contains(m.ID), "edited out", token);
|
||||
}
|
||||
post.Title = draft.Title == default ? post.Title : Clean(draft.Title);
|
||||
// as on Mastodon, an edit sends the whole post: no warning sent is no warning (the language alone falls back)
|
||||
post.SpoilerText = Clean(draft.SpoilerText);
|
||||
post.HasContentWarning = draft.Sensitive || post.SpoilerText != default;
|
||||
post.Text = draft.Text;
|
||||
@@ -380,6 +410,7 @@ namespace PrivaPub.Domain.Statuses
|
||||
.Modify(p => p.Revisions, new List<PostRevision>())
|
||||
.ExecuteAsync(token);
|
||||
await Fanout.Deleting(post, token);
|
||||
await _media.Trash(m => m.PostId == post.ID, "post deleted", token);
|
||||
await DB.Default.DeleteAsync<TimelineEntry>(e => e.PostId == post.ID || e.ReblogOfPostId == post.ID);
|
||||
// boosts of it end with it, as on Mastodon, so no count keeps them
|
||||
await DB.Default.Update<PostEntity>().Match(p => p.ReblogOfPostId == post.ID && !p.DeletedAt.HasValue)
|
||||
@@ -684,7 +715,8 @@ namespace PrivaPub.Domain.Statuses
|
||||
};
|
||||
}
|
||||
|
||||
async Task<List<MediaAttachment>> Media(LocalActor author, IReadOnlyList<string> ids, string postId, CancellationToken token)
|
||||
// the author's own uploads, not yet another post's (nor held by another scheduled post), nor trashed, nor pictures
|
||||
async Task<List<MediaAttachment>> Media(LocalActor author, IReadOnlyList<string> ids, string postId, string scheduledId, CancellationToken token)
|
||||
{
|
||||
if (ids == default || ids.Count == 0)
|
||||
return new List<MediaAttachment>();
|
||||
@@ -692,23 +724,38 @@ namespace PrivaPub.Domain.Statuses
|
||||
return default;
|
||||
var wanted = ids.Distinct().ToList();
|
||||
var found = await DB.Default.Find<MediaAttachment>()
|
||||
.Match(m => wanted.Contains(m.ID) && m.OwnerAvatarId == author.Id && (m.PostId == null || m.PostId == postId))
|
||||
.Match(m => wanted.Contains(m.ID) && m.OwnerAvatarId == author.Id && (m.PostId == null || m.PostId == postId)
|
||||
&& (m.ScheduledStatusId == null || m.ScheduledStatusId == scheduledId) && m.TrashedAt == null && m.ProfileOfAvatarId == null
|
||||
&& m.ProcessingState == null)
|
||||
.ExecuteAsync(token);
|
||||
return found.Count == wanted.Count ? wanted.Select(id => found.First(m => m.ID == id)).ToList() : default;
|
||||
}
|
||||
|
||||
static async Task Attach(IEnumerable<MediaAttachment> media, string postId, CancellationToken token)
|
||||
// attaches the media in one conditional update, true when all of them are the post's; when another post took one
|
||||
// meanwhile, the ones this call took go back as they were
|
||||
static async Task<bool> Claim(IReadOnlyList<MediaAttachment> media, string postId, string scheduledId, CancellationToken token)
|
||||
{
|
||||
var ids = media.Select(m => m.ID).ToList();
|
||||
if (ids.Count > 0)
|
||||
await DB.Default.Update<MediaAttachment>().Match(m => ids.Contains(m.ID))
|
||||
if (ids.Count == 0)
|
||||
return true;
|
||||
var claimed = await DB.Default.Update<MediaAttachment>()
|
||||
.Match(m => ids.Contains(m.ID) && (m.PostId == null || m.PostId == postId) && m.TrashedAt == null
|
||||
&& (m.ScheduledStatusId == null || m.ScheduledStatusId == scheduledId))
|
||||
.Modify(m => m.PostId, postId).Modify(m => m.AttachedAt, DateTime.UtcNow).ExecuteAsync(token);
|
||||
if (claimed.MatchedCount == ids.Count)
|
||||
return true;
|
||||
var taken = media.Where(m => m.PostId == null).Select(m => m.ID).ToList();
|
||||
await DB.Default.Update<MediaAttachment>().Match(m => taken.Contains(m.ID) && m.PostId == postId)
|
||||
.Modify(m => m.PostId, null).Modify(m => m.AttachedAt, null).ExecuteAsync(token);
|
||||
return false;
|
||||
}
|
||||
|
||||
PostMedia ToPostMedia(MediaAttachment attachment) => new()
|
||||
{
|
||||
AttachmentId = attachment.ID,
|
||||
ContentType = attachment.ContentType,
|
||||
Kind = attachment.Kind,
|
||||
DurationSeconds = attachment.DurationSeconds,
|
||||
URL = _media.Url(attachment.FilePath),
|
||||
PreviewURL = _media.Url(attachment.PreviewPath ?? attachment.FilePath),
|
||||
Description = attachment.Description,
|
||||
|
||||
@@ -21,7 +21,8 @@ namespace PrivaPub.Federation.Actors
|
||||
// FEP-8fcf, received: an account elsewhere tells, with its delivery, a digest of its followers here. When the personas
|
||||
// following it here digest otherwise, its partial list (read signed by the instance actor) says who they are: a follow
|
||||
// only PrivaPub remembers ends, a request it answered without our knowing is accepted, and a follow only it remembers
|
||||
// is undone by the persona, as Mastodon does. A follow ends only when the list is the one the digest describes.
|
||||
// is undone by the persona, as Mastodon does, or kept within RestoreRecord.FollowersGrace of a restore, which may have
|
||||
// lost it. A follow ends only when the list is the one the digest describes.
|
||||
public class FollowingSynchronization : IJobHandler
|
||||
{
|
||||
const int MaxPages = 10;
|
||||
@@ -87,6 +88,19 @@ namespace PrivaPub.Federation.Actors
|
||||
}
|
||||
if (await _localActors.FindByUri(uri, token) is not { Kind: LocalActorKind.Person } stranger)
|
||||
continue;
|
||||
// a follow the restore lost: kept, not undone
|
||||
if (await Infrastructure.Backup.RestoreRecord.InFollowersGrace(token))
|
||||
{
|
||||
await DB.Default.SaveAsync(new Following
|
||||
{
|
||||
AvatarId = stranger.Id,
|
||||
TargetActorURI = actor.ActorURI,
|
||||
TargetAccountId = actor.ID,
|
||||
TargetInboxURL = string.IsNullOrEmpty(actor.InboxURL) ? actor.SharedInboxURL : actor.InboxURL,
|
||||
State = FollowState.Accepted
|
||||
}, token);
|
||||
continue;
|
||||
}
|
||||
// (it has no id for a follow PrivaPub never made; it is undone by actor and object)
|
||||
var undo = new JsonObject
|
||||
{
|
||||
|
||||
@@ -188,10 +188,11 @@ namespace PrivaPub.Federation.Outbox
|
||||
});
|
||||
}
|
||||
|
||||
// FEP-8fcf: what a delivery to the persona's followers tells the receiving server of its followers there
|
||||
// FEP-8fcf: what a delivery to the persona's followers tells the receiving server of its followers there; nothing for
|
||||
// a while after a restore, whose losses are not the other server's to mend
|
||||
static async Task<(string, string)[]> Synchronization(LocalActor signer, string body, Uri inbox, CancellationToken token)
|
||||
{
|
||||
if (!FollowersSynchronization.Synchronizes(signer) || JsonNode.Parse(body) is not JsonObject activity || !FollowersSynchronization.ForFollowers(activity, signer))
|
||||
if (!FollowersSynchronization.Synchronizes(signer) || await Infrastructure.Backup.RestoreRecord.InFollowersGrace(token) || JsonNode.Parse(body) is not JsonObject activity || !FollowersSynchronization.ForFollowers(activity, signer))
|
||||
return [];
|
||||
var there = await FollowersSynchronization.On(signer, FollowersSynchronization.Origin(inbox.AbsoluteUri), token);
|
||||
return [(FollowersSynchronization.Header, FollowersSynchronization.HeaderValue(signer, FollowersSynchronization.Digest(there)))];
|
||||
|
||||
@@ -4,6 +4,7 @@ using PrivaPub.Federation.Actors;
|
||||
using PrivaPub.Federation.Objects;
|
||||
using PrivaPub.Models.Federation;
|
||||
using PrivaPub.Models.Post;
|
||||
using PrivaPub.Domain.Media;
|
||||
|
||||
using System.Globalization;
|
||||
using System.Net;
|
||||
@@ -386,7 +387,7 @@ namespace PrivaPub.Federation.Rendering
|
||||
};
|
||||
if (!string.IsNullOrEmpty(m.Blurhash))
|
||||
attachment["blurhash"] = m.Blurhash;
|
||||
if (m.Focus is { Length: 2 })
|
||||
if (FocalPoint.IsSound(m.Focus))
|
||||
attachment["focalPoint"] = new JsonArray(m.Focus[0], m.Focus[1]);
|
||||
if (m.Width.HasValue && m.Height.HasValue)
|
||||
{
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
using System.Text.Json;
|
||||
using System.Text.Json.Serialization;
|
||||
|
||||
namespace PrivaPub.Infrastructure.Backup
|
||||
{
|
||||
// What a backup holds (manifest.json at its root): enough to check it is whole, to restore it on this host only, and to
|
||||
// rebuild what Mongo had (each collection's indexes).
|
||||
public sealed class ArchiveManifest
|
||||
{
|
||||
public const int CurrentFormat = 1;
|
||||
public const string FileName = "manifest.json";
|
||||
|
||||
public int Format { get; set; } = CurrentFormat;
|
||||
public string Kind { get; set; }//nightly, manual, pre-deploy, pre-restore, uploaded
|
||||
public DateTime CreatedAt { get; set; } = DateTime.UtcNow;
|
||||
public string Host { get; set; }//BackendBaseAddress: URIs and media URLs are stored whole, so only this host can restore it
|
||||
public string AppCommit { get; set; }
|
||||
public string AppRef { get; set; }
|
||||
public string NewestMigration { get; set; }
|
||||
public int MigrationNumber { get; set; }
|
||||
public bool Consistent { get; set; }//read at one instant (a snapshot session on a replica set)
|
||||
public bool DbOnly { get; set; }//no media files, only their list (the deploy's, which can't link www-data's files)
|
||||
public List<CollectionEntry> Collections { get; set; } = [];
|
||||
public List<ExcludedEntry> Excluded { get; set; } = [];
|
||||
public MediaEntry Media { get; set; } = new();
|
||||
|
||||
public sealed class CollectionEntry
|
||||
{
|
||||
public string Name { get; set; }
|
||||
public long Count { get; set; }
|
||||
public long Bytes { get; set; }
|
||||
public string Sha256 { get; set; }
|
||||
public List<string> Indexes { get; set; } = [];//each as canonical Extended JSON
|
||||
}
|
||||
|
||||
public sealed class ExcludedEntry
|
||||
{
|
||||
public string Name { get; set; }
|
||||
public string Why { get; set; }
|
||||
}
|
||||
|
||||
public sealed class MediaEntry
|
||||
{
|
||||
public int Files { get; set; }
|
||||
public long Bytes { get; set; }
|
||||
public int Missing { get; set; }
|
||||
public List<string> List { get; set; } = [];
|
||||
}
|
||||
|
||||
static readonly JsonSerializerOptions Json = new() { WriteIndented = true, PropertyNamingPolicy = JsonNamingPolicy.CamelCase, DefaultIgnoreCondition = JsonIgnoreCondition.Never };
|
||||
|
||||
public static ArchiveManifest Read(string directory)
|
||||
{
|
||||
var path = Path.Combine(directory, FileName);
|
||||
return File.Exists(path) ? JsonSerializer.Deserialize<ArchiveManifest>(File.ReadAllText(path), Json) : default;
|
||||
}
|
||||
|
||||
public void Write(string directory)
|
||||
{
|
||||
using var file = new FileStream(Path.Combine(directory, FileName), FileMode.Create, FileAccess.Write);
|
||||
JsonSerializer.Serialize(file, this, Json);
|
||||
file.Flush(flushToDisk: true);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
namespace PrivaPub.Infrastructure.Backup
|
||||
{
|
||||
public class BackupOptions
|
||||
{
|
||||
public string Root { get; set; }//where backups are kept (/var/lib/privapub/backups); <media root>-backups by default
|
||||
public string NightlyAt { get; set; } = "03:30";//UTC
|
||||
public int KeepDaily { get; set; } = 7;
|
||||
public int KeepWeekly { get; set; } = 4;
|
||||
public int KeepPreDeploy { get; set; } = 3;
|
||||
public int KeepPreRestore { get; set; } = 3;
|
||||
public bool Nightly { get; set; } = true;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,224 @@
|
||||
using System.Formats.Tar;
|
||||
using System.Text;
|
||||
using System.Text.RegularExpressions;
|
||||
|
||||
namespace PrivaPub.Infrastructure.Backup
|
||||
{
|
||||
// A backup as one tar file, for the administrator to download and to upload again: every file under <id>/, its length
|
||||
// known before the first byte (so a download has a Content-Length and resumes with Range), written straight from the
|
||||
// backup's files, never held in memory.
|
||||
public sealed partial class BackupTar
|
||||
{
|
||||
const int Block = 512;
|
||||
|
||||
readonly List<Entry> _entries = [];
|
||||
|
||||
public long Length { get; private set; }
|
||||
|
||||
sealed record Entry(string Name, string Path, long Size, DateTime ModifiedAt, long Offset);
|
||||
|
||||
public static BackupTar Of(string backupsRoot, string id)
|
||||
{
|
||||
var directory = Path.Combine(backupsRoot, id);
|
||||
var tar = new BackupTar();
|
||||
var offset = 0L;
|
||||
foreach (var file in Directory.EnumerateFiles(directory, "*", SearchOption.AllDirectories).OrderBy(f => f, StringComparer.Ordinal))
|
||||
{
|
||||
var info = new FileInfo(file);
|
||||
var name = id + "/" + Path.GetRelativePath(directory, file).Replace('\\', '/');
|
||||
tar._entries.Add(new Entry(name, file, info.Length, info.LastWriteTimeUtc, offset));
|
||||
offset += Block + Padded(info.Length);
|
||||
}
|
||||
tar.Length = offset + 2 * Block;
|
||||
return tar;
|
||||
}
|
||||
|
||||
static long Padded(long size) => (size + Block - 1) / Block * Block;
|
||||
|
||||
/// <summary>Writes bytes from to to (inclusive) of the tar.</summary>
|
||||
public async Task Write(Stream output, long from, long to, CancellationToken token)
|
||||
{
|
||||
var buffer = new byte[81920];
|
||||
foreach (var entry in _entries)
|
||||
{
|
||||
var headerEnd = entry.Offset + Block;
|
||||
var dataEnd = headerEnd + entry.Size;
|
||||
var entryEnd = headerEnd + Padded(entry.Size);
|
||||
if (entryEnd <= from)
|
||||
continue;
|
||||
if (entry.Offset > to)
|
||||
return;
|
||||
if (from < headerEnd)
|
||||
{
|
||||
var header = Header(entry);
|
||||
var start = (int)Math.Max(0, from - entry.Offset);
|
||||
var end = (int)Math.Min(Block, to - entry.Offset + 1);
|
||||
await output.WriteAsync(header.AsMemory(start, end - start), token);
|
||||
}
|
||||
if (from < dataEnd && to >= headerEnd && entry.Size > 0)
|
||||
{
|
||||
var start = Math.Max(0, from - headerEnd);
|
||||
var end = Math.Min(entry.Size, to - headerEnd + 1);
|
||||
await using var file = new FileStream(entry.Path, FileMode.Open, FileAccess.Read, FileShare.Read, 1, FileOptions.SequentialScan);
|
||||
file.Seek(start, SeekOrigin.Begin);
|
||||
var left = end - start;
|
||||
while (left > 0)
|
||||
{
|
||||
var read = await file.ReadAsync(buffer.AsMemory(0, (int)Math.Min(buffer.Length, left)), token);
|
||||
if (read == 0)
|
||||
throw new IOException($"{entry.Name} shrank while it was sent");
|
||||
await output.WriteAsync(buffer.AsMemory(0, read), token);
|
||||
left -= read;
|
||||
}
|
||||
}
|
||||
if (to >= dataEnd && from < entryEnd)
|
||||
{
|
||||
var start = Math.Max(from, dataEnd);
|
||||
var end = Math.Min(entryEnd - 1, to);
|
||||
if (end >= start)
|
||||
await output.WriteAsync(new byte[end - start + 1], token);
|
||||
}
|
||||
}
|
||||
var trailer = Length - 2 * Block;
|
||||
if (to >= trailer)
|
||||
await output.WriteAsync(new byte[to - Math.Max(from, trailer) + 1], token);
|
||||
}
|
||||
|
||||
// a ustar header (name split into prefix and name past 100 bytes; sizes past 8 GiB in base 256)
|
||||
static byte[] Header(Entry entry)
|
||||
{
|
||||
var header = new byte[Block];
|
||||
var name = entry.Name;
|
||||
var prefix = string.Empty;
|
||||
if (Encoding.UTF8.GetByteCount(name) > 100)
|
||||
{
|
||||
var cut = name.LastIndexOf('/', Math.Min(name.Length - 1, 155));
|
||||
while (cut > 0 && Encoding.UTF8.GetByteCount(name[(cut + 1)..]) > 100)
|
||||
cut = name.IndexOf('/', cut + 1);
|
||||
if (cut <= 0 || Encoding.UTF8.GetByteCount(name[..cut]) > 155)
|
||||
throw new InvalidOperationException($"{name} is too long for a tar header");
|
||||
prefix = name[..cut];
|
||||
name = name[(cut + 1)..];
|
||||
}
|
||||
Text(header, 0, 100, name);
|
||||
Octal(header, 100, 8, Convert.ToInt32("640", 8));
|
||||
Octal(header, 108, 8, 0);
|
||||
Octal(header, 116, 8, 0);
|
||||
if (entry.Size < 1L << 33)
|
||||
Octal(header, 124, 12, entry.Size);
|
||||
else
|
||||
{
|
||||
header[124] = 0x80;
|
||||
for (var i = 0; i < 8; i++)
|
||||
header[135 - i] = (byte)(entry.Size >> (8 * i));
|
||||
}
|
||||
Octal(header, 136, 12, new DateTimeOffset(entry.ModifiedAt).ToUnixTimeSeconds());
|
||||
header[156] = (byte)'0';
|
||||
Text(header, 257, 6, "ustar");
|
||||
Text(header, 263, 2, "00");
|
||||
Text(header, 265, 32, "privapub");
|
||||
Text(header, 297, 32, "privapub");
|
||||
Text(header, 345, 155, prefix);
|
||||
for (var i = 148; i < 156; i++)
|
||||
header[i] = (byte)' ';
|
||||
var sum = header.Sum(b => (long)b);
|
||||
Text(header, 148, 7, Convert.ToString(sum, 8).PadLeft(6, '0'));
|
||||
header[155] = (byte)' ';
|
||||
return header;
|
||||
}
|
||||
|
||||
static void Text(byte[] header, int at, int length, string value)
|
||||
{
|
||||
var bytes = Encoding.UTF8.GetBytes(value);
|
||||
Array.Copy(bytes, 0, header, at, Math.Min(bytes.Length, length));
|
||||
}
|
||||
|
||||
static void Octal(byte[] header, int at, int length, long value) =>
|
||||
Text(header, at, length - 1, Convert.ToString(value, 8).PadLeft(length - 1, '0'));
|
||||
|
||||
[GeneratedRegex(@"^\d{8}-\d{6}-[a-z-]{1,20}$")]
|
||||
public static partial Regex BackupId();
|
||||
|
||||
/// <summary>
|
||||
/// A backup read from an uploaded tar into the backups' root: refused when it holds anything but plain files and
|
||||
/// folders under one backup's folder, a path leaving it, or more than the disk can take. The backup, or why not.
|
||||
/// </summary>
|
||||
public static async Task<(BackupInfo Backup, string Error)> Import(Stream tar, string backupsRoot, CancellationToken token)
|
||||
{
|
||||
const int MaxEntries = 2_000_000;
|
||||
var id = default(string);
|
||||
var partial = default(string);
|
||||
var entries = 0;
|
||||
var free = new DriveInfo(Path.GetFullPath(backupsRoot)).AvailableFreeSpace;
|
||||
var written = 0L;
|
||||
try
|
||||
{
|
||||
await using var reader = new TarReader(tar, leaveOpen: true);
|
||||
while (await reader.GetNextEntryAsync(copyData: false, token) is { } entry)
|
||||
{
|
||||
if (++entries > MaxEntries)
|
||||
return (default, "too many files");
|
||||
var name = entry.Name.TrimEnd('/');
|
||||
var slash = name.IndexOf('/');
|
||||
var top = slash < 0 ? name : name[..slash];
|
||||
if (id == default)
|
||||
{
|
||||
if (!BackupId().IsMatch(top))
|
||||
return (default, $"{top} is not a backup's folder");
|
||||
id = top;
|
||||
if (Directory.Exists(Path.Combine(backupsRoot, id)))
|
||||
return (default, $"{id} is already here");
|
||||
partial = Path.Combine(backupsRoot, id + ServerBackup.PartialSuffix);
|
||||
if (Directory.Exists(partial))
|
||||
Directory.Delete(partial, recursive: true);
|
||||
ServerBackup.MakeDirectory(partial);
|
||||
}
|
||||
if (top != id)
|
||||
return (default, $"{name} is outside {id}");
|
||||
if (entry.EntryType is TarEntryType.Directory)
|
||||
continue;
|
||||
if (entry.EntryType is not (TarEntryType.RegularFile or TarEntryType.V7RegularFile))
|
||||
return (default, $"{name} is not a plain file");
|
||||
if (slash < 0)
|
||||
return (default, $"{name} is not inside a folder");
|
||||
written += entry.Length;
|
||||
if (written > free - 512L * 1024 * 1024)
|
||||
return (default, "not enough free disk");
|
||||
var path = ServerBackup.Inside(partial, name[(slash + 1)..]);
|
||||
Directory.CreateDirectory(Path.GetDirectoryName(path)!);
|
||||
await using var file = OperatingSystem.IsWindows()
|
||||
? new FileStream(path, FileMode.CreateNew, FileAccess.Write)
|
||||
: new FileStream(path, new FileStreamOptions { Mode = FileMode.CreateNew, Access = FileAccess.Write,
|
||||
UnixCreateMode = UnixFileMode.UserRead | UnixFileMode.UserWrite | UnixFileMode.GroupRead });
|
||||
if (entry.DataStream != default)
|
||||
await entry.DataStream.CopyToAsync(file, token);
|
||||
}
|
||||
if (id == default)
|
||||
return (default, "the file holds nothing");
|
||||
var manifest = ArchiveManifest.Read(partial);
|
||||
if (manifest == default)
|
||||
return (default, "it has no manifest");
|
||||
manifest.Kind = "uploaded";
|
||||
manifest.Write(partial);
|
||||
Directory.Move(partial, Path.Combine(backupsRoot, id));
|
||||
partial = default;
|
||||
var problems = await ServerBackup.Verify(backupsRoot, id, token);
|
||||
if (problems.Count > 0)
|
||||
{
|
||||
ServerBackup.Delete(backupsRoot, id);
|
||||
return (default, string.Join("; ", problems.Take(5)));
|
||||
}
|
||||
return (ServerBackup.Find(backupsRoot, id), default);
|
||||
}
|
||||
catch (Exception ex) when (ex is FormatException or InvalidDataException or InvalidOperationException or EndOfStreamException or System.Text.Json.JsonException)
|
||||
{
|
||||
return (default, $"not a backup: {ex.Message}");
|
||||
}
|
||||
finally
|
||||
{
|
||||
if (partial != default && Directory.Exists(partial))
|
||||
Directory.Delete(partial, recursive: true);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
using Microsoft.Extensions.Options;
|
||||
|
||||
using MongoDB.Entities;
|
||||
|
||||
using PrivaPub.Domain.Media;
|
||||
using PrivaPub.Models;
|
||||
|
||||
namespace PrivaPub.Infrastructure.Backup
|
||||
{
|
||||
// The server's backups, wherever they are started from: the CLI, the nightly schedule, the administrator's page.
|
||||
public class Backups(IOptionsMonitor<BackupOptions> options, IOptionsMonitor<AppConfiguration> app, IMediaService media)
|
||||
{
|
||||
/// <summary>Where backups are kept: Backups:Root (production's /var/lib/privapub/backups), else beside the media root.</summary>
|
||||
public string Root => Path.GetFullPath(options.CurrentValue.Root ?? media.Root.TrimEnd(Path.DirectorySeparatorChar) + "-backups");
|
||||
|
||||
public BackupOptions Options => options.CurrentValue;
|
||||
|
||||
public string Host => app.CurrentValue.BackendBaseAddress?.TrimEnd('/');
|
||||
|
||||
public BackupContext Context() => new(DB.Default.Database(), Root, media.Root, media.TrashRoot, Host, options.CurrentValue);
|
||||
|
||||
public Task<(BackupInfo Backup, string Error)> Create(string kind, bool dbOnly, CancellationToken token) =>
|
||||
ServerBackup.Create(Context(), kind, dbOnly, token);
|
||||
|
||||
public List<BackupInfo> List() => ServerBackup.List(Root);
|
||||
|
||||
public BackupInfo Find(string id) => ServerBackup.Find(Root, id);
|
||||
|
||||
public Task<List<string>> Verify(string id, CancellationToken token) => ServerBackup.Verify(Root, id, token);
|
||||
|
||||
public bool Delete(string id) => ServerBackup.Delete(Root, id);
|
||||
}
|
||||
|
||||
// A nightly backup at Backups:NightlyAt (UTC), or as soon as the service is up after missing it; the old ones rotated
|
||||
// out as each is made. Personas' archives a week old go too.
|
||||
public class BackupScheduler(Backups backups, ILogger<BackupScheduler> logger) : BackgroundService
|
||||
{
|
||||
static readonly TimeSpan Interval = TimeSpan.FromMinutes(10);
|
||||
|
||||
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
|
||||
{
|
||||
while (!stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
try
|
||||
{
|
||||
await Task.Delay(Interval, stoppingToken);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
return;
|
||||
}
|
||||
try
|
||||
{
|
||||
await RunIfDue(DateTime.UtcNow, stoppingToken);
|
||||
await Domain.Portability.PersonaArchives.Forget(backups.Root, stoppingToken);
|
||||
}
|
||||
catch (Exception ex) when (ex is not OperationCanceledException)
|
||||
{
|
||||
logger.LogError(ex, "The nightly backup failed");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Backs up when the day's time has come and there is no nightly backup since: whether it did.</summary>
|
||||
public async Task<bool> RunIfDue(DateTime now, CancellationToken token)
|
||||
{
|
||||
var options = backups.Options;
|
||||
if (!options.Nightly || !TimeOnly.TryParse(options.NightlyAt, System.Globalization.CultureInfo.InvariantCulture, out var at)
|
||||
|| !IsDue(now, at, backups.List().Where(b => b.Kind == "nightly").Select(b => b.CreatedAt)))
|
||||
return false;
|
||||
var (made, error) = await backups.Create("nightly", dbOnly: false, token);
|
||||
if (made == default)
|
||||
{
|
||||
logger.LogWarning("The nightly backup was not made: {Error}", error);
|
||||
return false;
|
||||
}
|
||||
logger.LogInformation("Nightly backup {Id}: {Collections} collections, {Files} media files", made.Id, made.Manifest.Collections.Count, made.Manifest.Media.Files);
|
||||
return true;
|
||||
}
|
||||
|
||||
/// <summary>Whether the last time of day for a nightly backup has passed with no nightly backup made since.</summary>
|
||||
public static bool IsDue(DateTime now, TimeOnly at, IEnumerable<DateTime> nightlies)
|
||||
{
|
||||
var due = now.Date + at.ToTimeSpan();
|
||||
if (now < due)
|
||||
due = due.AddDays(-1);
|
||||
return !nightlies.Any(made => made >= due);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
using System.Runtime.InteropServices;
|
||||
|
||||
namespace PrivaPub.Infrastructure.Backup
|
||||
{
|
||||
// A second name for a file (.NET has no API for it): a backup links the live media, which costs no disk, since names are
|
||||
// random and files never change; deleting the live one leaves the backup's.
|
||||
static class HardLink
|
||||
{
|
||||
// strings go to libc as UTF-8 on Unix
|
||||
[DllImport("libc", EntryPoint = "link", SetLastError = true, CharSet = CharSet.Ansi)]
|
||||
static extern int Link(string existing, string created);
|
||||
|
||||
/// <summary>Links, or copies where a link can't be made (another disk, a filesystem refusing it).</summary>
|
||||
public static bool LinkOrCopy(string existing, string created)
|
||||
{
|
||||
Directory.CreateDirectory(Path.GetDirectoryName(created)!);
|
||||
if (OperatingSystem.IsLinux())
|
||||
{
|
||||
try
|
||||
{
|
||||
if (Link(existing, created) == 0)
|
||||
return true;
|
||||
}
|
||||
catch (DllNotFoundException)
|
||||
{
|
||||
}
|
||||
catch (EntryPointNotFoundException)
|
||||
{
|
||||
}
|
||||
}
|
||||
File.Copy(existing, created, overwrite: true);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
namespace PrivaPub.Infrastructure.Backup
|
||||
{
|
||||
// What Program asks before migrations: whether a restore waits, and what this process does about it. The service carries
|
||||
// it out; a command waits for it (exit 75, EX_TEMPFAIL), except asking how it went.
|
||||
public static class MaintenanceGate
|
||||
{
|
||||
/// <summary>The exit code that stops systemd restarting the service (RestartPreventExitStatus=75): a restore gave up.</summary>
|
||||
public const int TryLater = 75;
|
||||
|
||||
/// <summary>Null to go on booting; else the code to exit with.</summary>
|
||||
public static async Task<int?> Enter(IServiceProvider services, string[] command, CancellationToken token)
|
||||
{
|
||||
var backups = services.GetRequiredService<Backups>();
|
||||
var logger = services.GetRequiredService<ILoggerFactory>().CreateLogger(nameof(MaintenanceGate));
|
||||
if (command != default)
|
||||
{
|
||||
var waiting = RestoreMarker.Read(backups.Root);
|
||||
if (waiting == default || command is ["restore", ..])
|
||||
return default;
|
||||
Console.Error.WriteLine($"the restore of {waiting.Backup} is {waiting.State}{(waiting.Error == default ? string.Empty : $" ({waiting.Error})")}: try again once the service has carried it out (PrivaPub admin restore --status)");
|
||||
return TryLater;
|
||||
}
|
||||
return await ServerRestore.ApplyPending(backups.Context(), logger, token) switch
|
||||
{
|
||||
RestoreOutcome.Failed => 1,//systemd starts it again, and the next attempt redoes everything
|
||||
RestoreOutcome.GaveUp => TryLater,
|
||||
_ => (int?)null
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
// A restore asked for while the service runs (from the administrator's page or the CLI): the service stops within
|
||||
// seconds, systemd starts it again, and the restore runs before anything else.
|
||||
public class RestoreWatcher(Backups backups, IHostApplicationLifetime lifetime, ILogger<RestoreWatcher> logger) : BackgroundService
|
||||
{
|
||||
static readonly TimeSpan Interval = TimeSpan.FromSeconds(3);
|
||||
|
||||
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
|
||||
{
|
||||
while (!stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
try
|
||||
{
|
||||
await Task.Delay(Interval, stoppingToken);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
return;
|
||||
}
|
||||
if (RestoreMarker.Read(backups.Root) is { State: "pending" } marker)
|
||||
{
|
||||
logger.LogWarning("A restore of {Backup} was asked for by {RequestedBy}: stopping, to restore it at the next start", marker.Backup, marker.RequestedBy);
|
||||
lifetime.StopApplication();
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,75 @@
|
||||
using MongoDB.Driver;
|
||||
using MongoDB.Entities;
|
||||
|
||||
namespace PrivaPub.Infrastructure.Backup
|
||||
{
|
||||
// One backup or restore at a time, whoever starts it (the CLI, the nightly schedule, the administrator's page): a
|
||||
// document held with a heartbeat. A holder that died (no heartbeat for two minutes) is taken over. The collection is
|
||||
// never backed up, nor dropped by a restore.
|
||||
public class MaintenanceLock : Entity
|
||||
{
|
||||
public const string Name = "maintenance";
|
||||
static readonly TimeSpan Stale = TimeSpan.FromMinutes(2);
|
||||
static readonly TimeSpan Beat = TimeSpan.FromSeconds(30);
|
||||
|
||||
public string Owner { get; set; }
|
||||
public string What { get; set; }
|
||||
public DateTime Since { get; set; }
|
||||
public DateTime Heartbeat { get; set; }
|
||||
|
||||
/// <summary>Takes the lock for what is said, or null when someone else holds it; disposing the result frees it.</summary>
|
||||
public static async Task<Held> Take(string what, CancellationToken token)
|
||||
{
|
||||
var owner = $"{Environment.MachineName}:{Environment.ProcessId}:{Guid.NewGuid():N}";
|
||||
var now = DateTime.UtcNow;
|
||||
var stale = now - Stale;
|
||||
try
|
||||
{
|
||||
var taken = await DB.Default.UpdateAndGet<MaintenanceLock>()
|
||||
.Match(l => l.ID == Name && (l.Heartbeat < stale || l.Owner == null))
|
||||
.Modify(l => l.Owner, owner)
|
||||
.Modify(l => l.What, what)
|
||||
.Modify(l => l.Since, now)
|
||||
.Modify(l => l.Heartbeat, now)
|
||||
.Option(o => o.IsUpsert = true)
|
||||
.ExecuteAsync(token);
|
||||
return taken?.Owner == owner ? new Held(owner) : default;
|
||||
}
|
||||
catch (MongoCommandException ex) when (ex.Code == 11000)
|
||||
{
|
||||
return default;//held: the upsert met the live lock's id
|
||||
}
|
||||
catch (MongoWriteException ex) when (ex.WriteError?.Category == ServerErrorCategory.DuplicateKey)
|
||||
{
|
||||
return default;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Who holds it now and for what, or null.</summary>
|
||||
public static async Task<MaintenanceLock> Current(CancellationToken token)
|
||||
{
|
||||
var held = await DB.Default.Find<MaintenanceLock>().Match(l => l.ID == Name).ExecuteFirstAsync(token);
|
||||
return held?.Owner != null && held.Heartbeat >= DateTime.UtcNow - Stale ? held : default;
|
||||
}
|
||||
|
||||
public sealed class Held : IAsyncDisposable
|
||||
{
|
||||
readonly string _owner;
|
||||
readonly Timer _heartbeat;
|
||||
|
||||
public Held(string owner)
|
||||
{
|
||||
_owner = owner;
|
||||
_heartbeat = new Timer(_ => _ = DB.Default.Update<MaintenanceLock>().Match(l => l.ID == Name && l.Owner == owner)
|
||||
.Modify(l => l.Heartbeat, DateTime.UtcNow).ExecuteAsync(), default, Beat, Beat);
|
||||
}
|
||||
|
||||
public async ValueTask DisposeAsync()
|
||||
{
|
||||
await _heartbeat.DisposeAsync();
|
||||
await DB.Default.Update<MaintenanceLock>().Match(l => l.ID == Name && l.Owner == _owner)
|
||||
.Modify(l => l.Owner, null).Modify(l => l.Heartbeat, DateTime.MinValue).ExecuteAsync();
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,271 @@
|
||||
using MongoDB.Bson;
|
||||
using MongoDB.Driver;
|
||||
|
||||
namespace PrivaPub.Infrastructure.Backup
|
||||
{
|
||||
// A restore may lose what was made since the backup, but it never undoes a protective act (owner decision 2026-10-07).
|
||||
// From the pre-restore backup P, after the backup is imported:
|
||||
// - who follows whom is P's: a follower that left stays gone, so no followers-only post reaches it, and one gained is
|
||||
// kept; the persona's follows are P's too;
|
||||
// - blocks, mutes, domain blocks (the server's and the personas'), being blocked, reserved names, deletion tombstones,
|
||||
// reports, filters and OAuth applications are the union of both, P's row winning;
|
||||
// - deletions win: a root, persona, group, post or remote account deleted since is deleted again (P's row, as its
|
||||
// deletion left it), with what a deletion takes away; P's moderation of a persona (silenced, suspended, banned) stays;
|
||||
// - a root keeps P's password, e-mail, ban and policies;
|
||||
// - roots, personas and groups made since become tombstones: deleted, their names kept; local posts made since answer
|
||||
// 410; media made since go to the trash, as do media trashed since (and they leave the posts restored with them);
|
||||
// - every session ends, and every server's circuit is closed again.
|
||||
// Run on the imported database, it reads P's files only, so it can run again after a failed attempt.
|
||||
public static class ProtectiveMerge
|
||||
{
|
||||
static readonly (string Collection, string[] Key)[] Unions =
|
||||
[
|
||||
("ReservedName", ["Name"]),
|
||||
("DomainBlock", ["Domain"]),
|
||||
("Block", ["AvatarId", "TargetActorURI"]),
|
||||
("Mute", ["AvatarId", "TargetActorURI"]),
|
||||
("AccountDomainBlock", ["AvatarId", "Domain"]),
|
||||
("BlockedBy", ["AvatarId", "ActorURI"]),
|
||||
("DeletedObject", ["ObjectURI"]),
|
||||
("Report", ["_id"]),
|
||||
("PersonaFilter", ["_id"]),
|
||||
("openiddict.applications", ["client_id"])
|
||||
];
|
||||
|
||||
const string Since = "restore: made after the backup";
|
||||
|
||||
public static async Task Apply(IMongoDatabase database, string previous, RestoreReport report, CancellationToken token)
|
||||
{
|
||||
IEnumerable<BsonDocument> P(string collection) => ServerRestore.Read(Path.Combine(previous, collection + ".jsonl.gz"));
|
||||
var now = DateTime.UtcNow;
|
||||
|
||||
// who follows whom: P's
|
||||
foreach (var collection in new[] { "Follower", "Following" })
|
||||
{
|
||||
await database.DropCollectionAsync(collection, token);
|
||||
await ServerRestore.Insert(database.GetCollection<BsonDocument>(collection), P(collection), token);
|
||||
}
|
||||
|
||||
foreach (var (collection, key) in Unions)
|
||||
report.ProtectiveKept += await Union(database.GetCollection<BsonDocument>(collection), P(collection), key, token);
|
||||
|
||||
var roots = database.GetCollection<BsonDocument>("RootUser");
|
||||
foreach (var root in P("RootUser"))
|
||||
{
|
||||
var restored = await ById(roots, root["_id"], token);
|
||||
if (restored == default)
|
||||
{
|
||||
// made since: deleted, as RootRemoval leaves a root
|
||||
report.RootsTombstoned.Add(root.GetValue("UserName", "").ToString());
|
||||
root["UserName"] = $"deleted-{root["_id"]}";
|
||||
root["Email"] = BsonNull.Value;
|
||||
root["HashedPassword"] = BsonNull.Value;
|
||||
root["Policies"] = new BsonArray();
|
||||
root["IsBanned"] = false;
|
||||
root["IsEmailValidated"] = false;
|
||||
root["DeletedAt"] = Deleted(root, "DeletedAt", now);
|
||||
await roots.InsertOneAsync(root, cancellationToken: token);
|
||||
continue;
|
||||
}
|
||||
if (IsSet(root, "DeletedAt"))
|
||||
{
|
||||
if (!IsSet(restored, "DeletedAt"))
|
||||
report.RootsDeleted++;
|
||||
await roots.ReplaceOneAsync(Id(root["_id"]), root, cancellationToken: token);
|
||||
continue;
|
||||
}
|
||||
await roots.UpdateOneAsync(Id(root["_id"]), Copy(root, "HashedPassword", "Email", "IsEmailValidated", "IsBanned", "Policies",
|
||||
"CredentialsChangedAt", "ResetPasswordToken", "ResetPasswordTokenSentAt"), cancellationToken: token);
|
||||
}
|
||||
|
||||
var avatars = database.GetCollection<BsonDocument>("Avatar");
|
||||
foreach (var avatar in P("Avatar"))
|
||||
{
|
||||
var restored = await ById(avatars, avatar["_id"], token);
|
||||
if (restored == default)
|
||||
{
|
||||
report.PersonasTombstoned.Add(avatar.GetValue("UserName", "").ToString());
|
||||
avatar["DeletionAt"] = Deleted(avatar, "DeletionAt", now);
|
||||
await avatars.InsertOneAsync(avatar, cancellationToken: token);
|
||||
continue;
|
||||
}
|
||||
if (IsSet(avatar, "DeletionAt"))
|
||||
{
|
||||
if (!IsSet(restored, "DeletionAt"))
|
||||
report.PersonasDeleted++;
|
||||
await avatars.ReplaceOneAsync(Id(avatar["_id"]), avatar, cancellationToken: token);
|
||||
// what RootRemoval takes with a persona, which the backup still had
|
||||
foreach (var collection in new[] { "PersonaListMember", "PersonaList", "PersonaFilter", "FollowedTag", "ScheduledStatus" })
|
||||
await database.GetCollection<BsonDocument>(collection).DeleteManyAsync(new BsonDocument("AvatarId", avatar["_id"].ToString()), token);
|
||||
continue;
|
||||
}
|
||||
var moderated = new[] { "SilencedAt", "SuspendedAt", "BannedAt" }.Where(field => IsSet(avatar, field)).ToArray();
|
||||
if (moderated.Length > 0)
|
||||
await avatars.UpdateOneAsync(Id(avatar["_id"]), Copy(avatar, moderated), cancellationToken: token);
|
||||
}
|
||||
|
||||
var links = database.GetCollection<BsonDocument>("RootToAvatar");
|
||||
var linked = (await (await links.FindAsync(FilterDefinition<BsonDocument>.Empty, cancellationToken: token)).ToListAsync(token))
|
||||
.Select(l => l.GetValue("AvatarId", BsonNull.Value).ToString()).ToHashSet(StringComparer.Ordinal);
|
||||
await ServerRestore.Insert(links, P("RootToAvatar").Where(l => !linked.Contains(l.GetValue("AvatarId", BsonNull.Value).ToString())), token);
|
||||
|
||||
var groups = database.GetCollection<BsonDocument>("Group");
|
||||
foreach (var group in P("Group"))
|
||||
{
|
||||
var restored = await ById(groups, group["_id"], token);
|
||||
if (restored == default)
|
||||
{
|
||||
report.GroupsTombstoned.Add(group.GetValue("UserName", "").ToString());
|
||||
group["DeletionAt"] = Deleted(group, "DeletionAt", now);
|
||||
await groups.InsertOneAsync(group, cancellationToken: token);
|
||||
continue;
|
||||
}
|
||||
if (!IsSet(group, "DeletionAt"))
|
||||
continue;
|
||||
if (!IsSet(restored, "DeletionAt"))
|
||||
report.GroupsDeleted++;
|
||||
await groups.ReplaceOneAsync(Id(group["_id"]), group, cancellationToken: token);
|
||||
}
|
||||
|
||||
// remote accounts deleted or gone since: as P has them
|
||||
var foreign = database.GetCollection<BsonDocument>("ForeignAvatar");
|
||||
foreach (var account in P("ForeignAvatar").Where(a => IsSet(a, "DeletionAt") || IsSet(a, "ForgottenAt")))
|
||||
await foreign.ReplaceOneAsync(Id(account["_id"]), account, cancellationToken: token);
|
||||
|
||||
await Posts(database, P("Post"), report, token);
|
||||
await Media(database, P("MediaAttachment"), report, token);
|
||||
|
||||
// every session ends: a new stamp for each root's tokens, and the personas' OAuth tokens go
|
||||
foreach (var root in await (await roots.FindAsync(FilterDefinition<BsonDocument>.Empty, cancellationToken: token)).ToListAsync(token))
|
||||
await roots.UpdateOneAsync(Id(root["_id"]), Builders<BsonDocument>.Update
|
||||
.Set("SessionStamp", Guid.NewGuid().ToString("N"))
|
||||
.Set("CredentialsChangedAt", now), cancellationToken: token);
|
||||
foreach (var collection in new[] { "openiddict.tokens", "openiddict.authorizations" })
|
||||
report.SessionsEnded += (int)(await database.GetCollection<BsonDocument>(collection).DeleteManyAsync(FilterDefinition<BsonDocument>.Empty, token)).DeletedCount;
|
||||
|
||||
// what the backup remembered of servers being down is long past
|
||||
await database.GetCollection<BsonDocument>("RemoteInstance").UpdateManyAsync(FilterDefinition<BsonDocument>.Empty,
|
||||
Builders<BsonDocument>.Update.Set("ConsecutiveFailures", 0).Set("UnavailableUntil", BsonNull.Value), cancellationToken: token);
|
||||
}
|
||||
|
||||
// posts deleted since: deleted again, out of timelines and pins; local posts made since: there as deleted, as a
|
||||
// deletion leaves them, so they answer 410 and their ids and addresses are never given again
|
||||
static async Task Posts(IMongoDatabase database, IEnumerable<BsonDocument> previous, RestoreReport report, CancellationToken token)
|
||||
{
|
||||
var posts = database.GetCollection<BsonDocument>("Post");
|
||||
var now = DateTime.UtcNow;
|
||||
foreach (var batch in previous.Where(p => IsSet(p, "DeletedAt") || IsLocal(p)).Chunk(500))
|
||||
{
|
||||
var ids = new BsonArray(batch.Select(p => p["_id"]));
|
||||
var restored = (await (await posts.FindAsync(new BsonDocument("_id", new BsonDocument("$in", ids)), cancellationToken: token)).ToListAsync(token))
|
||||
.ToDictionary(p => p["_id"]);
|
||||
foreach (var post in batch)
|
||||
{
|
||||
if (!restored.TryGetValue(post["_id"], out var mine))
|
||||
{
|
||||
if (!IsLocal(post))
|
||||
continue;
|
||||
if (!IsSet(post, "DeletedAt"))
|
||||
{
|
||||
post["DeletedAt"] = now;
|
||||
foreach (var field in new[] { "Text", "ContentHtml", "Title", "SpoilerText" })
|
||||
post[field] = BsonNull.Value;
|
||||
post["Media"] = new BsonArray();
|
||||
post["Revisions"] = new BsonArray();
|
||||
}
|
||||
await posts.InsertOneAsync(post, cancellationToken: token);
|
||||
report.PostsGone++;
|
||||
continue;
|
||||
}
|
||||
if (!IsSet(post, "DeletedAt") || IsSet(mine, "DeletedAt"))
|
||||
continue;
|
||||
await posts.ReplaceOneAsync(Id(post["_id"]), post, cancellationToken: token);
|
||||
var id = post["_id"].ToString();
|
||||
await database.GetCollection<BsonDocument>("TimelineEntry").DeleteManyAsync(
|
||||
new BsonDocument("$or", new BsonArray { new BsonDocument("PostId", id), new BsonDocument("ReblogOfPostId", id) }), token);
|
||||
await database.GetCollection<BsonDocument>("Pin").DeleteManyAsync(new BsonDocument("PostId", id), token);
|
||||
report.PostsDeleted++;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// media trashed since, and media made since: in the trash (the janitor deletes their files after its grace), and out
|
||||
// of the posts restored; media of posts deleted now: in the trash too
|
||||
static async Task Media(IMongoDatabase database, IEnumerable<BsonDocument> previous, RestoreReport report, CancellationToken token)
|
||||
{
|
||||
var media = database.GetCollection<BsonDocument>("MediaAttachment");
|
||||
var posts = database.GetCollection<BsonDocument>("Post");
|
||||
var now = DateTime.UtcNow;
|
||||
foreach (var batch in previous.Chunk(1000))
|
||||
{
|
||||
var ids = new BsonArray(batch.Select(m => m["_id"]));
|
||||
var restored = (await (await media.FindAsync(new BsonDocument("_id", new BsonDocument("$in", ids)), cancellationToken: token)).ToListAsync(token))
|
||||
.ToDictionary(m => m["_id"]);
|
||||
var made = new List<BsonDocument>();
|
||||
foreach (var row in batch)
|
||||
{
|
||||
if (!restored.TryGetValue(row["_id"], out var mine))
|
||||
{
|
||||
if (!IsSet(row, "TrashedAt"))
|
||||
{
|
||||
row["TrashedAt"] = now;
|
||||
row["TrashReason"] = Since;
|
||||
}
|
||||
made.Add(row);
|
||||
continue;
|
||||
}
|
||||
if (!IsSet(row, "TrashedAt") || IsSet(mine, "TrashedAt"))
|
||||
continue;
|
||||
await media.UpdateOneAsync(Id(row["_id"]), Copy(row, "TrashedAt", "TrashReason"), cancellationToken: token);
|
||||
if (mine.GetValue("PostId", BsonNull.Value) is BsonString postId)
|
||||
await posts.UpdateOneAsync(Id(ObjectId.TryParse(postId.AsString, out var post) ? post : postId), Builders<BsonDocument>.Update.PullFilter<BsonDocument>("Media",
|
||||
new BsonDocument("AttachmentId", row["_id"].ToString())), cancellationToken: token);
|
||||
report.MediaTrashed++;
|
||||
}
|
||||
await ServerRestore.Insert(media, made, token);
|
||||
report.MediaTrashed += made.Count(m => m["TrashReason"] == Since);
|
||||
}
|
||||
|
||||
var deleted = await (await posts.FindAsync(new BsonDocument("DeletedAt", new BsonDocument("$ne", BsonNull.Value)),
|
||||
new FindOptions<BsonDocument> { Projection = new BsonDocument("_id", 1) }, token)).ToListAsync(token);
|
||||
foreach (var batch in deleted.Select(p => (BsonValue)p["_id"].ToString()).Chunk(1000))
|
||||
report.MediaTrashed += (int)(await media.UpdateManyAsync(
|
||||
new BsonDocument { { "PostId", new BsonDocument("$in", new BsonArray(batch)) }, { "TrashedAt", BsonNull.Value } },
|
||||
Builders<BsonDocument>.Update.Set("TrashedAt", now).Set("TrashReason", "restore: its post was deleted"), cancellationToken: token)).ModifiedCount;
|
||||
}
|
||||
|
||||
// P's rows added, each replacing the restored row with the same key: how many P had
|
||||
static async Task<int> Union(IMongoCollection<BsonDocument> collection, IEnumerable<BsonDocument> previous, string[] key, CancellationToken token)
|
||||
{
|
||||
var count = 0;
|
||||
foreach (var batch in previous.Chunk(500))
|
||||
{
|
||||
var writes = new List<WriteModel<BsonDocument>>();
|
||||
foreach (var row in batch)
|
||||
{
|
||||
var match = new BsonDocument(key.Select(field => new BsonElement(field, row.GetValue(field, BsonNull.Value))));
|
||||
writes.Add(new DeleteManyModel<BsonDocument>(match));
|
||||
writes.Add(new InsertOneModel<BsonDocument>(row));
|
||||
}
|
||||
await collection.BulkWriteAsync(writes, new BulkWriteOptions { IsOrdered = true, BypassDocumentValidation = true }, token);
|
||||
count += batch.Length;
|
||||
}
|
||||
return count;
|
||||
}
|
||||
|
||||
static bool IsLocal(BsonDocument post) =>
|
||||
post.GetValue("GroupUserId", BsonNull.Value) is BsonString && !post.GetValue("IsFederatedCopy", false).ToBoolean();
|
||||
|
||||
static bool IsSet(BsonDocument row, string field) => row.TryGetValue(field, out var value) && !value.IsBsonNull;
|
||||
|
||||
static BsonValue Deleted(BsonDocument row, string field, DateTime now) => IsSet(row, field) ? row[field] : now;
|
||||
|
||||
static FilterDefinition<BsonDocument> Id(BsonValue id) => new BsonDocument("_id", id);
|
||||
|
||||
static async Task<BsonDocument> ById(IMongoCollection<BsonDocument> collection, BsonValue id, CancellationToken token) =>
|
||||
await (await collection.FindAsync(Id(id), cancellationToken: token)).FirstOrDefaultAsync(token);
|
||||
|
||||
static UpdateDefinition<BsonDocument> Copy(BsonDocument from, params string[] fields) =>
|
||||
Builders<BsonDocument>.Update.Combine(fields.Select(field => Builders<BsonDocument>.Update.Set(field, from.GetValue(field, BsonNull.Value))));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
using System.Text.Json;
|
||||
|
||||
namespace PrivaPub.Infrastructure.Backup
|
||||
{
|
||||
// A restore asked for (by the administrator's page or the CLI) and carried out at the next boot, before anything
|
||||
// else (Program, MaintenanceGate): restore.json in the backups' directory, written whole or not at all.
|
||||
public sealed class RestoreMarker
|
||||
{
|
||||
public const string FileName = "restore.json";
|
||||
public const int MaxAttempts = 3;
|
||||
|
||||
public string Backup { get; set; }//the backup restored
|
||||
public string PreRestore { get; set; }//the backup taken just before, what the protective merge reads
|
||||
public string State { get; set; } = "pending";//pending, then running
|
||||
public string RequestedBy { get; set; }//who asked: a root's name, or "cli"
|
||||
public int Attempts { get; set; }
|
||||
public DateTime RequestedAt { get; set; } = DateTime.UtcNow;
|
||||
public string Error { get; set; }
|
||||
|
||||
public static string PathIn(string backupsRoot) => Path.Combine(backupsRoot, FileName);
|
||||
|
||||
public static RestoreMarker Read(string backupsRoot)
|
||||
{
|
||||
var path = PathIn(backupsRoot);
|
||||
if (!File.Exists(path))
|
||||
return default;
|
||||
try
|
||||
{
|
||||
return JsonSerializer.Deserialize<RestoreMarker>(File.ReadAllText(path));
|
||||
}
|
||||
catch (JsonException)
|
||||
{
|
||||
//never written half (Write is atomic): someone's hand; the server won't guess what was meant
|
||||
return new RestoreMarker { State = "unreadable", Attempts = MaxAttempts, Error = $"{FileName} can't be read" };
|
||||
}
|
||||
}
|
||||
|
||||
public void Write(string backupsRoot)
|
||||
{
|
||||
Directory.CreateDirectory(backupsRoot);
|
||||
var path = PathIn(backupsRoot);
|
||||
var part = path + ".part";
|
||||
using (var file = new FileStream(part, FileMode.Create, FileAccess.Write))
|
||||
{
|
||||
JsonSerializer.Serialize(file, this);
|
||||
file.Flush(flushToDisk: true);
|
||||
}
|
||||
File.Move(part, path, overwrite: true);
|
||||
}
|
||||
|
||||
public static void Clear(string backupsRoot)
|
||||
{
|
||||
var path = PathIn(backupsRoot);
|
||||
if (File.Exists(path))
|
||||
File.Delete(path);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
using MongoDB.Entities;
|
||||
|
||||
namespace PrivaPub.Infrastructure.Backup
|
||||
{
|
||||
// What a restore did (or why it was abandoned), kept for the administrator's page and for the followers' grace: for
|
||||
// FollowersGrace after a restore the followers' digests (FEP-8fcf) are neither sent nor acted on to undo a follow, since
|
||||
// what the restore lost of either side is not the other's fault.
|
||||
public class RestoreRecord : Entity
|
||||
{
|
||||
public static readonly TimeSpan FollowersGrace = TimeSpan.FromDays(14);
|
||||
|
||||
public string Backup { get; set; }
|
||||
public DateTime BackupCreatedAt { get; set; }
|
||||
public string PreRestore { get; set; }
|
||||
public string RequestedBy { get; set; }
|
||||
public DateTime RequestedAt { get; set; }
|
||||
public DateTime RestoredAt { get; set; } = DateTime.UtcNow;
|
||||
public int Attempts { get; set; }
|
||||
public bool Abandoned { get; set; }//nothing was changed: the server booted as it was
|
||||
public string Error { get; set; }
|
||||
public RestoreReport Report { get; set; } = new();
|
||||
|
||||
static (DateTime Until, DateTime Read) _grace;
|
||||
|
||||
/// <summary>Whether a restore ended less than FollowersGrace ago (read at most once a minute).</summary>
|
||||
public static async Task<bool> InFollowersGrace(CancellationToken token)
|
||||
{
|
||||
var now = DateTime.UtcNow;
|
||||
var cached = _grace;
|
||||
if (now - cached.Read > TimeSpan.FromMinutes(1))
|
||||
{
|
||||
var last = await DB.Default.Find<RestoreRecord>().Match(r => !r.Abandoned).Sort(r => r.RestoredAt, Order.Descending).ExecuteFirstAsync(token);
|
||||
cached = (last == default ? DateTime.MinValue : last.RestoredAt + FollowersGrace, now);
|
||||
_grace = cached;
|
||||
}
|
||||
return now < cached.Until;
|
||||
}
|
||||
|
||||
/// <summary>Forgets what was read, so a restore made in this process counts at once.</summary>
|
||||
public static void Forget() => _grace = default;
|
||||
}
|
||||
|
||||
public class RestoreReport
|
||||
{
|
||||
public int Collections { get; set; }
|
||||
public long Documents { get; set; }
|
||||
public int CollectionsDropped { get; set; }//live collections the backup had no documents in
|
||||
public int MediaRestored { get; set; }//files the live directory lacked, brought back
|
||||
public int MediaMissing { get; set; }
|
||||
public int RootsDeleted { get; set; }//deleted since the backup, deleted again
|
||||
public int PersonasDeleted { get; set; }
|
||||
public int GroupsDeleted { get; set; }
|
||||
public int PostsDeleted { get; set; }
|
||||
public List<string> RootsTombstoned { get; set; } = [];//made after the backup: deleted, their names kept
|
||||
public List<string> PersonasTombstoned { get; set; } = [];
|
||||
public List<string> GroupsTombstoned { get; set; } = [];
|
||||
public int PostsGone { get; set; }//local posts made after the backup: 410 from now on
|
||||
public int MediaTrashed { get; set; }
|
||||
public int ProtectiveKept { get; set; }//blocks, mutes, domain blocks, reserved names, reports, filters, deletions kept from since
|
||||
public int SessionsEnded { get; set; }
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,343 @@
|
||||
using MongoDB.Bson;
|
||||
using MongoDB.Bson.IO;
|
||||
using MongoDB.Driver;
|
||||
|
||||
using PrivaPub.Infrastructure.Data;
|
||||
|
||||
using System.Globalization;
|
||||
using System.IO.Compression;
|
||||
using System.Security.Cryptography;
|
||||
using System.Text;
|
||||
|
||||
namespace PrivaPub.Infrastructure.Backup
|
||||
{
|
||||
// What a backup needs to know: where things are, and whose host it is.
|
||||
public sealed record BackupContext(IMongoDatabase Database, string BackupsRoot, string MediaRoot, string TrashRoot, string Host, BackupOptions Options);
|
||||
|
||||
public sealed record BackupInfo(string Id, string Kind, DateTime CreatedAt, long Bytes, ArchiveManifest Manifest);
|
||||
|
||||
// The whole server, backed up as files (owner decision 2026-10-07: a whole-server backup, plain, protected by file
|
||||
// permissions). Each backup is a directory <stamp>-<kind> in the backups' root:
|
||||
// - manifest.json: what it holds (ArchiveManifest);
|
||||
// - db/<collection>.jsonl.gz: every document of each collection, one per line, in canonical Extended JSON (every BSON
|
||||
// type kept as it was), all read at one instant when Mongo is a replica set (a snapshot session);
|
||||
// - media/<path>: the media files rows hold, as hard links to the live ones (no disk spent; a deleted file stays here
|
||||
// until the backup is rotated out), copied where a link can't be made; a db-only backup lists them instead.
|
||||
// It is written as <id>.partial and renamed once whole. Left out: what belongs to the moment (Excluded). Everything is
|
||||
// readable by the service's user and group only: it holds every persona's private key and private posts.
|
||||
public static class ServerBackup
|
||||
{
|
||||
public const string PartialSuffix = ".partial";
|
||||
|
||||
public static readonly IReadOnlyDictionary<string, string> Excluded = new Dictionary<string, string>
|
||||
{
|
||||
["InteractionSalt"] = "the day's statistics salt never outlives its day (owner rule)",
|
||||
["Job"] = "work in flight: deliveries and inbox processing belong to the moment",
|
||||
["Delivery"] = "work in flight, from before jobs",
|
||||
["EmailRecovery"] = "recovery codes live an hour",
|
||||
["openiddict.tokens"] = "sessions: a restore ends every one",
|
||||
["openiddict.authorizations"] = "sessions: a restore ends every one",
|
||||
[nameof(MaintenanceLock)] = "who is backing up or restoring now",
|
||||
["AppConfiguration"] = "the configuration's copy holds the SMTP password, and is made again from appsettings at boot",
|
||||
[nameof(RestoreRecord)] = "what restores did outlives what they restore",
|
||||
["PersonaArchive"] = "personas' archives being made or imported belong to the moment"
|
||||
};
|
||||
|
||||
static readonly JsonWriterSettings Json = new() { OutputMode = JsonOutputMode.CanonicalExtendedJson, Indent = false };
|
||||
|
||||
// 2770: the service (www-data) and the deploy (in www-data's group) each read and rotate what the other wrote, and new
|
||||
// files take the directory's group
|
||||
const UnixFileMode DirectoryMode = UnixFileMode.UserRead | UnixFileMode.UserWrite | UnixFileMode.UserExecute
|
||||
| UnixFileMode.GroupRead | UnixFileMode.GroupWrite | UnixFileMode.GroupExecute | UnixFileMode.SetGroup;
|
||||
const UnixFileMode FileMode0640 = UnixFileMode.UserRead | UnixFileMode.UserWrite | UnixFileMode.GroupRead;
|
||||
|
||||
/// <summary>Takes the maintenance lock and backs the server up: the backup, or why not.</summary>
|
||||
public static async Task<(BackupInfo Backup, string Error)> Create(BackupContext context, string kind, bool dbOnly, CancellationToken token)
|
||||
{
|
||||
await using var held = await MaintenanceLock.Take("backup", token);
|
||||
if (held == default)
|
||||
return (default, "a backup or a restore is already running");
|
||||
return await CreateHeld(context, kind, dbOnly, token);
|
||||
}
|
||||
|
||||
/// <summary>Backs the server up with the maintenance lock already held (a restore's own backup, taken first).</summary>
|
||||
public static async Task<(BackupInfo Backup, string Error)> CreateHeld(BackupContext context, string kind, bool dbOnly, CancellationToken token)
|
||||
{
|
||||
var database = context.Database;
|
||||
var names = (await (await database.ListCollectionNamesAsync(cancellationToken: token)).ToListAsync(token))
|
||||
.Where(n => !n.StartsWith("system.", StringComparison.Ordinal))
|
||||
.OrderBy(n => n, StringComparer.Ordinal)
|
||||
.ToList();
|
||||
MakeDirectory(context.BackupsRoot);
|
||||
var stats = await database.RunCommandAsync<BsonDocument>(new BsonDocument("dbStats", 1), cancellationToken: token);
|
||||
var needed = (long)(stats.GetValue("dataSize", 0).ToDouble() * 1.1);
|
||||
if (new DriveInfo(Path.GetFullPath(context.BackupsRoot)).AvailableFreeSpace < needed)
|
||||
return (default, $"not enough free disk for a backup: about {needed / 1024 / 1024} MB needed");
|
||||
|
||||
var id = $"{DateTime.UtcNow.ToString("yyyyMMdd-HHmmss", CultureInfo.InvariantCulture)}-{kind}";
|
||||
var partial = Path.Combine(context.BackupsRoot, id + PartialSuffix);
|
||||
MakeDirectory(partial);
|
||||
MakeDirectory(Path.Combine(partial, "db"));
|
||||
try
|
||||
{
|
||||
var manifest = new ArchiveManifest
|
||||
{
|
||||
Kind = kind,
|
||||
Host = context.Host,
|
||||
AppCommit = BuildInfo.Commit,
|
||||
AppRef = BuildInfo.Ref,
|
||||
DbOnly = dbOnly
|
||||
};
|
||||
var replica = await MongoTopology.IsReplicaSet(database, token);
|
||||
var media = new SortedSet<string>(StringComparer.Ordinal);
|
||||
using var session = replica ? await database.Client.StartSessionAsync(new ClientSessionOptions { Snapshot = true }, token) : default;
|
||||
var window = replica ? await RaiseSnapshotWindow(database, token) : default(int?);
|
||||
try
|
||||
{
|
||||
foreach (var name in names)
|
||||
{
|
||||
if (Excluded.TryGetValue(name, out var why))
|
||||
{
|
||||
manifest.Excluded.Add(new ArchiveManifest.ExcludedEntry { Name = name, Why = why });
|
||||
continue;
|
||||
}
|
||||
manifest.Collections.Add(await Dump(database, session, name, partial, name == "MediaAttachment" ? media : default, token));
|
||||
}
|
||||
(manifest.NewestMigration, manifest.MigrationNumber) = await NewestMigration(database, session, token);
|
||||
}
|
||||
finally
|
||||
{
|
||||
if (window is { } previous)
|
||||
await SetSnapshotWindow(database, previous, CancellationToken.None);
|
||||
}
|
||||
manifest.Consistent = replica;
|
||||
|
||||
// the files it holds (or, db-only, lists); a row whose file was already gone is only counted
|
||||
foreach (var relative in media)
|
||||
{
|
||||
var source = new[] { context.MediaRoot, context.TrashRoot }.Select(root => Inside(root, relative)).FirstOrDefault(File.Exists);
|
||||
if (source == default)
|
||||
{
|
||||
manifest.Media.Missing++;
|
||||
continue;
|
||||
}
|
||||
if (!dbOnly)
|
||||
HardLink.LinkOrCopy(source, Inside(Path.Combine(partial, "media"), relative));
|
||||
manifest.Media.List.Add(relative);
|
||||
manifest.Media.Files++;
|
||||
manifest.Media.Bytes += new FileInfo(source).Length;
|
||||
}
|
||||
|
||||
manifest.Write(partial);
|
||||
Directory.Move(partial, Path.Combine(context.BackupsRoot, id));
|
||||
Retain(context.BackupsRoot, context.Options);
|
||||
return (Info(context.BackupsRoot, id), default);
|
||||
}
|
||||
catch
|
||||
{
|
||||
if (Directory.Exists(partial))
|
||||
Directory.Delete(partial, recursive: true);
|
||||
throw;
|
||||
}
|
||||
}
|
||||
|
||||
// a collection read as raw BSON in batches, each document a line of canonical Extended JSON, gzipped; the media rows'
|
||||
// files collected on the way
|
||||
static async Task<ArchiveManifest.CollectionEntry> Dump(IMongoDatabase database, IClientSessionHandle session, string name, string directory,
|
||||
ISet<string> media, CancellationToken token)
|
||||
{
|
||||
var collection = database.GetCollection<RawBsonDocument>(name);
|
||||
var path = Path.Combine(directory, "db", name + ".jsonl.gz");
|
||||
var count = 0L;
|
||||
await using (var file = NewFile(path))
|
||||
await using (var gzip = new GZipStream(file, CompressionLevel.Fastest))
|
||||
await using (var writer = new StreamWriter(gzip, new UTF8Encoding(false)))
|
||||
{
|
||||
var options = new FindOptions<RawBsonDocument> { BatchSize = 1000 };
|
||||
using var cursor = session == default
|
||||
? await collection.FindAsync(FilterDefinition<RawBsonDocument>.Empty, options, token)
|
||||
: await collection.FindAsync(session, FilterDefinition<RawBsonDocument>.Empty, options, token);
|
||||
while (await cursor.MoveNextAsync(token))
|
||||
foreach (var document in cursor.Current)
|
||||
using (document)
|
||||
{
|
||||
await writer.WriteLineAsync(document.ToJson(Json));
|
||||
count++;
|
||||
if (media != default)
|
||||
Collect(document, media);
|
||||
}
|
||||
}
|
||||
var indexes = await (await collection.Indexes.ListAsync(token)).ToListAsync(token);
|
||||
return new ArchiveManifest.CollectionEntry
|
||||
{
|
||||
Name = name,
|
||||
Count = count,
|
||||
Bytes = new FileInfo(path).Length,
|
||||
Sha256 = await Hash(path, token),
|
||||
Indexes = [.. indexes.Select(i => i.ToJson(Json))]
|
||||
};
|
||||
}
|
||||
|
||||
// a media row's files, unless it is trashed
|
||||
static void Collect(RawBsonDocument row, ISet<string> media)
|
||||
{
|
||||
if (row.TryGetValue("TrashedAt", out var trashed) && !trashed.IsBsonNull)
|
||||
return;
|
||||
foreach (var field in new[] { "FilePath", "PreviewPath" })
|
||||
if (row.TryGetValue(field, out var value) && value.IsString && Safe(value.AsString))
|
||||
media.Add(value.AsString);
|
||||
}
|
||||
|
||||
// MongoDB.Entities records each migration run with its class's number (_016_... is 16) and its name in words
|
||||
static async Task<(string Name, int Number)> NewestMigration(IMongoDatabase database, IClientSessionHandle session, CancellationToken token)
|
||||
{
|
||||
var history = database.GetCollection<BsonDocument>("_migration_history_");
|
||||
var options = new FindOptions<BsonDocument> { Sort = new BsonDocument("Number", -1), Limit = 1 };
|
||||
var newest = session == default
|
||||
? await (await history.FindAsync(FilterDefinition<BsonDocument>.Empty, options, token)).FirstOrDefaultAsync(token)
|
||||
: await (await history.FindAsync(session, FilterDefinition<BsonDocument>.Empty, options, token)).FirstOrDefaultAsync(token);
|
||||
return newest == default ? default : (newest.GetValue("Name", BsonNull.Value).ToString(), newest.GetValue("Number", 0).ToInt32());
|
||||
}
|
||||
|
||||
/// <summary>The newest migration this build has: a backup made by a newer one can't be restored by it.</summary>
|
||||
public static int CodeMigration() => typeof(ServerBackup).Assembly.GetTypes()
|
||||
.Where(t => typeof(MongoDB.Entities.IMigration).IsAssignableFrom(t) && !t.IsAbstract)
|
||||
.Select(t => int.TryParse(new string(t.Name.TrimStart('_').TakeWhile(char.IsDigit).ToArray()), out var number) ? number : 0)
|
||||
.DefaultIfEmpty(0)
|
||||
.Max();
|
||||
|
||||
// how long a snapshot stays readable: raised only while a backup reads (a longer window keeps old versions in the
|
||||
// small cache all day); the value it had, to set back
|
||||
static async Task<int?> RaiseSnapshotWindow(IMongoDatabase database, CancellationToken token)
|
||||
{
|
||||
var admin = database.Client.GetDatabase("admin");
|
||||
try
|
||||
{
|
||||
var current = await admin.RunCommandAsync<BsonDocument>(new BsonDocument { { "getParameter", 1 }, { "minSnapshotHistoryWindowInSeconds", 1 } }, cancellationToken: token);
|
||||
var previous = current.GetValue("minSnapshotHistoryWindowInSeconds", 300).ToInt32();
|
||||
await SetSnapshotWindow(database, Math.Max(previous, 3600), token);
|
||||
return previous;
|
||||
}
|
||||
catch (MongoCommandException)
|
||||
{
|
||||
return default;
|
||||
}
|
||||
}
|
||||
|
||||
static async Task SetSnapshotWindow(IMongoDatabase database, int seconds, CancellationToken token) =>
|
||||
await database.Client.GetDatabase("admin").RunCommandAsync<BsonDocument>(
|
||||
new BsonDocument { { "setParameter", 1 }, { "minSnapshotHistoryWindowInSeconds", seconds } }, cancellationToken: token);
|
||||
|
||||
/// <summary>The backups kept, newest first (not one still being written).</summary>
|
||||
public static List<BackupInfo> List(string backupsRoot)
|
||||
{
|
||||
if (!Directory.Exists(backupsRoot))
|
||||
return [];
|
||||
return Directory.EnumerateDirectories(backupsRoot)
|
||||
.Select(Path.GetFileName)
|
||||
.Where(name => !name.EndsWith(PartialSuffix, StringComparison.Ordinal) && File.Exists(Path.Combine(backupsRoot, name, ArchiveManifest.FileName)))
|
||||
.Select(name => Info(backupsRoot, name))
|
||||
.OrderByDescending(b => b.CreatedAt)
|
||||
.ToList();
|
||||
}
|
||||
|
||||
public static BackupInfo Find(string backupsRoot, string id) =>
|
||||
Safe(id) && !id.Contains('/') && Directory.Exists(Path.Combine(backupsRoot, id)) && !id.EndsWith(PartialSuffix, StringComparison.Ordinal)
|
||||
? Info(backupsRoot, id)
|
||||
: default;
|
||||
|
||||
static BackupInfo Info(string backupsRoot, string id)
|
||||
{
|
||||
var directory = Path.Combine(backupsRoot, id);
|
||||
var manifest = ArchiveManifest.Read(directory);
|
||||
var bytes = Directory.EnumerateFiles(Path.Combine(directory, "db")).Sum(f => new FileInfo(f).Length);
|
||||
return new BackupInfo(id, manifest?.Kind, manifest?.CreatedAt ?? Directory.GetCreationTimeUtc(directory), bytes, manifest);
|
||||
}
|
||||
|
||||
/// <summary>Whether every file of a backup is what its manifest says: the problems found (none when whole).</summary>
|
||||
public static async Task<List<string>> Verify(string backupsRoot, string id, CancellationToken token)
|
||||
{
|
||||
var problems = new List<string>();
|
||||
var backup = Find(backupsRoot, id);
|
||||
if (backup?.Manifest == default)
|
||||
return ["no such backup, or no manifest"];
|
||||
var directory = Path.Combine(backupsRoot, id);
|
||||
foreach (var collection in backup.Manifest.Collections)
|
||||
{
|
||||
var path = Path.Combine(directory, "db", collection.Name + ".jsonl.gz");
|
||||
if (!File.Exists(path))
|
||||
problems.Add($"{collection.Name}: missing");
|
||||
else if (await Hash(path, token) != collection.Sha256)
|
||||
problems.Add($"{collection.Name}: altered");
|
||||
}
|
||||
if (!backup.Manifest.DbOnly)
|
||||
foreach (var relative in backup.Manifest.Media.List.Where(r => !File.Exists(Inside(Path.Combine(directory, "media"), r))))
|
||||
problems.Add($"media {relative}: missing");
|
||||
return problems;
|
||||
}
|
||||
|
||||
/// <summary>Deletes a backup (its media links go; the live files stay).</summary>
|
||||
public static bool Delete(string backupsRoot, string id)
|
||||
{
|
||||
if (Find(backupsRoot, id) == default)
|
||||
return false;
|
||||
Directory.Delete(Path.Combine(backupsRoot, id), recursive: true);
|
||||
return true;
|
||||
}
|
||||
|
||||
// what is kept: the newest nightly ones for KeepDaily days and the newest of each of KeepWeekly weeks before, the
|
||||
// newest pre-deploy and pre-restore ones; manual and uploaded ones until deleted; a backup that died writing goes
|
||||
public static void Retain(string backupsRoot, BackupOptions options)
|
||||
{
|
||||
var all = List(backupsRoot);
|
||||
var nightly = all.Where(b => b.Kind == "nightly").OrderByDescending(b => b.CreatedAt).ToList();
|
||||
var kept = nightly.Take(options.KeepDaily).ToHashSet();
|
||||
foreach (var week in nightly.Skip(options.KeepDaily).GroupBy(b => (ISOWeek.GetYear(b.CreatedAt), ISOWeek.GetWeekOfYear(b.CreatedAt))).Take(options.KeepWeekly))
|
||||
kept.Add(week.First());
|
||||
var doomed = nightly.Where(b => !kept.Contains(b))
|
||||
.Concat(all.Where(b => b.Kind == "pre-deploy").OrderByDescending(b => b.CreatedAt).Skip(options.KeepPreDeploy))
|
||||
.Concat(all.Where(b => b.Kind == "pre-restore").OrderByDescending(b => b.CreatedAt).Skip(options.KeepPreRestore));
|
||||
var stale = Directory.EnumerateDirectories(backupsRoot, "*" + PartialSuffix).Where(p => Directory.GetLastWriteTimeUtc(p) < DateTime.UtcNow.AddDays(-1));
|
||||
foreach (var directory in doomed.Select(b => Path.Combine(backupsRoot, b.Id)).Concat(stale).ToList())
|
||||
try
|
||||
{
|
||||
Directory.Delete(directory, recursive: true);
|
||||
}
|
||||
catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
|
||||
{
|
||||
//another user's backup this one may not delete: the next rotation tries again
|
||||
}
|
||||
}
|
||||
|
||||
// a relative path from a database row or a manifest, never outside its root
|
||||
static bool Safe(string relative) =>
|
||||
!string.IsNullOrEmpty(relative) && !Path.IsPathRooted(relative) && !relative.Split('/', '\\').Any(part => part is ".." or ".");
|
||||
|
||||
public static string Inside(string root, string relative)
|
||||
{
|
||||
var full = Path.GetFullPath(Path.Combine(root, relative));
|
||||
if (!Safe(relative) || !full.StartsWith(Path.GetFullPath(root) + Path.DirectorySeparatorChar, StringComparison.Ordinal))
|
||||
throw new InvalidOperationException($"{relative} is not inside {root}");
|
||||
return full;
|
||||
}
|
||||
|
||||
static async Task<string> Hash(string path, CancellationToken token)
|
||||
{
|
||||
await using var file = File.OpenRead(path);
|
||||
return Convert.ToHexStringLower(await SHA256.HashDataAsync(file, token));
|
||||
}
|
||||
|
||||
static FileStream NewFile(string path) => OperatingSystem.IsWindows()
|
||||
? new FileStream(path, FileMode.CreateNew, FileAccess.Write)
|
||||
: new FileStream(path, new FileStreamOptions { Mode = FileMode.CreateNew, Access = FileAccess.Write, UnixCreateMode = FileMode0640 });
|
||||
|
||||
// set after creating, since the umask would take the group's write away
|
||||
public static void MakeDirectory(string path)
|
||||
{
|
||||
var existed = Directory.Exists(path);
|
||||
Directory.CreateDirectory(path);
|
||||
if (!existed && !OperatingSystem.IsWindows())
|
||||
File.SetUnixFileMode(path, DirectoryMode);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,247 @@
|
||||
using MongoDB.Bson;
|
||||
using MongoDB.Driver;
|
||||
|
||||
using System.IO.Compression;
|
||||
|
||||
namespace PrivaPub.Infrastructure.Backup
|
||||
{
|
||||
public enum RestoreOutcome
|
||||
{
|
||||
None,//nothing was asked
|
||||
Restored,
|
||||
Abandoned,//refused before anything changed: the server boots as it was
|
||||
Failed,//failed midway: the next boot tries again from the start
|
||||
GaveUp//failed MaxAttempts times: the server stays down until someone looks
|
||||
}
|
||||
|
||||
// A backup restored (owner decision 2026-10-07: from the CLI or the administrator's page). Asking writes restore.json
|
||||
// and the running service stops at once (RestoreWatcher); the restore itself runs at the next boot, before migrations,
|
||||
// indexes and every hosted service, with nothing else touching the database:
|
||||
// 1. a pre-restore backup P is taken, once (a retry reuses it);
|
||||
// 2. every collection the backup holds is dropped and imported raw, with its indexes; every other one is dropped,
|
||||
// except what a backup never holds (ServerBackup.Excluded), which stays as it is;
|
||||
// 3. media files the live directory lacks come back from the backup (or the trash); a restore deletes no file;
|
||||
// 4. ProtectiveMerge brings back from P every protective act made since the backup;
|
||||
// 5. every session ends, every server's circuit closes, and a RestoreRecord tells what happened.
|
||||
// Each attempt redoes everything, so one that dies midway converges on the next.
|
||||
public static class ServerRestore
|
||||
{
|
||||
static readonly TimeSpan LockWait = TimeSpan.FromMinutes(10);
|
||||
|
||||
/// <summary>Why a backup can't be restored here: none when it can.</summary>
|
||||
public static async Task<List<string>> Check(BackupContext context, string id, CancellationToken token)
|
||||
{
|
||||
var backup = ServerBackup.Find(context.BackupsRoot, id);
|
||||
if (backup?.Manifest == default)
|
||||
return ["no such backup"];
|
||||
var manifest = backup.Manifest;
|
||||
var problems = new List<string>();
|
||||
if (manifest.Format != ArchiveManifest.CurrentFormat)
|
||||
problems.Add($"its format is {manifest.Format}, this build reads {ArchiveManifest.CurrentFormat}");
|
||||
if (!string.Equals(manifest.Host?.TrimEnd('/'), context.Host, StringComparison.OrdinalIgnoreCase))
|
||||
problems.Add($"it was made by {manifest.Host}, not {context.Host}: every id and address in it names its host");
|
||||
var code = ServerBackup.CodeMigration();
|
||||
if (manifest.MigrationNumber > code)
|
||||
problems.Add($"it was made by a newer build (migration {manifest.MigrationNumber}; this one knows {code})");
|
||||
problems.AddRange(await ServerBackup.Verify(context.BackupsRoot, id, token));
|
||||
return problems;
|
||||
}
|
||||
|
||||
/// <summary>Asks for a backup to be restored at the next boot: why not, or null once asked.</summary>
|
||||
public static async Task<string> Request(BackupContext context, string id, string requestedBy, CancellationToken token)
|
||||
{
|
||||
var problems = await Check(context, id, token);
|
||||
if (problems.Count > 0)
|
||||
return string.Join("; ", problems.Take(5)) + (problems.Count > 5 ? $"; and {problems.Count - 5} more" : string.Empty);
|
||||
var waiting = RestoreMarker.Read(context.BackupsRoot);
|
||||
if (waiting != default && waiting.Attempts < RestoreMarker.MaxAttempts)
|
||||
return $"the restore of {waiting.Backup} is already {waiting.State}";
|
||||
if (await MaintenanceLock.Current(token) is { } held)
|
||||
return $"a {held.What} is running";
|
||||
new RestoreMarker { Backup = id, RequestedBy = requestedBy }.Write(context.BackupsRoot);
|
||||
return default;
|
||||
}
|
||||
|
||||
/// <summary>The restore asked for, carried out (at boot, before migrations): what came of it.</summary>
|
||||
public static async Task<RestoreOutcome> ApplyPending(BackupContext context, ILogger logger, CancellationToken token)
|
||||
{
|
||||
var marker = RestoreMarker.Read(context.BackupsRoot);
|
||||
if (marker == default)
|
||||
return RestoreOutcome.None;
|
||||
if (marker.Attempts >= RestoreMarker.MaxAttempts)
|
||||
{
|
||||
logger.LogCritical("The restore of {Backup} failed {Attempts} times ({Error}). The database may be half restored; {PreRestore} holds it as it was before. Restore that backup or another (PrivaPub admin restore <id>), or delete {Marker} to boot as it is",
|
||||
marker.Backup, marker.Attempts, marker.Error, marker.PreRestore ?? "no backup", RestoreMarker.PathIn(context.BackupsRoot));
|
||||
return RestoreOutcome.GaveUp;
|
||||
}
|
||||
|
||||
await using var held = await TakeLock(token);
|
||||
if (held == default)
|
||||
return await Abandon(context, marker, "a backup kept the maintenance lock for ten minutes", logger, token);
|
||||
|
||||
var problems = await Check(context, marker.Backup, token);
|
||||
if (problems.Count > 0 && marker.PreRestore == default)
|
||||
return await Abandon(context, marker, string.Join("; ", problems), logger, token);
|
||||
|
||||
if (marker.PreRestore == default)
|
||||
{
|
||||
var (taken, error) = await ServerBackup.CreateHeld(context, "pre-restore", dbOnly: false, token);
|
||||
if (taken == default)
|
||||
return await Abandon(context, marker, $"the pre-restore backup could not be made: {error}", logger, token);
|
||||
marker.PreRestore = taken.Id;
|
||||
}
|
||||
marker.State = "running";
|
||||
marker.Attempts++;
|
||||
marker.Error = default;
|
||||
marker.Write(context.BackupsRoot);
|
||||
logger.LogWarning("Restoring {Backup} (attempt {Attempt}); the server as it was is {PreRestore}", marker.Backup, marker.Attempts, marker.PreRestore);
|
||||
|
||||
try
|
||||
{
|
||||
if (problems.Count > 0)
|
||||
throw new InvalidOperationException(string.Join("; ", problems));
|
||||
var backup = ServerBackup.Find(context.BackupsRoot, marker.Backup);
|
||||
var report = new RestoreReport();
|
||||
await Import(context, backup, report, token);
|
||||
Media(context, backup, report);
|
||||
await ProtectiveMerge.Apply(context.Database, Path.Combine(context.BackupsRoot, marker.PreRestore, "db"), report, token);
|
||||
await Record(context, new RestoreRecord
|
||||
{
|
||||
Backup = backup.Id,
|
||||
BackupCreatedAt = backup.CreatedAt,
|
||||
PreRestore = marker.PreRestore,
|
||||
RequestedBy = marker.RequestedBy,
|
||||
RequestedAt = marker.RequestedAt,
|
||||
Attempts = marker.Attempts,
|
||||
Report = report
|
||||
}, token);
|
||||
RestoreRecord.Forget();
|
||||
RestoreMarker.Clear(context.BackupsRoot);
|
||||
logger.LogWarning("Restored {Backup}: {Documents} documents in {Collections} collections, {Media} media files brought back, {Tombstoned} accounts made since deleted",
|
||||
backup.Id, report.Documents, report.Collections, report.MediaRestored, report.RootsTombstoned.Count + report.PersonasTombstoned.Count + report.GroupsTombstoned.Count);
|
||||
return RestoreOutcome.Restored;
|
||||
}
|
||||
catch (Exception ex) when (ex is not OperationCanceledException)
|
||||
{
|
||||
marker.Error = ex.Message;
|
||||
marker.Write(context.BackupsRoot);
|
||||
logger.LogError(ex, "The restore of {Backup} failed (attempt {Attempt} of {Max})", marker.Backup, marker.Attempts, RestoreMarker.MaxAttempts);
|
||||
return marker.Attempts >= RestoreMarker.MaxAttempts ? RestoreOutcome.GaveUp : RestoreOutcome.Failed;
|
||||
}
|
||||
}
|
||||
|
||||
static async Task<MaintenanceLock.Held> TakeLock(CancellationToken token)
|
||||
{
|
||||
var until = DateTime.UtcNow + LockWait;
|
||||
while (true)
|
||||
{
|
||||
var held = await MaintenanceLock.Take("restore", token);
|
||||
if (held != default || DateTime.UtcNow > until)
|
||||
return held;
|
||||
await Task.Delay(TimeSpan.FromSeconds(5), token);
|
||||
}
|
||||
}
|
||||
|
||||
// refused before anything changed: recorded for the administrator's page, and the server boots as it was
|
||||
static async Task<RestoreOutcome> Abandon(BackupContext context, RestoreMarker marker, string why, ILogger logger, CancellationToken token)
|
||||
{
|
||||
logger.LogError("The restore of {Backup} was abandoned, nothing changed: {Why}", marker.Backup, why);
|
||||
await Record(context, new RestoreRecord
|
||||
{
|
||||
Backup = marker.Backup,
|
||||
RequestedBy = marker.RequestedBy,
|
||||
RequestedAt = marker.RequestedAt,
|
||||
Attempts = marker.Attempts,
|
||||
Abandoned = true,
|
||||
Error = why
|
||||
}, token);
|
||||
RestoreMarker.Clear(context.BackupsRoot);
|
||||
return RestoreOutcome.Abandoned;
|
||||
}
|
||||
|
||||
// in the database restored (a backup never holds these records: they outlive what they restore)
|
||||
static async Task Record(BackupContext context, RestoreRecord record, CancellationToken token)
|
||||
{
|
||||
record.ID = ObjectId.GenerateNewId().ToString();
|
||||
await context.Database.GetCollection<RestoreRecord>(nameof(RestoreRecord)).InsertOneAsync(record, cancellationToken: token);
|
||||
}
|
||||
|
||||
// each collection dropped and imported as the backup holds it, its indexes made again; the others dropped
|
||||
static async Task Import(BackupContext context, BackupInfo backup, RestoreReport report, CancellationToken token)
|
||||
{
|
||||
var database = context.Database;
|
||||
var directory = Path.Combine(context.BackupsRoot, backup.Id, "db");
|
||||
foreach (var entry in backup.Manifest.Collections)
|
||||
{
|
||||
await database.DropCollectionAsync(entry.Name, token);
|
||||
await database.CreateCollectionAsync(entry.Name, cancellationToken: token);
|
||||
report.Documents += await Insert(database.GetCollection<BsonDocument>(entry.Name), Read(Path.Combine(directory, entry.Name + ".jsonl.gz")), token);
|
||||
var indexes = entry.Indexes.Select(BsonDocument.Parse).Where(i => i.GetValue("name", "").AsString != "_id_").ToList();
|
||||
foreach (var index in indexes)
|
||||
index.Remove("ns");
|
||||
if (indexes.Count > 0)
|
||||
await database.RunCommandAsync<BsonDocument>(new BsonDocument { { "createIndexes", entry.Name }, { "indexes", new BsonArray(indexes) } }, cancellationToken: token);
|
||||
report.Collections++;
|
||||
}
|
||||
var held = backup.Manifest.Collections.Select(c => c.Name).ToHashSet(StringComparer.Ordinal);
|
||||
foreach (var name in await (await database.ListCollectionNamesAsync(cancellationToken: token)).ToListAsync(token))
|
||||
{
|
||||
if (held.Contains(name) || ServerBackup.Excluded.ContainsKey(name) || name.StartsWith("system.", StringComparison.Ordinal))
|
||||
continue;
|
||||
await database.DropCollectionAsync(name, token);
|
||||
report.CollectionsDropped++;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Documents inserted in batches, unordered and unvalidated, as they were: how many.</summary>
|
||||
public static async Task<long> Insert(IMongoCollection<BsonDocument> collection, IEnumerable<BsonDocument> documents, CancellationToken token)
|
||||
{
|
||||
var count = 0L;
|
||||
foreach (var batch in documents.Chunk(1000))
|
||||
{
|
||||
await collection.InsertManyAsync(batch, new InsertManyOptions { IsOrdered = false, BypassDocumentValidation = true }, token);
|
||||
count += batch.Length;
|
||||
}
|
||||
return count;
|
||||
}
|
||||
|
||||
/// <summary>A collection's documents in a backup, one by one; none when it has no file.</summary>
|
||||
public static IEnumerable<BsonDocument> Read(string file)
|
||||
{
|
||||
if (!File.Exists(file))
|
||||
yield break;
|
||||
using var gzip = new GZipStream(File.OpenRead(file), CompressionMode.Decompress);
|
||||
using var reader = new StreamReader(gzip);
|
||||
while (reader.ReadLine() is { } line)
|
||||
if (line.Length > 0)
|
||||
yield return BsonDocument.Parse(line);
|
||||
}
|
||||
|
||||
// the backup's media files the live directory lacks: linked from the backup, or moved back from the trash
|
||||
static void Media(BackupContext context, BackupInfo backup, RestoreReport report)
|
||||
{
|
||||
var kept = Path.Combine(context.BackupsRoot, backup.Id, "media");
|
||||
foreach (var relative in backup.Manifest.Media.List)
|
||||
{
|
||||
var live = ServerBackup.Inside(context.MediaRoot, relative);
|
||||
if (File.Exists(live))
|
||||
continue;
|
||||
var fromBackup = ServerBackup.Inside(kept, relative);
|
||||
var fromTrash = ServerBackup.Inside(context.TrashRoot, relative);
|
||||
if (File.Exists(fromBackup))
|
||||
HardLink.LinkOrCopy(fromBackup, live);
|
||||
else if (File.Exists(fromTrash))
|
||||
{
|
||||
Directory.CreateDirectory(Path.GetDirectoryName(live)!);
|
||||
File.Move(fromTrash, live);
|
||||
}
|
||||
else
|
||||
{
|
||||
report.MediaMissing++;
|
||||
continue;
|
||||
}
|
||||
report.MediaRestored++;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,164 @@
|
||||
using System.Collections.Concurrent;
|
||||
using System.Security.Cryptography;
|
||||
using System.Text.Json;
|
||||
|
||||
namespace PrivaPub.Infrastructure.Backup
|
||||
{
|
||||
// Backups leaving and arriving through the administrator's page, never through the client's memory:
|
||||
// - a download is a link with a ticket in its path, good for ten minutes for one backup (a browser downloads it, and
|
||||
// resumes it with Range), given for the administrator's password;
|
||||
// - an upload arrives in pieces appended at the offset already received (a piece that would leave a gap or overlap is
|
||||
// refused with what was received, so a broken upload resumes), into <backups>/.uploads, and is read into a backup
|
||||
// once whole. One left for a day is deleted.
|
||||
public class TransferStore(Backups backups)
|
||||
{
|
||||
public static readonly TimeSpan TicketLifetime = TimeSpan.FromMinutes(10);
|
||||
public const long ChunkBytes = 32L * 1024 * 1024;
|
||||
const string Uploads = ".uploads";
|
||||
|
||||
readonly ConcurrentDictionary<string, (string Backup, DateTime Expires)> _tickets = new();
|
||||
readonly ConcurrentDictionary<string, SemaphoreSlim> _appending = new();
|
||||
|
||||
public (string Ticket, DateTime Expires) Ticket(string backup)
|
||||
{
|
||||
foreach (var (ticket, stale) in _tickets.Where(t => t.Value.Expires < DateTime.UtcNow).ToList())
|
||||
_tickets.TryRemove(ticket, out _);
|
||||
var expires = DateTime.UtcNow + TicketLifetime;
|
||||
var token = Convert.ToHexStringLower(RandomNumberGenerator.GetBytes(32));
|
||||
_tickets[token] = (backup, expires);
|
||||
return (token, expires);
|
||||
}
|
||||
|
||||
/// <summary>The backup a ticket is for, while it is good.</summary>
|
||||
public string Redeem(string ticket) =>
|
||||
ticket != default && _tickets.TryGetValue(ticket, out var held) && held.Expires > DateTime.UtcNow ? held.Backup : default;
|
||||
|
||||
string Folder => Path.Combine(backups.Root, Uploads);
|
||||
|
||||
string File(string id) => Path.Combine(Folder, id + ".tar");
|
||||
|
||||
static bool IsId(string id) => id is { Length: 32 } && id.All(char.IsAsciiHexDigitLower);
|
||||
|
||||
public string Start(string by)
|
||||
{
|
||||
ServerBackup.MakeDirectory(backups.Root);
|
||||
ServerBackup.MakeDirectory(Folder);
|
||||
Forget(DateTime.UtcNow.AddDays(-1));
|
||||
var id = Convert.ToHexStringLower(RandomNumberGenerator.GetBytes(16));
|
||||
using (OperatingSystem.IsWindows()
|
||||
? new FileStream(File(id), FileMode.CreateNew, FileAccess.Write)
|
||||
: new FileStream(File(id), new FileStreamOptions { Mode = FileMode.CreateNew, Access = FileAccess.Write,
|
||||
UnixCreateMode = UnixFileMode.UserRead | UnixFileMode.UserWrite | UnixFileMode.GroupRead }))
|
||||
{
|
||||
}
|
||||
System.IO.File.WriteAllText(Path.Combine(Folder, id + ".json"), JsonSerializer.Serialize(new { by, at = DateTime.UtcNow }));
|
||||
return id;
|
||||
}
|
||||
|
||||
/// <summary>Who started an upload (an administrator's name, or the root a persona's archive belongs to).</summary>
|
||||
public string Owner(string id)
|
||||
{
|
||||
if (!IsId(id) || !System.IO.File.Exists(Path.Combine(Folder, id + ".json")))
|
||||
return default;
|
||||
try
|
||||
{
|
||||
return JsonDocument.Parse(System.IO.File.ReadAllText(Path.Combine(Folder, id + ".json"))).RootElement.GetProperty("by").GetString();
|
||||
}
|
||||
catch (JsonException)
|
||||
{
|
||||
return default;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>A whole upload moved where it is read from: its new path, or null when there is no such upload.</summary>
|
||||
public string Take(string id, string folder, string name)
|
||||
{
|
||||
if (!IsId(id) || !System.IO.File.Exists(File(id)))
|
||||
return default;
|
||||
ServerBackup.MakeDirectory(folder);
|
||||
var path = Path.Combine(folder, name);
|
||||
System.IO.File.Move(File(id), path, overwrite: true);
|
||||
System.IO.File.Delete(Path.Combine(Folder, id + ".json"));
|
||||
_appending.TryRemove(id, out _);
|
||||
return path;
|
||||
}
|
||||
|
||||
/// <summary>How much of an upload has arrived; -1 when there is no such upload.</summary>
|
||||
public long Received(string id) => IsId(id) && System.IO.File.Exists(File(id)) ? new FileInfo(File(id)).Length : -1;
|
||||
|
||||
/// <summary>A piece appended where the upload stands: what has arrived since, and whether it was taken.</summary>
|
||||
public async Task<(long Received, bool Taken)> Append(string id, long offset, Stream piece, CancellationToken token)
|
||||
{
|
||||
if (!IsId(id) || !System.IO.File.Exists(File(id)))
|
||||
return (-1, false);
|
||||
var gate = _appending.GetOrAdd(id, _ => new SemaphoreSlim(1, 1));
|
||||
await gate.WaitAsync(token);
|
||||
try
|
||||
{
|
||||
await using var file = new FileStream(File(id), FileMode.Open, FileAccess.Write);
|
||||
if (file.Length != offset)
|
||||
return (file.Length, false);
|
||||
file.Seek(offset, SeekOrigin.Begin);
|
||||
var buffer = new byte[81920];
|
||||
var total = 0L;
|
||||
int read;
|
||||
while ((read = await piece.ReadAsync(buffer, token)) > 0)
|
||||
{
|
||||
total += read;
|
||||
if (total > ChunkBytes)
|
||||
{
|
||||
file.SetLength(offset);
|
||||
return (offset, false);
|
||||
}
|
||||
await file.WriteAsync(buffer.AsMemory(0, read), token);
|
||||
}
|
||||
await file.FlushAsync(token);
|
||||
return (file.Length, true);
|
||||
}
|
||||
catch (IOException)
|
||||
{
|
||||
//the piece broke off: what arrived of it is dropped, so the next one starts where this one did
|
||||
await using var file = new FileStream(File(id), FileMode.Open, FileAccess.Write);
|
||||
file.SetLength(offset);
|
||||
return (offset, false);
|
||||
}
|
||||
finally
|
||||
{
|
||||
gate.Release();
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>The whole upload read into a backup: the backup, or why not. The upload goes either way.</summary>
|
||||
public async Task<(BackupInfo Backup, string Error)> Finish(string id, CancellationToken token)
|
||||
{
|
||||
if (!IsId(id) || !System.IO.File.Exists(File(id)))
|
||||
return (default, "no such upload");
|
||||
try
|
||||
{
|
||||
await using var tar = new FileStream(File(id), FileMode.Open, FileAccess.Read, FileShare.Read, 81920, FileOptions.SequentialScan);
|
||||
return await BackupTar.Import(tar, backups.Root, token);
|
||||
}
|
||||
finally
|
||||
{
|
||||
Cancel(id);
|
||||
}
|
||||
}
|
||||
|
||||
public bool Cancel(string id)
|
||||
{
|
||||
if (!IsId(id) || !System.IO.File.Exists(File(id)))
|
||||
return false;
|
||||
System.IO.File.Delete(File(id));
|
||||
System.IO.File.Delete(Path.Combine(Folder, id + ".json"));
|
||||
_appending.TryRemove(id, out _);
|
||||
return true;
|
||||
}
|
||||
|
||||
// uploads left behind
|
||||
void Forget(DateTime before)
|
||||
{
|
||||
foreach (var file in Directory.EnumerateFiles(Folder).Where(f => System.IO.File.GetLastWriteTimeUtc(f) < before))
|
||||
System.IO.File.Delete(file);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -3,6 +3,7 @@ using MongoDB.Entities;
|
||||
using PrivaPub.ClientModels;
|
||||
using PrivaPub.ClientModels.User;
|
||||
using PrivaPub.ClientModels.User.Avatar;
|
||||
using PrivaPub.Infrastructure.Backup;
|
||||
using PrivaPub.Models.User;
|
||||
using PrivaPub.Services;
|
||||
using PrivaPub.Services.ClientToServer.Private;
|
||||
@@ -22,8 +23,23 @@ namespace PrivaPub.Infrastructure.Cli
|
||||
usage: PrivaPub admin promote|demote <root>
|
||||
PrivaPub admin create-root <login> [--admin] the password is read from standard input
|
||||
PrivaPub admin smoke <persona> prints "<login> <password>" for the deploy's signed-in check
|
||||
PrivaPub admin media audit [--fix] media files against what holds them; --fix (as www-data) trashes
|
||||
what nothing holds and gives today's pictures their rows
|
||||
PrivaPub admin backup [--kind manual|pre-deploy] [--db-only]
|
||||
backs the server up; --db-only lists the media without
|
||||
linking them (the deploy's, which can't link www-data's files)
|
||||
PrivaPub admin backups the backups kept, newest first
|
||||
PrivaPub admin backup verify <id> whether a backup's files are what its manifest says
|
||||
PrivaPub admin restore <id> restores a backup: the service stops within seconds and
|
||||
restores it when systemd starts it again; everyone signs
|
||||
in again, and what was made since is lost, except what
|
||||
protects (blocks, deletions, ...)
|
||||
PrivaPub admin restore --status the restore waiting, and the last one done
|
||||
""";
|
||||
|
||||
/// <summary>Whether a command runs before migrations: backups are of the database as it was.</summary>
|
||||
public static bool BeforeMigrations(string[] args) => args is ["backup", ..] or ["backups"] or ["restore", ..];
|
||||
|
||||
public static async Task<int> Run(string[] args, IServiceProvider services, TextReader input = default, TextWriter output = default)
|
||||
{
|
||||
input ??= Console.In;
|
||||
@@ -36,12 +52,138 @@ namespace PrivaPub.Infrastructure.Cli
|
||||
return await CreateRoot(services, login, input.ReadLine(), flags.Contains("--admin"), output);
|
||||
case ["smoke", var persona]:
|
||||
return await Smoke(services, persona, output);
|
||||
case ["media", "audit", .. var flags] when flags.All(f => f == "--fix"):
|
||||
return await AuditMedia(services, flags.Contains("--fix"), output);
|
||||
case ["backup", "verify", var id]:
|
||||
return await VerifyBackup(services, id, output);
|
||||
case ["backup", .. var flags] when BackupFlags(flags, out var kind, out var dbOnly):
|
||||
return await BackUp(services, kind, dbOnly, output);
|
||||
case ["backups"]:
|
||||
return ListBackups(services, output);
|
||||
case ["restore", "--status"]:
|
||||
return await RestoreStatus(services, output);
|
||||
case ["restore", var id] when !id.StartsWith('-'):
|
||||
return await Restore(services, id, output);
|
||||
default:
|
||||
Console.Error.WriteLine(Usage);
|
||||
return 2;
|
||||
}
|
||||
}
|
||||
|
||||
static readonly string[] CommandKinds = ["manual", "pre-deploy"];
|
||||
|
||||
static bool BackupFlags(string[] flags, out string kind, out bool dbOnly)
|
||||
{
|
||||
kind = "manual";
|
||||
dbOnly = false;
|
||||
for (var i = 0; i < flags.Length; i++)
|
||||
switch (flags[i])
|
||||
{
|
||||
case "--db-only":
|
||||
dbOnly = true;
|
||||
break;
|
||||
case "--kind" when i + 1 < flags.Length && CommandKinds.Contains(flags[i + 1]):
|
||||
kind = flags[++i];
|
||||
break;
|
||||
default:
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
static async Task<int> BackUp(IServiceProvider services, string kind, bool dbOnly, TextWriter output)
|
||||
{
|
||||
var (made, error) = await services.GetRequiredService<Backups>().Create(kind, dbOnly, CancellationToken.None);
|
||||
if (made == default)
|
||||
{
|
||||
Console.Error.WriteLine(error);
|
||||
return 1;
|
||||
}
|
||||
var manifest = made.Manifest;
|
||||
output.WriteLine($"{made.Id}: {manifest.Collections.Count} collections, {manifest.Collections.Sum(c => c.Count)} documents, {made.Bytes / 1024} KiB");
|
||||
output.WriteLine(dbOnly
|
||||
? $"{manifest.Media.List.Count} media files listed, not linked"
|
||||
: $"{manifest.Media.Files} media files, {manifest.Media.Bytes / 1024 / 1024} MiB, {manifest.Media.Missing} missing");
|
||||
if (!manifest.Consistent)
|
||||
output.WriteLine("not read at one instant: Mongo is not a replica set");
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int ListBackups(IServiceProvider services, TextWriter output)
|
||||
{
|
||||
foreach (var backup in services.GetRequiredService<Backups>().List())
|
||||
output.WriteLine($"{backup.Id}\t{backup.Kind}\t{backup.CreatedAt:u}\t{backup.Bytes / 1024} KiB\t{backup.Manifest?.Media.Files ?? 0} media{(backup.Manifest?.DbOnly == true ? " (listed)" : string.Empty)}");
|
||||
return 0;
|
||||
}
|
||||
|
||||
static async Task<int> VerifyBackup(IServiceProvider services, string id, TextWriter output)
|
||||
{
|
||||
var problems = await services.GetRequiredService<Backups>().Verify(id, CancellationToken.None);
|
||||
foreach (var problem in problems)
|
||||
output.WriteLine(problem);
|
||||
if (problems.Count > 0)
|
||||
return 1;
|
||||
output.WriteLine($"{id}: whole");
|
||||
return 0;
|
||||
}
|
||||
|
||||
static async Task<int> Restore(IServiceProvider services, string id, TextWriter output)
|
||||
{
|
||||
var backups = services.GetRequiredService<Backups>();
|
||||
var refused = await ServerRestore.Request(backups.Context(), id, "cli", CancellationToken.None);
|
||||
if (refused != default)
|
||||
{
|
||||
Console.Error.WriteLine(refused);
|
||||
return 1;
|
||||
}
|
||||
output.WriteLine($"{id} will be restored: the running service stops within seconds and restores it when it starts again");
|
||||
output.WriteLine("(systemd starts it again; follow it with journalctl -u privapub, then PrivaPub admin restore --status)");
|
||||
return 0;
|
||||
}
|
||||
|
||||
static async Task<int> RestoreStatus(IServiceProvider services, TextWriter output)
|
||||
{
|
||||
var waiting = RestoreMarker.Read(services.GetRequiredService<Backups>().Root);
|
||||
if (waiting != default)
|
||||
output.WriteLine($"waiting: {waiting.Backup}, {waiting.State}, attempt {waiting.Attempts} of {RestoreMarker.MaxAttempts}, asked by {waiting.RequestedBy} at {waiting.RequestedAt:u}{(waiting.Error == default ? string.Empty : $": {waiting.Error}")}");
|
||||
var last = await DB.Default.Find<RestoreRecord>().Sort(r => r.RestoredAt, Order.Descending).ExecuteFirstAsync();
|
||||
if (last == default)
|
||||
{
|
||||
if (waiting == default)
|
||||
output.WriteLine("no restore asked for, none done");
|
||||
return 0;
|
||||
}
|
||||
if (last.Abandoned)
|
||||
{
|
||||
output.WriteLine($"last: {last.Backup} abandoned at {last.RestoredAt:u}, nothing changed: {last.Error}");
|
||||
return 0;
|
||||
}
|
||||
var report = last.Report;
|
||||
output.WriteLine($"last: {last.Backup} (made {last.BackupCreatedAt:u}) restored at {last.RestoredAt:u}, asked by {last.RequestedBy}; before it: {last.PreRestore}");
|
||||
output.WriteLine($" {report.Documents} documents in {report.Collections} collections, {report.MediaRestored} media files brought back, {report.MediaMissing} missing");
|
||||
output.WriteLine($" deleted again: {report.RootsDeleted} roots, {report.PersonasDeleted} personas, {report.GroupsDeleted} groups, {report.PostsDeleted} posts");
|
||||
output.WriteLine($" made since, now deleted: roots {Names(report.RootsTombstoned)}, personas {Names(report.PersonasTombstoned)}, groups {Names(report.GroupsTombstoned)}; {report.PostsGone} posts gone");
|
||||
output.WriteLine($" {report.ProtectiveKept} protective rows kept, {report.MediaTrashed} media trashed, {report.SessionsEnded} sessions ended");
|
||||
return 0;
|
||||
}
|
||||
|
||||
static string Names(List<string> names) => names.Count == 0 ? "none" : string.Join(", ", names);
|
||||
|
||||
static async Task<int> AuditMedia(IServiceProvider services, bool fix, TextWriter output)
|
||||
{
|
||||
var report = await Domain.Media.MediaAudit.Run(services.GetRequiredService<Domain.Media.IMediaService>(), fix, CancellationToken.None);
|
||||
output.WriteLine($"{report.Files} files served, {report.Held} rows holding media");
|
||||
output.WriteLine($"{report.Adopted} pictures personas show without a row{(fix ? ": given one" : string.Empty)}");
|
||||
output.WriteLine($"{report.OfDeleted} media of deleted posts or personas{(fix ? ": trashed" : string.Empty)}");
|
||||
output.WriteLine($"{report.MissingFiles} rows whose files are missing{(fix ? ": trashed" : string.Empty)}");
|
||||
output.WriteLine($"{report.Unheld} files nothing holds{(fix ? ": trashed" : string.Empty)}");
|
||||
foreach (var example in report.Examples)
|
||||
output.WriteLine($" {example}");
|
||||
if (!fix && report.Adopted + report.OfDeleted + report.MissingFiles + report.Unheld > 0)
|
||||
output.WriteLine("run again with --fix, as www-data, to apply this; trashed files are deleted after a day");
|
||||
return 0;
|
||||
}
|
||||
|
||||
static async Task<int> Promote(bool promote, string userName, TextWriter output)
|
||||
{
|
||||
userName = userName.ToLowerInvariant();
|
||||
|
||||
@@ -4,7 +4,22 @@ namespace PrivaPub.Infrastructure.Data
|
||||
{
|
||||
public static class EntityMaps
|
||||
{
|
||||
static readonly Lock Gate = new();
|
||||
static bool _warm;
|
||||
|
||||
// once per process, one map at a time: test hosts boot side by side
|
||||
public static void Warm()
|
||||
{
|
||||
lock (Gate)
|
||||
{
|
||||
if (_warm)
|
||||
return;
|
||||
Map();
|
||||
_warm = true;
|
||||
}
|
||||
}
|
||||
|
||||
static void Map()
|
||||
{
|
||||
var collection = typeof(DB).GetMethods()
|
||||
.Single(m => m.Name == nameof(DB.Collection) && m.IsGenericMethodDefinition && m.GetParameters().Length == 0);
|
||||
|
||||
@@ -31,6 +31,21 @@ namespace PrivaPub.Infrastructure.Data
|
||||
await Plain<Post>(token, p => p.ReblogOfPostId);
|
||||
await Plain<Post>(token, p => p.AnsweringToPostId);
|
||||
await DB.Default.Index<Post>().Key(p => p.Geo, KeyType.Geo2DSphere).CreateAsync(token);
|
||||
// a post imported from another actor's archive, once per persona: importing it twice changes nothing
|
||||
await DB.Default.Index<Post>().Key(p => p.GroupUserId, KeyType.Ascending).Key(p => p.ImportedFromURI, KeyType.Ascending)
|
||||
.Option(o =>
|
||||
{
|
||||
o.Unique = true;
|
||||
o.PartialFilterExpression = Builders<Post>.Filter.Gt(p => p.ImportedFromURI, "");
|
||||
})
|
||||
.CreateAsync(token);
|
||||
|
||||
// media: a post's, a persona's, what a scheduled post holds, the janitor's never-posted uploads and its trash
|
||||
await Plain<Models.Media.MediaAttachment>(token, m => m.PostId, m => m.ScheduledStatusId, m => m.CreatedAt);
|
||||
await Plain<Models.Media.MediaAttachment>(token, m => m.OwnerAvatarId);
|
||||
await Plain<Models.Media.MediaAttachment>(token, m => m.ScheduledStatusId);
|
||||
await Plain<Models.Media.MediaAttachment>(token, m => m.ProfileOfAvatarId);
|
||||
await Plain<Models.Media.MediaAttachment>(token, m => m.TrashedAt);
|
||||
|
||||
await Unique<DmPost>(p => p.ObjectURI, Builders<DmPost>.Filter.Gt(p => p.ObjectURI, ""), token);
|
||||
await Plain<DmPost>(token, p => p.GroupId, p => p.ID);
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
using MongoDB.Bson;
|
||||
using MongoDB.Driver;
|
||||
using MongoDB.Entities;
|
||||
|
||||
using PrivaPub.Models.Media;
|
||||
|
||||
using PostEntity = PrivaPub.Models.Post.Post;
|
||||
|
||||
namespace PrivaPub.Infrastructure.Data.Migrations
|
||||
{
|
||||
// A focal point of NaN or Infinity (taken as given before FocalPoint) made every status, timeline and Note holding its
|
||||
// post fail to serialise: it goes, from the upload and from the copy each post keeps.
|
||||
public class _016_focal_points_are_finite : IMigration
|
||||
{
|
||||
public async Task UpgradeAsync()
|
||||
{
|
||||
var unsound = new BsonDocument("$in", new BsonArray { double.NaN, double.PositiveInfinity, double.NegativeInfinity });
|
||||
await DB.Default.Collection<MediaAttachment>().UpdateManyAsync(new BsonDocument("Focus", unsound),
|
||||
new BsonDocument("$set", new BsonDocument("Focus", BsonNull.Value)));
|
||||
await DB.Default.Collection<PostEntity>().UpdateManyAsync(new BsonDocument("Media.Focus", unsound),
|
||||
new BsonDocument("$set", new BsonDocument("Media.$[m].Focus", BsonNull.Value)),
|
||||
new UpdateOptions { ArrayFilters = [new BsonDocumentArrayFilterDefinition<BsonDocument>(new BsonDocument("m.Focus", unsound))] });
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
using MongoDB.Bson;
|
||||
using MongoDB.Driver;
|
||||
|
||||
namespace PrivaPub.Infrastructure.Data
|
||||
{
|
||||
// Whether the mongod is a replica set member (production's is a one-member set, rs0): only then can a read session
|
||||
// hold one point in time across collections (a snapshot), which a backup needs to be consistent.
|
||||
public static class MongoTopology
|
||||
{
|
||||
public static async Task<bool> IsReplicaSet(IMongoDatabase database, CancellationToken token)
|
||||
{
|
||||
try
|
||||
{
|
||||
var hello = await database.Client.GetDatabase("admin").RunCommandAsync<BsonDocument>(new BsonDocument("hello", 1), cancellationToken: token);
|
||||
return hello.Contains("setName");
|
||||
}
|
||||
catch (MongoCommandException)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -29,6 +29,9 @@ namespace PrivaPub.Infrastructure.Http
|
||||
Task<HttpResponseMessage> OpenMedia(string url, System.Net.Http.Headers.RangeHeaderValue range, CancellationToken token);
|
||||
Task<HttpResponseMessage> Send(HttpRequestMessage request, CancellationToken token);
|
||||
Task<(byte[] Bytes, string ContentType)> GetMedia(string url, long maxBytes, CancellationToken token);
|
||||
/// <summary>Copies a media file into <paramref name="destination"/>, up to maxBytes: its type, or why it was refused
|
||||
/// ("too-large", "content-type", ...).</summary>
|
||||
Task<(string ContentType, string Refusal)> DownloadMedia(string url, long maxBytes, Stream destination, CancellationToken token);
|
||||
Task<(int Status, string Text)> GetText(string url, int maxBytes, CancellationToken token);
|
||||
Task<IReadOnlyList<string>> GetStringArray(string url, int maxItems, CancellationToken token);
|
||||
}
|
||||
@@ -389,14 +392,29 @@ namespace PrivaPub.Infrastructure.Http
|
||||
Uri.TryCreate(url, UriKind.Absolute, out var target) && _cache.TryGetValue(NegativeKey(target), out Refusal refusal) && refusal.Transient;
|
||||
|
||||
public async Task<(byte[] Bytes, string ContentType)> GetMedia(string url, long maxBytes, CancellationToken token)
|
||||
{
|
||||
var bytes = default(byte[]);
|
||||
var (contentType, _) = await ReadMedia(url, maxBytes, async (content, limit, t) =>
|
||||
{
|
||||
bytes = await ReadBounded(content, (int)Math.Min(limit, int.MaxValue), t);
|
||||
return bytes?.Length ?? -1;
|
||||
}, token);
|
||||
return contentType == default ? default : (bytes, contentType);
|
||||
}
|
||||
|
||||
public Task<(string ContentType, string Refusal)> DownloadMedia(string url, long maxBytes, Stream destination, CancellationToken token) =>
|
||||
ReadMedia(url, maxBytes, (content, limit, t) => CopyBounded(content, destination, limit, t), token);
|
||||
|
||||
// a media file read through consume (which answers how many bytes it took, or -1 past the limit): its type, or why not
|
||||
async Task<(string ContentType, string Refusal)> ReadMedia(string url, long maxBytes, Func<HttpContent, long, CancellationToken, Task<long>> consume, CancellationToken token)
|
||||
{
|
||||
var exchange = new Exchange(url, "media");
|
||||
try
|
||||
{
|
||||
var media = await GetMedia(url, maxBytes, exchange, token);
|
||||
if (media.Bytes == default && exchange.Outcome == Interactions.Ok)
|
||||
var contentType = await ReadMedia(url, maxBytes, consume, exchange, token);
|
||||
if (contentType == default && exchange.Outcome == Interactions.Ok)
|
||||
exchange.Refused("unusable");
|
||||
return media;
|
||||
return (contentType, contentType == default ? exchange.Reason ?? "unusable" : default);
|
||||
}
|
||||
finally
|
||||
{
|
||||
@@ -404,7 +422,7 @@ namespace PrivaPub.Infrastructure.Http
|
||||
}
|
||||
}
|
||||
|
||||
async Task<(byte[] Bytes, string ContentType)> GetMedia(string url, long maxBytes, Exchange exchange, CancellationToken token)
|
||||
async Task<string> ReadMedia(string url, long maxBytes, Func<HttpContent, long, CancellationToken, Task<long>> consume, Exchange exchange, CancellationToken token)
|
||||
{
|
||||
if (!Uri.TryCreate(url, UriKind.Absolute, out var target) || !IsAllowed(target))
|
||||
{
|
||||
@@ -451,14 +469,14 @@ namespace PrivaPub.Infrastructure.Http
|
||||
exchange.Refused("too-large");
|
||||
return default;
|
||||
}
|
||||
var bytes = await ReadBounded(response.Content, (int)Math.Min(maxBytes, int.MaxValue), timeout.Token);
|
||||
if (bytes == default)
|
||||
var taken = await consume(response.Content, maxBytes, timeout.Token);
|
||||
if (taken < 0)
|
||||
{
|
||||
exchange.Refused("too-large");
|
||||
return default;
|
||||
}
|
||||
exchange.Bytes = bytes.Length;
|
||||
return (bytes, mediaType);
|
||||
exchange.Bytes = taken;
|
||||
return mediaType;
|
||||
}
|
||||
exchange.Refused("too-many-redirects");
|
||||
return default;
|
||||
@@ -608,6 +626,23 @@ namespace PrivaPub.Infrastructure.Http
|
||||
return await _httpClientFactory.CreateClient(ClientName).SendAsync(request, HttpCompletionOption.ResponseHeadersRead, token);
|
||||
}
|
||||
|
||||
// copies at most limit bytes: how many, or -1 once there are more
|
||||
static async Task<long> CopyBounded(HttpContent content, Stream destination, long limit, CancellationToken token)
|
||||
{
|
||||
await using var stream = await content.ReadAsStreamAsync(token);
|
||||
var chunk = new byte[64 * 1024];
|
||||
var copied = 0L;
|
||||
int read;
|
||||
while ((read = await stream.ReadAsync(chunk, token)) > 0)
|
||||
{
|
||||
copied += read;
|
||||
if (copied > limit)
|
||||
return -1;
|
||||
await destination.WriteAsync(chunk.AsMemory(0, read), token);
|
||||
}
|
||||
return copied;
|
||||
}
|
||||
|
||||
public static async Task<byte[]> ReadBounded(HttpContent content, int limit, CancellationToken token)
|
||||
{
|
||||
await using var stream = await content.ReadAsStreamAsync(token);
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
using MongoDB.Bson;
|
||||
using MongoDB.Driver;
|
||||
using MongoDB.Entities;
|
||||
|
||||
@@ -29,8 +30,13 @@ namespace PrivaPub.Infrastructure.Jobs
|
||||
Task<bool> Enqueue(JobKind kind, string payload, string host, string dedupeKey, CancellationToken token);
|
||||
Task<string> Payload(string dedupeKey, CancellationToken token);
|
||||
Task<int> EnqueueMany(IEnumerable<Job> jobs, CancellationToken token);
|
||||
/// <summary>How long a lease lasts unless it is renewed.</summary>
|
||||
TimeSpan LeaseFor { get; }
|
||||
Task<Job> Lease(JobKind kind, IReadOnlyCollection<string> busyHosts, CancellationToken token);
|
||||
Task Finish(Job job, JobOutcome outcome, int maxAttempts, CancellationToken token);
|
||||
/// <summary>Extends a running job's lease; false once it is no longer this lease's (reaped and leased again).</summary>
|
||||
Task<bool> Renew(Job job, CancellationToken token);
|
||||
/// <summary>Records the outcome; false when the lease was lost meanwhile, and then nothing changes.</summary>
|
||||
Task<bool> Finish(Job job, JobOutcome outcome, int maxAttempts, CancellationToken token);
|
||||
Task<long> Reap(CancellationToken token);
|
||||
Task WaitForWork(JobKind kind, TimeSpan poll, CancellationToken token);
|
||||
}
|
||||
@@ -39,6 +45,8 @@ namespace PrivaPub.Infrastructure.Jobs
|
||||
{
|
||||
public static readonly TimeSpan LeaseTime = TimeSpan.FromMinutes(2);
|
||||
|
||||
// every lease is stamped with its own owner (the process and a fresh id): two workers of one process, the second
|
||||
// leasing a job the reaper took back from the first, must not renew nor finish each other's lease
|
||||
readonly string _owner = $"{Environment.MachineName}:{Environment.ProcessId}";
|
||||
readonly ConcurrentDictionary<JobKind, SemaphoreSlim> _signals = new();
|
||||
readonly FilterDefinition<Job> _scope = Builders<Job>.Filter.Empty;
|
||||
@@ -49,6 +57,15 @@ namespace PrivaPub.Infrastructure.Jobs
|
||||
|
||||
public JobQueue(Expression<Func<Job, bool>> scope) => _scope = Builders<Job>.Filter.Where(scope);
|
||||
|
||||
public JobQueue(TimeSpan leaseFor, Expression<Func<Job, bool>> scope = default)
|
||||
{
|
||||
LeaseFor = leaseFor;
|
||||
if (scope != default)
|
||||
_scope = Builders<Job>.Filter.Where(scope);
|
||||
}
|
||||
|
||||
public TimeSpan LeaseFor { get; } = LeaseTime;
|
||||
|
||||
public async Task<bool> Enqueue(JobKind kind, string payload, string host, string dedupeKey, CancellationToken token) =>
|
||||
await EnqueueMany(new[] { new Job { Kind = kind, Payload = payload, Host = host, DedupeKey = dedupeKey } }, token) == 1;
|
||||
|
||||
@@ -81,17 +98,26 @@ namespace PrivaPub.Infrastructure.Jobs
|
||||
return await DB.Default.UpdateAndGet<Job>()
|
||||
.Match(f => f.Where(j => j.Kind == kind && j.State == JobState.Pending && j.RunAt <= now && !busy.Contains(j.Host)) & _scope)
|
||||
.Modify(j => j.State, JobState.Running)
|
||||
.Modify(j => j.LeasedUntil, now + LeaseTime)
|
||||
.Modify(j => j.LeaseOwner, _owner)
|
||||
.Modify(j => j.LeasedUntil, now + LeaseFor)
|
||||
.Modify(j => j.LeaseOwner, $"{_owner}/{ObjectId.GenerateNewId()}")
|
||||
.Modify(b => b.Inc(j => j.Attempts, 1))
|
||||
.Option(o => o.Sort = Builders<Job>.Sort.Ascending(j => j.RunAt))
|
||||
.ExecuteAsync(token);
|
||||
}
|
||||
|
||||
public async Task Finish(Job job, JobOutcome outcome, int maxAttempts, CancellationToken token)
|
||||
public async Task<bool> Renew(Job job, CancellationToken token)
|
||||
{
|
||||
var result = await DB.Default.Update<Job>()
|
||||
.Match(j => j.ID == job.ID && j.State == JobState.Running && j.LeaseOwner == job.LeaseOwner)
|
||||
.Modify(j => j.LeasedUntil, DateTime.UtcNow + LeaseFor)
|
||||
.ExecuteAsync(token);
|
||||
return result.MatchedCount == 1;
|
||||
}
|
||||
|
||||
public async Task<bool> Finish(Job job, JobOutcome outcome, int maxAttempts, CancellationToken token)
|
||||
{
|
||||
var now = DateTime.UtcNow;
|
||||
var update = DB.Default.Update<Job>().MatchID(job.ID)
|
||||
var update = DB.Default.Update<Job>().Match(j => j.ID == job.ID && j.LeaseOwner == job.LeaseOwner)
|
||||
.Modify(j => j.LeasedUntil, null)
|
||||
.Modify(j => j.LeaseOwner, null)
|
||||
.Modify(j => j.LastError, outcome.Error);
|
||||
@@ -112,7 +138,7 @@ namespace PrivaPub.Infrastructure.Jobs
|
||||
update.Modify(j => j.State, JobState.Dead).Modify(j => j.FinishedAt, now);
|
||||
break;
|
||||
}
|
||||
await update.ExecuteAsync(token);
|
||||
return (await update.ExecuteAsync(token)).MatchedCount == 1;
|
||||
}
|
||||
|
||||
public async Task<long> Reap(CancellationToken token)
|
||||
|
||||
@@ -69,18 +69,27 @@ namespace PrivaPub.Infrastructure.Jobs
|
||||
|
||||
var host = job.Host ?? string.Empty;
|
||||
inFlight.AddOrUpdate(host, 1, (_, count) => count + 1);
|
||||
// the lease is renewed while the handler runs; once it is lost (another worker leased the job again) the
|
||||
// handler is cancelled and its outcome dropped
|
||||
using var running = CancellationTokenSource.CreateLinkedTokenSource(stoppingToken);
|
||||
var renewing = KeepLease(job, running);
|
||||
try
|
||||
{
|
||||
JobOutcome outcome;
|
||||
try
|
||||
{
|
||||
using var scope = HttpScope.Triggered(handler.Kind.ToString().ToLowerInvariant());
|
||||
outcome = await handler.Handle(job, stoppingToken);
|
||||
outcome = await handler.Handle(job, running.Token);
|
||||
}
|
||||
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
return;
|
||||
}
|
||||
catch (OperationCanceledException) when (running.IsCancellationRequested)
|
||||
{
|
||||
_logger.LogWarning("{Kind} job {Id} lost its lease and was stopped", handler.Kind, job.ID);
|
||||
continue;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(ex, "{Kind} job {Id} threw", handler.Kind, job.ID);
|
||||
@@ -90,7 +99,8 @@ namespace PrivaPub.Infrastructure.Jobs
|
||||
if (outcome.Result != JobResult.Done && job.Attempts >= handler.MaxAttempts && outcome.Result == JobResult.Retry)
|
||||
_logger.LogWarning("{Kind} job {Id} for {Host} is dead after {Attempts} attempts: {Error}",
|
||||
handler.Kind, job.ID, job.Host, job.Attempts, outcome.Error);
|
||||
await _queue.Finish(job, outcome, handler.MaxAttempts, CancellationToken.None);
|
||||
if (!await _queue.Finish(job, outcome, handler.MaxAttempts, CancellationToken.None))
|
||||
_logger.LogWarning("{Kind} job {Id} lost its lease before it finished; its outcome is dropped", handler.Kind, job.ID);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
@@ -98,11 +108,43 @@ namespace PrivaPub.Infrastructure.Jobs
|
||||
}
|
||||
finally
|
||||
{
|
||||
await running.CancelAsync();
|
||||
await renewing;
|
||||
inFlight.AddOrUpdate(host, 0, (_, count) => count - 1);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// renews the lease every third of its length until the handler ends; a lease found lost cancels the handler
|
||||
async Task KeepLease(Job job, CancellationTokenSource running)
|
||||
{
|
||||
var every = _queue.LeaseFor / 3;
|
||||
while (!running.IsCancellationRequested)
|
||||
{
|
||||
try
|
||||
{
|
||||
await Task.Delay(every, running.Token);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
if (await _queue.Renew(job, CancellationToken.None))
|
||||
continue;
|
||||
await running.CancelAsync();
|
||||
return;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
// a passing database error: the lease has two more thirds to go, so try again at the next turn
|
||||
_logger.LogWarning(ex, "{Worker} could not renew the lease of job {Id}", nameof(JobWorker), job.ID);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async Task Reap(CancellationToken stoppingToken)
|
||||
{
|
||||
while (!stoppingToken.IsCancellationRequested)
|
||||
|
||||
@@ -15,12 +15,18 @@ namespace PrivaPub.Infrastructure
|
||||
public int AccountsPerMinute { get; set; } = 10;//sign-ups, sign-ins and recoveries per client address
|
||||
public int InboxBurst { get; set; } = 300;//deliveries a sending origin may make at once
|
||||
public int InboxPerTenSeconds { get; set; } = 50;//and the rate it earns them back
|
||||
public int UploadsBurst { get; set; } = 30;//uploads (media, profile pictures) a session may make at once
|
||||
public int UploadsPerMinute { get; set; } = 10;//and the rate it earns them back
|
||||
public int ProxyBurst { get; set; } = 1200;//remote media a client address may ask for at once (a timeline is many pictures)
|
||||
public int ProxyPerMinute { get; set; } = 600;//and the rate it earns them back
|
||||
}
|
||||
|
||||
public static class RateLimiting
|
||||
{
|
||||
public const string Accounts = "accounts";
|
||||
public const string Inbox = "inbox";
|
||||
public const string Uploads = "uploads";
|
||||
public const string Proxy = "proxy";
|
||||
|
||||
static RateLimitOptions Limits(HttpContext context) => context.RequestServices.GetRequiredService<IOptions<RateLimitOptions>>().Value;
|
||||
|
||||
@@ -59,8 +65,36 @@ namespace PrivaPub.Infrastructure
|
||||
ReplenishmentPeriod = TimeSpan.FromSeconds(10),
|
||||
QueueLimit = 0
|
||||
}));
|
||||
options.AddPolicy(Proxy, context => RateLimitPartition.GetTokenBucketLimiter(
|
||||
context.Connection.RemoteIpAddress?.ToString() ?? "unknown",
|
||||
_ => new TokenBucketRateLimiterOptions
|
||||
{
|
||||
TokenLimit = Limits(context).ProxyBurst,
|
||||
TokensPerPeriod = Limits(context).ProxyPerMinute,
|
||||
ReplenishmentPeriod = TimeSpan.FromMinutes(1),
|
||||
QueueLimit = 0
|
||||
}));
|
||||
// per session: the limiter runs before authentication, so the credential sent stands for whoever sends it
|
||||
options.AddPolicy(Uploads, context => RateLimitPartition.GetTokenBucketLimiter(
|
||||
Credential(context.Request) ?? "anonymous:" + context.Connection.RemoteIpAddress,
|
||||
_ => new TokenBucketRateLimiterOptions
|
||||
{
|
||||
TokenLimit = Limits(context).UploadsBurst,
|
||||
TokensPerPeriod = Limits(context).UploadsPerMinute,
|
||||
ReplenishmentPeriod = TimeSpan.FromMinutes(1),
|
||||
QueueLimit = 0
|
||||
}));
|
||||
});
|
||||
|
||||
// a hash of the credential, never the credential itself
|
||||
static string Credential(HttpRequest request)
|
||||
{
|
||||
var authorization = request.Headers.Authorization.ToString();
|
||||
return string.IsNullOrEmpty(authorization)
|
||||
? default
|
||||
: Convert.ToHexStringLower(System.Security.Cryptography.SHA256.HashData(System.Text.Encoding.UTF8.GetBytes(authorization)))[..32];
|
||||
}
|
||||
|
||||
static string SenderOrigin(HttpRequest request)
|
||||
{
|
||||
var signature = request.Headers["Signature"].ToString();
|
||||
|
||||
@@ -121,6 +121,9 @@ namespace PrivaPub.Middleware
|
||||
.AddSingleton<IJobHandler, Federation.Actors.OutboxBackfill>()
|
||||
.AddSingleton<IJobHandler, Federation.Actors.FollowingSynchronization>()
|
||||
.AddSingleton<IJobHandler, PollCloseJob>()
|
||||
.AddSingleton<IJobHandler, Domain.Media.ProcessMediaJob>()
|
||||
.AddSingleton<IJobHandler, Domain.Portability.ExportArchiveJob>()
|
||||
.AddSingleton<IJobHandler, Domain.Portability.ImportArchiveJob>()
|
||||
.AddSingleton<IJobHandler, Domain.Statuses.PublishScheduledJob>()
|
||||
.AddSingleton<InstanceDescriber>()
|
||||
.AddSingleton<IJobHandler>(services => services.GetRequiredService<InstanceDescriber>())
|
||||
|
||||
@@ -35,7 +35,10 @@ namespace PrivaPub.Models.Jobs
|
||||
PublishScheduled,
|
||||
FetchReplies,
|
||||
BackfillOutbox,
|
||||
SynchronizeFollowing
|
||||
SynchronizeFollowing,
|
||||
ProcessMedia,
|
||||
ExportArchive,
|
||||
ImportArchive
|
||||
}
|
||||
|
||||
public enum JobState
|
||||
|
||||
@@ -16,9 +16,18 @@ namespace PrivaPub.Models.Media
|
||||
public int? Width { get; set; }
|
||||
public int? Height { get; set; }
|
||||
public float[] Focus { get; set; }
|
||||
public double? DurationSeconds { get; set; }
|
||||
// audio and video sent to /api/v2/media are processed by a job: "pending" until then, "failed" (with why) if it can't be
|
||||
public string ProcessingState { get; set; }
|
||||
public string ProcessingError { get; set; }
|
||||
public DateTime CreatedAt { get; set; } = DateTime.UtcNow;
|
||||
public DateTime? AttachedAt { get; set; }
|
||||
public string ScheduledStatusId { get; set; }//kept for a scheduled post until it is published or dropped
|
||||
public string ProfileOfAvatarId { get; set; }//a persona's avatar or header (Kind "avatar" or "header"), never a post's
|
||||
// set once nothing holds it any more (its post deleted, edited out, replaced, its root removed, never posted): its
|
||||
// files move out of what /media/files serves at once, and the janitor deletes them after a grace
|
||||
public DateTime? TrashedAt { get; set; }
|
||||
public string TrashReason { get; set; }
|
||||
}
|
||||
|
||||
public class MediaSecret : Entity
|
||||
|
||||
@@ -58,6 +58,7 @@ namespace PrivaPub.Models.Post
|
||||
|
||||
public bool IsFederatedCopy { get; set; }
|
||||
public string ObjectURI { get; set; }//the Note's id
|
||||
public string ImportedFromURI { get; set; }//a post imported from another actor's archive: the original's id (PersonaImport)
|
||||
public string ObjectType { get; set; }//the remote object's type: Note, Article, Page, Video, Event…; null for local posts
|
||||
public bool AsChatMessage { get; set; }//a local direct message sent as a ChatMessage: its one recipient writes to us that way
|
||||
public string ActivityURI { get; set; }//the Create's id
|
||||
|
||||
@@ -4,6 +4,7 @@ namespace PrivaPub.Models.Post
|
||||
{
|
||||
public Guid Id { get; set; } = Guid.NewGuid();
|
||||
public string ContentType { get; set; }
|
||||
public string Kind { get; set; }//a local upload's kind ("gifv" for a GIF made an mp4), where the type alone can't tell
|
||||
public string FileName { get; set; }
|
||||
public string Extension { get; set; }
|
||||
public string Path { get; set; }
|
||||
|
||||
+33
-2
@@ -68,6 +68,11 @@ try
|
||||
.AddSingleton<PrivaPub.Domain.Media.IMediaService, PrivaPub.Domain.Media.MediaService>()
|
||||
.AddSingleton<PrivaPub.Domain.Media.IMediaProxy, PrivaPub.Domain.Media.MediaProxy>()
|
||||
.AddHostedService<PrivaPub.Domain.Media.MediaJanitor>()
|
||||
.Configure<PrivaPub.Infrastructure.Backup.BackupOptions>(builder.Configuration.GetSection("Backups"))
|
||||
.AddSingleton<PrivaPub.Infrastructure.Backup.Backups>()
|
||||
.AddSingleton<PrivaPub.Infrastructure.Backup.TransferStore>()
|
||||
.AddHostedService<PrivaPub.Infrastructure.Backup.BackupScheduler>()
|
||||
.AddHostedService<PrivaPub.Infrastructure.Backup.RestoreWatcher>()
|
||||
.PrivaPubMiddlewareConfiguration();
|
||||
}
|
||||
catch (Exception ex)
|
||||
@@ -86,8 +91,6 @@ try
|
||||
var mongoSettings = builder.Configuration.GetSection(nameof(MongoSettings)).Get<MongoSettings>();
|
||||
await DB.InitAsync(mongoSettings.Database, MongoClientSettings.FromConnectionString(mongoSettings.ConnectionString));
|
||||
EntityMaps.Warm();
|
||||
await DB.Default.MigrateAsync<Program>();
|
||||
await Indexes.Create();
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
@@ -106,8 +109,35 @@ try
|
||||
throw;
|
||||
}
|
||||
|
||||
// A restore asked for runs here, before migrations, indexes and every hosted service; a command waits for it.
|
||||
var gate = await PrivaPub.Infrastructure.Backup.MaintenanceGate.Enter(app.Services, args is ["admin", .. var asked] ? asked : default, CancellationToken.None);
|
||||
if (gate is { } stop)
|
||||
{
|
||||
Environment.ExitCode = stop;
|
||||
return;
|
||||
}
|
||||
|
||||
// Commands get every service but start nothing: no Kestrel, no hosted services, no media directory. The deploy runs
|
||||
// them as its own user, which can read the configuration and reach the private mongod but owns no www-data directory.
|
||||
// Backups and restores are asked for before migrations: the deploy's backup is of the database as the live build left it.
|
||||
if (args is ["admin", .. var command] && AdminCommands.BeforeMigrations(command))
|
||||
{
|
||||
using var scope = app.Services.CreateScope();
|
||||
Environment.ExitCode = await AdminCommands.Run(command, scope.ServiceProvider);
|
||||
return;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
await DB.Default.MigrateAsync<Program>();
|
||||
await Indexes.Create();
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
Log.ForContext<Program>().Fatal(ex, $"{nameof(Program)}.{nameof(Program)}() DB Migrations");
|
||||
throw;
|
||||
}
|
||||
|
||||
if (args is ["admin", ..])
|
||||
{
|
||||
using var scope = app.Services.CreateScope();
|
||||
@@ -144,6 +174,7 @@ try
|
||||
{
|
||||
FileProvider = new Microsoft.Extensions.FileProviders.PhysicalFileProvider(mediaRoot),
|
||||
RequestPath = "/media/files",
|
||||
ContentTypeProvider = PrivaPub.Domain.Media.MediaService.ContentTypes,
|
||||
OnPrepareResponse = context =>
|
||||
{
|
||||
context.Context.Response.Headers["X-Content-Type-Options"] = "nosniff";
|
||||
|
||||
@@ -37,6 +37,7 @@ namespace PrivaPub.Services
|
||||
readonly IPasswordHasher _passwordHasher;
|
||||
readonly ILocalActorService _localActors;
|
||||
readonly IDeliveryService _delivery;
|
||||
readonly Domain.Media.IMediaProxy _proxy;
|
||||
readonly IStringLocalizer<GenericRes> _localizer;
|
||||
readonly ILogger<GroupUsersService> _logger;
|
||||
|
||||
@@ -45,8 +46,10 @@ namespace PrivaPub.Services
|
||||
ILocalActorService localActors,
|
||||
IDeliveryService delivery,
|
||||
IStringLocalizer<GenericRes> localizer,
|
||||
ILogger<GroupUsersService> logger)
|
||||
ILogger<GroupUsersService> logger,
|
||||
Domain.Media.IMediaProxy proxy)
|
||||
{
|
||||
_proxy = proxy;
|
||||
_dbEntities = dbEntities;
|
||||
_passwordHasher = passwordHasher;
|
||||
_localActors = localActors;
|
||||
@@ -387,12 +390,13 @@ namespace PrivaPub.Services
|
||||
}
|
||||
}
|
||||
|
||||
static ViewGroupMember Remote(string actorUri, Models.User.ForeignAvatar foreign, GroupRole role, DateTime since, bool pending) => new()
|
||||
// a remote member's picture through the media proxy, as everywhere else a client sees remote media
|
||||
ViewGroupMember Remote(string actorUri, Models.User.ForeignAvatar foreign, GroupRole role, DateTime since, bool pending) => new()
|
||||
{
|
||||
ActorUri = actorUri,
|
||||
Handle = foreign == default ? actorUri : $"{foreign.UserName}@{foreign.Domain}",
|
||||
Name = foreign?.Name ?? foreign?.UserName,
|
||||
PictureUrl = foreign?.PictureURL,
|
||||
PictureUrl = _proxy.Wrap(foreign?.PictureURL),
|
||||
IsPending = pending,
|
||||
Role = role.ToString().ToLowerInvariant(),
|
||||
Since = since
|
||||
|
||||
@@ -7,6 +7,7 @@ using PrivaPub.Models.Federation;
|
||||
using PrivaPub.Models.Social;
|
||||
using PrivaPub.Models.User;
|
||||
using PrivaPub.StaticServices;
|
||||
using PrivaPub.Domain.Media;
|
||||
|
||||
using System.Text.Json.Nodes;
|
||||
|
||||
@@ -30,9 +31,11 @@ namespace PrivaPub.Services
|
||||
readonly ILocalActorService _localActors;
|
||||
readonly IDeliveryService _delivery;
|
||||
readonly IRootSessions _sessions;
|
||||
readonly IMediaService _media;
|
||||
|
||||
public RootRemoval(DbEntities dbEntities, ILocalActorService localActors, IDeliveryService delivery, IRootSessions sessions)
|
||||
public RootRemoval(DbEntities dbEntities, ILocalActorService localActors, IDeliveryService delivery, IRootSessions sessions, IMediaService media)
|
||||
{
|
||||
_media = media;
|
||||
_dbEntities = dbEntities;
|
||||
_localActors = localActors;
|
||||
_delivery = delivery;
|
||||
@@ -66,6 +69,10 @@ namespace PrivaPub.Services
|
||||
await DB.Default.DeleteAsync<Models.Social.PersonaList>(l => l.AvatarId == avatar.ID);
|
||||
await DB.Default.DeleteAsync<Models.Social.PersonaFilter>(f => f.AvatarId == avatar.ID);
|
||||
await DB.Default.DeleteAsync<Models.Social.FollowedTag>(t => t.AvatarId == avatar.ID);
|
||||
// nothing of it publishes later, and none of its files stays served: its posts' media, its pictures and
|
||||
// what its scheduled posts held
|
||||
await DB.Default.DeleteAsync<Models.Social.ScheduledStatus>(s => s.AvatarId == avatar.ID);
|
||||
await _media.Trash(m => m.OwnerAvatarId == avatar.ID, "root removed", token);
|
||||
}
|
||||
|
||||
await DB.Default.Update<RootUser>().MatchID(root.ID)
|
||||
|
||||
@@ -4,6 +4,9 @@
|
||||
"LogsDatabase": "logs",
|
||||
"ConnectionString": "mongodb://localhost:27017"
|
||||
},
|
||||
"Backups": {
|
||||
"Nightly": false
|
||||
},
|
||||
"AppConfiguration": {
|
||||
"RequiresFirstTimeSetup": null,
|
||||
"Version": "0.0.0",
|
||||
|
||||
@@ -2,6 +2,9 @@
|
||||
"Media": {
|
||||
"Root": "/var/lib/privapub/media"
|
||||
},
|
||||
"Backups": {
|
||||
"Root": "/var/lib/privapub/backups"
|
||||
},
|
||||
"Registrations": {
|
||||
"Mode": "Invitations"
|
||||
},
|
||||
@@ -28,7 +31,7 @@
|
||||
"MongoSettings": {
|
||||
"Database": "PrivaPub",
|
||||
"LogsDatabase": "logs",
|
||||
"ConnectionString": "mongodb://127.0.0.1:27022"
|
||||
"ConnectionString": "mongodb://127.0.0.1:27022/?directConnection=true"
|
||||
},
|
||||
"AppConfiguration": {
|
||||
"Version": "0.0.0",
|
||||
@@ -74,7 +77,7 @@
|
||||
{
|
||||
"Name": "MongoDBCapped",
|
||||
"Args": {
|
||||
"databaseUrl": "mongodb://127.0.0.1:27022/logs",
|
||||
"databaseUrl": "mongodb://127.0.0.1:27022/logs?directConnection=true",
|
||||
"collectionName": "PrivaPub",
|
||||
"cappedMaxSizeMb": "1024",
|
||||
"cappedMaxDocuments": "10000"
|
||||
|
||||
@@ -13,6 +13,10 @@ echo "== directories"
|
||||
install -d -o "$RUNNER" -g www-data -m 755 /var/www/$HOST
|
||||
install -d -o "$RUNNER" -g "$RUNNER" -m 750 /var/backups/$HOST
|
||||
install -d -o www-data -g www-data -m 750 /var/lib/privapub /var/lib/privapub/mongo
|
||||
# backups: the service writes them, and so does the deploy (as $RUNNER, a member of www-data) before each deploy; a runner
|
||||
# just added to the group gets it when the runner restarts
|
||||
id -nG "$RUNNER" | grep -qw www-data || usermod -aG www-data "$RUNNER"
|
||||
install -d -o www-data -g www-data -m 2770 /var/lib/privapub/backups
|
||||
|
||||
echo "== sudoers"
|
||||
SUDOERS=/etc/sudoers.d/$RUNNER
|
||||
@@ -28,6 +32,18 @@ systemctl enable --now privapub-mongod >/dev/null
|
||||
systemctl enable $UNIT >/dev/null
|
||||
systemctl is-active privapub-mongod
|
||||
|
||||
echo "== replica set"
|
||||
# once: mongod restarted with --replSet (PrivaPub stopped meanwhile), the one-member set rs0 made, its primary awaited
|
||||
if [ "$(mongosh --quiet --port 27022 --eval 'db.hello().setName' 2>/dev/null)" != "rs0" ]; then
|
||||
systemctl stop $UNIT
|
||||
systemctl restart privapub-mongod
|
||||
for _ in $(seq 1 60); do mongosh --quiet --port 27022 --eval 'db.hello()' >/dev/null 2>&1 && break; sleep 1; done
|
||||
mongosh --quiet --port 27022 --eval "rs.initiate({_id: 'rs0', members: [{_id: 0, host: '127.0.0.1:27022'}]})" >/dev/null
|
||||
for _ in $(seq 1 60); do [ "$(mongosh --quiet --port 27022 --eval 'db.hello().isWritablePrimary')" = "true" ] && break; sleep 1; done
|
||||
systemctl start $UNIT
|
||||
fi
|
||||
echo "replica set: $(mongosh --quiet --port 27022 --eval 'db.hello().setName')"
|
||||
|
||||
echo "== nginx snippet and bootstrap vhost"
|
||||
install -m 644 "$SRC/nginx/privapub-headers.conf" /etc/nginx/snippets/privapub-headers.conf
|
||||
if [ -f /root/.acme.sh/${HOST}_ecc/fullchain.cer ]; then
|
||||
|
||||
@@ -1,3 +1,6 @@
|
||||
# backup uploads: a few pieces a second at most, per address
|
||||
limit_req_zone $binary_remote_addr zone=privapub_backup_uploads:1m rate=120r/m;
|
||||
|
||||
server {
|
||||
listen 80;
|
||||
listen [::]:80;
|
||||
@@ -48,6 +51,33 @@ server {
|
||||
proxy_read_timeout 300s;
|
||||
}
|
||||
|
||||
# the administrator's backups and the personas' archives (owner decision 2026-10-07): a download streams a whole one for
|
||||
# as long as it takes; an upload arrives in pieces of at most 32 MB, unbuffered
|
||||
location ~ ^/clientapi/(admin/backups|persona/archive)/download/ {
|
||||
proxy_buffering off;
|
||||
proxy_pass http://127.0.0.1:6970;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Connection "";
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_read_timeout 3600s;
|
||||
proxy_send_timeout 3600s;
|
||||
}
|
||||
location ~ ^/clientapi/(admin/backups|persona/archive)/uploads {
|
||||
client_max_body_size 40m;
|
||||
proxy_request_buffering off;
|
||||
limit_req zone=privapub_backup_uploads burst=30 nodelay;
|
||||
proxy_pass http://127.0.0.1:6970;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Connection "";
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_read_timeout 300s;
|
||||
}
|
||||
location ^~ /media/proxy/ {
|
||||
proxy_buffering off;
|
||||
proxy_pass http://127.0.0.1:6970;
|
||||
|
||||
@@ -6,7 +6,8 @@ After=network.target
|
||||
Type=exec
|
||||
User=www-data
|
||||
Group=www-data
|
||||
ExecStart=/usr/bin/mongod --dbpath /var/lib/privapub/mongo --port 27022 --bind_ip 127.0.0.1 --noauth --wiredTigerCacheSizeGB 0.25 --quiet
|
||||
# a one-member replica set (rs0), so a backup reads every collection at one instant; its oplog kept small
|
||||
ExecStart=/usr/bin/mongod --dbpath /var/lib/privapub/mongo --port 27022 --bind_ip 127.0.0.1 --noauth --wiredTigerCacheSizeGB 0.25 --replSet rs0 --oplogSize 990 --quiet
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
LimitNOFILE=64000
|
||||
|
||||
@@ -14,6 +14,11 @@ Environment=ASPNETCORE_ENVIRONMENT=Production
|
||||
Environment=DOTNET_CLI_TELEMETRY_OPTOUT=1
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
# a restore asked for stops the service, and the next start carries it out; one that failed three times exits 75 and
|
||||
# waits for someone to look (journalctl -u privapub, PrivaPub admin restore --status)
|
||||
RestartPreventExitStatus=75
|
||||
# its own /tmp: ffmpeg's and the uploads' temporary files are nobody else's
|
||||
PrivateTmp=true
|
||||
SyslogIdentifier=privapub
|
||||
|
||||
[Install]
|
||||
|
||||
@@ -288,6 +288,32 @@ and circles (see Owner decisions).
|
||||
| A post's `replies` and `context` | **Publish public threads.** Public and unlisted posts name their `replies` and `context` collections, which list only the public and unlisted replies PrivaPub holds; followers-only, circle, direct and local-only posts never do. |
|
||||
| FEP-8fcf followers synchronisation | **Send digests.** A delivery to a server that the persona's followers there would get carries a `Collection-Synchronization` header: a digest of the persona's followers on that server only, and where that server reads that partial list. |
|
||||
|
||||
### Owner decisions on media, backups and archives (2026-10-07)
|
||||
|
||||
The owner asked for media and file handling (posted and profile) to be audited for soundness, and for a backup and
|
||||
restore system (JSON mostly, plus the files).
|
||||
|
||||
| Question | Decision |
|
||||
|---|---|
|
||||
| What backups cover | **The whole server and a per-persona archive.** The server backup holds every collection as canonical Extended JSON and the media files (hard links); a persona's archive, in Mastodon's account-archive layout, can be restored into a persona. |
|
||||
| Who runs them | **The CLI, a nightly schedule and the administrator's page, which can also restore.** This approves the `/clientapi/admin/backups` endpoints in production (download for the password, restore for the password and the host typed out). |
|
||||
| Protection | **Plain archives, protected by file permissions** (`/var/lib/privapub/backups`, 2770, files 0640). No encryption. |
|
||||
| Consistency | **Production's mongod becomes a one-member replica set** (`rs0`), so a backup reads every collection at one instant. `setup.sh` converts it, which needs a planned restart of mongod. |
|
||||
|
||||
**Done 2026-10-07 (committed, not deployed):**
|
||||
- media: files live exactly as long as something holds them (trash with a day's grace, janitor), finite focal points,
|
||||
pictures checked before they are decoded, audio and video bounded and processed off the request, a bounded media
|
||||
proxy, scheduled posts' media reserved, profile pictures as rows with `DELETE`, edits that reach the post, storage
|
||||
counted, and `PrivaPub admin media audit [--fix]` for what came before;
|
||||
- the server backup and restore: nightly, before each deploy (replacing mongodump) and from the page; a restore runs at
|
||||
boot and never undoes a protective act (`tools/pasture/scenarios/restore.sh`, 19 checks, then `town.sh check` 2454/2454);
|
||||
- a persona's archive, exported and imported (`tools/pasture/scenarios/persona-archive.sh` imports a real Mastodon
|
||||
archive: 156 posts, nothing delivered).
|
||||
|
||||
**At the next deploy, in this order:** `setup.sh` (the replica set, the backups directory, the runner in www-data's
|
||||
group, the nginx locations), the deploy, `PrivaPub admin media audit` and, once its report is read, `--fix`; and the
|
||||
owner's word on deleting the old pre-deploy dumps, which hold statistics salts.
|
||||
|
||||
## Libraries (researched; no maintained .NET ActivityPub library exists, so Letterbook and Iceshrimp.NET both wrote their own)
|
||||
|
||||
| Area | Choice |
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
#!/usr/bin/env bash
|
||||
# Runs a command (normally `dotnet test`) with a throwaway mongod: a fresh data directory on a random localhost port,
|
||||
# stopped and deleted when the command ends. The box's own mongods (27017 shared, 27022 PrivaPub's) are never touched.
|
||||
# Like production's, it is a one-member replica set (rs0), so the backup's snapshot reads are what the tests exercise.
|
||||
# usage: tools/ci/with-test-mongod.sh dotnet test PrivaPub.sln -c Release
|
||||
set -euo pipefail
|
||||
|
||||
@@ -39,11 +40,11 @@ trap cleanup EXIT INT TERM
|
||||
if command -v mongod >/dev/null 2>&1; then
|
||||
mkdir -p "$dir/db"
|
||||
timeout 7200 mongod --dbpath "$dir/db" --port "$port" --bind_ip 127.0.0.1 --noauth \
|
||||
--wiredTigerCacheSizeGB 0.25 --quiet --logpath "$dir/mongod.log" &
|
||||
--wiredTigerCacheSizeGB 0.25 --replSet rs0 --quiet --logpath "$dir/mongod.log" &
|
||||
echo $! > "$dir/pid"
|
||||
how="mongod $(mongod --version | head -1)"
|
||||
elif command -v podman >/dev/null 2>&1; then
|
||||
podman run -d --rm -p "127.0.0.1:$port:27017" docker.io/library/mongo:8 --quiet > "$dir/container"
|
||||
podman run -d --rm -p "127.0.0.1:$port:27017" docker.io/library/mongo:8 --quiet --replSet rs0 > "$dir/container"
|
||||
how="podman mongo:8"
|
||||
else
|
||||
echo "::error::neither mongod nor podman is available for the test mongod"; exit 1
|
||||
@@ -51,7 +52,17 @@ fi
|
||||
|
||||
for _ in $(seq 1 60); do listening "$port" && break; sleep 0.5; done
|
||||
listening "$port" || { echo "::error::the test mongod did not start"; cat "$dir/mongod.log" 2>/dev/null | tail -20; exit 1; }
|
||||
echo "test mongod: $how on 127.0.0.1:$port"
|
||||
|
||||
export PRIVAPUB_TEST_MONGOD=1 PRIVAPUB_TEST_REQUIRE_MONGOD=1 PRIVAPUB_TEST_MONGO="mongodb://127.0.0.1:$port"
|
||||
# the one-member replica set, then wait until it is its primary
|
||||
shell() {
|
||||
if [ -f "$dir/container" ]; then podman exec "$(cat "$dir/container")" mongosh --quiet --eval "$1"
|
||||
else mongosh --quiet --port "$port" --eval "$1"; fi
|
||||
}
|
||||
member=$([ -f "$dir/container" ] && echo "127.0.0.1:27017" || echo "127.0.0.1:$port")
|
||||
shell "rs.initiate({_id: 'rs0', members: [{_id: 0, host: '$member'}]})" >/dev/null
|
||||
for _ in $(seq 1 60); do [ "$(shell 'db.hello().isWritablePrimary' 2>/dev/null)" = "true" ] && break; sleep 0.5; done
|
||||
[ "$(shell 'db.hello().isWritablePrimary' 2>/dev/null)" = "true" ] || { echo "::error::the test mongod never became its replica set's primary"; exit 1; }
|
||||
echo "test mongod: $how on 127.0.0.1:$port, replica set rs0"
|
||||
|
||||
export PRIVAPUB_TEST_MONGOD=1 PRIVAPUB_TEST_REQUIRE_MONGOD=1 PRIVAPUB_TEST_MONGO="mongodb://127.0.0.1:$port/?directConnection=true"
|
||||
"$@"
|
||||
@@ -2,7 +2,7 @@
|
||||
"MongoSettings": {
|
||||
"Database": "PrivaPub",
|
||||
"LogsDatabase": "logs",
|
||||
"ConnectionString": "mongodb://mongo:27017"
|
||||
"ConnectionString": "mongodb://mongo:27017/?directConnection=true"
|
||||
},
|
||||
"AppConfiguration": {
|
||||
"Version": "0.0.0",
|
||||
@@ -26,7 +26,8 @@
|
||||
"Relays": [ "https://relay.test/actor", "https://aoderelay.test/actor" ]
|
||||
},
|
||||
"Media": { "Root": "/tmp/privapub-media" },
|
||||
"RateLimits": { "AccountsPerMinute": 1000 },
|
||||
"Backups": { "Root": "/tmp/privapub-backups", "Nightly": false },
|
||||
"RateLimits": { "AccountsPerMinute": 1000, "UploadsBurst": 1000, "UploadsPerMinute": 1000, "ProxyBurst": 100000, "ProxyPerMinute": 100000 },
|
||||
"Registrations": { "Mode": "Open" },
|
||||
"Statistics": { "Geo": { "AutoUpdate": false } },
|
||||
"Kestrel": { "Endpoints": { "Http": { "Url": "http://0.0.0.0:80", "Protocols": "Http1AndHttp2" } } },
|
||||
|
||||
@@ -21,11 +21,16 @@ pasture_base_up() {
|
||||
podman volume exists pasture-caddy-data || podman volume create pasture-caddy-data >/dev/null
|
||||
# (PrivaPub's uploads live in a volume: a container made again must not lose them)
|
||||
podman volume exists pasture-privapub-media || podman volume create --label pasture=1 pasture-privapub-media >/dev/null
|
||||
podman run -d --replace --name pasture-mongo --network $net --network-alias mongo docker.io/library/mongo:8 --quiet >/dev/null
|
||||
# a one-member replica set, as production's, so backups read at one instant
|
||||
podman run -d --replace --name pasture-mongo --network $net --network-alias mongo docker.io/library/mongo:8 --quiet --replSet rs0 >/dev/null
|
||||
for _ in $(seq 1 60); do podman exec pasture-mongo mongosh --quiet --eval 'db.hello()' >/dev/null 2>&1 && break; sleep 1; done
|
||||
podman exec pasture-mongo mongosh --quiet --eval "rs.initiate({_id: 'rs0', members: [{_id: 0, host: 'mongo:27017'}]})" >/dev/null
|
||||
for _ in $(seq 1 60); do [ "$(podman exec pasture-mongo mongosh --quiet --eval 'db.hello().isWritablePrimary')" = "true" ] && break; sleep 1; done
|
||||
caddy_up
|
||||
local extra=()
|
||||
for setting in ${PRIVAPUB_ENV:-}; do extra+=(-e "$setting"); done
|
||||
podman run -d --replace --name pasture-privapub --network $net -p "127.0.0.1:$port:80" \
|
||||
# --restart=always, as systemd's unit: a restore asked for stops PrivaPub, and the next start carries it out
|
||||
podman run -d --replace --restart=always --name pasture-privapub --network $net -p "127.0.0.1:$port:80" \
|
||||
--sysctl net.ipv4.ip_unprivileged_port_start=0 -e ASPNETCORE_ENVIRONMENT=Pasture "${extra[@]}" -w /app -v "$publish:/app:Z,ro" \
|
||||
-v pasture-privapub-media:/tmp/privapub-media \
|
||||
mcr.microsoft.com/dotnet/runtime-deps:10.0 /app/PrivaPub >/dev/null
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
# A persona's archive (owner decision 2026-10-07), with a real one: mastouser's own account archive, made by Mastodon's
|
||||
# backup service, is imported into mover_archive, a PrivaPub persona mastouser follows. Its posts arrive, with their
|
||||
# media, on the persona's profile, and nobody is told: Mastodon, which would get anything delivered to the persona's
|
||||
# followers, holds none of the copies. Importing it again changes nothing. Then the persona's own archive is exported and
|
||||
# downloaded through its link, in Mastodon's layout. Needs the mastodon peer.
|
||||
M=https://mastodon.test:6443
|
||||
mcurl() { curl -sk --resolve mastodon.test:6443:127.0.0.1 "$@"; }
|
||||
. "$here/peers/mastodon.sh"
|
||||
m_rails() { podman exec pasture-mastodon bin/rails runner "$1" 2>/dev/null | tail -1; }
|
||||
|
||||
echo "persona-archive"
|
||||
JWT=$(privapub_root)
|
||||
RH="Authorization: Bearer $JWT"
|
||||
AT=$(privapub_token mover_archive)
|
||||
AH="Authorization: Bearer $AT"
|
||||
[ -n "$AT" ] && ok "PrivaPub token for mover_archive" || { ko "PrivaPub token for mover_archive"; return 1; }
|
||||
me=$(curl -s -H "$AH" "$P/api/v1/accounts/verify_credentials")
|
||||
me_id=$(echo "$me" | j "print(d['id'])")
|
||||
me_uri=$(echo "$me" | j "print(d['url'])" | sed 's|/@|/peasants/|')
|
||||
A="$P/clientapi/persona/$me_id/archive"
|
||||
# each run starts from a persona that has imported nothing
|
||||
podman exec pasture-mongo mongosh --quiet PrivaPub --eval "db.Post.deleteMany({GroupUserId:'$me_id', ImportedFromURI:{\$exists:true, \$ne:null}}); db.MediaAttachment.deleteMany({OwnerAvatarId:'$me_id'})" >/dev/null
|
||||
|
||||
m_follows() { m_rails "puts Follow.exists?(account: Account.find_local(\"mastouser\"), target_account: Account.find_by(uri: \"$me_uri\")) ? \"True\" : \"False\""; }
|
||||
if [ "$(m_follows)" != "True" ]; then
|
||||
on_m=$(mcurl -H "Authorization: Bearer $(mastodon_token)" "$M/api/v2/search?q=@mover_archive@privapub.test&resolve=true&type=accounts" | j "print(d['accounts'][0]['id'])")
|
||||
mcurl -o /dev/null -X POST -H "Authorization: Bearer $(mastodon_token)" "$M/api/v1/accounts/$on_m/follow"
|
||||
fi
|
||||
until_true 45 '[ "$(m_follows)" = "True" ]' && ok "mastouser follows mover_archive" || ko "mastouser never followed mover_archive"
|
||||
|
||||
dump=$(m_rails 'u = Account.find_local("mastouser").user; b = u.backups.create!; BackupService.new.call(b); puts b.reload.dump.path')
|
||||
podman cp "pasture-mastodon:$dump" "$work/mastodon.zip" 2>/dev/null
|
||||
[ -s "$work/mastodon.zip" ] && ok "Mastodon made mastouser's archive" || { ko "Mastodon made no archive"; return 1; }
|
||||
|
||||
import() {
|
||||
local upload
|
||||
upload=$(curl -s -X POST -H "$RH" "$P/clientapi/persona/archive/uploads" | j "print(d['id'])")
|
||||
curl -s -o /dev/null -X PUT -H "$RH" -H "Content-Type: application/octet-stream" --data-binary "@$work/mastodon.zip" "$P/clientapi/persona/archive/uploads/$upload?offset=0"
|
||||
curl -s -o /dev/null -w '%{http_code}' -X POST -H "$RH" -H 'Content-Type: application/json' "$A/import" -d "{\"uploadId\":\"$upload\",\"parts\":[\"posts\"]}"
|
||||
}
|
||||
state() { curl -s -H "$RH" "$A" | j "print(d['$1'])"; }
|
||||
count() { curl -s -H "$RH" "$A" | j "print(d['importCounts'].get('$1', 0))"; }
|
||||
|
||||
[ "$(import)" = "202" ] && ok "the import is asked for" || { ko "the import was refused"; return 1; }
|
||||
until_true 150 '[ "$(state importState)" = "done" ]' && ok "the import is done" || { ko "the import is $(state importState): $(state importError)"; return 1; }
|
||||
posts=$(count posts)
|
||||
[ "$posts" -gt 0 ] && ok "$posts of mastouser's posts imported ($(count 'boosts skipped') boosts and $(count 'direct or circle posts skipped') private posts left out)" \
|
||||
|| ko "no post was imported"
|
||||
[ "$(count media)" -gt 0 ] && ok "$(count media) pictures came with them" || ko "no picture came with them"
|
||||
copies=$(curl -s -H "$AH" "$P/api/v1/accounts/$me_id/statuses?limit=40")
|
||||
[ "$(echo "$copies" | j "print(len(d))")" -gt 0 ] && ok "they are on mover_archive's profile" || ko "the profile shows none of them"
|
||||
|
||||
# nobody told: what PrivaPub would have delivered would be at Mastodon by now
|
||||
sleep 20
|
||||
uris=$(echo "$copies" | j "print(' '.join(s['uri'] for s in d[:5]))")
|
||||
held=$("$here/town.sh" stored mastodon $uris | j "print(sum(1 for v in d.values() if v['exists']))")
|
||||
[ "$held" = "0" ] && ok "Mastodon holds none of the copies" || ko "Mastodon holds $held of the copies"
|
||||
|
||||
[ "$(import)" = "202" ] && until_true 150 '[ "$(state importState)" = "done" ]' \
|
||||
&& [ "$(count posts)" = "0" ] && [ "$(count 'posts already here')" = "$posts" ] \
|
||||
&& ok "importing it again changes nothing" || ko "the second import: $(count posts) posts, $(count 'posts already here') already here"
|
||||
|
||||
echo " exported"
|
||||
[ "$(curl -s -o /dev/null -w '%{http_code}' -X POST -H "$RH" "$A")" = "202" ] && ok "the persona's archive is asked for" || ko "the archive was refused"
|
||||
until_true 90 '[ "$(state exportState)" = "ready" ]' && ok "the archive is ready" || ko "the archive is $(state exportState): $(state exportError)"
|
||||
link=$(curl -s -X POST -H "$RH" "$A/ticket" | j "print(d['url'])")
|
||||
curl -s -o "$work/mover.zip" "$P$link"
|
||||
names=$(unzip -Z1 "$work/mover.zip" 2>/dev/null)
|
||||
grep -qx outbox.json <<< "$names" && grep -qx actor.json <<< "$names" \
|
||||
&& ok "it downloads, in Mastodon's layout" || ko "the download is not an archive"
|
||||
@@ -0,0 +1,89 @@
|
||||
# Backup and restore (owner decisions 2026-10-07), on the live pasture and through the administrator's endpoints: the
|
||||
# server is backed up, then life goes on: alice_restore deletes a post and makes another, mastouser follows her, she
|
||||
# blocks bob_restore, and carol_restore is made. The backup is restored: PrivaPub stops, restores it as it starts again,
|
||||
# and nothing protective is undone: the deleted post stays gone, the post made since answers as deleted, mastouser still
|
||||
# follows her, the block holds and carol's name stays taken. Every session ends, so the scenario signs in again, and at
|
||||
# the end the town's PrivaPub accounts too (town.sh renew privapub). Run it alone: it restores the whole server. Needs
|
||||
# the mastodon peer.
|
||||
M=https://mastodon.test:6443
|
||||
mcurl() { curl -sk --resolve mastodon.test:6443:127.0.0.1 "$@"; }
|
||||
. "$here/peers/mastodon.sh"
|
||||
m_rails() { podman exec pasture-mastodon bin/rails runner "$1" 2>/dev/null | tail -1; }
|
||||
p_mongo() { podman exec pasture-mongo mongosh --quiet PrivaPub --eval "$1"; }
|
||||
|
||||
echo "restore"
|
||||
podman exec -w /app pasture-privapub /app/PrivaPub admin promote "$ROOT_USER" >/dev/null 2>&1 || true
|
||||
JWT=$(privapub_root)
|
||||
RH="Authorization: Bearer $JWT"
|
||||
AT=$(privapub_token alice_restore)
|
||||
BT=$(privapub_token bob_restore)
|
||||
AH="Authorization: Bearer $AT"
|
||||
[ -n "$AT" ] && [ -n "$BT" ] && ok "PrivaPub tokens for alice_restore and bob_restore" || { ko "PrivaPub tokens for alice_restore and bob_restore"; return 1; }
|
||||
run=$(date +%s)
|
||||
alice=$(curl -s -H "$AH" "$P/api/v1/accounts/verify_credentials")
|
||||
alice_id=$(echo "$alice" | j "print(d['id'])")
|
||||
alice_uri=$(echo "$alice" | j "print(d['url'])" | sed 's|/@|/peasants/|')
|
||||
bob_id=$(curl -s -H "Authorization: Bearer $BT" "$P/api/v1/accounts/verify_credentials" | j "print(d['id'])")
|
||||
regret=$(curl -s -X POST -H "$AH" "$P/api/v1/statuses" -d "status=a post regretted later $run&visibility=public")
|
||||
regret_id=$(echo "$regret" | j "print(d['id'])"); regret_uri=$(echo "$regret" | j "print(d['uri'])")
|
||||
# mastouser follows nobody here yet: the follow comes after the backup
|
||||
m_follows() { m_rails "puts Follow.exists?(account: Account.find_local(\"mastouser\"), target_account: Account.find_by(uri: \"$alice_uri\")) ? \"True\" : \"False\""; }
|
||||
alice_on_m=$(mcurl -H "Authorization: Bearer $(mastodon_token)" "$M/api/v2/search?q=@alice_restore@privapub.test&resolve=true&type=accounts" | j "print(d['accounts'][0]['id'])")
|
||||
if [ "$(m_follows)" = "True" ]; then
|
||||
mcurl -o /dev/null -X POST -H "Authorization: Bearer $(mastodon_token)" "$M/api/v1/accounts/$alice_on_m/unfollow"
|
||||
until_true 30 '[ "$(m_follows)" = "False" ]'
|
||||
fi
|
||||
curl -s -o /dev/null -X POST -H "$AH" "$P/api/v1/accounts/$bob_id/unblock"
|
||||
|
||||
backups() { curl -s -H "$RH" "$P/clientapi/admin/backups"; }
|
||||
before=$(backups | j "print(' '.join(b['id'] for b in d['backups']))")
|
||||
[ "$(curl -s -o /dev/null -w '%{http_code}' -X POST -H "$RH" "$P/clientapi/admin/backups")" = "202" ] && ok "a backup is asked for" || ko "the backup was refused"
|
||||
backup=""
|
||||
newest() { backups | j "
|
||||
ids=[b['id'] for b in d['backups'] if b['id'] not in '$before'.split()]
|
||||
print(ids[0] if ids and d.get('running') is None else '')"; }
|
||||
until_true 300 '[ -n "$(newest)" ]' && backup=$(newest)
|
||||
[ -n "$backup" ] && ok "backed up as $backup" || { ko "the backup was never made"; return 1; }
|
||||
|
||||
echo " life goes on"
|
||||
curl -s -o /dev/null -X DELETE -H "$AH" "$P/api/v1/statuses/$regret_id"
|
||||
since=$(curl -s -X POST -H "$AH" "$P/api/v1/statuses" -d "status=made after the backup $run&visibility=public")
|
||||
since_uri=$(echo "$since" | j "print(d['uri'])")
|
||||
mcurl -o /dev/null -X POST -H "Authorization: Bearer $(mastodon_token)" "$M/api/v1/accounts/$alice_on_m/follow"
|
||||
until_true 45 '[ "$(m_follows)" = "True" ]' && ok "mastouser follows alice after the backup" || ko "mastouser never followed alice"
|
||||
until_true 30 '[ "$(curl -s -H "$AH" "$P/api/v1/accounts/$alice_id/followers" | j "print(any(a[\"acct\"]==\"mastouser@mastodon.test\" for a in d))")" = "True" ]' \
|
||||
&& ok "PrivaPub has mastouser following alice" || ko "PrivaPub never had the follower"
|
||||
curl -s -o /dev/null -X POST -H "$AH" "$P/api/v1/accounts/$bob_id/block"
|
||||
carol="carol_restore_$run"
|
||||
curl -s -o /dev/null -X POST -H "$RH" -H 'Content-Type: application/json' "$P/clientapi/avatar/private/insert" \
|
||||
-d "{\"userName\":\"$carol\",\"name\":\"carol\",\"biography\":\"made after the backup\"}"
|
||||
|
||||
echo " restored"
|
||||
[ "$(curl -s -o /dev/null -w '%{http_code}' -X POST -H "$RH" -H 'Content-Type: application/json' "$P/clientapi/admin/backups/$backup/restore" \
|
||||
-d "{\"password\":\"$ROOT_PASS\",\"host\":\"elsewhere.test\"}")" = "422" ] && ok "a restore with the wrong host is refused" || ko "a restore with the wrong host was taken"
|
||||
[ "$(curl -s -o /dev/null -w '%{http_code}' -X POST -H "$RH" -H 'Content-Type: application/json' "$P/clientapi/admin/backups/$backup/restore" \
|
||||
-d "{\"password\":\"$ROOT_PASS\",\"host\":\"privapub.test\"}")" = "202" ] && ok "the restore is asked for" || { ko "the restore was refused"; return 1; }
|
||||
# it stops within seconds, restores as it starts again, and every session ends: the old token is refused once it is back
|
||||
until_true 300 '[ "$(curl -s -o /dev/null -w "%{http_code}" -H "$RH" "$P/clientapi/admin/backups")" = "401" ]' \
|
||||
&& ok "PrivaPub came back, and the administrator's session ended" || { ko "PrivaPub never came back from the restore"; podman logs --tail 30 pasture-privapub; return 1; }
|
||||
JWT=$(privapub_root); RH="Authorization: Bearer $JWT"
|
||||
[ "$(backups | j "print(d['lastRestore']['backup'])")" = "$backup" ] && ok "the last restore is $backup" || ko "the last restore is not $backup"
|
||||
[ "$(curl -s -o /dev/null -w '%{http_code}' -H "$AH" "$P/api/v1/accounts/verify_credentials")" = "401" ] && ok "alice's old token is refused" || ko "alice's old token still works"
|
||||
AT=$(privapub_token alice_restore); AH="Authorization: Bearer $AT"
|
||||
|
||||
[ "$(curl -s -o /dev/null -w '%{http_code}' -H "$AH" "$P/api/v1/statuses/$regret_id")" = "404" ] && ok "the post deleted after the backup stays deleted" || ko "the deleted post came back"
|
||||
gone_unsigned "$regret_uri" && ok "its address answers as gone" || ko "its address answers $(pstatus "$regret_uri")"
|
||||
gone_unsigned "$since_uri" && ok "the post made after the backup answers as gone" || ko "the post made after the backup answers $(pstatus "$since_uri")"
|
||||
[ "$(curl -s -H "$AH" "$P/api/v1/accounts/$alice_id/followers" | j "print(any(a['acct']=='mastouser@mastodon.test' for a in d))")" = "True" ] \
|
||||
&& ok "mastouser still follows alice" || ko "the follower gained after the backup was lost"
|
||||
[ "$(m_follows)" = "True" ] && ok "Mastodon still has the follow" || ko "Mastodon lost the follow"
|
||||
[ "$(curl -s -H "$AH" "$P/api/v1/accounts/relationships?id[]=$bob_id" | j "print(d[0]['blocking'])")" = "True" ] \
|
||||
&& ok "alice still blocks bob_restore" || ko "the block made after the backup was undone"
|
||||
[ "$(curl -s -o /dev/null -w '%{http_code}' -X POST -H "$RH" -H 'Content-Type: application/json' "$P/clientapi/avatar/private/insert" \
|
||||
-d "{\"userName\":\"$carol\",\"name\":\"carol\",\"biography\":\"again\"}")" != "200" ] && ok "carol's name stays taken" || ko "carol's name was free again"
|
||||
status=$(podman exec -w /app pasture-privapub /app/PrivaPub admin restore --status 2>/dev/null | grep -o "personas [^,;]*" | head -1)
|
||||
echo "$status" | grep -q "$carol" && ok "the report names carol among the personas made since" || ko "the report: $status"
|
||||
|
||||
# the town signs in again; the record goes, so the followers' digests (FEP-8fcf, followsync) are not resting for two weeks
|
||||
"$here/town.sh" renew privapub >/dev/null && ok "the town's PrivaPub accounts signed in again" || ko "the town could not sign in again"
|
||||
p_mongo "db.RestoreRecord.deleteMany({})" >/dev/null
|
||||
@@ -1,4 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
# The town: a fake community seeded across every pasture peer, then checked for coherence (see town/town.py).
|
||||
# usage: tools/pasture/town.sh <command> [args] commands: selftest, seed, check, report, backlog, gaps, drive, stored, seen
|
||||
# usage: tools/pasture/town.sh <command> [args] commands: selftest, seed, check, report, backlog, gaps, drive, stored, seen,
|
||||
# renew <peer> (its accounts signed in again)
|
||||
exec python3 "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/town/town.py" "$@"
|
||||
@@ -5,6 +5,7 @@
|
||||
drive <peer> <user> <verb> '<json>' one action as one town account, printed as JSON
|
||||
stored <peer> <uri>... what a peer holds of each object (from its database, never fetching)
|
||||
seen <peer> <user> <uri>... what one account can see of each object
|
||||
renew <peer>... its accounts signed in again (after a PrivaPub restore ends every session)
|
||||
plan <spec> the deterministic plan of a spec, printed as JSON lines
|
||||
seed <spec> replays a spec's plan against the pasture, writing out/<run>/ledger.jsonl
|
||||
check [run] sweeps every peer for what the ledger expects
|
||||
@@ -47,6 +48,13 @@ def main(argv):
|
||||
rows = driver(args[0]).stored(args[1:])
|
||||
print(json.dumps({u: {k: v for k, v in r.__dict__.items() if k != "raw"} for u, r in rows.items()}, default=str))
|
||||
return 0
|
||||
if cmd == "renew":
|
||||
# a peer's town accounts signed in again (a PrivaPub restore ends every session): new tokens, nothing else
|
||||
for platform in args:
|
||||
stored = state.sessions(platform)
|
||||
state.remember(driver(platform).provision([s.account for s in stored]))
|
||||
print(f"{platform}: {len(stored)} accounts signed in again")
|
||||
return 0
|
||||
if cmd == "seen":
|
||||
s = state.session(args[0], args[1])
|
||||
print(json.dumps(driver(args[0]).seen(s, args[2:])))
|
||||
|
||||
Reference in new issue
Block a user