Compare commits

...
7 Commits
Author SHA1 Message Date
thepraandClaude Opus 5.5 436f7da464 CDNs found by themselves, and servers followed through time
Build / Build (push) Successful in 5m11s
Deploy / privapub.thepra.dev (push) Successful in 5m48s
PrivaPub now finds CDNs three ways, best first: the address ranges the
CDNs publish (Cloudflare, Fastly, Amazon CloudFront, Bunny, Gcore,
Imperva), downloaded daily by CdnUpdater and kept in CdnRangeSet; the
CDN's fingerprint in the responses it already gets from a server
(EdgeHintsHandler on the federation client); and the networks that carry
only a CDN. The fixed ASN list is gone; ASNs shared with plain hosting
(AWS, DataPacket) no longer hide a server. A server's Geo records the
CDN, its domain and how it was found, and weekly snapshots now keep the
city and coordinates too.

Servers through time (ServerPlaces): /instances/:host/history lists a
server's weekly snapshots, a CDN-fronted server's geo names the CDN's
domain and where the server was before it (before_cdn), and
/api/privapub/v1/cdns and /cdns/:domain group servers by CDN with week
by week who joined and who left. Owner decisions recorded in ROADMAP.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
2026-10-04 11:33:39 +02:00
thepraandClaude Opus 5.5 5d7edc4a4c ROADMAP: no size or bucket restrictions on the public statistics
The plan for the public /stargazing statistics showed per-server
activity only as order-of-magnitude buckets and only for servers
reporting at least 10 users. The owner never intended either
restriction; both are gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
2026-10-04 11:20:44 +02:00
thepraandClaude Opus 5.5 eb7552e8f3 Server locations: no user threshold
The public projection of a server's place showed only the country for
servers reporting fewer than 10 users. The owner never decided that
threshold: every located server now shows its city, coordinates (0.1°)
and network, and a CDN-fronted one still shows only its CDN, since the
address reached is the CDN's edge. Statistics:PublicCityMinUsers is
gone; the ROADMAP decision on server locations, CLAUDE.md and the
/stargazing explainer are corrected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
2026-10-04 11:12:02 +02:00
thepraandClaude Opus 5.5 31d1d21d1e Pixelfed places: a remote post's own place, shown and never federated again
A Note's `location` that is a Place with coordinates (Pixelfed sends
them as strings) is kept in the new Post.Place and shown as
Status.privapub.place {name, latitude, longitude, country}; an edit
replaces it. An Event's location stays its own `places`, and nothing
renders a place back out. Closes the INTEROP P2 Pixelfed location gap
and the ROADMAP long-tail item.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
2026-10-04 10:07:06 +02:00
thepraandClaude Opus 5.5 d9fb5c582f Where a server is, on the public instance API
/api/privapub/v1/instances/:host gains `geo`, the public projection of a
server's place already decided for public server locations
(PublicGeo.Project): city, coordinates and network for servers reporting
at least 10 users and not behind a CDN, the country otherwise, the CDN's
name for a CDN-fronted one, with DB-IP's attribution. `?host[]=`
answers up to 40 servers at once, and this server describes itself:
its host's address located once a day (SelfLocation), or
Statistics:Geo:Self when the owner sets it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
2026-10-04 10:03:59 +02:00
thepraandClaude Opus 5.5 2cd75176a4 One sign-in for decePubClient: the root JWT exchanged for a persona's token
The first-party client signs in on /clientapi and exchanges that JWT on
/oauth/token (RFC 8693, subject_token_type jwt, avatar_id) for one
persona's Mastodon token. Only the seeded public application `decepub`
holds the grant; RootJwtSubjectToken validates the JWT through RootJwt,
which JwtBearer now shares (signature, lifetime, ban, deletion, session
stamp). The issued token names the avatar, never the root.

Owner decision recorded in ROADMAP; it supersedes "moving decePubClient
onto the Mastodon API is out of scope".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
2026-10-04 10:03:46 +02:00
thepraandClaude Opus 5.5 c7e8760ffe docs: v1.20.0 deployed and verified; everything on in production
Build / Build (push) Successful in 4m34s
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELjqpznMFMNrJoJUj6K5p2
2026-10-04 04:27:40 +02:00
45 changed files with 2240 additions and 128 deletions

No files matched your search

+24 -5
View File
@@ -1,4 +1,4 @@
# CLAUDE.md
# CLAUDE.md
Guidance for working in this repository. `docs/ROADMAP.md` holds the owner's decisions and the phased plan to full
ActivityPub interop; read it before changing anything federation-, privacy- or API-shaped. `docs/INTEROP.md` is the
@@ -29,8 +29,8 @@ Privacy features in the model:
The intended client is the Mastodon client API (Tusky, Elk, Phanpy, Ivory), where each avatar logs in as its own
account. The private `/clientapi` covers what Mastodon can't express. `thepra/decePubClient`
(https://decepub.thepra.dev) is the owner's own Pleroma-FE-like PWA. It still shows mock data and is not wired to the
server.
(https://decepub.thepra.dev) is the owner's own Pleroma-FE-like PWA. It signs in once on `/clientapi` and exchanges that
JWT for each persona's Mastodon token (`PersonaExchange`), then uses both APIs.
## Route names are deliberate
@@ -221,6 +221,12 @@ group www-data and reaches the private mongod; `sudo -u www-data` works too.
1. **A token is one persona.** Its subject is the avatar id; the root id lives only in the fifteen-minute `/oauth`
cookie used while choosing the persona, and never in a token, an authorization or a response.
- **The first-party client exchanges the root JWT for one persona's token** (RFC 8693 on `/oauth/token`,
`subject_token_type` `urn:ietf:params:oauth:token-type:jwt`, `avatar_id`; `Api/Mastodon/Auth/PersonaExchange.cs`).
Only the seeded public application (`ClientApi:FirstPartyClientId`, default `decepub`) holds the grant.
`RootJwtSubjectToken` validates the JWT through `RootJwt`, the same rules as JwtBearer (signature, lifetime, ban,
deletion, session stamp). The issued token is the same kind as the authorization code flow's, so revocation and
`RootSessions` cover it.
2. `/api/*` authenticates with OpenIddict validation, everything else with the old JWT (`PrivaPub` policy scheme).
Every `/api` request re-checks that the persona's root is neither banned nor deleted (`MastodonController`).
3. Read parameters through `Params` (query, form and JSON merged Rails-style), never MVC binding. A value type read
@@ -234,7 +240,8 @@ group www-data and reaches the private mongod; `sudo -u www-data` works too.
because clients show quotes from it, so everything a 4.3+ client calls because of it must answer: grouped
notifications (`/api/v2/notifications`, its unread count, groups, accounts and dismiss) and the notification policy.
7. **What a Mastodon `Status` cannot say goes in `Status.privapub`** (`PrivaPubStatus`): object type, title, excerpt,
cover, the author's source, link, video, audio and event details, and up/down votes. Every media URL in it goes
cover, the author's source, link, video, audio and event details, a remote post's own place (Pixelfed's `location`,
never federated again), and up/down votes. Every media URL in it goes
through the proxy; only page links (`link.url`, an event's online link) point at the remote site, because following
one is the reader's choice.
@@ -313,7 +320,19 @@ group www-data and reaches the private mongod; `sudo -u www-data` works too.
revision. Delivery details reach them through `Arrival.Current`, which `InboxProcessor` sets for the handler's
duration. A fetched record has no signature, key or `@context` of the activity that caused the fetch (its activity
fields name that trigger), and every record stores its `Extensions` and `ContextNamespaces` for statistics. The raw form lives outside `Post` so timelines never load it. Read through
`/api/privapub/v1/statuses/:id/provenance` and `/api/privapub/v1/instances/:host`.
`/api/privapub/v1/statuses/:id/provenance` and `/api/privapub/v1/instances/:host` (or `?host[]=`, up to 40; this
server answers too).
- **A server's place leaves only through `PublicGeo.Project`:** the city, coordinates (0.1°) and network it was located
to, whatever its size (there is no user threshold), and only the CDN's name for a CDN-fronted one. This server's own
place is its host's address located once a day (`SelfLocation`), or `Statistics:Geo:Self` when the owner sets it.
- **CDNs are found three ways, best first** (`Infrastructure/Geo/CdnCatalog.cs`): the ranges a CDN publishes
(`CdnUpdater` downloads them daily into `CdnRangeSet`, keeping the old set when a download or parse fails), the CDN's
fingerprint in the responses PrivaPub already gets (`EdgeHintsHandler` on the federation client, memory only), and
a network that carries nothing but a CDN. Never list an ASN shared with plain hosting (AWS, Google, Microsoft,
DataPacket). A server's `Geo` records the CDN, its domain and how it was found.
- **Servers through time** (`Domain/Statistics/ServerPlaces.cs`, from the weekly `RemoteInstanceSnapshot`s):
`/api/privapub/v1/instances/:host/history`, a CDN-fronted server's `geo.before_cdn`, and the CDNs grouped by their
own domain (`/api/privapub/v1/cdns`, `/cdns/:domain` with weekly joined/left).
- **Remote objects are parsed for every shape in `ObjectShapes`:** `url`/`icon`/`image` as a value, an object or an
array; Markdown `content`; missing `mediaType`s inferred; thumbnails from any of their five places; and the typed
`Link`, `Video`, `Audio` and `Event` details. A Mastodon API card is built from those, never by fetching the linked page
+46 -1
View File
@@ -1,4 +1,4 @@
using PrivaPub.Api.Mastodon.Mappers;
using PrivaPub.Api.Mastodon.Mappers;
using PrivaPub.Federation.Objects;
using PrivaPub.Models.Post;
@@ -79,6 +79,51 @@ namespace PrivaPub.Tests.Federation
Assert.Equal("2026-10-03 10:00 – 13:00 (Europe/Rome) · Biblioteca", MastodonMapper.EventLine(note.Event));
}
[Fact]
public void Keeps_a_pixelfed_place_whose_coordinates_are_strings()
{
var note = Parse("""
{
"id": "https://pixel.example/p/ann/1", "type": "Note", "attributedTo": "https://pixel.example/users/ann", "content": "<p>view</p>",
"location": { "type": "Place", "name": "Tokyo", "longitude": "139.6917", "latitude": "35.6895", "country": "Japan" },
"commentsEnabled": true
}
""");
Assert.Equal(("Tokyo", 35.6895, 139.6917, "Japan"), (note.Place.Name, note.Place.Latitude, note.Place.Longitude, note.Place.Country));
Assert.Null(note.Event);
}
[Theory]
[InlineData("null")]
[InlineData("""{ "type": "Place", "name": "Nowhere" }""")]
[InlineData("""{ "type": "Place", "name": "Off the map", "latitude": 123, "longitude": 9 }""")]
[InlineData("""{ "type": "Place", "latitude": "north", "longitude": "9.19" }""")]
[InlineData("""{ "type": "VirtualLocation", "url": "https://meet.example/a", "latitude": 45, "longitude": 9 }""")]
public void Has_no_place_without_a_place_with_coordinates(string location)
{
var note = Parse($$"""
{ "id": "https://pixel.example/p/ann/2", "type": "Note", "attributedTo": "https://pixel.example/users/ann", "content": "<p>x</p>", "location": {{location}} }
""");
Assert.Null(note.Place);
}
[Fact]
public void An_events_location_stays_the_events_own()
{
var note = Parse("""
{
"id": "https://events.example/events/10", "type": "Event", "name": "Picnic", "startTime": "2026-10-03T10:00:00+02:00",
"location": { "type": "Place", "name": "Park", "latitude": 45.07, "longitude": 7.68 },
"attributedTo": "https://events.example/@club"
}
""");
Assert.Null(note.Place);
Assert.Equal("Park", Assert.Single(note.Event.Places).Name);
}
[Fact]
public void Keeps_a_lemmy_link_post_as_a_link_with_its_thumbnail()
{
@@ -0,0 +1,291 @@
using MongoDB.Entities;
using PrivaPub.Models.Jobs;
using PrivaPub.Tests.Support;
using PrivaPub.Tests.Support.Host;
using System.Net;
using System.Text.Json.Nodes;
namespace PrivaPub.Tests.Http
{
// Where a server is, on the public instance API: where it was located (city, coordinates and network, whatever its size),
// only the CDN's name for a CDN-fronted one, the batch form, and this server itself.
[Trait("Category", "Integration")]
public sealed class InstanceLocationTests : IAsyncLifetime
{
PrivaPubHost _host;
public async ValueTask InitializeAsync()
{
Assert.SkipUnless(MongoFixture.Enabled, MongoFixture.Skip);
_host = await PrivaPubHost.Shared();
}
public ValueTask DisposeAsync() => ValueTask.CompletedTask;
static CancellationToken Token => TestContext.Current.CancellationToken;
static async Task<string> Instance(int users, InstanceGeo geo, string name = "located")
{
var host = $"{name}{Guid.NewGuid():N}.example";
await DB.Default.SaveAsync(new RemoteInstance { Host = host, Software = "mastodon", UsersTotal = users, Geo = geo }, Token);
return host;
}
static InstanceGeo Milan(string cdn = default) => new()
{
Country = "IT",
City = "Milan",
Latitude = 45.5,
Longitude = 9.2,
Asn = 12345,
AsnOrg = "Example Hosting",
Cdn = cdn,
Ipv6 = true,
Source = "DB-IP Lite 2026-10",
LocatedAt = DateTime.UtcNow.AddDays(-2)
};
async Task<JsonNode> Geo(string host) => (await _host.Client().Get($"/api/privapub/v1/instances/{host}")).Ok().Body["geo"];
[Fact]
public async Task A_located_server_shows_its_city_coordinates_and_network()
{
var geo = await Geo(await Instance(50, Milan()));
Assert.Equal("city", geo.Text("precision"));
Assert.Equal(("IT", "Milan", "Example Hosting"), (geo.Text("country"), geo.Text("city"), geo.Text("network")));
Assert.Equal(45.5, geo["latitude"]!.GetValue<double>());
Assert.Equal(9.2, geo["longitude"]!.GetValue<double>());
Assert.Equal("DB-IP Lite 2026-10", geo.Text("source"));
Assert.NotNull(geo.Text("located_at"));
Assert.Null(geo["asn"]);
Assert.Null(geo["ipv6"]);
}
[Fact]
public async Task A_one_user_server_shows_its_city_too()
{
var geo = await Geo(await Instance(1, Milan()));
Assert.Equal("city", geo.Text("precision"));
Assert.Equal(("IT", "Milan", "Example Hosting"), (geo.Text("country"), geo.Text("city"), geo.Text("network")));
Assert.Equal(45.5, geo["latitude"]!.GetValue<double>());
}
[Fact]
public async Task A_server_located_only_to_its_country_shows_the_country()
{
var geo = await Geo(await Instance(50, new InstanceGeo { Country = "IT", Source = "DB-IP Lite 2026-10", LocatedAt = DateTime.UtcNow }));
Assert.Equal("country", geo.Text("precision"));
Assert.Equal("IT", geo.Text("country"));
Assert.Null(geo["city"]);
Assert.Null(geo["latitude"]);
}
[Fact]
public async Task A_cdn_fronted_server_shows_only_its_cdn()
{
var geo = await Geo(await Instance(5000, Milan(cdn: "Cloudflare")));
Assert.Equal("cdn", geo.Text("precision"));
Assert.Equal("Cloudflare", geo.Text("cdn"));
Assert.Null(geo["country"]);
Assert.Null(geo["latitude"]);
Assert.Null(geo["network"]);
}
[Fact]
public async Task An_unlocated_server_has_no_place()
{
Assert.Null(await Geo(await Instance(50, default)));
}
[Fact]
public async Task The_provenance_of_a_post_carries_the_same_projection()
{
var host = await Instance(3, Milan());
var described = (await _host.Client().Get($"/api/privapub/v1/instances/{host}")).Ok().Body;
Assert.Equal("city", described["geo"].Text("precision"));
Assert.Equal(described["geo"]!.ToJsonString(), (await Geo(host))!.ToJsonString());
}
[Fact]
public async Task Several_servers_answer_at_once_in_the_order_asked_without_the_unknown_ones()
{
var big = await Instance(50, Milan(), "big");
var small = await Instance(3, Milan(), "small");
var unknown = $"unknown{Guid.NewGuid():N}.example";
var answer = (await _host.Client().Get($"/api/privapub/v1/instances?host[]={small}&host[]={unknown}&host[]={big.ToUpperInvariant()}&host[]={small}")).Ok();
var hosts = answer.Body!.AsArray().Select(i => i.Text("host")).ToList();
Assert.Equal(new[] { small, big }, hosts);
Assert.Equal("city", answer.Body![0]["geo"].Text("precision"));
Assert.Equal("city", answer.Body![1]["geo"].Text("precision"));
Assert.Empty((await _host.Client().Get("/api/privapub/v1/instances")).Ok().Body!.AsArray());
}
[Fact]
public async Task At_most_forty_servers_answer_at_once()
{
var hosts = new List<string>();
for (var i = 0; i < 42; i++)
hosts.Add(await Instance(1, default, "many"));
var answer = (await _host.Client().Get("/api/privapub/v1/instances?" + string.Join("&", hosts.Select(h => "host[]=" + h)))).Ok();
Assert.Equal(hosts.Take(40), answer.Body!.AsArray().Select(i => i.Text("host")));
}
[Fact]
public async Task This_server_describes_itself()
{
var self = (await _host.Client().Get($"/api/privapub/v1/instances/{PrivaPubHost.Host}")).Ok().Body;
Assert.Equal(PrivaPubHost.Host, self.Text("host"));
Assert.Equal("privapub", self.Text("software"));
Assert.Equal("activitypub", Assert.Single(self["protocols"]!.AsArray())!.GetValue<string>());
Assert.Null(self["geo"]);
var batch = (await _host.Client().Get($"/api/privapub/v1/instances?host[]={PrivaPubHost.Host}")).Ok().Body!.AsArray();
Assert.Equal("privapub", Assert.Single(batch).Text("software"));
}
[Fact]
public async Task The_owner_may_place_this_server()
{
var placed = await SelfPlacedHost.Shared();
var geo = (await placed.Client().Get($"/api/privapub/v1/instances/{PrivaPubHost.Host}")).Ok().Body["geo"];
Assert.Equal("city", geo.Text("precision"));
Assert.Equal(("IT", "Milan", "configured"), (geo.Text("country"), geo.Text("city"), geo.Text("source")));
Assert.Equal(45.5, geo["latitude"]!.GetValue<double>());
Assert.Equal(9.2, geo["longitude"]!.GetValue<double>());
}
static string WeeksAgo(int weeks) => PrivaPub.Federation.Objects.InstanceDescriber.Week(DateTime.UtcNow.AddDays(-7 * weeks));
static Task Snapshot(string host, int weeksAgo, string cdn = default, string cdnDomain = default, string city = default, double? latitude = default,
double? longitude = default, string country = default) =>
DB.Default.SaveAsync(new RemoteInstanceSnapshot
{
Host = host, Week = WeeksAgo(weeksAgo), TakenAt = DateTime.UtcNow.AddDays(-7 * weeksAgo), Reachable = true, Software = "mastodon", UsersTotal = 3,
Cdn = cdn, CdnDomain = cdnDomain, CdnSource = cdn == default ? default : "ranges", Country = country, City = city, Latitude = latitude, Longitude = longitude
}, Token);
[Fact]
public async Task A_cdn_fronted_server_names_its_cdn_and_where_it_was_before_it()
{
var host = await Instance(3, new InstanceGeo
{
Country = "US", City = "San Francisco", Latitude = 37.8, Longitude = -122.4, Cdn = "Cloudflare", CdnDomain = "cloudflare.com", CdnSource = "ranges",
Source = "DB-IP Lite 2026-10", LocatedAt = DateTime.UtcNow
}, "fronted");
await Snapshot(host, 3, city: "Berlin", latitude: 52.5, longitude: 13.4, country: "DE");
await Snapshot(host, 2, city: "Hamburg", latitude: 53.6, longitude: 10.0, country: "DE");
await Snapshot(host, 1, cdn: "Cloudflare", cdnDomain: "cloudflare.com", city: "San Francisco", latitude: 37.8, longitude: -122.4, country: "US");
var geo = await Geo(host);
Assert.Equal(("cdn", "Cloudflare", "cloudflare.com", "ranges"), (geo.Text("precision"), geo.Text("cdn"), geo.Text("cdn_domain"), geo.Text("cdn_source")));
Assert.Null(geo["latitude"]);
Assert.Equal((WeeksAgo(2), "Hamburg", "DE"), (geo["before_cdn"].Text("week"), geo["before_cdn"].Text("city"), geo["before_cdn"].Text("country")));
Assert.Equal(53.6, geo["before_cdn"]!["latitude"]!.GetValue<double>());
}
[Fact]
public async Task A_server_history_is_weekly_newest_first_and_a_cdn_hides_its_place()
{
var host = await Instance(3, Milan(), "historic");
await Snapshot(host, 2, city: "Milan", latitude: 45.5, longitude: 9.2, country: "IT");
await Snapshot(host, 1, cdn: "Fastly", cdnDomain: "fastly.com", city: "Frankfurt", latitude: 50.1, longitude: 8.7, country: "DE");
var weeks = (await _host.Client().Get($"/api/privapub/v1/instances/{host}/history")).Ok().Body!.AsArray();
Assert.Equal(new[] { WeeksAgo(1), WeeksAgo(2) }, weeks.Select(w => w.Text("week")));
Assert.Equal(("cdn", "fastly.com"), (weeks[0]!["geo"].Text("precision"), weeks[0]!["geo"].Text("cdn_domain")));
Assert.Null(weeks[0]!["geo"]!["city"]);
Assert.Equal(("city", "Milan"), (weeks[1]!["geo"].Text("precision"), weeks[1]!["geo"].Text("city")));
Assert.Equal(3, weeks[1]!["users_total"]!.GetValue<long>());
Assert.Single((await _host.Client().Get($"/api/privapub/v1/instances/{host}/history?weeks=1")).Ok().Body!.AsArray());
Assert.Equal(HttpStatusCode.NotFound, (await _host.Client().Get($"/api/privapub/v1/instances/unknown{Guid.NewGuid():N}.example/history")).Status);
}
[Fact]
public async Task Servers_are_grouped_by_their_cdn_week_by_week()
{
var cdn = $"Edge{Guid.NewGuid():N}"[..20];
var key = cdn.ToLowerInvariant();
var stayed = await Instance(3, new InstanceGeo { Country = "FR", Cdn = cdn, CdnSource = "headers" }, "stayed");
var moved = await Instance(3, Milan(), "moved");
await Snapshot(moved, 3, cdn: cdn);
await Snapshot(moved, 2, cdn: cdn);
await Snapshot(stayed, 2, cdn: cdn);
await Snapshot(moved, 1, city: "Milan", latitude: 45.5, longitude: 9.2, country: "IT");
await Snapshot(stayed, 1, cdn: cdn);
var list = (await _host.Client().Get("/api/privapub/v1/cdns")).Ok().Body!.AsArray();
var group = list.Single(g => g.Text("key") == key);
Assert.Equal((cdn, 1), (group.Text("name"), group["servers"]!.GetValue<int>()));
var cloudflare = list.Single(g => g.Text("key") == "cloudflare.com");
Assert.Equal(new[] { "ranges", "headers", "asn" }, cloudflare["detected_by"]!.AsArray().Select(v => v!.GetValue<string>()));
var detail = (await _host.Client().Get($"/api/privapub/v1/cdns/{key}")).Ok().Body;
Assert.Equal(new[] { stayed }, detail["hosts"]!.AsArray().Select(h => h!.GetValue<string>()));
var weeks = detail["weeks"]!.AsArray();
Assert.Equal(new[] { WeeksAgo(1), WeeksAgo(2), WeeksAgo(3) }, weeks.Select(w => w.Text("week")));
Assert.Equal(new[] { 1, 2, 1 }, weeks.Select(w => w!["servers"]!.GetValue<int>()));
Assert.Equal(new[] { moved }, weeks[0]!["left"]!.AsArray().Select(h => h!.GetValue<string>()));
Assert.Equal(new[] { stayed }, weeks[1]!["joined"]!.AsArray().Select(h => h!.GetValue<string>()));
Assert.Equal(new[] { moved }, weeks[2]!["joined"]!.AsArray().Select(h => h!.GetValue<string>()));
Assert.Equal("Cloudflare", (await _host.Client().Get("/api/privapub/v1/cdns/cloudflare.com")).Ok().Body.Text("name"));
Assert.Equal(HttpStatusCode.NotFound, (await _host.Client().Get($"/api/privapub/v1/cdns/none{Guid.NewGuid():N}")).Status);
}
[Fact]
public async Task Junk_hosts_are_not_an_error()
{
Assert.Equal(HttpStatusCode.NotFound, (await _host.Client().Get($"/api/privapub/v1/instances/{new string('a', 300)}")).Status);
Assert.Empty((await _host.Client().Get($"/api/privapub/v1/instances?host[]=&host[]={new string('a', 300)}")).Ok().Body!.AsArray());
}
}
public sealed class SelfPlacedHost : PrivaPubHost
{
static readonly SemaphoreSlim Boot = new(1, 1);
static SelfPlacedHost _shared;
public static new async Task<SelfPlacedHost> Shared()
{
await PrivaPubHost.Shared();
await Boot.WaitAsync();
try
{
if (_shared == default)
{
var host = new SelfPlacedHost();
_ = host.Services;
_shared = host;
}
return _shared;
}
finally
{
Boot.Release();
}
}
protected override IEnumerable<KeyValuePair<string, string>> Settings() => base.Settings().Concat(new Dictionary<string, string>
{
["Statistics:Geo:Self:Latitude"] = "45.4642",
["Statistics:Geo:Self:Longitude"] = "9.19",
["Statistics:Geo:Self:City"] = "Milan",
["Statistics:Geo:Self:Country"] = "IT"
});
}
}
+218
View File
@@ -0,0 +1,218 @@
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.IdentityModel.JsonWebTokens;
using Microsoft.IdentityModel.Tokens;
using PrivaPub.Api.Mastodon.Auth;
using PrivaPub.Services;
using PrivaPub.Tests.Support;
using PrivaPub.Tests.Support.Host;
using System.Net;
using System.Security.Claims;
using System.Text;
namespace PrivaPub.Tests.Http
{
// The first-party client's one sign-in: the /clientapi JWT exchanged for one persona's Mastodon token (PersonaExchange).
[Trait("Category", "Integration")]
public sealed class PersonaExchangeTests : IAsyncLifetime
{
const string Grant = "urn:ietf:params:oauth:grant-type:token-exchange";
const string FirstParty = "decepub";
PrivaPubHost _host;
public async ValueTask InitializeAsync()
{
Assert.SkipUnless(MongoFixture.Enabled, MongoFixture.Skip);
_host = await PrivaPubHost.Shared();
}
public ValueTask DisposeAsync() => ValueTask.CompletedTask;
async Task<HttpResponseMessage> Exchange(string jwt, string avatarId, string clientId = FirstParty, string clientSecret = default,
string subjectTokenType = PersonaExchange.SubjectTokenType)
{
var fields = new List<(string, string)>
{
("grant_type", Grant),
("client_id", clientId),
("subject_token", jwt),
("subject_token_type", subjectTokenType),
(PersonaExchange.AvatarParameter, avatarId),
("scope", "read write follow")
};
if (clientSecret != default)
fields.Add(("client_secret", clientSecret));
using var client = _host.Client();
return await client.Form("/oauth/token", fields.Where(f => f.Item2 != default).ToArray());
}
async Task<string> Token(Persona persona)
{
var response = await Exchange(persona.Root.Jwt, persona.Id);
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
return (await response.JsonBody())["access_token"]!.GetValue<string>();
}
static async Task AssertRefused(HttpResponseMessage response, string error = "invalid_grant")
{
Assert.Equal(HttpStatusCode.BadRequest, response.StatusCode);
Assert.Equal(error, (await response.JsonBody())["error"]?.GetValue<string>());
}
async Task<HttpStatusCode> Me(string token)
{
using var api = _host.As(token);
return (await api.GetAsync("/api/v1/accounts/verify_credentials", TestContext.Current.CancellationToken)).StatusCode;
}
static string Jwt(string rootId, string key, DateTime expires) => new JsonWebTokenHandler().CreateToken(new SecurityTokenDescriptor
{
Issuer = PrivaPubHost.Base,
Audience = PrivaPubHost.Base,
Subject = new ClaimsIdentity(new[] { new Claim(ClaimTypes.UserData, rootId) }),
NotBefore = expires.AddHours(-2),
IssuedAt = expires.AddHours(-2),
Expires = expires,
SigningCredentials = new(new SymmetricSecurityKey(Encoding.UTF8.GetBytes(key)), SecurityAlgorithms.HmacSha512)
});
[Fact]
public async Task The_root_jwt_becomes_one_personas_token_that_never_names_the_root()
{
var persona = await _host.Persona(await _host.SignUp(), "exchange");
var root = persona.Root;
var response = await Exchange(root.Jwt, persona.Id);
var body = await response.Content.ReadAsStringAsync(TestContext.Current.CancellationToken);
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
Assert.DoesNotContain(root.Id, body);
Assert.DoesNotContain(root.UserName, body);
var token = (await response.JsonBody())["access_token"]!.GetValue<string>();
using var api = _host.As(token);
var account = await api.GetStringAsync("/api/v1/accounts/verify_credentials", TestContext.Current.CancellationToken);
Assert.Contains($"\"id\":\"{persona.Id}\"", account);
Assert.DoesNotContain(root.Id, account);
foreach (var entry in await ClientApi.StoredTokens(FirstParty))
{
var payload = entry.Contains("payload") && entry["payload"].IsString ? ClientApi.JwtPayload(entry["payload"].AsString) : string.Empty;
Assert.DoesNotContain(root.Id, entry.ToString() + payload);
Assert.DoesNotContain(root.UserName, entry.ToString() + payload);
}
}
[Fact]
public async Task Each_persona_of_the_root_gets_its_own_token()
{
var root = await _host.SignUp();
var first = await _host.Persona(root, "first");
var second = await _host.Persona(root, "second");
using var firstApi = _host.As(await Token(first));
using var secondApi = _host.As(await Token(second));
Assert.Contains($"\"id\":\"{first.Id}\"", await firstApi.GetStringAsync("/api/v1/accounts/verify_credentials", TestContext.Current.CancellationToken));
Assert.Contains($"\"id\":\"{second.Id}\"", await secondApi.GetStringAsync("/api/v1/accounts/verify_credentials", TestContext.Current.CancellationToken));
}
[Fact]
public async Task Another_roots_persona_is_refused()
{
var mine = await _host.SignUp();
var theirs = await _host.Persona(await _host.SignUp(), "theirs");
await AssertRefused(await Exchange(mine.Jwt, theirs.Id));
await AssertRefused(await Exchange(mine.Jwt, "000000000000000000000000"));
await AssertRefused(await Exchange(mine.Jwt, "not an id"));
}
[Fact]
public async Task A_token_that_is_not_a_valid_root_jwt_is_refused()
{
var persona = await _host.Persona(await _host.SignUp(), "forged");
var key = _host.Services.GetRequiredService<IConfiguration>()["AppConfiguration:Jwt:Key"]!;
await AssertRefused(await Exchange("garbage", persona.Id));
await AssertRefused(await Exchange(Jwt(persona.Root.Id, key, DateTime.UtcNow.AddMinutes(-10)), persona.Id));
await AssertRefused(await Exchange(Jwt(persona.Root.Id, new string('k', 64), DateTime.UtcNow.AddHours(1)), persona.Id));
}
[Fact]
public async Task Ended_sessions_refuse_the_jwt_and_revoke_the_exchanged_tokens()
{
var persona = await _host.Persona(await _host.SignUp(), "ended");
var token = await Token(persona);
Assert.Equal(HttpStatusCode.OK, await Me(token));
using (var scope = _host.Services.CreateScope())
await scope.ServiceProvider.GetRequiredService<IRootSessions>().Revoke(persona.Root.Id, TestContext.Current.CancellationToken);
await AssertRefused(await Exchange(persona.Root.Jwt, persona.Id));
Assert.Equal(HttpStatusCode.Unauthorized, await Me(token));
}
[Fact]
public async Task A_banned_root_is_refused()
{
var persona = await _host.Persona(await _host.SignUp(), "banned");
await ClientApi.Ban(persona.Root.Id);
try
{
await AssertRefused(await Exchange(persona.Root.Jwt, persona.Id));
}
finally
{
await ClientApi.Ban(persona.Root.Id, banned: false);
}
}
[Fact]
public async Task A_revoked_token_no_longer_works()
{
var persona = await _host.Persona(await _host.SignUp(), "revoked");
var token = await Token(persona);
using var client = _host.Client();
var revoked = await client.Form("/oauth/revoke", ("token", token), ("client_id", FirstParty));
Assert.Equal(HttpStatusCode.OK, revoked.StatusCode);
Assert.Equal(HttpStatusCode.Unauthorized, await Me(token));
}
[Fact]
public async Task Only_the_first_party_client_may_exchange()
{
var persona = await _host.Persona(await _host.SignUp(), "thirdparty");
using var client = _host.Client();
var app = await client.RegisterApp();
var response = await Exchange(persona.Root.Jwt, persona.Id, app.ClientId, app.ClientSecret);
Assert.Equal(HttpStatusCode.BadRequest, response.StatusCode);
Assert.Equal("unauthorized_client", (await response.JsonBody())["error"]?.GetValue<string>());
}
[Fact]
public async Task A_mastodon_token_is_not_a_subject_token()
{
var persona = await _host.Persona(await _host.SignUp(), "mastodon");
var token = await Token(persona);
await AssertRefused(await Exchange(token, persona.Id, subjectTokenType: "urn:ietf:params:oauth:token-type:access_token"), "invalid_request");
await AssertRefused(await Exchange(token, persona.Id));
}
[Fact]
public async Task Missing_parameters_are_a_client_error()
{
var persona = await _host.Persona(await _host.SignUp(), "missing");
await AssertRefused(await Exchange(persona.Root.Jwt, default));
await AssertRefused(await Exchange(default, persona.Id), "invalid_request");
await AssertRefused(await Exchange(persona.Root.Jwt, persona.Id, subjectTokenType: default), "invalid_request");
}
}
}
+106
View File
@@ -0,0 +1,106 @@
using MongoDB.Entities;
using PrivaPub.Models.Post;
using PrivaPub.Tests.Support;
using PrivaPub.Tests.Support.Host;
using System.Text.Json.Nodes;
namespace PrivaPub.Tests.Http
{
// A Pixelfed post's own place: stored from `location`, shown as `privapub.place`, changed by an edit.
[Trait("Category", "Integration")]
public sealed class PostPlaceTests : IAsyncLifetime
{
PrivaPubHost _host;
Peer _peer;
public async ValueTask InitializeAsync()
{
Assert.SkipUnless(MongoFixture.Enabled, MongoFixture.Skip);
_host = await PrivaPubHost.Shared();
_peer = await Peer.Start();
}
public async ValueTask DisposeAsync()
{
if (_peer != default)
await _peer.DisposeAsync();
}
static CancellationToken Token => TestContext.Current.CancellationToken;
static JsonObject Location(string name, string latitude, string longitude) => new()
{
["type"] = "Place", ["name"] = name, ["latitude"] = latitude, ["longitude"] = longitude, ["country"] = "Italy"
};
[Fact]
public async Task A_pixelfed_place_is_shown_and_follows_the_authors_edits()
{
var alice = await _host.Mastodon("alice");
var ann = new RemoteActor(_peer, "ann");
var post = await _host.PublicPostFrom(ann, alice, "<p>the duomo</p>", note => note["location"] = Location("Milan", "45.4642", "9.19"));
var place = (await alice.Client.Get($"/api/v1/statuses/{post.ID}")).Ok().Body["privapub"]!["place"];
Assert.Equal(("Milan", "Italy"), (place.Text("name"), place.Text("country")));
Assert.Equal(45.4642, place["latitude"]!.GetValue<double>());
Assert.Equal(9.19, place["longitude"]!.GetValue<double>());
await Edit(ann, alice, post, note => note["location"] = Location("Turin", "45.0703", "7.6869"));
Assert.Equal("Turin", (await alice.Client.Get($"/api/v1/statuses/{post.ID}")).Ok().Body["privapub"]!["place"].Text("name"));
await Edit(ann, alice, post, _ => { });
Assert.Null((await alice.Client.Get($"/api/v1/statuses/{post.ID}")).Ok().Body["privapub"]!["place"]);
Assert.Null((await DB.Default.Find<Post>().MatchID(post.ID).ExecuteFirstAsync(Token)).Place);
}
[Fact]
public async Task A_post_without_a_place_has_none()
{
var alice = await _host.Mastodon("alice");
var post = await _host.PublicPostFrom(new RemoteActor(_peer, "plain"), alice);
Assert.Null((await alice.Client.Get($"/api/v1/statuses/{post.ID}")).Ok().Body["privapub"]!["place"]);
}
[Fact]
public async Task A_local_post_never_carries_a_location()
{
var alice = await _host.Mastodon("alice");
var status = await alice.Status("made here");
using var client = _host.Client();
client.DefaultRequestHeaders.Accept.ParseAdd("application/activity+json");
var note = JsonNode.Parse(await client.GetStringAsync(status.Text("uri"), Token))!;
Assert.Null(note["location"]);
}
async Task Edit(RemoteActor author, Mastodon mentioned, Post post, Action<JsonObject> shape)
{
var note = new JsonObject
{
["id"] = post.ObjectURI,
["type"] = "Note",
["attributedTo"] = author.Id,
["to"] = new JsonArray(MastodonHelpers.Public),
["cc"] = new JsonArray(mentioned.Uri),
["content"] = "<p>the duomo, again</p>",
["published"] = post.CreationDate.ToString("O"),
["updated"] = DateTime.UtcNow.ToString("O"),
["tag"] = new JsonArray(new JsonObject { ["type"] = "Mention", ["href"] = mentioned.Uri, ["name"] = "@" + mentioned.UserName })
};
shape(note);
await _host.Deliver(author, mentioned.Mouth, new JsonObject
{
["id"] = $"{post.ObjectURI}/updates/{Guid.NewGuid():N}",
["type"] = "Update",
["actor"] = author.Id,
["to"] = new JsonArray(MastodonHelpers.Public),
["object"] = note
});
}
}
}
@@ -0,0 +1,111 @@
using PrivaPub.Infrastructure.Geo;
using PrivaPub.Infrastructure.Http;
using PrivaPub.Models.Jobs;
using System.Net;
namespace PrivaPub.Tests.Infrastructure
{
public class CdnCatalogTests
{
static HttpResponseMessage Response(params (string Name, string Value)[] headers)
{
var response = new HttpResponseMessage(HttpStatusCode.OK) { Content = new StringContent("{}") };
foreach (var (name, value) in headers)
response.Headers.TryAddWithoutValidation(name, value);
return response;
}
[Theory]
[InlineData("cf-ray", "8c1f0e2b4d5a1234-FRA", "cloudflare.com")]
[InlineData("server", "cloudflare", "cloudflare.com")]
[InlineData("x-amz-cf-id", "abc==", "cloudfront.net")]
[InlineData("via", "1.1 5a1b.cloudfront.net (CloudFront)", "cloudfront.net")]
[InlineData("x-served-by", "cache-fra-eddf8230045-FRA", "fastly.com")]
[InlineData("server", "BunnyCDN-DE1-1078", "bunny.net")]
[InlineData("akamai-grn", "0.1234", "akamai.com")]
[InlineData("x-azure-ref", "20261004T101010Z-abc", "azure.microsoft.com")]
[InlineData("x-vercel-id", "fra1::abc", "vercel.com")]
[InlineData("x-iinfo", "1-2-3", "imperva.com")]
public void A_cdn_is_recognised_by_its_fingerprint(string header, string value, string domain)
{
Assert.Equal(domain, CdnCatalog.FromResponse(Response((header, value)))?.Domain);
}
[Fact]
public void A_plain_server_has_no_cdn_and_an_unknown_cdn_names_itself()
{
Assert.Null(CdnCatalog.FromResponse(Response(("server", "nginx"), ("via", "1.1 varnish"))));
var named = CdnCatalog.FromResponse(Response(("x-cdn", "EdgeOne")));
Assert.Equal(("EdgeOne", null, "edgeone"), (named.Name, named.Domain, named.Key));
Assert.Null(CdnCatalog.FromResponse(Response(("x-cdn", "<script>"))));
}
[Fact]
public void Only_networks_that_carry_nothing_but_a_cdn_name_it()
{
Assert.Equal("Cloudflare", CdnCatalog.OfAsn(13335)?.Name);
Assert.Equal("Akamai", CdnCatalog.OfAsn(20940)?.Name);
Assert.Null(CdnCatalog.OfAsn(16509));//AWS: CloudFront only by its ranges or its headers
Assert.Null(CdnCatalog.OfAsn(60068));//DataPacket also rents servers
}
[Fact]
public void Every_published_list_format_is_read()
{
Assert.Equal(new[] { "173.245.48.0/20", "2400:cb00::/32" },
CdnCatalog.Parse(CdnListFormat.Lines, "173.245.48.0/20\n# comment\n\n2400:cb00::/32\n").Select(n => n.ToString()));
Assert.Equal(new[] { "89.187.188.227/32", "2400:52e0:1500::714:1/128" },
CdnCatalog.Parse(CdnListFormat.StringArray, """["89.187.188.227","2400:52e0:1500::714:1"]""").Select(n => n.ToString()));
Assert.Equal(new[] { "23.235.32.0/20", "2a04:4e40::/32" },
CdnCatalog.Parse(CdnListFormat.AddressesJson, """{"addresses":["23.235.32.0/20"],"ipv6_addresses":["2a04:4e40::/32"]}""").Select(n => n.ToString()));
Assert.Equal(new[] { "92.223.124.12/32", "2a03:90c0::/32" },
CdnCatalog.Parse(CdnListFormat.AddressesJson, """{"addresses":["92.223.124.12/32"],"addresses_v6":["2a03:90c0::/32"]}""").Select(n => n.ToString()));
Assert.Equal(new[] { "199.83.128.0/21", "2a02:e980::/29" },
CdnCatalog.Parse(CdnListFormat.ImpervaJson, """{"ipRanges":["199.83.128.0/21"],"ipv6Ranges":["2a02:e980::/29"],"res":0}""").Select(n => n.ToString()));
Assert.Equal(new[] { "13.32.0.0/15", "2600:9000::/28" },
CdnCatalog.Parse(CdnListFormat.AwsCloudFront, """
{"prefixes":[{"ip_prefix":"3.4.12.4/32","service":"AMAZON"},{"ip_prefix":"13.32.0.0/15","service":"CLOUDFRONT"}],
"ipv6_prefixes":[{"ipv6_prefix":"2600:9000::/28","service":"CLOUDFRONT"},{"ipv6_prefix":"2406:daba::/40","service":"EC2"}]}
""").Select(n => n.ToString()));
Assert.Throws<FormatException>(() => CdnCatalog.Parse(CdnListFormat.Lines, "173.245.48.0/20\nnot an address"));
}
[Fact]
public void The_published_ranges_name_the_cdn_of_an_address()
{
var ranges = new CdnRanges();
ranges.Replace(new CdnRangeSet { Key = "cloudflare.com", Name = "Cloudflare", Ranges = new() { "104.16.0.0/13", "2606:4700::/32" }, FetchedAt = DateTime.UtcNow });
Assert.Equal("Cloudflare", ranges.Of(IPAddress.Parse("104.18.2.3"))?.Name);
Assert.Equal("Cloudflare", ranges.Of(IPAddress.Parse("::ffff:104.18.2.3"))?.Name);
Assert.Equal("Cloudflare", ranges.Of(IPAddress.Parse("2606:4700::1"))?.Name);
Assert.Null(ranges.Of(IPAddress.Parse("203.0.113.7")));
Assert.Equal(2, ranges.Status["cloudflare.com"].Ranges);
}
[Fact]
public async Task Responses_leave_a_cdn_hint_for_their_host_until_one_comes_without()
{
var hints = new EdgeHints();
var answers = new Queue<HttpResponseMessage>(new[] { Response(("cf-ray", "1-FRA")), Response(("server", "nginx")) });
using var client = new HttpClient(new EdgeHintsHandler(hints) { InnerHandler = new Answering(answers) });
await client.GetAsync("https://social.example/.well-known/nodeinfo", TestContext.Current.CancellationToken);
Assert.Equal("Cloudflare", hints.Of("social.example")?.Name);
await client.GetAsync("https://social.example/nodeinfo/2.1", TestContext.Current.CancellationToken);
Assert.Null(hints.Of("social.example"));
}
sealed class Answering : HttpMessageHandler
{
readonly Queue<HttpResponseMessage> _answers;
public Answering(Queue<HttpResponseMessage> answers) => _answers = answers;
protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) =>
Task.FromResult(_answers.Dequeue());
}
}
}
@@ -0,0 +1,105 @@
using Microsoft.Extensions.Logging.Abstractions;
using MongoDB.Entities;
using PrivaPub.Infrastructure.Geo;
using PrivaPub.Infrastructure.Statistics;
using PrivaPub.Models.Jobs;
using PrivaPub.Tests.Support;
using System.Net;
namespace PrivaPub.Tests.Infrastructure
{
[Trait("Category", "Integration")]
public sealed class CdnUpdaterTests : IAsyncLifetime
{
Peer _peer;
CdnRanges _ranges;
CdnUpdater _updater;
string _key;
public async ValueTask InitializeAsync()
{
Assert.SkipUnless(MongoFixture.Enabled, MongoFixture.Skip);
_peer = await Peer.Start();
_ranges = new CdnRanges();
_updater = new CdnUpdater(new Clients(), new StaticOptions<StatisticsOptions>(new StatisticsOptions()), _ranges, NullLogger<CdnUpdater>.Instance);
_key = $"edge{Guid.NewGuid():N}.example";
}
public async ValueTask DisposeAsync()
{
if (_peer != default)
await _peer.DisposeAsync();
}
sealed class Clients : IHttpClientFactory
{
public HttpClient CreateClient(string name) => new();
}
CdnProvider Provider() => new("Test Edge", _key, Array.Empty<int>(), new[]
{
new CdnList($"{_peer.A}/edge/v4", CdnListFormat.Lines),
new CdnList($"{_peer.A}/edge/v6", CdnListFormat.StringArray)
}, _ => false);
static CancellationToken Token => TestContext.Current.CancellationToken;
[Fact]
public async Task The_published_ranges_are_downloaded_stored_and_used()
{
_peer.ServeText("/edge/v4", "198.51.100.0/24\n", "text/plain");
_peer.ServeText("/edge/v6", """["2001:db8::/32","192.0.2.9"]""", "application/json");
Assert.Equal(1, await _updater.Update(new[] { Provider() }, Token));
var stored = await DB.Default.Find<CdnRangeSet>().Match(s => s.Key == _key).ExecuteSingleAsync(Token);
Assert.Equal(new[] { "198.51.100.0/24", "2001:db8::/32", "192.0.2.9/32" }, stored.Ranges);
Assert.Equal("Test Edge", _ranges.Of(IPAddress.Parse("198.51.100.20"))?.Name);
Assert.Equal("Test Edge", _ranges.Of(IPAddress.Parse("192.0.2.9"))?.Name);
var restarted = new CdnRanges();
await restarted.Load(Token);
Assert.Equal("Test Edge", restarted.Of(IPAddress.Parse("2001:db8::7"))?.Name);
}
[Theory]
[InlineData("not a network")]
[InlineData("")]
public async Task A_broken_or_empty_list_keeps_the_stored_ranges(string broken)
{
_peer.ServeText("/edge/v4", "198.51.100.0/24", "text/plain");
_peer.ServeText("/edge/v6", "[]", "application/json");
Assert.Equal(1, await _updater.Update(new[] { Provider() }, Token));
_peer.ServeText("/edge/v4", broken, "text/plain");
Assert.Equal(0, await _updater.Update(new[] { Provider() }, Token));
Assert.Equal(new[] { "198.51.100.0/24" }, (await DB.Default.Find<CdnRangeSet>().Match(s => s.Key == _key).ExecuteSingleAsync(Token)).Ranges);
Assert.Equal("Test Edge", _ranges.Of(IPAddress.Parse("198.51.100.1"))?.Name);
}
// The real lists, from the CDNs themselves: set PRIVAPUB_TEST_CDN_LIVE=1 to check that every format still parses.
[Fact]
public async Task Every_cdns_real_list_still_parses()
{
Assert.SkipUnless(Environment.GetEnvironmentVariable("PRIVAPUB_TEST_CDN_LIVE") == "1", "set PRIVAPUB_TEST_CDN_LIVE=1 to download the real lists");
var listed = CdnCatalog.All.Where(p => p.Lists.Length > 0).ToList();
Assert.Equal(listed.Count, await _updater.Update(listed, Token));
Assert.Equal("Cloudflare", _ranges.Of(IPAddress.Parse("104.16.0.1"))?.Name);
foreach (var provider in listed)
Assert.True(_ranges.Status[provider.Key].Ranges > 0, provider.Name);
}
[Fact]
public async Task An_unreachable_list_changes_nothing()
{
Assert.Equal(0, await _updater.Update(new[] { Provider() }, Token));
Assert.False(await DB.Default.Find<CdnRangeSet>().Match(s => s.Key == _key).ExecuteAnyAsync(Token));
}
}
}
@@ -1,4 +1,4 @@
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Logging.Abstractions;
using PrivaPub.Infrastructure.Geo;
using PrivaPub.Infrastructure.Http;
@@ -45,7 +45,7 @@ namespace PrivaPub.Tests.Infrastructure
Assert.NotNull(fix);
Assert.Equal(2, fix.Country.Length);
Assert.Equal(13335, fix.Asn);
Assert.Equal("Cloudflare", CdnNetworks.Of(fix.Asn));
Assert.Equal("Cloudflare", CdnCatalog.OfAsn(fix.Asn)?.Name);
Assert.NotNull(fix.AsnOrg);
Assert.NotNull(fix.Latitude);
Assert.Equal(Math.Round(fix.Latitude.Value, 1), fix.Latitude);
@@ -61,9 +61,9 @@ namespace PrivaPub.Tests.Infrastructure
Assert.Equal(IPAddress.Parse("203.0.113.7"), connected.Of("social.example"));
Assert.Null(connected.Of("other.example"));
Assert.Equal("Fastly", CdnNetworks.Of(54113));
Assert.Null(CdnNetworks.Of(64496));
Assert.Null(CdnNetworks.Of(default));
Assert.Equal("Fastly", CdnCatalog.OfAsn(54113)?.Name);
Assert.Null(CdnCatalog.OfAsn(64496));
Assert.Null(CdnCatalog.OfAsn(default));
}
}
}
+34 -8
View File
@@ -1,4 +1,4 @@
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Logging.Abstractions;
using MongoDB.Entities;
@@ -78,16 +78,16 @@ namespace PrivaPub.Tests.Statistics
}
[Fact]
public void The_public_location_depends_on_size_and_on_a_cdn()
public void The_public_location_is_where_the_server_was_located_unless_a_cdn_hides_it()
{
var geo = new InstanceGeo { Country = "DE", City = "Berlin", Latitude = 52.5, Longitude = 13.4, AsnOrg = "Hetzner", Asn = 24940 };
Assert.Equal(new PublicLocation("DE", "Berlin", 52.5, 13.4, "Hetzner", null), PublicGeo.Project(new RemoteInstance { Geo = geo, UsersTotal = 10 }, 10));
Assert.Equal(new PublicLocation("DE", null, null, null, null, null), PublicGeo.Project(new RemoteInstance { Geo = geo, UsersTotal = 1 }, 10));
Assert.Equal(new PublicLocation("DE", null, null, null, null, null), PublicGeo.Project(new RemoteInstance { Geo = geo }, 10));
Assert.Equal(new PublicLocation("DE", "Berlin", 52.5, 13.4, "Hetzner", null), PublicGeo.Project(geo));
Assert.Equal(new PublicLocation("DE", null, null, null, null, null), PublicGeo.Project(new InstanceGeo { Country = "DE" }));
Assert.Equal(new PublicLocation(null, null, null, null, null, "Cloudflare"),
PublicGeo.Project(new RemoteInstance { Geo = new InstanceGeo { Country = "US", City = "SF", Cdn = "Cloudflare" }, UsersTotal = 5000 }, 10));
Assert.Null(PublicGeo.Project(new RemoteInstance(), 10));
PublicGeo.Project(new InstanceGeo { Country = "US", City = "SF", Latitude = 37.8, Longitude = -122.4, Cdn = "Cloudflare" }));
Assert.Null(PublicGeo.Project(new InstanceGeo()));
Assert.Null(PublicGeo.Project(null));
}
}
@@ -112,7 +112,8 @@ namespace PrivaPub.Tests.Statistics
{
readonly string _peer;
public PeerDescriber(IFederationHttp http, IGeoLocator geo, IConnectedAddresses addresses, string peer) : base(http, geo, addresses) => _peer = peer;
public PeerDescriber(IFederationHttp http, IGeoLocator geo, IConnectedAddresses addresses, string peer, ICdnRanges ranges = default,
IEdgeHints edges = default) : base(http, geo, addresses, ranges, edges) => _peer = peer;
protected override string Address(string url) => _peer + new Uri(url).PathAndQuery;
}
@@ -155,6 +156,31 @@ namespace PrivaPub.Tests.Statistics
Assert.Equal(InstanceDescriber.Week(DateTime.UtcNow), snapshot.Week);
}
[Fact]
public async Task A_cdn_is_found_by_its_published_ranges_first_then_by_its_headers()
{
var token = TestContext.Current.CancellationToken;
var listed = $"listed{Guid.NewGuid():N}.example";
var hinted = $"hinted{Guid.NewGuid():N}.example";
ServeServer("/" + listed, "mastodon");
ServeServer("/" + hinted, "mastodon");
var addresses = new ConnectedAddresses();
var ranges = new CdnRanges();
ranges.Replace(new CdnRangeSet { Key = "cloudflare.com", Name = "Cloudflare", Ranges = new() { "127.0.0.0/8" }, FetchedAt = DateTime.UtcNow });
var edges = new EdgeHints();
edges.Remember("127.0.0.1", CdnCatalog.ByKey("fastly.com"));
await new PeerDescriber(Peer.Http(connected: addresses), new FixedGeo(), addresses, _peer.A + "/" + listed, ranges, edges).Describe(listed, crawled: false, default, token);
await new PeerDescriber(Peer.Http(connected: addresses), new FixedGeo(), addresses, _peer.A + "/" + hinted, new CdnRanges(), edges).Describe(hinted, crawled: false, default, token);
var listedOne = await DB.Default.Find<RemoteInstance>().Match(i => i.Host == listed).ExecuteSingleAsync(token);
Assert.Equal(("Cloudflare", "cloudflare.com", "ranges"), (listedOne.Geo.Cdn, listedOne.Geo.CdnDomain, listedOne.Geo.CdnSource));
var hintedOne = await DB.Default.Find<RemoteInstance>().Match(i => i.Host == hinted).ExecuteSingleAsync(token);
Assert.Equal(("Fastly", "fastly.com", "headers"), (hintedOne.Geo.Cdn, hintedOne.Geo.CdnDomain, hintedOne.Geo.CdnSource));
var snapshot = await DB.Default.Find<RemoteInstanceSnapshot>().Match(s => s.Host == listed).ExecuteSingleAsync(token);
Assert.Equal(("Cloudflare", "cloudflare.com", "ranges", "Amsterdam", 52.4), (snapshot.Cdn, snapshot.CdnDomain, snapshot.CdnSource, snapshot.City, snapshot.Latitude!.Value));
}
[Fact]
public async Task Without_api_v2_the_v1_api_is_read_and_an_unreachable_week_is_recorded_too()
{
+2 -1
View File
@@ -1,4 +1,4 @@
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.DataProtection;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Http;
@@ -75,6 +75,7 @@ namespace PrivaPub.Tests.Support.Host
["Federation:FetchLinkPreviews"] = "false",
["Registrations:Mode"] = "Open",
["Statistics:Geo:AutoUpdate"] = "false",
["Statistics:Cdn:AutoUpdate"] = "false",
["Media:Root"] = _mediaRoot,
["Logging:LogLevel:Default"] = "Warning",
["Serilog:MinimumLevel:Default"] = Environment.GetEnvironmentVariable("PRIVAPUB_TEST_LOGS") == "1" ? "Information" : "Fatal"
+8 -1
View File
@@ -49,7 +49,14 @@ namespace PrivaPub.Api.Mastodon.Auth
.SetTokenEndpointUris("/oauth/token")
.SetRevocationEndpointUris("/oauth/revoke")
.SetConfigurationEndpointUris("/.well-known/openid-configuration", "/.well-known/oauth-authorization-server");
options.AllowAuthorizationCodeFlow().AllowClientCredentialsFlow();
options.AllowAuthorizationCodeFlow().AllowClientCredentialsFlow().AllowTokenExchangeFlow();
//the only subject token accepted is the root JWT, for PersonaExchange; OpenIddict's own tokens are not exchanged
options.Configure(server =>
{
server.SubjectTokenTypes.Clear();
server.SubjectTokenTypes.Add(PersonaExchange.SubjectTokenType);
});
options.AddEventHandler(RootJwtSubjectToken.Descriptor);
options.RegisterScopes(MastodonScopes.All);
options.UseReferenceAccessTokens();
options.SetAccessTokenLifetime(null);
@@ -0,0 +1,105 @@
using Microsoft.IdentityModel.Tokens;
using OpenIddict.Abstractions;
using OpenIddict.Server;
using PrivaPub.Services;
using PrivaPub.StaticServices;
using System.Security.Claims;
using static OpenIddict.Abstractions.OpenIddictConstants;
using static OpenIddict.Server.OpenIddictServerEvents;
namespace PrivaPub.Api.Mastodon.Auth
{
// One sign-in for the first-party client (decePubClient): it signs in on /clientapi, then exchanges that root JWT for one
// persona's Mastodon token (RFC 8693, grant_type urn:ietf:params:oauth:grant-type:token-exchange, subject_token_type
// urn:ietf:params:oauth:token-type:jwt, avatar_id). Only the seeded first-party application holds the grant permission,
// and the issued token is the same kind the authorization code flow issues: subject the avatar, never the root.
public static class PersonaExchange
{
public const string SubjectTokenType = "urn:ietf:params:oauth:token-type:jwt";
public const string AvatarParameter = "avatar_id";
//the root the subject token belongs to; it lives on the subject token's principal only, which is never stored or issued
public const string RootClaim = "privapub_root";
public static string FirstPartyClientId(IConfiguration configuration) =>
configuration["ClientApi:FirstPartyClientId"] is { Length: > 0 } clientId ? clientId : "decepub";
//the first-party application: public (a browser keeps no secret), token exchange only, no redirect
public static async Task EnsureFirstPartyClient(IServiceProvider services, CancellationToken token)
{
using var scope = services.CreateScope();
var applications = scope.ServiceProvider.GetRequiredService<IOpenIddictApplicationManager>();
var clientId = FirstPartyClientId(scope.ServiceProvider.GetRequiredService<IConfiguration>());
var descriptor = new OpenIddictApplicationDescriptor
{
ClientId = clientId,
ClientType = ClientTypes.Public,
ConsentType = ConsentTypes.Implicit,
DisplayName = "decePubClient",
Permissions =
{
Permissions.Endpoints.Token,
Permissions.Endpoints.Revocation,
Permissions.GrantTypes.TokenExchange
}
};
foreach (var scopeName in new[] { "read", "write", "follow" })
descriptor.Permissions.Add(Permissions.Prefixes.Scope + scopeName);
var existing = await applications.FindByClientIdAsync(clientId, token);
if (existing == default)
{
await applications.CreateAsync(descriptor, token);
return;
}
await applications.PopulateAsync(existing, descriptor, token);
await applications.UpdateAsync(existing, token);
}
}
// Validates a root JWT given as a token exchange subject token, before OpenIddict tries it as one of its own tokens.
// The principal it builds names the root under PersonaExchange.RootClaim for TokenController, which signs in a fresh
// identity for the avatar.
public sealed class RootJwtSubjectToken : IOpenIddictServerHandler<ValidateTokenContext>
{
public static OpenIddictServerHandlerDescriptor Descriptor { get; } = OpenIddictServerHandlerDescriptor.CreateBuilder<ValidateTokenContext>()
.UseScopedHandler<RootJwtSubjectToken>()
.SetOrder(OpenIddictServerHandlers.Protection.ValidateIdentityModelToken.Descriptor.Order - 500)
.SetType(OpenIddictServerHandlerType.Custom)
.Build();
readonly IConfiguration _configuration;
readonly DbEntities _dbEntities;
public RootJwtSubjectToken(IConfiguration configuration, DbEntities dbEntities)
{
_configuration = configuration;
_dbEntities = dbEntities;
}
public async ValueTask HandleAsync(ValidateTokenContext context)
{
if (context.ValidTokenTypes.Count != 1 || !context.ValidTokenTypes.Contains(PersonaExchange.SubjectTokenType))
return;
var root = await RootJwt.Root(context.Token, _configuration, _dbEntities, context.Transaction.CancellationToken);
if (root == default)
{
context.Reject(Errors.InvalidGrant, "The session token is not valid.");
return;
}
var identity = new ClaimsIdentity(TokenValidationParameters.DefaultAuthenticationType, Claims.Name, Claims.Role);
identity.SetClaim(PersonaExchange.RootClaim, root.ID);
var principal = new ClaimsPrincipal(identity)
.SetTokenType(PersonaExchange.SubjectTokenType)
.SetCreationDate(DateTimeOffset.UtcNow);
if (context.ValidPresenters.Count > 0)
principal.SetPresenters(context.ValidPresenters);
context.Principal = principal;
}
}
}
+25 -1
View File
@@ -1,4 +1,4 @@
using Microsoft.AspNetCore;
using Microsoft.AspNetCore;
using Microsoft.AspNetCore.Authentication;
using Microsoft.AspNetCore.Mvc;
using Microsoft.IdentityModel.Tokens;
@@ -48,9 +48,33 @@ namespace PrivaPub.Api.Mastodon.Auth
return SignIn(new ClaimsPrincipal(identity), OpenIddictServerAspNetCoreDefaults.AuthenticationScheme);
}
if (request.IsTokenExchangeGrantType())
return await Persona(request, token);
return Refuse(Errors.UnsupportedGrantType, "The grant type is not supported.");
}
//PersonaExchange: the root JWT, validated by RootJwtSubjectToken, for a token of one of the root's personas
async Task<IActionResult> Persona(OpenIddictRequest request, CancellationToken token)
{
var subject = (await HttpContext.AuthenticateAsync(OpenIddictServerAspNetCoreDefaults.AuthenticationScheme)).Principal;
var rootId = subject?.GetClaim(PersonaExchange.RootClaim);
var avatarId = (string)request[PersonaExchange.AvatarParameter];
if (string.IsNullOrEmpty(rootId) || string.IsNullOrEmpty(avatarId))
return Refuse(Errors.InvalidGrant, "The persona can no longer be used.");
var owns = await _dbEntities.RootToAvatars.Match(r => r.RootId == rootId && r.AvatarId == avatarId).ExecuteAnyAsync(token);
var avatar = owns ? await _dbEntities.Avatars.MatchID(avatarId).ExecuteFirstAsync(token) : default;
if (avatar == default || !await Usable(avatarId, token))
return Refuse(Errors.InvalidGrant, "The persona can no longer be used.");
var identity = new ClaimsIdentity(TokenValidationParameters.DefaultAuthenticationType, Claims.Name, Claims.Role);
identity.SetClaim(Claims.Subject, avatar.ID);
identity.SetClaim(Claims.Name, avatar.UserName);
identity.SetScopes(MastodonScopes.Parse(request.Scope).Intersect(new[] { "read", "write", "follow" }));
identity.SetDestinations(_ => new[] { Destinations.AccessToken });
return SignIn(new ClaimsPrincipal(identity), OpenIddictServerAspNetCoreDefaults.AuthenticationScheme);
}
async Task<bool> Usable(string avatarId, CancellationToken token)
{
if (string.IsNullOrEmpty(avatarId))
@@ -0,0 +1,82 @@
using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Mvc;
using PrivaPub.Api.Mastodon.Infrastructure;
using PrivaPub.Domain.Statistics;
namespace PrivaPub.Api.Mastodon.Controllers
{
// The CDNs in front of the servers this server knows, each named by its own domain (owner decision 2026-10-04): how many
// servers are behind it, how it is recognised, and week by week who joined and who left.
public class CdnsController : MastodonController
{
const int MaxWeeks = 260;
readonly IServerPlaces _places;
public CdnsController(IServerPlaces places) => _places = places;
[HttpGet("/api/privapub/v1/cdns"), Scope("read", requiresUser: false), AllowAnonymous]
public async Task<IActionResult> List(CancellationToken token) => Json((await _places.Cdns(token)).Select(Summary).ToList());
[HttpGet("/api/privapub/v1/cdns/{key}"), Scope("read", requiresUser: false), AllowAnonymous]
public async Task<IActionResult> Detail(string key, CancellationToken token)
{
var history = await _places.Cdn(key, Math.Clamp(Params.Int("weeks") ?? 52, 1, MaxWeeks), token);
if (history == default)
return NotFoundError();
var detail = new CdnDetail
{
Hosts = history.Hosts,
HostsTruncated = history.Group.Servers > history.Hosts.Count,
Weeks = history.Weeks.Select(w => new CdnWeekEntry { Week = w.Week, Servers = w.Servers, Joined = w.Joined, Left = w.Left }).ToList()
};
Fill(detail, history.Group);
return Json(detail);
}
static CdnSummary Summary(CdnGroup group)
{
var summary = new CdnSummary();
Fill(summary, group);
return summary;
}
static void Fill(CdnSummary summary, CdnGroup group)
{
summary.Key = group.Key;
summary.Name = group.Name;
summary.Domain = group.Domain;
summary.Servers = group.Servers;
summary.Ranges = group.Ranges;
summary.RangesUpdatedAt = group.RangesUpdatedAt is { } at ? MastodonJson.Time(at) : default;
summary.DetectedBy = group.DetectedBy;
}
public class CdnSummary
{
public string Key { get; set; }
public string Name { get; set; }
public string Domain { get; set; }
public int Servers { get; set; }
public int Ranges { get; set; }
public string RangesUpdatedAt { get; set; }
public List<string> DetectedBy { get; set; } = new();
}
public class CdnDetail : CdnSummary
{
public List<string> Hosts { get; set; } = new();
public bool HostsTruncated { get; set; }
public List<CdnWeekEntry> Weeks { get; set; } = new();
}
public class CdnWeekEntry
{
public string Week { get; set; }
public int Servers { get; set; }
public List<string> Joined { get; set; } = new();
public List<string> Left { get; set; } = new();
}
}
}
@@ -1,11 +1,16 @@
using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Options;
using MongoDB.Entities;
using PrivaPub.Api.Mastodon.Infrastructure;
using PrivaPub.Domain.Privacy;
using PrivaPub.Domain.Statistics;
using PrivaPub.Federation.Actors;
using PrivaPub.Federation.Objects;
using PrivaPub.Infrastructure;
using PrivaPub.Infrastructure.Geo;
using PrivaPub.Models.Federation;
using PrivaPub.Models.Jobs;
using PrivaPub.Models.Post;
@@ -19,9 +24,25 @@ namespace PrivaPub.Api.Mastodon.Controllers
{
public class ProvenanceController : MastodonController
{
readonly DbEntities _dbEntities;
const int MaxHosts = 40;
public ProvenanceController(DbEntities dbEntities) => _dbEntities = dbEntities;
readonly DbEntities _dbEntities;
const int MaxWeeks = 260;
readonly ISelfLocation _self;
readonly IServerPlaces _places;
readonly ILocalActorService _localActors;
readonly IOptionsMonitor<RegistrationOptions> _registrations;
public ProvenanceController(DbEntities dbEntities, ISelfLocation self, IServerPlaces places, ILocalActorService localActors,
IOptionsMonitor<RegistrationOptions> registrations)
{
_dbEntities = dbEntities;
_self = self;
_places = places;
_localActors = localActors;
_registrations = registrations;
}
[HttpGet("/api/privapub/v1/statuses/{id}/provenance"), Scope("read:statuses", requiresUser: false), AllowAnonymous]
public async Task<IActionResult> Status(string id, CancellationToken token)
@@ -38,18 +59,104 @@ namespace PrivaPub.Api.Mastodon.Controllers
?? await DB.Default.Find<ObjectRecord>().Match(r => r.ObjectURI == post.ObjectURI).ExecuteFirstAsync(token);
var host = record?.Host ?? (Uri.TryCreate(post.ObjectURI, UriKind.Absolute, out var uri) ? uri.Host.ToLowerInvariant() : default);
var instance = host == default ? default : await DB.Default.Find<RemoteInstance>().Match(i => i.Host == host).ExecuteFirstAsync(token);
return Json(Describe(post, record, instance));
var beforeCdn = instance?.Geo?.Cdn == default ? default : (await _places.BeforeCdn(new[] { instance.Host }, token)).GetValueOrDefault(instance.Host);
return Json(Describe(post, record, instance, beforeCdn));
}
[HttpGet("/api/privapub/v1/instances/{host}"), Scope("read", requiresUser: false), AllowAnonymous]
public async Task<IActionResult> Instance(string host, CancellationToken token)
{
host = host?.Trim().ToLowerInvariant();
var instance = await DB.Default.Find<RemoteInstance>().Match(i => i.Host == host).ExecuteFirstAsync(token);
return instance == default ? NotFoundError() : Json(Describe(instance));
var described = await Describe(new[] { host }, token);
return described.Count == 0 ? NotFoundError() : Json(described[0]);
}
static Provenance Describe(PostEntity post, ObjectRecord record, RemoteInstance instance)
//up to MaxHosts at once (host[]=...), in the order asked; hosts this server does not know are left out
[HttpGet("/api/privapub/v1/instances"), Scope("read", requiresUser: false), AllowAnonymous]
public async Task<IActionResult> Instances(CancellationToken token) => Json(await Describe(Params.List("host"), token));
//the server's weekly snapshots, newest first (weeks=, at most MaxWeeks): its place as shown publicly, its CDN, its size
[HttpGet("/api/privapub/v1/instances/{host}/history"), Scope("read", requiresUser: false), AllowAnonymous]
public async Task<IActionResult> History(string host, CancellationToken token)
{
host = host?.Trim().ToLowerInvariant();
if (string.IsNullOrEmpty(host) || host.Length > 253 || !await DB.Default.Find<RemoteInstance>().Match(i => i.Host == host).ExecuteAnyAsync(token))
return NotFoundError();
var weeks = Math.Clamp(Params.Int("weeks") ?? 52, 1, MaxWeeks);
return Json((await _places.History(host, weeks, token)).Select(s => new InstanceWeek
{
Week = s.Week,
TakenAt = MastodonJson.Time(s.TakenAt),
Reachable = s.Reachable,
Software = s.Software,
Version = s.SoftwareVersion,
UsersTotal = s.UsersTotal,
UsersActiveMonth = s.UsersActiveMonth,
LocalPosts = s.LocalPosts,
OpenRegistrations = s.OpenRegistrations,
Geo = Located(PublicGeo.Project(new InstanceGeo
{
Country = s.Country, City = s.City, Latitude = s.Latitude, Longitude = s.Longitude, Asn = s.Asn,
Cdn = s.Cdn, CdnDomain = s.CdnDomain, CdnSource = s.CdnSource, Ipv6 = s.Ipv6
}), new InstanceGeo { Cdn = s.Cdn, CdnDomain = s.CdnDomain ?? CdnCatalog.ByKey(s.Cdn)?.Key, CdnSource = s.CdnSource })
}).ToList());
}
async Task<List<InstanceDescription>> Describe(IEnumerable<string> hosts, CancellationToken token)
{
var wanted = hosts.Select(h => h?.Trim().ToLowerInvariant()).Where(h => !string.IsNullOrEmpty(h) && h.Length <= 253)
.Distinct().Take(MaxHosts).ToList();
var selfHost = Uri.TryCreate(_localActors.BaseAddress, UriKind.Absolute, out var self) ? self.Host.ToLowerInvariant() : default;
var remote = await DB.Default.Find<RemoteInstance>().Match(i => wanted.Contains(i.Host)).ExecuteAsync(token);
var beforeCdn = await _places.BeforeCdn(remote.Where(i => i.Geo?.Cdn != default).Select(i => i.Host), token);
var described = new List<InstanceDescription>();
foreach (var host in wanted)
{
if (host == selfHost)
described.Add(await DescribeSelf(host, token));
else if (remote.FirstOrDefault(i => i.Host == host) is { } instance)
described.Add(Describe(instance, beforeCdn.GetValueOrDefault(host)));
}
return described;
}
async Task<InstanceDescription> DescribeSelf(string host, CancellationToken token)
{
var geo = await _self.Get(token);
return new InstanceDescription
{
Host = host,
Software = "privapub",
Version = BuildInfo.Ref,
NodeName = "PrivaPub",
Protocols = new() { "activitypub" },
OpenRegistrations = _registrations.CurrentValue.IsOpen,
DescribedAt = MastodonJson.Time(DateTime.UtcNow),
Geo = Located(PublicGeo.Project(geo), geo)
};
}
//the public projection of a server's place (PublicGeo); for a CDN-fronted one, the CDN's domain, how it was found and
//where the server was before it
static InstanceLocation Located(PublicLocation location, InstanceGeo geo, PlaceBeforeCdn beforeCdn = default) => location == default ? default : new InstanceLocation
{
Precision = location.Cdn != default ? "cdn" : location.Latitude != default ? "city" : "country",
Country = location.Country,
City = location.City,
Latitude = location.Latitude,
Longitude = location.Longitude,
Network = location.Network,
Cdn = location.Cdn,
CdnDomain = location.Cdn != default ? ServerPlaces.KeyOf(geo) : default,
CdnSource = location.Cdn != default ? geo?.CdnSource : default,
BeforeCdn = location.Cdn == default || beforeCdn == default ? default : new InstancePlace
{
Week = beforeCdn.Week, Country = beforeCdn.Country, City = beforeCdn.City, Latitude = beforeCdn.Latitude, Longitude = beforeCdn.Longitude
},
Source = geo?.Source,
LocatedAt = geo?.LocatedAt is { } at ? MastodonJson.Time(at) : default
};
Provenance Describe(PostEntity post, ObjectRecord record, RemoteInstance instance, PlaceBeforeCdn beforeCdn)
{
var raw = record?.Raw == default ? default : JsonNode.Parse(record.Raw) as JsonObject;
return new Provenance
@@ -86,11 +193,11 @@ namespace PrivaPub.Api.Mastodon.Controllers
RawHash = r.RawHash,
Raw = r.Raw == default ? default : JsonNode.Parse(r.Raw)
}).ToList() ?? new(),
Instance = instance == default ? default : Describe(instance)
Instance = instance == default ? default : Describe(instance, beforeCdn)
};
}
static InstanceDescription Describe(RemoteInstance instance) => new()
static InstanceDescription Describe(RemoteInstance instance, PlaceBeforeCdn beforeCdn) => new()
{
Host = instance.Host,
Software = instance.Software,
@@ -107,7 +214,8 @@ namespace PrivaPub.Api.Mastodon.Controllers
UnavailableUntil = instance.UnavailableUntil is { } until ? MastodonJson.Time(until) : default,
LastSuccessAt = instance.LastSuccessAt is { } success ? MastodonJson.Time(success) : default,
LastFailureAt = instance.LastFailureAt is { } failure ? MastodonJson.Time(failure) : default
}
},
Geo = Located(PublicGeo.Project(instance.Geo), instance.Geo, beforeCdn)
};
public class Provenance
@@ -170,6 +278,48 @@ namespace PrivaPub.Api.Mastodon.Controllers
public string DescriptionError { get; set; }
public JsonNode NodeInfo { get; set; }
public InstanceDelivery Delivery { get; set; }
public InstanceLocation Geo { get; set; }
}
//precision: "city" (coordinates to 0.1°, city and network), "country" (located only to a country) or "cdn" (the
//CDN's name only)
public class InstanceLocation
{
public string Precision { get; set; }
public string Country { get; set; }
public string City { get; set; }
public double? Latitude { get; set; }
public double? Longitude { get; set; }
public string Network { get; set; }
public string Cdn { get; set; }
public string CdnDomain { get; set; }
public string CdnSource { get; set; }//"ranges", "headers" or "asn"
public InstancePlace BeforeCdn { get; set; }//where a CDN-fronted server was located before the CDN, by its snapshots
public string Source { get; set; }
public string LocatedAt { get; set; }
}
public class InstancePlace
{
public string Week { get; set; }
public string Country { get; set; }
public string City { get; set; }
public double? Latitude { get; set; }
public double? Longitude { get; set; }
}
public class InstanceWeek
{
public string Week { get; set; }
public string TakenAt { get; set; }
public bool Reachable { get; set; }
public string Software { get; set; }
public string Version { get; set; }
public long? UsersTotal { get; set; }
public long? UsersActiveMonth { get; set; }
public long? LocalPosts { get; set; }
public bool? OpenRegistrations { get; set; }
public InstanceLocation Geo { get; set; }
}
public class InstanceDelivery
+3 -1
View File
@@ -1,4 +1,4 @@
namespace PrivaPub.Api.Mastodon.Entities
namespace PrivaPub.Api.Mastodon.Entities
{
public class Account
{
@@ -73,6 +73,7 @@ namespace PrivaPub.Api.Mastodon.Entities
public PrivaPubVideo Video { get; set; }
public PrivaPubAudio Audio { get; set; }
public PrivaPubEvent Event { get; set; }
public PrivaPubPlace Place { get; set; }//a remote post's own place (Pixelfed), shown only
public PrivaPubVotes Votes { get; set; } = new();
}
@@ -151,6 +152,7 @@ namespace PrivaPub.Api.Mastodon.Entities
{
public string Name { get; set; }
public string Address { get; set; }
public string Country { get; set; }//a post's own place only
public double? Latitude { get; set; }
public double? Longitude { get; set; }
public string Url { get; set; }
@@ -1,4 +1,4 @@
using MongoDB.Entities;
using MongoDB.Entities;
using PrivaPub.Api.Mastodon.Entities;
using PrivaPub.Api.Mastodon.Infrastructure;
@@ -457,6 +457,10 @@ namespace PrivaPub.Api.Mastodon.Mappers
Status = post.Event.Status,
Category = post.Event.Category
},
Place = post.Place == default ? default : new PrivaPubPlace
{
Name = post.Place.Name, Latitude = post.Place.Latitude, Longitude = post.Place.Longitude, Country = post.Place.Country
},
Votes = new PrivaPubVotes { Up = post.FavouritesCount, Down = post.DownvotesCount }
};
+5 -6
View File
@@ -6,18 +6,17 @@ namespace PrivaPub.Domain.Statistics
public static class PublicGeo
{
//owner decision: a CDN-fronted server shows only its CDN (the address is an edge), a server reporting at least
//minUsers users its city, coordinates and network, any other only its country
public static PublicLocation Project(RemoteInstance instance, int minUsers)
//a server's place as it is shown publicly: where it was located (country, city, coordinates to 0.1°, network),
//except that a CDN-fronted server shows only its CDN, because the address we reached is the CDN's edge
public static PublicLocation Project(InstanceGeo geo)
{
var geo = instance?.Geo;
if (geo == default)
return default;
if (geo.Cdn != default)
return new PublicLocation(default, default, default, default, default, geo.Cdn);
if (instance.UsersTotal >= minUsers)
if (geo.Country == default && geo.Latitude == default)
return default;
return new PublicLocation(geo.Country, geo.City, geo.Latitude, geo.Longitude, geo.AsnOrg, default);
return geo.Country == default ? default : new PublicLocation(geo.Country, default, default, default, default, default);
}
}
}
+109
View File
@@ -0,0 +1,109 @@
using Microsoft.Extensions.Options;
using PrivaPub.Infrastructure.Geo;
using PrivaPub.Infrastructure.Statistics;
using PrivaPub.Models;
using PrivaPub.Models.Jobs;
using System.Net;
using System.Net.Sockets;
namespace PrivaPub.Domain.Statistics
{
public interface ISelfLocation
{
//where this server is, or default when it cannot tell; Source is "configured" when the owner set it (Statistics:Geo:Self)
Task<InstanceGeo> Get(CancellationToken token);
}
// This server's own place, for the instance API (the client's globe draws every server from it). The owner may set it,
// for a server behind a proxy or a CDN; otherwise the server's own host name is resolved and located like any other
// server's, once a day. Never the address of a request.
public class SelfLocation : ISelfLocation
{
static readonly TimeSpan Lifetime = TimeSpan.FromDays(1);
readonly IGeoLocator _geo;
readonly ICdnRanges _cdnRanges;
readonly IOptionsMonitor<StatisticsOptions> _options;
readonly IOptionsMonitor<AppConfiguration> _app;
readonly ILogger<SelfLocation> _logger;
readonly SemaphoreSlim _gate = new(1, 1);
InstanceGeo _located;
DateTime _locatedAt = DateTime.MinValue;
public SelfLocation(IGeoLocator geo, IOptionsMonitor<StatisticsOptions> options, IOptionsMonitor<AppConfiguration> app, ILogger<SelfLocation> logger,
ICdnRanges cdnRanges = default)
{
_geo = geo;
_cdnRanges = cdnRanges;
_options = options;
_app = app;
_logger = logger;
}
public async Task<InstanceGeo> Get(CancellationToken token)
{
if (_options.CurrentValue.Geo.Self is { Latitude: { } latitude, Longitude: { } longitude } self)
return new InstanceGeo
{
Country = self.Country,
City = self.City,
Latitude = Math.Round(latitude, 1),
Longitude = Math.Round(longitude, 1),
Source = "configured"
};
if (DateTime.UtcNow - _locatedAt < Lifetime)
return _located;
await _gate.WaitAsync(token);
try
{
if (DateTime.UtcNow - _locatedAt >= Lifetime)
{
_located = await Locate(token);
_locatedAt = DateTime.UtcNow;
}
return _located;
}
finally
{
_gate.Release();
}
}
async Task<InstanceGeo> Locate(CancellationToken token)
{
try
{
if (!Uri.TryCreate(_app.CurrentValue.BackendBaseAddress, UriKind.Absolute, out var self))
return default;
var addresses = await Dns.GetHostAddressesAsync(self.Host, token);
var address = addresses.FirstOrDefault(a => a.AddressFamily == AddressFamily.InterNetwork) ?? addresses.FirstOrDefault();
var fix = address == default ? default : _geo.Locate(address);
var (cdn, cdnSource) = Federation.Objects.InstanceDescriber.Cdn(address, self.Host, fix?.Asn, _cdnRanges, default);
return fix == default ? default : new InstanceGeo
{
Country = fix.Country,
City = fix.City,
Latitude = fix.Latitude,
Longitude = fix.Longitude,
Asn = fix.Asn,
AsnOrg = fix.AsnOrg,
Cdn = cdn?.Name,
CdnDomain = cdn?.Domain,
CdnSource = cdnSource,
Ipv6 = address.AddressFamily == AddressFamily.InterNetworkV6,
Source = _geo.Source,
LocatedAt = DateTime.UtcNow
};
}
catch (Exception ex) when (ex is SocketException or ArgumentException)
{
_logger.LogInformation("{Service}: this server's host does not resolve", nameof(SelfLocation));
return default;
}
}
}
}
+163
View File
@@ -0,0 +1,163 @@
using Microsoft.Extensions.Caching.Memory;
using MongoDB.Entities;
using PrivaPub.Federation.Objects;
using PrivaPub.Infrastructure.Geo;
using PrivaPub.Models.Jobs;
namespace PrivaPub.Domain.Statistics
{
// Where a server was before it went behind a CDN, as its weekly snapshots last saw it.
public sealed record PlaceBeforeCdn(string Week, string Country, string City, double? Latitude, double? Longitude);
// A CDN as a group of servers: those behind it now, and week by week how many were behind it, who joined and who left.
public sealed record CdnGroup(string Key, string Name, string Domain, int Servers, int Ranges, DateTime? RangesUpdatedAt, List<string> DetectedBy);
public sealed record CdnWeek(string Week, int Servers, List<string> Joined, List<string> Left);
public sealed record CdnHistory(CdnGroup Group, List<string> Hosts, List<CdnWeek> Weeks);
public interface IServerPlaces
{
Task<Dictionary<string, PlaceBeforeCdn>> BeforeCdn(IEnumerable<string> hosts, CancellationToken token);
Task<List<RemoteInstanceSnapshot>> History(string host, int weeks, CancellationToken token);
Task<List<CdnGroup>> Cdns(CancellationToken token);
Task<CdnHistory> Cdn(string key, int weeks, CancellationToken token);
}
// Servers through time (owner decision 2026-10-04): each server's weekly snapshots (place, CDN, size), and the servers
// grouped by the CDN in front of them, named by the CDN's own domain. Read from the weekly snapshots InstanceDescriber
// writes; the CDN pages are kept for half an hour, because a large CDN reads many snapshots.
public class ServerPlaces : IServerPlaces
{
public const int MaxHosts = 500;
const int MaxChanges = 50;
static readonly TimeSpan CacheFor = TimeSpan.FromMinutes(30);
readonly ICdnRanges _ranges;
readonly IMemoryCache _cache;
public ServerPlaces(ICdnRanges ranges, IMemoryCache cache)
{
_ranges = ranges;
_cache = cache;
}
// the CDN key a server's place belongs to: the CDN's domain, or its name for a CDN that has none
public static string KeyOf(InstanceGeo geo) =>
geo?.Cdn == default ? default : geo.CdnDomain ?? CdnCatalog.ByKey(geo.Cdn)?.Key ?? geo.Cdn.ToLowerInvariant();
static string KeyOf(RemoteInstanceSnapshot snapshot) =>
snapshot.Cdn == default ? default : snapshot.CdnDomain ?? CdnCatalog.ByKey(snapshot.Cdn)?.Key ?? snapshot.Cdn.ToLowerInvariant();
public async Task<Dictionary<string, PlaceBeforeCdn>> BeforeCdn(IEnumerable<string> hosts, CancellationToken token)
{
var wanted = hosts.Where(h => h != default).Distinct().ToList();
if (wanted.Count == 0)
return new();
var located = await DB.Default.Find<RemoteInstanceSnapshot>()
.Match(s => wanted.Contains(s.Host) && s.Cdn == null && (s.Country != null || s.Latitude != null))
.Sort(s => s.Week, Order.Descending)
.ExecuteAsync(token);
return located.GroupBy(s => s.Host)
.Select(g => g.First())
.ToDictionary(s => s.Host, s => new PlaceBeforeCdn(s.Week, s.Country, s.City, s.Latitude, s.Longitude));
}
public async Task<List<RemoteInstanceSnapshot>> History(string host, int weeks, CancellationToken token) =>
await DB.Default.Find<RemoteInstanceSnapshot>()
.Match(s => s.Host == host)
.Sort(s => s.Week, Order.Descending)
.Limit(weeks)
.ExecuteAsync(token);
public async Task<List<CdnGroup>> Cdns(CancellationToken token)
{
var fronted = await Fronted(token);
var status = _ranges.Status;
var keys = CdnCatalog.All.Select(p => p.Key).Concat(fronted.Select(i => KeyOf(i.Geo))).Distinct(StringComparer.OrdinalIgnoreCase);
return keys.Select(key => Group(key, fronted, status)).OrderByDescending(g => g.Servers).ThenBy(g => g.Name).ToList();
}
public async Task<CdnHistory> Cdn(string key, int weeks, CancellationToken token)
{
key = key?.Trim().ToLowerInvariant();
if (string.IsNullOrEmpty(key) || key.Length > 100)
return default;
return await _cache.GetOrCreateAsync($"cdn-history|{key}|{weeks}", async entry =>
{
entry.AbsoluteExpirationRelativeToNow = CacheFor;
var fronted = await Fronted(token);
var provider = CdnCatalog.ByKey(key);
if (provider == default && fronted.All(i => KeyOf(i.Geo) != key))
return default;
var group = Group(provider?.Key ?? key, fronted, _ranges.Status);
var hosts = fronted.Where(i => KeyOf(i.Geo) == group.Key).Select(i => i.Host).OrderBy(h => h).Take(MaxHosts).ToList();
return new CdnHistory(group, hosts, await Weeks(group, weeks, token));
});
}
async Task<List<CdnWeek>> Weeks(CdnGroup group, int weeks, CancellationToken token)
{
var since = InstanceDescriber.Week(DateTime.UtcNow.AddDays(-7 * weeks));
var everBehind = (await DB.Default.Find<RemoteInstanceSnapshot>()
.Match(f => f.Gte(s => s.Week, since) & (f.Eq(s => s.CdnDomain, group.Key) | f.Eq(s => s.Cdn, group.Name)))
.Project(s => new RemoteInstanceSnapshot { Host = s.Host })
.ExecuteAsync(token))
.Select(s => s.Host).Distinct().ToList();
if (everBehind.Count == 0)
return new();
var snapshots = await DB.Default.Find<RemoteInstanceSnapshot>()
.Match(f => f.In(s => s.Host, everBehind) & f.Gte(s => s.Week, since))
.Project(s => new RemoteInstanceSnapshot { Host = s.Host, Week = s.Week, Cdn = s.Cdn, CdnDomain = s.CdnDomain })
.ExecuteAsync(token);
// a server is behind the CDN from the week a snapshot shows it there until one shows it elsewhere
var behind = new Dictionary<string, bool>();
var history = new List<CdnWeek>();
foreach (var week in snapshots.GroupBy(s => s.Week).OrderBy(w => w.Key, StringComparer.Ordinal))
{
var joined = new List<string>();
var left = new List<string>();
foreach (var snapshot in week)
{
var now = KeyOf(snapshot) == group.Key;
var before = behind.GetValueOrDefault(snapshot.Host);
if (now && !before)
joined.Add(snapshot.Host);
else if (!now && before)
left.Add(snapshot.Host);
behind[snapshot.Host] = now;
}
history.Add(new CdnWeek(week.Key, behind.Count(b => b.Value), joined.OrderBy(h => h).Take(MaxChanges).ToList(),
left.OrderBy(h => h).Take(MaxChanges).ToList()));
}
history.Reverse();
return history;
}
static async Task<List<RemoteInstance>> Fronted(CancellationToken token) =>
await DB.Default.Find<RemoteInstance>()
.Match(i => i.Geo.Cdn != null)
.Project(i => new RemoteInstance { Host = i.Host, Geo = i.Geo })
.ExecuteAsync(token);
static CdnGroup Group(string key, List<RemoteInstance> fronted, IReadOnlyDictionary<string, (int Ranges, DateTime FetchedAt)> status)
{
var provider = CdnCatalog.ByKey(key);
var servers = fronted.Where(i => KeyOf(i.Geo) == key).ToList();
var detectedBy = new List<string>();
if (provider?.Lists.Length > 0)
detectedBy.Add("ranges");
if (provider != default)
detectedBy.Add("headers");
if (provider?.Asns.Length > 0)
detectedBy.Add("asn");
var ranges = status.TryGetValue(key, out var known) ? known : default;
return new CdnGroup(key, provider?.Name ?? servers.Select(i => i.Geo.Cdn).FirstOrDefault() ?? key, provider?.Domain, servers.Count, ranges.Ranges,
ranges.Ranges > 0 ? ranges.FetchedAt : null, detectedBy);
}
}
}
+1 -13
View File
@@ -1,10 +1,7 @@
using Microsoft.AspNetCore.Authentication;
using Microsoft.IdentityModel.Tokens;
using PrivaPub.Services;
using System.Text;
namespace PrivaPub.Extensions
{
public static class AddAuthExtension
@@ -15,16 +12,7 @@ namespace PrivaPub.Extensions
#if DEBUG
options.RequireHttpsMetadata = false;
#endif
options.TokenValidationParameters = new()
{
ValidateIssuer = true,
ValidateAudience = true,
ValidateLifetime = true,
ValidateIssuerSigningKey = true,
ValidIssuer = configuration["AppConfiguration:Jwt:Issuer"],
ValidAudience = configuration["AppConfiguration:Jwt:Audience"],
IssuerSigningKey = new SymmetricSecurityKey(Encoding.UTF8.GetBytes(configuration["AppConfiguration:Jwt:Key"]))
};
options.TokenValidationParameters = RootJwt.Parameters(configuration);
options.Events = new JwtEvents();
});
return builder;
+2 -1
View File
@@ -1,4 +1,4 @@
using MongoDB.Entities;
using MongoDB.Entities;
using PrivaPub.Domain.Statuses;
using PrivaPub.Federation.Actors;
@@ -33,6 +33,7 @@ namespace PrivaPub.Federation.Inbox
post.Video = note.Video ?? post.Video;
post.Audio = note.Audio ?? post.Audio;
post.Event = note.Event ?? post.Event;
post.Place = note.Place;
if (!IsEdit(note, post))
{
await DB.Default.SaveAsync(post, token);
+2 -1
View File
@@ -1,4 +1,4 @@
using MongoDB.Driver;
using MongoDB.Driver;
using MongoDB.Entities;
using PrivaPub.Infrastructure.Http;
@@ -90,6 +90,7 @@ namespace PrivaPub.Federation.Inbox
Video = note.Video,
Audio = note.Audio,
Event = note.Event,
Place = note.Place,
HasContentWarning = note.Sensitive,
Text = note.ContentHtml,
ContentHtml = note.ContentHtml,
@@ -1,4 +1,4 @@
using MongoDB.Entities;
using MongoDB.Entities;
using PrivaPub.Infrastructure.Geo;
using PrivaPub.Infrastructure.Http;
@@ -18,12 +18,17 @@ namespace PrivaPub.Federation.Objects
readonly IFederationHttp _http;
readonly IGeoLocator _geo;
readonly IConnectedAddresses _addresses;
readonly ICdnRanges _cdnRanges;
readonly IEdgeHints _edges;
public InstanceDescriber(IFederationHttp http, IGeoLocator geo = default, IConnectedAddresses addresses = default)
public InstanceDescriber(IFederationHttp http, IGeoLocator geo = default, IConnectedAddresses addresses = default, ICdnRanges cdnRanges = default,
IEdgeHints edges = default)
{
_http = http;
_geo = geo;
_addresses = addresses;
_cdnRanges = cdnRanges;
_edges = edges;
}
public JobKind Kind => JobKind.DescribeInstance;
@@ -145,35 +150,57 @@ namespace PrivaPub.Federation.Objects
.Modify(s => s.RegistrationMode, instance?.RegistrationMode)
.Modify(s => s.Languages, instance?.Languages)
.Modify(s => s.Country, instance?.Geo?.Country)
.Modify(s => s.City, instance?.Geo?.City)
.Modify(s => s.Latitude, instance?.Geo?.Latitude)
.Modify(s => s.Longitude, instance?.Geo?.Longitude)
.Modify(s => s.Asn, instance?.Geo?.Asn)
.Modify(s => s.Cdn, instance?.Geo?.Cdn)
.Modify(s => s.CdnDomain, instance?.Geo?.CdnDomain)
.Modify(s => s.CdnSource, instance?.Geo?.CdnSource)
.Modify(s => s.Ipv6, instance?.Geo?.Ipv6)
.Option(o => o.IsUpsert = true)
.ExecuteAsync(token);
return facts != default;
}
// Where the address we reached sits, and the CDN in front of it, found by its published ranges, then by the
// fingerprint in the responses we just got, then by a network that carries only a CDN.
InstanceGeo Locate(string host)
{
var reached = _addresses?.Of(new Uri(Address($"https://{host}/")).Host);
var reachedHost = new Uri(Address($"https://{host}/")).Host;
var reached = _addresses?.Of(reachedHost);
var fix = reached == default ? default : _geo?.Locate(reached);
if (fix == default)
var (cdn, cdnSource) = Cdn(reached, reachedHost, fix?.Asn, _cdnRanges, _edges);
if (fix == default && cdn == default)
return default;
return new InstanceGeo
{
Country = fix.Country,
City = fix.City,
Latitude = fix.Latitude,
Longitude = fix.Longitude,
Asn = fix.Asn,
AsnOrg = fix.AsnOrg,
Cdn = CdnNetworks.Of(fix.Asn),
Ipv6 = reached.AddressFamily == AddressFamily.InterNetworkV6,
Source = _geo.Source,
Country = fix?.Country,
City = fix?.City,
Latitude = fix?.Latitude,
Longitude = fix?.Longitude,
Asn = fix?.Asn,
AsnOrg = fix?.AsnOrg,
Cdn = cdn?.Name,
CdnDomain = cdn?.Domain,
CdnSource = cdnSource,
Ipv6 = reached?.AddressFamily == AddressFamily.InterNetworkV6,
Source = _geo?.Source,
LocatedAt = DateTime.UtcNow
};
}
public static (CdnProvider Cdn, string Source) Cdn(System.Net.IPAddress address, string host, int? asn, ICdnRanges ranges, IEdgeHints edges)
{
if (ranges?.Of(address) is { } listed)
return (listed, "ranges");
if (edges?.Of(host) is { } hinted)
return (hinted, "headers");
if (CdnCatalog.OfAsn(asn) is { } network)
return (network, "asn");
return default;
}
//where an https address is fetched from: itself, except in tests, which point it at a plain-http peer
protected virtual string Address(string url) => url;
}
+4 -2
View File
@@ -1,4 +1,4 @@
using PrivaPub.Models.Post;
using PrivaPub.Models.Post;
using System.Globalization;
using System.Text;
@@ -44,6 +44,7 @@ namespace PrivaPub.Federation.Objects
public VideoDetails Video { get; init; }
public AudioDetails Audio { get; init; }
public EventDetails Event { get; init; }
public PostPlace Place { get; init; }
}
public static class NoteParser
@@ -110,7 +111,8 @@ namespace PrivaPub.Federation.Objects
Link = ObjectShapes.Link(note),
Video = ObjectShapes.Video(note),
Audio = ObjectShapes.Audio(note),
Event = ObjectShapes.Event(note)
Event = ObjectShapes.Event(note),
Place = ObjectShapes.NotePlace(note)
};
}
+26 -1
View File
@@ -1,4 +1,4 @@
using PrivaPub.Models.Post;
using PrivaPub.Models.Post;
using System.Globalization;
using System.Net;
@@ -293,6 +293,31 @@ namespace PrivaPub.Federation.Objects
};
}
//a post's own place, as Pixelfed sends it: a Place with coordinates (numbers, or numbers as strings); an Event's
//places are its own (Event)
public static PostPlace NotePlace(JsonObject note)
{
if (Value(note, "type") == "Event")
return default;
var place = Objects(note["location"]).FirstOrDefault(l => Value(l, "type") == "Place");
var latitude = Coordinate(place, "latitude");
var longitude = Coordinate(place, "longitude");
if (latitude is not (>= -90 and <= 90) || longitude is not (>= -180 and <= 180))
return default;
return new PostPlace
{
Name = Text(Value(place, "name"), 200),
Latitude = Math.Round(latitude.Value, 4),
Longitude = Math.Round(longitude.Value, 4),
Country = Text(Value(place, "country"), 100)
};
}
static double? Coordinate(JsonObject node, string property) =>
Double(node, property) ?? (double.TryParse(Value(node, property), NumberStyles.Float, CultureInfo.InvariantCulture, out var number) && double.IsFinite(number)
? number
: (double?)null);
static EventPlace Place(JsonObject location)
{
var type = Value(location, "type");
+4 -1
View File
@@ -1,4 +1,4 @@
using MongoDB.Bson;
using MongoDB.Bson;
using MongoDB.Driver;
using MongoDB.Entities;
@@ -131,6 +131,9 @@ namespace PrivaPub.Infrastructure.Data
await DB.Default.Index<RemoteInstanceSnapshot>().Key(s => s.Host, KeyType.Ascending).Key(s => s.Week, KeyType.Ascending)
.Option(o => o.Unique = true).CreateAsync(token);
await Plain<RemoteInstanceSnapshot>(token, s => s.Week);
await Plain<RemoteInstanceSnapshot>(token, s => s.CdnDomain, s => s.Week);
await Plain<RemoteInstanceSnapshot>(token, s => s.Cdn, s => s.Week);
await Plain<CdnRangeSet>(token, s => s.Key);
await Plain<RemoteInstance>(token, i => i.Seen);
await Plain<RemoteInstance>(token, i => i.LastCrawledAt);
}
+156
View File
@@ -0,0 +1,156 @@
using System.Net;
using System.Text.Json;
namespace PrivaPub.Infrastructure.Geo
{
// How a published list of a CDN's addresses is written.
public enum CdnListFormat
{
Lines,//one CIDR per line (Cloudflare)
StringArray,//a JSON array of addresses or CIDRs (Bunny)
AddressesJson,//{"addresses": [...], "ipv6_addresses"/"addresses_v6": [...]} (Fastly, Gcore)
AwsCloudFront,//AWS ip-ranges.json, CLOUDFRONT entries only
ImpervaJson//{"ipRanges": [...], "ipv6Ranges": [...]}
}
public sealed record CdnList(string Url, CdnListFormat Format, string Method = "GET", string Body = default);
// A CDN, or any edge network whose addresses say nothing about where a server runs. Domain names it in statistics.
public sealed record CdnProvider(string Name, string Domain, int[] Asns, CdnList[] Lists, Func<HttpResponseMessage, bool> Fingerprint)
{
public string Key => Domain ?? Name.ToLowerInvariant();
}
// owner decision (2026-10-04): PrivaPub finds CDNs by itself, three ways, best first. The address ranges a CDN
// publishes (downloaded daily by CdnUpdater), the CDN's fingerprint in the responses PrivaPub already gets from a server
// (EdgeHintsHandler), and the networks (ASN) that carry nothing but a CDN. A network shared with plain hosting (AWS,
// Google, Microsoft, DataPacket) is never listed by ASN: only ranges or fingerprints name those CDNs.
public static class CdnCatalog
{
public static readonly IReadOnlyList<CdnProvider> All = new CdnProvider[]
{
new("Cloudflare", "cloudflare.com", new[] { 13335, 209242 },
new[] { new CdnList("https://www.cloudflare.com/ips-v4", CdnListFormat.Lines), new CdnList("https://www.cloudflare.com/ips-v6", CdnListFormat.Lines) },
r => Has(r, "cf-ray") || ServerIs(r, "cloudflare")),
new("Fastly", "fastly.com", new[] { 54113 },
new[] { new CdnList("https://api.fastly.com/public-ip-list", CdnListFormat.AddressesJson) },
r => Has(r, "x-fastly-request-id") || Has(r, "fastly-restarts") || Value(r, "x-served-by")?.StartsWith("cache-", StringComparison.OrdinalIgnoreCase) == true),
new("Amazon CloudFront", "cloudfront.net", Array.Empty<int>(),
new[] { new CdnList("https://ip-ranges.amazonaws.com/ip-ranges.json", CdnListFormat.AwsCloudFront) },
r => Has(r, "x-amz-cf-id") || Has(r, "x-amz-cf-pop") || Value(r, "via")?.Contains("cloudfront", StringComparison.OrdinalIgnoreCase) == true),
new("Akamai", "akamai.com", new[] { 20940, 16625, 16702, 21342 }, Array.Empty<CdnList>(),
r => Has(r, "akamai-grn") || Has(r, "x-akamai-transformed") || Has(r, "akamai-cache-status") || ServerStarts(r, "AkamaiGHost") || ServerStarts(r, "AkamaiNetStorage")),
new("Bunny", "bunny.net", new[] { 200325 },
new[] { new CdnList("https://api.bunny.net/system/edgeserverlist", CdnListFormat.StringArray), new CdnList("https://api.bunny.net/system/edgeserverlist/IPv6", CdnListFormat.StringArray) },
r => ServerStarts(r, "BunnyCDN") || Has(r, "cdn-pullzone") || Has(r, "cdn-requestid")),
new("Gcore", "gcore.com", Array.Empty<int>(),
new[] { new CdnList("https://api.gcore.com/cdn/public-ip-list", CdnListFormat.AddressesJson) },
r => ServerStarts(r, "gcore")),
new("Imperva", "imperva.com", new[] { 19551 },
new[] { new CdnList("https://my.imperva.com/api/integration/v1/ips", CdnListFormat.ImpervaJson, "POST", "resp_format=json") },
r => Has(r, "x-iinfo") || Value(r, "x-cdn") is "Imperva" or "Incapsula"),
new("CDN77", "cdn77.com", Array.Empty<int>(), Array.Empty<CdnList>(),
r => Has(r, "x-77-nzt") || Has(r, "x-77-cache") || ServerStarts(r, "CDN77")),
new("Edgio", "edg.io", new[] { 15133 }, Array.Empty<CdnList>(),
r => ServerStarts(r, "ECAcc") || ServerStarts(r, "ECS (") || Has(r, "x-ec-custom-error")),
new("Azure Front Door", "azure.microsoft.com", Array.Empty<int>(), Array.Empty<CdnList>(),
r => Has(r, "x-azure-ref")),
new("Google Cloud", "cloud.google.com", Array.Empty<int>(), Array.Empty<CdnList>(),
r => Value(r, "via")?.Contains("google", StringComparison.OrdinalIgnoreCase) == true),
new("Vercel", "vercel.com", Array.Empty<int>(), Array.Empty<CdnList>(),
r => Has(r, "x-vercel-id") || ServerIs(r, "Vercel")),
new("Netlify", "netlify.com", Array.Empty<int>(), Array.Empty<CdnList>(),
r => Has(r, "x-nf-request-id") || ServerIs(r, "Netlify")),
new("Sucuri", "sucuri.net", Array.Empty<int>(), Array.Empty<CdnList>(),
r => Has(r, "x-sucuri-id") || ServerStarts(r, "Sucuri")),
new("DDoS-Guard", "ddos-guard.net", Array.Empty<int>(), Array.Empty<CdnList>(),
r => ServerIs(r, "ddos-guard"))
};
public static CdnProvider ByKey(string key) =>
key == default ? default : All.FirstOrDefault(p => string.Equals(p.Key, key, StringComparison.OrdinalIgnoreCase) || string.Equals(p.Name, key, StringComparison.OrdinalIgnoreCase));
public static CdnProvider OfAsn(int? asn) => asn is { } number ? All.FirstOrDefault(p => p.Asns.Contains(number)) : default;
// The CDN a response passed through: a known fingerprint, or a CDN that names itself in X-CDN.
public static CdnProvider FromResponse(HttpResponseMessage response)
{
if (response == default)
return default;
var known = All.FirstOrDefault(p => p.Fingerprint(response));
if (known != default)
return known;
var named = Value(response, "x-cdn")?.Trim();
return named is { Length: > 1 and <= 40 } && named.All(c => char.IsLetterOrDigit(c) || c is ' ' or '-' or '.' or '_')
? new CdnProvider(named, default, Array.Empty<int>(), Array.Empty<CdnList>(), _ => false)
: default;
}
// The networks a published list holds; an entry that is not an address or a network is an error, never skipped.
public static List<IPNetwork> Parse(CdnListFormat format, string body)
{
IEnumerable<string> entries = format switch
{
CdnListFormat.Lines => body.Split('\n', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries).Where(l => !l.StartsWith('#')),
CdnListFormat.StringArray => Root(body),
CdnListFormat.AddressesJson => Json(body, "addresses", "ipv6_addresses", "addresses_v6"),
CdnListFormat.ImpervaJson => Json(body, "ipRanges", "ipv6Ranges"),
CdnListFormat.AwsCloudFront => CloudFront(body),
_ => Enumerable.Empty<string>()
};
return entries.Select(Network).ToList();
}
static IPNetwork Network(string entry)
{
if (IPNetwork.TryParse(entry, out var network))
return network;
if (IPAddress.TryParse(entry, out var address))
return new IPNetwork(address, address.AddressFamily == System.Net.Sockets.AddressFamily.InterNetwork ? 32 : 128);
throw new FormatException($"not an address or a network: {entry}");
}
static List<string> Root(string body)
{
using var document = JsonDocument.Parse(body);
return Strings(document.RootElement);
}
static IEnumerable<string> Json(string body, params string[] properties)
{
using var document = JsonDocument.Parse(body);
var entries = new List<string>();
foreach (var property in properties)
if (document.RootElement.TryGetProperty(property, out var array))
entries.AddRange(Strings(array));
return entries;
}
static IEnumerable<string> CloudFront(string body)
{
using var document = JsonDocument.Parse(body);
var entries = new List<string>();
foreach (var (array, field) in new[] { ("prefixes", "ip_prefix"), ("ipv6_prefixes", "ipv6_prefix") })
if (document.RootElement.TryGetProperty(array, out var prefixes))
foreach (var prefix in prefixes.EnumerateArray())
if (prefix.TryGetProperty("service", out var service) && service.GetString() == "CLOUDFRONT" && prefix.TryGetProperty(field, out var value))
entries.Add(value.GetString());
return entries;
}
static List<string> Strings(JsonElement array) =>
array.ValueKind == JsonValueKind.Array ? array.EnumerateArray().Select(e => e.GetString()).ToList() : throw new FormatException("not a list");
static bool Has(HttpResponseMessage response, string header) =>
response.Headers.Contains(header) || response.Content?.Headers.Contains(header) == true;
static string Value(HttpResponseMessage response, string header) =>
response.Headers.TryGetValues(header, out var values) || response.Content?.Headers.TryGetValues(header, out values) == true
? string.Join(", ", values)
: default;
static bool ServerIs(HttpResponseMessage response, string name) => string.Equals(Value(response, "server")?.Trim(), name, StringComparison.OrdinalIgnoreCase);
static bool ServerStarts(HttpResponseMessage response, string prefix) => Value(response, "server")?.TrimStart().StartsWith(prefix, StringComparison.OrdinalIgnoreCase) == true;
}
}
+65
View File
@@ -0,0 +1,65 @@
using MongoDB.Entities;
using PrivaPub.Models.Jobs;
using System.Net;
namespace PrivaPub.Infrastructure.Geo
{
public interface ICdnRanges
{
// the CDN whose published ranges hold the address, or default
CdnProvider Of(IPAddress address);
void Replace(CdnRangeSet set);
Task Load(CancellationToken token);
// per CDN key: how many ranges are known and when they were downloaded
IReadOnlyDictionary<string, (int Ranges, DateTime FetchedAt)> Status { get; }
}
// The published CDN ranges in memory, rebuilt from the stored CdnRangeSets at start and replaced set by set as
// CdnUpdater downloads them. A few thousand networks: a linear scan per describe is cheap, and describes are weekly.
public class CdnRanges : ICdnRanges
{
readonly object _gate = new();
Dictionary<string, (CdnProvider Provider, IPNetwork[] Networks, DateTime FetchedAt)> _sets = new(StringComparer.OrdinalIgnoreCase);
public IReadOnlyDictionary<string, (int Ranges, DateTime FetchedAt)> Status
{
get
{
lock (_gate)
return _sets.ToDictionary(s => s.Key, s => (s.Value.Networks.Length, s.Value.FetchedAt), StringComparer.OrdinalIgnoreCase);
}
}
public CdnProvider Of(IPAddress address)
{
if (address == default)
return default;
if (address.IsIPv4MappedToIPv6)
address = address.MapToIPv4();
Dictionary<string, (CdnProvider Provider, IPNetwork[] Networks, DateTime FetchedAt)> sets;
lock (_gate)
sets = _sets;
foreach (var (provider, networks, _) in sets.Values)
foreach (var network in networks)
if (network.BaseAddress.AddressFamily == address.AddressFamily && network.Contains(address))
return provider;
return default;
}
public void Replace(CdnRangeSet set)
{
var provider = CdnCatalog.ByKey(set.Key) ?? new CdnProvider(set.Name, set.Key, Array.Empty<int>(), Array.Empty<CdnList>(), _ => false);
var networks = set.Ranges.Select(r => IPNetwork.TryParse(r, out var n) ? n : (IPNetwork?)null).Where(n => n != default).Select(n => n!.Value).ToArray();
lock (_gate)
_sets = new(_sets, StringComparer.OrdinalIgnoreCase) { [set.Key] = (provider, networks, set.FetchedAt) };
}
public async Task Load(CancellationToken token)
{
foreach (var set in await DB.Default.Find<CdnRangeSet>().ExecuteAsync(token))
Replace(set);
}
}
}
+114
View File
@@ -0,0 +1,114 @@
using Microsoft.Extensions.Options;
using MongoDB.Entities;
using PrivaPub.Infrastructure.Statistics;
using PrivaPub.Models.Jobs;
using System.Text;
namespace PrivaPub.Infrastructure.Geo
{
// Keeps the CDNs' published address ranges current by itself (owner decision 2026-10-04): at start it loads the stored
// ones, then once a day it downloads each list again. A list that fails to download or to parse, or comes back empty,
// leaves the stored one in place and is tried the next day. Like GeoUpdater, plain HTTPS to fixed hosts, not federation.
public sealed class CdnUpdater : BackgroundService
{
public const string ClientName = "cdn";
static readonly TimeSpan FirstWait = TimeSpan.FromMinutes(1);
static readonly TimeSpan Interval = TimeSpan.FromHours(24);
const int MaxBytes = 16 * 1024 * 1024;
const int MaxRanges = 200_000;
readonly IHttpClientFactory _http;
readonly IOptionsMonitor<StatisticsOptions> _options;
readonly ICdnRanges _ranges;
readonly ILogger<CdnUpdater> _logger;
public CdnUpdater(IHttpClientFactory http, IOptionsMonitor<StatisticsOptions> options, ICdnRanges ranges, ILogger<CdnUpdater> logger)
{
_http = http;
_options = options;
_ranges = ranges;
_logger = logger;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
try
{
await _ranges.Load(stoppingToken);
await Task.Delay(FirstWait, stoppingToken);
while (!stoppingToken.IsCancellationRequested)
{
if (_options.CurrentValue.Cdn.AutoUpdate)
await Update(CdnCatalog.All, stoppingToken);
await Task.Delay(Interval + TimeSpan.FromMinutes(Random.Shared.Next(60)), stoppingToken);
}
}
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
{
}
catch (Exception ex)
{
_logger.LogWarning(ex, "{Service} stopped", nameof(CdnUpdater));
}
}
// The number of CDNs whose ranges were replaced.
public async Task<int> Update(IEnumerable<CdnProvider> providers, CancellationToken token)
{
var replaced = 0;
foreach (var provider in providers.Where(p => p.Lists.Length > 0))
{
try
{
var ranges = new List<string>();
foreach (var list in provider.Lists)
ranges.AddRange(CdnCatalog.Parse(list.Format, await Download(list, token)).Select(n => n.ToString()));
ranges = ranges.Distinct().ToList();
if (ranges.Count == 0 || ranges.Count > MaxRanges)
throw new InvalidDataException($"{ranges.Count} ranges");
var set = await DB.Default.Find<CdnRangeSet>().Match(s => s.Key == provider.Key).ExecuteFirstAsync(token) ?? new CdnRangeSet { Key = provider.Key };
set.Name = provider.Name;
set.Ranges = ranges;
set.Sources = provider.Lists.Select(l => l.Url).ToList();
set.FetchedAt = DateTime.UtcNow;
await DB.Default.SaveAsync(set, token);
_ranges.Replace(set);
replaced++;
}
catch (Exception ex) when (ex is HttpRequestException or InvalidDataException or FormatException or System.Text.Json.JsonException
or TaskCanceledException && !token.IsCancellationRequested)
{
_logger.LogWarning(ex, "Could not update the published ranges of {Cdn}", provider.Name);
}
}
return replaced;
}
async Task<string> Download(CdnList list, CancellationToken token)
{
using var request = new HttpRequestMessage(list.Method == "POST" ? HttpMethod.Post : HttpMethod.Get, list.Url);
request.Headers.Accept.ParseAdd("application/json, text/plain;q=0.9");
if (list.Body != default)
request.Content = new StringContent(list.Body, Encoding.UTF8, "application/x-www-form-urlencoded");
using var response = await _http.CreateClient(ClientName).SendAsync(request, HttpCompletionOption.ResponseHeadersRead, token);
response.EnsureSuccessStatusCode();
if (response.Content.Headers.ContentLength > MaxBytes)
throw new InvalidDataException($"more than {MaxBytes} bytes");
await using var stream = await response.Content.ReadAsStreamAsync(token);
using var buffer = new MemoryStream();
var chunk = new byte[81920];
int read;
while ((read = await stream.ReadAsync(chunk, token)) > 0)
{
if (buffer.Length + read > MaxBytes)
throw new InvalidDataException($"more than {MaxBytes} bytes");
buffer.Write(chunk, 0, read);
}
return Encoding.UTF8.GetString(buffer.ToArray());
}
}
}
+1 -20
View File
@@ -1,4 +1,4 @@
using MaxMind.Db;
using MaxMind.Db;
using Microsoft.Extensions.Options;
@@ -165,23 +165,4 @@ namespace PrivaPub.Infrastructure.Geo
_asn?.Dispose();
}
}
public static class CdnNetworks
{
static readonly Dictionary<int, string> Known = new()
{
[13335] = "Cloudflare",
[209242] = "Cloudflare",
[54113] = "Fastly",
[20940] = "Akamai",
[16625] = "Akamai",
[16702] = "Akamai",
[21342] = "Akamai",
[200325] = "Bunny",
[60068] = "CDN77",
[15133] = "Edgio"
};
public static string Of(int? asn) => asn is { } number && Known.TryGetValue(number, out var name) ? name : default;
}
}
+3 -3
View File
@@ -1,4 +1,4 @@
using MaxMind.Db;
using MaxMind.Db;
using Microsoft.Extensions.Options;
@@ -11,8 +11,8 @@ namespace PrivaPub.Infrastructure.Geo
{
// Keeps the DB-IP Lite databases current by itself, so a deploy needs no timer and no root: once a day it checks
// whether each database was built this month and, if not, fetches this month's (or, early in the month, last month's),
// checks it opens as the right kind of database and swaps it in. This and SMTP are the server's only traffic that is not
// federation, which is why it has its own client instead of IFederationHttp.
// checks it opens as the right kind of database and swaps it in. This, CdnUpdater and SMTP are the server's only traffic
// that is not federation, which is why each has its own client instead of IFederationHttp.
public sealed class GeoUpdater : BackgroundService
{
public const string ClientName = "geo";
+55
View File
@@ -0,0 +1,55 @@
using PrivaPub.Infrastructure.Geo;
using System.Collections.Concurrent;
namespace PrivaPub.Infrastructure.Http
{
public interface IEdgeHints
{
void Remember(string host, CdnProvider cdn);
CdnProvider Of(string host);
}
// The CDN each remote host's responses came through, as their headers tell it (CdnCatalog fingerprints), for the
// weekly description of the server. Like ConnectedAddresses, memory only: what a host answers is never stored per
// request.
public class EdgeHints : IEdgeHints
{
const int MaxHosts = 50_000;
readonly ConcurrentDictionary<string, CdnProvider> _hints = new(StringComparer.OrdinalIgnoreCase);
public void Remember(string host, CdnProvider cdn)
{
if (string.IsNullOrEmpty(host))
return;
if (cdn == default)
{
_hints.TryRemove(host, out _);
return;
}
if (_hints.Count >= MaxHosts)
_hints.Clear();
_hints[host] = cdn;
}
public CdnProvider Of(string host) => host != default && _hints.TryGetValue(host, out var cdn) ? cdn : default;
}
// On the federation client: reads each response's CDN fingerprint for its host. A response without one forgets the
// host's hint, so a server that leaves its CDN is seen as itself again.
public class EdgeHintsHandler : DelegatingHandler
{
readonly IEdgeHints _hints;
public EdgeHintsHandler(IEdgeHints hints) => _hints = hints;
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
var response = await base.SendAsync(request, cancellationToken);
if (request.RequestUri is { } uri && (int)response.StatusCode is >= 200 and < 500)
_hints.Remember(uri.Host, CdnCatalog.FromResponse(response));
return response;
}
}
}
@@ -1,12 +1,18 @@
namespace PrivaPub.Infrastructure.Statistics
namespace PrivaPub.Infrastructure.Statistics
{
public class StatisticsOptions
{
public bool Enabled { get; set; } = true;
public string GeoDirectory { get; set; } = "/var/lib/privapub/geo";
public int PublicCityMinUsers { get; set; } = 10;
public CrawlerOptions Crawler { get; set; } = new();
public GeoOptions Geo { get; set; } = new();
public CdnOptions Cdn { get; set; } = new();
}
//owner decision (2026-10-04): the server downloads the CDNs' published address ranges daily (CdnUpdater)
public class CdnOptions
{
public bool AutoUpdate { get; set; } = true;
}
//owner decision (2026-10-04): the server keeps its DB-IP Lite databases current by itself
@@ -16,6 +22,16 @@ namespace PrivaPub.Infrastructure.Statistics
public string DownloadBase { get; set; } = "https://download.db-ip.com/free";
public long MaxDownloadBytes { get; set; } = 300L * 1024 * 1024;
public long MaxDatabaseBytes { get; set; } = 1024L * 1024 * 1024;
//this server's place, when locating its own address would be wrong (behind a proxy or a CDN): the owner's call
public SelfGeoOptions Self { get; set; } = new();
}
public class SelfGeoOptions
{
public double? Latitude { get; set; }
public double? Longitude { get; set; }
public string City { get; set; }
public string Country { get; set; }
}
//owner decision: off by default (on in production since 2026-10-04); when on, one server a minute, each at most weekly, at most MaxHosts servers
+15 -2
View File
@@ -1,4 +1,4 @@
using Microsoft.AspNetCore.Authentication.JwtBearer;
using Microsoft.AspNetCore.Authentication.JwtBearer;
using Microsoft.AspNetCore.ResponseCompression;
using OpenIddict.Validation.AspNetCore;
@@ -61,9 +61,11 @@ namespace PrivaPub.Middleware
client.DefaultRequestHeaders.UserAgent.ParseAdd($"PrivaPub/{BuildInfo.Ref} (+{baseAddress}/)");
})
.ConfigurePrimaryHttpMessageHandler(provider =>
SafeHttpHandlerFactory.Create(provider.GetRequiredService<IOptions<FederationOptions>>().Value, provider.GetRequiredService<IConnectedAddresses>()));
SafeHttpHandlerFactory.Create(provider.GetRequiredService<IOptions<FederationOptions>>().Value, provider.GetRequiredService<IConnectedAddresses>()))
.AddHttpMessageHandler(provider => new EdgeHintsHandler(provider.GetRequiredService<IEdgeHints>()));
return service
.AddSingleton<IConnectedAddresses, ConnectedAddresses>()
.AddSingleton<IEdgeHints, EdgeHints>()
.AddSingleton<IFederationHttp, FederationHttp>()
.AddSingleton<IDomainBlocks, DomainBlocks>()
.AddSingleton<IContentRenderer, ContentRenderer>()
@@ -119,6 +121,13 @@ namespace PrivaPub.Middleware
client.Timeout = TimeSpan.FromMinutes(15);
client.DefaultRequestHeaders.UserAgent.ParseAdd($"PrivaPub/{BuildInfo.Ref} (+{baseAddress}/)");
});
// not federation either: the CDNs' published address ranges (CdnUpdater), plain HTTPS to the CDNs' own hosts
service.AddHttpClient(CdnUpdater.ClientName, (provider, client) =>
{
var baseAddress = provider.GetRequiredService<IOptionsMonitor<AppConfiguration>>().CurrentValue.BackendBaseAddress?.TrimEnd('/');
client.Timeout = TimeSpan.FromMinutes(2);
client.DefaultRequestHeaders.UserAgent.ParseAdd($"PrivaPub/{BuildInfo.Ref} (+{baseAddress}/)");
});
return service
.Configure<StatisticsOptions>(configuration.GetSection("Statistics"))
.AddSingleton<InteractionSalts>()
@@ -127,6 +136,10 @@ namespace PrivaPub.Middleware
.AddHostedService(services => services.GetRequiredService<InteractionLedger>())
.AddSingleton<IJobHandler, RollupJob>()
.AddSingleton<IGeoLocator, DbIpLocator>()
.AddSingleton<ICdnRanges, CdnRanges>()
.AddHostedService<CdnUpdater>()
.AddSingleton<Domain.Statistics.ISelfLocation, Domain.Statistics.SelfLocation>()
.AddSingleton<Domain.Statistics.IServerPlaces, Domain.Statistics.ServerPlaces>()
.AddHostedService<GeoUpdater>()
.AddSingleton<Domain.Statistics.StatisticsQueries>()
.AddSingleton<IJobHandler, Federation.Crawler.CrawlPlanner>()
+15
View File
@@ -0,0 +1,15 @@
using MongoDB.Entities;
namespace PrivaPub.Models.Jobs
{
// The address ranges one CDN publishes, as CdnUpdater last downloaded them: kept, so a restart needs no download and a
// failed download keeps the previous list.
public class CdnRangeSet : Entity
{
public string Key { get; set; }//the CDN's domain (CdnProvider.Key)
public string Name { get; set; }
public List<string> Ranges { get; set; } = new();
public List<string> Sources { get; set; } = new();
public DateTime FetchedAt { get; set; }
}
}
+8 -1
View File
@@ -1,4 +1,4 @@
using MongoDB.Entities;
using MongoDB.Entities;
namespace PrivaPub.Models.Jobs
{
@@ -57,6 +57,8 @@ namespace PrivaPub.Models.Jobs
public int? Asn { get; set; }
public string AsnOrg { get; set; }
public string Cdn { get; set; }//the CDN in front of it, whose edge address says nothing about where it runs
public string CdnDomain { get; set; }//the CDN's own domain, which groups its servers (CdnProvider.Key)
public string CdnSource { get; set; }//how the CDN was found: "ranges", "headers" or "asn"
public bool? Ipv6 { get; set; }
public string Source { get; set; }
public DateTime? LocatedAt { get; set; }
@@ -79,8 +81,13 @@ namespace PrivaPub.Models.Jobs
public string RegistrationMode { get; set; }
public List<string> Languages { get; set; }
public string Country { get; set; }
public string City { get; set; }
public double? Latitude { get; set; }
public double? Longitude { get; set; }
public int? Asn { get; set; }
public string Cdn { get; set; }
public string CdnDomain { get; set; }
public string CdnSource { get; set; }
public bool? Ipv6 { get; set; }
}
}
+3 -1
View File
@@ -1,4 +1,4 @@
using MongoDB.Bson.Serialization.Attributes;
using MongoDB.Bson.Serialization.Attributes;
using MongoDB.Entities;
namespace PrivaPub.Models.Post
@@ -39,6 +39,8 @@ namespace PrivaPub.Models.Post
[BsonIgnoreIfNull]
public EventDetails Event { get; set; }
[BsonIgnoreIfNull]
public PostPlace Place { get; set; }
[BsonIgnoreIfNull]
public PostPoll Poll { get; set; }
public List<float> Location { get; set; } = new();
public float RangeKm { get; set; } = 5.0f;
+10 -1
View File
@@ -1,4 +1,4 @@
namespace PrivaPub.Models.Post
namespace PrivaPub.Models.Post
{
public class CustomEmoji
{
@@ -104,6 +104,15 @@ namespace PrivaPub.Models.Post
public string Category { get; set; }
}
//a remote post's own place (Pixelfed's location): shown, never federated again
public class PostPlace
{
public string Name { get; set; }
public double Latitude { get; set; }
public double Longitude { get; set; }
public string Country { get; set; }
}
public class EventPlace
{
public string Name { get; set; }
+2 -1
View File
@@ -1,4 +1,4 @@
using Microsoft.AspNetCore.HttpOverrides;
using Microsoft.AspNetCore.HttpOverrides;
using Microsoft.Extensions.Options;
using MongoDB.Bson;
@@ -186,6 +186,7 @@ try
var passwordHasher = app.Services.GetService(typeof(IPasswordHasher)) as IPasswordHasher;
await dbClient.Init(passwordHasher);
await app.Services.GetRequiredService<PrivaPub.Federation.Moderation.IDomainBlocks>().Reload(CancellationToken.None);
await PrivaPub.Api.Mastodon.Auth.PersonaExchange.EnsureFirstPartyClient(app.Services, CancellationToken.None);
}
catch (Exception ex)
{
+3 -8
View File
@@ -28,15 +28,10 @@ namespace PrivaPub.Services
? default
: await context.HttpContext.RequestServices.GetRequiredService<DbEntities>().RootUsers.MatchID(rootId)
.ExecuteFirstAsync(context.HttpContext.RequestAborted);
if (root is not { IsBanned: false, DeletedAt: null })
var refusal = RootJwt.Refusal(root, context.Principal);
if (refusal != default)
{
context.Fail("The account can no longer be used.");
return;
}
// a password recovery ends every session made before it (RootSessions gives the root a new stamp)
if ((root.SessionStamp ?? string.Empty) != (context.Principal.FindFirst(AuthTokenManager.SessionStamp)?.Value ?? string.Empty))
{
context.Fail("The account's sessions were ended.");
context.Fail(refusal);
return;
}
if (context.Principal.Identity is not ClaimsIdentity identity)
+53
View File
@@ -0,0 +1,53 @@
using Microsoft.IdentityModel.JsonWebTokens;
using Microsoft.IdentityModel.Tokens;
using PrivaPub.Extensions;
using PrivaPub.Models.User;
using PrivaPub.StaticServices;
using System.Security.Claims;
using System.Text;
namespace PrivaPub.Services
{
// The /clientapi token: how it is validated (JwtBearer through AddPrivaPubAuth, and the persona token exchange on
// /oauth/token), and when a valid one can no longer be used. One place, so the two never drift apart.
public static class RootJwt
{
public static TokenValidationParameters Parameters(IConfiguration configuration) => new()
{
ValidateIssuer = true,
ValidateAudience = true,
ValidateLifetime = true,
ValidateIssuerSigningKey = true,
ValidIssuer = configuration["AppConfiguration:Jwt:Issuer"],
ValidAudience = configuration["AppConfiguration:Jwt:Audience"],
IssuerSigningKey = new SymmetricSecurityKey(Encoding.UTF8.GetBytes(configuration["AppConfiguration:Jwt:Key"]))
};
//why a token that validated can no longer be used, or default when it can: a token outlives a ban, a removal and a
//password recovery (RootSessions gives the root a new stamp), so the database has the last word
public static string Refusal(RootUser root, ClaimsPrincipal principal)
{
if (root is not { IsBanned: false, DeletedAt: null })
return "The account can no longer be used.";
if ((root.SessionStamp ?? string.Empty) != (principal.FindFirst(AuthTokenManager.SessionStamp)?.Value ?? string.Empty))
return "The account's sessions were ended.";
return default;
}
//the root a token belongs to, when the token is valid and can still be used; default otherwise, never an exception
public static async Task<RootUser> Root(string jwt, IConfiguration configuration, DbEntities dbEntities, CancellationToken token)
{
if (string.IsNullOrEmpty(jwt))
return default;
var result = await new JsonWebTokenHandler { MapInboundClaims = false }.ValidateTokenAsync(jwt, Parameters(configuration));
if (!result.IsValid)
return default;
var principal = new ClaimsPrincipal(result.ClaimsIdentity);
var rootId = principal.GetUserId();
var root = string.IsNullOrEmpty(rootId) ? default : await dbEntities.RootUsers.MatchID(rootId).ExecuteFirstAsync(token);
return Refusal(root, principal) == default ? root : default;
}
}
}
+5 -3
View File
@@ -1,4 +1,4 @@
@page "/stargazing"
@page "/stargazing"
@model PrivaPub.Web.Pages.StargazingModel
@{
ViewData["Title"] = "Stargazing";
@@ -13,8 +13,10 @@
never accounts: what kind of software a server runs, what it exchanges with us, and how reliably. Distinct accounts are
only counted, through a key that is destroyed at the end of each day.</p>
<p>A server we exchange activities with is described once a week, from its public NodeInfo and, when it has one, its
Mastodon instance API. Its location comes from the address we reached, looked up in an offline database: only the
country is shown publicly for small servers, and only the CDN for servers behind one.</p>
Mastodon instance API. Its location comes from the address we reached, looked up in an offline database, and is shown
to the city; for a server behind a CDN, only the CDN is shown, since the address is the CDN's. CDNs are recognised from
the address lists they publish and from their marks in the answers we get, and servers are grouped by their CDN week
by week.</p>
@if (Model.GeoSource is { } geo)
{
<p>Locations come from <a href="https://db-ip.com" rel="nofollow noopener">@geo</a>, licensed under
+10 -4
View File
@@ -1,4 +1,4 @@
# Interop: what every peer sends, what it expects, and what PrivaPub still drops
# Interop: what every peer sends, what it expects, and what PrivaPub still drops
Research as of **2026-10-01**: the platforms' source on their default branches, live ActivityPub fetches from large
instances, release notes and the FEP repository. Version numbers are what was current that day. Claims the
@@ -631,8 +631,10 @@ attachment. The player is a card whose iframe loads from the remote host, which
- **Gaps:**
- **P1 outbound:** keep JPEG/PNG renditions with an explicit `mediaType`.
- **P1 inbound:** Loops' `preview` poster.
- **P2:** Pixelfed `location` → own `privapub.place`, display only and never re-federated; `commentsEnabled: false`
disables replies.
- ~~**P2:** Pixelfed `location` → own `privapub.place`, display only and never re-federated~~ done:
`ObjectShapes.NotePlace` keeps a `Place` with coordinates (Pixelfed sends them as strings) in `Post.Place`, an edit
replaces it, and nothing renders it back out.
- **P2:** `commentsEnabled: false` disables replies.
- **P3:** ignore `Add{Story}` without an error; answer `FeatureRequest` with `Reject`.
### Long-form: WordPress plugin 9.3.1, Ghost 6, WriteFreely 0.17.2
@@ -807,7 +809,11 @@ a remote host.
`GET /api/privapub/v1/instances/:host` returns cached NodeInfo (`metadata` extras: Misskey `themeColor`/`maxNoteTextLength`,
Pleroma `features[]`/`federation.mrf_policies`), the instance API's icon and description, the delivery health our
circuit breaker sees, and which signature scheme worked.
circuit breaker sees, and which signature scheme worked. Its `geo` is the public projection of where the server is
(`precision` `city`, `country` or `cdn`, ROADMAP owner decision on server locations); a CDN-fronted server's names the
CDN's domain, how it was found (`ranges`, `headers`, `asn`) and `before_cdn`, its last place before the CDN. `?host[]=`
answers up to 40 at once, and this server describes itself the same way. `/instances/:host/history` lists its weekly
snapshots; `/api/privapub/v1/cdns` and `/cdns/:domain` group servers by CDN, week by week.
`software.name` → family, for display:
+17 -9
View File
@@ -1,4 +1,4 @@
# PrivaPub roadmap
# PrivaPub roadmap
Written 2026-10-01 from the original 2023 code, the decePubClient UI, a federation gap audit of commit `075c222`, research on .NET ActivityPub libraries, and the owner's decisions. Each phase ends in a tagged deploy plus verification; tick phases off here as they land.
@@ -31,7 +31,7 @@ Written 2026-10-01 from the original 2023 code, the decePubClient UI, a federati
Both open items were closed without a root step (owner decisions, 2026-10-04): the server fetches its geolocation
databases itself, and the deploy makes and signs in as @thepra.
- [ ] Everything on in production (owner decisions 2026-10-04):
- [x] Everything on in production (owner decisions 2026-10-04):
- v1.18.0, deployed and verified 2026-10-04: geolocation that updates itself (DB-IP Lite 2026-10 loaded), the deploy
signing in as @thepra (undiscoverable), the crawler on (1010 servers known within the hour), sign-up by invitation
with one registrations switch;
@@ -41,7 +41,7 @@ Written 2026-10-01 from the original 2023 code, the decePubClient UI, a federati
- SecureMode is on: a persona answers 401 unsigned, a browser is redirected, and all six pasture peers pass;
- account privacy: sign-in and recovery say nothing, recovery codes are hashed for an hour, a recovered password
ends every session, and a deleted root's personas and groups are deleted everywhere;
- v1.20.0, 2026-10-04: one answer everywhere:
- v1.20.0, deployed and verified 2026-10-04: one answer everywhere:
- one counting rule for posts and users;
- gone accounts out of every count;
- replies_count only for public replies;
@@ -151,7 +151,7 @@ and circles (see Owner decisions).
| Question | Decision |
|---|---|
| Client interface | **Mastodon client API**, so Tusky, Elk, Phanpy, Ivory and the official apps work. Each avatar is its own Mastodon account: at OAuth authorize, the logged-in RootUser picks the avatar the token is for. PrivaPub-only features (avatars, groups, contacts, range posts) stay on `/clientapi`. Moving decePubClient onto the Mastodon API is out of scope. |
| Client interface | **Mastodon client API**, so Tusky, Elk, Phanpy, Ivory and the official apps work. Each avatar is its own Mastodon account: at OAuth authorize, the logged-in RootUser picks the avatar the token is for. PrivaPub-only features (avatars, groups, contacts, range posts) stay on `/clientapi`. ~~Moving decePubClient onto the Mastodon API is out of scope.~~ *Superseded 2026-10-04: decePubClient uses both APIs with one sign-in, see "Owner decisions on decePubClient".* |
| Personas | **Unlinkable to other users and servers.** The admin can still see the link in the database. No root IDs next to IPs in logs, NodeInfo counts nothing that links personas, blocks, mutes and notifications are per avatar, and invitation signup no longer names the avatar after the root username. |
| Location-ranged posts | **Local only, never federated.** Shown to local users within the radius, with coordinates rounded on storage. |
| Groups | **Per group, two kinds.** A *community* federates per FEP-1b12, Lemmy-compatible: it Announces the activity, uses `audience`, and accepts posts from non-followers. A *circle* is invitation-only: posts are addressed to the members collection, and objects are served only to signed requests from members. |
@@ -178,7 +178,7 @@ and circles (see Owner decisions).
| Local side in statistics | **Only the kind of local actor** (person, group, application), **and only on public and unlisted traffic.** DMs, followers-only and circle traffic are one "private" class, never broken out per server in public. Circles are never named, whether as a kind or as a reason. Fetches of our own documents are counted per day, never per server. |
| Reading-driven traffic | **Counted per day, never logged per event:** the media proxy, lookups a client asks for, and the client API per endpoint group (admin only). Client app names are not recorded. |
| Describing servers | **Every server we exchange activities with is described weekly**, from its NodeInfo (including the user counts it publishes) and its Mastodon instance API, never its contact account. These requests are unsigned, because they are not ActivityPub documents. Never on read. |
| Server locations | **City and network (ASN) from the offline DB-IP Lite databases** (CC BY 4.0, attributed), downloaded monthly (by the server itself since 2026-10-04). The location comes from the address we connected to; an inbound sender's address is never recorded, and no address is stored. In public: city and network only for servers reporting at least 10 users and not behind a CDN; the country otherwise; only the CDN's name for CDN-fronted servers. The admin sees everything. |
| Server locations | **City and network (ASN) from the offline DB-IP Lite databases** (CC BY 4.0, attributed), downloaded monthly (by the server itself since 2026-10-04). The location comes from the address we connected to; an inbound sender's address is never recorded, and no address is stored. In public: the city, coordinates and network the server was located to; only the CDN's name for CDN-fronted servers. The admin sees everything. *Corrected 2026-10-04: this row used to show only the country for servers reporting fewer than 10 users; the owner never decided that threshold, and it is gone.* |
| Crawler | **Off by default** (`Statistics:Crawler:Enabled`); **on in production since 2026-10-04**, see below. When on, it identifies as `PrivaPub-Stargazer/<version> (+https://privapub.thepra.dev/stargazing)`, where `/stargazing` explains it and how to opt out. It honours robots.txt (an unreachable robots.txt means "keep out") and domain blocks. It visits one server a minute, each at most weekly, and at most 5000 servers. It reads only robots.txt, NodeInfo, the instance API and the peers list, never accounts, posts or directories. Crawled servers stay marked as crawled. |
| A remote account deletes itself | **Its posts are kept but hidden everywhere** (`Post.AuthorGone`): from timelines, profiles, search and lookups by id. Its follows and timeline rows go, as before. |
| Signed-in smoke check in production | **An undiscoverable persona**, the deploy checks `verify_credentials`, home and notifications with it. *Superseded 2026-10-04: the deploy makes and keeps the persona itself, below.* |
@@ -197,6 +197,15 @@ and circles (see Owner decisions).
| What circles and located posts reveal | **Unchanged**: circles still answer WebFinger, and circle and located posts still count in a persona's post count, "a good balance for the fediverse to work". |
| Public `/stargazing` statistics | **Later**, as decided on 2026-10-03; the crawler and the admin API keep collecting meanwhile. |
### Owner decisions on decePubClient (2026-10-04)
| Question | Decision |
|---|---|
| Sign-in | **One login.** decePubClient signs in on `/clientapi` and exchanges that JWT for one persona's Mastodon token (RFC 8693 token exchange on `/oauth/token`, `PersonaExchange`), one token per persona it uses. Only the seeded first-party application may exchange; the token names the persona, never the root, like any other. |
| CDN detection | **The server finds CDNs by itself.** It downloads the address ranges CDNs publish (Cloudflare, Fastly, Amazon CloudFront, Bunny, Gcore, Imperva) once a day from the CDNs' own hosts, reads CDN fingerprints in the responses it already gets from each server (`cf-ray`, `x-served-by`, `x-amz-cf-id`…), and keeps a short list of networks that carry nothing but a CDN (Akamai, Edgio…). |
| CDN-fronted servers | **Kept apart, and followed through time.** Their place stays hidden (the address is the CDN's edge). They are grouped per CDN, named by the CDN's own domain, with week by week how many servers were behind it and who joined or left. Each server's weekly history (place, CDN, size) is public, including where it was before it went behind a CDN. |
| Server locations on the instance API | **Public, as decided for public server locations** (2026-10-03, corrected): city, coordinates to 0.1° and network for every located server whatever its size, the CDN's name for a CDN-fronted one, with DB-IP's attribution. decePubClient's globe draws posts at their author's server. This server's own place comes from its host's address, or from `Statistics:Geo:Self` when the owner sets it (a server behind a proxy). |
## Libraries (researched; no maintained .NET ActivityPub library exists, so Letterbook and Iceshrimp.NET both wrote their own)
| Area | Choice |
@@ -587,7 +596,7 @@ it, raw where it doesn't.
- Misskey actor extras;
- book reviews (`relatedWith`, `rating`);
- Funkwhale and Castopod metadata;
- Pixelfed `place` (display only, never re-federated);
- ~~Pixelfed `place` (display only, never re-federated)~~ done 2026-10-04: `Post.Place`, `privapub.place`;
- Bluesky bridging per persona.
### Sweep and statistics (T and M, planned 2026-10-03, between P6 and P7)
@@ -606,8 +615,8 @@ statistics gain value with every day recorded.
Later, and not part of these steps: the public `/stargazing` statistics. It is anonymous, cached and rate-limited, and
covers per-server software, self-published counts, location as projected by the rule above, availability buckets and
the public activity mix. Per-server activity is shown only as order-of-magnitude buckets, and only for servers that
report at least 10 users; smaller ones fold into one "small servers" aggregate.
the public activity mix. *Corrected 2026-10-04: an earlier version showed per-server activity only as order-of-magnitude
buckets and only for servers reporting at least 10 users; the owner never intended either restriction.*
### Cut or deferred (deliberately)
- **Cut:**
@@ -621,7 +630,6 @@ report at least 10 users; smaller ones fold into one "small servers" aggregate.
- S3 storage.
- JSON-LD and LD-signature processing. FEP-8b32 proofs need neither (JCS), so they are in P8.
- **Deferred:** video transcoding (remux only for now).
- **Out of scope:** moving decePubClient onto the Mastodon API.
## Verification (per phase)