actually why don’t fedi servers/media proxies use bittorrent to fetch media from each other instead of hammering the origin server? this seems like a low-hanging fruit that can help immensely with load balancing

@bartholin no.

@lanodan no it doesn’t. have you seen webtorrent?

you don’t need to rely on dht either. the origin can be a tracker and speed up connection setup that way.

@bartholin ipfs is high-latency, not meant for ephemeral media, and does nothing to mitigate congestion. bittorrent is a better fit.

@lanodan i mean, servers could also speak webtorrent to each other and negotiate the connections via webrtc, but i feel like that’s gonna be a pain for software not written in node.js

@xaetacore some CDNs literally use webrtc to allow clients to share files, and peertube uses it for live streaming fragments as well. there is no technical reason not to do this.

also, modifying files on fedi is not really a thing. they always get a new URI.

@xaetacore webrtc, as well as http/3, were both designed to reduce the need for, and the load on, CDNs. fedi is decentralized, but i see no reason why parts of it can’t be distributed as well.

in fact i think the CDN architecture you’re suggesting is a terrible idea.

notes and media on fedi are ephemeral. storage like this is a concern only for caching and things like misskey’s drive feature. so it’s the fetch, not the storage, that needs to be prioritized.

servers could also use webtorrent (i.e. set up the data channels via webrtc instead of also being torrent trackers).

@xaetacore this isn’t about clients. that would not be a low-hanging fruit because it needs work on backends, frontend, apps…

i’m talking about the backend side of things. instances already talk to each other, they already have established peer connections, so why wouldn’t they distribute media like this?

@xaetacore also, building on top of that, if the nodes with complete copies are known, the load could be distributed by allowing clients to fetch from these alternate locations if they are in closer geographical proximity. that might be less trivial in practice though, because you would need end-to-end cryptographic verification of file contents for trust reasons, and preferably without stalling presentation until all the data is available

@xaetacore and for the record, i don’t share your worries about exposing IP addresses. it’s the fucking internet, and it’s not the 90s where you’d bitchslap.exe someone on IRC, nor does that address disclose who you are. if you’re not using tor there is no way in hell this is a real concern to you, and if you are using tor it’s a non-issue to begin with.

@icedquinn it solves a very different problem poorly (distributed storage, not distributed network load)

@icedquinn @xaetacore i’ve used webtorrent as an example because that is used by peertube for live streaming fragments. people have used it for small files on static websites too. it’s really not that slow at all, and it has better congestion management than anything TCP-based.

it’d be optional anyway. like, don’t bother if the roundtrips and stuff to establish data connections would take longer than just serving the file over http under current network conditions.

@i i think you misunderstand. this is about more efficiently fetching content that you were already going to fetch anyway, not storage/caching. just network load balancing.

@i so like, if you fully federate with an instance and don’t have their media marked as risky, your instance could also distribute the files that it already downloads. whether it keeps serving them after the fact or stops once the immediate demand subsides is a separate question.

@xaetacore at least some of this can be addressed by restricting connections to other peers based on user preferences (there can be some granularity there, like only connecting to people on the same instance).

though i don’t think client-side stuff like this will make a difference unless your instance is already too big to manage. the real problem is posting a video and getting your instance DDoS’d by a million mastodon instances trying to grab and cache everything they see

@i distributing the load does matter. and until http/3 rolls out everywhere, torrents actually have much better latency and behavior in congested networks (which is a big deal to anyone hosting an instance from home)

@i i mean, there is a time window shortly after publishing a note with a file attachment during which demand for that attachment spikes. that’s the main target of this optimization, so there’s a good chance there will be enough active peers for torrents to make a real difference.

@icedquinn @xaetacore it uses webrtc to establish peer connections. it’s not the bittorrent you’re familiar with.