@lucy and iirc even that header is fairly ambiguous and could easily occur in any random byte stream…

@snacks @lucy i blame the very concept of file systems because both name extensions and headers/magic numbers are a bad form of in-band signalling

@snacks @lucy the idea is to, ideally, never store or transmit random blobs of data without identifying metadata

in general, metadata is not handled well on any popular system. we have dozens of audio tag formats, xmp + sidecar files, .DS_Store/.directory and others, extended file attributes etc. etc.
it’s terrible. there’s got to be a better way, and maybe files and folders are just a bad concept to design computer storage after

@desea @lucy @snacks i’m not a systems engineer

i think data storage should have more than one possible abstraction. how the records are ultimately stored should not be that important, but i would want to have different views of the data depending on the use case and my own habits and preferences

@lucy @snacks @desea and as a programmer i just HATE file systems because they’re so fucking painful to use. where do i store shit? how do i serialize my data structures? how do i ensure ACID? what do i do if a transfer is interrupted? what’s the optimal allocation/access strategy for this filesystem and my particular workload? and so on and so forth

@lucy @snacks @desea like, look at how e.g. mastodon stores media files, or how nginx organizes cache directories. they do some weird ass directory trees to avoid taking too much of a performance hit on ext4 or something

@lucy @snacks @desea ceph sort of has the right idea, it’s just way too enterprise-oriented