Select branch based on file properties

[DerSpatz]

Right now, the branch selection is only based on the properties and structures of the underlying file systems. Wouldn't it be useful to also take into account the properties of the file that needs to be written. I am thinking specifically about (initial) file size and file type.

The user could add SSDs and HDDs into the same pool, and mark certain drives/branches as preferred for certain files. Let's say all files smaller than 10MB are written to the SSD(s), while all files with an .iso type are written to the HDDs (those are just examples). After this preselection, mergerfs follows the set policy applied to the selected subbranch to write the file.

There should be some settings to handle edge cases (like if the SSD is full, write files to the HDDs or give back an error).

Additionally, there should be a tool which can run automatically or manually, which sorts the files into the subbranches initially, and regularly checks if conditions are met to transfer a file to a different subbranch (most likely a file got larger than the limit for this subbranch), maybe with a hysteresis (initial limit is 10MB, but move only if larger than 15MB).

The above described behavior would give a lot of advantages:

• faster responsiveness (small files are easier handled by SSDs)
• persistent cache like behavior without the need for an actual cache (and complicated setup)
• transparent optimized handling for files in web services etc.

To further explain the last point: jellyfin (for example) can be set up to store metadata next to the media files, or in dedicated metadata directories. One is easier to manage for the user, while the other usually loads faster because it is then stored on a SSD. In my case the disks also need to spin up, only to show the posters for the movies and TV shows.

If mergerfs would be able to automatically write all smaller files to the SSD subbranch, this would give the advantages of both options, and it would be fully transparent to jellyfin. The disks would only spin up on actual video playback.

Because this behavior would happen on a file system level, it would also work for many other services while at the same time making configuration of those services easier, as the user would only need to mount the mergerfs pool as storage and not think about what should be stored on which drive.

Lastly the user should be warned that if he uses snapraid, the SSDs should also be included in the snapraid array.

[trapexit]

In the future please submit individual discussion entries or tickets per topic. It is difficult to discuss details when there are multiple topics in flight.

I am thinking specifically about (initial) file size and file type.

The initial file size is always 0. There is no such thing as the creation of a file that isn't zero. Also there is no "type" unless you mean file vs directory vs fifo, etc. At best you can do whitelist/blacklists on branches based on file names or paths and that idea has already been discussed at length. It's trivial to implement but it isn't easy to create a language for the user to configure in the current config setup. It however is a pretty niche request and as such not something I've put effort into till other more generally important things are done.

There should be some settings to handle edge cases (like if the SSD is full, write files to the HDDs or give back an error).

There is. moveonenospc.

Additionally, there should be a tool

People can create any tooling they wish. Such a tool has nothing to do with mergerfs.

In my case the disks also need to spin up, only to show the posters for the movies and TV shows.

This is not really accurate. Plex and I'm pretty sure Jellyfin store metadata they collect in a bespoke location. Not in the directory with the media. Most people simply put that directory on an SSD already as part of their container setup. Additionally, mergerfs has no choice but to read from the HDDs with media when software request the contents of a directory and when scanning they also look for metadata in the files which means they open the file which means they must spin up.

I've written about this in detail.

https://github.com/trapexit/mergerfs/wiki/Limit-Drive-Spinup

[DerSpatz]

Thank you for your detailed reply!

To be honest, I don't know enough about how writing/copying files is handled in Linux. I assumed file size and type would be known before the files is written.

At least in jellyfin, the user can choose where additional metadata is stored. Of course this does not prevent spinup if the media file's metadata is needed.

I guess I'll try to replicate this behaviour manually by adding an SSD to the branch as NC, and move the files over by hand. If I see any benefits in this, I will write a script that does this automatically.

[trapexit]

I assumed file size and type would be known before the files is written.

There is no such thing as a file type outside "regular", "directory", "socket", "fifo", "symlink", etc. Unix doesn't even have "extensions" like Windows. It's just convention. ".iso" or ".png" is meaningless to the filesystem. It's just part of the file name. That doesn't mean mergerfs can't do something based on the name but it would be imperfect. Lots of software write to files with a temporary name and then rename so by the time the name has meaning to you it is too late to pick where it goes. At most it could be moved later.

Size is always zero when you create it. There is no such thing as "copy" or "move" in the filesystem. There is only "create file", "write data", "close file". There is no obvious correlation between any of these at a high level. Especially across multiple files. You could be writing a file from generated data, from data read from the network, from data read from another filesystem... mergerfs has no idea.