Home

Welcome to the pdwfs wiki!

Ideas

• implement I/O libc calls to allow removing an entire directory (i.e. have "rm -r directory" works): the -at calls family, unlinkat, newstatat, etc,
• remove the inode layer and the handling of directories in pdwfs (basically just intercept the read/write calls using the full path as a key in Redis), workflow setup process would consist in making sure it works on normal filesystem (to test file and directory operations), then switch to pdwfs,
• implement different data distribution strategy among the Redis instances for better locality awareness: per stripe, per file, per producer,
• SLURM plugin to setup and teardown the Redis servers infrastructure,
• writers for asynchronous I/O (software burst buffer), processes running next to Redis instances that write new Redis file data on filesystem
• I/O operation stats gathering (like Selfie) in Redis instances, dump stats in file or database on teardown
• try aternative Redis-protocol-speaking database (e.g. ARDB + RocksDB, the latter being adapted for fast storage system like SSD, Optane?)
• re-coded alternative to Redis adapted to pdwfs (e.g. https://github.com/JCapul/kvdroid), feature: binary protocol instead of ascii protocol, use low-level communication protocols for high-speed network fabric (e.g. OFI libfabric) instead of TCP, what else ???
• redis-protocol proxy server for a hierarchical/tiered data transfer infrastructure (to cope with hero runs with +100,000 clients)
• secured data transfer infrastructure with spiped
• read-cache mode: Redis instances used as read cache for files on FS, 2 options: files are read by the application and sent in Redis for caching or a dedicated reader process receives read notifications from a channel and push file data in Redis instances