Is there a way to combine multiple folders into one folder using symlinks?

Question

At my work we generate a lot of data for analysis, such that we have to keep purchasing new NASs for data storage. This has created a situation where on our computational servers we have many NASs mounted and we have to keep of track which dataset is on which NAS. Each NAS has a similar directory structure so for example if datasetA was on NAS foo then it would be in /mnt/foo/data/datasetA on the server, while datasetB on NAS bar would be at /mnt/bar/data/datasetB. I am thinking of creating a single data folder on the local filesystem and symlinking all the datasets from the mounted NASs onto it, e.g.:

mkdir /data
ln -s /mnt/foo/data/* /data
ln -s /mnt/bar/data/* /data

Doing this, we would just access the datasets from /data and wouldn't care about which NAS the data was actually stored on. I think this would work, except whenever a new dataset appears on an NAS, we would need to manually symlink it into /data. Is there a way to set up these symlinks such that the new directories appear automatically? I guess one option is just to set up a cron job that re-runs the linking commands periodically, but maybe there's a better solution.

Answer 1

Especially if these are read-only, an overlayfs setup might since the issue.

However,

we have to keep purchasing new NASs for data storage

Sounds like both an expensive and an unreliable solution. It certainly sounds like something I would advise against, as it on top of that is also administratively intensive - and I mean not only in terms of IT administration, but also in terms of administrative overhead of buying NASes.

The usual solution here is to buy a storage server with a beefy network interface and a lot of drive bays, and add new drives as you go along, and retire old drives as they start to degrade with age. Set up a raid 6, which you can do relatively trivially using lvmraid. Put all your data on that one logical storage device.

Once we're not talking about Terabytes but Petabytes, moving to something more clusterable would become the method of choice. I'm not experienced at all, but I hear good things about ceph, which is a storage cluster solution for cases where your storage starts to fill racks.