Is it safe to raise that value and what would be the consequences of a too high value?
This is from an old link, so might have changed, but the point is, raise it as high as you want:
inotify watch is 40 bytes
inotify device is 68 bytes
inotify event is 272 bytes
Is there a way to find out what are the currently set watches and which process set them to be able to determine if the reached limit is not caused by a faulty software?
I don't expect that there should be a way to find out who changes the value (here referring to /proc/sys/fs/inotify/max_user_watches). I guess very few codes need the values higher than the default (~8k). The only one such code I know is Tracker (a desktop indexer, among other things), and it makes such heavy use of the feature that the Debian package sets it to ~64k.
NOTE: If you want the setting to be permanent, set it in "/etc/sysctl.conf". Mine looks like:
fs.inotify.max_user_watches=100000
EDIT: Since september '13 /etc/sysctl.conf is deprecated. To make this setting permanent, you should edit any file in the /etc/sysctl.d/ directory. (e.g. /etc/sysctl.d/99-sysctl.conf)
You can now manually reload the systemd configuration by executing this command as root to apply the new configuration:
sysctl --system
