I don't expect any problem as far as each directory have different file names and you don't use the --delete flag. The caveat is that you have to guarantee the file name uniqueness across the sources.
If you have a file called foobar (or a directory called foobar) in more than one source, each rsync process would replace the destination foobar with its own source version.