Primarily what you require is issuing an external command on your OS from your browser. But this is usually tough due to obvious security reasons. However, there is an add-on for Firefox which can achieve this : Commandrun. But I haven't tried this addon so can't comment on it.
A Reasonably Good Solution
However, I'd suggest another alternative solution using Autokey. This is not going to be the best solution as per your needs but it can be a reasonable one.
So, first install autokey:
sudo apt-get install autokey-gtk #Assuming you're on a Debian-based linux
Now, in Autokey, create a new script and write the following lines in that script:
import subprocess
keyboard.send_keys("<ctrl>+l") #Bring cursor to the location bar
time.sleep(0.15)
keyboard.send_keys("<ctrl>+c") #Now copy the location from the location bar
time.sleep(0.15)
keyboard.send_keys("<ctrl>+w") #Now close the window
time.sleep(0.15)
text = clipboard.get_selection() #copy the clipboard content i.e. the URL
subprocess.call(["vlc", text]) #Make the call to vlc player
Now, assign some hotkey to this script such as SuperL or anything else that you prefer. Also, assign a filter so that this script will run only on Firefox windows and not any other window. To apply the filter just add .*Firefox.* in the Filter.
How this works:
Whenever you are on a youtube video page on Firefox, you can now press SuperL and this script will run. It will first copy the url and then close the youtube tab. Then it will take this url and make a call to vlc to run this video.
Further extension
You can further easily extend the script to make an ssh connection to your RaspberryPi and run the video there.