X server, keyboard & mouse lockup with corrupted display

Question

My computer (which has an NVIDIA card) has a rendering problem that is very annoying.

Sometimes the X server 'locks up' after freezing (or whatever's going on in there because VLC music loops at the current packet then) for ~1 second, the pointer becomes invisible, the keyboard & mouse won't work and I get a screen that looks like this:

except the colors are different each time. I enabled ctrl-alt-backspace and alt-sysrq-[rk] keys and not even those have an effect, to recover I must do a cold reboot.

Sometimes instead of getting the above screen I get a console full of kernel errors like nouveau: failed to idle channel XXX. Then after a few seconds all programs get killed (while in the tty). This might be an X server because my xgamma settings (which are reset every logout) are even then still in effect.

And then even the console itself disappears, no login screen, just black (and the only way out is also a cold reboot).

The strangest part is that KTorrent (which was downloading a file when this happened) was still transmitting data during the lockup, according to my modem LED. All programs are killed, can reproduce with VLC playing music.

In particular, I can't run any JavaFX 8 programs (not even the simplest one which creates an empty window) because when I run java -jar my.jar, the window appears but in a degenerated/rendering-out-of-sync form, and either a) the screen locks up after a few seconds of b) after a few (un)maximizing/resizing the above happens. Also happens sometimes while I use firefox or even when there are no windows open.

wmctrl -l shows that the lockup screen is not a window. Also, as said above, windows/programs are still open so all GUI processes wouldn't be killed. This is very confusing and I'm out of ideas.

Answer 1

By the way: alt-sysrq-[rk] won't help here - it actually kills all the programs itself and putting you in that console scenario. However, that is not the problem. In a nutshell, it's nouveau.

From what you're saying, the X server is still functioning but the graphics "connection" appear to be cut off from your display.

Since a nouveau driver is for NVIDIA cards, java, firefox and your other programs are trying to utilize the card thinking it's available. This is called video (aka. hardware) acceleration.

The screen "lockup" is not from the X server then, it's from the lack of hardware capabilities in nouveau for your card (3D rendering, video decoding, etc.), so when a program attempts to use one of them, it crashes nouveau.

Consider the GPU code names and video engines for NVIDIA cards. Assuming you have a GeForce 6/7 series card, it ends up in the NV40 (Curie) family, which in turn use VPE1, VPE2 and VP1 video engines, none of which supports video (hardware) acceleration (except for XvMC from VPE2). That is what's making your X server "fail". (Be warned, the proprietary NVIDIA driver will not help in this case because of the missing capabilities.)

The solution is to disable hardware acceleration entirely, and just stick to software rendering until you get a supported card. In java 8, you can do this: java -Dprism.order=sw -jar my.jar, works fine on my system. And firefox has an option in its advanced preferences to enable hardware acceleration (if it's available, which mistakenly is but I don't know why), make sure that's unchecked.

Update: nouveau was never good at utilizing NVIDIA hardware in the first place, so if you're using one and your distro (e.g. RHEL 7) makes it difficult to get proprietary drivers I recommend you replace the card with a Radeon/Intel whose open-source drivers are better supported.