Return to Question

I have several Linux VMs on VMware + SAN.

What happened

A problem occured on the SAN (failed path) so that for some time, there were I/O errors on the Linux VMs drives. When the path failover had been done, it was too late: every Linux machine considered most of its drives as not "trustworthy" anymore, setting them as read-only devices. The root filesystem's drives were also impacted.

What I tried

• mount -o rw,remount / without success,
• echo running > /sys/block/sda/device/state without success,
• dug into /sys to find a solution without success.

Finally...

I had to reboot all my Linux VMs. The Windows VMs were fine...

Some more info from VMware...

The problem is described here. VMware suggests to increase the Linux scsi timeout to prevent this problem to happen.

The question!

However, when the problem does eventually happen, is there a way to get the drives back to read-write mode? (once the SAN is back to normal)

I have several Linux VMs on VMware + SAN.

What happened

A problem occured on the SAN (failed path) so that for some time, there were I/O errors on the Linux VMs drives. When the path failover had been done, it was too late: every Linux machine considered most of its drives as not "trustworthy" anymore, setting them as read-only devices. The root filesystem's drives were also impacted.

What I tried

• mount -o rw,remount / without success,
• echo running > /sys/block/sda/device/state without success,
• dug into /sys to find a solution without success.

What I may have not tried

• blockdev --setrw /dev/sda

Finally...

I had to reboot all my Linux VMs. The Windows VMs were fine...

Some more info from VMware...

The problem is described here. VMware suggests to increase the Linux scsi timeout to prevent this problem to happen.

The question!

However, when the problem does eventually happen, is there a way to get the drives back to read-write mode? (once the SAN is back to normal)

I have several Linux VMs on VMware + SAN.

Recently, a problem occured on the SAN (failed path) so that for some time, there were I/O errors on the Linux VMs drives. When the path failover had been done, it was too late: every Linux machine considered most of its drives as not trustworthy anymore, setting them as read-only devices. The root filesystem's drives were also impacted.

The problem is described here by VMware. They suggest to increase the Linux scsi timeout to prevent this problem to happen.

However, when the problem does eventually happen, is there a way to get the drives back to read-write mode? (once the SAN is back to normal)

When it happened (reading OK, but no writing), I tried mount -o rw,remount / without success, then I dug into /sys to find a solution without success, and finally had to reboot all my Linux VMs. The Windows VMs were fine...

Hint: would something like echo running > /sys/block/sda/device/state help?

I have several Linux VMs on VMware + SAN.

What happened

A problem occured on the SAN (failed path) so that for some time, there were I/O errors on the Linux VMs drives. When the path failover had been done, it was too late: every Linux machine considered most of its drives as not "trustworthy" anymore, setting them as read-only devices. The root filesystem's drives were also impacted.

What I tried

• mount -o rw,remount / without success,
• echo running > /sys/block/sda/device/state without success,
• dug into /sys to find a solution without success.

Finally...

I had to reboot all my Linux VMs. The Windows VMs were fine...

Some more info from VMware...

The problem is described here. VMware suggests to increase the Linux scsi timeout to prevent this problem to happen.

The question!

However, when the problem does eventually happen, is there a way to get the drives back to read-write mode? (once the SAN is back to normal)

I have several Linux VMs on VMware + SAN.

Recently, a problem occured on the SAN (failed path) so that for some time, there were I/O errors on the Linux VMs drives. When the path failover had been done, it was too late: every Linux machine considered most of its drives as not trustworthy anymore, setting them as read-only devices. The root filesystem's drives were also impacted.

The problem is described here by VMware. They suggest to increase the Linux scsi timeout to prevent this problem to happen.

However, when the problem does eventually happen, is there a way to get the drives back to read-write mode? (once the SAN is back to normal)

When it happened (reading OK, but no writing), I dug into /sys without success to find a solution, and finally had to reboot all my Linux VMs. The Windows VMs were fine...

Hint: would something like echo running > /sys/block/sda/device/state help?

I have several Linux VMs on VMware + SAN.

Recently, a problem occured on the SAN (failed path) so that for some time, there were I/O errors on the Linux VMs drives. When the path failover had been done, it was too late: every Linux machine considered most of its drives as not trustworthy anymore, setting them as read-only devices. The root filesystem's drives were also impacted.

The problem is described here by VMware. They suggest to increase the Linux scsi timeout to prevent this problem to happen.

However, when the problem does eventually happen, is there a way to get the drives back to read-write mode? (once the SAN is back to normal)

When it happened (reading OK, but no writing), I tried mount -o rw,remount / without success, then I dug into /sys to find a solution without success, and finally had to reboot all my Linux VMs. The Windows VMs were fine...

Hint: would something like echo running > /sys/block/sda/device/state help?