Wednesday, May 2, 2012

Removed LUN and multipath reports "tur checker reports path is down"

When removing a LUN from a Red Hat Enterprise Linux 5 (RHEL 5) server, I ran into this interesting issue:
[root@server ~]# multipath -ll
sdat: checker msg is "tur checker reports path is down"
sdbt: checker msg is "tur checker reports path is down"
sdct: checker msg is "tur checker reports path is down"
sdt: checker msg is "tur checker reports path is down"
This is annoying and problematic since running pvs or something similar will cause the system to hang as it tries to access the missing LUN. It's briefly covered below, but see also the official documentation: http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Online_Storage_Reconfiguration_Guide/removing_devices.html
  1. Stop using the volume and umount
  2. Remove it from software RAID or LVM volume group
  3. Use multipath -l to get the block devices names (e.g., sda, sdb, etc.)
  4. Use multipath -f mpathX
  5. To flush outstanding I/O ("sync"), use blockdev –flushbufs device
  6. Make sure applications and things are no longer referencing the device(s)
  7. echo 1 > /sys/block/device-name/device/delete

1 comment:

  1. I had to clean up lots of luns, and was lazy..
    WARNING: Potentially dangerous!!

    #!/bin/bash
    # script to remove luns from device-mapper and scsi layer
    #

    if [ -z $1 ]
    then
    echo "usage: ./removelun "
    echo "example: ./removelun mpath42"
    exit
    fi

    MPATH=$1

    echo query device $MPATH..
    DEVICES=`multipath -ll $MPATH | grep ':.:.:' | awk '{print $3}' `
    echo $DEVICES

    echo flushing multipath device $MPATH
    multipath -f $MPATH

    echo flush outstanding i/o to sd devices ..
    for DEVICE in $DEVICES
    do
    echo $DEVICE..
    blockdev --flushbufs /dev/$DEVICE
    done
    echo deleting sd devices..
    for DEVICE in $DEVICES
    do
    echo $DEVICE..
    echo 1 > /sys/block/$DEVICE/device/delete
    done

    echo done, now you can unpresent on disk array

    ReplyDelete