TSM + HP Library + HP LTO Drive = I/O error on drive

[nia]

Hi Mark,

There is still pending issues with TSM and non-IBM LTO library with non-IBM LTO drives ... in The past, I could not even get that far as seen below.

I have this configuration:

Library: 70 CHANNEL: 3 TARGET: 00 LUN: 00
 Vendor identification: HP
 Product identification: MSL6000 Series
 Product revision level: 2.00
 Unit serial number: 80000070
 NAA: 70:11:22:33:ab:3:00:00

Drive: 71 CHANNEL: 3 TARGET: 00 LUN: 01
 Library ID: 70 Slot: 01
 Vendor identification: HP
 Product identification: Ultrium 4-SCSI
 Product revision level: N11G
 Unit serial number: 80000071
 NAA: 70:11:22:33:ab:3:00:01
 Compression: factor 1 enabled 1

Drive: 72 CHANNEL: 3 TARGET: 00 LUN: 2
 Library ID: 70 Slot: 2
 Vendor identification: HP
 Product identification: Ultrium 4-SCSI
 Product revision level: N11G
 Unit serial number: 80000072
 NAA: 70:11:22:33:ab:3:00:2
 Compression: factor 1 enabled 1

I get this in TSM:

ANR8302E I/O error on drive DRIVE1 (/dev/tsmscsi/mt0) with volume HPS002L4 (OP=WRITE, Error Number=0, CC=305, rc = 12, KEY=40,
ASC=00, ASCQ=07,
SENSE=F0.00.40.00.00.00.00.58.00.00.00.00.00.07.00.00.00.00.00.00.00.00.,
Description=Drive failure).  Refer to
Appendix C in the 'Messages' manual for recommended action.
ANR8792E Unrecoverable drive failures on drive DRIVE1 (/dev/tsmscsi/mt0); drive is now marked offline.
ANR0515I Process 2 closed volume HPS002L4.
ANR1227E BACKUP STGPOOL: Process 2 terminated - internal server error detected.
ANR9999D Thread<60> issued message 1227 from:
ANR9999D Thread<60>  0x00000000b8d93a outRptf
ANR9999D Thread<60>  0x00000000631352 EndBackupProc
ANR9999D Thread<60>  0x00000000631856 AfBackupPoolThread
ANR9999D Thread<60>  0x00000000bfdf84 StartThread
ANR9999D Thread<60>  0x0000380ea0673d *UNKNOWN*
ANR9999D Thread<60>  0x0000380ded44bd *UNKNOWN*

Attached is V3 syslog & dump_tape

hp.lto.messages

dump_tape.HPS002L4


I think this is an old issue with MHVTL+TSM .. some how it does not like a Library that is non IBM when using LTO type drives

I believe, it may have to do with Drive Device Type being detected by TSM which is being thrown off by the Library model, vendor type ? is there a relationship ?

Let me know if you need further testing.

-nia

[nia]

FYI, Please note, above example involved copying data reading from IBM LTO-4 drive attached to IBM Library 3584 emulation then writing to HP Library hardware with HP LTO-4 drive. The Problem occurs with the HP LTO Drive/Media. The same would have occurred if isolated to just the HP LTO Library/drive anyway.

-nia

[Mark Harvey]

(OP=WRITE, KEY=40,  ASC=0, ASCQ=7)

This is an end of tape (EOM bit set), ASC/ASCQ 0/07 is program early warning.

Not near computer at the moment, so I cant check right now.. Perhaps tomorrow.

Cheers
Mark

[nia]

I have performed additional testings also .. Please see new message files attached. This Test maybe a bit cleaner than the first one.

hp.lto.tsm.write.messages

hp.lto.tsm.init.messages

dump_tape.HPS003L4

Thanks

-nia

[Mark Harvey]

The HP SCSI reference guide says this about Programmable Early Warning Size (PEWZ):

The device server shall return CHECK CONDITION status, with the sense key set to NO SENSE, the EOM bit set to one and the additional sense code set to PROGRAMMABLE EARLY WARNING DETECTED at the
completion of a command that caused the medium to transition into the PEWZ if that command is:
a) WRITE(6);
b) WRITE(16);
c) WRITE FILEMARKS(6); or
d) WRITE FILEMARKS(16).

The error you are seeing is the tape emulation returning PEWZ.

[Mark Harvey]

And the snippet from the syslog:

Sep 11 21:11:01 viper vtltape[2829]: write_tape_block: Successfully wrote block: 3765
Sep 11 21:11:01 viper vtltape[2829]: writeBlock: End of Medium - Programmable Early Warning
Sep 11 21:11:01 viper vtltape[2829]: mkSenseBuf: Sense buf: 0x6210f0
Sep 11 21:11:01 viper vtltape[2829]: mkSenseBuf: SENSE [Key/ASC/ASCQ] [40 00 07]

[Mark Harvey]

The PEWZ has been reverted for the time being.

A 'better' version is in the works.

Please let retest from latest github and confirm it now works.

Thanks
Mark

[nia]

Looks a little better or at least the I/O error is now more clear ..

ANR8302E I/O error on drive DRIVE2 (/dev/tsmscsi/mt1) with volume HPS002L4 (OP=WRITE, Error Number=0, CC=400, rc = 1, KEY=4D, ASC=00,
ASCQ=02,
SENSE=F0.00.4D.00.00.00.00.58.00.00.00.00.00.02.00.00.00.00.00.00.00.00.,
Description=Physical end of media encountered).  Refer
to Appendix C in the 'Messages' manual for recommended action.
ANR1411W Access mode for volume HPS002L4 now set to "read-only" due to write error.
ANR0514I Session 2 closed volume HPS002L4.

I have attached the syslog and dump tape also.

dump_tape.HPS001L4
dump_tape.HPS002L4
hp.lto.messages.zip

-nia

[nia]

Mark, aside from my update in my previous reply for the HP LTO drive issue, I think the change for that maybe affecting the IBM LTO drives now, see below: I will need to back rev my code to test this further .. I am not sure if I have tested the IBM LTO long enough to see this from before or if this is new.

I am now getting this in TSM when using the IBM LTO-4 tape drives ..

ANR0514I Session 2 closed volume IBM001L5.
ANR0511I Session 2 opened output volume IBM001L5.
ANR9999D_2628632188 DumpBlock(pvrntp.c:6134) Thread<59>: DumpBlock error on drive DRIVE1 (/dev/IBMtape0); already reached EOV once.
ANR9999D Thread<59> issued message 9999 from:
ANR9999D Thread<59>  0x00000000b92302 OutDiagToCons
ANR9999D Thread<59>  0x00000000b94e09 outDiagfExt
ANR9999D Thread<59>  0x00000000c5443f DumpBlock
ANR9999D Thread<59>  0x00000000c564b1 NtpWriteNC
ANR9999D Thread<59>  0x00000000c62718 LtoWriteNC
ANR9999D Thread<59>  0x000000008c00c0 AgentThread
ANR9999D Thread<59>  0x00000000bfdf84 StartThread
ANR9999D Thread<59>  0x0000380ea0673d *UNKNOWN*
ANR9999D Thread<59>  0x0000380ded44bd *UNKNOWN*
ANR1411W Access mode for volume IBM001L5 now set to "read-only" due to write error.
ANR0514I Session 2 closed volume IBM001L5.
ANR0523W Transaction failed for session 2 for node TSM08 (Linux86) - error on output storage device.

Attached is message and tape _dump for that.

ibm.lto.zip
dump_tape.zip

Thanks
-nia

[Mark Harvey]

OK, just getting around to having another look at this.

I am now getting this in TSM when using the IBM LTO-4 tape drives ..

ANR0514I Session 2 closed volume IBM001L5.

And the syslog / dump_tape confirm the 'LTO-5 media' being loaded into an LTO-4 drive..

PCL is : IBM001L5
Media density code: 0x58
Media type code   : 0x0b
Media description : Ultrium 5/16T
Tape Capacity     : 524288000

Sep 13 00:31:39 viper vtltape[7211]: loadTape: Setting MediumDensityCode to LTO5 (0x58)

Hmmmm. Not sure about higher rev media into a drive.
Documentation describes higher rev drive with lower rev media, but not the other way around.

Can you pls re-test using LTO-4 media ?

Cheers

[nia]

Mark Harvey wrote

And the syslog / dump_tape confirm the 'LTO-5 media' being loaded into an LTO-4 drive..

That is really odd .. I can see that TSM mounts the correct tape on the correct drive which is on a different library even using a different device driver .. and the error that keeps flagging is  DumpBlock error on drive DRIVE1 (/dev/IBMtape0); already reached EOV once.


I will remove the LTO-4 library/drives and only test again with the IBM LTO-5 drive to isolate this ...

Thanks Mark.

-nia

[nia]

nia wrote

I am now getting this in TSM when using the IBM LTO-4 tape drives ..

That was a typo .. I meant to say LTO-5 .. Basically I had no problem with the IBM drives before the patch for the HP LTO-4

Do you still need me to retest ?


-nia

[Mark Harvey]

OK, with a patch by Sebastian and another minor fixup by myself, I think github branch is ready for another test - when you get a chance.

Cheers
Mark

[nia]

Ok, IBM LTO  working normal again ... But HP LTO still not behaving ..

ANR8302E I/O error on drive DRIVE1 (/dev/tsmscsi/mt0) with volume HPS001L4 (OP=WRITE, Error Number=0, CC=305, rc = 12, KEY=40,
ASC=00, ASCQ=00,
SENSE=F0.00.40.00.00.00.00.58.00.00.00.00.00.00.00.00.00.00.00.00.00.00.,
Description=Drive failure).  Refer to
Appendix C in the 'Messages' manual for recommended action.
ANR8792E Unrecoverable drive failures on drive DRIVE1 (/dev/tsmscsi/mt0); drive is now marked offline.
ANR0515I Process 20 closed volume HPS001L4.
ANR1227E BACKUP STGPOOL: Process 20 terminated - internal server error detected.

I will provide syslog  soon ...

-nia

[Mark Harvey]

Don't worry about syslog. I'll apply IBM LTO 

fix to HP LTO ;)

Sent from my iPhone

On 16/09/2011, at 8:01, "nia [via MHVTL - Linux Virtual Tape Library - Community Forums]"<[hidden email]> wrote:

Ok, IBM LTO  working normal again ... But HP LTO still not behaving ..

ANR8302E I/O error on drive DRIVE1 (/dev/tsmscsi/mt0) with volume HPS001L4 (OP=WRITE, Error Number=0, CC=305, rc = 12, KEY=40,
ASC=00, ASCQ=00,
SENSE=F0.00.40.00.00.00.00.58.00.00.00.00.00.00.00.00.00.00.00.00.00.00.,
Description=Drive failure).  Refer to
Appendix C in the 'Messages' manual for recommended action.
ANR8792E Unrecoverable drive failures on drive DRIVE1 (/dev/tsmscsi/mt0); drive is now marked offline.
ANR0515I Process 20 closed volume HPS001L4.
ANR1227E BACKUP STGPOOL: Process 20 terminated - internal server error detected.

I will provide syslog  soon ...

-nia

---

If you reply to this email, your message will be added to the discussion below:

http://mhvtl-linux-virtual-tape-library-community-forums.966029.n3.nabble.com/TSM-HP-Library-HP-LTO-Drive-I-O-error-on-drive-tp3327355p3340343.html

To start a new topic under MHVTL - Linux Virtual Tape Library - Community Forums, email [hidden email]
To unsubscribe from MHVTL - Linux Virtual Tape Library - Community Forums, click here.

[nia]

Mark Harvey wrote

Don't worry about syslog. I'll apply IBM LTO
fix to HP LTO ;)

Mark, fyi .. It is still not working ...Unrecoverable drive failures

ANR8302E I/O error on drive DRIVE1 (/dev/tsmscsi/mt0) with volume HPS001L4 (OP=WRITE, Error Number=0, CC=305, rc = 12, KEY=40, ASC=00, ASCQ=00, SENSE=F0.00.40.00.00.00.00.58.00.00.00.00.00.00.00.00.00.00.00.00.00.00.,

-nia

[Mark Harvey]

Thanks for testing.

Can you provide the syslog pls ;)

Cheers

[nia]

Here is a new test with syslog and dump_tape

TSM produces this error:

ANR8302E I/O error on drive DRIVE1 (/dev/tsmscsi/mt0) with volume HPS003L4 (OP=WRITE, Error Number=0, CC=305, rc = 12, KEY=40, ASC=00, ASCQ=00, SENSE=F0.00.40.00.00.00.00.58.00.00.00.00.00.00.00.00.00.00.00.00.00.00.,
Description=Drive failure).  Refer to Appendix C in the 'Messages' manual
for recommended action.
ANR8792E Unrecoverable drive failures on drive DRIVE1 (/dev/tsmscsi/mt0); drive is now marked offline.
ANR0514I Session 2 closed volume HPS003L4.
ANR2186W Transaction was aborted for session 2 for node TSM08 (Linux86).
ANR8468I LTO volume HPS003L4 dismounted from drive DRIVE1 (/dev/tsmscsi/mt0) in library MLS6000.

hp.lto.issue.zip
dump_tape_HPS003L4.zip

Thanks
-nia

[Mark Harvey]

HP personality module has been updated - yet again.

This time the initialisation is almost exactly the same as the IBM one.
I found an extra VPD page in the HP doco that I included. But this
should not make any difference.


On Sat, Sep 17, 2011 at 3:17 PM, nia [via MHVTL - Linux Virtual Tape
Library - Community Forums] <[hidden email]>
wrote:

> Here is a new test with syslog and dump_tape
>
> TSM produces this error:
>
> ANR8302E I/O error on drive DRIVE1 (/dev/tsmscsi/mt0) with volume HPS003L4
> (OP=WRITE, Error Number=0, CC=305, rc = 12, KEY=40, ASC=00, ASCQ=00,
> SENSE=F0.00.40.00.00.00.00.58.00.00.00.00.00.00.00.00.00.00.00.00.00.00.,
> Description=Drive failure).  Refer to Appendix C in the 'Messages' manual
> for recommended action.
> ANR8792E Unrecoverable drive failures on drive DRIVE1 (/dev/tsmscsi/mt0);
> drive is now marked offline.
> ANR0514I Session 2 closed volume HPS003L4.
> ANR2186W Transaction was aborted for session 2 for node TSM08 (Linux86).
> ANR8468I LTO volume HPS003L4 dismounted from drive DRIVE1 (/dev/tsmscsi/mt0)
> in library MLS6000.
>
> hp.lto.issue.zip
> dump_tape_HPS003L4.zip
>
> Thanks
> -nia
>
> ________________________________
> If you reply to this email, your message will be added to the discussion
> below:
> http://mhvtl-linux-virtual-tape-library-community-forums.966029.n3.nabble.com/TSM-HP-Library-HP-LTO-Drive-I-O-error-on-drive-tp3327355p3343709.html
> To start a new topic under MHVTL - Linux Virtual Tape Library - Community
> Forums, email [hidden email]
> To unsubscribe from MHVTL - Linux Virtual Tape Library - Community Forums,
> click here.

[nia]

Sorry, no luck yet .. same error:

ANR8302E I/O error on drive DRIVE2 (/dev/tsmscsi/mt1) with volume HPS002L4 (OP=WRITE, Error Number=0, CC=305, rc = 12, KEY=40, ASC=00, ASCQ=00, SENSE=F0.00.40.00.00.00.00.58.00.00.00.00.00.00.00.00.00.00.00.00.00.00.,

-nia