Library config issue with BakBone NetVault

[IKP]

Have been investigasting using BakBone with mhvtl.

So far have identified a couple of issues, some of which have been able to resolve others need further investigation or would help if Mark were to take a look at.

The first of the issues is if the device size is less than 2GigaByte in size the software will not write to it - this is just to document the resolution - create devices larger then 4GB

The second and more important, if the library is created with more than 4 tape slots the software wil not discover morte than 4. I have attached details from the test envirnment to help further investigation -

Files attached - System messages, screen shot and MHVTL config files

NVU-library-config.PNG
mhvtl.conf
messages
device.conf
library_contents.10

Sorry for the size of the messages file ... mhvtl verbosity set to 3

Hopefully the slot count issue can be resolved.



Will continue to test using 4 slots for now ..


IKP

[Mark Harvey]

Thanks for the information.

Re: Tape size < 4G
This sounds like a BakBone feature to me. Which doesn't actually surprise me that much.

I need to look at the messages file first to confirm. LTO media is expected to be 50G+, so I can imagine where backup s/w would query the tape capacity and reject it - if it had less than XX G free.

What sort of message does BakBone report when the media is < 2G ?

Can you create one (or more) media of <2G in size and attempt to get BakBone to use the media. Then provide the messages file for analysis :)


Re: messages file.
Can you advise what occurred during the time of the messages file ?

I can see it is from the system boot (which is good, as I see all the startup info too).

However, I can't see any media being loaded.

The only MODE SENSE information logged is from the vtllibrary (but this could be a 0.19-0 bug, I'll check the logging at my end).

I've realised a rather nasty bug with 0.19-0 where Albert & nia reported that it's not honoring the end-of-capacity.
Looking at the code, this is actually a rather bad bug where I'm not returning correct sense data.. (i.e. It's not a straight forward bug fix).
The 0.19-0 also is not performing MODE SELECT correctly (Already fixed). However, from your messages file, there are no MODE SELECTs logged.

I'm assuming this is a messages file when BakBone ran a "discover library with 6 drives".

Can I ask you to retry using the 0.18-12 release please.

Many thanks
Mark

[IKP]

Mark Harvey wrote

Thanks for the information.

Re: Tape size < 4G
This sounds like a BakBone feature to me. Which doesn't actually surprise me that much.

I need to look at the messages file first to confirm. LTO media is expected to be 50G+, so I can imagine where backup s/w would query the tape capacity and reject it - if it had less than XX G free.

What sort of message does BakBone report when the media is < 2G ?

Can you create one (or more) media of <2G in size and attempt to get BakBone to use the media. Then provide the messages file for analysis :)

Netvault is configured to expect an anmount of "free space" at the end of the tape, which by default is 2G. So drives less than 4G during the initial import and label get rejected.

Mark Harvey wrote

Re: messages file.
Can you advise what occurred during the time of the messages file ?

The message file contains

1) System boot.
2) NetVault startup -

"Nov 27 07:50:48 vtl19nvb NetVault[5003]: NetVault: Client: 'vtl19nvb' Class: 'System' Job: 'N/A' Warnlevel: 'Startup' Msg: '============================= LOG DAEMON STARTED ===================='"


3) NetVault library detect lauched - successfully finds the L700
4) L700 selected and the drives reported as 6* IBM LTO
5) The drives assigned   physical drive 1 == libraray drive 1  etc ... thru to drive 6 == drive 6
6) NetVault reports the drives and slots as per the originally attached screen image including a process where each of the tapes reported in the library are loaded, and a "BLANK" operation applied to the media header to then allow software functionionality to label in to appropriate media groups (pools) as appropriate

I can see it is from the system boot (which is good, as I see all the startup info too).

Mark Harvey wrote

However, I can't see any media being loaded.

As can be seen from the screen shot, the software has been thru' a media load/label/unload process.

Mark Harvey wrote

I'm assuming this is a messages file when BakBone ran a "discover library with 6 drives".

Can I ask you to retry using the 0.18-12 release please.

Many thanks
Mark

I will rebuild the environment later today with veriosn 0.18-12 and repeat the above process. Asa for the media size less than 4G unless you are really conserned about the software rejecting the media then this really just needs to be docuemnted.

Cheers
IKP

[IKP]

Adjunct to previous message ...

Have tested NetVault with 0.18-12mad got the same results.

Attached config and messages file -  this time messages include label of the 4 tapes in to a "Default" pool.

Message log -

System boot
Start NetVault service from command line
Detect
- Library   L700
- Devices  6 * LTO
- Assign Library drives to OS drives
- Label media in to a pool

As before media created at 4GB


mhvtl.conf
device.conf
library_contents.10
messages

Let me know if more needed

IKP

[Mark Harvey]

Received, thanks.

I won't be in a position to review the data in any detail until tomorrow at the earliest.

Cheers
Mark

[IKP]

Mark Harvey wrote

Received, thanks.

I won't be in a position to review the data in any detail until tomorrow at the earliest.

Cheers
Mark

Mark, anything else you need to review the data ?


IKP

[mxt]

1. < 2GB.
You can still use it. You need to change the default 2GB reserve space setting. Here is how to change it:
Device Management->Add(or Modify) Library->Configure->"Double Click on the tape drive"->Configuration->"Amount of media reserved at end of tape(MB)". I changed it to 100. That means I can still use 1.9 GB of the space.
2. See only 4 slots:
I have the same issue.

[Mark Harvey]

Sorry about the delay.

Some how or other, this one slipped thru the cracks.

We can see NetVault requesting a 'READ ELEMENT STATUS' but only allocates 252 bytes of storage...

Nov 28 08:24:57 vtl18 vtllibrary[4003]: CDB (543) b8 14 00 01 00 06 00 00 00 fc 00 00
Nov 28 08:24:57 vtl18 vtllibrary[4003]: smc_read_element_status: READ ELEMENT STATUS (543) **
Nov 28 08:24:57 vtl18 vtllibrary[4003]: smc_read_element_status:  Element type(4) => Data Transfer Elements
Nov 28 08:24:57 vtl18 vtllibrary[4003]: smc_read_element_status:   Starting Element Address: 1
Nov 28 08:24:57 vtl18 vtllibrary[4003]: smc_read_element_status:   Number of Elements      : 6
Nov 28 08:24:57 vtl18 vtllibrary[4003]: smc_read_element_status:   Allocation length       : 252
Nov 28 08:24:57 vtl18 vtllibrary[4003]: smc_read_element_status:   Device ID: No, voltag: Yes
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_page: Element type: Data Transfer (drive) Element, min: 1 num: 6
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_page: max_count: 6, max_bytes: 252
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_page: avail: 6, count: 4, space: 4 *cur_count: 0
                                                                                    ^^^^^^^^^^^^^^^^^^ Here we can see that we have 6 drives, but only enough room in the buffer for 4.

Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_status_page_hdr: Element Status Page Header: 04 80 00 34 00 00 00 d0
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_page: Slot: 1
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_descriptor: Slot location: 1
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_descriptor: DVCID: 0, VOLTAG: 1, Index: 12
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_descriptor: Returning 52 bytes
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_page: Slot: 2
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_descriptor: Slot location: 2
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_descriptor: DVCID: 0, VOLTAG: 1, Index: 12
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_descriptor: Returning 52 bytes
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_page: Slot: 3
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_descriptor: Slot location: 3
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_descriptor: DVCID: 0, VOLTAG: 1, Index: 12
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_descriptor: Returning 52 bytes
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_page: Slot: 4
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_descriptor: Slot location: 4
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_descriptor: DVCID: 0, VOLTAG: 1, Index: 12
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_descriptor: Returning 52 bytes
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_status_data_hdr: Building READ ELEMENT STATUS Header struct
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_status_data_hdr:  Starting slot: 1, number of configured slots: 4
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_status_data_hdr:  Element Status Data HEADER: 00 01 00 04 00 00 00 d8
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_status_data_hdr:  Decoded:
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_status_data_hdr:   First element Address    : 1
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_status_data_hdr:   Number elements reported : 4
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_status_data_hdr:   Total byte count         : 216

This I think is wrong. We should be returning '6' drives, but only valid data for 4 as that is all that will fit into the initiators allocated buffer.

Any chance of running "mtx -f /dev/sgXX status" against the library with 6 drives.
mtx should allocate a bigger buffer and it should return 6 drives.

[IKP]

mxt wrote

1. < 2GB.
You can still use it. You need to change the default 2GB reserve space setting. Here is how to change it:
Device Management->Add(or Modify) Library->Configure->"Double Click on the tape drive"->Configuration->"Amount of media reserved at end of tape(MB)". I changed it to 100. That means I can still use 1.9 GB of the space.

There is a reason why the software requires the 2GB of free space and reducing this can lead to issues when attempting to span the backup across the media boundary. As such the software should be left configured to require the 2GB free space at the end of the media and Mark’s software configured with tape size greater than 4GByte.

IKP

[IKP]

[IKP]

Mark Harvey wrote

Sorry about the delay.

Some how or other, this one slipped thru the cracks.

We can see NetVault requesting a 'READ ELEMENT STATUS' but only allocates 252 bytes of storage...

Nov 28 08:24:57 vtl18 vtllibrary[4003]: CDB (543) b8 14 00 01 00 06 00 00 00 fc 00 00
Nov 28 08:24:57 vtl18 vtllibrary[4003]: smc_read_element_status: READ ELEMENT STATUS (543) **
Nov 28 08:24:57 vtl18 vtllibrary[4003]: smc_read_element_status:  Element type(4) => Data Transfer Elements
Nov 28 08:24:57 vtl18 vtllibrary[4003]: smc_read_element_status:   Starting Element Address: 1
Nov 28 08:24:57 vtl18 vtllibrary[4003]: smc_read_element_status:   Number of Elements      : 6
Nov 28 08:24:57 vtl18 vtllibrary[4003]: smc_read_element_status:   Allocation length       : 252
Nov 28 08:24:57 vtl18 vtllibrary[4003]: smc_read_element_status:   Device ID: No, voltag: Yes
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_page: Element type: Data Transfer (drive) Element, min: 1 num: 6
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_page: max_count: 6, max_bytes: 252
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_page: avail: 6, count: 4, space: 4 *cur_count: 0
                                                                                    ^^^^^^^^^^^^^^^^^^ Here we can see that we have 6 drives, but only enough room in the buffer for 4.

Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_status_page_hdr: Element Status Page Header: 04 80 00 34 00 00 00 d0
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_page: Slot: 1
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_descriptor: Slot location: 1
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_descriptor: DVCID: 0, VOLTAG: 1, Index: 12
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_descriptor: Returning 52 bytes
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_page: Slot: 2
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_descriptor: Slot location: 2
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_descriptor: DVCID: 0, VOLTAG: 1, Index: 12
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_descriptor: Returning 52 bytes
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_page: Slot: 3
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_descriptor: Slot location: 3
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_descriptor: DVCID: 0, VOLTAG: 1, Index: 12
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_descriptor: Returning 52 bytes
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_page: Slot: 4
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_descriptor: Slot location: 4
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_descriptor: DVCID: 0, VOLTAG: 1, Index: 12
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_descriptor: Returning 52 bytes
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_status_data_hdr: Building READ ELEMENT STATUS Header struct
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_status_data_hdr:  Starting slot: 1, number of configured slots: 4
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_status_data_hdr:  Element Status Data HEADER: 00 01 00 04 00 00 00 d8
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_status_data_hdr:  Decoded:
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_status_data_hdr:   First element Address    : 1
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_status_data_hdr:   Number elements reported : 4
Nov 28 08:24:57 vtl18 vtllibrary[4003]: fill_element_status_data_hdr:   Total byte count         : 216

This I think is wrong. We should be returning '6' drives, but only valid data for 4 as that is all that will fit into the initiators allocated buffer.

Any chance of running "mtx -f /dev/sgXX status" against the library with 6 drives.
mtx should allocate a bigger buffer and it should return 6 drives.

Mark,

As requested -

[root@vtl18 ~]#
[root@vtl18 ~]# mtx -f /dev/sg8 status
  Storage Changer /dev/sg8:6 Drives, 21 Slots ( 5 Import/Export )
Data Transfer Element 0:Empty
Data Transfer Element 1:Empty
Data Transfer Element 2:Empty
Data Transfer Element 3:Empty
Data Transfer Element 4:Empty
Data Transfer Element 5:Empty
      Storage Element 1:Full :VolumeTag=E01001L4
      Storage Element 2:Full :VolumeTag=E01002L4
      Storage Element 3:Full :VolumeTag=E01003L4
      Storage Element 4:Full :VolumeTag=E01004L4
      Storage Element 5:Full :VolumeTag=E01005L4
      Storage Element 6:Full :VolumeTag=E01006L4
      Storage Element 7:Full :VolumeTag=E01007L4
      Storage Element 8:Full :VolumeTag=E01008L4
      Storage Element 9:Full :VolumeTag=E01009L4
      Storage Element 10:Full :VolumeTag=E01010L4
      Storage Element 11:Empty
      Storage Element 12:Empty
      Storage Element 13:Empty
      Storage Element 14:Empty
      Storage Element 15:Empty
      Storage Element 16:Full :VolumeTag=CLN101L4
      Storage Element 17 IMPORT/EXPORT:Empty
      Storage Element 18 IMPORT/EXPORT:Empty
      Storage Element 19 IMPORT/EXPORT:Empty
      Storage Element 20 IMPORT/EXPORT:Empty
      Storage Element 21 IMPORT/EXPORT:Empty
[root@vtl18 ~]#

Let me know id you need anything else ?

IKP

[Mark Harvey]

Looking at the READ ELEMENT STATUS code and trying to keep my sanity..

I'm in the wrong.. In several areas :(

I've got one issue resolved (return correct number of elements, if number of requested elements do not fit into initiator allocated buffer).
I'm now working on generating the correct response if a READ ELEMENT STATUS is sent with only 8 bytes of initiator allocated buffer (a valid combo according to the specs).

From the logs I can see BakBone sending both types of requests..

I hope to have something to test in the next day or two.

Cheers
Mark

[IKP]

Mark Harvey wrote

Looking at the READ ELEMENT STATUS code and trying to keep my sanity..

I'm in the wrong.. In several areas :(

I've got one issue resolved (return correct number of elements, if number of requested elements do not fit into initiator allocated buffer).
I'm now working on generating the correct response if a READ ELEMENT STATUS is sent with only 8 bytes of initiator allocated buffer (a valid combo according to the specs).

From the logs I can see BakBone sending both types of requests..

I hope to have something to test in the next day or two.

Cheers
Mark

Mark,

Now most of us are back at work, did you ever get the work you were planning to complete to allow the both forms of requests ? Therefore needing testing ?

Let me know


IKP

[Mark Harvey]

I do have something for you to test..

It is still not pretty..

One of these days, when I'm in the right frame of mind, I'll recode READ ELEMENT STATUS..
3rd time lucky.

Can you test this build please (and report back on success/failure).

mhvtl-2011-01-06-0.18.13beta1-git-0.18-13.tgz

Cheers
Mark

[aigmm]

Mark Harvey wrote

I do have something for you to test..

It is still not pretty..

One of these days, when I'm in the right frame of mind, I'll recode READ ELEMENT STATUS..
3rd time lucky.

Can you test this build please (and report back on success/failure).

mhvtl-2011-01-06-0.18.13beta1-git-0.18-13.tgz

Cheers
Mark

Up to 1.1-0 release NetVault backup still cannot work with more than 4 tape slots configured... the reason should be that, per SCSI spec, the returned data buffer shall not be adjusted to match the allocation length field. I temporarily hacked the returned buffer in smc_read_element_status() and now NVBU turns to work flawlessly.

"The allocation length is used to limit the maximum amount of variable length data (e.g., mode data, log data, diagnostic data) returned to an application client. If the information being transferred to the Data-In Buffer includes fields containing counts of the number of bytes in some or all of the data, then the contents of these fields shall not be altered to reflect the truncation, if any, that results from an insufficient ALLOCATION LENGTH value, unless the standard that describes the Data-In Buffer format states otherwise."

[Mark Harvey]

aigmm wrote

Up to 1.1-0 release NetVault backup still cannot work with more than 4 tape slots configured... the reason should be that, per SCSI spec, the returned data buffer shall not be adjusted to match the allocation length field. I temporarily hacked the returned buffer in smc_read_element_status() and now NVBU turns to work flawlessly.

"The allocation length is used to limit the maximum amount of variable length data (e.g., mode data, log data, diagnostic data) returned to an application client. If the information being transferred to the Data-In Buffer includes fields containing counts of the number of bytes in some or all of the data, then the contents of these fields shall not be altered to reflect the truncation, if any, that results from an insufficient ALLOCATION LENGTH value, unless the standard that describes the Data-In Buffer format states otherwise."

Any chance of posting your changes (or emailing me directly) so I can see what works for BakBone ?

Last time I looked at this, I thought I had it nailed... Oh well.

Cheers
Mark

[aigmm]

Mark Harvey wrote

Any chance of posting your changes (or emailing me directly) so I can see what works for BakBone ?

I only set max_bytes to SMC_BUF_SIZE in fill_element_page() --- provided that the actual buffer used is always SMC_BUF_SIZE long it won't cause a buffer overflow...  though this is a really bad fix :(

diff --git a/usr/smc.c b/usr/smc.c
index 720620d..1b064bd 100644
--- a/usr/smc.c
+++ b/usr/smc.c
@@ -743,7 +743,7 @@ static uint32_t fill_element_page(struct scsi_cmd *cmd, uint16_t start,
        uint32_t max_bytes;
 
        max_count = get_unaligned_be16(&cdb[4]);
-       max_bytes = 0xffffff & get_unaligned_be32(&cdb[6]);
+       max_bytes = 0x100000; /* SMC_BUF_SIZE */
 
        smc_p = cmd->lu->lu_private;

[Mark Harvey]

I see the problem now..

A request with 8 byte buffer (just enough to fit the header) returns the correct length needed for ALL elements.
# sg_raw -r 1k /dev/sg6 b8 10 00 00 00 ff 00 00 00 08 00 00
SCSI Status: Good

Sense Information:
sense buffer empty

Received 8 bytes of data:
 00     00 01 00 30 00 00 09 e0

vs:
# sg_raw -r 1k /dev/sg6 b8 10 00 00 00 ff 00 00 01 08 00 00
SCSI Status: Good

Sense Information:
sense buffer empty

Received 224 bytes of data:
 00     00 01 00 30 00 00 00 d8
                                ^^^^^^ Incorrect length of all data here.


However, if the allocated buffer is > 8 bytes but < required size to fit all the data, the returned 'correct length' is incorrectly calculated. i.e. In the above setup, '00 09 e0' should be returned no matter what size the initiator indicated it's buffer is.

I'll work on a fix shortly.

Cheers & many thanks for your persistence with this bug.
Mark

[Mark Harvey]

Any chance of testing this patch ?

0001-READ-ELEMENT-STATUS-Fix-header-data-count.patch

Cheers
Mark

[ade01]

Mark,
I tested this new patch. The last slot still has problem. The barcode can not be read correctly. There also be another issue. All the tape drives are not working. Please check the messages.
qbadl01


messages.zipNV.log