mhvtl 1.1.0-8f01a32 with Debian 6.0 - 2.6.39-2

[nia]

Hi Mark,

I am having some issues with mhvtl 1.1.0-8f01a32 running on Debian 6.0 - 2.6.39-2. Since mhvtl upgrade, mhvtl starts up with no errors yet no mhvtl devices shows up when doing lsscsi. When trying to stop mhvtl, it says it not running, but then when trying to start it  says it is already running .. I can see vtlxxx processes running but will not kill if I do /etc/init.d/mhvtl shutdown and command hangs .. Not sure what is going on yet.

mhvtl ~ # cat /proc/version
Linux version 2.6.39-2-amd64 (Debian 2.6.39-2) ([e-mail]) (gcc version 4.4.6 (Debian 4.4.6-3) ) #1 SMP Wed Jun 8 11:01:04 UTC 2011

-nia

[Mark Harvey]

Anything interesting in the syslog ?

Try a "vtlcmd <index number> exit" to see if the process will terminate.

Can you confirm the binaries are 64bit and not 32bit binaries ?

I've seen similar symptoms when a 64bit kernel with 32bit daemons are used..

Cheers

[nia]

It register this when started :

mhvtl: version 0.18.11 [20110801-0], opts=0x0

Then after a while .. daemons crashes and more messages show up in the syslog ..

Nov 22 23:58:45 mhvtl kernel: [  402.852036] scsi 3:1:0:1: Device offlined - not ready after error recovery
Nov 22 23:59:46 mhvtl kernel: [  463.844043] scsi 3:1:0:2: Device offlined - not ready after error recovery

Nov 23 00:00:02 mhvtl kernel: [  480.325315] vtltape         D ffff88003bca15a0     0  2819      1 0x00000000
Nov 23 00:00:02 mhvtl kernel: [  480.325322]  ffff88003bca15a0 0000000000000086 0000000000000028 ffff88003fa50040
Nov 23 00:00:02 mhvtl kernel: [  480.325329]  0000000000013b40 ffff88003bf91fd8 ffff88003bf91fd8 0000000000013b40
Nov 23 00:00:02 mhvtl kernel: [  480.325334]  ffff88003bca15a0 ffff88003bf90010 ffff88003d3fb180 0000000000000286
Nov 23 00:00:02 mhvtl kernel: [  480.325339] Call Trace:
Nov 23 00:00:02 mhvtl kernel: [  480.325352]  [<ffffffff81331e3d>] ? __mutex_lock_common+0x127/0x193
Nov 23 00:00:02 mhvtl kernel: [  480.325357]  [<ffffffff81331f75>] ? mutex_lock+0x1a/0x33
Nov 23 00:00:02 mhvtl kernel: [  480.325389]  [<ffffffffa002af4b>] ? __scsi_add_device+0x84/0xfc [scsi_mod]
Nov 23 00:00:02 mhvtl kernel: [  480.325395]  [<ffffffffa01ef3b9>] ? vtl_add_lu_action+0x2ae/0x2ec [mhvtl]
Nov 23 00:00:02 mhvtl kernel: [  480.325402]  [<ffffffff810b7677>] ? filemap_fault+0x1a3/0x34e
Nov 23 00:00:02 mhvtl kernel: [  480.325408]  [<ffffffff810d23c8>] ? __do_fault+0x3f9/0x430
Nov 23 00:00:02 mhvtl kernel: [  480.325413]  [<ffffffff810bc182>] ? __alloc_pages_nodemask+0x13d/0x7c4
Nov 23 00:00:02 mhvtl kernel: [  480.325418]  [<ffffffff8110c732>] ? dput+0x29/0xe8
Nov 23 00:00:02 mhvtl kernel: [  480.325424]  [<ffffffff810e6230>] ? alloc_pages_current+0xa7/0xca
Nov 23 00:00:02 mhvtl kernel: [  480.325430]  [<ffffffff8114ed16>] ? sysfs_write_file+0xe0/0x11c
Nov 23 00:00:02 mhvtl kernel: [  480.325436]  [<ffffffff810fbfe8>] ? vfs_write+0xa4/0xff
Nov 23 00:00:02 mhvtl kernel: [  480.325440]  [<ffffffff810fc0f9>] ? sys_write+0x45/0x6e
Nov 23 00:00:02 mhvtl kernel: [  480.325445]  [<ffffffff81338dd2>] ? system_call_fastpath+0x16/0x1b

[Mark Harvey]

Ouch..

This is not good.
Any kernel oops is a bad thing..

A. You need to reboot the box. Prefer to not start user daemons (vtltape / vtllibrary) at boot time.
b. Start one daemon in 'debug' mode:
e.g.
 vtltape -q 11 -d | tee /tmp/vtltape-1.out
 vtltape -q 12 -d | tee /tmp/vtltape-2.out
(etc)

See if you can re-produce the crash.

Forward the syslog + /tmp/vtltape* files for analysis or if possible, describe what actually triggered the fault.

Cheers & good luck.

[Mark Harvey]

FWIW: The kernel module hasn't had any changes since before 1.0-1 release in September 2011.

Not that those changes would have caused this sort of panic.

Cheers
Mark

[nia]

Hi Mark, everything was running fine until I pulled the latest git ..

Here is a debug output from one device : Full log is attached. Nothing registered in the syslog:

This error shows at the end of the run.

vtltape: main: vtltape: version 1.1.0-git-8f01a32, verbose log: 9
vtltape: readline: 

vtltape: readline: VERSION: 5
vtltape: readline: 
.
.
.
.
vtltape: main: starting...
vtltape: init_default_ssc: +++ Trace +++
.
.
.
vtltape: chrdev_get_major: Can't open /sys/bus/pseudo/drivers/mhvtl/major: No such file or directory
vtltape: chrdev_create: ** Incorrect version of kernel module loaded **
vtltape: main: Unable to create device node mhvtl31

vtltape-1.out

-nia

[Mark Harvey]

Does that 'major' file exist under the /sys ?

If so, can you cat the contents and let me know what it is.

I'll look at the logs in more detail shortly.

Cheers (from India)

Sent from my iPad

On Nov 24, 2011, at 8:11, "nia [via MHVTL - Linux Virtual Tape Library - Community Forums]"<[hidden email]> wrote:

Hi Mark, everything was running fine until I pulled the latest git ..

Here is a debug output from one device : Full log is attached. Nothing registered in the syslog:

This error shows at the end of the run.

vtltape: main: vtltape: version 1.1.0-git-8f01a32, verbose log: 9
vtltape: readline: 

vtltape: readline: VERSION: 5
vtltape: readline: 
.
.
.
.
vtltape: main: starting...
vtltape: init_default_ssc: +++ Trace +++
.
.
.
vtltape: chrdev_get_major: Can't open /sys/bus/pseudo/drivers/mhvtl/major: No such file or directory
vtltape: chrdev_create: ** Incorrect version of kernel module loaded **
vtltape: main: Unable to create device node mhvtl31

vtltape-1.out

-nia

---

If you reply to this email, your message will be added to the discussion below:

http://mhvtl-linux-virtual-tape-library-community-forums.966029.n3.nabble.com/mhvtl-1-1-0-8f01a32-with-Debian-6-0-2-6-39-2-tp3529746p3532691.html

To start a new topic under MHVTL - Linux Virtual Tape Library - Community Forums, email [hidden email]
To unsubscribe from MHVTL - Linux Virtual Tape Library - Community Forums, click here.
NAML

[Mark Harvey]

Somethining is not happy.

Device 31 is a T10000A, yet default personaliy being loaded, not the T10000 ?

Did you get a kernel oops at this point ?

Perhaps the kernel module oops and hence no /sys/....../major file.

Sent from my iPad

On Nov 24, 2011, at 8:11, "nia [via MHVTL - Linux Virtual Tape Library - Community Forums]"<[hidden email]> wrote:

Hi Mark, everything was running fine until I pulled the latest git ..

Here is a debug output from one device : Full log is attached. Nothing registered in the syslog:

This error shows at the end of the run.

vtltape: main: vtltape: version 1.1.0-git-8f01a32, verbose log: 9
vtltape: readline: 

vtltape: readline: VERSION: 5
vtltape: readline: 
.
.
.
.
vtltape: main: starting...
vtltape: init_default_ssc: +++ Trace +++
.
.
.
vtltape: chrdev_get_major: Can't open /sys/bus/pseudo/drivers/mhvtl/major: No such file or directory
vtltape: chrdev_create: ** Incorrect version of kernel module loaded **
vtltape: main: Unable to create device node mhvtl31

vtltape-1.out

-nia

---

If you reply to this email, your message will be added to the discussion below:

http://mhvtl-linux-virtual-tape-library-community-forums.966029.n3.nabble.com/mhvtl-1-1-0-8f01a32-with-Debian-6-0-2-6-39-2-tp3529746p3532691.html

To start a new topic under MHVTL - Linux Virtual Tape Library - Community Forums, email [hidden email]
To unsubscribe from MHVTL - Linux Virtual Tape Library - Community Forums, click here.
NAML

[nia]

Mark Harvey wrote

Does that 'major' file exist under the /sys ?

If so, can you cat the contents and let me know what it is.

not when doing vtltape -q 31  -d | tee /tmp/vtltape-1.out, but if I start mhvtl via /etc/init.d/mhvtl start then file exist

~ # cat /sys/bus/pseudo/drivers/mhvtl/major
251

[Mark Harvey]

Thanks for the clarification.

I may not be in a position to test/trouble-shoot any further until I get home next week..

I'll look back at the change log and see if there is anything obviously different.

p.s. Did you re-compile the kernel module with the latest git or use the older kernel module (you stated that the problems only started with latest git code).

Just interested to see if the problem occurs if you go back to the kernel module shipped with the 1.0 release.. Or if it is the changes in user-space daemons that are triggering a fault in the kernel module.

Any chance of tested a slightly earlier version (either 1.0 or 1.1 release) and report if it works on this kernel 2.6.39-2 version OK.

If nothing else, I'll then be able to identify what's changed between working & non-working versions.

Cheers
Mark

[nia]

I went back to release : markh794-mhvtl-97d9001 and things are normal again.

I will try to upgrade one commit at time to see which one in between caused this start.

[nia]

Mark Harvey wrote

p.s. Did you re-compile the kernel module with the latest git or use the older kernel module (you stated that the problems only started with latest git code).

Yes, I always do re-compile the kernel module with the latest git I pull down.

[Mark Harvey]

nia wrote

I went back to release : markh794-mhvtl-97d9001 and things are normal again.

I will try to upgrade one commit at time to see which one in between caused this start.

That would be great..

Major changes which could have upset things include:
  ---------------
commit 49639c28e0d2c3599d07866bddb215265e7df69b
Author: Mark Harvey <[e-mail]>
Date:   Tue Oct 18 10:39:38 2011 +1100

    Track fifo usage & clean up resources when finished
  ---------------

The other commits should not have effected operation as they are simply bug fixes.
But only testing will tell :)

[nia]

Tested up to fc757e3 and was fine so far ...

[Mark Harvey]

can you try with the 'fifo' entries removed from device.conf ?

I suspect the problem deals with failing to create the IPC related to that patch.

Removing the fifo entries should result in this not being called.

[nia]

Just applied commit 49639c28e0d2c3599d07866bddb215265e7df69b and now having issues..
Removing the fifo entries from device.conf does not help.

-nia

[Mark Harvey]

Ok, thanks.

Back to the drawing board..

Can you provide syslog at VERBOSE = 3

during startup.. It might provide something I've missed.

Sent from my iPad

On Nov 24, 2011, at 22:27, "nia [via MHVTL - Linux Virtual Tape Library - Community Forums]"<[hidden email]> wrote:

Just applied commit 49639c28e0d2c3599d07866bddb215265e7df69b and now having issues..
Removing the fifo entries from device.conf does not help.

-nia

---

If you reply to this email, your message will be added to the discussion below:

http://mhvtl-linux-virtual-tape-library-community-forums.966029.n3.nabble.com/mhvtl-1-1-0-8f01a32-with-Debian-6-0-2-6-39-2-tp3529746p3534126.html

To start a new topic under MHVTL - Linux Virtual Tape Library - Community Forums, email [hidden email]
To unsubscribe from MHVTL - Linux Virtual Tape Library - Community Forums, click here.
NAML

[nia]

mhvtl ~ # ps -ef | grep vtl
root      1366     1  0 14:01 pts/0    00:00:00 vtltape -q 11 -v -v -v
root      1369     1  0 14:01 pts/0    00:00:00 vtltape -q 12 -v -v -v
root      1372     1  0 14:01 pts/0    00:00:00 vtltape -q 13 -v -v -v
root      1375     1  0 14:01 pts/0    00:00:00 vtltape -q 14 -v -v -v
root      1378     1  0 14:01 pts/0    00:00:00 vtltape -q 31 -v -v -v
root      1381     1  0 14:01 pts/0    00:00:00 vtltape -q 32 -v -v -v
root      1384     1  0 14:01 pts/0    00:00:00 vtltape -q 33 -v -v -v
root      1387     1  0 14:01 pts/0    00:00:00 vtltape -q 34 -v -v -v
root      1391     1  0 14:01 pts/0    00:00:00 vtllibrary -q 10 -v -v -v
root      1394     1  0 14:01 pts/0    00:00:00 vtllibrary -q 30 -v -v -v

Nov 24 14:01:42 mhvtl kernel: [   56.054389] scsi3 : mhvtl: version 0.18.11 [20110801-0], opts=0x0
Nov 24 14:01:42 mhvtl rsyslogd-2177: imuxsock begins to drop messages from pid 1365 due to rate-limiting
Nov 24 14:01:42 mhvtl rsyslogd-2177: imuxsock begins to drop messages from pid 1368 due to rate-limiting
Nov 24 14:01:42 mhvtl rsyslogd-2177: imuxsock begins to drop messages from pid 1371 due to rate-limiting
Nov 24 14:01:42 mhvtl rsyslogd-2177: imuxsock begins to drop messages from pid 1374 due to rate-limiting
Nov 24 14:01:42 mhvtl rsyslogd-2177: imuxsock begins to drop messages from pid 1377 due to rate-limiting
Nov 24 14:01:42 mhvtl rsyslogd-2177: imuxsock begins to drop messages from pid 1380 due to rate-limiting
Nov 24 14:01:42 mhvtl rsyslogd-2177: imuxsock begins to drop messages from pid 1383 due to rate-limiting
Nov 24 14:01:42 mhvtl rsyslogd-2177: imuxsock begins to drop messages from pid 1386 due to rate-limiting
Nov 24 14:01:42 mhvtl rsyslogd-2177: imuxsock begins to drop messages from pid 1390 due to rate-limiting
Nov 24 14:01:42 mhvtl rsyslogd-2177: imuxsock begins to drop messages from pid 1393 due to rate-limiting

Nov 24 14:03:03 mhvtl kernel: [  136.904042] scsi 3:0:1:0: Device offlined - not ready after error recovery

[nia]

I turned Rate-limiting to off in /etc/rsyslog.conf .. so I don't see imuxsock begins to drop messages from .. messages

Here is more stuff that shows in syslog also.

Nov 24 14:22:40 mhvtl kernel: [   64.174279] scsi3 : mhvtl: version 0.18.11 [20110801-0], opts=0x0
Nov 24 14:24:02 mhvtl kernel: [  145.864038] scsi 3:0:1:0: Device offlined - not ready after error recovery
Nov 24 14:25:23 mhvtl kernel: [  226.888048] scsi 3:0:2:0: Device offlined - not ready after error recovery
Nov 24 14:25:36 mhvtl kernel: [  240.329330] vtltape         D ffff88003d3da720     0  1425      1 0x00000000
Nov 24 14:25:36 mhvtl kernel: [  240.329338]  ffff88003d3da720 0000000000000082 0000000000000028 ffff8800370c6b20
Nov 24 14:25:36 mhvtl kernel: [  240.329344]  0000000000013b40 ffff880037455fd8 ffff880037455fd8 0000000000013b40
Nov 24 14:25:36 mhvtl kernel: [  240.329349]  ffff88003d3da720 ffff880037454010 ffff88003d3c4428 ffffffff811a36b1
Nov 24 14:25:36 mhvtl kernel: [  240.329354] Call Trace:
Nov 24 14:25:36 mhvtl kernel: [  240.329368]  [<ffffffff811a36b1>] ? kobject_set_name_vargs+0x51/0x58
Nov 24 14:25:36 mhvtl kernel: [  240.329375]  [<ffffffff81331e3d>] ? __mutex_lock_common+0x127/0x193
Nov 24 14:25:36 mhvtl kernel: [  240.329379]  [<ffffffff81331f75>] ? mutex_lock+0x1a/0x33
Nov 24 14:25:36 mhvtl kernel: [  240.329411]  [<ffffffffa00d0f4b>] ? __scsi_add_device+0x84/0xfc [scsi_mod]
Nov 24 14:25:36 mhvtl kernel: [  240.329417]  [<ffffffffa01fb3b9>] ? vtl_add_lu_action+0x2ae/0x2ec [mhvtl]
Nov 24 14:25:36 mhvtl kernel: [  240.329424]  [<ffffffff810b7677>] ? filemap_fault+0x1a3/0x34e
Nov 24 14:25:36 mhvtl kernel: [  240.329432]  [<ffffffff810d23c8>] ? __do_fault+0x3f9/0x430
Nov 24 14:25:36 mhvtl kernel: [  240.329437]  [<ffffffff810bc182>] ? __alloc_pages_nodemask+0x13d/0x7c4
Nov 24 14:25:36 mhvtl kernel: [  240.329442]  [<ffffffff8110c732>] ? dput+0x29/0xe8
Nov 24 14:25:36 mhvtl kernel: [  240.329449]  [<ffffffff810e6230>] ? alloc_pages_current+0xa7/0xca
Nov 24 14:25:36 mhvtl kernel: [  240.329455]  [<ffffffff8114ed16>] ? sysfs_write_file+0xe0/0x11c
Nov 24 14:25:36 mhvtl kernel: [  240.329460]  [<ffffffff810fbfe8>] ? vfs_write+0xa4/0xff
Nov 24 14:25:36 mhvtl kernel: [  240.329464]  [<ffffffff810fc0f9>] ? sys_write+0x45/0x6e
Nov 24 14:25:36 mhvtl kernel: [  240.329469]  [<ffffffff81338dd2>] ? system_call_fastpath+0x16/0x1b
Nov 24 14:25:36 mhvtl kernel: [  240.330489] vtltape         D ffff88003beaafc0     0  1428      1 0x00000000
Nov 24 14:25:36 mhvtl kernel: [  240.330495]  ffff88003beaafc0 0000000000000082 0000000000000028 ffff88003cfc81c0
Nov 24 14:25:36 mhvtl kernel: [  240.330500]  0000000000013b40 ffff88003dfbffd8 ffff88003dfbffd8 0000000000013b40
Nov 24 14:25:36 mhvtl kernel: [  240.330504]  ffff88003beaafc0 ffff88003dfbe010 ffff88003d3c4828 ffffffff811a36b1
Nov 24 14:25:36 mhvtl kernel: [  240.330509] Call Trace:
Nov 24 14:25:36 mhvtl kernel: [  240.330513]  [<ffffffff811a36b1>] ? kobject_set_name_vargs+0x51/0x58
Nov 24 14:25:36 mhvtl kernel: [  240.330517]  [<ffffffff81331e3d>] ? __mutex_lock_common+0x127/0x193
Nov 24 14:25:36 mhvtl kernel: [  240.330521]  [<ffffffff81331f75>] ? mutex_lock+0x1a/0x33
Nov 24 14:25:36 mhvtl kernel: [  240.330531]  [<ffffffffa00d0f4b>] ? __scsi_add_device+0x84/0xfc [scsi_mod]
Nov 24 14:25:36 mhvtl kernel: [  240.330536]  [<ffffffffa01fb3b9>] ? vtl_add_lu_action+0x2ae/0x2ec [mhvtl]
Nov 24 14:25:36 mhvtl kernel: [  240.330540]  [<ffffffff810b7677>] ? filemap_fault+0x1a3/0x34e
Nov 24 14:25:36 mhvtl kernel: [  240.330544]  [<ffffffff810d23c8>] ? __do_fault+0x3f9/0x430
Nov 24 14:25:36 mhvtl kernel: [  240.330549]  [<ffffffff810bc182>] ? __alloc_pages_nodemask+0x13d/0x7c4
Nov 24 14:25:36 mhvtl kernel: [  240.330552]  [<ffffffff8110c732>] ? dput+0x29/0xe8
Nov 24 14:25:36 mhvtl kernel: [  240.330557]  [<ffffffff810e6230>] ? alloc_pages_current+0xa7/0xca
Nov 24 14:25:36 mhvtl kernel: [  240.330562]  [<ffffffff8114ed16>] ? sysfs_write_file+0xe0/0x11c
Nov 24 14:25:36 mhvtl kernel: [  240.330566]  [<ffffffff810fbfe8>] ? vfs_write+0xa4/0xff
Nov 24 14:25:36 mhvtl kernel: [  240.330569]  [<ffffffff810fc0f9>] ? sys_write+0x45/0x6e
Nov 24 14:25:36 mhvtl kernel: [  240.330573]  [<ffffffff81338dd2>] ? system_call_fastpath+0x16/0x1b
Nov 24 14:25:36 mhvtl kernel: [  240.331689] vtltape         D ffff88003beaa8e0     0  1431      1 0x00000000
Nov 24 14:25:36 mhvtl kernel: [  240.331694]  ffff88003beaa8e0 0000000000000086 0000000000000028 ffff88003cfc88a0
Nov 24 14:25:36 mhvtl kernel: [  240.331699]  0000000000013b40 ffff88003ce79fd8 ffff88003ce79fd8 0000000000013b40
Nov 24 14:25:36 mhvtl kernel: [  240.331704]  ffff88003beaa8e0 ffff88003ce78010 ffff88003d3c4c28 ffffffff811a36b1
Nov 24 14:25:36 mhvtl kernel: [  240.331708] Call Trace:
Nov 24 14:25:36 mhvtl kernel: [  240.331713]  [<ffffffff811a36b1>] ? kobject_set_name_vargs+0x51/0x58
Nov 24 14:25:36 mhvtl kernel: [  240.331717]  [<ffffffff81331e3d>] ? __mutex_lock_common+0x127/0x193
Nov 24 14:25:36 mhvtl kernel: [  240.331720]  [<ffffffff81331f75>] ? mutex_lock+0x1a/0x33
Nov 24 14:25:36 mhvtl kernel: [  240.331729]  [<ffffffffa00d0f4b>] ? __scsi_add_device+0x84/0xfc [scsi_mod]
Nov 24 14:25:36 mhvtl kernel: [  240.331734]  [<ffffffffa01fb3b9>] ? vtl_add_lu_action+0x2ae/0x2ec [mhvtl]
Nov 24 14:25:36 mhvtl kernel: [  240.331738]  [<ffffffff810b7677>] ? filemap_fault+0x1a3/0x34e
Nov 24 14:25:36 mhvtl kernel: [  240.331742]  [<ffffffff810d23c8>] ? __do_fault+0x3f9/0x430
Nov 24 14:25:36 mhvtl kernel: [  240.331747]  [<ffffffff810bc182>] ? __alloc_pages_nodemask+0x13d/0x7c4
Nov 24 14:25:36 mhvtl kernel: [  240.331751]  [<ffffffff8110c732>] ? dput+0x29/0xe8
Nov 24 14:25:36 mhvtl kernel: [  240.331755]  [<ffffffff810e6230>] ? alloc_pages_current+0xa7/0xca
Nov 24 14:25:36 mhvtl kernel: [  240.331760]  [<ffffffff8114ed16>] ? sysfs_write_file+0xe0/0x11c
Nov 24 14:25:36 mhvtl kernel: [  240.331764]  [<ffffffff810fbfe8>] ? vfs_write+0xa4/0xff
Nov 24 14:25:36 mhvtl kernel: [  240.331768]  [<ffffffff810fc0f9>] ? sys_write+0x45/0x6e
Nov 24 14:25:36 mhvtl kernel: [  240.331771]  [<ffffffff81338dd2>] ? system_call_fastpath+0x16/0x1b
Nov 24 14:25:36 mhvtl kernel: [  240.332837] vtltape         D ffff8800370c6b20     0  1434      1 0x00000000
Nov 24 14:25:36 mhvtl kernel: [  240.332842]  ffff8800370c6b20 0000000000000082 0000000000000028 ffff88003beb8240
Nov 24 14:25:36 mhvtl kernel: [  240.332847]  0000000000013b40 ffff880037403fd8 ffff880037403fd8 0000000000013b40
Nov 24 14:25:36 mhvtl kernel: [  240.332852]  ffff8800370c6b20 ffff880037402010 ffff88003d36d428 ffffffff811a36b1
Nov 24 14:25:36 mhvtl kernel: [  240.332856] Call Trace:
Nov 24 14:25:36 mhvtl kernel: [  240.332861]  [<ffffffff811a36b1>] ? kobject_set_name_vargs+0x51/0x58
Nov 24 14:25:36 mhvtl kernel: [  240.332865]  [<ffffffff81331e3d>] ? __mutex_lock_common+0x127/0x193
Nov 24 14:25:36 mhvtl kernel: [  240.332869]  [<ffffffff81331f75>] ? mutex_lock+0x1a/0x33
Nov 24 14:25:36 mhvtl kernel: [  240.332877]  [<ffffffffa00d0f4b>] ? __scsi_add_device+0x84/0xfc [scsi_mod]
Nov 24 14:25:36 mhvtl kernel: [  240.332882]  [<ffffffffa01fb3b9>] ? vtl_add_lu_action+0x2ae/0x2ec [mhvtl]
Nov 24 14:25:36 mhvtl kernel: [  240.332886]  [<ffffffff810b7677>] ? filemap_fault+0x1a3/0x34e
Nov 24 14:25:36 mhvtl kernel: [  240.332891]  [<ffffffff810d23c8>] ? __do_fault+0x3f9/0x430
Nov 24 14:25:36 mhvtl kernel: [  240.332895]  [<ffffffff810bc182>] ? __alloc_pages_nodemask+0x13d/0x7c4
Nov 24 14:25:36 mhvtl kernel: [  240.332899]  [<ffffffff8110c732>] ? dput+0x29/0xe8
Nov 24 14:25:36 mhvtl kernel: [  240.332904]  [<ffffffff810e6230>] ? alloc_pages_current+0xa7/0xca
Nov 24 14:25:36 mhvtl kernel: [  240.332909]  [<ffffffff8114ed16>] ? sysfs_write_file+0xe0/0x11c
Nov 24 14:25:36 mhvtl kernel: [  240.332913]  [<ffffffff810fbfe8>] ? vfs_write+0xa4/0xff
Nov 24 14:25:36 mhvtl kernel: [  240.332917]  [<ffffffff810fc0f9>] ? sys_write+0x45/0x6e
Nov 24 14:25:36 mhvtl kernel: [  240.332920]  [<ffffffff81338dd2>] ? system_call_fastpath+0x16/0x1b
Nov 24 14:25:36 mhvtl kernel: [  240.333971] vtltape         D ffff88003cfc81c0     0  1437      1 0x00000000
Nov 24 14:25:36 mhvtl kernel: [  240.333976]  ffff88003cfc81c0 0000000000000082 0000000000000028 ffff88003beaa200
Nov 24 14:25:36 mhvtl kernel: [  240.333981]  0000000000013b40 ffff8800376bbfd8 ffff8800376bbfd8 0000000000013b40
Nov 24 14:25:36 mhvtl kernel: [  240.333986]  ffff88003cfc81c0 ffff8800376ba010 ffff8800377e5428 ffffffff811a36b1
Nov 24 14:25:36 mhvtl kernel: [  240.333990] Call Trace:
Nov 24 14:25:36 mhvtl kernel: [  240.333994]  [<ffffffff811a36b1>] ? kobject_set_name_vargs+0x51/0x58
Nov 24 14:25:36 mhvtl kernel: [  240.333998]  [<ffffffff81331e3d>] ? __mutex_lock_common+0x127/0x193
Nov 24 14:25:36 mhvtl kernel: [  240.334002]  [<ffffffff81331f75>] ? mutex_lock+0x1a/0x33
Nov 24 14:25:36 mhvtl kernel: [  240.334011]  [<ffffffffa00d0f4b>] ? __scsi_add_device+0x84/0xfc [scsi_mod]
Nov 24 14:25:36 mhvtl kernel: [  240.334016]  [<ffffffffa01fb3b9>] ? vtl_add_lu_action+0x2ae/0x2ec [mhvtl]
Nov 24 14:25:36 mhvtl kernel: [  240.334020]  [<ffffffff810b7677>] ? filemap_fault+0x1a3/0x34e
Nov 24 14:25:36 mhvtl kernel: [  240.334024]  [<ffffffff810d23c8>] ? __do_fault+0x3f9/0x430
Nov 24 14:25:36 mhvtl kernel: [  240.334029]  [<ffffffff810bc182>] ? __alloc_pages_nodemask+0x13d/0x7c4
Nov 24 14:25:36 mhvtl kernel: [  240.334032]  [<ffffffff8110c732>] ? dput+0x29/0xe8
Nov 24 14:25:36 mhvtl kernel: [  240.334037]  [<ffffffff810e6230>] ? alloc_pages_current+0xa7/0xca
Nov 24 14:25:36 mhvtl kernel: [  240.334042]  [<ffffffff8114ed16>] ? sysfs_write_file+0xe0/0x11c
Nov 24 14:25:36 mhvtl kernel: [  240.334046]  [<ffffffff810fbfe8>] ? vfs_write+0xa4/0xff
Nov 24 14:25:36 mhvtl kernel: [  240.334050]  [<ffffffff810fc0f9>] ? sys_write+0x45/0x6e
Nov 24 14:25:36 mhvtl kernel: [  240.334053]  [<ffffffff81338dd2>] ? system_call_fastpath+0x16/0x1b
Nov 24 14:25:36 mhvtl kernel: [  240.336118] vtllibrary      D ffff88003beaa200     0  1441      1 0x00000000
Nov 24 14:25:36 mhvtl kernel: [  240.336127]  ffff88003beaa200 0000000000000086 0000000000000028 ffff88003beb8240
Nov 24 14:25:36 mhvtl kernel: [  240.336132]  0000000000013b40 ffff88003cc57fd8 ffff88003cc57fd8 0000000000013b40
Nov 24 14:25:36 mhvtl kernel: [  240.336136]  ffff88003beaa200 ffff88003cc56010 ffff88003beb4428 ffffffff811a36b1
Nov 24 14:25:36 mhvtl kernel: [  240.336141] Call Trace:
Nov 24 14:25:36 mhvtl kernel: [  240.336148]  [<ffffffff811a36b1>] ? kobject_set_name_vargs+0x51/0x58
Nov 24 14:25:36 mhvtl kernel: [  240.336153]  [<ffffffff81331e3d>] ? __mutex_lock_common+0x127/0x193
Nov 24 14:25:36 mhvtl kernel: [  240.336156]  [<ffffffff81331f75>] ? mutex_lock+0x1a/0x33
Nov 24 14:25:36 mhvtl kernel: [  240.336172]  [<ffffffffa00d0f4b>] ? __scsi_add_device+0x84/0xfc [scsi_mod]
Nov 24 14:25:36 mhvtl kernel: [  240.336178]  [<ffffffffa01fb3b9>] ? vtl_add_lu_action+0x2ae/0x2ec [mhvtl]
Nov 24 14:25:36 mhvtl kernel: [  240.336183]  [<ffffffff810b7677>] ? filemap_fault+0x1a3/0x34e
Nov 24 14:25:36 mhvtl kernel: [  240.336188]  [<ffffffff810d23c8>] ? __do_fault+0x3f9/0x430
Nov 24 14:25:36 mhvtl kernel: [  240.336192]  [<ffffffff810bc182>] ? __alloc_pages_nodemask+0x13d/0x7c4
Nov 24 14:25:36 mhvtl kernel: [  240.336197]  [<ffffffff8110c732>] ? dput+0x29/0xe8
Nov 24 14:25:36 mhvtl kernel: [  240.336202]  [<ffffffff810e6230>] ? alloc_pages_current+0xa7/0xca
Nov 24 14:25:36 mhvtl kernel: [  240.336207]  [<ffffffff8114ed16>] ? sysfs_write_file+0xe0/0x11c
Nov 24 14:25:36 mhvtl kernel: [  240.336211]  [<ffffffff810fbfe8>] ? vfs_write+0xa4/0xff
Nov 24 14:25:36 mhvtl kernel: [  240.336215]  [<ffffffff810fc0f9>] ? sys_write+0x45/0x6e
Nov 24 14:25:36 mhvtl kernel: [  240.336219]  [<ffffffff81338dd2>] ? system_call_fastpath+0x16/0x1b
Nov 24 14:25:36 mhvtl kernel: [  240.337277] vtllibrary      D ffff88003cfc88a0     0  1444      1 0x00000000
Nov 24 14:25:36 mhvtl kernel: [  240.337283]  ffff88003cfc88a0 0000000000000086 0000000000000028 ffff88003bea1660
Nov 24 14:25:36 mhvtl kernel: [  240.337288]  0000000000013b40 ffff88003d3c9fd8 ffff88003d3c9fd8 0000000000013b40
Nov 24 14:25:36 mhvtl kernel: [  240.337292]  ffff88003cfc88a0 ffff88003d3c8010 ffff88003beb4028 ffffffff811a36b1
Nov 24 14:25:36 mhvtl kernel: [  240.337297] Call Trace:
Nov 24 14:25:36 mhvtl kernel: [  240.337301]  [<ffffffff811a36b1>] ? kobject_set_name_vargs+0x51/0x58
Nov 24 14:25:36 mhvtl kernel: [  240.337305]  [<ffffffff81331e3d>] ? __mutex_lock_common+0x127/0x193
Nov 24 14:25:36 mhvtl kernel: [  240.337309]  [<ffffffff81331f75>] ? mutex_lock+0x1a/0x33
Nov 24 14:25:36 mhvtl kernel: [  240.337318]  [<ffffffffa00d0f4b>] ? __scsi_add_device+0x84/0xfc [scsi_mod]
Nov 24 14:25:36 mhvtl kernel: [  240.337323]  [<ffffffffa01fb3b9>] ? vtl_add_lu_action+0x2ae/0x2ec [mhvtl]
Nov 24 14:25:36 mhvtl kernel: [  240.337327]  [<ffffffff810b7677>] ? filemap_fault+0x1a3/0x34e
Nov 24 14:25:36 mhvtl kernel: [  240.337331]  [<ffffffff810d23c8>] ? __do_fault+0x3f9/0x430
Nov 24 14:25:36 mhvtl kernel: [  240.337336]  [<ffffffff810bc182>] ? __alloc_pages_nodemask+0x13d/0x7c4
Nov 24 14:25:36 mhvtl kernel: [  240.337340]  [<ffffffff8110c732>] ? dput+0x29/0xe8
Nov 24 14:25:36 mhvtl kernel: [  240.337345]  [<ffffffff810e6230>] ? alloc_pages_current+0xa7/0xca
Nov 24 14:25:36 mhvtl kernel: [  240.337349]  [<ffffffff8114ed16>] ? sysfs_write_file+0xe0/0x11c
Nov 24 14:25:36 mhvtl kernel: [  240.337353]  [<ffffffff810fbfe8>] ? vfs_write+0xa4/0xff
Nov 24 14:25:36 mhvtl kernel: [  240.337357]  [<ffffffff810fc0f9>] ? sys_write+0x45/0x6e
Nov 24 14:25:36 mhvtl kernel: [  240.337361]  [<ffffffff81338dd2>] ? system_call_fastpath+0x16/0x1b

[nia]

FYI, this issue does not seem to affect my other Linux box that is running Oracle Linux 5 : 2.6.32-100.0.19.el5