MHVTL : Getting Started

[[nia]]

MHVTL : Linux Virtual Tape Library By [e-mail] or [e-mail]

For more information, please visit MHVTL development site at http://sites.google.com/site/linuxvtl2/

License: GPL v2

Download:
http://sites.google.com/site/linuxvtl2/home#vtl-download-old
http://sites.google.com/site/linuxvtl2/home#vtl-download-stable
http://sites.google.com/site/linuxvtl2/home#vtl-download-devel

Or obtain most up-to-date releases via github:

To setup your own git repository:

Create a working directory
$ mkdir mhvtl
cd there
$ cd mhvtl
Initialise this directory as a git repository
$ git init
Pull the source code
$ git pull http://github.com/markh794/mhvtl.git


*** ( There is also Web GUI add-on provided by the community )
http://mhvtl-linux-virtual-tape-library-community-forums.966029.n3.nabble.com/ANNOUNCE-Web-Console-GUI-add-on-for-mhvtl-tp3136431p3136431.html



MHVTL : Linux Virtual Tape Library consists of several components:

LLD

LLD - A low level driver implemented as a kernel module - mhvtl
Target devices - Daemons vtltape(1) and vtllibrary(1) which implement SCSI target device(s) in user-space
Utility commands mktape(1), vtlcmd(1)
And startup scripts build_library_config(1), make_vtl_devices(1)
The kernel module is based on the scsi_debug kernel module (http://sg.danny.cz/sg/sdebug26.html). mhvtl.ko is a pseudo HBA (LLD). Note: As of 0.16, there are no default devices.
The support scripts will add Drives (SSC devices) & library (SMC) depending on the contents of /etc/mhvtl/library_contents and /etc/mhvtl/device.conf.
A char device back-end has been included with the vtl LLD driver This allows data and SCSI commands to be passed from the LLD to user-mode daemons (SCSI targets) which constently poll the driver and process any outstanding SCSI commands.

vtltape

vtltape(1) is the usermode SSC target daemon which writes/reads data to data files in the /opt/vtl directory (if a virtual tape has been loaded). The virtual tape files include a Medium Auxiliary Memory (MAM) data structure to store persistent data (number of tape loads, total data written/read, media type etc).

vtllibrary

vtllibrary(1) is the usermode SMC target daemon which reads its configuration from the file /etc/mhvtl/library_contents(5) at startup. The number of storage slots are built dynamically when the daemon starts. Hence changing the number of storage slots and media access slots are a matter of modifying the file contents and restarting the vtllibrary(1) daemon. All 'library' commands are performed on data structures in memory ONLY.

vtlcmd

A utility vtlcmd(1) is used to administrator the daemons vtltape(1) and vtllibrary.
Message queue (key 0x4d61726b) is used to pass messages between vtlcmd(1), vtllibrary(1) and vtltape(1)

When a SCSI 'move medium' from a storage slot to a tape drive is requested, the media location is updated in vtllibrary(1) data structures, and the barcode of the media id is passed via the message queue to the vtltape(1) daemon in question. A file open of the barcode is attempted in the /opt/vtl directory and if successful, the vtltape(1) daemon will now return a ASC/ASCQ 0x0/0x0 to any Test Unit Ready requests. An unload SCSI command to the tape drive will close the data file.

Media can be moved out of the VTL via the Media Access Port. Once media is logically moved into the MAP slots, the MAP entries can be cleared using the vxcmd:

# vtlcmd library empty map
The media can be 'moved back again' by re-starting the VTL user-space daemons:

Media can be loaded into the MAP via a :

# vtlcmd library load map <barcode>
or restart the vtl
# /etc/init.d/mhvtl restart
TapeAlert

TapeAlert flags can be set using the vtlcmd(1)

e.g. vtlcmd [index|library] TapeAlert [flags]

Where index is the message Q offset associated with the drive (or the string 'library').

e.g. To set flag 14h (Clean now) the 20th bit (14h) needs to be set:
Pull out the binary to decimal calculator
1000 0000 0000 0000 0000 (20 bits) => hex => 80000

# vtlcmd 1 TapeAlert 80000
A listing of TapeAlert flags can be found at t10 home page for SSC devices Annex A.

Examples

mhvtl is registered as HBA #3:
# lsscsi -g
[0:0:0:0] disk MAXTOR ATLAS10K4_36SCA DFM0 /dev/sda /dev/sg0
[0:0:6:0] process PE/PV 1x3 SCSI BP 1.1 - /dev/sg1
[2:0:0:0] disk SEAGATE ST336607FC 0003 /dev/sdb /dev/sg2
[2:0:1:0] disk SEAGATE ST336607FC 0003 /dev/sdc /dev/sg3
[2:0:2:0] mediumx ATL 1500 6.0 - /dev/sg4
[2:0:2:1] tape QUANTUM SuperDLT1 2323 /dev/st0 /dev/sg5
[2:0:2:2] tape QUANTUM SuperDLT1 2323 /dev/st1 /dev/sg6
[2:0:2:3] process ATTO 310-LV 1.42 - /dev/sg7
[3:0:0:0] mediumx STK  L700   vtl0 - /dev/sg8
[3:0:0:1] tape SONY SDX-900V 5400 /dev/st2 /dev/sg9
[3:0:0:2] tape SONY SDX-900V 5400 /dev/st3 /dev/sg10
[3:0:0:3] tape QUANTUM SDLT600 5400 /dev/st4 /dev/sg11
[3:0:0:4] tape QUANTUM SDLT600 5400 /dev/st5 /dev/sg12
[3:0:0:5] tape QUANTUM SDLT600 5400 /dev/st6 /dev/sg13
[3:0:0:6] tape IBM ULT3580-TD3 5400 /dev/st7 /dev/sg14
[3:0:0:7] tape IBM ULT3580-TD3 5400 /dev/st8 /dev/sg15
[3:0:0:8] tape IBM ULT3580-TD3 5400 /dev/st9 /dev/sg16

Getting Started

RPM based Linux distributions:

To install and get the package running:

1. Install both src.rpm & x86/x86_64 (depending on your CPU/OS version - check using uname -p)

2. Build kernel module from source you just installed:
  cd /usr/src/packages/BUILD
  tar xvfz ../SOURCES/mhvtl-2009-12-16.tgz
  cd mhvtl-0.16/kernel
  make
  make install

3. Start the package using the rc script.
  /etc/init.d/mhvtl start

4. Check it's all running:
  lsscsi -g
  ps -ef|grep vtl (default with 5 daemons - 1 vtllibrary & 4 vtltape)

5. Configure your backup software ;)

NON-RPM based Linux distributions:

To install and get the package running:

1. Download the 'mhvtl-YYYY-MM-DD.tgz

2. Extract source code
 cd /some/where/safe
 tar xvfz /where/you/downloaded/mhvtl-YYYY-MM-dd.tgz
3. Build user-space daemons
 cd mhvtl-0.18 (or mhvtl-0.16 if you downloaded the 'stable' version)
 make
4. Add a user 'vtl'
 sudo useradd -c "mhvtl user" vtl

5. Install binaries
 sudo make install
6. Build kernel module from source you just installed:
  cd kernel
  make
  make install

7. Start the package using the rc script.
  /etc/init.d/mhvtl start

8. Check it's all running:
  lsscsi -g
  ps -ef|grep vtl (default with 5 daemons - 1 vtllibrary & 4 vtltape)

9. Configure your backup software ;)



Long term goals

Eventually, the functionality will be moved across to the SCSI Target Framework (stgt) http://stgt.berlios.de/
The stgt is a very nicely designed SCSI taret framwork which can support all target types along with multiple SCSI transports in a very structured way.

However, until time permits (and my c coding standards improve), features and improvements on this code base will continue.


There are many way of providing assistance

Testing with backup software X and reporting success/failure. If possible, test any patches that may result of testing.
If you are willing, patches are very welcome.
Please read the Linux Documentation/CodingStyle
Note: I only discovered this documentation guide recently, so patches to update old code style to the CodingStyle are also welcome.
Please run any patches thru the Linux 'mhvtl-<version>/scripts/checkpatch.pl' for coding style sanity check before sending.

Nice things people are saying about the mhvtl

It's really caught on here, I've noticed that some of our developers have stopped using real tape hardware altogether, it's just easier to use the VTL. Plus our customer support has started doing customer training with the VTL instead of real tape hardware. Pretty impressive! A fine piece of software, wish I had found it a few years earlier.
The VTL has been very helpful in my testing and for customer demos. Your hard work on this is very much appreciated by myself and others on my team.
I am one more user of your magical software vtl here at Symantec. Just want to say that it rocks
I finally got round to trying to get this set up and I have to say.. You are a genius ! My laptop now has NBU 6.5.1 on Redhat 4 AS (on VMware) with a VTL robot and 8 drives !! Fantastic ! Thanks for all your work on this ! This will be very useful for me.

By [e-mail] or [e-mail]