kbnlresearch / tapeimgr Goto Github PK

This utility should traverse all 1st-order subdirs, look for a tapeimgr metadata file (naming based on config file, with option to override to user-defined name), and create a summary file in CSV format that lists, for each tape:

• tape directory file path (relative to parent dir)
• identifier
• description
• acquisitionStart
• acquisitionEnd
• successFlag

Note that you may (or may not) need to swap endian-ness if the tape was written on a big-endian machine

https://unix.stackexchange.com/a/480553/187090

But see: #19, pkexec makes this quite difficult.

Specifically this is about the dd exit status in the block size estimation function, and the final mt status at the end of the tape (both of these might alarm users).

Apparently block size can be variable within a file that's written to tape, see:

https://github.com/torvalds/linux/blob/master/Documentation/scsi/st.rst

In variable block mode, the byte count in write() determines the size
of the physical block on tape. When reading, the drive reads the next
tape block and returns to the user the data if the read() byte count
is at least the block size. Otherwise, error ENOMEM is returned.

Don't know how common this is, but this is something to keep in mind if tapeimgr fails on a tape. In theory support for variable-block tapes could be added by reading 1-block at a time, and then do a new blockSize estimation once a block results in read errors.

TODO: see if variable-block tapes can be emulated with mhvtl, and use that for testing.

It also implies that the 'filll failed blocks' option is likely to yield corrupted content for variable-block tapes.

See also:

https://unix.stackexchange.com/a/366217/187090

and:

https://www.mkssoftware.com/docs/man4/tape.4.asp

which says:

SCSI tape drives can support variable block mode, fixed block mode, or both. Older drives typically support only fixed block mode. Newer drives often either support variable block mode only, or both variable block and fixed block modes.

So if true this means older tapes are unlikely to be affected.

Same as:

KBNLresearch/omimgr#13

See omimgr commits 25-3-2019:

KBNLresearch/omimgr@1724f59

KBNLresearch/omimgr@0e59da0

KBNLresearch/omimgr@d73c233

KBNLresearch/omimgr@7512117

The exact same function is now replicated in both cli.py and gui.py:

It would be better to move it to the Tape class. This would require some changes to the Tape class and how it is called by gui.py and cli.py. In particular:

• Define default values for all Tape class arguments
• Create a Tape class instance before Submit function is called
• Then use additional function calls to update the Tape instance variables.

• New annotation options / interface
• Metadata output file
• New items in configuration file
• Default dir in config file

Change to sth like this:

<action id="org.freedesktop.policykit.pkexec.run-tapeimgr">

The current launcher uses gksudo to run tapeimgr as root. But gksudo is not installed by default on BitCurator / Ubuntu 18.04LTS (Bionic). BitCurator does have the pkexec tool (which it uses to launch Guymager as root), but running the command:

pkexec tapeimgr

Results in this error:

_tkinter.TclError: no display name and no $DISPLAY environment variable

It seems that this not a bug, but a result of the pkexec/PolicyKit policy settings. See also:

https://groups.google.com/forum/#!topic/comp.lang.python/s4WMWIdkXMY

More on differences between pkexec and gksudo:

https://askubuntu.com/questions/78352/when-to-use-pkexec-vs-gksu-gksudo

And specifically on Python:

https://askubuntu.com/questions/288087/can-i-use-pkexec-in-a-python-script-or-a-desktop-file

IOError, PermissionError for log file; check if dirOut already contains files, etc.

Sometime the following error occurs at the end of a tapeimgr run:

2018-12-04 14:34:48,206 - ERROR - 'int' object is not callable
Traceback (most recent call last):
  File "/home/johan/.local/lib/python3.5/site-packages/tapeimgr/gui.py", line 341, in main
    time.sleep(0.1)
TypeError: 'int' object is not callable

Ther error refers to the following line (sleep value for main loop in GUI):

 time.sleep(0.1)

Similar to Guymager. Fields can be left empty. Use .json file to store some general metadata as well, such as:

• file names + checksum + checksumType (in addition to existing checksums file)
• tapeimgr version
• acquisition start / end dateTime

https://github.com/bitsgalore/tapeimgr/blob/6e1fad2d02e80de539f3c1b3e5b6e30462fef05c/tapeimgr/tapeimgr.py#L192

Change to sth like (for 10-char filename + extension):

        ofName = prefix + str(session).zfill(10 - len(prefix)) + '.' + extension

Same as diskimgr, device list needs filtering to display only tape devices.

Trying to recover old DDS-90 tapes on a Sony DAT drive. Keeps reading only the first 32K block. I let it run on a tape and stopped it when the first 40+ .dd files were identical. Tried a couple others and get either this or it just keeps increasing the test size and never reads a block.

Installed fresh Ubuntu 20.x and kept using apt-get until it ran.

Current code:

self.fillblocks_entry = tk.Checkbutton(self, variable=self.tape.fillBlocks)

Shouldn't this be:

self.fillblocks_entry = tk.Checkbutton(self, variable=self.tape.fBlocks)

Also code in reset_gui doesn't look quit right. See also omimgr.

Strangely it works normally when the desktop launcher is used. Happens with both global and user install. Probably a permissions thing.

Very odd, posted a question below:

https://stackoverflow.com/questions/53375415/python-3-keyboardinterrupt-in-background-thread-not-detected-by-main-thread-un

For now the effect is that the "Completed processing this tape" dialog doesn't show up until the user either hovers the mouse over the UI, or presses a key.

See:

https://wiki.debian.org/SystemGroups

So what might work is to let the configuration script add the current user to the tape group. In that case there would be no need for root access, user installs would work and all pkexec-related stuff could be dropped.

Example:

"tapeimagrVersion": "0.4.0"

Should be:

"tapeimgrVersion":"0.4.0" 

For DDS-3 tape, fsr exit status after 3 sessions is 0, but subsequent reads with dd result in:

dd: error reading '/dev/nst0': Input/output error
0+0 records in
0+0 records out
0 bytes copied, 0.0206987 s, 0.0 kB/s

Result is that tapeimgr get stuck in an infinite loop, producing 0-byte files for each iteration. Unclear if cause is tape, reader or something else. A possible fix is to do a check on dd's exit status, and quit if it isn't 0 (but this could have some unintended side-effects).

In that case:

• configuration would be stored in ~/.config/tapeimgr directory:
  https://stackoverflow.com/questions/1024114/location-of-ini-config-files-in-linux-unix
• menu entry in ~/.local/share/applications

Global install goes to:

/usr/local/lib/python3.5/dist-packages/

User install goes to:

/home/johan/.local/lib/python3.5/site-packages/

So rule, sth like:

• If packageDir in $HOME :

  • configuration to ~/.config/tapeimgr directory
  • menu entry in ~/.local/share/applications
  • desktop launcher to ~/Desktop

• Else:

  • configuration to /etc/tapeimgr directory
  • menu entry in /usr/share/applications
  • desktop launcher to ~/Desktop

When attempting to install this tape imager using the command
sudo install pip3 tapeimgr
I encounter this error.

I'm running Mint 19.3 64-bit MATE.
I copied the error into a note because it shows up as red text in the terminal and is difficult to read.
I'm not sure how to continue.

Would need an overall success/fail flag, based on the dd / mt exit codes. If fail, notify user via pop-up at end of imaging run (and refer to log file for details)

Note: tapeimgr needs root access, so menu/desktop launcher command should be sth like gksudo tapeimgr. This might need some kind of post-install hook.

This can mean the tape device is not accessible, or tapeimgr wasn't launched as sudo.

Value is now set to user's home dir, but it would be convenient to have the option to use a user-defined location instead (e.g. when all images are written to an external disk).