tehmaze / xmodem

XMODEM protocol implementation for Python

XMODEM protocol implementation

Documentation available at http://packages.python.org/xmodem/

Python Package Index (PyPI) page is available at https://pypi.python.org/pypi/xmodem

xmodem has been tested with python 2.7 through python 3.11

Usage

Create a function to get and put character data (to a serial line for example):

>>> import serial
>>> from xmodem import XMODEM
>>> ser = serial.Serial('/dev/ttyUSB0', timeout=0) # or whatever port you need
>>> def getc(size, timeout=1):
...     return ser.read(size) or None
...
>>> def putc(data, timeout=1):
...     return ser.write(data)  # note that this ignores the timeout
...
>>> modem = XMODEM(getc, putc)

Now, to upload a file, use the send method:

>>> stream = open('/etc/fstab', 'rb')
>>> modem.send(stream)

To download a file, use the recv method:

>>> stream = open('output', 'wb')
>>> modem.recv(stream)

For more information, take a look at the documentation.

Changes

0.4.7:

• bugfix: stall on some kinds of error in recv(), PR #56.
• bugfix: sequence number miscalculation in send(), PR #52.
• enhancement: callback function added for recv(), PR #53.
• bugfix: receiving empty file and stall condition in recv(), PR #50.
• bugfix: callback is now called for some kinds of errors and some CLI fixes, 8a798e8b.
• bugfix: remove DepreactionWarning for logging.warn(), PR #49.

0.4.6:

• bugfix: Abort send on EOT in startup-sequence Issue #34.
• enhancement: Include LICENSE file in the source distribution.

0.4.5:

• bugfix: Remove bogus assert False code in recv() that resulted in AssertionError introduced in version 0.4.0 commit-id 9b03fc20, PR #29.

0.4.4:

• bugfix: Large file transfers in send() were more likely to fail for small values of retry: This value should be the maximum failures per block transfer as documented, but was improperly implemented as the number of failures allowed for the total duration of the transfer, PR #21.
• bugfix: send(retry=n) and recv(retry=n) should retry n times as documented, was retrying n - 1.

0.4.3:

• bugfix: putc() callback was called in series, 3 times for each part of xmodem block header, data, and checksum during block transfer. Now all three data blocks are sent by single putc() call. This resolves issues when integrating with microcontrollers or equipment sensitive to timing issues at stream boundaries, PR #19.

0.4.2:

• bugfix: documentation files missing from the release tarball Issue #16.

0.4.1

• bugfix: re-transmit in send() on NAK or timeout, previously re-transmissions (wrongly) occurred only on garbage bytes. PR #12.

0.4.0

• enhancement: support for python 3 PR #8.
• bugfix: CRC failures in XMODEM.recv() were not renegotiated correctly PR #11.

xmodem's Issues

Got AssertionError when receive file

Hey, I tried to use the latest version, 0.4.4, to transfer files between nova instance and host, mainly instance to host. I've redirected the instance's serial port to host's unix domain socket.
On the instance

import serial

ser = serial.Serial(port = '/dev/ttyS2',baudrate=9600,timeout=3)
ser.flushInput()
def getc(size,timeout=1):
    data = ser.read(size)
    return data

def putc(data,timeout=1):
    return ser.write(data)

from xmodem import XMODEM
modem = XMODEM(getc,putc)
test = file('test.txt','rb')
print modem.send(test)
ser.close()

On host

import socket
import sys

if len(sys.argv)!=2:
    print 'usage:python '
unix_socket = sys.argv[1]
s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
s.connect(unix_socket)
def getc(size,timeout=1):
    data = s.recv(size)
    return data

def putc(data,timeout=1):
    return s.send(data)

from xmodem import XMODEM
modem = XMODEM(getc,putc)
test = file('test.txt','wb')
print modem.recv(test)
s.close()

Then I ran these two scripts, the guest send a file and the host try to receive it. But the host constantly got errors and then received the file finally.

root@node-51:~# python receivefile.py /tmp/virtualsocket/0a125fca-efac-4b4d-b681-7c78f3087713 
No handlers could be found for logger "xmodem.XMODEM"
Traceback (most recent call last):
  File "receivefile.py", line 19, in <module>
    print modem.recv(test)
  File "build/bdist.linux-x86_64/egg/xmodem/__init__.py", line 531, in recv
AssertionError: h
root@node-51:~# python receivefile.py /tmp/virtualsocket/0a125fca-efac-4b4d-b681-7c78f3087713 
No handlers could be found for logger "xmodem.XMODEM"
Traceback (most recent call last):
  File "receivefile.py", line 19, in <module>
    print modem.recv(test)
  File "build/bdist.linux-x86_64/egg/xmodem/__init__.py", line 531, in recv
AssertionError:  
root@node-51:~# python receivefile.py /tmp/virtualsocket/0a125fca-efac-4b4d-b681-7c78f3087713 
No handlers could be found for logger "xmodem.XMODEM"
Traceback (most recent call last):
  File "receivefile.py", line 19, in <module>
    print modem.recv(test)
  File "build/bdist.linux-x86_64/egg/xmodem/__init__.py", line 531, in recv
AssertionError:  
root@node-51:~# python receivefile.py /tmp/virtualsocket/0a125fca-efac-4b4d-b681-7c78f3087713 
No handlers could be found for logger "xmodem.XMODEM"
Traceback (most recent call last):
  File "receivefile.py", line 19, in <module>
    print modem.recv(test)
  File "build/bdist.linux-x86_64/egg/xmodem/__init__.py", line 531, in recv
AssertionError:  
root@node-51:~# python receivefile.py /tmp/virtualsocket/0a125fca-efac-4b4d-b681-7c78f3087713 
No handlers could be found for logger "xmodem.XMODEM"
Traceback (most recent call last):
  File "receivefile.py", line 19, in <module>
    print modem.recv(test)
  File "build/bdist.linux-x86_64/egg/xmodem/__init__.py", line 531, in recv
AssertionError:  
root@node-51:~# python receivefile.py /tmp/virtualsocket/0a125fca-efac-4b4d-b681-7c78f3087713 
No handlers could be found for logger "xmodem.XMODEM"
Traceback (most recent call last):
  File "receivefile.py", line 19, in <module>
    print modem.recv(test)
  File "build/bdist.linux-x86_64/egg/xmodem/__init__.py", line 531, in recv
AssertionError:  
root@node-51:~# python receivefile.py /tmp/virtualsocket/0a125fca-efac-4b4d-b681-7c78f3087713 
No handlers could be found for logger "xmodem.XMODEM"
Traceback (most recent call last):
  File "receivefile.py", line 19, in <module>
    print modem.recv(test)
  File "build/bdist.linux-x86_64/egg/xmodem/__init__.py", line 531, in recv
AssertionError:  
root@node-51:~# python receivefile.py /tmp/virtualsocket/0a125fca-efac-4b4d-b681-7c78f3087713 
128
root@node-51:~# 

I checked the xmodem/init.py line 531

assert False, data

Did I make a mistake or is this a bug?

version planning

Hi, the release is only 0.4.5 here. However, I find the 0.4.6 version is available on the Pypi: https://pypi.org/project/xmodem/. Will there be any later versions than 0.4.6? Will the version be updated?

Example code required

Hi,

It would be good to have some fully functional XMODEM example, for getting the file that is sent using Linux sz tool. (https://linux.die.net/man/1/sz)

My other questions are,

1. Is it actively maintained?
2. Does it support Python 3.10

Thanks in advance, let me know if I can be helpful in any sort.

Command-line tool

xmodem/__init__.py provides a main function. As far as I can see, it does not use pyserial to send/receive data but sz and rz utilities.

Are you planning to rewrite it to use the internal functions only?

It would be also useful if this utility could be used as follows:

python3 -m xmodem send filename filename

i.e. the way you can use some python classes:

python3 -m http.server 8888

LICENSE file

Please include a LICENSE file in the pypi source distribution. Thanks.

On CRC checksum failure and block re-transmission, send() calls getc() for next block

Needs a test, but I discovered while working through some porting issues that when xmodem.send receives a NAK for retransmission -- although acknowledged, it still subsequently calls self.getc() for the next block -- thereby skipping a sequence block all-together.

pip installation fails with python 3

Here's the output I get when installing into a virtualenv with python 3:

(venv3)allen@localhost:~/scratch
➤ pip install xmodem
Collecting xmodem
  Using cached xmodem-0.4.1.tar.gz
Building wheels for collected packages: xmodem
  Running setup.py bdist_wheel for xmodem
  Complete output from command /home/allen/scratch/venv3/bin/python3 -c "import setuptools;__file__='/tmp/pip-build-rim3dtub/xmodem/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /tmp/tmp5_m7wzkupip-wheel-:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib
  creating build/lib/xmodem
  copying xmodem/__init__.py -> build/lib/xmodem
  running egg_info
  writing dependency_links to xmodem.egg-info/dependency_links.txt
  writing top-level names to xmodem.egg-info/top_level.txt
  writing xmodem.egg-info/PKG-INFO
  warning: manifest_maker: standard file '-c' not found

  reading manifest file 'xmodem.egg-info/SOURCES.txt'
  writing manifest file 'xmodem.egg-info/SOURCES.txt'
  installing to build/bdist.linux-x86_64/wheel
  running install
  running install_lib
  creating build/bdist.linux-x86_64
  creating build/bdist.linux-x86_64/wheel
  creating build/bdist.linux-x86_64/wheel/xmodem
  copying build/lib/xmodem/__init__.py -> build/bdist.linux-x86_64/wheel/xmodem
  running install_data
  creating build/bdist.linux-x86_64/wheel/xmodem-0.4.1.data
  creating build/bdist.linux-x86_64/wheel/xmodem-0.4.1.data/data
  creating build/bdist.linux-x86_64/wheel/xmodem-0.4.1.data/data/doc
  error: can't copy 'doc/XMODEM.TXT': doesn't exist or not a regular file

  ----------------------------------------
  Failed building wheel for xmodem
Failed to build xmodem
Installing collected packages: xmodem
  Running setup.py install for xmodem
    Complete output from command /home/allen/scratch/venv3/bin/python3 -c "import setuptools, tokenize;__file__='/tmp/pip-build-rim3dtub/xmodem/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-9flb_zhb-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/allen/scratch/venv3/include/site/python3.5/xmodem:
    running install
    running build
    running build_py
    running egg_info
    writing dependency_links to xmodem.egg-info/dependency_links.txt
    writing top-level names to xmodem.egg-info/top_level.txt
    writing xmodem.egg-info/PKG-INFO
    warning: manifest_maker: standard file '-c' not found

    reading manifest file 'xmodem.egg-info/SOURCES.txt'
    writing manifest file 'xmodem.egg-info/SOURCES.txt'
    running install_lib
    creating /home/allen/scratch/venv3/lib/python3.5/site-packages/xmodem
    copying build/lib/xmodem/__init__.py -> /home/allen/scratch/venv3/lib/python3.5/site-packages/xmodem
    byte-compiling /home/allen/scratch/venv3/lib/python3.5/site-packages/xmodem/__init__.py to __init__.cpython-35.pyc
    running install_data
    creating /home/allen/scratch/venv3/doc
    error: can't copy 'doc/XMODEM.TXT': doesn't exist or not a regular file

    ----------------------------------------
Command "/home/allen/scratch/venv3/bin/python3 -c "import setuptools, tokenize;__file__='/tmp/pip-build-rim3dtub/xmodem/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-9flb_zhb-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/allen/scratch/venv3/include/site/python3.5/xmodem" failed with error code 1 in /tmp/pip-build-rim3dtub/xmodem

This pulls down the xmodem-0.4.1.tar.gz file from pypi, whereas installing with python2 uses the xmodem-0.4.1-py2-none-any.whl file. The xmodem-0.4.1.tar.gz is missing the doc/ directory.

Reworking the example

First of all, is xmodem UNIX only project or does it truly support Windows? If it only supports UNIX-like operating systems, would it make sense to add select() call to the getc() routine so that it can benefit from the timeout parameter?

def getc(size, timeout=1):
    buf = None
    readable,_,_ = select.select([ser], [], [], timeout)
    if ser in readable:
        buf = ser.read(size)

    return buf

I would also replace serial.Serial() with serial.serial_for_url().

What about Py2 support? Does it still make sense to support it?

broken API for calc_checksum

the example for checksum is simply:

csum = modem.calc_checksum('hello')

but this cause error:

return (sum(data) + checksum) % 256
TypeError: unsupported operand type(s) for +: 'int' and 'str'

quick fix:

return (sum(data) + checksum) % 256

should instead be

csum = sum(bytearray('hello', "utf-8")) % 256

or the API should be changed to

csum = modem.calc_checksum(bytearray('hello'))

up for adoption: supporting y and z-modem batch file transfers

Let us complete @tehmaze vision of the python 'modem' module, https://pypi.python.org/pypi/modem/1.0 and https://github.com/tehmaze-labs/modem

• design: how should an API for batch file send/recv appear? Agree on a new send/recv pattern.
• research: is z-modem still a lot of work to implement?
• laborers: we're all too busy, any volunteers?

Another API, https://pypi.python.org/pypi/serialXfer is released, based on this one, with only YMODEM support, though documented as ZMODEM as TODO. Can we collaborate with this Jason Morgan?

We also have partial y-modem send() contributed as PR #24.

Incidentally, I also own the modem.xyz domain for documentation.

Some mistake in clac_crc() comment.

I used the algorithm in the code, it works good. But the comment of calc_crc() function puzzled me a lot.

'''
Calculate the Cyclic Redundancy Check for a given block of data, can
also be used to update a CRC.
crc = modem.calc_crc('hello')
crc = modem.calc_crc('world', crc)
hex(crc)
'0xd5e3'
'''

Should the hex(crc) be 0x4ab3 instead of 0xd5e3? I checked many times. The algorithm is right. But the example result listed above is far different from whatever I could get.

coveralls.io support

please enable this repository in your coveralls.io account, so we may add coveralls to after_success of travis.yaml

Sending EOT cause Handler Error

When sending a file, If the receiver cancels the transfer sending the EOT 0x04 then I get an error like this:
No handlers can be found for logger "xmodem.XMODEM"

Turning on the logging in the debug mode I then show this:

DEBUG:xmodem.XMODEM:Begin start sequence, packet_size=128
ERROR:xmodem.XMODEM:send error: expected NAK, CRC, or CAN; got '\x04'
INFO:xmodem.XMODEM:send error: error_count reached 16, aborting.

Shouldn't the Xmodem abort immediately after receiving the EOT char?

bug? xmodem 0.2.4 transfers where 0.3.2+ does not.

The xmodem package on PyPi is out of date. It is at version 0.3.2, but 0.4.1 is released (or tagged) and there is another recent bug fix committed post 0.4.1. Could you please update so I can pip install the latest / most stable version.

Is there a forum to discuss issues with xmodem?
I currently have an issue where xmodem 0.3.2 (using python 2.7 on OS X Yosemite 10.10) is giving me timeout errors. The same code works ok with xmodem 0.2.3 (using python 2.6). I'm hoping a later version will fix things. What is the best way to install the latest version (given PyPi is out of date).

send error: expected NAK, CRC, EOT or CAN; got ***

I try to use XMODEM to send a txt file to my embedded device.
However, the error information is:

send error: expected NAK, CRC, EOT or CAN; got b'\x00'
send error: expected NAK, CRC, EOT or CAN; got b'r'
send error: expected NAK, CRC, EOT or CAN; got b't'
send error: expected NAK, CRC, EOT or CAN; got b'O'
send error: expected NAK, CRC, EOT or CAN; got b'e'
send error: expected NAK, CRC, EOT or CAN; got b'W'
send error: expected NAK, CRC, EOT or CAN; got b':'
send error: expected NAK, CRC, EOT or CAN; got b'm'
send error: expected NAK, CRC, EOT or CAN; got b'#'
send error: expected NAK, CRC, EOT or CAN; got b'r'
send error: expected NAK, CRC, EOT or CAN; got b' '
send error: expected NAK, CRC, EOT or CAN; got b' '
send error: expected NAK, CRC, EOT or CAN; got b' '
send error: expected NAK, CRC, EOT or CAN; got b'1'
send error: expected NAK, CRC, EOT or CAN; got b't'
send error: expected NAK, CRC, EOT or CAN; got b'1'
send error: expected NAK, CRC, EOT or CAN; got b'\n'

...................

my code on my computer is:

def getc(size, timeout=10):
return mSerial.port.read(size) or None

def putc(data, timeout=10):
return mSerial.port.write(data) # note that this ignores the timeout

stream = open('test1.txt', 'rb')
modem = XMODEM(getc, putc)
modem.send(stream)

---

On the embedded device, I use command "rx -X -t 100 test1.txt" to wait for receiving the file

Missing LICENSE file when downloading from pypi

There is no LICENSE file included when downloading from pypi: https://files.pythonhosted.org/packages/29/5d/a20d7957f207fc4c4c143881ca7b9617ab7700c153012372ef0a934c7710/xmodem-0.4.6.tar.gz

Hence it makes it difficult to attach license file automatically in projects such as Buildroot: https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/

I broke the tests, fix travis

ACK after receiving EOT

It looks to me that after the EOT is recv'd the sender expects an ACK. In your code, the method returns the incoming byte count, but no ACK. Can you confirm this is a bug?

Thanks! David

In recv, if char is something unexpected, it will loop until error_count > retry without re-evaluating char

In xmodem/init.py, between lines 472 and 510, we have a while loop and if clauses to determine the appropriate action to take. I had a case where I would have the variable char being something unexpected and then the execution would go in the else statement. However, char is never re-evaluated in that else. It would then just loop for retries in the else. I believe that adding this statement: char = self.getc(1, timeout) after line 504 would solve this undesirable behavior.

Thanks for your time!

Problems with pypi page

I noticed that the PyPi page shows:

YMODEM: | © 1985 Chunk Forsberg, Omen Technology Inc.
ZMODEM: | © 1986 Chunk Forsberg, Omen Technology Inc.

Pretty sure his name was Chuck Forsberg, unless he had an alternate career as a Chippendale of which I was previously unaware.

Fails on large files

I've tried this over and over in my python 2.7 script and it works perfect up to the point about 24 packets. This is sending xmodem to an older machine, which will fill it's memory, stop the transfer, then when there is more room it receives more xmodem packets. Xmodem code here doesn't seem to handle that and just goes crazy trying to send the same packet over and over then things go south.

2020-05-24 16:15:15,646 ERROR send error: expected ACK; got None for block 24

Block 23
06 01 17 E8 20 59 2D 31 2E 38 30 31 0D 0A 4E 35 39 30 20 58 31 2E 38 31 37 33 20 59 2D 31 2E 37 35 36 35 0D 0A 4E 35 39 35 20 58 31 2E 38 36 30 34 20 59 2D 31 2E 37 31 30 38 0D 0A 4E 36 30 30 20 58 31 2E 39 30 32 33 20 59 2D 31 2E 36 36 33 39 0D 0A 4E 36 30 35 20 58 31 2E 39 34 33 31 20 59 2D 31 2E 36 31 36 32 0D 0A 4E 36 31 30 20 58 31 2E 39 38 32 37 20 59 2D 31 2E 35 36 37 34 0D 0A 4E 36 31 FA (OKAY)

Block 24
06 01 18 E7 35 20 58 32 2E 30 32 31 20 59 2D 31 2E 35 31 37 35 0D 0A 4E 36 32 30 20 58 32 2E 30 35 38 31 20 59 2D 31 2E 34 36 36 38 0D 0A 4E 36 32 35 20 58 32 2E 30 39 34 20 59 2D 31 2E 34 31 35 33 0D 0A 4E 36 33 30 20 58 32 2E 31 32 38 35 20 59 2D 31 2E 33 36 32 37 0D 0A 4E 36 33 35 20 58 32 2E 31 36 31 37 20 59 2D 31 2E 33 30 39 34 0D 0A 4E 36 34 30 20 58 32 2E 31 39 33 37 20 59 2D 31 2E 32 1E

Block 24 retry
01 18 E7 35 20 58 32 2E 30 32 31 20 59 2D 31 2E 35 31 37 35 0D 0A 4E 36 32 30 20 58 32 2E 30 35 38 31 20 59 2D 31 2E 34 36 36 38 0D 0A 4E 36 32 35 20 58 32 2E 30 39 34 20 59 2D 31 2E 34 31 35 33 0D 0A 4E 36 33 30 20 58 32 2E 31 32 38 35 20 59 2D 31 2E 33 36 32 37 0D 0A 4E 36 33 35 20 58 32 2E 31 36 31 37 20 59 2D 31 2E 33 30 39 34 0D 0A 4E 36 34 30 20 58 32 2E 31 39 33 37 20 59 2D 31 2E 32 1E

Below is a packet send with SZ
06 01 18 E7 0A 4E 36 31 35 20 58 32 2E 30 32 31 20 59 2D 31 2E 35 31 37 35 0D 0A 4E 36 32 30 20 58 32 2E 30 35 38 31 20 59 2D 31 2E 34 36 36 38 0D 0A 4E 36 32 35 20 58 32 2E 30 39 34 20 59 2D 31 2E 34 31 35 33 0D 0A 4E 36 33 30 20 58 32 2E 31 32 38 35 20 59 2D 31 2E 33 36 32 37 0D 0A 4E 36 33 35 20 58 32 2E 31 36 31 37 20 59 2D 31 2E 33 30 39 34 0D 0A 4E 36 34 30 20 58 32 2E 31 39 33 37 20 59 1F

How is the last part of the packet getting corrupt identical each time?
Here it is with this xmodem, in ascii.... that resends the same block each time...
N61úç5 X2.021 Y-1.5175
N620 X2.0581 Y-1.4668
N625 X2.094 Y-1.4153
N630 X2.1285 Y-1.3627
N635 X2.1617 Y-1.3094
N640 X2.1937 Y-1.2ç5 X2.021 Y-1.5175
N620 X2.0581 Y-1.4668
N625 X2.094 Y-1.4153
N630 X2.1285 Y-1.3627
N635 X2.1617 Y-1.3094
N640 X2.1937 Y-1.2ç5 X2.021 Y-1.5175
N620 X2.0581 Y-1.4668
N625 X2.094 Y-1.4153
N630 X2.1285 Y-1.3627
N635 X2.1617 Y-1.3094
N640 X2.1937 Y-1.2__ç5 X2.021 Y-1.5175

I used the Linux sz -X from the cli and it worked perfect.
I've also used hyperterminal xmodem from windows and another terminal program to send and it also works perfect.

I have ascii and hex logging of the transactions, and I also have the debug log from xmodem that I can send whomever is interested.

doc/*.txt licenses

The licenses of the doc/*.txt files are not clear. Please clarify them.

New pypi package

The latest package on pypi is 0.3.2. Could we get the latest (0.4.1) uploaded? I'd love to be able to install via pip with python 3.

Zmodem suggest

Hi,
I find you are trying to understand the Zmodem protocol and port it into python.
But why not we use subprocess.popen() to call rz and sz to recv and send file?

I just use it to send an file by zmodem and also have good performance.

Is it a good idea? If so, maybe I can send an PR for it.