naturalhpc / cerulean

A Python 3 library for talking to HPC clusters and supercomputers

Cerulean

Cerulean is a Python 3 library for talking to HPC clusters and supercomputers. It lets you copy files between local and SFTP filesystems using a pathlib-like API, it lets you start processes locally and remotely via SSH, and it lets you submit jobs to schedulers such as Slurm and Torque/PBS.

Documentation and Help

Cerulean can be installed as usual using pip:

pip install cerulean

Instructions on how to use Cerulean can be found in the Cerulean documentation.

Code of Conduct

Before we get to asking questions and reporting bugs, we'd like to point out that this project is governed by a code of conduct, as described in CODE_OF_CONDUCT.rst, and we expect you to adhere to it. Please be nice to your fellow humans.

Questions

If you have a question that the documentation does not answer for you, then you have found a bug in the documentation. We'd love to fix it, but we need a bit of help from you to do so. Please do the following:

1. use the search functionality here to see if someone already filed the same issue;
2. if your issue search did not yield any relevant results, make a new issue;
3. apply the "Question" label; apply other labels when relevant.

We'll answer your question, and improve the documentation where necessary. Thanks!

Bugs

Like most software, Cerulean is made by humans, and we make mistakes. If you think you've found a bug in Cerulean, please let us know! Reporting bugs goes as follows.

1. Use the search functionality here to see if someone already filed the same issue.
2. If your issue search did not yield any relevant results, make a new issue. Please explain: - what you were trying to achieve, - what you did to make that happen, - what you expected the result to be, - what happened instead. It really helps to have the actual code for a simple example that demonstrates the issue, but excerpts and error messages and a description are welcome too.
3. Finally, apply any relevant labels to the newly created issue.

With that, we should be able to fix the problem, but we may ask for some more information if we can't figure it out right away.

Development

More information for Cerulean developers may be found in the Cerulean documentation.

License

Copyright (c) 2018, The Netherlands eScience Center and VU University Amsterdam

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

cerulean's Issues

scheduler.wait() should allow setting sleep time

The scheduler.wait() function waits until a job is done. You can give it a timeout, and it will check to see if the job is done at intervals of 5% of the timeout, with a minimum of 0.1s. 5% of the expected time is a reasonable check interval, but the timeout is not the expected time, it's the maximum time you're willing to wait before you decide that something has gone wrong. So if you have a job that usually doesn't take much time, but sometimes does, then this doesn't work well.

Since we're not sure about the situation, it's probably best to let the user specify the sleep time as an optional parameter, and fall back to the present behaviour if they don't specify anything. Perhaps a minimum of 1s would be better (schedulers are not that quick anyway, and if you expect the command to finish really quickly, you should use a Terminal anyway), and/or a lower percentage.

CopyCallback docstring is broken

This appears to be a bug in Sphinx autodoc, see sphinx-doc/sphinx#4422

Slurm submission not error-checked

SlurmScheduler doesn't check the return value of sbatch (well, only to log it) on submitting a job. If this fails, it should be detected, and an exception should be raised.

JobDescription API docs have a formatting issue

At the end of the description, the "Note that stdout_file and stderr_file will receive..." part should be formatted as plain text.

Add a RelativePath class?

Currently in cerulean, paths are always absolute. That's a good thing, because a current working directory is a bit of a tricky concept when you're dealing with remote file systems and network errors and such. However, we sometimes need relative paths, like in a recursive copy of a directory, where we need to obtain the relative path of a file inside the tree to be copied with respect to the root, so we can rebase it on the target directory to get an absolute path to copy the file to.

The chrono library in C++ has time_point and duration classes. A time_point is an absolute point in time, while a duration is the difference between two time_points. So you can add a duration to a time_point to get another time_point, or subtract it, or add two durations to get another duration. This concept seems like a good solution here.

So let's add a cerulean.RelativePath here. A RelativePath is abstract, and it's not associated with a FileSystem like a normal Path. RelativePaths can be concatenated using / to form a new RelativePath, and you can write path / relative_path to get a Path containing the concatenation.

A RelativePath is pure, which means you cannot access files through it. Which methods should it have? Some subset of PurePosixPath? TBD...

Tests time out on Travis

The tests are very slow, and even slower on Travis, where they have now started to time out regularly. This should be improved.

• Add measurement and get the timings of a test run. How long does it take to start and stop the containers? When does each test start and stop? Are things running in parallel or not? Output ideally to the console, so that we can test on Travis as well.
• Try to speed up the tests, possibly by manually parallelising the scheduler tests?
• Travis times out after not receiving any console output for 10 minutes. We have all the scheduler tests in a single file, which puts them on a single line, and it seems that there's a line buffer somewhere. Can we remove that buffer, or split the tests over multiple files so that Travis sees output more frequently?

FileSystem should have root()?

It'd be nice if we could get a Path to the root of a FileSystem using .root(). Workaround is to use fs / '', but that's not very pretty or clear.

ISTR that I had this at some point, then removed or decided against it. Should see if I can find back why...

File systems should be comparable

File systems are currently compared equal based on object equality, but they should be compared based on the attributes, so that e.g. (LocalFileSystem() / 'tmp') == (LocalFileSystem() / 'tmp'). That's currently False, but should be True.

Of course file systems may alias, e.g. (remote_sftp / 'test') == (remote_webdav / 'test') may actually refer to the same file, which Cerulean cannot know. So we'll never get it completely right. The current semantics are that if two paths compare equal, that they're for sure equal, but if they compare unequal they may still be equal. That's preserved with this proposal, so it's not making things worse at least, and it wouldn't be surprising to a programmer. A note in the documentation would still be good though.

Leftover print statement in Path.walk()

Remove...

Path.match is not implemented

Python's pathlib PurePath has a match() method, which tests the path against a glob-style pattern. We're not implementing it, but could, and therefore should.

Copy leaks paramiko exception

If the connection drops during a file copy operation, then a paramiko.ssh_exception.SSHConnectionError is raised. It would probably be better if you got an ordinary Python ConnectionError, or a Cerulean-specific exception type, as the fact that we use paramiko is really an implementation detail that should not affect the public API.

Backtrace:

  File "/usr/local/lib/python3.5/dist-packages/cerulean/copy_files.py", line 82, in copy
    copy_permissions, source_path, callback, 0, size)
  File "/usr/local/lib/python3.5/dist-packages/cerulean/copy_files.py", line 124, in _copy
    already_written, size)
  File "/usr/local/lib/python3.5/dist-packages/cerulean/copy_files.py", line 258, in _copy_dir
    permission, source_path.has_permission(permission))
  File "/usr/local/lib/python3.5/dist-packages/cerulean/path.py", line 516, in set_permission
    self.filesystem._set_permission(self.__path, permission, value)
  File "/usr/local/lib/python3.5/dist-packages/cerulean/sftp_file_system.py", line 283, in _set_permission
    self._chmod(lpath, mode)
  File "/usr/local/lib/python3.5/dist-packages/cerulean/sftp_file_system.py", line 288, in _chmod
    self.__sftp.chmod(str(lpath), mode)
  File "/usr/local/lib/python3.5/dist-packages/paramiko/sftp_client.py", line 541, in chmod
    self._request(CMD_SETSTAT, path, attr)
  File "/usr/local/lib/python3.5/dist-packages/paramiko/sftp_client.py", line 813, in _request
    return self._read_response(num)
  File "/usr/local/lib/python3.5/dist-packages/paramiko/sftp_client.py", line 845, in _read_response
    raise SSHException("Server connection dropped: {}".format(e))
paramiko.ssh_exception.SSHException: Server connection dropped: 

Filesystem / '/path' yields a double slash at the beginning

If you use the / operator on a file system with a string that starts with a /, then you'll end up with a Path that has a double / in it, which while logical is not what you want. Leading and trailing slashes should be stripped from these strings. (Note that forward slashes are illegal in file names both on Posix and on Windows.)