There is a zero division error when calculating progress; but even with that taken care of, there is some issue with multi-part-uploading an empty file.

This old discussion might be of use: kr/s3#30

I was trying to get the limits for my user on the API, but I get None for all the three calls:

import sevenbridges as sbg
c = sbg.Config(profile='sbpla')
api = sbg.Api(config=c)
# Let's print the rate limit:
print(api.limit) # prints None
print(api.remaining) # prints None
print(api.reset_time) # prints None

However, if I do call my user first I get the expected result:

api.users.me()
print(api.limit)
print(api.remaining)
print(api.reset_time)

Output

<User: username=mmattioni>
1000
999
2016-10-05 15:34:54

I guess this is a minor bug, or if it is intended to be used like this, the docs should be updated: http://sevenbridges-python.readthedocs.io/en/latest/quickstart/#rate-limit

I guess it's @SenadI and @damirkrstanovic call

Although sevenbridges-python has support for proxies, by perusing same methodology as requests library (setting environment variables), it would be nice to support proxies in the configuration, for example $HOME/.sbgrc would look like:

[sbpla]
auth-token = <TOKEN HERE>
api-url = https://api.sbgenomics.com/v2
http-proxy = http://proxy.company.com:3128
https-proxy = http://proxy.company.com:3128

In the Quickstart section of the documentation there is this example of an .sbgrc file:

[sbpla]
api-url = 'https://api.sbgenomics.com/v2'
auth-token = 700992f7b24a470bb0b028fe813b8100

[cgc]
api-url = 'https://api.sbgenomics.com/v2'
auth-token = 910975f5b24a470bb0b028fe813b8100

This example doesn't work (using a proper auth-token), because the URL should not be quoted. I'm using Python 3, apparently the issue does not come up in Python 2.

On the server there is an ability to disable batching for task. Expose this in sevenbridges-python

You have from version 0.2.0

Download a file to the current working directory

new_file.download(wait=True)

should be something like

new_file.download(path=new_file.name)

[via @jdagilliland, @ysaletore]

If a user passes in an id like
id = name/project/app_id/revision
then this method works. But since the method is also passed revision, this seems redundant - so the user might try instead
id = name/project/app_id

Now the app_id will be split off (instead of the revision) and there seems to be no check that it was a string. In the project, the user will have an app called
name/project/revision/0
where revision was passed into create_app_revision()

Currently collection can not be used for input, it has to be casted into list. This needs to be adapted user should not worry about the cast.

app.create_revision no longer functions as expected, when using id as returned by app.id, a revision number, and the raw app content.
The reason seems to be https://github.com/sbg/sevenbridges-python/blob/develop/sevenbridges/models/app.py#L100-L104, which seems meant to trim the version number off of an ID string.
The problem with this is that the ID string returned by the id property is already so trimmed: https://github.com/sbg/sevenbridges-python/blob/develop/sevenbridges/models/app.py#L31-L37.

pypi does not support markdown files only reStructuredText. This is a minor improvement so that the readme will be properly displayed on pypi.

When inspecting a file via the task.outputs property, I can't access any of the files properties, other than id. When I use id to query for the file directly, everything works fine. An example:

>>> task.outputs["some file"].download_info()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-22-75c8150e8f7e> in <module>()
----> 1 task.outputs["some file"].download_info()

[...]lib/python3.4/site-packages/sevenbridges/models/file.py in download_info(self)
    103         :return: Download info object.
    104         """
--> 105         info = self._api.get(url=self._URL['download_info'].format(id=self.id))
    106         return DownloadInfo(api=self._api, **info.json())
    107 

AttributeError: 'NoneType' object has no attribute 'get'

However, the following works just fine:

>>> api.files.get(id=task.outputs["some file"].id).download_info()
<DownloadInfo: url=[...]>

I am using Python 3.

Steps to replicate

1. Start a batch task
2. Get execution details
3. Retrieve jobs attribute
4. TypeError results

An example with an existing task is as follows

bt = api.tasks.get(id='f198c7a8-f246-48fc-b335-298fa64de0d3')    #  A completed batch task
bt.get_execution_details().jobs
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-569-c9cbeaba8660> in <module>()
----> 1 bt.get_execution_details().jobs

/Users/kghose/miniconda2/envs/bench/lib/python2.7/site-packages/sevenbridges/meta/fields.pyc in __get__(self, instance, owner)
     80     def __get__(self, instance, owner):
     81         return [self.cls(api=instance._api, **item) for item in
---> 82                 instance._data[self.name]]
     83 
     84 

TypeError: 'NoneType' object is not iterable

This works fine with regular tasks. Thanks

Noticed that the API throws an error when there are single quotes within the project description,

Code:
project_name = 'Copy of Cancer Cell Line Encyclopedia (CCLE)'
my_project = [p for p in api.projects.query(limit=100).all()
if p.name == project_name][0]
print('Project description: {} \n'.format(my_project.description))

Error:
UnicodeEncodeError Traceback (most recent call last)
in ()
---> 4 print('Project description: {} \n'.format(my_project.description))

UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in position 2299: ordinal not in range(128)

Library silently fails on api.tasks.get(''), it should throw an exception. The id of the resource being fetched should be validated before retrieval.

When I fetch a specific older revision of an app, and then start a task using that revision, the task will be spawned with the newest revision of that app:

>>> app = api.apps.get_revision(id='whatever', revision=0)
>>> task = api.tasks.create('name', project='some-project', app, inputs={}, run=True)
>>> task.app
'whatever/3'

I'm running version 0.5.3 on Python 3.5.

imp = api.imports.submit_import(volume=volume_import, project=my_project, location=bucket_location)
time.sleep(30)

Wait until the import finishes

while True:
      import_status = imp.reload().state

imp.reload() was returning None which lead to error
this is inspite of the import actually taking place
what worked for me was:
imp = api.imports.submit_import(volume=svolume_im, project=smy_project, location=sfilename)

Wait until the import finishes    

import_status = ''
time.sleep(30)
i = api.imports.get(id=imp.id)
print i.state

User flow

• make a project "my-project"
• copy a Public Reference file to "my-project"
• try to export one of those files with

for f in selected_files:
    export = api.exports.submit_export(
        file=f,                        
        volume=my_volume,                     
        location=f.name) 

I get the error

Forbidden: Requested file cannot be exported. Check the documentation.

But I don't have a link to "the documentation" so I don't know where to look. I don't see this mentioned in readthedocs and don't recall it on docs.sevenbridges.com either.

Cannot authenticate with OS environmental variables:

os.environ['API_URL'] = 'https://cgc-api.sbgenomics.com/v2'
os.environ['AUTH_TOKEN'] = authToken
api_config = sbg.Config()
api = sbg.Api(config=api_config)

gives:
proxies = config.proxies
AttributeError: 'Config' object has no attribute 'proxies'

Looking at source code in config.py ->

        self.auth_token = os.environ.get('AUTH_TOKEN')
        self.oauth_token = os.environ.get('OAUTH_TOKEN')
        self.api_url = os.environ.get('API_URL')
        if not self.auth_token and not self.oauth_token:
            raise SbgError(
                'auth-token or oauth-token environment variable missing.'
            )
        if not self.api_url:
            raise SbgError('api-url variable environment missing.')

self.proxies is never initialized in init function.

When SevenBridges/CGC API is under maintenance, APi returns http status code = 503 and the api code is 0. This can be added to check_error() decorator to handle this gracefully and fire UnderMaintenanceError exception.

Missing comma before 'run' here:

try:
task = api.tasks.create(name=name, project=project, app=app,
inputs=inputs, batch_input=batch_input, batch_by=batch_by run=True)

Issue

I used:

all_export_jobs = my_volume.get_exports(project=my_project)

for j in all_export_jobs:
    print('File {} export is {}'
         .format(j.destination, j.state))
print('\n')

and see the three files I exported from my_project and two other files I exported from other projects. Setting project does not seem to have the desired effect.

Additional detail

Files were exported using

# Loop through selected files, start one job for each.
exports = []
for f in my_files:
    export = api.exports.submit_export(
        file=f,                        
        volume=my_volume,                     
        location=f.name)

    exports.append(export)

    print("File {} is {} for export to your cloud storage \n"
          .format(f.name,exports[-1].state))

Retrieving non-current revisions of apps through the API is a challenge because the app IDs attached to the app objects include the number of the latest revision.
This means that when using the App.get_revision() function, it tries to append the desired revision number on to the current revision number (resulting in an invalid request).
I'm not quite sure how to fix this, because the instances of App that are retrieved through api.apps.query() already come with the revision number attached, but I imagine it would not be too hard for someone who knows the ins and outs of your API wrapper, because the return values of list-all-apps-available-to-you do not include revision numbers.

I stumbled across this while trying to prevent a script from bumping up against the rate limit all the time, but the rate limit attributes (which are fundamentally integral in nature) should in fact be integers.

I happened to be just plainly comparing them to an int, to see if I had enough requests remaining, but the strs were always comparing greater than or equal to n.

Make billing group a kwarg param. Incoming feature may require this. Make it so that it stays backwards compatible.

Apps can not be deleted. Remove the delete() method from api APP documentation on readthedocs.

f.delete() works well. I saw the URL in models-file but not the method for it. It would be good to have in readthedocs quickstart if possible

[via Erik L]

Running

# [USER INPUT] file names to upload:
file_list = ['files_listAll.ipynb',
            'files_copyFromMyProject.ipynb',
            'files_copyFromPublicReference.ipynb',
            'files_detailOne.ipynb',
            'files_upload_and_setMetadata.ipynb']

for f in file_list:
    api.files.upload(project = my_project, path = f)

# List all files in the project
my_files = api.files.query(project = my_project)
print('In project {}, you have {} files.\n'.format(
    my_project.name, my_files.total))
for f in my_files:
    print(f.name)

# Set file metadata
base_md = {
    'toy_example': True,
    'extension': 'ipynb',
    'revision_number': 7,
    'Hello':'Nope!'
}

for ii, f in enumerate(my_files):
    f.metadata = base_md
    if ii == 2:
        print(ii)
        f.metadata['Hello'] = "is it me you're looking for?"
    f.save()

# List files based on metadata
my_matched_files = api.files.query(
    project=my_project, 
    metadata = {'Hello' : "is it me you're looking for?"})

print('In project {}, you have {} matching files.\n'.format(
    my_project.name, my_matched_files.total))

for f in my_matched_files:
    print(f.name)

generates:

In project Shiny & New, you have 5 files.

files_upload_and_setMetadata.ipynb
files_copyFromPublicReference.ipynb
files_detailOne.ipynb
files_copyFromMyProject.ipynb
files_listAll.ipynb
0
1
2
2
3
4
In project Shiny & New, you have 3 matching files.

files_detailOne.ipynb
files_copyFromMyProject.ipynb
files_listAll.ipynb

Bug reported by jdagilliland - quickstart mentions save() method for apps, while there is no such method.

In order to create new app, create_revision() method should be used, for example:

app = api.apps.query(limit=1)[0]
new_version = app.create_revision(app.id,app.revision+1, raw = app.raw)

Additionally, user can manipulate raw content of the app - create new CWL manually, etc. before creating a new revision.

If I open a volume object via api.volumes.get(<ID>), and then try to edit components of its service object (in particular the elements of "credentials"), when I try to save my changes back to the platform, I get a ResourceNotModified exception, which tells me that "no relevant changes were detected in order to update the resource on the server".

e.g.

vol.service["credentials"]["access_key_id"] = "AKIAII4RAZIQUWAOB4QQ"
vol.service["credentials"]["secret_access_key"] = “<censored>"
vol.save()
...
ResourceNotModified: No relevant changes were detected in order to update the resource on the server.

The current operation and help for the class function api.files.copy is:

api.files.copy?
Signature: api.files.copy(self, project, name=None)
Docstring:
Copies the current file.
:param project: Destination project.
:param name: Destination file name.
:return: Copied File object.

This is expected for a file.copy method of a file object, but I expect api.files.copy to be a generalized copy function, taking a source file and a destination project with possibly a new name, all explicitly supplied to the function. The current function and help are ambiguous and unexpected.

Thanks!

Even though the metadata class subclasses dict, the typical dict methods like update do not work with it (properly), e.g.

my_files = api.files.query(project=project_id, names=[fname])
my_files[0].metadata.update(new_metadata)
my_files[0].save()
print(my_files[0].metadata)

Does not actually save the new metadata to the platform, although the last step seems to print the updated metadata that the user wishes had gone to the platform.

My best guess is that it has to do with the __setitem__ method; it seems that that method does special stuff with the metadata object (above and beyond the regular dict stuff), and perhaps that isn't being carried through to an update method call?

Given an app, it would be very useful to get a dictionary with the default inputs for that app. The use case for this is when trying to check if an app has previously been run with a given set of inputs. I can get all tasks for a given app, and then compare the inputs. However, since the inputs on the task objects contain all the default values, I cannot simply compare the dictionaries (also, see #6 for problems with comparing file inputs).

Transform is not working for python 2 str types. Please instead of returning resource when resource is str, return six.text_types(resource). This covers it!

I constantly receive "failed to complete: _submit_part" error while trying to upload a small FASTQ file (~81mb).

I reproduced this issue with multiple other files and I decided to test out different file sizes to see whether it's something related to the file size. I tried about 10 runs of attached script and almost in every trial upload failed for any file larger than 2 MB. Sometimes upload of a 2 MB file was also a failure, and one time a 5 MB file was successfully uploaded as well.

api_upload_test.txt

Also, a new exception was thrown while handling the original exception. The full error text was as following:

Exception in thread Thread-5:
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/sevenbridges/transfer/upload.py", line 509, in run
    for _ in parted_file:
  File "/usr/local/lib/python3.5/site-packages/sevenbridges/transfer/upload.py", line 182, in __iter__
    yield future.result()
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/concurrent/futures/_base.py", line 405, in result
    return self.__get_result()
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
    raise self._exception
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.5/site-packages/sevenbridges/transfer/upload.py", line 101, in _upload_part
    session, part_url, part, timeout
  File "/usr/local/lib/python3.5/site-packages/sevenbridges/decorators.py", line 52, in wrapper
    threading.current_thread().getName(), f.__name__)
sevenbridges.errors.SbgError: Thread-4: failed to complete: _submit_part

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.5/site-packages/sevenbridges/transfer/upload.py", line 528, in run
    raise SbgError(six.text_type(e))
sevenbridges.errors.SbgError: Thread-4: failed to complete: _submit_part

I'd appreciate any pointers.

macOS Sierra 10.12.2, Python v3.5.2, sevenbridges-python v0.6.1

might be a nice feature to be able to pick a prior version of the app, was straightforward with the HTTP calls. low priority

Environment - Python 2.7, miniconda, OSX, also reported to cause issues on Windows 7.
Code to reproduce:

bgs = api.billing_groups.query(limit=1)
new_proj = api.projects.create(name='test123',billing_group=bgs[0].id)

Throws exception:

---------------------------------------------------------------------------
SbgError                                  Traceback (most recent call last)
<ipython-input-3-ea199f5c776e> in <module>()
----> 1 new_proj = api.projects.create(name='test123',billing_group=bgs[0].id)

/usr/local/lib/python2.7/site-packages/sevenbridges/models/project.pyc in create(cls, name, billing_group, description, tags, api)
     68         api = api if api else cls._API
     69
---> 70         billing_group = Transform.to_billing_group(billing_group)
     71         if name is None:
     72             raise SbgError('Project name is required!')

/usr/local/lib/python2.7/site-packages/sevenbridges/meta/transformer.pyc in to_billing_group(billing_group)
     73             return billing_group
     74         else:
---> 75             raise SbgError('Invalid billing group parameter!')
     76
     77     @staticmethod

SbgError: Invalid billing group parameter!

Both using string or BillingGroup object works without issues, but using billing_group.id causes this issue.

Currently the only way to get tasks newer than a given date is to fetch all tasks and then filter. This is impractical for projects that contain large numbers of tasks. It would be great to either have direct support for this, or the option to iterate over the collection of tasks sorted by decreasing start-time.

Right now the upload function does not return the file identifier, so a user does not have the ability to update the metadata just right off it.

Right now it looks like:

api.files.upload(name, to_project)
file_uploaded_identifier = self.to_platform.files.query(to_project, names=[name])[0]
# File operation

Would be nicer to have somenthig like:

file_uploaded_identifier = api.files.upload(name, to_project)
# file operation here for example metadata

Just as submit_export does. Thanks!

Currently when a list of file names is passed into the query function the returned order is not related to the list of file names supplied. In many cases it would be very useful to have this ordering. Thanks!

I tried the async uploader and got this message - not sure what it is about.

When copying task inputs from another task the api repr somehow ends up in the inputs object. When the inputs are saved next time wrapper will throw Json Not Serializable Exception

Fetch a file
f=api.files.get(id='57574236e4b083c873cc5eba')

Check name

f.name
Out[191]: u'haha.i.changed.this'

Check metadata (I believe this fetches the metadata from the platform. I like it's lazy nature!)

f.metadata
`` (There's no metadata)

Change file name
f.name = 'oops.i.changed.it.again'

Verify the local version has the changed file name
f.name
'oops.i.changed.it.again'

Now ask to see the metadata again. What I did not expect was that a) this would be a fresh fetch of the metadata and, b) this would overwrite the file name

f.metadata
`` (No metadata as expected)

f.name
'haha.i.changed.this' <---- Zowie! Was not expecting this

Need to do f.save() first

This code always fails for me. I tried it with three different files in this project. I can change the name fine via the UI. Thanks!

import os
import sevenbridges as sbg

api = sbg.Api(url=os.environ['SBG_API_URL'], token=os.environ['SBG_AUTH_TOKEN'])
f = api.files.get(id='57c3337de4b06ea683b426bd')
# f = api.files.get(id='57c3337de4b06ea683b426bf')
# f = api.files.get(id='57c3337de4b06ea683b426bb')
print(f.name)

f.name = 'this.txt'
f.save()
print(f.name)

Output:

f2.txt
Traceback (most recent call last):
  File "file_save_api_bug.py", line 11, in <module>
    f.save()
  File "/Users/kghose/miniconda2/envs/bench/lib/python2.7/site-packages/sevenbridges/decorators.py", line 23, in wrapped
    api_object = method(obj, *args, **kwargs)
  File "/Users/kghose/miniconda2/envs/bench/lib/python2.7/site-packages/sevenbridges/models/file.py", line 204, in save
    url=self._URL['get'].format(id=self.id), data=modified_data
  File "/Users/kghose/miniconda2/envs/bench/lib/python2.7/site-packages/sevenbridges/http/client.py", line 170, in patch
    data=data, append_base=append_base)
  File "/Users/kghose/miniconda2/envs/bench/lib/python2.7/site-packages/sevenbridges/decorators.py", line 100, in wrapper
    raise SbgError(message=str(e))
sevenbridges.errors.SbgError: No JSON object could be decoded

Currently file objects cannot be compared in a meaningful way. Example:

>>> api.files.get(id='asdf') == api.files.get(id='asdf')
False

It is possible to compare them by their id attribute, but that becomes difficult when comparing lists of files.