closeio / closeio-api Goto Github PK

We should:

• Make sure our client works well in Python v3.x.
• Make Circle run tests on 2.7 + 3 (similar to https://github.com/closeio/tasktiger/blob/master/.circleci/config.yml).
• Add a classifier to setup.py.

Guys, I would have made a pull request, but it seems that you didn't updated the repository with the latest version of the API published at Pip.

So, you must change the name of the variable due to it had become a reserved keyword in you init file:
def init(self, base_url, api_key, tz_offset=None, async=False, max_retries=5):

Hello, I'm trying to use your APIs implementation to retrieve data of our leads (about 45k records).
While the number of retrieved records is correct, there are a lot of duplicates (about 8k) that don't match the number of duplicates i get from the export from the close.io website. Any idea about what could be causing that?

Example: a Lead has 3 custom fields, I want to update one of them. If I sent an update with only that one field, it seems to null out the other 2 fields.

Do I really need to do a round trip to get existing values, merge in my change, then send the update?

Try:

  $opportunity = new Opportunity();
  $opportunity->setConfidence("50");

  $lead->setOpportunities(array($opportunity));
  $response = $leadsApi->addLead($lead);

Nothing happens :(

We need a script that associates contact-less activities (i.e. calls and emails with contact_id: null) with given lead's contacts based on a phone number or an email address.

Traverse through all the leads for given API key and:

• match contacts based on email['envelope']['to']['email'] if email['direction'] == 'outgoing' else email['envelope']['from']['email']
• match contacts based on call['remote_phone']
• if multiple contacts match a given phone number/email, pick the first matching contact
• at the end, print a report of activities where contact_id is empty and no matching contacts were found

Should take a search query as input and 2 API keys (from different orgs)

Transfer all the leads matching the search query

As well as all the nested objects on the leads (Contacts, Tasks, Opportunities), and the Calls & Note Activities

Emails don't need to transfer since we'll sync them ourselves but all other lead data should transfer over.

If lead statuses or opportunity statuses don't exist it should clearly print out that you need to set up those statuses within the destination organization first.

It would be very helpful to have this on PyPI. Dependencies would be easily resolved when installing, and it would not need to be vendored into projects (making upgrading easier, etc).

https://github.com/ross/requests-futures

It's the future: spyoungtech/grequests#22

Also handles errors much better: #7

Can you add some release tags so I can reference them in my requirements.txt file? It's not that big as you can reference individual commits but using tags would be much nicer.

for the from/to params

The user_reassign script doesn't update all the objects. It paginates through objects that match certain criteria (i.e. the old user), reassigns them to the new user, and then increases the offset to fetch the next batch. The problem is that when we reassign old objects, the results don't contain them anymore, and we end up skipping objects. We instead need to keep the offset at 0 and loop until no more objects are left (except for in the dry run).

Currently the code is assuming that we will have some JSON to parse out when we receive 429.

https://github.com/closeio/closeio-api/blob/master/closeio_api/__init__.py#L111-L116

This is no longer true in our current infrastructure setup as you can get rate limited on the ingress level instead of the app level, if it happens there will be no JSON payload to parse and decide how many seconds to sleep.

We should have a strategy to backoff in such case, maybe just a very pessimistic sleep but not sure how exactly we would want to do it.

We support OAuth now: https://developer.close.com/topics/authentication-oauth2/

At very least we should support authentication with Authorization: Bearer ... header but there are many other aspects of OAuth like token expiration. refreshing, invalidation etc. Probably needs a little bit of design upfront.

Default user agent should include something like "python closeio v{VERSION} {whatever requests already includes}" so we can identify usages of this client and which version people are on.

Hi all,

I'm using your library on Python 3 and the current version 1.0 released on PyPi raise an error. The problem was already found and solved in #89 but wasn't released on PyPi.

The only way I found to use it, is pointing directly to Github but this is tricky using setuptools due to some open problems (here are some links to implement a workaround [1], [2]).

A version bump and a release on PyPi would solve everything.

Thanks

Have been using this and I ended up getting very confused by the fact that it wasn't actually editing my data till I added this argument.

I suggest within the wiki that you should mention this step to actually perform the action to merge the contacts. I assumed that during the print out the action was being taken when actually it was just displaying without taking the action.

This repo has no OSS license.

lead_results = api.get('lead', params={
    '_limit': 10,
    '_fields': 'id,display_name,status_label',
    'query': 'custom.my_custom_field:"some_value" status:"Potential" sort:updated'
})

should be changed to:

lead_results = api.get('lead', params={
    '_limit': 10,
    '_fields': 'id,display_name,status_label',
    'custom.my_custom_field':'some_value',
    'status': 'Potential',
    'sort':'updated'
})

Trying to run the following query will not return a filtered result:

api.get('activity/status_change/opportunity', {'_limit':100, '_fields': 'opportunity_id,lead_id', 'query':'opportunity_id:"some_id"'})

Hey close.io devs, this is Nick Sweeting with drchrono, thanks for writing this python library, it makes our lives a lot easier.

I'm writing a PR for the TODO you guys put in init.py:71
grequests now supports passing an exception_handler function so we no longer have to deal with this APIError() monkey business. I've started updating the code and should have a pull request ready in a few days.

the new code will follow this pattern:

def map(self, reqs, max_retries=None):
    import grequests
    max_retries = max_retries or self.max_retries

    retries = 0
    failed = []
    responses = grequests.map(reqs, exception_handler=lambda req, excp: failed.append(req))

    while failed and retries < self.max_retries:
        succeeded = grequests.map(failed, exception_handler=lambda req, excp: pass)       # leave the ones that continue failing in failed
        failed = [ req for req in failed if req not in reversed(succeeded) ]              # checking if reversed succeeded contains our request is faster because we append good ones to the end
        responses += succeeded
        retries += 1
        time.sleep(2)

    return [resp.json() for resp in responses]

python scripts/merge_leads.py --field email -k API_KEY

Traceback (most recent call last):
  File "scripts/merge_leads.py", line 206, in <module>
    pbar.update(pbar.currval + len(leads))
  File "/home/sam/sources/closeio-api/venv/local/lib/python2.7/site-packages/progressbar/progressbar.py", line 252, in update
    raise ValueError('Value out of range')
ValueError: Value out of range

Added a bit of logging and I could see that in the first iteration it found 1371 leads:

2016-10-17 15:30:26,676 - closeio.api.merge_leads - INFO - Total Leads: (1371), Offset: (0)

Then at some point it tries to update from 1300 with 100 and thus value out of range.

2016-10-17 15:40:29,934 - closeio.api.merge_leads - INFO - Currval: (1300), len(leads) (100)

Is this assumption correct total_leads = resp['total_results']?

user_reassign.py

Input:

• API key
• --confirmed flag
• --from-user-id
• --to-user-id
• --tasks: if given, update tasks
• --opportunities: if given, update opportunities

Either --tasks or --opportunities (or both) should be specified

What the script should do:

• Assign all tasks and/or opportunities that are assigned from one user to another

(closeio)--- Projects/closeio-api ‹master» python scripts/bulk_update_leads_info.py ~/Downloads/test.csv --api-key <redacted> --confirmed
[2015-05-15 18:59:23,018] INFO Starting new HTTPS connection (1): app.close.io
[2015-05-15 18:59:23,539] INFO line 2 updated: lead_iyTYlkzfVWKn0rH0x1yGrPX2vMOzzCsOdBK205WrzLk Plumbing Medic
[2015-05-15 18:59:23,539] INFO summary: updated[1], new[0], skipped[0]
(closeio)--- Projects/closeio-api ‹master» cat ~/Downloads/test.csv
lead_id,contact_name,contact_phone,,,,,,
lead_iyTYlkzfVWKn0rH0x1yGrPX2vMOzzCsOdBK205WrzLk,Conference Call Line,+1818-452-3980,,,,,,

No contact added to https://app.close.io/lead/lead_iyTYlkzfVWKn0rH0x1yGrPX2vMOzzCsOdBK205WrzLk/

Currently, pip install closeio install version 0.3 of the API client, and pip install closeio_api installs version 0.1 of the API client. setup.py claims the latest version is 0.4, and the latest version in master is different, too. We should unify this, and also update the README.

Let's have this repo just be our Python API client and move example scripts to their own repo called closeio-api-scripts

bulk_update_address_countries.py

inputs:

• api key
• old country code
• new country code
• leads search query (defaults to * sort:created if none) -- can be coded in script since it's hard to pass complicated queries (e.g. quotes) to cli args

python cli script that takes these items as inputs. it should loop through each lead in the given search query, and if there are any addresses on that lead with the old country code then it should update the address with the new country code

Would also like a read-only preview mode of what will happen and then a --confirmed flag

@congocongo Can you work on this one after the other script?

When trying to create a new lead as follows:

from closeio_api import Client
import os

api_key = os.getenv('CLOSEIO_API_KEY') 
api = Client(api_key)
api.post('lead', data={'name': 'Test lead'})

I get the following error: TypeError: expected string or bytes-like object

Apparently, the problem is the value of the X-TZ-Offset header used by default:

from closeio_api.utils import local_tz_offset
type(local_tz_offset()) # -> float

Converting that value to a string seems to fix the problem:

api.tz_offset = str(api.tz_offset)
api.post('lead', data={'name': 'Test lead'}) # Success!

Environment:

• Ubuntu 15.10
• python 3.5

Would love a more robust example of creating a Lead and a Contact as a single POST request..

dispatch function breaks when number of retries is greater than self.max_retries:

    def dispatch(self, method_name, endpoint, data=None):
        method = getattr(self.requests, method_name)

        retries = 0
        while True and retries < self.max_retries:
            try:
            response = method(...)
                break
            except requests.exceptions.ConnectionError:
                time.sleep(2)
                retries += 1

        if self.async:
            return response
        else:
            if response.ok:
                return response.json()
            else:
                raise APIError(response.text)

since there's ConnectionError, response is never defined.

When trying to run the merge_leads.py script, I get this error:

Traceback (most recent call last):
  File "./merge_leads.py", line 47, in <module>
    is_duplicate = last_lead and lead['name'].strip() and last_lead['name'].strip().lower() == lead['name'].strip().lower()
AttributeError: 'NoneType' object has no attribute 'strip'

I believe this is because if lead doesn't have a name, it returns None.

When a request fails because of an aborted connection or other unhandled exception the resulting APIError and the stack trace generated with it is not very informative. We should make the string representation of APIError include more detail.

We want to update the behavior of our api and client to follow the draft RFC related to communicating rate limiting.

As an example, today I ran this request:

resp = api.get('report/activity/ORGID', params=
{'user_id':'USERID', 'date_start':'2018-01-02', 'date_end':'2018-01-02'})

The response I got back was:

{
    "revenue_created_annual_created": 0, 
    "opportunities_created": 0, 
    "revenue_lost_annual_created": 0, 
    "sms_sent": 0, 
    "leads_created": 0, 
    "opportunities_lost": 0, 
    "revenue_won_monthly": 0, 
    "revenue_won_annual": 0, 
    "leads_contacted": 3, 
    "revenue_won_one_time": 0, 
    "revenue_lost_annual": 0, 
    "revenue_lost_one_time_created": 0, 
    "revenue_lost_one_time": 0, 
    "revenue_created_monthly": 0, 
    "revenue_won_annual_created": 0, 
    "opportunities_won": 0, 
    "opportunities_created_created": 0, 
    "emails_sent": 0, 
    "revenue_created_monthly_created": 0, 
    "emails_received": 0, 
    "calls_duration_total": 0, 
    "sms_received": 0, 
    "calls_duration_average": 0, 
    "revenue_won_one_time_created": 0, 
    "revenue_created_one_time": 0, 
    "revenue_won_monthly_created": 0, 
    "revenue_lost_monthly_created": 0, 
    "opportunities_won_created": 0, 
    "opportunities_lost_created": 0, 
    "revenue_lost_monthly": 0, 
    "_queries": {
        "leads_contacted": "call(duration > 0  date >= 2018-01-01 date <= 2018-01-01) or email(direction:sent  date >= 2018-01-01 date <= 2018-01-01) or sms(direction:sent  date >= 2018-01-01 date <= 2018-01-01)", 
        "emails_received": "email(direction:received  date >= 2018-01-01 date <= 2018-01-01)", 
        "calls": "call( date >= 2018-01-01 date <= 2018-01-01)", 
        "opportunities_created": "opportunity( created >= 2018-01-01 created <= 2018-01-01)", 
        "opportunities_won_created": "opportunity(status_type:won  closed >= 2018-01-01 closed <= 2018-01-01)", 
        "opportunities_lost_created": "opportunity(status_type:lost  lost >= 2018-01-01 lost <= 2018-01-01)", 
        "sms_sent": "sms(direction:sent  date >= 2018-01-01 date <= 2018-01-01)", 
        "leads_created": " created >= 2018-01-01 created <= 2018-01-01", 
        "opportunities_won": "opportunity(status_type:won  closed >= 2018-01-01 closed <= 2018-01-01)", 
        "opportunities_created_created": "opportunity( created >= 2018-01-01 created <= 2018-01-01)", 
        "emails_sent": "email(direction:sent  date >= 2018-01-01 date <= 2018-01-01)", 
        "sms_received": "smsdate >= 2018-01-01 date <= 2018-01-01)", 
        "opportunities_lost": "opportunity(status_type:lost" lost >= 2018-01-01 lost <= 2018-01-01)"
    }, 
    "revenue_created_one_time_created": 0, 
    "calls": 0, 
    "revenue_created_annual": 0
}

since the bounding dates are 2018-01-01 and 2018-01-01, it means that we aren't correctly factoring in timezone when passing through date params.

We need to add the tz.offset*-1 to the datetime in both directions for it to appear the same as it does in app.

We need a script that takes a CSV file with some or all of these columns:

• lead_id
• company
• description
• url
• status
• contact1_name
• contact1_title
• contact1_phone1
• contact1_email1
• contact1_url1
• custom.foo bar ("foo bar" can be any valid custom field that already exists)

The basic idea is that the script should help you bulk edit leads in Close.io and bulk-add contacts to existing leads. For lead-specific fields (description, url, status, custom fields) it will update the lead and it will always add contacts to the leads.

• Have a --confirmed flag and the script should be read-only for testing until this is set. See https://github.com/elasticsales/closeio-api/blob/master/scripts/move_custom_field_to_contact_info.py for an example but feel free to improve it
• Make sure the script is well documented with command line instructions. Assume the user isn't sure how it works :)
• Assume that the user running the script will make a mistake with the headers so put in some nice checks to warn them if the script has unrecognized headers and make them correct it before continuing.
• Use the custom fields endpoint to make sure that any custom fields listed are valid
• Per row there can be up to 9 contacts (e.g. contact9_name) and each contact can have up to 9 phone numbers (e.g. contact5_phone3), emails, and urls.
• Require that either lead_id or company name must be there
• Some rows may not have all the fields filled out. Some rows may be trying to add 3 contacts some rows may be trying to add 1 contact.
• Loop through each CSV row:
  • Lookup to find an existing lead to edit. IF lead_id is a column in the CSV then only edit based on the lead_id.
  • If lead_id doesn't exist then for each row look up the lead to update based on the 'company' field. Use the ?query param with a "company:" prefix to do this. If you find multiple leads you should print a warning about this but if its in confirmed mode (see below) then just pick the first lead.
  • By default you'll create new leads if you don't find a lead from the lead_id or company name lookup. But have a --disable-create option to turn this off.
• Have a nice printout of which rows were rejected due to 400 errors / bad data / validation errors. The user can fix those rows and then re-run if they want.

@congocongo Can you work on this asap? If your trial account is expired just shoot an email to [email protected] asking for us to extend it b/c you're working on this script. Let me know if anything needs any clarification :)

Hi

I've followed the README instructions but Python still can't find closeio_api when I'm executing the merge_leads script. What could be wrong? I guess somehow the closeio_api isn't being included when I'm executing the script.

(venv) TK-Max:closeio-api tk$ pwd
/Users/tk/closeio-api
(venv) TK-Max:closeio-api tk$ python ./scripts/merge_leads.py
Traceback (most recent call last):
  File "./scripts/merge_leads.py", line 40, in <module>
    from closeio_api import Client as CloseIO_API
ImportError: No module named closeio_api

If I install closeio_api through pip it will get past this error but then there it seems to be incompatible with what the script expects from the get command:

(venv) TK-Max:closeio-api tk$ python ./scripts/merge_leads.py  --api-key API_KEY
Traceback (most recent call last):
  File "./scripts/merge_leads.py", line 173, in <module>
    '_fields': 'id,display_name,name,contacts,status_label,opportunities'
TypeError: get() got an unexpected keyword argument 'params'

How do I include the closeio_api since python doesn't seem to pick it up

Thanks

Currently the client sends the Content-Type header with every request.

https://github.com/closeio/closeio-api#running-a-script

Change:
$ cd closeio-api

To:
$ cd closeio_api

The library mentions using a development server but just refers to localhost, is there a specific local implementation anywhere that we can download and run with fake data or any ideas around implementing a test endpoint in the future at api.close.io?

Drop the io from pypi package name, python module and user-agent (#104)

We should have unit tests for this package + CircleCI + tox (to ensure PY2 & PY3 compat).

For both the confirmed and dry run cases it should end with printing:

total tasks / opportunities
total tasks / opportunities that had errors

When integrating with the Close API, we need to include a forward slash at the end of every API call to avoid an unnecessary (+200ms) redirect. Is this correct?

Also listed at https://apitracker.io/a/closeio

(closeio-api)--- lib/closeio-api ‹master» ./scripts/user_reassign.py --to-user-id user_wWDo6ETmCC9mSL3YQt0wdierIIKfZnR0LShSiVxEXBF --from-user-id user_fLuLxKSVFjJTePCe660YI2f2b5JpwoBjWQaPqWUjm5V -k <redacted> --all-tasks --all-opportunities -c

Running the above command seems to modify the tasks and opportunities but if you run it again the see the same output / nothing actually changes.