servicenow / pysnc Goto Github PK

from master to main

tried using get_attachments() and it gave me zero records back when manually querying them worked. Need to figure out why.

Wrong functions called and unclosed sockets e.g.

test_serialize_all_batch (test_snc_serialization.TestSerialization) ... /Users/matthew.gill/.pyenv/versions/anaconda3-5.3.1/lib/python3.7/_collections_abc.py:848: ResourceWarning: unclosed <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.2.177', 61001), raddr=('149.96.6.119', 443)>
  for key, value in kwds.items():
/Users/matthew.gill/.pyenv/versions/anaconda3-5.3.1/lib/python3.7/_collections_abc.py:848: ResourceWarning: unclosed <socket.socket fd=9, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.2.177', 61002), raddr=('149.96.6.119', 443)>
  for key, value in kwds.items():
/Users/matthew.gill/.pyenv/versions/anaconda3-5.3.1/lib/python3.7/_collections_abc.py:848: ResourceWarning: unclosed <socket.socket fd=10, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.2.177', 61004), raddr=('149.96.6.119', 443)>
  for key, value in kwds.items():
/Users/matthew.gill/.pyenv/versions/anaconda3-5.3.1/lib/python3.7/_collections_abc.py:848: ResourceWarning: unclosed <socket.socket fd=12, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('192.168.2.177', 61005), raddr=('149.96.6.119', 443)>
  for key, value in kwds.items():
ok

Include and doc-default to the usage of mTLS

I see in the dev notes this hasn’t been implemented yet. But it’s not marked as “not possible”, so it seems to be possible. I’m having trouble finding resources online for doing this. Does anyone know the API request to do a set_new_guid_value on a record in service now? If someone can point me in the right direction, I can help implement. thanks

We can use the API to query, then execute UI Actions. The following is the current method to execute:

params = dict(sysparm_table='x_snc_some_table', sysparm_sys_id=sys_id)
r = self.client.session.post(f"{self.client.instance}/api/now/ui/ui_action/{ui_action_sys_id}", params=params, json={})

But we could easily make this part of the client. Something like self.client.execute_ui_action(...) or perhaps even a self.client.UiAction(table, ...) deal like attachment?

Similarly the record object has a gr.add_attachment so maybe we could have also gr.exeute_ui_action?

'GlideRecord.add_encoded_query()' method should throw any error if the URL length limit is crossed, this will help in debugging script

when ipaccess control is enabled, a 403 is returned with HTML. however client.py:116 attempts to parse it as json.

Need to handle this error path so it's more debuggable

The topic of asynchronous requests was brought up. This seems mostly useful for the get() method.

Perhaps something like:

gr.get('sysidhere', lambda x: for e in x: print(e))

or we can deal with Futures

future = gr.get('sysidhere')
future.result()

...

await gr.get('sysidhere')

Or

client.await([gr.get('asdf'), gr.get('1234')])

setup.py is a clunker and poetry just makes more sense than pipenv

https://docs.python.org/3/library/pickle.html#pickle-protocol
https://stackoverflow.com/questions/1939058/simple-example-of-use-of-setstate-and-getstate

It would be useful to read/write these to disk.

Uses the legacy mobile oauth provider which isnt out of box on newest instances. Deprecate this feature and update unit tests.

ERROR: test_oauth (test_snc_auth.TestAuth)
ERROR: test_oauth_builtin (test_snc_auth.TestAuth)

By exporting Query and QueryCondition classes in the __init__.py file will help to create custom queries programatically.

from pysnc import Query, QueryCondition, GlideRecord

# Using Query
q = Query("incident")

q.add_active_query()
q.add_query("description", "=", "123456")

raw_query = q.generate_query()

gr = GlideRecord("ServiceNowClient", "incident")

gr.add_encoded_query(raw_query)

# Using now the QueryCondition
q = QueryCondition("description", "CONTAINS", "testing")
q.add_or_query("impact", "=", "2")

raw_condition = q.generate()

gr = GlideRecord("instance", "incident")
gr.add_encoded_query(raw_query)

Another important point will be to unify the interface for Query and QueryCondition. Perhaps, Query.generate_query() and QueryCondition.generate()

https://docs.servicenow.com/bundle/sandiego-application-development/page/integrate/inbound-rest/concept/batch-api.html

we need to support efficient operations for large data set modifications/etc.

project should build, test, and release via GH actions

Support the headers

X-WantSessionDebugMessages: true
X-WantSessionNotificationMessages: true

For:

"session":{"notifications":[],"debug_logs":[]}

These, however, require actual sessions

Hello, first of all, amazing work on this API! I've just started using it today and it already helped me A LOT.

For my question: does it support dot walking the reference fields, like gr.caller_id.user_name? It seems that caller_id is simply a string and not an object pointing to the sys_user, or am I wrong?

I'm trying to extract data from two tables that are linked using cmdb_rel_ci. I was expecting to be able to add a join_query as described by https://servicenow.github.io/PySNC/user/advanced.html#join-queries but it seems that this is a functionality to query linked or dotwalked attributes (not sure how they are called in servicenow - reference field?) with join_table_field.

This is pseudo code I would like to be able to write :

gr = client.GlideRecord('cmdb_ci_server')
join_query = gr.add_join_query('cmdb_ci_ip_address', join_relationship_field='Owned by::Owns')
gr.query()

Inspired by schema https://docs.servicenow.com/en-US/bundle/tokyo-servicenow-platform/page/product/configuration-management/concept/class-server.html

That would return records of servers with their related ip_address.

Is this already possible with the existing code ?

If you specify gr.fields = 'active,short_description' for example and then try to update it will error - because sys_id is never pulled and we don't know what record we're pulling.

Should always pull sys_id no matter what.

When doing something as such:

gr = client.GlideRecord(table)
gr.initialize()
gr.field = 'some value'
gr.get('non_existant_id')

We expect the gr.field to now be None -- in actuality it is still some value as the failed get() did not update _current.

Actual GlideRecord behaves in this matter:

let gr = new GlideRecord('incident')
gr.initialize()
gr.short_description = 'this is a test'
gs.info(gr.short_description)
gr.get('sys_id')
gs.info(gr.short_description)

will give us

my_scope: this is a test
my_scope: 

On insert, for example, you get a 200 return with a JSON blob instead of anything meaningful.

The ask is to improve feedback to make this easier to debug.

In [26]: gr.insert()
---------------------------------------------------------------------------
InsertException                           Traceback (most recent call last)
<ipython-input-26-38bda16632a5> in <module>
----> 1 gr.insert()
/usr/local/lib/python3.8/site-packages/pysnc/record.py in insert(self)
    304             raise AuthenticationException(response.json()['error'])
    305         else:
--> 306             raise InsertException(response.json(), status_code=code)
    307
    308     def update(self):
InsertException: (InsertException(...), "200 - {'result': [...

Good afternoon,
First, thank you for all the work you've put into creating this Python package.

Problem Statement
to_pandas() call is extremely slow if the returned data has a lot of columns.

Example: a simple query to the user table will return 7 rows of data, but will also return 100-200 columns. The to_pandas() call in this case will take several minutes to complete even though there are only 7 rows.

Request

1. Print a warning when running that function if number of columns is larger than 20.
2. Update the function docstring and recommend specifying the desired fields with gr.fields

Hi Team, hopefully this is right place to ask, if not, I'd appreciate if you can direct me.

I'm the founder of cloudquery.io, a high performance open source ELT framework.

Our users are interested in a ServiceNow plugin, but as we cannot maintain all the plugins ourselves, I was curious if this would be an interesting collaboration, where we would help implement an initial source plugin, and you will help maintain it.

This will give your users the ability to sync ServiceNow APIs to any of their datalakes/data-warehouses/databases easily using any of the growing list of CQ destination plugins.

Best,
Yevgeny

When using record = gr.pop_record() to extract and item, if I try to modify it and then update it record.update(), it fails with

AttributeError: 'NoneType' object has no attribute '_put'

In the following knowledge base documentation : https://support.servicenow.com/kb?id=kb_article_view&sysparm_article=KB0854577 a ReST API to extract information from relations with a depth level seems to indicate the possibility of getting related items in ServiceNow. This type of functionality would be super useful for the way we use PySNC. Any plans of looking into this and offering a binding to this functionality ?

(probably related is #54)

If i'm doing python for x in y i should not have to call rewind - it should always just start from 0

Good afternoon! 👋

I ran into an interesting error today when querying a table with pysnc: JSONDecodeError: Expecting value: line 1 column 1 (char 0).

Looking at the full error trace (attached at the bottom of this issue) I see that the line that is failing is in pysnc/client.py in _validate_response. It appears as though it is attempting to parse the server response as json but unfortunately the data that is being returned looks like HTML (at bottom of this issue).

Using my dapper debugger, I see that the pertinent lines in this HTML response are http.400 and java.lang.IllegalArgumentException: Request header is too large. It doesn't like the size of my request. If I make my request smaller and send it again it does not error out.

I would recommend modifying the below piece of code to first check whether the returned data is json or not before attempting to parse it. Let me know if you have questions!

pysnc/client.py in _validate_response(self, response)
    116             raise AuthenticationException(rjson)
    117         if code == 400:
--> 118             rjson = response.json()
    119             raise RequestException(rjson)

Repro steps

sys_ids = [<SYS_IDS>] # place 300+ sys ids here
company_sys_ids_joined = ",".join(sys_ids)

gr = client_cpt.GlideRecord('<TABLE_NAME>')
query = 'companyIN' + company_sys_ids_joined # this is a string that contains ~300 sys ids
gr.add_encoded_query(query)
gr.fields = ['company',  'name']
gr.query()

HTML server response

<html><head><title>ServiceNow - Error report</title><style><!--h1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} h2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} h3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} body {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} b {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} p {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;} a {color:black;} a.name {color:black;} .line {height:1px;background-color:#525D76;border:none;}--></style> </head><body><h1>HTTP Status 400 – </h1><HR size="1" noshade="noshade"><p><b>type</b> Exception Report</p><p><b>Message</b> <u></u></p><p><b>Description</b> <u>http.400</u></p><p><b>Exception</b> <pre>java.lang.IllegalArgumentException: Request header is too large
	org.apache.coyote.http11.Http11InputBuffer.fill(Http11InputBuffer.java:726)
	org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:464)
	org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:682)
	org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
	org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:810)
	org.apache.tomcat.util.net.Nio2Endpoint$SocketProcessor.doRun(Nio2Endpoint.java:1777)
	org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	org.apache.tomcat.util.net.AbstractEndpoint.processSocket(AbstractEndpoint.java:1082)
	org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$2.completed(Nio2Endpoint.java:569)
	org.apache.tomcat.util.net.Nio2Endpoint$Nio2SocketWrapper$2.completed(Nio2Endpoint.java:547)
	java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
	java.base/sun.nio.ch.Invoker$2.run(Invoker.java:219)
	java.base/sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
	java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	java.base/java.lang.Thread.run(Thread.java:834)
</pre></p><p><b>Note</b> <u>The full stack trace of the root cause is available in the server logs.</u></p><HR size="1" noshade="noshade"><h3>ServiceNow</h3></body></html>

Error trace

---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
<ipython-input-25-cdc177364535> in <module>
----> 1 get_company_instances(0)

<ipython-input-24-98142c971225> in get_company_instances(index_start, index_end)
     13     gr.add_encoded_query(query)
     14     gr.fields = ['company',  'name']
---> 15     gr.query()
     16 
     17     return pd.DataFrame(gr.to_pandas())

~/Documents/projects/sandbox-python/.venv/lib/python3.9/site-packages/pysnc/record.py in query(self)
    237             :RequestException: If the transaction is canceled due to execution time
    238         """
--> 239         response = self._client._list(self)
    240         code = response.status_code
    241         if code == 200:

~/Documents/projects/sandbox-python/.venv/lib/python3.9/site-packages/pysnc/client.py in _list(self, record)
    125                              params=params,
    126                              headers=dict(Accept="application/json"))
--> 127         self._validate_response(r)
    128         return r
    129 

~/Documents/projects/sandbox-python/.venv/lib/python3.9/site-packages/pysnc/client.py in _validate_response(self, response)
    116             raise AuthenticationException(rjson)
    117         if code == 400:
--> 118             rjson = response.json()
    119             raise RequestException(rjson)
    120 

~/Documents/projects/sandbox-python/.venv/lib/python3.9/site-packages/requests/models.py in json(self, **kwargs)
    898                     # used.
    899                     pass
--> 900         return complexjson.loads(self.text, **kwargs)
    901 
    902     @property

~/.pyenv/versions/3.9.1/lib/python3.9/json/__init__.py in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    344             parse_int is None and parse_float is None and
    345             parse_constant is None and object_pairs_hook is None and not kw):
--> 346         return _default_decoder.decode(s)
    347     if cls is None:
    348         cls = JSONDecoder

~/.pyenv/versions/3.9.1/lib/python3.9/json/decoder.py in decode(self, s, _w)
    335 
    336         """
--> 337         obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    338         end = _w(s, end).end()
    339         if end != len(s):

~/.pyenv/versions/3.9.1/lib/python3.9/json/decoder.py in raw_decode(self, s, idx)
    353             obj, end = self.scan_once(s, idx)
    354         except StopIteration as err:
--> 355             raise JSONDecodeError("Expecting value", s, err.value) from None
    356         return obj, end

JSONDecodeError: Expecting value: line 1 column 1 (char 0)