badstreff / git2jss

Easily sync your scripts and EA's from a git repo with the JSS

git2jss

A fast asynchronous python library for syncing your scripts in git with your JSS easily. This allows admins to keep their script in a version control system for easy updating rather than googling and copy-pasting from resources that they find online.

Getting Started

1. Fork the Project
2. Install Python version 3.6 or higher. (this is because of the async requirements)
3. Run python3.6 -m pip install -r requirements.txt to install required modules
4. Run ./tools/download.py --url https://your.jss.url:8443 --username api_user to download all scripts and extension attributes to the repository
5. Run ./sync.py --url https://your.jss.url:8443 --username api_user to sync all scripts back to your JSS

Optional flags for download.py:

• --password for CI/CD (Will prompt for password if not set)
• --do_not_verify_ssl to skip ssl verification
• --overwrite to overwrite all scripts and extension attributes

Optional flags for sync.py:

• --password for CI/CD (Will prompt for password if not set)
• --do_not_verify_ssl to skip ssl verification
• --overwrite to overwrite all scripts and extension attributes
• --limit to limit max connections (default=25)
• --timeout to limit max connections (default=60)
• --verbose to add additional logging
• --update_all to upload all resources in ./extension_attributes and ./scripts
• --jenkins to write a Jenkins file:jenkins.properties with $scripts and $eas and compare $GIT_PREVIOUS_COMMIT with $GIT_COMMIT

ConfigParser (Optional):

A config file can be created in the project root or the users home folder. When a config file exists, the script will not promt for a password.

A jamfapi.cfg file can provide the following variables:

• username
• password
• url

Prerequisites

git2jss requires Python 3.6 and the python modules listed in requirements.txt

Deployment

The project can be ran ad-hoc with the example listed above, but ideally you setup webhooks and integrate into a CI/CD pipeline so each time a push is made to the repo your scripts are re-uploaded to the JSS.

Contributing

PR's are always welcome!

git2jss's Issues

No interpreter directive found for (zsh)

For scripts with the shebang line of #!/bin/zsh — performing a download yields the following message:

No interpreter directive found for:

DeprecationWarning: with timeout() is deprecated, use async with timeout() instead

Have been looking to get this going for our org for a while now and finally got all the pieces in place only to be stumped by what appears to be a deprecated function.

Wondering if you or anybody else has come across this and has a quick fix?

Jamf API changes

Before going down the path to fix ourselves, wondering if anyone has done the work to update what seems to just be aiojss.py to support the new tokenized API auth vs old basic auth. Seems like it should be relatively easy, but always prefer avoiding re-inventing the wheel. =)

Thanks!

Not supplying --password produces error

From what I understand, you can not supply --password in the command to generate a password prompt. However, when the password is entered in this prompt it appears to not be accepted, as I get the following error:

python sync.py --url https://<jss>.com:8443 --username api_git2jss --do_not_verify_ssl
Password: 
Traceback (most recent call last):
  File "sync.py", line 429, in <module>
    loop.run_until_complete(main())
  File "uvloop/loop.pyx", line 1417, in uvloop.loop.Loop.run_until_complete
  File "sync.py", line 385, in main
    session, args.url, args.username, args.password, semaphore)
  File "sync.py", line 355, in get_existing_categories
    auth = aiohttp.BasicAuth(user, passwd)
  File "/Users/ap.orlebeke/.virtualenv/git2jss/lib/python3.7/site-packages/aiohttp/helpers.py", line 127, in __new__
    raise ValueError('None is not allowed as password value')
ValueError: None is not allowed as password value

Not Fully Asynchronous When Syncing Scripts

store command line args in keychain

Currently each time sync is run the username and password to the JSS need typed in the command line, it would be nice to have the ability to cache them in something like the keychain and verify they are correct so the user doesn't have to keep typing them

refactor directory structure

It looks like there is a practice followed my mac admins to separate their scripts by folders within their repo.

Wanted to start the discussion of possibly refactoring the script and documentation to depend on this structure.

sync not uploading EAs/Scripts if jamf has none

We've been discussing this in #git2jss on Slack, but I thought it was worth opening an issue. When trying to use sync to upload EAs/scripts to a Jamf Pro instance without any scripts for EAs (e.g., a brand new test instance) no EAs or scripts get uploaded, even when using --update_all

Changed Extention Attributes:  []
Changed Scripts:  []
No Changes in Scripts
No Changes in Extension Attributes

It sounds like a fix has already been identified. Just wanted an issue to attach to it.

ElementTree errors on parsing "&"

Receive the error: File "/usr/local/lib/python3.6/xml/etree/ElementTree.py", line 1314, in XML parser.feed(text) xml.etree.ElementTree.ParseError: reference to invalid character number: line 35, column 20

While trying to download and/or parse our scripts. It seems to be only breaking on &. I have been doing some research on how to mitigate this. The only thing I could see is to read the XML for each line and handle the & before the ElementTree gets it.

Add Flag for Disabling Validation of cert

In some environments eg a dev environment that is internal only the cert may be self signed..

Would it be possible to add a flag for disabling this verification of the cert?

Traceback (most recent call last):
  File "/home/gitlab-runner/.local/lib/python3.5/site-packages/aiohttp/connector.py", line 796, in _wrap_create_connection
    return (yield from self._loop.create_connection(*args, **kwargs))
  File "uvloop/loop.pyx", line 1769, in create_connection
  File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/sslproto.py", line 493, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "/usr/lib/python3.5/asyncio/sslproto.py", line 201, in feed_ssldata
    self._sslobj.do_handshake()
  File "/usr/lib/python3.5/ssl.py", line 633, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./sync.py", line 163, in <module>
    loop.run_until_complete(main(args))
  File "uvloop/loop.pyx", line 1364, in uvloop.loop.Loop.run_until_complete
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
    result = coro.throw(exc)
  File "./sync.py", line 147, in main
    await upload_scripts(session, args.url, args.username, args.password, semaphore)
  File "./sync.py", line 92, in upload_scripts
    responses = await asyncio.gather(*tasks)
  File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
    result = coro.throw(exc)
  File "./sync.py", line 105, in upload_script
    auth=auth) as resp:
  File "/home/gitlab-runner/.local/lib/python3.5/site-packages/aiohttp/client.py", line 690, in __aenter__
    self._resp = yield from self._coro
  File "/home/gitlab-runner/.local/lib/python3.5/site-packages/aiohttp/client.py", line 267, in _request
    conn = yield from self._connector.connect(req)
  File "/home/gitlab-runner/.local/lib/python3.5/site-packages/aiohttp/connector.py", line 402, in connect
    proto = yield from self._create_connection(req)
  File "/home/gitlab-runner/.local/lib/python3.5/site-packages/aiohttp/connector.py", line 748, in _create_connection
    _, proto = yield from self._create_direct_connection(req)
  File "/home/gitlab-runner/.local/lib/python3.5/site-packages/aiohttp/connector.py", line 859, in _create_direct_connection
    raise last_exc
  File "/home/gitlab-runner/.local/lib/python3.5/site-packages/aiohttp/connector.py", line 831, in _create_direct_connection
    req=req, client_error=client_error)
  File "/home/gitlab-runner/.local/lib/python3.5/site-packages/aiohttp/connector.py", line 801, in _wrap_create_connection
    raise ClientConnectorSSLError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host <JSS>:8443 ssl:True [[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)]

using export_path with download.py caused errors in sync.py with sync_path

Short version, I think add sync_path needs to be added to the front of the joins near lines 152, 207, 293, and 335. (the line numbers are from my fork and are a bit off, as I took the long road to get there and changed a bit more thinking it was necessary)

More info on what I saw here: https://macadmins.slack.com/archives/C04QVP86E/p1697777935371559?thread_ts=1697751693.656709&cid=C04QVP86E
More info about my conclusions here: https://macadmins.slack.com/archives/CDLQMGB6E/p1697873801412639

This was with the fork that was just merged.

Add support for Configuration Profiles

Hello

First of all: Thank you for writing this great tool!

This is a feature request and not an issue as such. Do you have any planes to add support for configuration profiles?
Or would you be willing to accept a PR with this functionality?

Timeout on on prem v10

Hello!

While further testing this in prep for our go live. We've also tested it on v10.
I'm seeing an error through the CI runner hitting the webapp(hosted on the same server), and if i run it locally hitting the same server, or on the server hitting the webapp.

What it seems is aiosync is spawning too many connections in a short period. Which Jamv10 doesn't seem to like. I can confirm this on 10.0.0 and 10.1.1.

I've gone ahead and added to mysql & tomcat threads and still experience the same error.

Traceback (most recent call last):
  File "sync.py", line 49, in upload_extension_attribute
    resp = await session.put(put_url, auth=auth, data=ET.tostring(template), headers=headers)
  File "/home/rderewianko/.local/lib/python3.5/site-packages/aiohttp/helpers.py", line 104, in __await__
    ret = yield from self._coro
  File "/home/rderewianko/.local/lib/python3.5/site-packages/aiohttp/client.py", line 277, in _request
    yield from resp.start(conn, read_until_eof)
  File "/home/rderewianko/.local/lib/python3.5/site-packages/aiohttp/client_reqrep.py", line 624, in start
    (message, payload) = yield from self._protocol.read()
  File "/home/rderewianko/.local/lib/python3.5/site-packages/aiohttp/streams.py", line 554, in read
    yield from self._waiter
  File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 266, in result
    raise CancelledError
concurrent.futures._base.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "sync.py", line 159, in <module>
    loop.run_until_complete(main(args))
  File "uvloop/loop.pyx", line 1364, in uvloop.loop.Loop.run_until_complete
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
    result = coro.throw(exc)
  File "sync.py", line 145, in main
    await upload_extension_attributes(session, args.url, args.username, args.password)
  File "sync.py", line 31, in upload_extension_attributes
    responses = await asyncio.gather(*tasks)
  File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
    result = coro.throw(exc)
  File "sync.py", line 52, in upload_extension_attribute
    resp = await session.post(post_url, auth=auth, data=ET.tostring(template), headers=headers)
  File "/home/rderewianko/.local/lib/python3.5/site-packages/async_timeout/__init__.py", line 35, in __exit__
    self._do_exit(exc_type)
  File "/home/rderewianko/.local/lib/python3.5/site-packages/async_timeout/__init__.py", line 80, in _do_exit
    raise asyncio.TimeoutError
concurrent.futures._base.TimeoutError

There is a difference in the number of submissions before failure.
CI Runner - 63
Locally - 112

publish on pypi

Would be nice to be able to install this via pip and point it at a directory instead of having the script expect the root dir

sync.py do_not_verify_ssl

When trying to run the sync.py I'm getting the following error when the --do_not_verify_ssl flag is set. This same flag is working with the download.py.

./sync.py:288: DeprecationWarning: verify_ssl is deprecated, use ssl=False instead async with aiohttp.ClientSession(connector=aiohttp.TCPConnector(verify_ssl=args.do_not_verify_ssl)) as session: Traceback (most recent call last): File "./sync.py", line 330, in <module> loop.run_until_complete(main(args)) File "uvloop/loop.pyx", line 1446, in uvloop.loop.Loop.run_until_complete File "./sync.py", line 289, in main categories = await get_existing_categories(session, args.url, args.username, args.password, semaphore) File "./sync.py", line 281, in get_existing_categories return [c.find('name').text for c in [e for e in ET.fromstring(await resp.text()).findall('category')]] File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xml/etree/ElementTree.py", line 1315, in XML parser.feed(text) xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 1, column 0