esri / overwrite-hosted-features Goto Github PK

Feedback from @Erodenberg and Transportation team

Customers may have many GDBs that need to be synced and it's not intuitive how the script can be used for multiple GDBs. If we parameterize the config file, it will be easier for customers to manage more than one FGDB or Feature Collection updates

add a check in the script to verify the type matches what's expected for the Feature Service and Feature Collection item ids passed into the script and report a friendly message to the user if the item type isn't what was expected.

Repro Steps:

1. Stand up a feature service and feature collection pair on a portal using data store
2. Configure the overwrite script against the portal items
3. Run script: Receive error
   Failed to update Transportation511 feature service: {u'services': [{u'success': False, u'error': {u'message': u"Service name 'Transportation511' already exists for '0123456789ABCDEF'"}}]}
   

Resources:
Transportation 511 sample data
One of the public portals the dev team maintains

@NikkiGolding @jmhauck fyi

Should this script retain the original service projection? It doesn't seem to.
I ran the script against a hosted feature layer with required Projection (NZTM(2193)). The zipped FGDB is in the same projection. After running the tool the features are successfully added to the service but the projection of each of the service layers is updated to (WebMerc(102100)).
I checked over the script but couldn't see a projection reference. Can anyone help with this? Thanks.

Overwrite/Upload of current information (.gdb) to Online fails due to orphaned files within ArcGIS Online. The files that are orphaned are low level AGO files and are not visible within the UI. KY DOT has run into this issue twice now.

Note: The cause of the files being orphaned has occurred when an update was occurring and Online encountered an issue. The repro steps are not simple to recreate the root cause.

When an attempt is made to update the gdb a message is returned:

Traceback info:
File "overwrite_hosted_features.py", line 548, in run
self._upload_fgdb()
File "overwrite_hosted_features.py", line 427, in _upload_fgdb
response = self._url_request(url, request_parameters, files=files, error_text='Failed to upload file geodatabase')
File "overwrite_hosted_features.py", line 298, in _url_request
raise Exception("{0}: {1}".format(error_text, response_json))

Error Info:
Failed to upload file geodatabase: {u'error': {u'message': u"Item 'WazeAlerts.gdb.zip' already exists.", u'code': 409, u'details': []}}
Elapsed Time: 0:00:15.265000

additional context below:

We believe the issue was introduced when the Online infrastructure was under significant load. Due to the significant load, there were resulting issues across the system. I presume the script was in the middle of a delete operation when the 'blip' occurred. This left the KY files in a state where they were inaccessible to the script in a way we were searching for them.

Moving forward, we can make changes to the script's delete operation to "bypass the indexes" to try to ensure this does not happen again. However, we will need to test the scripts more thoroughly before we can release them. There is a fairly large caveat, we wont have another repro case to test the new version of the script unless the above circumstances with the Online infrastructure repeat.

Note: There is a fairly easy workaround for the issue if we ever get in a pinch again, we can rename the files being uploaded and alter the names for the files in the script.

To support feature service only deployments, make the export of the feature collection optional

From KY DOT:

This has occurred 2 or maybe 3 times, but should be easily fixed. For some reason, the file geodatabase in AGOL are not being deleted at the end of the overwrite_hosted_features.py script. So when the script runs the next time, it is not uploading the file geodatabase because it is saying it already exists in AGOL. This results in data not being updated for long periods of time (until someone is notified and manually deletes it off AGOL). I propose that the delete function also be applied at the beginning of the script. That way if it was not deleted the first time, it will catch it the next time around.

Feedback from Transportation team

For organizations that want to incorporate the script into larger processes, engineer the script to work as a module in other scripts in addition to working stand alone.

Known things that would need to be modified are:

• remove the main method
• make sub-functions private

From Iowa DOT:

Updating feature collections from the python update script: When feature collections are updated using the Overwrite Hosted Features python script provided, the symbology, popups, attribute settings all remain as you would expect. However, if you have your icons set to be rotated based on an attribute, this functionality gets wiped out after every update of the feature collection. What this means is that when a user adds the feature collection to a new map, they will have to set the rotation. If they have the feature collection in an existing map, the rotation is preserved.

Logged under the widget as well for better visibility while in planning phase:
https://devtopia.esri.com/john4818/arcgis-webappbuilder/issues/298

To better support troubleshooting of errors encountered during the execution of the script:

• capture more detailed error information
• capture job ids
• if possible, capture web request associated with failure

With the Feb 29 AGOL update, the ability to directly overwrite a feature collection on export was added. this was added as a requirement from the Transportation. we need to evaluate this capability and implement as appropriate.

Evaluate removing ArcREST dependency in the script

• reduces requirements for deployment
• improves error reporting

Receive message: <urlopen error [Errno 11004] getaddrinfo failed> when the baseURL parameter doesn't end with a fore-slash --

eg
this fails: http://stategovernment.maps.arcgis.com
this passes: http://stategovernment.maps.arcgis.com/

i think we could handle this error better. will add something in the doc for now.

Input from Transportation Team

Some organizations will have existing processes in place to update a hosted feature layer. In these cases, the file geodatabase upload and overwrite of the feature service are not needed.

enhance the script to make the file geodatabase upload and feature service overwrite sections optional.

using a service definition reduces script logic to maintain layer order and non-simple rendering.

need to evaluate:

• the approach without requiring ArcGIS license
• whether it adds additional dependencies