esri / arcgis-rest-js

compact, modular JavaScript wrappers for the ArcGIS REST API

@esri/arcgis-rest-js

compact, modular JavaScript wrappers for the ArcGIS REST API that run in Node.js and modern browsers.

Table of Contents

• Example
• API Reference
• Instructions
• Packages
• FAQ
• Issues
• Versioning
• Contributing
• Code of Conduct
• CHANGELOG
• License

Example

import { request } from "@esri/arcgis-rest-request";

const url =
  "https://www.arcgis.com/sharing/rest/content/items/6e03e8c26aad4b9c92a87c1063ddb0e3/data";

request(url).then((response) => {
  console.log(response); // WebMap JSON
});

Get Started

To get started, go to ArcGIS REST JS on the ArcGIS Developers website.

If you are on version 3, the previous documentation can be found at https://esri.github.io/arcgis-rest-js.

Documentation

The documentation is published at https://developers.arcgis.com/arcgis-rest-js/ and is maintained in a private repository and managed by the ArcGIS Developer Experience team. The API reference is generated automatically by TypeDoc via the npm run typedoc command and the typedoc.json configuration file.

Instructions

You can install dependencies by cloning the repository and running:

npm install && npm run build

This will install all dependencies and do an initial build. Afterward, you can run any of the demo apps by cd'ing by following the README for the specific demo. For a list of all available commands run npm run.

For all packages:

• npm run build - builds all distributions of every package with ultra, inside each package builds are done in parallel with npm-run-all. Output is errors only.
• npm run build:esm, npm run build:cjs, npm run build:bundled - as as the above but only one of our target distributions.
• npm run dev:esm, npm run dev:cjs, npm run dev:bundled - runs the appropriate watch command in all packages.

For a specific package:

• npm run build -w @esri/arcgis-rest-request - run all builds in a specific workspace
• npm run dev -w @esri/arcgis-rest-request - run all dev commands in a specific workspace
• npm run build:esm -w @esri/arcgis-rest-request - run the esm build in a specific workspace
• npm run dev:esm -w @esri/arcgis-rest-request - run the esm dev command in a specific workspace
• npm run build:cjs -w @esri/arcgis-rest-request - run the common js build in a specific workspace
• npm run dev:cjs -w @esri/arcgis-rest-request - run the common js dev command in a specific workspace
• npm run build:bundled -w @esri/arcgis-rest-request - run the rollup build in a specific workspace
• npm run dev:bundled -w @esri/arcgis-rest-request - run the rollup dev command in a specific workspace

Packages

• @esri/arcgis-rest-request - Core module implementing basic request code, shared TypeScript types and common utilities.
• @esri/arcgis-rest-portal - Methods for working with ArcGIS Online/Enterprise content and users.
• @esri/arcgis-rest-feature-service - Functions for querying, editing, and administering hosted feature layers and feature services.
• @esri/arcgis-rest-geocoding - Wrapper around geocoding services.
• @esri/arcgis-rest-routing - Wrapper around routing and directions services.
• @esri/arcgis-rest-demographics - Wrapper around demographic data services.
• @esri/arcgis-rest-places - Wrapper around place finding and place data services.

Issues

If something isn't working the way you expected, please take a look at previously logged issues first. Have you found a new bug? Want to request a new feature? We'd love to hear from you.

If you're looking for help you can also post issues on Stack Overflow with the esri-oss tag.

Versioning

For transparency into the release cycle and in striving to maintain backward compatibility, @esri/arcgis-rest-js is maintained under Semantic Versioning guidelines and will adhere to these rules whenever possible.

For more information on SemVer, please visit http://semver.org/.

Contributing

Esri welcomes contributions from anyone and everyone. Please see our guidelines for contributing.

License

Copyright © 2017-2023 Esri

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

A copy of the license is available in the repository's LICENSE file.

arcgis-rest-js's Issues

Clean up README.md

@jgravois we should probably go over the readme.md and spruce it up before open sourcing.

How much wrapping of the API do we do?

Only thing that gives me pause is the core-types - are we planning to parse the response json into types with behavior? I could see this massively increasing the scope of this effort if that is the case. My personal preference is to keep arcgis-rest-js a thin wrapper over the API idiosyncrasies, but still allowing a consumer to work with the json directly. If they choose to pump the json into "models" of some sort, so be it, but let's not force that.

@dbouwman in https://github.com/ArcGIS/arcgis-rest-js/pull/6#pullrequestreview-54540194

This is a pretty legitimate question. I definitely agree with @dbouwman that we should not attempt to put everything in our own models.

However I DO think we should provide TypeScript types for request and response parameters. For example here the response for http://geocode.arcgis.com/arcgis/rest/services/World/GeocodeServer/findAddressCandidates?Address=380+new+york+st&City=redlands&Region=CA&Postal=92373&forStorage=false&f=pjson

{
  spatialReference: {
    wkid: 4326,
    latestWkid: 4326
  },
  candidates: [
    {
      address: "380 New York St, Redlands, California, 92373",
      location: {
        x: -117.1956703176181,
        y: 34.05648811930892
      },
      score: 100,
      attributes: {},
      extent: {
        xmin: -117.1963135,
        ymin: 34.05510800000001,
        xmax: -117.19431349999999,
        ymax: 34.05710800000001
      }
    }
  ]
}

Below would be the corresponding typescript interface for this request using @esri/rest-common-types:

import { IExtent, ISpatialReference } from '@esri/rest-common-types';

interface IGeocodeResponse {
  spatialReference: ISpatialReference,
  candidates: IGeocodeCanidate[]
}

interface IGeocodeCanidate {
  address: string;
  location: {
    x: number,
    y: number
  },
  score: number,
  attrubutes: {
    [key: string]: any; //index type allows anything under here... https://www.typescriptlang.org/docs/handbook/advanced-types.html
  },
  extent: IExtent,
  [key: string]: any; // allow any additional params we might not have typed...
}

We can then write the geocode function like this:

import { IRequestOptions } from '@esri/rest-request';
import { IGeocodeResponse } from './IGeocodeResponse';

interface IGeocodeParams {
// stuff...
}

interface IGeocodeOptions extends IRequestOptions {
  geocodeService: string;
}

export function geodode(params: IGeocode, options: IGeocodeOptions) : Promise<IGeocodeResponse>{
  // stuff
  return request(/*...*/)
}

I think we should author typescript types for requests and responses. @esri/rest-common-types was simply envisioned to be a repository of the common types we spread across the different packages.

I think authoring TypeScript types for request params and responses is important for a couple of reasons.

1. TypeScript users will expect this. Deferring everything to an <any> type and deferring people to the REST API will not provide the best experience.
2. It allows us to annotate the response in our own docs first since we export the interfaces providing comments specific to our own library before shelling out to the REST API docs.
3. TypeScript users (and JavaScript users in VSCode) will get Intelisense from the typings.
4. In some cases (feature layer admin) we will HAVE to provide our own params and response types since it will have to wrap multiple calls. This means we will need to write a good chunk of @esri/rest-common-types anyway.

Since the typings are fairly easy to write and we will have to write a ton anyway for more complex use cases (like feature layer admin) I think we should strive to be consistant and provide at least some basic types since they are easy to write.

---

But we do still need to discuss what level of wrapping DOES any sense for the API? I have 3 examples:

1. In the above geocoding response, the location of each candidate does not have a spatialReference. If a user wants to pass the location of a match to another API they will need to manually add the spatialReference property to it. Should we wrap this for users?
2. How would we build an API for sharing items? There are 3 different sharing APIs (itemid/share, groupid/share, and users/username/share) how should would we implement this? Should we make an easier sharing API? Simply wrapping all 3 methods seems inadequate but a better sharing API gets away from the idea of a "simple request" wrapper.
3. Feature layer creation wraps 4-7 requests in most cases. how should we handle this? How much validation should we do?

---

My point being that should probably decide on a series of principles to guide how much wrapping we are going to do.

create example which demonstrates how to batch geocode a .csv

ref: Esri/geoservices-js#95

Example Apps

In order to ensure that the api works for our desired use-cases, I propose we build some simple applications to validate the library.

The following demos seem useful:

• jquery app
• dojo app
• ember js app
• angular js app
• react app
• node cli app

For the browser demos, the app should use oAuth to get a token, then instantiate a session and execute a simple, authenticated search @ AGO.

The node example should take the creds as command-line args, and iterate over all the items the user owns, listing the titles.

At this point, lets keep these as vanilla as possible - just bootstrap + minimal code needed. We can build idiomatic examples down the road.

I'll sign-up to create the arcgis-rest-js-demos repo, and build the jquery, ember and node examples. Need some Dojo/Angular/React people to jump in as well @tomwayson @mjuniper??

Switch all methods to use a single options object

Between #73 and #74 it seems we are struggling to figure out the right order of options and how to pass commonly used things like owner and portal. I am going to suggest the following:

1. Every method accepts a single options object.
2. That object is strictly typed and inherits from IRequestOptions or another options interface that inherits from IRequestOptions.
3. request now looks like

   interface IRequestOptions () {
     url: string;
     params?: IParams;
     httpMethod?: HTTPMethods;
     authentication?: IAuthenticationManager;
     fetch?: (input: RequestInfo, init?: RequestInit) => Promise<Response>;
   }
   
   request(options: IRequestOptions) {
     /** request **/
   }

4. Methods that wrap request are now implemented like:

   interface IReassignItemOptions extends IRequestOptions() {
     /**
      * `url` is not allowed in `IReassignItemOptions`. Use `portal` instead.
      */
     url: void;
     itemId: string;
     owner: string;
     newOwner: string;
     newFolder?: string;
     portal?: string;
   }
   
   reassignItem(options: IReassignItemOptions) {
     /*
      * `portal` and `currentOwner` can default to `options.authentication.username`
      * and `options.authentication.portal` or be overridden by the `portal` and `owner` options.
      */
   }

5. portal should be removed from IRequestOptions since not all methods require it https://github.com/Esri/arcgis-rest-js/blob/master/packages/arcgis-rest-request/src/request.ts#L51.

Broken links due to code comments

The below broken links are markdown links within code comments. These links don't get processed by Acetate and thus don't get the base url appended. We might want to figure out a way to autogenerate these in-comment links so we don't have to manually set and maintain them.

Appears on	Link Text	404ing URL
https://esri.github.io/arcgis-rest-js/api/request/checkForErrors/	ArcGISAuthError	https://esri.github.io/api/arcgis-core/ArcGISAuthError
https://esri.github.io/arcgis-rest-js/api/request/checkForErrors/	ArcGISRequestError	https://esri.github.io/api/arcgis-core/ArcGISRequestError
https://esri.github.io/arcgis-rest-js/api/request/IRequestOptions/	request()	https://esri.github.io/api/arcgis-core/request
https://esri.github.io/arcgis-rest-js/api/auth/UserSession/	request	https://esri.github.io/api/request/request

Start planning out packages

@dbouwman @jgravois @noahmulfinger @dmfenton @araedavis I think this repo is getting close to the point where we can start thinking about planning out the other packages that we want. I think we can still do this regardless of the auth decisions in https://github.com/ArcGIS/arcgis-rest-js/issues/5.

I think the first step is audit all of our projects and produce the following:

• A list of all API endpoints we call on server and portal
• All params we use on those endpoints
• How are our codebases currently organized
• What would we like to prioritize for inclusion as a package

I think once we have these audits complete and we know what we are dealing with we can better plan out a sane package organization.

I will add that I want to create a rest-core-types repo so we can share common types and interfaces across the project for things like geometries.

I also want to make rest-geocoding based on https://developers.arcgis.com/rest/geocode/api-reference/overview-world-geocoding-service.htm.

I want to do rest-core-types and rest-geocoding for 2 reasons:

1. Validate that the build/test/docs/lint systems all work properly.
2. Lay out examples for contributing new packages.
3. Verify that the arcgis-request and rest-core-types can both be easily used to create rest-geocoding

Design API for @esri/arcgis-rest-search module

API for interacting with the portal search API https://resources.arcgis.com/en/help/arcgis-rest-api/#/Search/02r3000000mp000000/.

Add support for f=geojson

Right now exactly 1 endpoint (FeatureServer/LAYER/query) supports f=geojson.

Should this library support handling f=geojson for other API endpoints by transforming the response with terraformer-arcgis-parser? Should we accept GeoJSON input the same way?

I'm more on the side of not doing this since it would be a lot of extra effort and it would mean that requests could have 2 different JSON outputs which would be hard for TypeScript. It would also mean extra dependencies.

refactor demos/vanilla sample to demonstrate popup:false as well

i did a little bit of head scratching this afternoon and i still can't figure out exactly how we want browser developers to implement OAuth2 when they don't want to use a popup.

it'd be cool to tweak our existing sample so that it can be toggled to demo both or just write a new one.

// no popup
var session = arcgisRest.UserSession.completeOAuth2({
    clientId: ClientId,
    redirectUri: "http://localhost:8080/post-sign-in.html",
    popup: false,
    portal: "https://www.arcgis.com/sharing/rest"
  })

session // how can i pass a session like this from the redirect page back to the original app?

Livereload docs when source files change

Just like the title says. This will probably need a custom script to watch the source files for changes, rerun TypeDoc and then tell Acetate to reload.

Application Session needs to take a Portal param

Right now we could not use this against devext/qaext

Design API for @esri/arcgis-rest-applications module

Design and API for interacting with applications, creating the item, oauth, proxy management. @ajturner @dbouwman and @tomwayson might have additional requirements for creating template applications.

Portal / AGOL only or also ArcGIS Server

So far it seems this project only contains code for calling Portal or ArcGIS Online. Will support for ArcGIS Server services be added at some point, or is that beyond the intended scope of this project?

At the moment, I am tasked with creating a JavaScript application that will call the Esri Linear Referencing REST API, which is part of the Roads and Highways product. Should I fork and add the LRS API support code as a new package within this project, or should I just create a new project that references @esri/arcgis-rest-request as a dependency?

Design API for @esri/arcgis-rest-groups module

API for interacting with the groups APIs https://resources.arcgis.com/en/help/arcgis-rest-api/#/Group/02r30000006m000000/.

add deprecation note to prior art

once this is open sourced, ill reply to open issues in past projects and include a deprecation notice at the top of those READMEs.

Esri/geoservices-js#91
Esri/node-arcgis#91

kinda creepin me out that those issue numbers match.

Automatically deploy to gh-pages when merging into master

Details are in https://docs.travis-ci.com/user/deployment/pages/. This should make it do we don't have to build the site locally to to update GH pages just merge it into master. This also means that the travis CI command will have to build the docs in addition to running the tests.

HTTPMethods links in doc are broken

when i reviewed 32face1 i noticed that i overwrote some links to nowhere with actual '404's.

everything looks okay in the typedoc.json, but we aren't yet linking correctly to HTTPMethods.

Investigate duration of refresh tokens

According to the REST API docs at http://resources.arcgis.com/en/help/arcgis-rest-api/#/Authentication/02r30000009z000000/ refresh_token has a limited life span:

The lifetime of the refresh token that's returned by this call is controllable by the app. The default expiry time for the refresh token returned by this flow is two weeks.

However you can request longer lived access tokens:

Using this flow, you can request a refresh token that's valid for a longer period by passing an expiration (in minutes) parameter during authorization.

and

For authorization grants, the default validity of access_tokens is 30 minutes and refresh_tokens is 2 weeks. The expiration parameter, if specified, overrides the validity period of refresh_tokens up to a max of 90 days.

-http://resources.arcgis.com/en/help/arcgis-rest-api/#/Authorize/02r300000214000000/

But:

The refresh token that's returned may be valid for a shorter period than requested based on the maximum expiry time set by the user's organization or the platform.

So while I can request a refresh token for up to 90 days it MIGHT be limited by the users organization or platform.

Question 1: "How do I know when my refresh_token will expire?"

It appears to be possible to use grant_type=exchange_refresh_token on http://resources.arcgis.com/en/help/arcgis-rest-api/#/Token/02r300000213000000/ to exchange your old refresh_token for a new one.

grant_type=exchange_refresh_token—Issues a new access_token and refresh_token by exchanging the old refresh_token obtained before. Old refresh_token will be invalidated upon issuing a new one. All the access_token obtained with the old refresh_token will also be invalidated. The newly exchanged refresh_token will have the same expiration minutes as the old one, the newly obtained access_token will have an expiration of 30 minutes.

Particularly this line:

The newly exchanged refresh_token will have the same expiration minutes as the old one

Question 2: Does this mean the same duration as was originally obtained during authorization? Or does this mean that the new refresh_token have the same number of minutes left until it expires as the old token?

---

@dbouwman @dmfenton you guys do long running work with portal and server. Do you know the answer?

Handling authentication

Authentication is a problem that has been bugging me for awhile about this repo. I'm note entirely sure how to resolve it. As a reference here are some of our design goals for this:

• Avoid use of a singleton like IdentityManager
• Make it easy for a developer to configure authentication once and forget about it no matter what library they are using.
• We want to properly federate requests (a la JS API) as much as possible with as minimal overhead as possible.

I feel like these goals are at odds with each other. No matter what I seem to always come up with designs where you pass the authentication in or query the authentication object for information.

---

Design 1

Below is a rehash of my inital design:

import { request } from 'arcgis-rest-core';
import { UserAuthentication, AppAuthentication } from 'arcgis-rest-auth';

const auth = new AppAuthentication({
  clientId: '123'; // required
  clientSecret: '123' // required
  token: '123' // optional, an access token if you have one
  expires: Date.now() // when the provided token expires
  portal: 'https://www.arcgis.com/sharing/rest' // which portal generated this token
});

auth.on('error', (error) => {
  // unable to authenticate a request so get a new token and retry.
  auth.refreshToken(/* ... */).then((token) => {
    error.request.retry(token);
  });
});

request({
  url: '...'
  authentication: auth,
  params: {
    //...
  }
}).then((response) => {
  // Success!
}).catch((error) => {
  // Error! Both HTTP AND server errors will end up here...
});

This approach will work fine, except that most of the interaction this this repo won't be directly with request it will be with help methods that we will write like geocode. So this starts to break down when we do this:

import {request} from 'arcgis-core';

export function geocode (/* ... */) {
  // since request is internal it won't be authenticated. :(
  return request(/* ... */).then(/* ... */);
}

---

Design 2

import { request } from 'arcgis-rest-core';
import { UserAuthentication, AppAuthentication } from 'arcgis-rest-auth';

const session = new AppAuthentication({
  clientId: '123'; // required
  clientSecret: '123' // required
  token: '123' // optional, an access token if you have one
  expires: Date.now() // when the provided token expires
  portal: 'https://www.arcgis.com/sharing/rest' // which portal generated this token
});

session.on('error', (error) => {
  // unable to authenticate a request so get a new token and retry.
  session.refreshToken(/* ... */).then((token) => {
    error.request.retry(token);
  });
});

// The UserAuthentication and AppAuthentication methods will expose a `authenticate`
// method that accepts a Promise from `request()` it then returns that promise with
// additional error handling.
session.authenticate(request({
  url: '...'
  params: {
    //...
  }
})).then((response) => {
  // Success!
}).catch((error) => {
  // Error! Both HTTP AND server errors will end up here...
});

// this would also work for anything that returned a Promise from `request()` so...
session.authenticate(geocode({/* ... */})).then((response) => {
  // Success!
}).catch((error) => {
  // Error! Both HTTP AND server errors will end up here...
});

This method has 1 major drawback though. The inital request will ALWAYS be unauthenticated, since the token was never passed in the inital params. This sucks but we should be able to recover from the error but it will happen every time whereas the JS API is smart enough to not fail and retry with a token all the time.

---

Design 3

This design is like the inverse of the one above. We simply expose the authentication option on all methods use request under the hood and teach request how to use the passed authentication object to handle auth failures.

import { request } from 'arcgis-rest-core';
import { UserAuthentication, AppAuthentication } from 'arcgis-rest-auth';

const session = new AppAuthentication({
  clientId: '123'; // required
  clientSecret: '123' // required
  token: '123' // optional, an access token if you have one
  expires: Date.now() // when the provided token expires
  portal: 'https://www.arcgis.com/sharing/rest' // which portal generated this token
});

session.on('error', (error) => {
  // unable to authenticate a request so get a new token and retry.
  session.refreshToken(/* ... */).then((token) => {
    error.request.retry(token);
  });
});

// we would have to teach request how to use `AppAuthentication` to recover from
// auth failures. Hopefully via an interface so it doesn't bloat the core repo.
request({
  url: '...'
  authentication: session
  params: {
    //...
  }
})).then((response) => {
  // Success!
}).catch((error) => {
  // Error! Both HTTP AND server errors will end up here...
});

// we would have to pass the `authentication` object down throguh the 
geocode({
  authentication: session
})).then((response) => {
  // Success!
}).catch((error) => {
  // Error! Both HTTP AND server errors will end up here...
});

---

Design 4 - Singleton Authentication Manager

import { request } from 'arcgis-rest-core';
import { AuthenticationManager } from 'arcgis-rest-auth';

// register a client id and client secret
AuthenticationManager.registerOauthCredentials({
  clientId: '123'; // required
  clientSecret: '123' // required
  token: '123' // optional, an access token if you have one
  expires: Date.now() // when the provided token expires
  portal: 'https://www.arcgis.com/sharing/rest' // which portal generated this token
});

// register an existing token or token from user auth
AuthenticationManager.registerToken({
  clientId: '123'; // required
  token: '123' // optional, an access token if you have one
  expires: Date.now() // when the provided token expires
  portal: 'https://www.arcgis.com/sharing/rest' // which portal generated this token
});

AuthenticationManager.on('authentication-required', (error) => {
  // unable to authenticate a request user will need to prompt for auth to a service.
});

// request now talks to the `AuthenticationManager` for all auth decisions.
request({
  url: '...'
  params: {
    //...
  }
})).then((response) => {
  // Success!
}).catch((error) => {
  // Error! Both HTTP AND server errors will end up here...
});

// since geocode will impliment `request` it will also use the `AuthenticationManager` singleton.
geocode({
  authentication: session
})).then((response) => {
  // Success!
}).catch((error) => {
  // Error! Both HTTP AND server errors will end up here...
});

---

I'm not super liking any of these options here. @jgravois @dbouwman @ajturner @tomwayson.

Setup headless Chrome and Firefox tests on Travis

Once we open source the repo we should have Travis CI run our Node tests on Node 6 and Node 8 and our browser tests through Sauce Labs https://wiki.saucelabs.com/display/DOCS/Setting+Up+Karma+for+JavaScript+Unit+Testing for all browsers that we support.

It would also be nice to use CodeCov or Coveralls to automatically upload the coverage files generated by Karma.

Inconsistent signatures...

searchItems and searchGroups signatures are inconsistent;

searchItems(search: string | ISearchRequestOptions) : Promise<ISearchResult> has one arg...
and ISearchRequestOptions contains a searchForm...

where as searchGroups takes the searchForm as the first arg.

searchGroups(searchForm: IGroupSearchRequest, requestOptions: IRequestOptions)

Either is fine, but consistency makes the lib much easier to work with.

As we port the ember-arcgis-portal-services over to using this lib, I'm sure we will find other things like this :)

Pseudocode

Pseudocode examples of how we would use the arcgis-rest-js auth + Identity Manager with:

• node js
• browser app (any framework)
• browser with JSAPI

Scenario:

• fetch private Map Service item from AGO that ref's an external, secured Map Service
• query a layer in the map service

push up live doc site

have we decided where this is going to live yet? gh-pages? surge?

Documentation build fails on Windows

When running npm run docs:serve on Windows, the script fails.

D:\Repos\rest-js>npm run docs:serve

> @esri/[email protected] predocs:serve D:\Repos\rest-js
> npm run docs:typedoc


> @esri/[email protected] docs:typedoc D:\Repos\rest-js
> node docs/build-typedoc.js

events.js:160
      throw er; // Unhandled 'error' event
      ^

Error: spawn typedoc ENOENT
    at exports._errnoException (util.js:1020:11)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:197:32)
    at onErrorNT (internal/child_process.js:376:16)
    at _combinedTickCallback (internal/process/next_tick.js:80:11)
    at process._tickCallback (internal/process/next_tick.js:104:9)
    at Module.runMain (module.js:606:11)
    at run (bootstrap_node.js:389:7)
    at startup (bootstrap_node.js:149:9)
    at bootstrap_node.js:502:3
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! @esri/[email protected] docs:typedoc: `node docs/build-typedoc.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the @esri/[email protected] docs:typedoc script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\<user>\AppData\Roaming\npm-cache\_logs\2017-09-21T19_43_34_456Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! @esri/[email protected] predocs:serve: `npm run docs:typedoc`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the @esri/[email protected] predocs:serve script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     C:\Users\<user>\AppData\Roaming\npm-cache\_logs\2017-09-21T19_43_34_490Z-debug.log

D:\Repos\rest-js>

Design API for @esri/arcgis-rest-feature-layers module

API design for a module with support for creating, updating and deleting feature both feature layers and their features.

Install script fails on Windows

I am unable to get a clean install by following the instructions on the README.md, While executing the npm install process, it fails at the lerna command npm run prepublish. I get the same error when using Cygwin64 Terminal or Git Bash. Below are my system details.

• Windows 8.1 Enterprise
• node 6.10.2
• npm 5.4.2
• lerna 2.2.0

I've attached a text file of the exact messages. rest-js_npm_install_log.txt

Docs: Formatting issue with long strings

The property table on the doc pages expands too far when there's a long string without a word break (see the sampleserver url below):

Add code of conduct

In the intereste of making this a friendly and safe open source project we should add a COC and stick to it. https://help.github.com/articles/adding-a-code-of-conduct-to-your-project/

FYI: existing type definitions

A while ago I contributed the @types/arcgis-rest-api type definitions to the DefinitelyTyped repository. If they meet your needs you could use these instead of creating them from scratch. (Available via NPM here)

I have another version with additional types, but that hasn't been contributed yet. I've attached that version below, in case you want to use it.
arcgis-rest-api.d.ts.zip

(The only reason that all of the types are in a single index.d.ts file is simply because that's the way the DefinitelyTyped project expects them to be.)

Incorrect path for "browser" field in package.json

Problem

When building a bundle with browserify, I received the very unhelpful error message that the module could not be found even though it was installed in ./node_modules via npm.

Error: Cannot find module './node_modules/@esri/arcgis-rest-auth'

Solution

After much confusion, poking, and prodding. I realized that the "browser" path hint in package.json was pointing to the wrong location. After fixing this locally browserify was able to find the correct distribution and build the bundle.

Example

https://github.com/Esri/arcgis-rest-js/blob/master/packages/arcgis-rest-request/package.json#L6
Presumably it's an issue for all packages.

Build release automation tooling

This will probably be one of my least favorite tasks ever but after automating this for Leaflet hopefully this can be OK.

General idea of the process:

1. Run lerna publish --no-git. This will run npm publish in each package and update the version key in lerna.json but skip doing anything in Git.
2. Update our package.json version with the new version from 'lerna.json'.
3. Run a script to generate a Changelog for the release. I'll post some ideas I have for this later.
4. Now run a script similar to https://github.com/Esri/esri-leaflet/blob/master/scripts/release.sh that will commit the built files in dist folders, the CHANGELOG.md, the updated lerna.json and the updated package.json to GitHub.
5. Next create a .zip file of all the umd builds.
6. run gh-release to create a new release on GitHub using our .zip file and our newly updated version from package.json. This will also tag the release on GitHub.

add README to each individual package (for npm)

ref:

@esri/arcgis-rest-request public

Sadly, this package has no readme. Go write one!

smart support for GET (which switches to POST automatically)?

our current implementation of esriRequest defaults to POST and allows a GET to be issued manually.

@patrickarlt's preference is to recommend that packages default to POST and skip implementing logic to switch from a GET to a POST after the commonly accepted ~2000 character limit has been exceeded.

this issue is intended to track interest from other developers who would like to piggyback off a smart request method instead of defaulting to POST (or implementing the logic above in their own package).

anecdotally, it didn't take me long to identify an ArcGIS Server operation that appears to only support GET

• http://sampleserver6.arcgisonline.com/arcgis/rest/services/Census/MapServer/dynamicLayer
• http://resources.arcgis.com/en/help/arcgis-rest-api/index.html#/Dynamic_Layer_Table/02r3000000q0000000/

write the docs!

patrickarlt [4:15 PM]
if you do want to mess with Typedoc you need TypeStrong/typedoc#549 to make it work.

patrickarlt [4:16 PM]
typedoc --out docs packages/arcgis-core/src --readme none

[4:16]
i dont know how to make typedoc stop treating every file as an external module…

new messages
[4:17]
there supposedly is a --mode modules but it doesn’t do anything.
Dojo has a bunch of custom tooling but their doc ends up looking pretty good https://dojo.io/api/

based on the trouble so far, i'll probably take a stab at running jsdoc on the compiled ES5 source first instead of trying to be a trailblazer.

Design API for @esri/arcgis-rest-item module

API for interacting with the portal content APIs https://resources.arcgis.com/en/help/arcgis-rest-api/#/Content_Root/02r300000093000000/.

Naming things!

@dbouwman @jgravois @noahmulfinger @dmfenton @araedavis @ajturner I think we need to discuss standardizing naming of our packages.

Currently we have 1 package called arcgis-core. This seems way to general to me. I would propose:

• arcgis-rest-client-js - this would mean all our packages would be arcgis-rest-client-js-*. Kinda long.
• arcgis-rest-js - this would mean all our packages would be arcgis-rest-js-*. Kinda long.
• @esri/rest -client - no need to use js in the NPM and package names. npm install @esri/rest-client-core.

Come up with something else like Koop, Winnow, Terraformer, ect... I've pulled from architectural terms before so I'll suggest some:

• truss - taken on npm. Could do arcgis-truss
• quoin - abandoned name from my first attempt at a Dojo based framework.
• arris - A modem company, but also unclaimed on NPM.
• trestle - also taken on NPM could do arcgis-trestle

Implement `UserCredentials` and `ApplicationCredentials`

Based on discussion in https://github.com/ArcGIS/arcgis-rest-js/issues/5 we are going to implement authentication as follows:

import { request, UserSession, ErrorTypes } from 'arcgis-rest-core';

const session = new UserSession(/* ... */);

session.on('error', (error) => {
  // I'm still not sure if we should actually allow retries here or not, see comments...
});

// we would have to teach request how to use `AppAuthentication` to recover from
// auth failures. Hopefully via an interface so it doesn't bloat the core repo.
request(url, params, {
  authentication: session
}).then(/* .. */).catch((error) => {
  switch (error.name) {
    case ErrorTypes.ArcGISRequestError:
      // general error from the server
      break;
    case ErrorTypes.ArcGISAuthError:
      // authentication error from the server
      break;
    default:
      // some other kind of error, probably from the network or fetch
      break;
  }
});

// we would have to pass the `authentication` object down through all calls.
geocode(params, {
  url: '...', // override default geocode URL?
  authentication: session
}).then(/* ... */).catch(/* ... */);

Since the request method and the UserSession/AuthSession objects will be closely coupled now I think this should all go into arcgis-rest-core.

At a glance this means adding:

• A new error type ArcGISAuthError.
• A new enum ErrorTypes which will expose ArcGISAuthError and ArcGISRequestError
• checkForErrors will have to throw ArcGISAuthError or ArcGISRequestError depending on what went wrong.

Now for some of the more challenging things:

Retrying authentication requests

The more I think about it the more I feel that we shouldn't try to implement automatic request retrying for a few reasons:

1. Implementation of retrying is hard and highly app specific.
2. Retrying is often async since you have to prompt for user input or wait for an async call. This is hard when resolving a promise chain.
3. Retrying (as should above) means that Session have to be event emitters which means more dependencies.

I wouldn't be opposed to "simple retrying" like so:

const session = new UserSession(/* ... */);

function retryRequest ([url, params, options]) {
  return new Promise((resolve, reject) => {
    // prompt for auth would prompt the user for sign in then return a new UserSession
    promptForAuth(url).then((newSession) => {
      resolve(session);
    })
  })
}

request(url, params, {
  authentication: session
})

// handle an auth error
.catch((error) => {
  if (error.name === ErrorTypes.ArcGISAuthError) {
    return error.retry(retryRequest(error.originalRequestParams));
  }

  throw error; // if this isn't an auth error keep dying...
})
.catch(error => {
  // this could still be an auth error from your retry or a different error...
})

// handle a successful request from `error.retry` or elsewhere.
.then(response => {
  console.log(response);
});

This makes retrying requests when auth fails relatively simple and it makes sense in terms of the promise-based implementation we have today. If we want to implement retries this is the way I would handle it.

"Federation"

In the JS API the identity manager does "Federation" between servers and portals. It works like this:

1. esri.request is called.
2. IdentityManager looks up to see if the server being called is on the list of CORS enabled servers it knows about.
   a. If the server is on the list of CORS enabled servers the request is sent. IdentityManager saves this in its internal cache of "server infos".
   b. If the request is not on the list of CORS enabled servers a request is first sent to a request is sent to ${SERVER}/arcgis/rest/info?f=json if the request responds we know the server supports CORS then the request is sent. IdentityManager saves this in its internal cache of "server infos".
3. If the request sent in step 2 fails with an auth error the JS API try the following steps in order:
   1. Ask IdentityManager for the token for the server the service sits on.
   2. If there is no token lookup if we know the server info this the server this service sits on:
      a. If we know the server info (see 2-b or 3-2-b) then proceed to step 4
      b. If we do not know the servers server info look it up (step 2-b). IdentityManager saves this in its internal cache of "server infos" and we can proceed to step 4
4. we now know the server info of the service we are trying to make a request to. We need to see if we have a token for the servers owningSystemUrl. if the system has no owningSystemUrl then the user authenticates directly with the server but I'm not entirely sure how that works.
   a. we have a token for the owningSystemUrl cached in IdentityManager so make a request to ${owningSystemUrl}/sharing/generateToken and ask it to generate a token for the server our service is sitting on. This token is cached in IdentityManager. Goto step 5
   b. We do not have a token for the owningSystemUrl, popup the identity manager and ask the user to authenticate into the owningSystemUrl. Once the user signs their token is cached in IdentityManager. repeat step 4-a.
5. Go to step 1 but with the token in the params.

In practice here is how this works for the following:

require(["esri/request"], function(request, id) {
  request('https://traffic.arcgis.com/arcgis/rest/services/World/Traffic/MapServer?f=json');
});

1. since traffic.arcgis.com is not in the list of CORS enabled servers we need to make a request to https://traffic.arcgis.com/arcgis/rest/info?f=json.
2. Now that we know the service supports CORS (because that request succeeded) we make a request to https://traffic.arcgis.com/arcgis/rest/services/World/Traffic/MapServer?f=json which fails with "Token Required".
3. Another request to https://traffic.arcgis.com/arcgis/rest/info?f=json is made. We know the owning system URL is https://www.arcgis.com.
4. Internally we have no token for https://traffic.arcgis.com.
5. Internally we have no token for https://www.arcgis.com.
6. Popup the auth dialog.

Here is how the same request works with a token registered.

require([
  "esri/request",
  "esri/identity/IdentityManager"
], function(request, id) {
  id.registerToken({
    token: "TOKEN",
    server: "https://www.arcgis.com/sharing/rest"
  });
});

Replace token with a token from a user or app.

1. since traffic.arcgis.com is not in the list of CORS enabled servers we need to make a request to https://traffic.arcgis.com/arcgis/rest/info?f=json.
2. Now that we know the service supports CORS (because that request succeeded) we make a request to https://traffic.arcgis.com/arcgis/rest/services/World/Traffic/MapServer?f=json which fails with "Token Required".
3. Another request to https://traffic.arcgis.com/arcgis/rest/info?f=json is made. We know the owning system URL is https://www.arcgis.com.
4. Internally we have no token for https://traffic.arcgis.com.
5. Internally we have no token for https://www.arcgis.com.
6. We DO have a token for https://www.arcgis.com so we can call https://www.arcgis.com/sharing/generateToken and ask it for a token for the https://traffic.arcgis.com/arcgis/rest/services/World/Traffic/MapServer
7. Retry the request with out shiny new token!
8. Request successful.

In short this is really complicated but does accomplish a few goals.

• Security - the users portal token is never sent to a server. This prevents malicious attacks like standing up a fake service and stealing portal tokens.
• Handling CORS support - by testing servers for CORS support we can also support older browsers and servers where people have CORS turned off by recommending a proxy to them.

This entire infrastructure is setup most to support the security use case. Doing the full generate token dance, especially with known services like traffic.arcgis.com seems silly. If someone has man-in-the-middled you and broken HTTPS (somehow) they can see your portal key anyway when it gets sent to /sharing/generateToken. If you send your portal token to a malicious service you 1 open up something with a malicious resource in it, 2 make a request to that resource, 3 sign in. It would probably be easier to just spoof the JS API experience and steal portal tokens in a simple phishing attack.

I would propose the following:

1. Each session maintains a list of trustedServers which defaults to any server under the arcgis.com. Developers can add additional servers.
2. If the request is to a trustedServer in that session attach the token and make the request. This works because tokens from the owningSytemUrl are also valid on any federated servers.
3. If the request is not to a trustedServer do the following:
   • Get the server URL of the request
   • Lookup the server info or call server/info to get the owningSystemUrl.
   • Lookup to see if we have a token for the owningSystemUrl
   • Call owningSystemUrl/generateToken to get a token for that server. Save this token and server info so we can look it up again later.
   • make the request with our new token.

This minimizes extra requests as much as possible. Standard AGOL users should see no additional requests. Users on Enterprise should see an extra server/info and owningSystemUrl/generateToken once per ArcGIS Server they call.

---

Final Design

import { request, UserSession, ErrorTypes } from 'arcgis-rest-core';

const session = new UserSession(/* ... */);

function getNewSession ({url, params, options}) {
  return new Promise((resolve, reject) => {
    // prompt for auth would prompt the user for sign in then return a new UserSession
    promptForAuth(url).then((newSession) => {
      resolve(session);
    })
  })
}

// we would have to teach request how to use `AppAuthentication` to recover from
// auth failures. Hopefully via an interface so it doesn't bloat the core repo.
request(url, params, {
  authentication: session
})

// handle errors first so we can retry...
.catch((error) => {
  switch (error.name) {
    case ErrorTypes.ArcGISRequestError:
      // general error from the server
      break;
    case ErrorTypes.ArcGISAuthError:
      return error.retryRequest(getNewSession(error.requestOptions))
      break;
    default:
      // some other kind of error, probably from the network or fetch
      break;
  }
})

// handle success either from the original request or retryRequest
.then(/* .. */);

Add CONTRIBUTING.md

If we are thinking about open sourcing this soon we should really create the CONTRIBUTING.md.

@jgravois I think you probably handle this but I think we should specifically mention:

1. Recommend installing the TypeScript, TSLint, Prettier and EditorConfig extensions for your text editor.
2. How to run and debug the tests
3. How to run the docs site locally
4. How to edit guides in the docs

Design API for @esri/arcgis-rest-geoenrichment module

Design an API for accessing the Geoenrichment API https://developers.arcgis.com/rest/geoenrichment/api-reference/geoenrichment-service-overview.htm.

Throw authentication error when unable to refresh user's session

Currently when UserSession is unable to refresh the session, it just throws a generic error. It should throw an ArcGISAuthError.

https://github.com/ArcGIS/arcgis-rest-js/blob/7e5f2e5e862e7f2aaffcbc76a19ca37a44621c47/packages/arcgis-rest-auth/src/UserSession.ts#L513-L523

Link to docs

Are the existing docs pushed to GH Pages or anything yet?

How to handle null, undefined and boolean, values?

After investigating why tests in #16 were failing I found the root cause to be that if you pass null or undefined as a value the form-data library in Node will throw an error. It appears form-data/form-data#336 is the root cause of this but form-data/form-data#137 might also be an issue.

In general based on http://resources.arcgis.com/en/help/arcgis-rest-api/#/Query_Feature_Service_Layer/02r3000000r1000000/ and http://resources.arcgis.com/en/help/arcgis-rest-api/#/Update_Item/02r30000009s000000/ that we should treat boolean values as strings "true" or "false".

However what to do about null or undefined? In general I would argue that if a value is null or undefined we shouldn't send that key to the API. However in some cases like http://resources.arcgis.com/en/help/arcgis-rest-api/#/Update_Item/02r30000009s000000/ with the clearEmptyFields param you might want to send empty fields to the API.

I'm going to implement this as follows:

• processParams will now exclude the following from being encoded: null, undefined, function.
• processParams will encode true or false as the strings "true" and "false".

In general this will mean that if you want to send an empty value to the API you will have to manually declare it like so:

updateItem(itemid, {
   description: "",
   clearEmptyFields: true
});

This seems pretty sensible to me.

Design API for @esri/arcgis-rest-routing module

Design API with support for the routing services https://resources.arcgis.com/en/help/arcgis-rest-api/#/Overview_of_network_analysis_services/02r30000001s000000/ probably using https://github.com/Esri/esri-leaflet-routing as a base.

Design API for @esri/arcgis-rest-sharing module

API for interacting with the sharing API (items, groups, ect...).

Design API for @esri/arcgis-rest-portal module

Module that will encapsulate CRUD methods for the portal resource.

Initially, has getPortal and getSelf

clearly doc requirements for lib use

• fetch polyfill
• user endpoints need to support CORS
• browser/node versions
• ?

when to open source?

are there any drawbacks to:

• getting this repo into a state that it can be open sourced
• adding an experimental flag
• putting out a call for public feedback
• tagging v1.0.0 once we've got something we're fairly happy with?

putting on my open source evangelism hat, i think it'd be a 💯 story to show devs from different teams across the company collaborating on a low-level tool from the ground up 'in the open'

Handling owner/username in portal API calls

When working with user content, its common that we have to use an endpoint similar to

https://www.arcgis.com/sharing/rest/content/users/${username}/...

which requires getting the user's username. The items package handles this by having the user input an owner parameter for functions that require it. However, in the feature-services-admin package I've been working on, I've been assuming that the authentication is a UserSession and pulling the username from requestOptions.authentication.username.

Is it always the case that user authentication is needed when calling an endpoint like https://www.arcgis.com/sharing/rest/content/users/${username}/...? If so, I think pulling from UserSession makes more sense. However, this would make requestOptions required, which I'm not sure matches the expected use of requestOptions.

I want to get people's thoughts on this so we have a consistent pattern. @patrickarlt @dbouwman @tomwayson @jgravois @araedavis

As a side note, issues like this make me think it would be good to set up an API design reference in order to keep track of cross-package design considerations.

Build script to host on gh-pages

We should have a script that builds the site and deploys it to gh-pages. For extra bonus points we could automatically deploy to gh-pages with travis CI https://docs.travis-ci.com/user/deployment/pages/.