Coder Social home page Coder Social logo

adobe / aio-lib-core-networking Goto Github PK

View Code? Open in Web Editor NEW
3.0 23.0 10.0 251 KB

This repository contains all HTTP client related Utils to be used by all SDKs

Home Page: https://adobe.io

License: Apache License 2.0

JavaScript 100.00%
aio-lib aio-lib-core adobe adobe-io networking aio-sdk-core serverless nodejs npm

aio-lib-core-networking's Introduction

Version Downloads/week Node.js CI License Codecov Coverage

Adobe I/O Core Networking Lib

Installing

$ npm install @adobe/aio-lib-core-networking

Usage

  1. Initialize the SDK
const { HttpExponentialBackoff, createFetch } = require('@adobe/aio-lib-core-networking')
const fetchRetry = new HttpExponentialBackoff()
const proxyFetch = createFetch()
  1. Call methods using the initialized SDK
const { HttpExponentialBackoff, createFetch } = require('@adobe/aio-lib-core-networking')
const fetchRetry = new HttpExponentialBackoff()
async function sdkTest() {

  return new Promise((resolve, reject) => {
    fetchRetry.exponentialBackoff(url, requestOptions, retryOptions, retryOn, retryDelay)
    .then((response) => {
      if (!response.ok) {
        throw Error(reduceError(response))
      }
      resolve(response.json())
    })
    .catch(err => {
      reject(
        new codes.ERROR_GET_SOMETHING({ sdkDetails, messageValues: err }))
    })
  }) 
}

let proxyFetch 
// this will get the proxy settings from the the HTTP_PROXY or HTTPS_PROXY environment variables, if set
proxyFetch = createFetch()

// this will use the passed in proxy settings. Embed basic auth in the url, if required
proxyFetch = createFetch({ proxyUrl: 'http://my.proxy:8080' })

// if the proxy settings are not passed in, and not available in the HTTP_PROXY or HTTPS_PROXY environment variables, it falls back to a simple fetch
const simpleFetch = createFetch()

Classes

HttpExponentialBackoff

This class provides methods to implement fetch with retries. The retries use exponential backoff strategy with defaults set to max of 3 retries and initial Delay as 100ms

ProxyFetch

This provides a wrapper for fetch that facilitates proxy auth authorization.

Functions

createFetch([proxyAuthOptions])function

Return the appropriate Fetch function depending on proxy settings.

parseRetryAfterHeader(header)number

Parse the Retry-After header Spec: https://tools.ietf.org/html/rfc7231#section-7.1.3

Typedefs

RetryOptions : object

Fetch Retry Options

ProxyAuthOptions : object

Proxy Auth Options

HttpExponentialBackoff

This class provides methods to implement fetch with retries. The retries use exponential backoff strategy with defaults set to max of 3 retries and initial Delay as 100ms

Kind: global class

httpExponentialBackoff.exponentialBackoff(url, requestOptions, [retryOptions], [retryOn], [retryDelay]) ⇒ Promise.<Response>

This function will retry connecting to a url end-point, with exponential backoff. Returns a Promise.

Kind: instance method of HttpExponentialBackoff
Returns: Promise.<Response> - Promise object representing the http response

Param Type Description
url string endpoint url
requestOptions object | Request request options
[retryOptions] RetryOptions (optional) retry options
[retryOn] function | Array (optional) Function or Array. If provided, will be used instead of the default
[retryDelay] function | number (optional) Function or number. If provided, will be used instead of the default

ProxyFetch

This provides a wrapper for fetch that facilitates proxy auth authorization.

Kind: global class

new ProxyFetch(authOptions)

Initialize this class with Proxy auth options

Param Type Description
authOptions ProxyAuthOptions the auth options to connect with

proxyFetch.fetch(resource, options) ⇒ Promise.<Response>

Fetch function, using the configured NTLM Auth options.

Kind: instance method of ProxyFetch
Returns: Promise.<Response> - Promise object representing the http response

Param Type Description
resource string | Request the url or Request object to fetch from
options object the fetch options

createFetch([proxyAuthOptions]) ⇒ function

Return the appropriate Fetch function depending on proxy settings.

Kind: global function
Returns: function - the Fetch API function

Param Type Description
[proxyAuthOptions] ProxyAuthOptions the proxy auth options

parseRetryAfterHeader(header) ⇒ number

Parse the Retry-After header Spec: https://tools.ietf.org/html/rfc7231#section-7.1.3

Kind: global function
Returns: number - Number of milliseconds to sleep until the next call to getEventsFromJournal

Param Type Description
header string Retry-After header value

RetryOptions : object

Fetch Retry Options

Kind: global typedef
Properties

Name Type Description
maxRetries number the maximum number of retries to try (default:3)
initialDelayInMillis number the initial delay in milliseconds (default:100ms)
proxy ProxyAuthOptions the (optional) proxy auth options

ProxyAuthOptions : object

Proxy Auth Options

Kind: global typedef
Properties

Name Type Description
proxyUrl string the proxy's url
rejectUnauthorized boolean set to false to not reject unauthorized server certs

Debug Logs

LOG_LEVEL=debug <your_call_here>

Prepend the LOG_LEVEL environment variable and debug value to the call that invokes your function, on the command line. This should output a lot of debug data for your SDK calls.

Contributing

Contributions are welcome! Read the Contributing Guide for more information.

Licensing

This project is licensed under the Apache V2 License. See LICENSE for more information.

aio-lib-core-networking's People

Contributors

amulyakashyap09 avatar arjuncooliitr avatar dependabot[bot] avatar greenkeeper[bot] avatar himavanth avatar macdonst avatar michaelgoberling avatar nicdard avatar pahupe avatar purplecabbage avatar sandeep-paliwal avatar sangeetha5491 avatar shazron avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

aio-lib-core-networking's Issues

An in-range update of fetch-mock is breaking the build 🚨


☝️ Important announcement: Greenkeeper will be saying goodbye 👋 and passing the torch to Snyk on June 3rd, 2020! Find out how to migrate to Snyk and more at greenkeeper.io


The devDependency fetch-mock was updated from 9.3.1 to 9.4.0.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

fetch-mock is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details

Commits

The new version differs by 10 commits.

  • 3cdc7ba fixed broken tests
  • b1f33ec fix linting
  • 87a2758 Merge pull request #532 from chet-manley/feat-return-node-fetch-interface
  • d7a6577 feat: mock node-fetch's export pattern
  • 3500688 Merge pull request #526 from pabloosso/patch-1
  • 8c231ac Fixed typo
  • e25f30a Merge pull request #525 from willheslam/patch-1
  • 5817bbf Fix typo in mock_response.md
  • 41b238f Update cheatsheet.md
  • aa8311e Update cheatsheet.md

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

proxy config - special character handling (username & password) - align with tools like curl

Context:
A customer has an existing proxy server running at http://proxy.company.com:7070. The proxy requires authentication. A technical proxy user with a password has been setup where the user name contains a special character.

Example:
Username: domain\user
Password: test

The customer expects to use this technical proxy user to authenticate all outgoing calls, e.g. curl, aio, ...

curl:
curl supports all of the following proxy configurations (unescaped, url-encoded uppercase, url-encoded lowercase):

HTTPS_PROXY='http://domain\user:[email protected]:7070' curl -v https://cloudmanager.adobe.io/api/programs

HTTPS_PROXY='http://domain%5Cuser:[email protected]:7070' curl -v https://cloudmanager.adobe.io/api/programs

HTTPS_PROXY='http://domain%5cuser:[email protected]:7070' curl -v https://cloudmanager.adobe.io/api/programs

All three calls will authenticate at the proxy server, will reach the service endpoint and return "Oauth token is missing."

It shall be noted that the curl debug output (headers) indicates that (in all three cases) the special character \ is not URL-encoded anymore before it gets Base64-encoded:

...
Proxy auth using Basic with user 'domain\user'
Proxy-Authorization: Basic ZG9tYWluXHVzZXI6dGVzdA== 
...

(which after base64-decode is domain\user:test)

Expected Behaviour

It is expected that aio with proxy support works similar to tools like curl, i.e. the same proxy + user can be used with curl and with aio. This is unfortunately not the case.

Actual Behaviour

When trying to use the same proxy with aio, all requests fail:

HTTPS_PROXY='http://domain\user:[email protected]:7070' aio login -d
=> errors out due to a Node-specific URL decomposition issue (treats 'domain\user' as a path, not as a user name)

HTTPS_PROXY='http://domain%5Cuser:[email protected]:7070' aio login -d
=> errors out with a proxy authentication error (HTTP 407)

HTTPS_PROXY='http://domain%5cuser:[email protected]:7070' aio login -d
=> errors out with a proxy authentication error (HTTP 407)

While the first case cannot be fixed (here), the second and third case should work. The reason why the proxy authentication fails is that the username and password URL-encoding is not reversed before the proxy URL is converted to proxyOpts and passed to the respective proxy agent.

Reproduce Scenario (including but not limited to)

Steps to Reproduce

Setup a local proxy server (e.g. using docker ubuntu-squid) and define a user: 'domain\user' with password 'test'
In the example above, it is listening on port 7070 using the http protocol.

Setup aio:

nvm use 16.16.0                               

aio config:set ims.contexts.aio-cli-plugin-cloudmanager ~/config.json --file --json
aio config:set ims.contexts.aio-cli-plugin-cloudmanager.private_key ~/private.key --file
aio config:set cloudmanager_orgid A1…@AdobeOrg
aio config:set cloudmanager_programid 12345

aio auth ctx -s aio-cli-plugin-cloudmanager

Then run the scripts above: (curl and aio login -d) with the proxy configured

Platform and Version

Mac on MacOS 13.1
aio v9.1.1 with aio-lib-core-networking v3.0.0

aio info:

System:
    OS: macOS 13.1
    CPU: (10) arm64 Apple M1 Max
    Memory: 564.42 MB / 32.00 GB
    Shell: 5.8.1 - /bin/zsh
  Binaries:
    Node: 16.16.0 - ~/.nvm/versions/node/v16.16.0/bin/node
    Yarn: Not Found
    npm: 8.11.0 - ~/.nvm/versions/node/v16.16.0/bin/npm
  Virtualization:
    Docker: 20.10.21 - /usr/local/bin/docker
  npmGlobalPackages:
    @adobe/aio-cli: 9.1.1

  Proxies:
    http: (not set)
    https: http://domain%5cuser:[email protected]:7070 (scheme mismatch)
  CLI plugins:
    core:
      @adobe/aio-cli 9.1.1
      @adobe/aio-cli-plugin-app 10.0.3
      @adobe/aio-cli-plugin-app-templates 1.5.1
      @adobe/aio-cli-plugin-auth 3.0.1
      @adobe/aio-cli-plugin-certificate 1.0.1
      @adobe/aio-cli-plugin-config 4.0.1
      @adobe/aio-cli-plugin-console 4.0.1
      @adobe/aio-cli-plugin-events 2.0.2
      @adobe/aio-cli-plugin-info 3.0.1
      @adobe/aio-cli-plugin-runtime 6.0.1
      @adobe/aio-cli-plugin-telemetry 1.1.0
      @oclif/plugin-autocomplete 1.3.8
      @oclif/plugin-help 5.1.20
      @oclif/plugin-not-found 2.3.11
      @oclif/plugin-plugins 2.1.9
      @oclif/plugin-warn-if-update-available 2.0.17
    user:
      @adobe/aio-cli-plugin-aem-rde 0.0.1-alpha.6
      @adobe/aio-cli-plugin-cloudmanager 4.0.0
    link:

Sample Code that illustrates the problem

Logs taken while reproducing problem

curl can access the service endpoint:
HTTPS_PROXY='http://domain%5Cuser:[email protected]:7070' curl https://cloudmanager.adobe.io/api/programs {"error_code":"403010","message":"Oauth token is missing."}

aio can't:
HTTPS_PROXY='http://domain%5Cuser:[email protected]:7070' aio login -d › Error: Cannot get token for context 'aio-cli-plugin-cloudmanager': 407 › (Proxy Authentication Required)

HTTP/HTTPS Proxy Agent Logic

if (proxyOpts.protocol.startsWith('https')) {
return new HttpsProxyAgent(proxyOpts)
} else {
return new HttpProxyAgent(proxyOpts)
}

This bases whether to use HTTP/HTTPS proxy based on what the proxy protocol is, however this logic is wrong, it should be based on what the end point protocol is:

Type Proxy Server
HttpProxyAgent HTTP HTTP
HttpProxyAgent HTTPS HTTP
HttpsProxyAgent HTTP HTTPS
HttpsProxyAgent HTTPS HTTPS

As a result, using an http proxy and connecting to an https endpoint causes the logic to pick HttpProxyAgent which then returns an error from the server as we are trying to connect to it via non HTTPS methods

An in-range update of @adobe/eslint-config-aio-lib-config is breaking the build 🚨


☝️ Important announcement: Greenkeeper will be saying goodbye 👋 and passing the torch to Snyk on June 3rd, 2020! Find out how to migrate to Snyk and more at greenkeeper.io


The devDependency @adobe/eslint-config-aio-lib-config was updated from 1.1.0 to 1.2.0.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

@adobe/eslint-config-aio-lib-config is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details

Release Notes for add plugin:node/recommended to check for extraneous/missing requires
  • feat: add plugin:node/recommended to check for extraneous/missing requires (#7) 15102de
  • chore(package): update eslint-plugin-jsdoc to version 23.0.0 (#6) 849c15b
  • chore(package): update eslint-plugin-jsdoc to version 22.0.0 (#5) a3788dd
  • Update README.md 7c2e18e
  • Update dependencies to enable Greenkeeper 🌴 (#4) 3a9468d

1.1.0...1.2.0

Commits

The new version differs by 6 commits.

  • f6a0062 1.2.0
  • 15102de feat: add plugin:node/recommended to check for extraneous/missing requires (#7)
  • 849c15b chore(package): update eslint-plugin-jsdoc to version 23.0.0 (#6)
  • a3788dd chore(package): update eslint-plugin-jsdoc to version 22.0.0 (#5)
  • 7c2e18e Update README.md
  • 3a9468d Update dependencies to enable Greenkeeper 🌴 (#4)

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

feat: add our own http client (with proxy support) and our own http.agent proxy

Related: adobe/aio-cli#224

Description

  1. client: Base it off the fetch API. Pay particular attention to the needs of of swagger-client custom userFetch property: https://github.com/swagger-api/swagger-js/blob/dc84649dd63b7a3a0733b24f828753b070ee2832/docs/usage/http-client.md#custom-fetch This will automatically use our http.agent Proxy below
  2. proxy: for adobe/aio-cli#224 - this can be used for example in the openwhisk library, where we can't specify our own client, but we can set the proxy/http.agent https://github.com/apache/openwhisk-client-js#constructor-options

Further Steps

All our aio libs, instead of using node-fetch or the global fetch, should use this implementation instead, and third-party libraries like swagger-client and others should use this custom fetch override. Similarly for openwhisk and the custom proxy

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.