luckymarmot / api-flow Goto Github PK

we should be able to see the name of requests via comments in cURL-exporter

With the addition of names to auth methods, it is possible to put auths methods that are shared in a reference environment.

Some additional work in Paw is still required to support variations on a shared object (e.g. scopes in OAuth2) before this can be implemented.

Swagger v3 will soon be stable. it would be nice to have the possibility to migrate swagger v2 to swagger v3

null authentication is slightly different from no authentication.

From a semantic point of view, it means that the request can be done without any authentication, while no authentication could also mean that the authentications were not provided in the file for various reason (e.g. the format does not support this auth format).

API-Flow should support imports from cURL.

• Improve Reference support through the use of FileReference and RemoteReference
• Support URI Parameters through Environment variables

The current architecture and models could be strongly improved through a careful refactoring of the app.

Issues

General Issues

Here are some of the current issues that we encounter:

• no support for multiple bodyTypes
• no support for multiple contentTypes
• no support for urlParameters
• no support for parameter constraints
• no support for flattening nested groups
• no support for remote files
• no support for examples
• limited support of responses
• limited support of schemas

Structural issues of the RequestContext object

• RequestContext.schema was added to solve schema references issues with the Swagger parser
• RequestContext.environments was added to solve Environment variables issues with the Postman parser

These fields feel like ad-hoc solutions to every format we meet. This is bad design, and we should move away from it. Ultimately, environments variables and schemas are only references that can be resolved locally.

Enhancements

Here are cool things we could integrate:

• json-schema-faker. A pretty cool library that allows one to generate random json objects that comply with a schema
• additional authentication method support.
• improved schema parsing when it's given as a raw string instead of an object

Going Forward

Here is a rough draft of the modification we suggest to make:

• Rename KeyValue to Parameter model, that will be used for queryParams, urlEncodedParams and formDataParams, and headers.
• Create a Constraint model that defines under which constraints a Parameter can be used. Each Parameter would therefore have an external list of Constraint under which they can be used/included.
• Add a Constraint list as an internal field for Parameter, that would describe the constraint the the Parameter itself must respect when being sent.
• Regroup headers, queryParams, urlEncodedParams, and formDataParams as fields of a ParameterContainer model.
• Create a Body model, that contains a Constraint list. It should be possible to filter the ParameterContainer based on this Constraint list to generate the actual content of the request.
• Create a bodies field in Request, that would contain a list of Body.
• Add description, name and example fields everywhere.
• Add a flatten method to Group
• Add a generate method to Parameter, that uses Faker.js or Chance.js to create realistic and compliant data. This would be useful for on the fly example generation.
• Add name, version, description, and other descriptive fields directly to RequestContext
• Move all the current exporters in exporters/paw -- changed to serializers/paw
• Rename Importer files as Exporter -- changed to Serializer
• Rename Parser files as Importer -- kept as Parser
• Rename immutables directory as models
• Rename RequestContext to Core
• Improve Auth module structure by regrouping all the authentication methods in a single Auth object.
• Improve Core module structure by grouping the multiple classes intelligently (Didn't look into it yet)
• Improve Context by removing the schema and environment fields. Create a references field instead that would contain all the references that appear in the model. Maybe use a ReferenceContainer, similar to the ParameterContainer?

TODO issues

• Parser - Swagger
• Parser - RAML
• Serializer - Swagger

Potential Growth

• CLI

Hi,

I'm trying to import postman's data, I did as README told, clone the repo, run "make install" with exit code 0, but there's no "./node_modules/lib/api-flow.js" file, if I try to use "./bin/api-flow.js", error comes:

node bin/api-flow.js /tmp/v2.json -f postman-2 -t raml
/tmp/tmp.jDw4AygtOF/API-Flow/lib/parsers/swagger/Parser.js:105
throw new Error(m + _tv2.default.error);
^

Error: Invalid Swagger File (invalid schema / version < 2.0):
ValidationError: Invalid type: string (expected object)
at SwaggerParser.parse (/tmp/tmp.jDw4AygtOF/API-Flow/lib/parsers/swagger/Parser.js:105:23)
at FlowCLI.run (/tmp/tmp.jDw4AygtOF/API-Flow/lib/runners/flow-node.js:269:34)
at Object. (/tmp/tmp.jDw4AygtOF/API-Flow/bin/api-flow.js:4:5)
at Module._compile (module.js:541:32)
at Object.Module._extensions..js (module.js:550:10)
at Module.load (module.js:458:32)
at tryModuleLoad (module.js:417:12)
at Function.Module._load (module.js:409:3)
at Module.runMain (module.js:575:10)
at run (bootstrap_node.js:352:7)

currently OAuth1Auth is not defined using the correct parameters.
It should.

See Paw-SwaggerImporter issue#9 for more information

./random.sh 2>1 is not supported

It feels heavy to import only one request in a group (especially for curl). Can we make it so if there's only one request to be imported it's not wrapped in a group?

We are using Swagger in an admittedly unusual way by annotating our swagger.yaml with some internal info, e.g. which controller class of our service will handle the request.

This looks something like this:

  /somePath:
    x-swagger-router-controller: somePathController
    get:
      description: Returns the some data

Now when using the Swagger Importer, it parses x-swagger-router-controller as just another HTTP method which us understandable but doesn't make sense in our case.
Would you be open to maybe add either a filtering for e.g. everything that starts with x- or maybe simply ignore anything that's not in a valid-HTTP-methods whitelist?

cURL importer is an object with field "requests".
Should be "context" and "group"

Core optimizations

• add an in field in Parameter
• rename Body object to RequestEnvironment or something like that, since that's what it actually is - it has nothing to do with the body content
• improve the Schema use.
• add default field to Parameter, for use case where the Parameter.value field has to be overloaded with other definitions (e.g. when type is array)
• Add a name field to Auth methods

Parsers optimizations:

RAML

• Support external definitions for key components, such as paths

Serializers optimizations:

RAML

• Use tags and type to extrapolate traits and resource types
• Support multiple auths of the same type being defined
• Support exotic auths (hawk, negotiate, etc.)

Swagger

• extrapolate global consumes/produces
• extrapolate basePath,
• extrapolate global parameter definitions

Resolvers optimizations:

• allow deep linking. Context-dependent links such as ../root.json#/definitions/User should be supported even in files that are resolved
• maybe pass the resolution stage before the parsing stage, to preemptively load the dependencies?

See Paw-SwaggerImporter issue#4 for more information

Swagger v3 will soon be stable. it would be nice to have the possibility to import swagger v3.

Currently the swagger importer does not extract scopes from the swagger file, although it could be done relatively easily.

It would be nice to have a method to rapidly detect the format of a file, if it supported.

_loadSwaggerCollection should only try to parse in one format (JSON/YAML) if an extension parameter is provided.

when exporting the API's from POW to PostMan or swagger, groups get ignored,.

API-Flow should support configuration files to give users a finer control on the behavior of the tool.

Here are some of the options which could be useful:

Parsers

General

• name: [swagger]: the name of the parser (e.g. swagger, raml, postman, curl, etc.). Default is swagger.
• instance: [null] if name is custom, an instance field could be used to pass a custom parser

Swagger

• no-x: [false]. If this option is set to true, the parser will not attempt to parse x-* patterned fields that it recognizes
• version: [2.0]. Not supported in v0.1. In the future, changing this option will allow user to parse v1.2 swagger files. If set to auto, the parser will attempt to guess which version is the most suited.

RAML

• warn-on-missing: [no-warn]. The level at which missing files raise warnings for the user. It can be any of [group, request, parameter, schema, example, none]

Postman:

• flatten: [null]. If flatten is set to a number, it will represent the maximum depth of nested references that are supported before resolution. e.g. if flatten: 1 the following reference user{{{{domain}}-userId}} will be flattened to user{{production-userId}}. set flatten to 0 to resolve immediately.

Curl:

• bash: [inline]. Not supported in v0.1. Search for bash environment variables.
• speech-marks: [false]. If set to true, the parser will try to parse unicode speech marks as regular speech marks potentially delimiting keywords.
• 'strip-newlines: [false]'. when encountering a curl command, strips out new lines from the input arguments if set to true.

Resolver

General

• name: [web]: the name of the resolver (e.g. node, web, paw, atom, etc.)
• instance: [null]. If name is custom, an instance field could be used to pass a custom resolver
• cwd: [.]. Not supported in v0.1. the directory/url from which to resolve all the uris.
• resolve: [true]. resolve accepts either a boolean, an object, or an array, which define how the granularity of the resolution.

Resolve parameter:

• [true/false]. if set to false, no resolution is done. if set to true, the resolver will try to resolve as much as possible.
• [object]: the object should be of the following form:

{
    remote: [true/false],
    local: [true/false],
    custom: [array]
}

• [array], is equivalent to [object] with remote and local set to true, and custom set to [array]. The array should respect the following form:

[
    {
         relative: '#/some/relative/uri',
         resolve: [true/false],
         value: [null/*]
    },
    ...
]

Serializers

General

• name: [web]: the name of the serializer (e.g. postman, raml, swagger, curl, etc.)
• instance: [null]. If name is custom, an instance field could be used to pass a custom serializer
• obfuscate: [false]. if set to true, the serializer will try to obfuscate commonly used authentication fields, such as user, username, password, secret, etc.

Swagger:

• no-x: [false] if set to true, drops all x-* patterned fields that provide additional informations created by API-Flow.
• format: [json]. alternative value is yaml

Paw

• domain: [null], if domain is set, use that name instead of the ones from API-Flow as a domain name.

Postman

• environment: [API-Flow]. if environment is set, use that name instead of the ones from API-Flow as an environment name.

curl

• no-comments: [false]. if set to true, output will have no comments to help readability.
• no-vars: [false]. if set to true, output will have no environment variables.

There are probably other options that could be added.

example of a file using parameters at the path level:
https://raw.githubusercontent.com/APIs-guru/api-models/master/APIs/googleapis.com/siteVerification/v1/swagger.yaml

Currently, the RAML Parser does not check for file references inside its content, despite having a shimming file reader that replaces missing files with a ::fileRef:: reference file tag.

We should be able to parse the content to check if there's a fileRef in it.

requested by a user

See Paw-SwaggerImporter issue#12 for more information.

It seems to me like a confusing name for users who haven't heard about JSF yet. Maybe somehting like "Use value randomizers for environment variables" etc.

If the basePath is /, it is not properly trimmed, given results such as https://api.github.com//feeds instead of https://api.github.com/feeds

I think the package.json can be improved. I don't know well the standards in organizing a npm module, but probably having a main index.js script would make sense? We point to src/CurlImporter.js as a main, and it's not a valid path.

I just imported the Swagger file from Instagram (on APIs.guru) and it doesn't set the Authorization URL for me:

Clearly the Swagger has it:

securityDefinitions:
  api_key:
    in: query
    name: access_token
    type: apiKey
  instagram_auth:
    authorizationUrl: 'https://instagram.com/oauth/authorize/'
    flow: implicit
    scopes:
      basic: to read a user's profile info and media (granted by default)
      comments: to post and delete comments on a user's behalf
      follower_list: to read the list of followers and followed-by users
      likes: to like and unlike media on a user's behalf
      public_content: to read any public profile info and media on a user’s behalf
      relationships: to follow and unfollow accounts on a user's behalf
    type: oauth2

Swagger define a root level field named parameters, which can be used to share parameters between requests, and the Swagger Importer could move these into environment variables.

See Paw-SwaggerImporter issue#5 for more information

We have been using a char-by-char string parsing which may lead to a performance hit. As per this comment on PR #29:

What's the goal of this char-by-char string parsing? Why not simply a regexp to match {{env}}? /\{\{[^}]+\}\}/ would work nicely…
If this is to be able to resolve nested env variables, I'm concerned that we are trading a "nice to have" feature that is likely to not even be officially supported by Postman – and may not be even used by anyone (why on Earth would you nest environment variable names?!) – for a massive performance hit. Most JavaScript regexp engines are probably written in plain C, and is super efficient, doing this by hand here, on every single string we import, may be really slow…

Maybe should we keep this function around, as an alternative? And quickly make a simpler one (can be done in a few lines I think). It's also about code complexity, I wouldn't want to resolve bugs in this myself :)

Would be nice to look into this later, and see what can be done. I'm thinking of an option to resolve nested environment names, which would trigger this char-by-char stack based parsing instead of a simpler and more efficient regexp-based parser.

Swagger and RAML require requests to have unique paths/method product.

If a user has multiple requests that share the same path and method, the export is not fully compatible with Swagger and RAML, and it would be a good idea to raise a warning or something.

API-Flow should raise an error on failure, instead of the silent logging behavior we currently have.

So we make sure our code is always nice.

Currently, our postman parser is a bit lacking on OAuth2.
With more documentation/samples, it should be possible to improve it.

TODO

• Improve support for parameters of type array, as they behave very differently from other types
• Improve path parameter support by saving them in environment variables
• Remove schema field from Response, since it has been moved into parameters

Potential TODO

• Support tags and operationId ?
• Support ^x-.* patterns ?
• Merge _extractResponseExternals and _extractExternals
• Merge _extractResponseBodies and _extractBodies -- behaviors are somewhat different, not merged

Note: This list will be updated as we progress in the migration

There can be multiple auths of the same type, which are hard to tell apart from each other when trying to export them. This is an issue for swagger and raml, as they use a root folder containing the security definitions

• How do we deal with host/schemes parameters?

note: this will be updated as we progress in the migration

When importing to paw the text-body newlines are represented as \n dynamic values rather than \n chars.