azure / openapi-diff

Command line tool to detect breaking changes between two openapi specifications

About openapi-diff

This repository contains source code of openapi-diff aka oad aka "Breaking change detector tool" npm package. This package is invoked internally by the azure-rest-api-specs and azure-rest-api-specs-pr repos Swagger Breaking Change and Breaking Change(Cross-Version) GitHub checks, validating PRs submitted to them.

For description of the overall process of which oad is part of, see https://aka.ms/azsdk/specreview.

npm package

• @azure/oad
• package.json

Caution

Do not use the package oad . It is deprecated and obsolete.

How to run locally

See relevant section in CONTRIBUTING.md

openapi-diff's Issues

Add Gulp task to facilitate developer scenarios

• clean
• Build
• Test

Add a switch --strict to perform checks irrespective of the api-version change

[Rule] Adding read-only field to response

New property which can only be set by the server can be added to the response

[Rule] Adding new APIs to an existing service

When a new resource types is added, it does not require API version to be updated for existing types.

Check-in initial version of the existing tool & codebase

Process is terminated due to StackOverflowException

E:\repos\openapi-diff>oad compare E:\repos\azure-rest-api-specs\arm-resources\resources\2016-09-01\swagger\resources.json E:\repos\azure-rest-api-specs\arm-resources\resources\2017-05-10\swagger\resources.json
(node:16864) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: Command failed: C:\Users\vissha\.autorest\frameworks\dotnet.exe C:\Users\vissha\AppData\Roaming\npm\node_modules\oad\lib\dlls\OpenApiDiff.dll -o C:\Users\vissha\AppData\Local\Temp\old.json -n C:\Users\vissha\AppData\Local\Temp\new.json -JsonValidationMessages

Process is terminated due to StackOverflowException.

(node:16864) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

[Rule] Property type has changed

Property “foo” was a “boolean” in v1 but is changed to a "string". A client using the existing api-version tries to set it as a bool, but the service will fail since its now expecting a string. So, the api-version must be updated.

Enable exception chaining in Node CLI

Enable CI system on the repo

Build command line Node CLI

[Rule] Property name has changed

If a property was called “foo” in v1 of API and the name will be changed to “bar”, this requires an API-version change since this will result in a breaking change for the client.

Release version 0.1.0

Enable consumption from npm install -g oad

Filter output (Info/Warning/Error) based on cli parameter

Output of the CLI should be filtered based on the CLI flag

[Rule] Property is made required (from optional)

If property “foo” was optional in the request body of v1 and now it is required, this should result in an api-version change. If not changed, clients relying on older api-version will fail if this property is not passed.

Add configuration based rule execution / suppression

Food for thought

• Naming of the repo & command line
• CLI structure align with the AutoRest CLI and it's command-line experience
• long term we should leverage JSON-RPC to perform in-memory spec resolution
• Each comparison should handle minimal model of swagger 2.0
• Move towards TypeScript model from NodeJS based CLI

Feel free to add more stuff as you come across.

fyi: @dsgouda @veronicagg

[Rule] Behavior of existing API has changed

There is a functional change in what the API was doing. This will need to be determined on a case by case basis.

[Rule] Bug fixes to existing API

Bug fixes to existing API which don’t fall into one of the above categories of breaking changes as described above are fine

Think about how to leverage AutoRest for the tool

Initialize all the rules from the configuration file

Class ComparisonMessages should be initialized from the json file for all the rules.

• Initialization from config file
• Categorization from config file
• Run tools in context (Azure v/s NonAzure, SDK v/s NonSDK)
• Initialize the severity from the config file
• Generate documentation from master list of rules

Open door for CLI to consume config file that autorest understands

As of today the CLI takes individual swagger and resolves it to pass it along to the breaking change tool. As autorest input is very flexible in also consuming the config files, we should allow the consumption of config files as well

cc
@olydis 👍 Thanks for pointing this out

[Rule] Resource naming rules should not change

This could result in failures which would have earlier succeeded. Even if the rules become less strict, clients relying on earlier name constraints to perform local validation will fail.

[Rule] Allowed values for an enum have changed

Enum “foo” had allowed values as “val1” and “val2” in v1 of API. If now, the values allowed/accepted by the service are “val1”, “val2” and “val3”, client will fail to de-serialize if “val3” comes back in the response.

[Rule] API has been removed or renamed

V1 of API contract supported PUT /resourceType1/{resourceType1_name} but the service no longer supports this method. This scenario should follow the proper Azure API deprecation policy and must be done in an updated api-version.

Package the tool for consumption

[Rule] New property added to response

If a new property/field is added to the response an API, the GET-PUT pipeline will be broken. Consider the case where from portal a customer updates the value of a new property "A". Another customer does a GET of this resource using the SDK. The SDK will ignore the property since it does not understand it. From the SDK, the customer does a PUT using the model that was returned from the GET. This will overwrite the change made by the first customer from the portal.

Add minimal Guidance/documentation around each rules

Understand breaking change v/s Smooth changes

• Now we have tool integrated into the CI, keep monitoring it for false positives

Hookup AutoRest Linter PR Bot with the breaking change tool

Enhance and fix following Rules

Update / Correct each rules

• VersionsReversed
• ProtocolNoLongerSupported
• RequestBodyFormatNoLongerSupported
• RemovedPath
• RemovedDefinition
• RemovedClientParameter
• ModifiedOperationId
• RemovedRequiredParameter
• AddingRequiredParameter
• AddingResponseCode
• RemovedResponseCode
• AddingHeader
• RemovingHeader
• ParameterInHasChanged
• ConstantStatusHasChanged
• ReferenceRedirection
• RemovedEnumValues
• RemovedEnumValue
• AddedEnumValues
• AddedAdditionalProperties
• RemovedAdditionalProperties
• TypeFormatChanged
• ConstraintIsStronger
• RequiredStatusChange
• TypeChanged
• DefaultValueChanged
• ArrayCollectionFormatChanged
• ReadonlyPropertyChanged2
• DifferentDiscriminator
• DifferentExtends
• DifferentAllOf
• RemovedProperty1
• AddedRequiredProperty1
• RemovedOperation
• ConstraintChanged
• ConstraintIsWeaker
• NoVersionChange
• ResponseBodyFormatNowSupported

Install autorest as part of installation of the cli

Add solid test cases for each of the rules

Eliminate duplicate messages

Think about whether do we need to eliminate duplicate messages. In the current system given a graph, if one node is dirty then all the paths reaching to that node is reported as dirty.

Make sure the reporting is based on the old or new file not intermingled

Enable Tool into the CI Pipeline to get early feedbacks

Add tool into azure-rest-api-specs travis CI in "Allowed Failure" for the reviewers

How to support Azure Stack & profiles for the tool

Enahnce messaging formates

• Support for text & json format
• Entire output should be emitted as valid json object

Add .NET core diff tool in the Node based cli

Enable easy release system via Jenkins / Travis

Should be able to release from travis or jenkins

[Rule] Operation Id has been changed

Operation Id has been changed from one version of the swagger to the another is considered a breaking change in SDK as the generated code would have method with different name.

Add test cases around Node CLI entry point

[Rule] New required property added to request

If a new property is made required in the request body, clients will have no way to set this and the request will fail.

Add AutoRest spec resolver into CLI

Bring the tool to minimal condition

[Rule] URL format has changed

Resource parameter names change from /resourceType1/{resourceType1_name} to /resourceType1/{resourceType1_id}. This will impact code generation.

[Rule] Existing Property is removed

If a property called "foo" was present in v1 of the API need to be removed, it should be done in a newer api-version

Enable integration with the VS Code

• Scenario 1:
  Developer copy existing folder -> Paste to new api-version -> make commit -> then update that swager
  i.e braking change tool should work on previous commit v/s uncommitted changes

• Scenario 2:
  Developer updates existing swagger inside a folder ->
  i.e braking change tool should work on previous commit v/s uncommitted changes

• Tell me two commits and i'll work on it

Enable debug & verbose logging in .NET project

Usage of AutoRest in the pipeline should be optional

Enable flag to enable/disable usage of AutoRest for spec resolution

Telemetry on the usage

Think about how to get insights of the usage and telemetry data points