spenceclark / newman-reporter-json-summary

A Newman JSON Reporter that strips the results down to a minimum

JSON Summary Newman Reporter

This generates a very cut down version of the standard JSON reporter output. It is needed where large test packs which generate 1000's of tests results can blow the limits of JSON.Stringify

This outputs:

• Collection.Info.Name
• Collection.Info.Id
• Run.Stats.Requests.*
• Run.Stats.Assertions.*
• Run.Timings.*
• Run.Failures[n].Parent.Name
• Run.Failures[n].Parent.Id
• Run.Failures[n].Source.Name
• Run.Failures[n].Source.Id
• Run.Failures[n].Error.Message
• Run.Failures[n].Error.Test

---

Usage:

newman run collection -r json-summary --reporter-summary-json-export output

Note: If reporter-json-summary-export parameter is not supplied, the output is written to a "newman" sub-folder and the file will contain the timestamp in its name

---

Schema of the output JSON: https://raw.githubusercontent.com/spenceclark/newman-reporter-json-summary/master/schema.json

---

Based on the https://www.npmjs.com/package/newman-reporter-json-light package, but slimmed down further.

newman-reporter-json-summary's Issues

Reporter fails to handle failures within pre-request scripts

Reporter fails to handle failures that occur within pre-request scripts.

The following error message is returned.

/home/john/forge/postman-collections/node_modules/newman-reporter-json-summary/index.js:27
                'Name': failure.parent.name,
                                       ^

TypeError: Cannot read property 'name' of undefined
    at /home/john/forge/postman-collections/node_modules/newman-reporter-json-summary/index.js:27:40
    at Array.forEach (<anonymous>)
    at createSummary (/home/john/forge/postman-collections/node_modules/newman-reporter-json-summary/index.js:23:26)
    at EventEmitter.<anonymous> (/home/john/forge/postman-collections/node_modules/newman-reporter-json-summary/index.js:70:37)
    at EventEmitter.emit (/home/john/forge/postman-collections/node_modules/eventemitter3/index.js:203:33)
    at done (/home/john/forge/postman-collections/node_modules/newman/lib/run/index.js:314:29)
    at /home/john/forge/postman-collections/node_modules/postman-runtime/lib/backpack/index.js:58:34
    at PostmanCollectionRun._process (/home/john/forge/postman-collections/node_modules/postman-runtime/lib/runner/run.js:163:13)
    at PostmanCollectionRun.<anonymous> (/home/john/forge/postman-collections/node_modules/postman-runtime/lib/runner/run.js:169:76)
    at listOnTimeout (node:internal/timers:556:17)
    at processTimers (node:internal/timers:499:7)

Taking a closer look at the failure object returned, you can see that when a failure occurs within a pre-request script the parent is left undefined.

FAILURE: {
  error: SerialisedError {
    errno: -3008,
    code: 'ENOTFOUND',
    syscall: 'getaddrinfo',
    hostname: 'auth-incorrect.test.com',
    name: 'Error',
    message: 'getaddrinfo ENOTFOUND auth-incorrect.test.com',
    stack: 'Error: getaddrinfo ENOTFOUND auth-incorrect.test.com\n' +
      '    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:69:26)',
    checksum: '31d5dc918783eb7f70b5f40b233e2629',
    id: 'removed',
    timestamp: 1633441749292,
    stacktrace: [ [Object] ]
  },
  at: 'request',
  source: PostmanItem {
    id: 'removed',
    request: PostmanRequest {
      url: [PostmanUrl],
      headers: [HeaderList],
      method: 'POST',
      body: [PostmanRequestBody]
    },
    responses: PostmanPropertyList {
      members: [],
      reference: {},
      Type: [Function]
    },
    events: PostmanEventList { members: [], reference: {}, Type: [Function] }
  },
  parent: undefined,
  cursor: {
    ref: '30d3f642-02b5-41db-a034-42fc3253e42a',
    length: 43,
    cycles: 1,
    position: 1,
    iteration: 0,
    scriptId: 'removed',
    execution: 'removed',
    httpRequestId: 'removed'
  }
}

Assuming that simply discounting these failures when building the summary object would be sufficient. Unless there is a particular need to include these failures within the final report summary?

Newman summary JSON having incorrect endpoints

If a request contains pm.sendRequest() in its test then, the summary object created post the execution of collection using newman.run yields following incorrect values:

1. Under summary.run.executions we would have 2 object with identical values . One corresponding to request and second corresponding to pm.sendRequest.
2. The url of parent request will also be incorrectly replaced with the url of pm.sendrequest() under summary.run.executions[index].request.url

reporter-json-summary-export is not creating output file

newman run collection -r json-summary,cli --reporter-json-summary-export output is not generating the output json file with 1.0.11 version. Its working on 1.0.10.

reporter is not found

Hello!

I've installed json-summary reporter together with newman but when trying to run a collection I'm getting this:

newman: could not find "json-summary" reporter
ensure that the reporter is installed in the same directory as newman
please install reporter using npm

I have:
Node.js v15.1.0 | npm -v 7.0.8 | newman -v 5.2.1 | 18.04.1-Ubuntu

npm list -g -depth=0
/usr/lib
+-- [email protected]
+-- [email protected]
`-- [email protected]

Have you ever encountered anything similar? If yes, what is the solution? I did install it using npm and with -g option, it is installed into the same directory as newman (node_modules).

I would like to use this reporter to pass newman test results to prometheus, I hope it will be suitable for this purpose.

Thank you!

--reporter-json-summary-export does not work together with standard JSON reporter

If I run this reporter by itself or together with another reporter (e.g. -r html,json-summary), the --reporter-json-summary-export flag is respected and the JSON file is generated where I request it.

However, if I run it with the standard json reporter (-r json,json-summary), both JSON files are generated correctly, but, the json-summary file ends up under the default newman folder, and custom file names are not respected.

.git folder in latest npm package 1.0.11

I installed newman-reporter-json-summary locally and then was unable to install newman-reporter-html in the same project as I got an npm error EISGIT
EISGIT
npm ERR! git ...\postman\testrunner\node_modules\newman-reporter-json-summary: Appears to be a git repo or submodule.
npm ERR! git ...\postman\testrunner\node_modules\newman-reporter-json-summary

I had to manually remove the .git folder from node_modules\newman-reporter-json-summary

Cant specify reporter options

Im using the json-summary reporter and I'm not able to specify the export path using the newman node module.

I have tried numerous variations of the variables "jsonSummary" and "summaryJsonExport" but with no luck.
How do I specify the export path using the newman module?

newman.run({
	collection: 'postman_collection.json',
	environment: 'postman_environment.json',
	reporters: [
		'cli', 
		'junit', 
		'htmlextra', 
		'json-summary'
	],
	reporter: {
		junit: {
			export: 'artifact/report.xml'
		},
		htmlextra: {
			export: 'artifact/report.html',
			logs: true
		},
		jsonSummary: {
			summaryJsonExport: 'artifact/report.json'
		}
	}
}, function (err) {
	if (err) { throw err; }

	console.log('done')
});

adding a CR/LF at the end of the json document

Hi,

I am using the reporter, and content-wise, it's good.

But I have one issue : the json file is supposed to be read by Filebeat to send to Logstash/ElasticSearch, and Filebeat won't send the line if there's no CR/LF at the end

So I know it's a bit of a stupid request, but could you please add one ?

thanks