virtualansoftware / idaithalam Goto Github PK

Idaithalam has a problem with the folder structure in a Postman collection.
Although it does process the folder itself correctly, it stops to recognize then and only then test defined outside of the folder.

I made a test and could reproduce the issue.
All tests are correctly identified (like for e.g. 5 tests) withouf folders.
But after adding a folder and put 1 of 5 test into that folder (does not come up with an empty folder), it
recognizes only 1 test instead of 5.

Having a large number of fields (>50) it will be very helpful to see which field caused the failed test.
I don't see that in the Cucumber report.
Is it possible to see which field did not pass ?

Idaithalam throws an error if the API request (like defined in a Postman collection) has duplicate API header keys.
Duplicate or more API header keys are valid REST API requests.
The exception are thrown in Cucumblan.

Scenario: Get the test result links for the last test session.. - GET api call # target/testreports/1/feature/virtualan-contract.0.feature:5 Okt 24, 2021 8:46:24 AM io.virtualan.cucumblan.core.BaseStepDefinition before INFORMATION: scenario ID:: file:///media/amrit/f62417df-6c96-4797-a3d9-a06fb4352655/workspace/api/apitesting/idaithalam/modules/idaithalam/target/testreports/1/feature/virtualan-contract.0.feature:5 Given a user perform a api action # io.virtualan.cucumblan.core.BaseStepDefinition.readRequestByPathParam(java.lang.String) And add request with given header params # io.virtualan.cucumblan.core.BaseStepDefinition.readAllHeaderParams(java.lang.String,java.util.Map<java.lang.String, java.lang.String>) io.cucumber.core.exception.CucumberException: Could not convert arguments for step [add (.*) with given header params$] defined at 'io.virtualan.cucumblan.core.BaseStepDefinition.readAllHeaderParams(java.lang.String,java.util.Map<java.lang.String, java.lang.String>)'. at io.cucumber.core.runner.PickleStepDefinitionMatch.couldNotConvertArguments(PickleStepDefinitionMatch.java:112) at io.cucumber.core.runner.PickleStepDefinitionMatch.runStep(PickleStepDefinitionMatch.java:56) at io.cucumber.core.runner.ExecutionMode$1.execute(ExecutionMode.java:10) at io.cucumber.core.runner.TestStep.executeStep(TestStep.java:92) at io.cucumber.core.runner.TestStep.run(TestStep.java:64) at io.cucumber.core.runner.PickleStepTestStep.run(PickleStepTestStep.java:51) at io.cucumber.core.runner.TestCase.run(TestCase.java:104) at io.cucumber.core.runner.Runner.runPickle(Runner.java:73) at io.cucumber.core.runtime.Runtime.lambda$execute$5(Runtime.java:110) at io.cucumber.core.runtime.CucumberExecutionContext.runTestCase(CucumberExecutionContext.java:117) at io.cucumber.core.runtime.Runtime.lambda$execute$6(Runtime.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at io.cucumber.core.runtime.Runtime$SameThreadExecutorService.execute(Runtime.java:233) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at io.cucumber.core.runtime.Runtime.lambda$run$2(Runtime.java:86) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.stream.SliceOps$1$1.accept(SliceOps.java:204) at java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1361) at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126) at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) at io.cucumber.core.runtime.Runtime.run(Runtime.java:87) at io.cucumber.core.cli.Main.run(Main.java:92) at io.virtualan.idaithalam.contract.IdaithalamExecutor.validateContract(IdaithalamExecutor.java:120) at io.virtualan.idaithalam.core.api.ParallelExecutor.call(ParallelExecutor.java:49) at io.virtualan.idaithalam.core.api.ParallelExecutor.call(ParallelExecutor.java:15) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: io.cucumber.datatable.CucumberDataTableException: Can't convert DataTable to Map<java.lang.String, java.lang.String>. Encountered duplicate key X-API-TEST1 with values test1 and test2 at io.cucumber.datatable.CucumberDataTableException.cantConvertToMap(CucumberDataTableException.java:28) at io.cucumber.datatable.CucumberDataTableException.duplicateKeyException(CucumberDataTableException.java:33) at io.cucumber.datatable.DataTableTypeRegistryTableConverter.createMap(DataTableTypeRegistryTableConverter.java:299) at io.cucumber.datatable.DataTableTypeRegistryTableConverter.toMap(DataTableTypeRegistryTableConverter.java:287) at io.cucumber.datatable.DataTableTypeRegistryTableConverter.convert(DataTableTypeRegistryTableConverter.java:81) at io.cucumber.datatable.DataTable.convert(DataTable.java:360) at io.cucumber.core.stepexpression.StepExpressionFactory.lambda$createExpression$0(StepExpressionFactory.java:66) at io.cucumber.core.stepexpression.DataTableArgument.getValue(DataTableArgument.java:19) at io.cucumber.core.runner.PickleStepDefinitionMatch.runStep(PickleStepDefinitionMatch.java:47) ... 33 more

If the response object is not found, it throws an exception.

I found a bracket error.
In class ParallelExecutor the check fails if there is an Excel input file.
It always resolves to true and tries to generate from an Excel file although the field inputexcel is Null.
That is because there is a missing bracket in the if statement for the or condition.
There the and is bound to the second or condition.
The or condition must have brackets on their own to separate them from the and condition.
Here the correct brackets:

        if (((System.getenv("IDAITHALAM") == null
            || !"PROD".equalsIgnoreCase(System.getenv("IDAITHALAM")))
            && apiExecutorParam.getInputExcel() != null)) {
          ExcelToCollectionGenerator.createCollection(apiExecutorParam);
        } 

It could happen that the feature file is missing (like execution: EXECUTE).
There is currently no error message in that case.

It will be better to have a proper error message for that.

It would be convenient if you could define fields with regular expressions.
Like it is with excludeProperties, just instead to ignore the fields it is tested againg a regular expression (mail format, timestamp etc).

Currently idaithalam does not process variables defined in the collection.

I will make a PR for that.
Please check and let me know if that is suitable.

I am not sure for what reason but if there are query parameter,
a line is added in the feature file for header paramater as well.
"And add request with given header params"
But no line below that does not work (means there are no header parameter).

If the response contains text with Umlaute, like the german ä,ü,ö the check fails.

I have a requirement to integrate Allure Reports with current framework.

Path variables are not processed, and the url keeps the placeholder (starting with colon).
URL must be changed according to the path variables.

io.cucumber.core.exception.CucumberException: Step [^Verify-all (.*) api includes following in the response$] is defined with 2 parameters at 'io.virtualan.cucumblan.core.BaseStepDefinition.verifyResponseMapson(java.lang.String,io.cucumber.datatable.DataTable)'.
However, the gherkin step has 1 arguments:

I checked the feature file, and it has 2 arguments.
It is not clear where that exception is coming from.

Postman response result.

Usually you don't have a Spring context in the JUnit tests.
I would like to have placeholders which can be defined programmatically.

Like you define in the yaml file:
apiExecutor[0].outputDir: ${file.path}

And define the value for the Idaithalam executor like in a Hashmap:
VirtualanTestPlanExecutor.setValues(HashMap map);

That will help a lot if you have several configuration files containing same values.
I can make a PR for that if that is ok.

cucumber tags will have multiple comma separated values and escape the commas.

If an error occurs like file not found it does not come up in the console.

The header that can be defined in a Postman collection for Authorization for all requests are not processed.
Those are added like the API Key authorization to all headers of all API request.
Idaithalam is skipping them but should add them to all header of all requests.

It will be very helpful if you can disable the checking of a field value.
It will be true contract testing, if the structure is checked (field names are there), but not the values.

If you have umlaute in the text like the Scenario text it is only a non-presentable character.
It must be encoded for correct presentation in the Cucumber html report.

Clarification question:
Right now headerList is valid for all requests since it is made for authentication API keys.
Maybe it would be nice to have the option to overwrite in the configuration just some API header for defined endpoints ?

The expression to ignore fields is not yet 100% clear for "excludeProperties".
It seems on the left side (url of endpoint) you can use a wildcard with ".*" while on the right side
you cannot use that wildcard (instead it makes a ".contains()" check).

So why the dot for the left side ?
Tutorial should make that clear for the right side as well.

excludeProperties works only for the last in the configuration yaml file.

I have put two Postman collections inside the yaml file like with

• reportTitle: 'First test'

  excludeProperties:
  /url1/: fieldone

• reportTitle: 'Second test'

  excludeProperties:
  /url2/: fieldtwo

  Now excludeProperties from second test works fine.
  But excludeProperties from first test fails.

  However, first test works fine if I remove second test from file (or having in second file).
  If I change position, it works for first test (in second position) and fails for the second.

  Means last one wins.

Looks like being an overwrite issue

If you disable a parameter in a Postman collection, it is processed anyway by Idaithalam.
That can have unexpected result since you don't expect disabled parameters to be part of the tests.

You can see those parameter easily in a Postman collection, they have the field "disabled: true".

Idaithalam throws an exception if the
query has multiple times the same query parameter.
A query parameter like: ?tags=tag1,tags=tag2,tag=tag3
are 100% common and valid REST API queries.
But Idaithalam throws an error for that.

Postman has the behaviour to cut number ending with null.
It changes 12.50 to 12.5.
Since the original value is 12.50, Idaithalam test fails because the Postman collection stored 12.5 instead.

Idaithalam shall check if it is a number and compare number values instead of string compare.

Idaithalam defines fix the expected content-type as application/json.
Result is that you get a 406 from the server if the content type is anything else but application/json.

That must be taken from either Postman collection, left null (which is what Postman is usually doing, it does not put that into the request) or it must be configurable in the yaml configuration file.

My proposal is to set default to null, means it is not send in the request.
If to be defined in the yaml file, it has to be separated by each URL request.

With my last PR I resolved the variables for path parameters.
That has to be done for the query parameters as well.