wso2 / apim-test-integration

apim-test-integration

Automation scripts of the APIM business scenario implementation and the related artifacts are maintained in this repo.

List Of Scenarios

#	Scenario
1	API Development
	* Managing Public, Partner vs Private APIs
	* Ownership, permission and collaborative API development
	* Developer Optimized APIs Development"
2	API security
	* Protecting Businesses by Anomaly Detection
3	App Development with APIs
4	API Lifecycle Management
5	API Versioning
6	API Governance
7	API Rate Limiting
	* API Monetization
8	Manage the Value Chain for APIs
	* Business Insights by API analytics
9	Developer Enablement and Community Building
10	Support for API Types
11	Geographical Distributed wide API Management
12	API SLA
13	Micro API Gateway
14	Hybrid Cloud Pattern Support

apim-test-integration's Issues

No pre and post scripts written for scenario03

Description:
scenario03 is missing pre-scenario-steps.sh and post-scenario-steps.sh. Therefore, the host and port information are not replaced in user.properties file resulting in test failures.

Suggested Labels:
Type/Bug

Suggested Assignees:

Affected Product Version:
Any APIM version tests are run against

scenario04 - 04-pre-scenario-steps.sh does not fulfill the pre-requisites

Description:
scenario04 has some prerequisites related to starting MSF4J. All the prerequisites should be fulfilled through 04-pre-scenario-steps.sh and any should be destroyed in the 04-post-scenario-steps.sh.

Suggested Labels:
Type/Bug

Affected Product Version:
Any APIM version tests are run against

scenario-mapper.yaml should be renamed and modified as testgrid.yaml

Description:
scenario-mapper.yaml should be renamed to testgrid.yaml and should have the structure specified in https://github.com/wso2/testgrid/blob/master/docs/Infrastructure-Deployment-Scenarios-Repository-Structure.md#the-testgridyaml-for-scenario-test-configuration-v2

Suggested Labels:
Type/New Feature

The version parse logic can be removed

Description:

Currently in configure_product.py we have a string-split logic to get the version value. This is incorrect since the name can include multiple "-" s (which is the split character).
This can be changed as to maintain a new global variable to version and assign the exact value at the time we read the distribution pom file (in run-intg-test-runner.sh).
Suggested Labels:

Suggested Assignees:

Affected Product Version:

OS, DB, other environment details and versions:

Steps to reproduce:

Related Issues:

scenario05 - Prerequisites not fulfilled in 05-pre-scenario-steps.sh

Description:
According to the README, scenario05 contains some pre-requisites. These should be fulfilled in 05-pre-scenario-steps.sh and reverted in 05-post-scenario-steps.sh.

Suggested Labels:
Type/Bug

Affected Product Version:
Any APIM version tests are run against

scenario04 - The properties in user.properties does not match with the ones in the jmx file

Description:
Some of the properties does not have matching values in the jmx file.

Suggested Labels:
Type/Bug

Related Issues
#28

running integration test on ubuntu16.04 with mysql5.7 and oracle jdk

Creating Integration tests against Infra combinations: CentOS - Oracle JDK 8 - SQL Server

Creating run-scenario.bat to run integration tests in Windows

Tasks

• Include Maven Install
• Include JAVA_HOME Setting
• Include Git Install
• Clone Product Repo
• Do changes in master-datasources.xml, copy db jdbc jar, etc
• Zip and copy the modified pack in /target folder
• Run integration tests

Create and update UI test cases in QASE

Description:

• Create and update test cases in QASE
• Check on cypress integration with QASE to publish test results

Suggested Labels:

Suggested Assignees:

Affected Product Version:

OS, DB, other environment details and versions:

Steps to reproduce:

Related Issues:

Scenario03 - jmx script should be improved

Description:

The jmx contains hard-coded values which should be parameterized.

Suggested Labels:
Type/Improvement

Running test on infra combinations and analyzing issues

Creating Integration tests against Infra combinations: CentOS - Oracle JDK 8 - Oracle

Description:

Suggested Labels:

Suggested Assignees:

Affected Product Version:

OS, DB, other environment details and versions:

Steps to reproduce:

Related Issues:

scenario04 - addNotificationsSubscriptionsRest shows an error in the wso2carbon.log

Description:
This causes the ms4fjVerifyNotification to fail and the cause could be the remoteInstance URL in registry.xml being localhost and so the response of addNotificationsSubscriptionsRest is as follows.

<soapenv:Envelope
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
soapenv:Body
<ns:subscribeRESTResponse
xmlns:ns="http://services.info.registry.carbon.wso2.org">
<ns:return
xmlns:ax2773="http://api.registry.carbon.wso2.org/xsd"
xmlns:ax2776="http://beans.common.registry.carbon.wso2.org/xsd"
xmlns:ax2777="http://utils.beans.common.registry.carbon.wso2.org/xsd"
xmlns:ax2772="http://exceptions.core.registry.carbon.wso2.org/xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="ax2776:SubscriptionBean">
ax2776:errorMessageFailed to subscribe to information of the resource /_system/governance/apimgt/applicationdata/provider/admin/TetstAPIJmeter2/1.0.0/api. You cannot directly subscribe to a Remote Resource. Use the Registry Browser User Interface to add a Remote Subscription.</ax2776:errorMessage>
ax2776:loggedIntrue</ax2776:loggedIn>
<ax2776:pathWithVersion xsi:nil="true"/>
ax2776:roleAccessLevel0</ax2776:roleAccessLevel>
<ax2776:roles xsi:nil="true"/>
<ax2776:subscriptionInstances xsi:nil="true"/>
ax2776:userAccessLevel0</ax2776:userAccessLevel>
<ax2776:userName xsi:nil="true"/>
ax2776:versionViewfalse</ax2776:versionView>
</ns:return>
</ns:subscribeRESTResponse>
</soapenv:Body>
</soapenv:Envelope>

The wso2carbon.log contains the following exception:

TID: [-1234] [] [2018-04-29 19:48:55,295] ERROR {org.wso2.carbon.registry.info.services.utils.SubscriptionBeanPopulator} -  Failed to subscribe to information of the resource /_system/governance/apimgt/applicationdata/provider/admin/TetstAPIJmeter2/1.0.0/api. {org.wso2.carbon.registry.info.services.utils.SubscriptionBeanPopulator}
java.lang.UnsupportedOperationException: You cannot directly subscribe to a Remote Resource. Use the Registry Browser User Interface to add a Remote Subscription.
	at org.wso2.carbon.registry.info.services.utils.SubscriptionBeanPopulator.subscribeAndPopulate(SubscriptionBeanPopulator.java:526)
	at org.wso2.carbon.registry.info.services.InfoService.subscribeREST(InfoService.java:149)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
	at org.apache.axis2.rpc.receivers.RPCMessageReceiver.invokeBusinessLogic(RPCMessageReceiver.java:117)
	at org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40)
	at org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
	at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
	at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:173)
	at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:147)
	at org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:232)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
	at org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
	at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
	at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
	at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.owasp.csrfguard.CsrfGuardFilter.doFilter(CsrfGuardFilter.java:88)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:65)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:124)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
	at org.wso2.carbon.identity.context.rewrite.valve.TenantContextRewriteValve.invoke(TenantContextRewriteValve.java:80)
	at org.wso2.carbon.identity.authz.valve.AuthorizationValve.invoke(AuthorizationValve.java:91)
	at org.wso2.carbon.identity.auth.valve.AuthenticationValve.invoke(AuthenticationValve.java:60)
	at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:99)
	at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
	at org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:57)
	at org.wso2.carbon.event.receiver.core.internal.tenantmgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:48)
	at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
	at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62)
	at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:159)
	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
	at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1115)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1775)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1734)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)

Suggested Labels:
Type/Bug

Creating Integration tests for Infra combinations: CentOS - Open JDK 8 - MySQL

Description:

Suggested Labels:

Suggested Assignees:

Affected Product Version:

OS, DB, other environment details and versions:

Steps to reproduce:

Related Issues:

run-scenario.sh script should contain all the execution scripts itself

Description:
Currently run-scenario.sh the script only contains the jmx file, whereas it should contain all the execution scripts including pre, post etc.

Example:
Execute following files using run-scenario.sh scripts
01-pre-scenario.sh
01-<test_plan_name>.jmx
01-post-scenario.sh

scenario03 - "Invoking API via JAVA SDK embeded in MSF4J Service" test case fails

Description:
"Invoking API via JAVA SDK embeded in MSF4J Service" test fails with the following stacktrace in Phone Service Msf4J service

org.wso2.client.api.ApiException: java.io.IOException: unexpected end of stream on com.squareup.okhttp.Address@458151fe
	at org.wso2.client.api.ApiClient.execute(ApiClient.java:984)
	at org.wso2.client.api.ApiClient.execute(ApiClient.java:964)
	at org.wso2.client.api.AppDevelopmentAPI.DefaultApi.getMockGetWithHttpInfo(DefaultApi.java:128)
	at org.wso2.client.api.AppDevelopmentAPI.DefaultApi.getMockGet(DefaultApi.java:115)
	at org.phone.service.PhoneOrder.get(PhoneOrder.java:52)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.wso2.msf4j.internal.router.HttpMethodInfo.invokeResource(HttpMethodInfo.java:187)
	at org.wso2.msf4j.internal.router.HttpMethodInfo.invoke(HttpMethodInfo.java:143)
	at org.wso2.msf4j.internal.MSF4JHttpConnectorListener.dispatchMethod(MSF4JHttpConnectorListener.java:217)
	at org.wso2.msf4j.internal.MSF4JHttpConnectorListener.lambda$onMessage$57(MSF4JHttpConnectorListener.java:128)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: unexpected end of stream on com.squareup.okhttp.Address@458151fe
	at com.squareup.okhttp.internal.http.Http1xStream.readResponse(Http1xStream.java:201)
	at com.squareup.okhttp.internal.http.Http1xStream.readResponseHeaders(Http1xStream.java:127)
	at com.squareup.okhttp.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:737)
	at com.squareup.okhttp.internal.http.HttpEngine.access$200(HttpEngine.java:87)
	at com.squareup.okhttp.internal.http.HttpEngine$NetworkInterceptorChain.proceed(HttpEngine.java:722)
	at com.squareup.okhttp.internal.http.HttpEngine.readResponse(HttpEngine.java:576)
	at com.squareup.okhttp.Call.getResponse(Call.java:287)
	at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:243)
	at com.squareup.okhttp.Call.getResponseWithInterceptorChain(Call.java:205)
	at com.squareup.okhttp.Call.execute(Call.java:80)
	at org.wso2.client.api.ApiClient.execute(ApiClient.java:980)
	... 15 more
Caused by: java.io.EOFException: \n not found: size=7 content=15030300020250...
	at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:201)
	at com.squareup.okhttp.internal.http.Http1xStream.readResponse(Http1xStream.java:186)
	... 25 more
May 03, 2018 10:30:14 AM org.phone.service.PhoneOrder get
INFO: Encountered an error while invoking the api: org.wso2.client.api.ApiException: java.io.IOException: unexpected end of stream on com.squareup.okhttp.Address@458151fe

Suggested Labels:
Type/bug
APIM 2.2.0 (wum updated)

Creating Integration tests for Infra combinations: CentOS - Oracle JDK 8 - MySQL

Error: File SubscriptionValueSet.csv must exist and be readable

Description:

When running the APIM tests in testgrid, we see following exception in the log. We think this is some kind of bug in the jmeter scripts. Could you please check?

[09:53:24,241] Started thread group number 7
[09:53:24,241] Waiting for thread group: Subscribe to APIs to finish before starting next group
[09:53:24,249] Thread started: Subscribe to APIs 7-1
[09:53:24,250] Stored: ../resources/SubscriptionValueSet.csv
[09:53:24,250] Test failed! java.lang.IllegalArgumentException: File SubscriptionValueSet.csv must exist and be readable
	at org.apache.jmeter.services.FileServer.createBufferedReader(FileServer.java:430)
	at org.apache.jmeter.services.FileServer.readLine(FileServer.java:346)
	at org.apache.jmeter.config.CSVDataSet.iterationStart(CSVDataSet.java:203)
	at org.apache.jmeter.control.GenericController.fireIterationStart(GenericController.java:393)
	at org.apache.jmeter.control.GenericController.fireIterEvents(GenericController.java:385)
	at org.apache.jmeter.control.GenericController.next(GenericController.java:158)
	at org.apache.jmeter.control.LoopController.next(LoopController.java:123)
	at org.apache.jmeter.control.GenericController.nextIsAController(GenericController.java:219)
	at org.apache.jmeter.control.GenericController.next(GenericController.java:173)
	at org.apache.jmeter.control.LoopController.next(LoopController.java:123)
	at org.apache.jmeter.threads.AbstractThreadGroup.next(AbstractThreadGroup.java:87)
	at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:253)
	at java.lang.Thread.run(Thread.java:748)

�[m�[32m[09:53:24,251] Thread finished: Subscribe to APIs 7-1

Suggested Labels:
Type/Bug

Suggested Assignees:
Chaminda, Shashika

Affected Product Version:
??

OS, DB, other environment details and versions:
testgrid
https://testgrid-live.private.wso2.com/job/wso2apim-2.1.0-LTS/11/console

Steps to reproduce:

Related Issues:

Prepare APIM 4.2.0 regression scenarios document for QA

Description:
Prepare APIM 4.2.0 regression scenarios doc for QA

Affected Product Version:

• 4.2.0

OS, DB, other environment details and versions:

Steps to reproduce:

Related Issues:

Multiple automation.log files in product intg tests

Description:
As I've seen, there are multiple automation.log files under the tests-backend/target/logs folder of api-m. Seems there's a file-rolling based on file size (~10MB). We need to download all these files.

> ls tests-backend/target/logs
automation.log 
automation.log.1 
automation.log.2

As you see, the pattern seems to be automation.log\.?[0-9]*

OS, DB, other environment details and versions:
product-apim intg tests

Steps to reproduce:
just mvn clean install

Related Issues:

Creating run-scenario.sh to run integration tests in Centos

Purpose

• Include needed packs installation (mvn, git, etc..)
• Include JAVA_HOME Setting
• Clone Product Repo and bild it
• Do changes in master-datasources.xml, copy db jdbc jar, etc
• Zip and copy the modified pack
• Run integration tests

Modifying UI test cases with proper naming convention

Description:

• Modifying the test cases with proper naming convention in UI test suit.

Suggested Labels:

Suggested Assignees:

Affected Product Version:

OS, DB, other environment details and versions:

Steps to reproduce:

init.sh should be improved

Description:
init.sh should be improved in following ways.

• Connection to ports of MSF4J services should be checked
• EC2 instance being created should be given tags when launching to identify its purpose

Suggested Labels:
Improvement