ITM TEPS still eating memory about grafana-ibm-apm

[rafal-szypulka]

Do you access Grafana via nginx or directly?
I noticed that on ITM 6 the HTTP DELETE method is rejected because of invalid Referer HTTP header (HTTP GET works). Check TEPS WAS SystemOut.log to make sure (path vary depending on platform, on Linux it is similar to: /opt/IBM/ITM/lx8266/iw/profiles/ITMProfile/logs/ITMServer/SystemOut.log), grep for Referer string. If you find something similar to:

00000014 rest          E com.ibm.tivoli.rest.RestRequestProcessor service ##### 
Rejecting request due to invalid Referer attribute

it means the ITM REST API rejects DELETE request because invalid Referer. The workaround (installation of nginx reverse proxy and clearing the Referer header) is described here: https://github.com/rafal-szypulka/grafana-ibm-apm#grafana-installation. Please let me know if it solved your problem and I will update my documentation (so far I saw this problem only for APM8).

[jripollv]

It looks like we have in ITM the same referer problem.
This is part of the SystemOut.log file before applying the firefox referer workaround:
[11/2/17 9:20:00:749 CET] 00000020 rest E com.ibm.tivoli.rest.RestRequestProcessor service ##### Rejecting request due to invalid Referer attribute:http://spmonreport:3000
[11/2/17 9:20:00:749 CET] 00000020 webapp E com.ibm.ws.webcontainer.webapp.WebApp logServletError SRVE0293E: [Servlet Error]-[RestServlet]: com.ibm.ws.webcontainer.webapp.WebAppErrorReport: SRVE0295E: Error reported: 500
at com.ibm.ws.webcontainer.webapp.WebAppDispatcherContext.sendError(WebAppDispatcherContext.java:626)
at com.ibm.ws.webcontainer.webapp.WebAppDispatcherContext.sendError(WebAppDispatcherContext.java:656)
at com.ibm.ws.webcontainer.srt.SRTServletResponse.sendError(SRTServletResponse.java:1236)
at com.ibm.ws.webcontainer.srt.SRTServletResponse.sendError(SRTServletResponse.java:1210)
at com.ibm.tivoli.rest.RestRequestProcessor.handleRequest(RestRequestProcessor.java:243)
at org.apache.wink.server.internal.servlet.RestServlet.service(RestServlet.java:124)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1230)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:775)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:457)
at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:79)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:928)
at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1025)
at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3763)
at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:304)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:975)
at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:195)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:453)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:515)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:306)
at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:83)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:166)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1691)
An this is after Firefox workaround:
[11/2/17 9:59:44:834 CET] 0000001f rest E com.ibm.tivoli.rest.RestRequestProcessor service ##### The referer field does not exist in the HTTP request. The lack of a referer field could be considered a CSRF attack when the request is performing a state change.null

Although the memory used by TEPS increases around 400MB after the metrics loading it is more stable now.
Now I've got another problem but I'm going to open another issue.

[rafal-szypulka]

IMO it looks fine, it is just a harmless warning. If you want to fix it, put "http://<teps_ip>:15200" in the Referer value instead of "" (empty Referer).