tal-tech / conan Goto Github PK
View Code? Open in Web Editor NEW柯南平台开源版本,为用户提供流量回放全流程解决方案
License: MIT License
柯南平台开源版本,为用户提供流量回放全流程解决方案
License: MIT License
2022-01-27 10:24:39.615 [流量回放线程-8] INFO c.t.w.conan.agent.service.impl.RecordServiceImpl - scroll_id=FGluY2x1ZGVfY29udGV4dF91dWlkDnF1ZXJ5VGhlbkZldGNoAA==
2022-01-27 10:24:39.615 [流量回放线程-8] INFO c.t.w.conan.agent.service.impl.RecordServiceImpl - 查询到总流量0条
2022-01-27 10:24:39.615 [流量回放线程-8] INFO c.t.w.conan.agent.service.impl.RecordServiceImpl - 查询到总流量 length0
ElasticsearchStatusException[Unable to parse response body]; nested: ResponseException[method [DELETE], host [http://10.33.40.22:9200], URI [/_search/scroll], status line [HTTP/1.1 404 Not Found]
{"succeeded":true,"num_freed":0}];
at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1872)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1626)
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1583)
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1553)
at org.elasticsearch.client.RestHighLevelClient.clearScroll(RestHighLevelClient.java:1225)
at com.tal.wangxiao.conan.agent.service.impl.RecordServiceImpl.getFlowByEsClientAndSaveFLow(RecordServiceImpl.java:379)
at com.tal.wangxiao.conan.agent.service.impl.RecordServiceImpl.record(RecordServiceImpl.java:262)
at com.tal.wangxiao.conan.agent.service.impl.RecordServiceImpl.startRecord(RecordServiceImpl.java:208)
at com.tal.wangxiao.conan.agent.service.impl.RecordServiceImpl$$FastClassBySpringCGLIB$$12c31d87.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at com.alibaba.druid.support.spring.stat.DruidStatInterceptor.invoke(DruidStatInterceptor.java:73)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
at com.tal.wangxiao.conan.agent.service.impl.RecordServiceImpl$$EnhancerBySpringCGLIB$$c9e46460.startRecord(<generated>)
at com.tal.wangxiao.conan.agent.config.KafkaConsumerListener.record(KafkaConsumerListener.java:99)
at sun.reflect.GeneratedMethodAccessor238.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:774)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Suppressed: ParsingException[Failed to parse object: expecting field with name [error] but found [succeeded]]
at org.elasticsearch.common.xcontent.XContentParserUtils.ensureFieldName(XContentParserUtils.java:50)
at org.elasticsearch.ElasticsearchException.failureFromXContent(ElasticsearchException.java:592)
at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:179)
at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1892)
at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1869)
... 31 more
Caused by: org.elasticsearch.client.ResponseException: method [DELETE], host [http://10.33.40.22:9200], URI [/_search/scroll], status line [HTTP/1.1 404 Not Found]
{"succeeded":true,"num_freed":0}
at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:272)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:246)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613)
... 30 more
2022-01-27 10:24:39.621 [流量回放线程-8] ERROR c.t.w.conan.agent.config.KafkaConsumerListener - 录制异常,异常code:Unable to parse response body
rt
不合理原因:
在录制时,com.tal.wangxiao.conan.agent.service.impl.RecordServiceImpl#getFlowByEsClientAndSaveFLow
和此参数有关的
过滤条件为 + 空格 + :
BoolQueryBuilder queryBuilder = QueryBuilders.boolQuery()
.must(QueryBuilders.wildcardQuery(url + ".keyword", method + " " + requestQuery.getApi() + "*"))
.must(shouldQuery);
但同样在录制逻辑中,实际入库完整 request 内容的 com.tal.wangxiao.conan.agent.service.impl.RecordServiceImpl#saveFlowInDb
,查询条件变为了纯 api path ,相关逻辑:
...
String apiKey = esConditionSetting.getApi();
...
String apiName = RegexUtils.getMsgByRegex(esFlowMap.get(apiKey) + "", esConditionSetting.getApiRegex());
...
List<Api> apiList = apiRepository.findByNameAndMethod(apiName, HttpMethodConstants.valueOf(method).getValue());
这样会导致如果没有配置 "接口正则表达式" ,且保证可以过滤掉 http method ,就会导致获取到的 apiName 为类似 GET /api/case/list
的值,导致无法匹配数据库中已定义的接口,进而出现录制是成功的(条数是对的),但实际没有任何数据入库到 bss_record_result
表(因为没有匹配到任何已定义的接口),没法回放
是否可以调整为,条数的查询条件里,去掉 http method 合并到 url 的做法,而是单独多开一个查询条件,专门查 http method ?本身 es 配置里也有要求配置单独对应 http method 的字段了,应该是可以适用的
在接口管理里配置的接口名称:/api/case/list
在 域名管理中的 es 配置:
ES中_source内接口名称对应的原始值:GET /api/case/list?pageSize=10&pageNum=1&productLineId=1&caseType=0&channel=1&bizId=83b0bcf4
接口正则表达式:/.*(?=\?)
录制完毕后,回放时,提示如下错误:
2021-06-22 05:24:39 [INFO] --开始执行回放, 回放ID:replay_id=12,task_execution_id=32,record_id=32\n2021-06-22 05:24:44 [INFO] --回放url=http://127.0.0.1:8094/api/case/list,body=,response={code:10400,msg:Required Integer parameter 'channel' is not present,data:null}
2021-06-22 05:24:44 [INFO] --回放url=http://127.0.0.1:8094/api/case/list,body=,response={code:10400,msg:Required Integer parameter 'channel' is not present,data:null}
...
错误原因为 url 里缺失了 queryString 部分。查看源码发现,回放时接口请求完整url拼接逻辑为:
com.tal.wangxiao.conan.agent.service.impl.ReplayServiceImpl#doReplay
...
Optional<Api> apiOptional = apiRepository.findById(apiId);
...
String domainName = domainOptional.get().getName();
String method = EnumUtil.getByField(HttpMethodConstants.class, "getValue", String.valueOf(api.getMethod())).getLabel();
String urlStr = "http://" + domainName + api.getName();
...
response = replayByHttp(urlStr, method, body, headerMap);
log.info("回放url={},method={},body={},response={}", urlStr, method, body, response);
逻辑里面,url 用的是 api.getName() ,即数据库中的接口名称。这个名称是不会带有 queryString 的,所以导致回放时缺失这部分。
是我使用姿势不对吗?还是目前不支持带有 QueryString 的 Get 接口录制回放?
当前的Web系统的认证类型较多,当前的系统仅仅支持 采用 header里 使用 Cookie的类型的。
比如我的产品使用了AKSK认证。 是在header中增加了Authorization参数。且认证逻辑比较复杂。
[AKSK认证信息]参考 https://support.huaweicloud.com/devg-apisign/api-sign-provide-aksk.html
sql脚本里的表名和代码里的表名是不匹配的,比如脚本里的表是'diff_detail',代码里的表是'bss_diff_detail'
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.