Coder Social home page Coder Social logo

tal-tech / conan Goto Github PK

View Code? Open in Web Editor NEW
757.0 757.0 164.0 2.08 MB

柯南平台开源版本,为用户提供流量回放全流程解决方案

License: MIT License

Java 57.80% JavaScript 6.04% Batchfile 0.07% HTML 0.24% Vue 27.84% SCSS 0.80% PLpgSQL 7.04% Shell 0.17%
conan-platform multi-rule-traffic traffic-collection traffic-playback

conan's People

Contributors

dazzlingn avatar dengkunnan avatar notjackson avatar saviorghost avatar wujunze avatar zhangyongxin5-tal avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

conan's Issues

录制异常,异常code:Unable to parse response body

2022-01-27 10:24:39.615 [流量回放线程-8] INFO  c.t.w.conan.agent.service.impl.RecordServiceImpl - scroll_id=FGluY2x1ZGVfY29udGV4dF91dWlkDnF1ZXJ5VGhlbkZldGNoAA==
2022-01-27 10:24:39.615 [流量回放线程-8] INFO  c.t.w.conan.agent.service.impl.RecordServiceImpl - 查询到总流量0条
2022-01-27 10:24:39.615 [流量回放线程-8] INFO  c.t.w.conan.agent.service.impl.RecordServiceImpl - 查询到总流量 length0
ElasticsearchStatusException[Unable to parse response body]; nested: ResponseException[method [DELETE], host [http://10.33.40.22:9200], URI [/_search/scroll], status line [HTTP/1.1 404 Not Found]
{"succeeded":true,"num_freed":0}];
        at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1872)
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1626)
        at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1583)
        at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1553)
        at org.elasticsearch.client.RestHighLevelClient.clearScroll(RestHighLevelClient.java:1225)
        at com.tal.wangxiao.conan.agent.service.impl.RecordServiceImpl.getFlowByEsClientAndSaveFLow(RecordServiceImpl.java:379)
        at com.tal.wangxiao.conan.agent.service.impl.RecordServiceImpl.record(RecordServiceImpl.java:262)
        at com.tal.wangxiao.conan.agent.service.impl.RecordServiceImpl.startRecord(RecordServiceImpl.java:208)
        at com.tal.wangxiao.conan.agent.service.impl.RecordServiceImpl$$FastClassBySpringCGLIB$$12c31d87.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
        at com.alibaba.druid.support.spring.stat.DruidStatInterceptor.invoke(DruidStatInterceptor.java:73)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)
        at com.tal.wangxiao.conan.agent.service.impl.RecordServiceImpl$$EnhancerBySpringCGLIB$$c9e46460.startRecord(<generated>)
        at com.tal.wangxiao.conan.agent.config.KafkaConsumerListener.record(KafkaConsumerListener.java:99)
        at sun.reflect.GeneratedMethodAccessor238.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:774)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)
        at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        Suppressed: ParsingException[Failed to parse object: expecting field with name [error] but found [succeeded]]
                at org.elasticsearch.common.xcontent.XContentParserUtils.ensureFieldName(XContentParserUtils.java:50)
                at org.elasticsearch.ElasticsearchException.failureFromXContent(ElasticsearchException.java:592)
                at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:179)
                at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1892)
                at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1869)
                ... 31 more
Caused by: org.elasticsearch.client.ResponseException: method [DELETE], host [http://10.33.40.22:9200], URI [/_search/scroll], status line [HTTP/1.1 404 Not Found]
{"succeeded":true,"num_freed":0}
        at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:272)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:246)
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613)
        ... 30 more
2022-01-27 10:24:39.621 [流量回放线程-8] ERROR c.t.w.conan.agent.config.KafkaConsumerListener - 录制异常,异常code:Unable to parse response body

ES 查询条件配置的 ”ES中_source内接口名称“ ,进行流量获取时过滤条件要求包含 http method 不大合理

特指此配置项:
image

不合理原因:
在录制时,com.tal.wangxiao.conan.agent.service.impl.RecordServiceImpl#getFlowByEsClientAndSaveFLow 和此参数有关的
过滤条件为 + 空格 + :

BoolQueryBuilder queryBuilder = QueryBuilders.boolQuery()
                .must(QueryBuilders.wildcardQuery(url + ".keyword", method + " " + requestQuery.getApi() + "*"))
                .must(shouldQuery);

但同样在录制逻辑中,实际入库完整 request 内容的 com.tal.wangxiao.conan.agent.service.impl.RecordServiceImpl#saveFlowInDb ,查询条件变为了纯 api path ,相关逻辑:

...
String apiKey = esConditionSetting.getApi();
...
String apiName = RegexUtils.getMsgByRegex(esFlowMap.get(apiKey) + "", esConditionSetting.getApiRegex());
...
List<Api> apiList = apiRepository.findByNameAndMethod(apiName, HttpMethodConstants.valueOf(method).getValue());

这样会导致如果没有配置 "接口正则表达式" ,且保证可以过滤掉 http method ,就会导致获取到的 apiName 为类似 GET /api/case/list 的值,导致无法匹配数据库中已定义的接口,进而出现录制是成功的(条数是对的),但实际没有任何数据入库到 bss_record_result 表(因为没有匹配到任何已定义的接口),没法回放

是否可以调整为,条数的查询条件里,去掉 http method 合并到 url 的做法,而是单独多开一个查询条件,专门查 http method ?本身 es 配置里也有要求配置单独对应 http method 的字段了,应该是可以适用的

录制不到数据

执行结果显示录制成功,域名 查询条件也配置了,但是实际录制条数为0
1L3XFGE%B @{2$O5@B%4SER
PM_Z)R%872CEM6T~LLYV0PJ
image

怎么录制回放带有 queryString 的 get 接口?

在接口管理里配置的接口名称:/api/case/list

在 域名管理中的 es 配置:

ES中_source内接口名称对应的原始值:GET /api/case/list?pageSize=10&pageNum=1&productLineId=1&caseType=0&channel=1&bizId=83b0bcf4

接口正则表达式:/.*(?=\?)

录制完毕后,回放时,提示如下错误:

2021-06-22 05:24:39 [INFO]  --开始执行回放, 回放ID:replay_id=12,task_execution_id=32,record_id=32\n2021-06-22 05:24:44 [INFO]   --回放url=http://127.0.0.1:8094/api/case/list,body=,response={code:10400,msg:Required Integer parameter 'channel' is not present,data:null}
2021-06-22 05:24:44 [INFO]   --回放url=http://127.0.0.1:8094/api/case/list,body=,response={code:10400,msg:Required Integer parameter 'channel' is not present,data:null}
...

错误原因为 url 里缺失了 queryString 部分。查看源码发现,回放时接口请求完整url拼接逻辑为:

com.tal.wangxiao.conan.agent.service.impl.ReplayServiceImpl#doReplay

...
Optional<Api> apiOptional = apiRepository.findById(apiId);
...
String domainName = domainOptional.get().getName();
            String method = EnumUtil.getByField(HttpMethodConstants.class, "getValue", String.valueOf(api.getMethod())).getLabel();
            String urlStr = "http://" + domainName + api.getName();
...
response = replayByHttp(urlStr, method, body, headerMap);
                log.info("回放url={},method={},body={},response={}", urlStr, method, body, response);

逻辑里面,url 用的是 api.getName() ,即数据库中的接口名称。这个名称是不会带有 queryString 的,所以导致回放时缺失这部分。

是我使用姿势不对吗?还是目前不支持带有 QueryString 的 Get 接口录制回放?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.