ssssssss-team / spider-flow

新一代爬虫平台，以图形化方式定义爬虫流程，不写代码即可完成爬虫。

介绍 | 特性 | 插件 | DEMO站点 | 文档 | 更新日志 | 截图 | 其它开源 | 免责声明

介绍

平台以流程图的方式定义爬虫,是一个高度灵活可配置的爬虫平台

特性

• 支持Xpath/JsonPath/css选择器/正则提取/混搭提取
• 支持JSON/XML/二进制格式
• 支持多数据源、SQL select/selectInt/selectOne/insert/update/delete
• 支持爬取JS动态渲染(或ajax)的页面
• 支持代理
• 支持自动保存至数据库/文件
• 常用字符串、日期、文件、加解密等函数
• 支持插件扩展(自定义执行器，自定义方法）
• 任务监控,任务日志
• 支持HTTP接口
• 支持Cookie自动管理
• 支持自定义函数

插件

• Selenium插件
• Redis插件
• OSS插件
• Mongodb插件
• IP代理池插件
• OCR识别插件
• 电子邮箱插件

项目部分截图

爬虫列表

爬虫测试

Debug

日志

其它开源项目

• spider-flow-vue，spider-flow的前端
• magic-api，一个以XML为基础自动映射为HTTP接口的框架
• magic-api-spring-boot-starter

免责声明

请勿将spider-flow应用到任何可能会违反法律规定和道德约束的工作中，请友善使用spider-flow，遵守蜘蛛协议，不要将spider-flow用于任何非法用途。如您选择使用spider-flow即代表您遵守此协议，作者不承担任何由于您违反此协议带来任何的法律风险和损失，一切后果由您承担。

spider-flow's Issues

站点www.spiderflow.org 403了

站点www.spiderflow.org 403了

403 Forbidden

大佬,这个框架非常好用啊,简单易懂,看了更新日志有0.5.1,还有在维护吗

大佬,这个框架非常好用啊,简单易懂,看了更新日志有0.5.1,还有在维护吗

当选择每隔几个小时执行一次时，会变成0-N小时，每隔一个小时执行一次

Language file

Hi ! Would like to know if i can translate Spider Flow in english ? Looks like really cool tool but it's a bit confusing for me ^^'

Thanks !

Dependency org.yaml:snakeyaml, leading to CVE problem

Hi, In /，there is a dependency org.yaml:snakeyaml:1.19 that calls the risk method.

CVE-2022-25857

The scope of this CVE affected version is ** [0,1.31)**

After further analysis, in this project, the main Api called is org.yaml.snakeyaml.composer.Composer: composeNode(org.yaml.snakeyaml.nodes.Node)Lorg.yaml.snakeyaml.nodes.Node;

Risk method repair link : GitHub

CVE Bug Invocation Path--

Path Length : 6

CVE Bug Invocation Path : 
org.spiderflow.model.SpiderLog: init(java.lang.String,java.lang.String,java.util.List)V /.m2/repository/org/springframework/boot/spring-boot-starter-quartz/2.0.7.RELEASE/spring-boot-starter-quartz-2.0.7.RELEASE.jar
org.yaml.snakeyaml.Yaml$1: next()Ljava.lang.Object; /.m2/repository/org/springframework/boot/spring-boot-starter-quartz/2.0.7.RELEASE/spring-boot-starter-quartz-2.0.7.RELEASE.jar
org.yaml.snakeyaml.constructor.BaseConstructor: getData()Ljava.lang.Object; /.m2/repository/org/springframework/boot/spring-boot-starter-quartz/2.0.7.RELEASE/spring-boot-starter-quartz-2.0.7.RELEASE.jar
org.yaml.snakeyaml.composer.Composer: getNode()Lorg.yaml.snakeyaml.nodes.Node; /.m2/repository/org/springframework/boot/spring-boot-starter-quartz/2.0.7.RELEASE/spring-boot-starter-quartz-2.0.7.RELEASE.jar
org.yaml.snakeyaml.composer.Composer: composeDocument()Lorg.yaml.snakeyaml.nodes.Node; /.m2/repository/org/springframework/boot/spring-boot-starter-quartz/2.0.7.RELEASE/spring-boot-starter-quartz-2.0.7.RELEASE.jar
org.yaml.snakeyaml.composer.Composer: composeNode(org.yaml.snakeyaml.nodes.Node)Lorg.yaml.snakeyaml.nodes.Node;

Dependency tree--

INFO] org.spiderflow:spider-flow:pom:0.5.0
[INFO] +- org.springframework.boot:spring-boot-starter-web:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- org.springframework.boot:spring-boot:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- org.springframework.boot:spring-boot-starter-logging:jar:2.0.7.RELEASE:compile
[INFO] |  |  |  +- ch.qos.logback:logback-classic:jar:1.2.3:compile
[INFO] |  |  |  |  \- ch.qos.logback:logback-core:jar:1.2.3:compile
[INFO] |  |  |  +- org.apache.logging.log4j:log4j-to-slf4j:jar:2.10.0:compile
[INFO] |  |  |  |  \- org.apache.logging.log4j:log4j-api:jar:2.10.0:compile
[INFO] |  |  |  \- org.slf4j:jul-to-slf4j:jar:1.7.25:compile
[INFO] |  |  +- javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO] |  |  \- org.yaml:snakeyaml:jar:1.19:runtime
[INFO] |  +- org.springframework.boot:spring-boot-starter-json:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.9.7:compile
[INFO] |  |  |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.9.0:compile
[INFO] |  |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.9.7:compile
[INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.9.7:compile
[INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.9.7:compile
[INFO] |  |  \- com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.9.7:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter-tomcat:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- org.apache.tomcat.embed:tomcat-embed-core:jar:8.5.35:compile
[INFO] |  |  +- org.apache.tomcat.embed:tomcat-embed-el:jar:8.5.35:compile
[INFO] |  |  \- org.apache.tomcat.embed:tomcat-embed-websocket:jar:8.5.35:compile
[INFO] |  +- org.hibernate.validator:hibernate-validator:jar:6.0.13.Final:compile
[INFO] |  |  +- javax.validation:validation-api:jar:2.0.1.Final:compile
[INFO] |  |  +- org.jboss.logging:jboss-logging:jar:3.3.2.Final:compile
[INFO] |  |  \- com.fasterxml:classmate:jar:1.3.4:compile
[INFO] |  +- org.springframework:spring-web:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.springframework:spring-webmvc:jar:5.0.11.RELEASE:compile
[INFO] |     +- org.springframework:spring-aop:jar:5.0.11.RELEASE:compile
[INFO] |     \- org.springframework:spring-expression:jar:5.0.11.RELEASE:compile
[INFO] +- org.springframework.boot:spring-boot-starter-quartz:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework:spring-context-support:jar:5.0.11.RELEASE:compile
[INFO] |  +- org.springframework:spring-tx:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.quartz-scheduler:quartz:jar:2.3.0:compile
[INFO] |     \- com.mchange:mchange-commons-java:jar:0.2.11:compile
[INFO] +- org.springframework.boot:spring-boot-starter-mail:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework:spring-context:jar:5.0.11.RELEASE:compile
[INFO] |  \- com.sun.mail:javax.mail:jar:1.6.2:compile
[INFO] |     \- javax.activation:activation:jar:1.1:compile
[INFO] +- org.springframework:spring-jdbc:jar:5.0.11.RELEASE:compile
[INFO] |  +- org.springframework:spring-beans:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.springframework:spring-core:jar:5.0.11.RELEASE:compile
[INFO] |     \- org.springframework:spring-jcl:jar:5.0.11.RELEASE:compile
[INFO] +- org.springframework.boot:spring-boot-starter-websocket:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework:spring-messaging:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.springframework:spring-websocket:jar:5.0.11.RELEASE:compile
[INFO] +- com.baomidou:mybatis-plus-boot-starter:jar:3.1.0:compile
[INFO] |  +- com.baomidou:mybatis-plus:jar:3.1.0:compile
[INFO] |  |  \- com.baomidou:mybatis-plus-extension:jar:3.1.0:compile
[INFO] |  |     +- com.baomidou:mybatis-plus-core:jar:3.1.0:compile
[INFO] |  |     |  +- com.baomidou:mybatis-plus-annotation:jar:3.1.0:compile
[INFO] |  |     |  +- org.mybatis:mybatis:jar:3.5.0:compile
[INFO] |  |     |  \- com.github.jsqlparser:jsqlparser:jar:1.4:compile
[INFO] |  |     \- org.mybatis:mybatis-spring:jar:2.0.0:compile
[INFO] |  +- org.springframework.boot:spring-boot-autoconfigure:jar:2.0.7.RELEASE:compile
[INFO] |  \- org.springframework.boot:spring-boot-starter-jdbc:jar:2.0.7.RELEASE:compile
[INFO] |     \- com.zaxxer:HikariCP:jar:2.7.9:compile
[INFO] +- mysql:mysql-connector-java:jar:5.1.47:compile
[INFO] +- com.alibaba:fastjson:jar:1.2.83:compile
[INFO] +- com.alibaba:druid-spring-boot-starter:jar:1.1.16:compile
[INFO] |  +- com.alibaba:druid:jar:1.1.16:compile
[INFO] |  \- org.slf4j:slf4j-api:jar:1.7.25:compile
[INFO] +- com.alibaba:transmittable-thread-local:jar:2.11.5:compile
[INFO] +- org.apache.commons:commons-text:jar:1.6:compile
[INFO] |  \- org.apache.commons:commons-lang3:jar:3.7:compile
[INFO] +- org.apache.commons:commons-csv:jar:1.8:compile
[INFO] +- commons-io:commons-io:jar:2.7:compile
[INFO] +- commons-codec:commons-codec:jar:1.11:compile
[INFO] +- com.google.guava:guava:jar:28.2-jre:compile
[INFO] |  +- com.google.guava:failureaccess:jar:1.0.1:compile
[INFO] |  +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] |  +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] |  +- org.checkerframework:checker-qual:jar:2.10.0:compile
[INFO] |  +- com.google.errorprone:error_prone_annotations:jar:2.3.4:compile
[INFO] |  \- com.google.j2objc:j2objc-annotations:jar:1.3:compile
[INFO] +- org.jsoup:jsoup:jar:1.11.3:compile
[INFO] \- us.codecraft:xsoup:jar:0.3.1:compile
[INFO]    \- org.assertj:assertj-core:jar:3.9.1:compile
[INFO] 
[INFO] -------------------< org.spiderflow:spider-flow-api >-------------------
[INFO] Building spider-flow-api 0.5.0                                     [2/4]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-dependency-plugin:3.0.2:tree (default-cli) @ spider-flow-api ---
[INFO] org.spiderflow:spider-flow-api:jar:0.5.0
[INFO] +- org.springframework.boot:spring-boot-starter-web:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- org.springframework.boot:spring-boot:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- org.springframework.boot:spring-boot-starter-logging:jar:2.0.7.RELEASE:compile
[INFO] |  |  |  +- ch.qos.logback:logback-classic:jar:1.2.3:compile
[INFO] |  |  |  |  \- ch.qos.logback:logback-core:jar:1.2.3:compile
[INFO] |  |  |  +- org.apache.logging.log4j:log4j-to-slf4j:jar:2.10.0:compile
[INFO] |  |  |  |  \- org.apache.logging.log4j:log4j-api:jar:2.10.0:compile
[INFO] |  |  |  \- org.slf4j:jul-to-slf4j:jar:1.7.25:compile
[INFO] |  |  +- javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO] |  |  \- org.yaml:snakeyaml:jar:1.19:runtime
[INFO] |  +- org.springframework.boot:spring-boot-starter-json:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.9.7:compile
[INFO] |  |  |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.9.0:compile
[INFO] |  |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.9.7:compile
[INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.9.7:compile
[INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.9.7:compile
[INFO] |  |  \- com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.9.7:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter-tomcat:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- org.apache.tomcat.embed:tomcat-embed-core:jar:8.5.35:compile
[INFO] |  |  +- org.apache.tomcat.embed:tomcat-embed-el:jar:8.5.35:compile
[INFO] |  |  \- org.apache.tomcat.embed:tomcat-embed-websocket:jar:8.5.35:compile
[INFO] |  +- org.hibernate.validator:hibernate-validator:jar:6.0.13.Final:compile
[INFO] |  |  +- javax.validation:validation-api:jar:2.0.1.Final:compile
[INFO] |  |  +- org.jboss.logging:jboss-logging:jar:3.3.2.Final:compile
[INFO] |  |  \- com.fasterxml:classmate:jar:1.3.4:compile
[INFO] |  +- org.springframework:spring-web:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.springframework:spring-webmvc:jar:5.0.11.RELEASE:compile
[INFO] |     +- org.springframework:spring-aop:jar:5.0.11.RELEASE:compile
[INFO] |     \- org.springframework:spring-expression:jar:5.0.11.RELEASE:compile
[INFO] +- org.springframework.boot:spring-boot-starter-quartz:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework:spring-context-support:jar:5.0.11.RELEASE:compile
[INFO] |  +- org.springframework:spring-tx:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.quartz-scheduler:quartz:jar:2.3.0:compile
[INFO] |     \- com.mchange:mchange-commons-java:jar:0.2.11:compile
[INFO] +- org.springframework.boot:spring-boot-starter-mail:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework:spring-context:jar:5.0.11.RELEASE:compile
[INFO] |  \- com.sun.mail:javax.mail:jar:1.6.2:compile
[INFO] |     \- javax.activation:activation:jar:1.1:compile
[INFO] +- org.springframework:spring-jdbc:jar:5.0.11.RELEASE:compile
[INFO] |  +- org.springframework:spring-beans:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.springframework:spring-core:jar:5.0.11.RELEASE:compile
[INFO] |     \- org.springframework:spring-jcl:jar:5.0.11.RELEASE:compile
[INFO] +- org.springframework.boot:spring-boot-starter-websocket:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework:spring-messaging:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.springframework:spring-websocket:jar:5.0.11.RELEASE:compile
[INFO] +- com.baomidou:mybatis-plus-boot-starter:jar:3.1.0:compile
[INFO] |  +- com.baomidou:mybatis-plus:jar:3.1.0:compile
[INFO] |  |  \- com.baomidou:mybatis-plus-extension:jar:3.1.0:compile
[INFO] |  |     +- com.baomidou:mybatis-plus-core:jar:3.1.0:compile
[INFO] |  |     |  +- com.baomidou:mybatis-plus-annotation:jar:3.1.0:compile
[INFO] |  |     |  +- org.mybatis:mybatis:jar:3.5.0:compile
[INFO] |  |     |  \- com.github.jsqlparser:jsqlparser:jar:1.4:compile
[INFO] |  |     \- org.mybatis:mybatis-spring:jar:2.0.0:compile
[INFO] |  +- org.springframework.boot:spring-boot-autoconfigure:jar:2.0.7.RELEASE:compile
[INFO] |  \- org.springframework.boot:spring-boot-starter-jdbc:jar:2.0.7.RELEASE:compile
[INFO] |     \- com.zaxxer:HikariCP:jar:2.7.9:compile
[INFO] +- mysql:mysql-connector-java:jar:5.1.47:compile
[INFO] +- com.alibaba:fastjson:jar:1.2.83:compile
[INFO] +- com.alibaba:druid-spring-boot-starter:jar:1.1.16:compile
[INFO] |  +- com.alibaba:druid:jar:1.1.16:compile
[INFO] |  \- org.slf4j:slf4j-api:jar:1.7.25:compile
[INFO] +- com.alibaba:transmittable-thread-local:jar:2.11.5:compile
[INFO] +- org.apache.commons:commons-text:jar:1.6:compile
[INFO] |  \- org.apache.commons:commons-lang3:jar:3.7:compile
[INFO] +- org.apache.commons:commons-csv:jar:1.8:compile
[INFO] +- commons-io:commons-io:jar:2.7:compile
[INFO] +- commons-codec:commons-codec:jar:1.11:compile
[INFO] +- com.google.guava:guava:jar:28.2-jre:compile
[INFO] |  +- com.google.guava:failureaccess:jar:1.0.1:compile
[INFO] |  +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] |  +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] |  +- org.checkerframework:checker-qual:jar:2.10.0:compile
[INFO] |  +- com.google.errorprone:error_prone_annotations:jar:2.3.4:compile
[INFO] |  \- com.google.j2objc:j2objc-annotations:jar:1.3:compile
[INFO] +- org.jsoup:jsoup:jar:1.11.3:compile
[INFO] \- us.codecraft:xsoup:jar:0.3.1:compile
[INFO]    \- org.assertj:assertj-core:jar:3.9.1:compile
[INFO] 
[INFO] ------------------< org.spiderflow:spider-flow-core >-------------------
[INFO] Building spider-flow-core 0.5.0                                    [3/4]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-dependency-plugin:3.0.2:tree (default-cli) @ spider-flow-core ---
[INFO] org.spiderflow:spider-flow-core:jar:0.5.0
[INFO] +- org.spiderflow:spider-flow-api:jar:0.5.0:compile
[INFO] +- org.springframework.boot:spring-boot-starter-web:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- org.springframework.boot:spring-boot:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- org.springframework.boot:spring-boot-starter-logging:jar:2.0.7.RELEASE:compile
[INFO] |  |  |  +- ch.qos.logback:logback-classic:jar:1.2.3:compile
[INFO] |  |  |  |  \- ch.qos.logback:logback-core:jar:1.2.3:compile
[INFO] |  |  |  +- org.apache.logging.log4j:log4j-to-slf4j:jar:2.10.0:compile
[INFO] |  |  |  |  \- org.apache.logging.log4j:log4j-api:jar:2.10.0:compile
[INFO] |  |  |  \- org.slf4j:jul-to-slf4j:jar:1.7.25:compile
[INFO] |  |  +- javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO] |  |  \- org.yaml:snakeyaml:jar:1.19:runtime
[INFO] |  +- org.springframework.boot:spring-boot-starter-json:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.9.7:compile
[INFO] |  |  |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.9.0:compile
[INFO] |  |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.9.7:compile
[INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.9.7:compile
[INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.9.7:compile
[INFO] |  |  \- com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.9.7:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter-tomcat:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- org.apache.tomcat.embed:tomcat-embed-core:jar:8.5.35:compile
[INFO] |  |  +- org.apache.tomcat.embed:tomcat-embed-el:jar:8.5.35:compile
[INFO] |  |  \- org.apache.tomcat.embed:tomcat-embed-websocket:jar:8.5.35:compile
[INFO] |  +- org.hibernate.validator:hibernate-validator:jar:6.0.13.Final:compile
[INFO] |  |  +- javax.validation:validation-api:jar:2.0.1.Final:compile
[INFO] |  |  +- org.jboss.logging:jboss-logging:jar:3.3.2.Final:compile
[INFO] |  |  \- com.fasterxml:classmate:jar:1.3.4:compile
[INFO] |  +- org.springframework:spring-web:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.springframework:spring-webmvc:jar:5.0.11.RELEASE:compile
[INFO] |     +- org.springframework:spring-aop:jar:5.0.11.RELEASE:compile
[INFO] |     \- org.springframework:spring-expression:jar:5.0.11.RELEASE:compile
[INFO] +- org.springframework.boot:spring-boot-starter-quartz:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework:spring-context-support:jar:5.0.11.RELEASE:compile
[INFO] |  +- org.springframework:spring-tx:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.quartz-scheduler:quartz:jar:2.3.0:compile
[INFO] |     \- com.mchange:mchange-commons-java:jar:0.2.11:compile
[INFO] +- org.springframework.boot:spring-boot-starter-mail:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework:spring-context:jar:5.0.11.RELEASE:compile
[INFO] |  \- com.sun.mail:javax.mail:jar:1.6.2:compile
[INFO] |     \- javax.activation:activation:jar:1.1:compile
[INFO] +- org.springframework:spring-jdbc:jar:5.0.11.RELEASE:compile
[INFO] |  +- org.springframework:spring-beans:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.springframework:spring-core:jar:5.0.11.RELEASE:compile
[INFO] |     \- org.springframework:spring-jcl:jar:5.0.11.RELEASE:compile
[INFO] +- org.springframework.boot:spring-boot-starter-websocket:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework:spring-messaging:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.springframework:spring-websocket:jar:5.0.11.RELEASE:compile
[INFO] +- com.baomidou:mybatis-plus-boot-starter:jar:3.1.0:compile
[INFO] |  +- com.baomidou:mybatis-plus:jar:3.1.0:compile
[INFO] |  |  \- com.baomidou:mybatis-plus-extension:jar:3.1.0:compile
[INFO] |  |     +- com.baomidou:mybatis-plus-core:jar:3.1.0:compile
[INFO] |  |     |  +- com.baomidou:mybatis-plus-annotation:jar:3.1.0:compile
[INFO] |  |     |  +- org.mybatis:mybatis:jar:3.5.0:compile
[INFO] |  |     |  \- com.github.jsqlparser:jsqlparser:jar:1.4:compile
[INFO] |  |     \- org.mybatis:mybatis-spring:jar:2.0.0:compile
[INFO] |  +- org.springframework.boot:spring-boot-autoconfigure:jar:2.0.7.RELEASE:compile
[INFO] |  \- org.springframework.boot:spring-boot-starter-jdbc:jar:2.0.7.RELEASE:compile
[INFO] |     \- com.zaxxer:HikariCP:jar:2.7.9:compile
[INFO] +- mysql:mysql-connector-java:jar:5.1.47:compile
[INFO] +- com.alibaba:fastjson:jar:1.2.83:compile
[INFO] +- com.alibaba:druid-spring-boot-starter:jar:1.1.16:compile
[INFO] |  +- com.alibaba:druid:jar:1.1.16:compile
[INFO] |  \- org.slf4j:slf4j-api:jar:1.7.25:compile
[INFO] +- com.alibaba:transmittable-thread-local:jar:2.11.5:compile
[INFO] +- org.apache.commons:commons-text:jar:1.6:compile
[INFO] |  \- org.apache.commons:commons-lang3:jar:3.7:compile
[INFO] +- org.apache.commons:commons-csv:jar:1.8:compile
[INFO] +- commons-io:commons-io:jar:2.7:compile
[INFO] +- commons-codec:commons-codec:jar:1.11:compile
[INFO] +- com.google.guava:guava:jar:28.2-jre:compile
[INFO] |  +- com.google.guava:failureaccess:jar:1.0.1:compile
[INFO] |  +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] |  +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] |  +- org.checkerframework:checker-qual:jar:2.10.0:compile
[INFO] |  +- com.google.errorprone:error_prone_annotations:jar:2.3.4:compile
[INFO] |  \- com.google.j2objc:j2objc-annotations:jar:1.3:compile
[INFO] +- org.jsoup:jsoup:jar:1.11.3:compile
[INFO] \- us.codecraft:xsoup:jar:0.3.1:compile
[INFO]    \- org.assertj:assertj-core:jar:3.9.1:compile
[INFO] 
[INFO] -------------------< org.spiderflow:spider-flow-web >-------------------
[INFO] Building spider-flow-web 0.5.0                                     [4/4]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-dependency-plugin:3.0.2:tree (default-cli) @ spider-flow-web ---
[INFO] org.spiderflow:spider-flow-web:jar:0.5.0
[INFO] +- org.spiderflow:spider-flow-core:jar:0.5.0:compile
[INFO] |  \- org.spiderflow:spider-flow-api:jar:0.5.0:compile
[INFO] +- org.springframework.boot:spring-boot-starter-web:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- org.springframework.boot:spring-boot:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- org.springframework.boot:spring-boot-starter-logging:jar:2.0.7.RELEASE:compile
[INFO] |  |  |  +- ch.qos.logback:logback-classic:jar:1.2.3:compile
[INFO] |  |  |  |  \- ch.qos.logback:logback-core:jar:1.2.3:compile
[INFO] |  |  |  +- org.apache.logging.log4j:log4j-to-slf4j:jar:2.10.0:compile
[INFO] |  |  |  |  \- org.apache.logging.log4j:log4j-api:jar:2.10.0:compile
[INFO] |  |  |  \- org.slf4j:jul-to-slf4j:jar:1.7.25:compile
[INFO] |  |  +- javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO] |  |  \- org.yaml:snakeyaml:jar:1.19:runtime
[INFO] |  +- org.springframework.boot:spring-boot-starter-json:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.9.7:compile
[INFO] |  |  |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.9.0:compile
[INFO] |  |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.9.7:compile
[INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.9.7:compile
[INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.9.7:compile
[INFO] |  |  \- com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.9.7:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter-tomcat:jar:2.0.7.RELEASE:compile
[INFO] |  |  +- org.apache.tomcat.embed:tomcat-embed-core:jar:8.5.35:compile
[INFO] |  |  +- org.apache.tomcat.embed:tomcat-embed-el:jar:8.5.35:compile
[INFO] |  |  \- org.apache.tomcat.embed:tomcat-embed-websocket:jar:8.5.35:compile
[INFO] |  +- org.hibernate.validator:hibernate-validator:jar:6.0.13.Final:compile
[INFO] |  |  +- javax.validation:validation-api:jar:2.0.1.Final:compile
[INFO] |  |  +- org.jboss.logging:jboss-logging:jar:3.3.2.Final:compile
[INFO] |  |  \- com.fasterxml:classmate:jar:1.3.4:compile
[INFO] |  +- org.springframework:spring-web:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.springframework:spring-webmvc:jar:5.0.11.RELEASE:compile
[INFO] |     +- org.springframework:spring-aop:jar:5.0.11.RELEASE:compile
[INFO] |     \- org.springframework:spring-expression:jar:5.0.11.RELEASE:compile
[INFO] +- org.springframework.boot:spring-boot-starter-quartz:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework:spring-context-support:jar:5.0.11.RELEASE:compile
[INFO] |  +- org.springframework:spring-tx:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.quartz-scheduler:quartz:jar:2.3.0:compile
[INFO] |     \- com.mchange:mchange-commons-java:jar:0.2.11:compile
[INFO] +- org.springframework.boot:spring-boot-starter-mail:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework:spring-context:jar:5.0.11.RELEASE:compile
[INFO] |  \- com.sun.mail:javax.mail:jar:1.6.2:compile
[INFO] |     \- javax.activation:activation:jar:1.1:compile
[INFO] +- org.springframework:spring-jdbc:jar:5.0.11.RELEASE:compile
[INFO] |  +- org.springframework:spring-beans:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.springframework:spring-core:jar:5.0.11.RELEASE:compile
[INFO] |     \- org.springframework:spring-jcl:jar:5.0.11.RELEASE:compile
[INFO] +- org.springframework.boot:spring-boot-starter-websocket:jar:2.0.7.RELEASE:compile
[INFO] |  +- org.springframework:spring-messaging:jar:5.0.11.RELEASE:compile
[INFO] |  \- org.springframework:spring-websocket:jar:5.0.11.RELEASE:compile
[INFO] +- com.baomidou:mybatis-plus-boot-starter:jar:3.1.0:compile
[INFO] |  +- com.baomidou:mybatis-plus:jar:3.1.0:compile
[INFO] |  |  \- com.baomidou:mybatis-plus-extension:jar:3.1.0:compile
[INFO] |  |     +- com.baomidou:mybatis-plus-core:jar:3.1.0:compile
[INFO] |  |     |  +- com.baomidou:mybatis-plus-annotation:jar:3.1.0:compile
[INFO] |  |     |  +- org.mybatis:mybatis:jar:3.5.0:compile
[INFO] |  |     |  \- com.github.jsqlparser:jsqlparser:jar:1.4:compile
[INFO] |  |     \- org.mybatis:mybatis-spring:jar:2.0.0:compile
[INFO] |  +- org.springframework.boot:spring-boot-autoconfigure:jar:2.0.7.RELEASE:compile
[INFO] |  \- org.springframework.boot:spring-boot-starter-jdbc:jar:2.0.7.RELEASE:compile
[INFO] |     \- com.zaxxer:HikariCP:jar:2.7.9:compile
[INFO] +- mysql:mysql-connector-java:jar:5.1.47:compile
[INFO] +- com.alibaba:fastjson:jar:1.2.83:compile
[INFO] +- com.alibaba:druid-spring-boot-starter:jar:1.1.16:compile
[INFO] |  +- com.alibaba:druid:jar:1.1.16:compile
[INFO] |  \- org.slf4j:slf4j-api:jar:1.7.25:compile
[INFO] +- com.alibaba:transmittable-thread-local:jar:2.11.5:compile
[INFO] +- org.apache.commons:commons-text:jar:1.6:compile
[INFO] |  \- org.apache.commons:commons-lang3:jar:3.7:compile
[INFO] +- org.apache.commons:commons-csv:jar:1.8:compile
[INFO] +- commons-io:commons-io:jar:2.7:compile
[INFO] +- commons-codec:commons-codec:jar:1.11:compile
[INFO] +- com.google.guava:guava:jar:28.2-jre:compile
[INFO] |  +- com.google.guava:failureaccess:jar:1.0.1:compile
[INFO] |  +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] |  +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] |  +- org.checkerframework:checker-qual:jar:2.10.0:compile
[INFO] |  +- com.google.errorprone:error_prone_annotations:jar:2.3.4:compile
[INFO] |  \- com.google.j2objc:j2objc-annotations:jar:1.3:compile
[INFO] +- org.jsoup:jsoup:jar:1.11.3:compile
[INFO] \- us.codecraft:xsoup:jar:0.3.1:compile
[INFO]    \- org.assertj:assertj-core:jar:3.9.1:compile

Suggested solutions:

Update dependency version

Thank you very much.

文档仓库

https://www.spiderflow.org/ 这个地址挂了，有文档仓库啥的嘛

demo站点挂了

http://demo.spiderflow.org/ 打不开

啊啪啪啪

奇绩创坛投资Reach Out

我是奇绩创坛投资研究团队的罗泽铭，主要关注软件生态和底层infra的投资机会。
现阶段在寻找优质的开源项目，对spider-flow 图形化爬虫挺感兴趣的。

无论是创业项目投资，还是类似基金会早期支持，都希望和你进一步聊一聊。

手机号/微信：18123892890

期待回信，祝生活愉快，谢谢。
罗泽铭

【背景信息】
奇绩创坛是一家加速器和创投基金，由前百度总裁兼COO陆奇博士创立，希望助力**技术驱动早期创新创业生态，前身是Y Combinator**。

我们过去一年投资了80多支初创团队，其中有不少的优秀开源项目，包括代码质量评估的思码逸 Merico、远程桌面控制工具 Rustdesk、面向产业IoT的开源开发平台 Shifu OS、云端研发工具链和协作平台 TeamCode等相关项目。他们大多都是从一个早期的开源项目开始的。

我们尤其希望支持优秀的开源项目找到商业化应用场景落地。技术团队创业，除了需要过硬的技术背景，在产品、团队、商业和战略等方面，同样存在许多潜在挑战。陆奇博士及核心团队会像联合创始人一样，帮助公司从零到一、稳健地走出第一步

DockerFile构建失败

我使用Docker部署项目，编译DockerFile时出现缺少重要文件。

ADD ./spider-flow-web/target/spider-flow.jar ./

spider-flow-web目录下无target文件夹及文件。

麻烦可以上传一下吗？

How can i do some AJAX requests by selenium in spiderflow?

I don't know how to build AJAX requests by dragging and dropping
Bro, if you give me some examples, i will thank for your help really!

大佬们这个截图要怎么操作呀

webElement.screen()截下来看到的是空文件

连接线节点-怎么没有出现?

连接线节点-怎么没有出现?

已爬取网页的过滤功能

您好，不知道目前平台是否有过滤已爬取网页的功能？或者说这部分功能，需要自己单独写插件来支持？

大佬，class里空格怎么操作？

一个class里面有空格，然后获取不到内容，其他下划线都正常。

demo is invalid

http://demo.spiderflow.org/ gives "An error condition occurred while reading data from the network. Please retry your request."

English translation

Hello @javamxd
Is there a way to translate Interface to other languages?
I like how the interface looks, and the project seems very interesting, buy I am unable to read the letters, so it would be nice if you made some translation to English.

启动两个爬虫的时候会死循环

filter的时候没数据

大佬们这个自定义变量怎么用呀

能不能定义成数组或者list的格式

这个不支持获取解析之后得html.比如天猫的价格信息.这个是获取不到的

目前项目还有在维护更新嘛？

如题

在vscode上引入了nashorn包后，启动springboot显示找不到主类。求解决

本地运行没问题,发布出现问题. org.apache.catalina.LifecycleException

失败 - 上下文路径[/flow]下，应用程序无法启动
失败 - 发生异常[org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[/flow]]]

IP代理池功能

Ip代理池的功能还会有吗，现在对应的插件拿不到了，是否会集成

能不能跟magic-api一样，可以设置密码 security-config

能不能跟magic-api一样，可以设置密码 security-config

大佬，网站停用了吗？有什么替用的域名吗

模拟 ip 怎么搞？？

模拟 ip 怎么搞？？

生成的文件目录怎么配置啊？

生成的文件目录怎么配置啊？默认生成在根目录

该项目还维护吗？

org.jsoup.validateTLSCertificates函数不存在

maser分支的org.spiderflow.core.io.HttpRequest.java，中jsoup.version:1.15.3版本导致代码不兼容，要改成1.11.3版本，才能编译通过，connection没有validateTLSCertificates函数

@SuppressWarnings("deprecation")
public HttpRequest validateTLSCertificates(boolean value){
	this.connection.validateTLSCertificates(value);
	return this;
}

你好，请问你的流程编辑是用什么插件完成的呢

循环节点会打乱数据的顺序

顺序的数据

生成乱序的数据

爬虫
sp_flow.txt

这个问题是由多线程引起的，除了使用单线程执行外，还有没有别的办法让数据按顺序插入数据库吗

Selenium 结点 添加 命令无效

如图，添加命令没有执行

保存数据到数据库是否支持去重？

保存数据到数据库是否支持去重？

定时执行，抓取数据，然后保存到数据库，插入前需要校验下是否存在该内容，如果存在则忽略，否则会存在多条一样的数据。

css选择器能否增加如jq筛选能力

如题

增加Puppeteer这种方式来抓取动态页面数据,有啥好的方式集成进来没？

增加Puppeteer这种方式来抓取动态页面数据,有啥好的方式集成进来没？

也就是：https://github.com/fanyong920/jvppeteer

通过插件的形式么？

Linux下安装提示,jdk.nashorn.api.scripting does not exist,但是jdk是1.8的

[ERROR] /spider-flow/spider-flow-core/src/main/java/org/spiderflow/core/script/ScriptManager.java:[3,33] package jdk.nashorn.api.scripting does not exist

jdk.nashorn.api.scripting程序包不存在

群主，这个项目不继续搞了吗

请问项目还在维护吗？

官网（包含文档）仓库

https://github.com/ssssssss-team/spider-flow-site

需要登录怎么抓取啊？？？