Coder Social home page Coder Social logo

Comments (7)

Horrypotter avatar Horrypotter commented on August 20, 2024

WeChatd58bc9ed24bf04fd4ca6363e627a1837

我在服务器上跑的,总共26w多条数据,slots是4,超级慢啊!!!

from chunjun.

Horrypotter avatar Horrypotter commented on August 20, 2024

同样的数据和环境用datax就十几秒

from chunjun.

yangsishu avatar yangsishu commented on August 20, 2024

你用的是那个版本

from chunjun.

yangsishu avatar yangsishu commented on August 20, 2024

最近我们也在做性能测试,几十万条记录跑这么慢肯定是有问题的,能发下你任务的的json配置吗

from chunjun.

yelijun avatar yelijun commented on August 20, 2024

1亿多 30分钟左右(每条数据字节数小),如果每条数据内容多会更慢,看了数据,每秒900条左右

from chunjun.

Horrypotter avatar Horrypotter commented on August 20, 2024

从mysql往mysql抽取,速度正常;
从mysql往oushuDB抽取,就是超级慢;
原因可能是他这个数据库对inset貌似支持不好。用datax向oushuDB插入的时候使用的插件是GpdbWriter插件,这个插件使用了copy from 来代替insert into ,速度超快。
我看这里的插入都是用insert into来搞的,也许可以改为copy from

from chunjun.

yangsishu avatar yangsishu commented on August 20, 2024

oushuDB 这个我们是没有测试过的

from chunjun.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.