Coder Social home page Coder Social logo

Comments (5)

jawabuu avatar jawabuu commented on June 21, 2024

Hey @zehweh
Could you try this
https://gist.github.com/jawabuu/da27bb6bbb562d9caa5d1692955c892a

from vitess-operator.

zehweh avatar zehweh commented on June 21, 2024

Thanks @jawabuu for your quick response.

Unfortunately, I get connection errors again.
You can see the logs here: https://gist.github.com/zehweh/c55ebe1aedcee5acd498e954baa1cd27

I noticed, that vttablet complains about mysql running in read-only mode... This looks suspicious.

Some additional information I forgot to add:

  • kubernetes version 1.22.4
  • flannel for networking

from vitess-operator.

jawabuu avatar jawabuu commented on June 21, 2024

@zehweh Are you on #vitess slack channel?

from vitess-operator.

jawabuu avatar jawabuu commented on June 21, 2024

I'd like to see your
kubectl get po after running my gist

from vitess-operator.

zehweh avatar zehweh commented on June 21, 2024

I found out what was causing the issues:

1: The resource limits were set too low in my VitessCluster manifest which resulted in occasional OOM kills of the mysql container and the logs you see above.

2: I'm working with a master-master replication on the external database (unmanaged tablet). The master I'm connecting to has multiple sets of GTIDs:

mysql> SHOW SLAVE STATUS\G

*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
[...]
            Executed_Gtid_Set: 01af65be-d03d-11ec-a14f-0cc47a6af5da:1-28538331:544939690-544939696,
d048674a-e588-11ec-80cd-0cc47a6c65c8:1-65287695
[...]

Because of this, the MoveTables workflow wouldn't start:

❯ vtctlclient MoveTables  --source testdb -all Create vitesstest.testremote2vitess                                          
Waiting for workflow to start:
0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... 0% ... E0606 09:56:06.404528    5131 main.go:67] E0606 09:56:06.404251 vtctl.go:2582] workflow did not start within 30s
MoveTables Error: rpc error: code = Unknown desc = workflow did not start within 30s
E0606 09:56:06.406961    5131 main.go:76] remote error: rpc error: code = Unknown desc = workflow did not start within 30s

After switching to a master-slave setup and resetting the GTIDs, I was able to successfully start the VReplication.

Thanks @jawabuu for your help!

from vitess-operator.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.