Unable to create the new target Tablets when resharding about vitess-operator

[enisoc]

For number 1, usually it's informative to check kubectl describe vts <vts-name>. The events log should show why the operator was unable to elect a primary.

For number 2, vtctl DropSources cleans up the Vitess topology, but it doesn't know how to actually undeploy the k8s resources. For that, you'd need to edit the VitessCluster CRD to remove the unwanted partitioning and then reapply it.

For number 3, can you share the YAML for the VitessCluster CRD?

[biostone]

For number 1, here is the output of kubectl describe vts <vts-name>:

NOTE: in this case, shard 80-aa, both tablets are replica and both pods in READY 2/3, and STATUS Running . This 80-aa is from the second partitioning of a keyspace in the VitessCluster CRD YAML.

> kubectl describe vts  example-customer-80-aa-67c9b737
Name:         example-customer-80-aa-67c9b737
Namespace:    default
Labels:       planetscale.com/cluster=example
              planetscale.com/keyspace=customer
              planetscale.com/shard=80-aa
Annotations:  <none>
API Version:  planetscale.com/v2
Kind:         VitessShard
Metadata:
  Creation Timestamp:  2021-01-07T21:17:01Z
  Generation:          1
  Owner References:
    API Version:           planetscale.com/v2
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  VitessKeyspace
    Name:                  example-customer-085639aa
    UID:                   7c576a5a-b594-4ffc-a3cf-a718f04a03d5
  Resource Version:        22782
  Self Link:               /apis/planetscale.com/v2/namespaces/default/vitessshards/example-customer-80-aa-67c9b737
  UID:                     d85f2fc4-4ebe-4d6b-9a1f-23e6148841f4
Spec:
  Database Init Script Secret:
    Key:   init_db.sql
    Name:  example-cluster-config
  Global Lockserver:
    Address:         example-etcd-faf13de3-client.default.svc:2379
    Implementation:  etcd2
    Root Path:       /vitess/example/global
  Image Pull Policies:
  Images:
    Mysqld:
      mysql56Compatible:  vitess/lite:v8.0.0
    Mysqld Exporter:      prom/mysqld-exporter:v0.11.0
    Vtbackup:             vitess/lite:v8.0.0
    Vttablet:             vitess/lite:v8.0.0
  Key Range:
    End:    aa
    Start:  80
  Name:     80-aa
  Replication:
    Initialize Backup:         true
    Initialize Master:         true
    Recover Restarted Master:  true
  Tablet Pools:
    Cell:  zone1
    Data Volume Claim Template:
      Access Modes:
        ReadWriteOnce
      Resources:
        Requests:
          Storage:  4Gi
    Mysqld:
      Resources:
        Requests:
          Cpu:     100m
          Memory:  256Mi
    Replicas:      2
    Type:          replica
    Vttablet:
      Extra Flags:
        db_charset:  utf8mb4
      Resources:
        Requests:
          Cpu:     100m
          Memory:  256Mi
  Topology Reconciliation:
    Prune Cells:             true
    Prune Keyspaces:         true
    Prune Shard Cells:       true
    Prune Shards:            true
    Prune Srv Keyspaces:     true
    Prune Tablets:           true
    Register Cells:          true
    Register Cells Aliases:  true
  Update Strategy:
    Type:  Immediate
  Zone Map:
    zone1:
Status:
  Cells:
    zone1
  Has Initial Backup:     Unknown
  Has Master:             False
  Idle:                   True
  Lowest Pod Generation:  1
  Observed Generation:    1
  Serving Writes:         False
  Tablets:
    zone1-1015435830:
      Available:          False
      Data Volume Bound:  True
      Index:              1
      Pool Type:          replica
      Ready:              False
      Running:            True
      Type:               replica
    zone1-3421714772:
      Available:          False
      Data Volume Bound:  True
      Index:              2
      Pool Type:          replica
      Ready:              False
      Running:            True
      Type:               replica
Events:
  Type     Reason               Age                   From                               Message
  ----     ------               ----                  ----                               -------
  Normal   Updated              4m9s                  vitessshard-controller             updated Pod example-vttablet-zone1-3421714772-ef2f5b63
  Normal   Created              4m9s                  vitessshard-controller             created PersistentVolumeClaim example-vttablet-zone1-3421714772-ef2f5b63
  Normal   Created              4m9s                  vitessshard-controller             created Pod example-vttablet-zone1-1015435830-40cbdc57
  Normal   Created              4m9s                  vitessshard-controller             created Pod example-vttablet-zone1-3421714772-ef2f5b63
  Normal   PVCResizeWaiting     4m9s (x4 over 4m9s)   vitessshard-controller             Waiting for PVC example-vttablet-zone1-1015435830-40cbdc57 to be ready for filesystem resize.
  Normal   Updated              4m9s                  vitessshard-controller             updated Pod example-vttablet-zone1-1015435830-40cbdc57
  Normal   Created              4m9s                  vitessshard-controller             created PersistentVolumeClaim example-vttablet-zone1-1015435830-40cbdc57
  Warning  TopoGetFailed        4m7s (x7 over 4m9s)   vitessshard-controller             failed to get tablet records: node doesn't exist: /vitess/example/global/keyspaces/customer/shards/80-aa/Shard
  Warning  TopoGetFailed        4m7s (x8 over 4m9s)   vitessshard-controller             failed to get shard info: node doesn't exist: /vitess/example/global/keyspaces/customer/shards/80-aa/Shard
  Warning  TopoGetFailed        4m3s (x13 over 4m9s)  vitessshardreplication-controller  failed to get shard record: node doesn't exist: /vitess/example/global/keyspaces/customer/shards/80-aa/Shard
  Warning  NotReconcilingDrain  4m1s                  vitessshardreplication-controller  Shard is in an unhealthy state: tablet zone1-3421714772 is not Available
  Warning  NotReconcilingDrain  3m56s (x5 over 4m1s)  vitessshardreplication-controller  Shard is in an unhealthy state: tablet zone1-1015435830 is not Available
  Normal   InitShardWaiting     3m56s (x6 over 4m1s)  vitessshardreplication-controller  can't initialize shard: tablet zone1-1015435830 is not running

For number 2, I did edit the yaml file, to remove the source partitioning of the reshard. What I reported is the symptom after I applied the new yaml file. What I am saying is that kubectl get pods still returns the pods that is not in the VitessCluster CRD YAML. Reapplying the updated yaml does not help at all. Those unwanted pods get stuck in READY 2/3, and STATUS Running state.

[biostone]

And here is the yaml file for number 3:
306_news_shards.yaml.txt

Thanks for look into this case! Really appreciate your help.

[biostone]

With vitess-9.0.0-rc1 paired with operator:latest, the above errors are gone.