Blocks engine: Ingester TSDB directory is not writable about cortex-helm-chart

[grzegdl]

Same went for store-gateway. It had an error:
level=warn ts=2021-01-12T11:07:50.560293065Z caller=bucket_stores.go:127 msg="failed to synchronize TSDB blocks" err="mkdir tsdb-sync: read-only file system"

I fixed it by specifying it's data dir:

  blocks_storage:
    bucket_store:
      sync_dir: "data"

Assuming that's how it should be configured it would be good to add it to the helm chart.

[MartinHell]

I think this is related to the fact that the tsdb configuration defaults to the directory "tsdb". Have a look here https://cortexmetrics.io/docs/configuration/configuration-file/#blocks_storage_config and scroll down to the tsdb section.

I've also noticed that when using blocks_storage the /data directory is not being used. My guess is /data is used for chunks storage and therefore the helm chart mounts that as a pvc (if you're running the ingester as a statefulset).

If that is true then I think the tsdb should be stored in /data as well. To do that all you'd need to set is the following:

  blocks_storage:
    backend: s3
    tsdb:
      dir: "/data"

Also if you're using the WAL you can't have the tsdb located in an emptyDir since the WAL would be located alongside it IIRC (so the wal would be lost on pod failures/restarts).

If the above is correct then I think the documentation needs to be updated.

[sbaier1]

For the record this can be worked around by setting these properties in the values.yaml, even with the current version:

ingester:
  extraVolumes:
    - name: tsdb
      emptyDir: {}
  extraVolumeMounts:
    - name: tsdb
      mountPath: /tsdb

[rubycut]

Why is this issue closed if it is not fixed? The workaround doesn't mean that issue is fixed.