yu-iskw / tensorflow-hub-with-ml-engine

Example of transfer learning with tensorflow-hub and Google ML Engine

Retrain flower data with tensorflow-hub

This is an example code to reproduce an issue of tensorflow-hub with Google ML Engine. The reason why I made the repository is to reproduce the issue that tensorflow-hub doesn't work on Google ML Engine at the time I am making the repository.

• How to use TF Hub on a distributed setting? · Issue #48 · tensorflow/hub
• Tensorflow-Hub doesn't work on the runtime version 1.7. [78898344] - Visible to Public - Issue Tracker

OP_REQUIRES failed at save_restore_tensor.cc:170 : Invalid argument: Unsuccessful TensorSliceReader constructor: Failed to get matching files on /tmp/tfhub_modules/11d9faf945d073033780fd924b2b09ff42155763/variables/variables: Not found: /tmp/tfhub_modules/11d9faf945d073033780fd924b2b09ff42155763/variables; No such file or directory

Requirements

• Anaconda
• Google Cloud SDK

Prepare for the python environment

# Create conda environment
make create-conda

# Remove conda environment
make remove-conda

Hot wo run

1. Download flower data. The script downloads the flower data in data/flower_photos

bash ./dev/prepare-dataset.sh

2. Create TFRecord data.

python crete_tfrecord.py \
  --input ./data/flower_photos \
  --train_output ./train.tfrecord \
  --eval_output ./eval.tfrecord

3. Upload TFRecord data to Google Cloud Storage.

PROJECT_ID=...
GCS_BUCKET=...

# Create a bucket if necessary.
gsutil mb -p "$PROJECT_ID" "gs://${GCS_BUCKET}"

# Copy TFRecord files.
gsutil cp -p train.tfrecord "gs://${GCS_BUCKET}/train.tfrecord"
gsutil cp -p eval.tfrecord "gs://${GCS_BUCKET}/eval.tfrecord"

# Check the uploaded files.
gsutil ls "gs://${GCS_BUCKET}/train.tfrecord"
gsutil ls "gs://${GCS_BUCKET}/eval.tfrecord"

4. Train a model.

# Run on your local machine to test.
GCS_BUCKET=...
TRAIN_DATA="gs://${GCS_BUCKET}/train.tfrecord"
EVAL_DATA="gs://${GCS_BUCKET}/eval.tfrecord"
bash run_local.sh "${TRAIN_DATA}" "${EVAL_DATA}"

# Run on ML Engine
PROJECT_ID=...
GCS_BUCKET=...
MODEL_DIR="gs://${GCS_BUCKET}/model/"
TRAIN_DATA="gs://${GCS_BUCKET}/train.tfrecord"
EVAL_DATA="gs://${GCS_BUCKET}/eval.tfrecord"
bash run_cloud.sh \
  "$PROJECT_ID" \
  "$GCS_BUCKET" \
  "$MODEL_DIR" \
  "$TRAIN_DATA" \
  "$EVAL_DATA"

# Monitor the training
tensorboard --logdir="$MODEL_DIR"

tensorflow-hub-with-ml-engine's Issues

Security Policy violation Branch Protection

This issue was automatically created by Allstar.

Security Policy Violation
No protection found for branch master

---

This issue will auto resolve when the policy is in compliance.

Issue created by Allstar. See https://github.com/ossf/allstar/ for more information. For questions specific to the repository, please contact the owner or maintainer.

Security Policy violation SECURITY.md

This issue was automatically created by Allstar.

Security Policy Violation
Security policy not enabled.
A SECURITY.md file can give users information about what constitutes a vulnerability and how to report one securely so that information about a bug is not publicly visible. Examples of secure reporting methods include using an issue tracker with private issue support, or encrypted email with a published key.

To fix this, add a SECURITY.md file that explains how to handle vulnerabilities found in your repository. Go to https://github.com/yu-iskw/tensorflow-hub-with-ml-engine/security/policy to enable.

For more information, see https://docs.github.com/en/code-security/getting-started/adding-a-security-policy-to-your-repository.

---

This issue will auto resolve when the policy is in compliance.

Issue created by Allstar. See https://github.com/ossf/allstar/ for more information. For questions specific to the repository, please contact the owner or maintainer.