pren1 / a_pipeline_of_pretraining_bert_on_google_tpu

A tutorial of pertaining Bert on your own dataset using google TPU

About Me

Hello! I am an enthusiastic and driven individual who is passionate about programming, research, learning, and problem-solving. I strive to leverage my technical abilities and insatiable curiosity to tackle real-world challenges and make a positive impact through technology.

My Loss Function

$MyLoss = D_{KL}(work||life) + patience * \left | \left | bugs \right | \right | _2$

Maintaining a healthy work-life balance can be a daunting task for anyone, especially for programmers who are constantly grappling with debugging issues. However, I am a firm believer in the power of patience and persistence. I know that with these qualities, I can overcome any obstacle and continue to grow as a professional.

My Projects

Here you will find a selection of my most interesting projects. Some are related to my research, while others are just for fun. Regardless, I'm proud of all of them and I hope you enjoy exploring them as much as I enjoyed creating them.

🎉 Empowering the Visually Impaired with an Innovative Indoor Navigation App

Introducing my iPhone app, designed to provide reliable and accurate navigation assistance to visually impaired individuals navigating indoor spaces. Utilizing advanced particle filter technology and programmed in Swift, my app offers a dependable solution for indoor navigation challenges faced by those with visual impairments.

To see the full capabilities of my app, check out this video demonstration. I'm excited to offer this tool to empower and enhance the independence of visually impaired individuals.

💡 Deep Learning Projects

• Pretraining BERT on Google TPU A tutorial I wrote to help others take advantage of Google's fast TPUs for pretraining BERT models. With 40 stars, it's been well received!

• MMD Variational Autoencoder A MMD-VAE implemented in Keras, for unsupervised representation learning.

• Chinese Voice Synthesizer A synthesizer developed for several Chinese vtubers. The demo video is available on Bilibili.

• VAST A model for separating mixed voices from two Japanese speakers.

• Artificial DD A fun project that sends context-based fake Chinese chat messages. The demo video is available on Bilibili.

💻 Machine Learning Projects

• Emoticon Discriminator A fast implementation of a Naive Bayes classifier for emoticon discrimination.

• Fans Kalman A Kalman filter for predicting the fans of any vtuber.

📖 Class Projects

• Primary Type Classifier A primary type classifier based on Naive Bayes.

• JPCT Demo A face model render app for Android, with detailed documentation.

🌟 Machine Learning Tutorials

I also write tutorials on basic machine learning topics, such as SVM, logistic regression, and Naive Bayes. These are available on Bilibili (in Chinese).

Get in Touch

If you have any questions or just want to chat about my projects, please feel free to reach out to me via my email. I would love to hear from you and discuss any questions or feedback you may have.

Thank you for visiting my GitHub page, and I look forward to hearing from you soon!

a_pipeline_of_pretraining_bert_on_google_tpu's Issues

VM and Disk size when traing Bert in google TPU

Hello. Thank you very much for sharing your experience.
I also want to pretrain a bert with google v3-8, and I want to ask about how to choose the VM Type and Disk size to ensure a bert training.
I noticed that one example said that with 4GBs training corpus, in the create_pretraining_data step, there may cause about 400MB RAM. Is the Virtual Machine and Disk size need to bigger than the RAM maty take? (my corpus is about 1GB)

best wishes.