alm-lab / pace

PACE (Podcast AI for Chapters and Episodes) is a semantic search engine that helps you find the information you need, inter- and intra-podcasts (Project for the AssemblyAI Winter 2022 Hackathon).

🎧 PACE: Podcast AI for Chapters and Episodes

DEMO: Try it out for limited time on our demo page.

📝 Table of Contents

• 🎯 Our IDEA
• 🔎 How does it work?
• 📺 Demo
• ⏳ Pipeline
• 🙏🏻 Acknowledgements
• 🤌🏻 About the project

---

🎯 Our IDEA

How many times have you been listening to a podcast and you wanted to go back to a specific part, but you don't remember the exact time? We have all been there. Now we got you covered!

We present PACE, Podcast AI for Chapters and Episodes, which allows you to search for podcasts at your own pace.
PACE is a semantic search engine that helps you find the information you need, in a fast and easy way.

To get more superpowers and let the magic begin, switch the checkbox on. PACE will use the power of AI to search inside the podcasts and give you back the exact part you're looking for.

In addition to providing fast and easy access to information within podcasts, PACE also uses advanced text-to-image generative models to automatically create chapter covers for each podcast episode. This makes it easy for listeners to quickly identify and navigate to the specific information they are looking for within an episode.

With PACE, accessing the information you need within podcasts has never been easier.
Get ready to experience the future of podcasting. Everything you need, just a click away!

---

🔎 How does it work?

Podcasts usually are long-form audio content. They could contain many interesting topics, but you may be interested in only one or some of them.

PACE got you covered!

• 🔈 Speech Processing: generate chapters from audio tracks
• 📃 NLP: search for episodes or chapters using natural language
• 🎨 Computer Vision: generate artwork for chapters using text-to-image models
• 💻 Web: Flask + Elasticsearch to build a web app

---

📺 Demo & Slides

Find the slides of our project here.

---

⏳ Pipeline

This is the pipeline of our project.

We ingest podcast rss feeds, and we enrich the dataframe with episodes and related metadata. We then filter episodes' description using BERT Podcast Classifier.

We encode episodes' description using sentence BERT, and we index them in Elasticsearch.

We then select a bunch of podcasts for intra-podcast search, and we use AssemblyAI to transcribe podcasts and retrieve chapters inside episodes. We index episodes' chapters in elasticsearch, using cohere to embed the chapters.

We finally leverage stable diffusion models to create an image/gif for each of the generated chapter.

---

🕒 PHASE 1

1. Ingest podcasts rss
2. Enrich dataframe with episodes and related metadata
3. Filter episodes' description using BERT
4. Encode episodes' description using sentence BERT
5. Index episodes in elasticsearch
6. TEST: Query elasticsearch using sentence BERT

END OF PHASE 1

---

🕕 PHASE 2

7. Select a bunch of podcasts for intra-podcast search (limited by AssemblyAI API)

8. Index episodes' chapters in elasticsearch

   • Podcast name
   • Chapter description
   • Start & End time
   • Original episode (mp3 link)

9. TEST: Query elasticsearch using cohere

10. Web app

    • Implement flask-based web app
    • Implement search bar
    • Implement search results
    • Implement player
    • Implement intra-podcast search

END OF PHASE 2

---

🕘 PHASE 3

11. Generative models (generate gifs for chapters)
12. Implement storage mechanism for images/gifs
13. Implement image/gif retrieval in web app
14. TEST: Test the overall correct behavior of the web app, in all its parts

END OF PHASE 3

---

🕛 PHASE 4

15. Slides & Demo
16. Add a "search by voice" button to the web app -- Future Work
17. Add "Fake News" detection to the web app - Future Work

END OF PHASE 4

---

🙏🏻 Acknowledgements

• AssemblyAI for the outstanding experience (and the free credits)
• Cohere for the amazing API
• 🎬 plyr.io for the media player design
• 🤗 Huggingface for the amazing models
• ☕️ ☕️ 🧃 for the energy

---

🤌🏻 About the project

PACE is a project created by Alkis Koudounas, Lorenzo Vaiani, and Moreno La Quatra. It is part of the AssemblyAI 50K Hackathon - Winter 2022.