Coder Social home page Coder Social logo

Comments (3)

dilzilla avatar dilzilla commented on July 22, 2024 1

This code fixed the problems for me. Note that this is block one and two.


import openai
from langchain.chat_models.openai import ChatOpenAI
from concurrent.futures import ThreadPoolExecutor
import tiktoken
from pathlib import Path
from langchain.schema import (
    HumanMessage,
    SystemMessage
)

YOUR_OPENAI_API_KEY = "Key Here "  # Replace with your actual OpenAI API key

chat = ChatOpenAI(
    model="gpt-3.5-turbo",
    temperature=0.2,
    max_tokens=500,
    openai_api_key=YOUR_OPENAI_API_KEY
)

def load_text(file_path):
    with Path(file_path).open("r") as file:
        return file.read()

def save_to_file(responses, output_file):
    with Path(output_file).open('w') as file:
        file.write("\n".join(responses))

def call_openai_api(chunk):
    messages = [
        SystemMessage(content="Clean the following transcripts of all grammatical mistakes, misplaced words, and identify the speakers."),
        HumanMessage(content=chunk)
    ]
    response = chat(messages)
    return response.content.strip()

def split_into_chunks(text, n_tokens=300):
    encoding = tiktoken.encoding_for_model('gpt-3.5-turbo')
    tokens = encoding.encode(text)
    chunks = []
    for i in range(0, len(tokens), n_tokens):
        chunks.append(' '.join(encoding.decode(tokens[i:i + n_tokens])))
    return chunks

def process_chunks(input_file, output_file, delay=0):  # delay in seconds (if you hit a rate limit error)
    text = load_text(input_file)
    chunks = split_into_chunks(text)[:5]
    responses = []
    for chunk in tqdm(chunks):
        responses.append(call_openai_api(chunk))

    save_to_file(responses, output_file)

if __name__ == "__main__":
    input_file = "YouTube.txt"
    output_file = "clean_transcript.txt"
    process_chunks(input_file, output_file)

    # Can take up to a few minutes to run depending on the size of your data input

from youtube-to-chatbot.

Hanalia avatar Hanalia commented on July 22, 2024

I added the below and it works

!pip install google-api-python-client
import googleapiclient.discovery
from tqdm import tqdm
from youtube_transcript_api import YouTubeTranscriptApi

from youtube-to-chatbot.

emmethalm avatar emmethalm commented on July 22, 2024

Big thanks to you both for helping debug this error. I've just pushed this patch to main.

from youtube-to-chatbot.

Related Issues (6)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.