Comments (3)
This code fixed the problems for me. Note that this is block one and two.
import openai
from langchain.chat_models.openai import ChatOpenAI
from concurrent.futures import ThreadPoolExecutor
import tiktoken
from pathlib import Path
from langchain.schema import (
HumanMessage,
SystemMessage
)
YOUR_OPENAI_API_KEY = "Key Here " # Replace with your actual OpenAI API key
chat = ChatOpenAI(
model="gpt-3.5-turbo",
temperature=0.2,
max_tokens=500,
openai_api_key=YOUR_OPENAI_API_KEY
)
def load_text(file_path):
with Path(file_path).open("r") as file:
return file.read()
def save_to_file(responses, output_file):
with Path(output_file).open('w') as file:
file.write("\n".join(responses))
def call_openai_api(chunk):
messages = [
SystemMessage(content="Clean the following transcripts of all grammatical mistakes, misplaced words, and identify the speakers."),
HumanMessage(content=chunk)
]
response = chat(messages)
return response.content.strip()
def split_into_chunks(text, n_tokens=300):
encoding = tiktoken.encoding_for_model('gpt-3.5-turbo')
tokens = encoding.encode(text)
chunks = []
for i in range(0, len(tokens), n_tokens):
chunks.append(' '.join(encoding.decode(tokens[i:i + n_tokens])))
return chunks
def process_chunks(input_file, output_file, delay=0): # delay in seconds (if you hit a rate limit error)
text = load_text(input_file)
chunks = split_into_chunks(text)[:5]
responses = []
for chunk in tqdm(chunks):
responses.append(call_openai_api(chunk))
save_to_file(responses, output_file)
if __name__ == "__main__":
input_file = "YouTube.txt"
output_file = "clean_transcript.txt"
process_chunks(input_file, output_file)
# Can take up to a few minutes to run depending on the size of your data input
from youtube-to-chatbot.
I added the below and it works
!pip install google-api-python-client
import googleapiclient.discovery
from tqdm import tqdm
from youtube_transcript_api import YouTubeTranscriptApi
from youtube-to-chatbot.
Big thanks to you both for helping debug this error. I've just pushed this patch to main.
from youtube-to-chatbot.
Related Issues (6)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from youtube-to-chatbot.