cclljj / google_colab_ipynb

Ling-Jyh Chen's Github Page 👋

LJ Stats

google_colab_ipynb's Issues

`summarize_text` fails when running `colab_notebook_YouTube_transcription.ipynb` on longer videos

For example, on the video with id EW5ezacJa_U , the following error is observed

---------------------------------------------------------------------------
InvalidRequestError                       Traceback (most recent call last)
[<ipython-input-20-f0a75f3a603a>](https://localhost:8080/#) in <cell line: 2>()
     52 
     53         # call the summarize_text function and append the result to the summary variable
---> 54         r = summarize_text(m, lang)
     55         summary += r
     56 

5 frames
[/usr/local/lib/python3.9/dist-packages/openai/api_requestor.py](https://localhost:8080/#) in _interpret_response_line(self, rbody, rcode, rheaders, stream)
    681         stream_error = stream and "error" in resp.data
    682         if stream_error or not 200 <= rcode < 300:
--> 683             raise self.handle_error_response(
    684                 rbody, rcode, resp.data, rheaders, stream_error=stream_error
    685             )

InvalidRequestError: This model's maximum context length is 4097 tokens. However, your messages resulted in 28280 tokens. Please reduce the length of the messages.

The root cause of the error seems to be that split_article relies on existence of punctuation in transcribed text, which may not be the case with zh content. It may be desireable to integrate native zh segmentation models such as Jieba, CKIP, or monpa to achieve initial segmentation to limit the tokens sent to ChatGPT.