KeyError: 'tokens' about whisper-jax

[sanchit-gandhi]

Is port 8000 the right port? Usually gradio defaults to port 7860 - could you check with:

API_URL=http://0.0.0.0:7860/generate/ API_URL_FROM_FEATURES=http://0.0.0.0:7860/generate_from_features/ python app.py

(also generate_from_features was spelled, wrong, which could have been the culprit)

Either way, I agree that we should have a better error message! Will open a PR for this

[sanchit-gandhi]

Using the main branch now should give you a more detailed Gradio error message @syedmustafa54!

[syedmustafa54]

@sanchit-gandhi Thanks for the update but now I am facing a different issue on GPU it's not giving output its stuck I have modified demo.launch(server_name="0.0.0.0",server_port=9504, show_api=False) and just ran this command python direct_app.py

To create a public link, set `share=True` in `launch()`.
2023-04-24 14:51:03;INFO;Loading audio file...
2023-04-24 14:51:04;INFO;Pre-processing audio file...

its stuck for long time there

[syedmustafa54]

lly gradio defaults to port 7860

port 8000 is for sh launch_app.sh and for app.py gradio its 9504 I have modified it

[sanchit-gandhi]

Cool! The model is probably JIT compiling (which takes a few minutes the first time you run it, but will be fast thereafter). See https://github.com/sanchit-gandhi/whisper-jax#pipeline-usage for details.

[syedmustafa54]

@sanchit-gandhi any guess on T4 I have waited for 25-30 mins but still no results is this normal?

[sanchit-gandhi]

Which checkpoint are you using? And how long is the audio file? If it's stuck on the pre-processing stage then it might be that you're maxed out on CPU resource for pre-processing the audio file.

[syedmustafa54]

I just ran direct_app.py. the audio file is 3 sec. I have checked the CPU usage its only 1 process is using 100% rest is at 0%

[sanchit-gandhi]

Hey @syedmustafa54 - could you first check that the Flax Whisper Pipeline works standalone with this audio file, i.e. not as a gradio app? This will give you better error logs for any environment / resource errors. You can follow the instructions from https://github.com/sanchit-gandhi/whisper-jax#pipeline-usage

[syedmustafa54]

Hey I tried to run the below code by adding audio.mp3 in the folder but still its not giving any error just not showing any output

from whisper_jax import FlaxWhisperPipline

# instantiate pipeline
pipeline = FlaxWhisperPipline("openai/whisper-large-v2")

# JIT compile the forward call - slow, but we only do once
text = pipeline("audio.mp3")

# used cached function thereafter - super fast!!
text = pipeline("audio.mp3")```

[syedmustafa54]

Hello @sanchit-gandhi I have started a new Instance install cuda-11.8 cudnn-8.8 and install all the requirements
and ran the below code

from whisper_jax import FlaxWhisperPipline

# instantiate pipeline
pipeline = FlaxWhisperPipline("openai/whisper-large-v2")

# JIT compile the forward call - slow, but we only do once
text = pipeline("audio.mp3")

# used cached function thereafter - super fast!!
text = pipeline("audio.mp3")
print(text)

Now this code is working fine its taking 1.5 mins to give the output on 3 sec audio on T4 GPU.
After confirming the code is working I ran python direct_app.py it has pass 800 sec still no results output was

2023-04-25 10:40:23;INFO;Loading audio file...

2023-04-25 10:40:23;INFO;Pre-processing audio file...

according to me when I debug the issue is in the below for loop logic it has printed Transcribing... {'stride': [(25728, 0, 0)], 'input_features': array([[[-0.58101606, -0.58101606, -0.38316262, ..., -0.58101606, and stuck at there after that for loop I have also added a print but that is not getting a trigger.
This is the for loop

   for batch, _ in zip(dataloader, progress.tqdm(dummy_batches, desc="Transcribing...")):
            print("Transcribing...", batch.shape)
            model_outputs.append(
                pipeline.forward(batch, batch_size=BATCH_SIZE, task=task, return_timestamps=return_timestamps)
            )

[sanchit-gandhi]

Hey @syedmustafa54 - could you maybe try and pinpoint the exact line the app is failing on? Perhaps you can quit the app when it hangs (CTRL + C) and paste the traceback here? For reference, I'm using direct_app.py exactly as is to host the Whisper JAX demo on the HF Hub!

There are some more verbose instructions for creating a demo here: https://github.com/sanchit-gandhi/whisper-jax#creating-an-endpoint

[syedmustafa54]

@sanchit-gandhi for your case you are just hosting the front end on hugging face and back end at Google TPU I think that's why you are not getting any issues.
this are the logs after CTRL+C

Initialized persistent compilation cache at ./jax_cache
/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/inputs.py:319: UserWarning: Usage of gradio.inputs is deprecated, and will not be supported in the future, please import your components from gradio.components
  warnings.warn(
/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/deprecation.py:40: UserWarning: `optional` parameter is deprecated, and it has no effect
  warnings.warn(value)
/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/inputs.py:183: UserWarning: Usage of gradio.inputs is deprecated, and will not be supported in the future, please import your component from gradio.components
  warnings.warn(
/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/inputs.py:121: UserWarning: Usage of gradio.inputs is deprecated, and will not be supported in the future, please import your component from gradio.components
  warnings.warn(
/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/outputs.py:22: UserWarning: Usage of gradio.outputs is deprecated, and will not be supported in the future, please import your components from gradio.components
  warnings.warn(
/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/inputs.py:27: UserWarning: Usage of gradio.inputs is deprecated, and will not be supported in the future, please import your component from gradio.components
  warnings.warn(
/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/deprecation.py:40: UserWarning: `numeric` parameter is deprecated, and it has no effect
  warnings.warn(value)
/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/blocks.py:863: UserWarning: api_name predict already exists, using predict_1
  warnings.warn(
/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/blocks.py:863: UserWarning: api_name predict already exists, using predict_2
  warnings.warn(
Running on local URL:  http://0.0.0.0:7860

To create a public link, set `share=True` in `launch()`.
2023-04-26 12:00:30;INFO;Loading audio file...
2023-04-26 12:00:31;INFO;Pre-processing audio file...
^CKeyboard interruption in main thread... closing server.
Process ForkPoolWorker-4:
Process ForkPoolWorker-6:
Process ForkPoolWorker-8:
Process ForkPoolWorker-7:
Process ForkPoolWorker-5:
Process ForkPoolWorker-3:
Process ForkPoolWorker-1:
Process ForkPoolWorker-2:
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/queues.py", line 365, in get
    with self._rlock:
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
Traceback (most recent call last):
Traceback (most recent call last):
KeyboardInterrupt
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/queues.py", line 365, in get
    with self._rlock:
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
Traceback (most recent call last):
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/queues.py", line 366, in get
    res = self._reader.recv_bytes()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/connection.py", line 221, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/connection.py", line 419, in _recv_bytes
    buf = self._recv(4)
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/connection.py", line 384, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/queues.py", line 365, in get
    with self._rlock:
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/queues.py", line 365, in get
    with self._rlock:
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/queues.py", line 365, in get
    with self._rlock:
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/queues.py", line 365, in get
    with self._rlock:
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/queues.py", line 365, in get
    with self._rlock:
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
KeyboardInterrupt
^CTraceback (most recent call last):
  File "/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/blocks.py", line 1984, in block_thread
    time.sleep(0.1)
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/whisper-jax/app/direct_app.py", line 236, in <module>
    demo.launch(server_name="0.0.0.0", show_api=False)
  File "/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/blocks.py", line 1901, in launch
    self.block_thread()
  File "/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/blocks.py", line 1987, in block_thread
    self.server.close()
  File "/opt/conda/envs/whisper/lib/python3.9/site-packages/gradio/networking.py", line 43, in close
    self.thread.join()
  File "/opt/conda/envs/whisper/lib/python3.9/threading.py", line 1029, in join
    self._wait_for_tstate_lock()
  File "/opt/conda/envs/whisper/lib/python3.9/threading.py", line 1045, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
KeyboardInterrupt
^CProcess ForkPoolWorker-9:
Traceback (most recent call last):
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/queues.py", line 366, in get
    res = self._reader.recv_bytes()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/connection.py", line 221, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/connection.py", line 419, in _recv_bytes
    buf = self._recv(4)
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/connection.py", line 384, in _recv
    chunk = read(handle, remaining)
KeyboardInterrupt
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/util.py", line 300, in _run_finalizers
    finalizer()
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/util.py", line 224, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 692, in _terminate_pool
    cls._help_stuff_finish(inqueue, task_handler, len(pool))
  File "/opt/conda/envs/whisper/lib/python3.9/multiprocessing/pool.py", line 672, in _help_stuff_finish
    inqueue._rlock.acquire()
KeyboardInterrupt
^C

[sanchit-gandhi]

Hey @syedmustafa54 - the application is agnostic to whether we host on HF / locally. How many preprocessing workers are you using? Is this more than the number of CPUs you have available? If so, then multiprocessing will hang indefinitely.

Could you check the number of CPUs you have through:

lscpu

And make sure that this number is less than the product of NUM_PROC:

whisper-jax/app/app.py

Line 22 in 7159222

NUM_PROC = 8

And concurrency_count:

whisper-jax/app/app.py

Line 244 in 7159222

demo.queue(concurrency_count=1, max_size=5)

(so lscpu should be less than NUM_PROC * concurrency_count )