bigbluebutton / bbb-transcription-controller

BigBlueButton

BigBlueButton is an open-source virtual classroom designed to help teachers teach and learners learn.

BigBlueButton supports real-time sharing of audio, video, slides (with whiteboard annotations), chat, and the screen. Instructors can engage remote students with polling, emojis, multi-user whiteboards, shared notes, and breakout rooms. During the session, BigBlueButton generates analytics that are visible to moderators in the Learning Analytics Dashboard.

Presenters can record and playback content for later sharing with others.

We designed BigBlueButton for online learning, it can be used for many other applications as well. The educational use cases for BigBlueButton are

• Online tutoring (one-to-one)
• Flipped classrooms (recording content ahead of your session)
• Group collaboration (many-to-many)
• Online classes (one-to-many)

The latest version is BigBlueButton 2.7. You can install BigBlueButton 2.7 on Ubuntu 20.04 using bbb-install.sh within 30 minutes (or your money back 😉).

For full technical documentation of BigBlueButton -- including architecture, features, API, and GreenLight (the default front-end) -- see https://docs.bigbluebutton.org/.

BigBlueButton and the BigBlueButton Logo are trademarks of BigBlueButton Inc.

bbb-transcription-controller's Issues

Closed captioning chat for deafblind users

Problem:

Current automatic speech recognition (ASR) solutions for video conferencing, including BigBlueButton, do not adequately address the needs of deafblind users. While real-time transcriptions can be helpful for deaf users, they are not accessible to blind users who rely on screen readers. Additionally, real-time transcriptions are not helpful for deafblind users who cannot see the captions or hear the audio.

Deafblindness is a condition that affects both vision and hearing. It can be caused by a variety of factors, including congenital conditions, injuries, and diseases. Deafblind people experience the world in a unique way, and they have specific needs that must be considered when designing technology.

Deafblind people are the most vulnerable and isolated population due to a lack of technology that is adapted to their needs. They often have difficulty accessing education, employment, and social activities.

Solution:

We propose a new feature that would create a closed captioning chat for deafblind users. This chat would display the transcription of the conversation in real time, allowing deafblind users to read it at their own pace. Additionally, the chat would allow users to:

Scroll back through the conversation history
Export the transcript for later review
Distinguish between speakers (if available)
Benefits:

This feature would greatly improve the accessibility of video conferencing for deafblind users, allowing them to participate more fully in online meetings and classes. It would also make BigBlueButton a more inclusive platform, setting an example for other video conferencing providers.

Implementation:

This feature could be implemented by adding a new tab to the BigBlueButton interface that displays the closed captioning chat. The chat would be populated with the real-time transcription of the conversation, and users would be able to scroll back through the history, export the transcript, and distinguish between speakers.

Call to action:

We urge BigBlueButton to consider implementing this feature as soon as possible. It would make a significant difference in the lives of deafblind users, and it would help to make BigBlueButton a more inclusive platform.

Syntax error MEDIA_ESL_AUTHENTICATION_ERROR is not defined

I am setting up bbb-transcription-controller. I am using nodeenv. The docker container is running. When I execute
node app.js

I get

2023-01-11T17:59:00.150Z - error: [transcript-manager] FSESL connection authentication error
/home/andrew/bbb/projects/bbb-transcription-controller/lib/esl-wrapper.js:149
this.error = this._normalizeError(MEDIA_ESL_AUTHENTICATION_ERROR, error.message)
^

ReferenceError: MEDIA_ESL_AUTHENTICATION_ERROR is not defined

If a password is involved where should I set it?

Warning no provider set, not transcribing

Hello all,

we tried to implement this in our bbb setup but we are having some issues.
When starting the nodeJS server and starting a conference with voice stream, we get the message "warn: [transcript-manager] No provider set, not transcribing"

Also we disabled gladia, because we don't want our services talking to an external service. Is this supported?

This is our config:

log:
  filename: /var/log/bigbluebutton/bbb-transcription-controller.log
  level: info
  stdout: true
  gladiaProxy: /var/log/bigbluebutton/gladia-proxy.log

redis:
  host: 0.0.0.0
  port: "6379"
  # password: foobared
  publishChannel: 'from-akka-apps-redis-channel'

includePartialResults: true

# If set to false connection to the transcription provider will be kept open from
# the call start to the call hangup. Otherwise it will be opened/closed whenever a
# user start/stops talking. Setting to true might yield lower quality results
disconnectOnSilence: false
# If disconnectOnSilence is true we'll wait this time in seconds
# before closing the connection. If this is too low we might have too much
# connections/disconnections and final results might be cut too short
closeConnectionAfterSeconds: 10

# in mhz
# Valid values are '16' and '8', but 8 seems to be
# too low to get accurate transcriptions
sampleRate: '16'

# Each transcription server has its own config section
# startMessage and endMessage are the messages sent whenever mod_audio_fork's
# connection starts/ends

# servers/server registers transcription server websocket URLs
# I've configured my servers using an nginx reverse proxy for each
# language, see the example below

vosk:
  startMessage: '{"config": { "sample_rate": "" } }'
  endMessage: '{"eof" : 1 }'
  servers:
    en-US: wss://127.0.0.1/voskEN
#    es-ES: wss://HOST/voskES
#    pt-BR: wss://HOST/voskPT
#    fr-FR: wss://HOST/voskFR

# TODO: whispering is still not supported, need to get
# the start/end messages right

gladia:
#  startMessage: '{"x_gladia_key": "", "sample_rate": 0, "bit_depth": 16, "model_type": "fast", "endpointing": 300 }'
#  endMessage: '{}'
#  server: ws://localhost:8777
  proxy:
    enabled: false
#    address: "wss://api.gladia.io/audio/text/audio-transcription"
#    minPartialDuration: 3
#    languageManual: true

#whispering:
#  startMessage: '{"config": { "sample_rate": ""} }'
#  endMessage: '{"eof" : 1 }'
#  server:  wss://HOST/whispering

# Setup freeswitch credentials to make ESL work

freeswitch:
  ip: 0.0.0.0
  port: 8021
  password: REDACTED

And this is our log:

2024-03-20T10:31:34.452Z - info: [transcript-manager] Connected to FreeSWITCH ESL
2024-03-20T10:31:52.475Z - info: [transcript-manager] FS: Associating channel fc99362d-325b-43c1-b324-36dbc87afbc9 2pbp9fvaajpj2dhbh7hc userId: w_hstui65d0oje
2024-03-20T10:31:52.477Z - info: [transcript-manager] Start mod_audio_fork connection fc99362d-325b-43c1-b324-36dbc87afbc9 w_hstui65d0oje
2024-03-20T10:31:52.483Z - warn: [transcript-manager] No provider set, not transcribing
2024-03-20T10:31:52.533Z - info: [transcript-manager] FS: floor changed 96954 10
2024-03-20T10:31:56.546Z - info: [transcript-manager] FS: Start talking fc99362d-325b-43c1-b324-36dbc87afbc9 userId: w_hstui65d0oje
2024-03-20T10:31:59.034Z - info: [transcript-manager] FS: Stop Talking fc99362d-325b-43c1-b324-36dbc87afbc9 userId: w_hstui65d0oje
2024-03-20T10:32:00.399Z - info: [transcript-manager] FS: Start talking fc99362d-325b-43c1-b324-36dbc87afbc9 userId: w_hstui65d0oje
2024-03-20T10:32:04.033Z - info: [transcript-manager] FS: Stop Talking fc99362d-325b-43c1-b324-36dbc87afbc9 userId: w_hstui65d0oje