microsoft / immersive-reader-sdk

SDK for using the Immersive Reader through Cognitive Services

Azure Applied AI Immersive Reader - Immersive Reader SDK

The Immersive Reader SDK is a set of libraries that allow you to easily and quickly integrate the Immersive Reader into your application.

Usage

• JavaScript
• Usage

Contributing

We welcome contributions to this project.

• Submit bugs and help us verify fixes as they are checked in.
• Submit and review source code changes.
• Join the discussion on StackOverflow and Twitter.

Please submit pull requests to the dev branch.

Reporting Security Issues

Security issues and bugs should be reported privately, via email, to the Microsoft Security Response Center (MSRC) at [email protected]. You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Further information, including the MSRC PGP key, can be found in the Security TechCenter.

License

Copyright (c) Microsoft Corporation. All rights reserved.

Licensed under the MIT License.

immersive-reader-sdk's Issues

Close panel button should be a minimum of 24x24 px relative to the viewport resolution on mobile devices

Need to make this a bit larger on smaller screens.

On some smaller screens the flex panel left and right insets are too wide

Need to adjust the padding here for small screens.

Play/Pause button should not require extra space for apps that suppress the app bar footer in Andorid and iOS Native Apps

There's a known issue on iOS where the app bar footer occludes content of web applications and is not factored into the height of the viewport. Apple has resolved this 'As designed'. As a workaround, the Immersive Reader applies extra padding to the bottom of the play/pause button. Native apps do not require this space.

There's an SDK option 'useWebview' tha may be used to set layout porperties of the web app. The Immersive Reader web app requires a few changes to ensure this boolean removes the extra space.

There's also extra space in the menu options panel that should be removed in native apps:

Unauthorized

I'm trying to run a sample
but it always returns
message: "Unauthorized"
in this request
"
Request URL: https://learningtools.onenote.com/learningtoolsapi/cognitive/v2.0/GetContentModelForReader
Request Method: POST
Status Code: 401

Accept: application/json, text/javascript, /; q=0.01
Authorization: CogSvcsAccessToken {Token}
Content-Type: application/json
MS-Int-AppId: Cognitive
"

Add option to turn off the exit button

Testing

Test issue, please ignore.

readaloudoptions not working

Configuring readAloudOptions on launch does not have any effect.

Have tried variations of this (e.g. autoplay -> autoPlay, male -> Male without success. The reader opens and requires manual starting and configuration.

Browsers tested Chrome, Edge.

This is the JS configuration:

const options = {
        "readAloudOptions": {
            "autoplay": true,
            "voice": 'male'
        },
        "onExit": exitCallback,
        "uiZIndex": 1000000
    };

Line breaks is stripped from the content for chrome webview

Line breaks from example "p" and "br" tags is stripped when using a webview on chrome apps.
Using the same code for both our chrome and electron app (except for using webview on chrome) results in different results.

Correct format using an iframe:

Incorrect format using a chrome webview:

Quality drop on every new sentence

Hi,

I integrated the immersive reader recently in the UWP app and my manager who lives in Portugal tried it. Another colleague in Portugal has the same issue. I cannot repro on my side. As a reference, the Immersive Reader is a black box and it is a website hosted by Microsoft essentially.

Problem: When the reader arrives at the end of the sentence and starts a new sentence, the quality drops. You can see the word “Other” and “These” in the video.
https://user-images.githubusercontent.com/1608424/115884669-282d5b00-a41d-11eb-9448-9ef27f204dc8.mp4. It does that for every sentence. I even created the Azure resource in Europe along with East US, but no change.

Scrollbar disappears after closing Immersive Reader

Publish immersive-reader-sdk on npm

Any chance that you can publish the built SDK on npm when releasing new versions? It would help for dependency management of the SDK when bundling it into an Electron app.

Some Fluent UI Styles component background colors are hidden in Android apps

Need to adjust the styles to work in specific user agents.

Let user override styles of the iframe container. [Internal SDK v.0.5.5]

The iframe container currently has a position: relative. This causes an issue when the iframe container is appended to a parent element other than the document's body. I would suggest letting the consumer of the SDK override the styles of the iframe container.

iframe src domain not set correctly

When not using a custom domain, the default domain is not set.
Currently options.customDomain is checked against null, it should instead be a truthy check since customDomain is undefined if not set.

Sudden TokenExpired message on using SDK

I have a project that was using the SDK with Azure AD Application and IR resource. It was working for several months until recently where now every request opens the frame for IR but now returns:

I changed some of the settings for subdomain, Secrets etc as a test and when changed the IR times out completely which leaves me to believe the settings I have are correct. The code to get the token is successful and using console.log I can see both it and the subdomain are ok before ImmersiveReader.LaunchAsync fires.

In order to check it wasn't me, my machine, my code etc I grabbed another dev example online: https://github.com/kasuken/ImmersiveReaderCore . I tested with the settings as is first and IR opened ok. I changed the settings to my own TenantID etc and the error returned. I have raised a support ticket with Azure as it was working before and feels like an account issue but if anyone can help I would appreciate it as its used in an application for specialist students.

Immersive reader does not reset state on errors

Steps to reproduce:

1. Open up the simple sample for Immersive Reader while online
2. Go offline (either for real or toggle "Offline" in dev tools)
3. Click on a button to open the Immersive Reader

Expected outcome: Immersive reader should not open up (or close again when the page fails to load)

Actual outcome: Immersive reader opens up full screen but fail to load, user ends up with a white page.

I think that this could be solved by calling the reset() function instead of resetTimeout() when ImmersiveReader-TokenExpired, ImmersiveReader-Throttled or ErrorCode.Timeout occurs in launcAsync.

Another note is that this problem becomes a bit harder to work around as a developer due to lack of classes and/or id on the iframeContainer element since it's error prone to programmatically access it today.

UWP sample or UWP SDK is missing

In the provided samples for the Immersive Reader, only iOS has a SDK. It would be nice if UWP gets some love here.

Thanks

Support for setting default display options

Python documentation

Can you please add python documentation for Immersive Reader, specifically a Flask example would be very helpful.

Play/pause button trails offscreen in iOS Safari

When using iOS Safari, the Play/Pause button get pushed offscreen when there are multiple tabs open.

with tabs:

without tabs:

What is the latest npm version

Hello the latest version on Github is 1.1.0 but on npm a yarn add @microsoft/immersive-reader-sdk adds version 0.5.2 that was released few days ago.

I am confused with those recent versions

source : https://www.npmjs.com/package/@microsoft/immersive-reader-sdk?activeTab=readme

Sorry, we couldn't get the right information to show the text correctly

I'm getting this error when I try to use the sdk.

Sorry, we couldn't get the right information to show the text correctly.

I checked the network debugger and everything is 200OK except for this:

Any ideas on how to fix?

Doesn't work with Chrome apps

Move subscription keys out of code in samples

Content button label selection includes line height in mobile apps

When users select thee button labels of the content the line height of the space between the buttons is greater due to the em spacing. Need to adjust the UA selection styles to match the height of the buttons.

Add declaration file(s)

Either here or in DefinitelyTyped

Sorry, we couldn't get the right information to show the text correctly.

Getting this error on every w=browser, used Webview option as well but no help. Used default Quickstart C# program from repo. Used susbcription Key from Azure

why there is no immersive reader in linux? at least make it SaaS

hey there i really love immersive reader when i was using Windows. it is amazing. it just perfect. but now i am using Linux due to system performance issue. i have Azure Account and why don't just charge me when i want use Immersive Reader in linux using AAD auth. i am really dissapointed by this decision. this technology should accessible to anyone, even for people "tech savvy" because not all people have time to build their own Immersive Reader. i try to build Chromium Extentions using microsoft account log in. i wish i can find a way to make this tech more accessible to many people. please help me.

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Browser default language - translation function does not work

When the browser default language is Chinese, the translation function cannot translate English to Chinese. Cases that cannot be translated:
(1) When the browser default language is Chinese, the text content is "How are you", and then choose to translate into Chinese.
(2) When the browser default language is English, the text content is "你好嗎", and then choose to translate into English.

SDK version: 1.1.0
Browser version: Chrome 86.0.4240.198

How to use immersive reader in uwp app?

I see that immersive reader is essentially a JavaScript based control. I'm trying to figure out how to add this into a c# xaml uwp app. Any ideas? I'd be happy to spearhead this, but I'd like to get some pointers since I'm not sure where to start

Microsoft branding should be hidden in Microsoft apps

Some Microsoft applications use the Immersive Reader SDK, there's no need to show the logo in these apps.

Provide types for `onPreferencesChanged` string

In the handler for onPreferencesChanged, it would be great to be able to parse the string using known types provided by the SDK. Today, the VSCode for Education Extension uses manually created types based on the preferences string we received at the time. If they change at any point, the current method of fixing it would be to once again manually take a look at the string rather than updating a package to obtain the new types.

At minimum, the documentation, could define what's expected in the onPreferencesChanged string.

Some of the types returned in the prefs string were documented, but not all. Below is an example.

export type TranslationLanguageData = {
    tlc: string;
    slc: string;
};

export type TranslationLanguage = {
    key: string;
    text: string;
    data: TranslationLanguageData;
};

See https://github.com/microsoft/vscode-edu-extension/pull/1532/files for context. If you need access to the repository, please comment or message offline.

Can't reopen Immersive Reader without refreshing if user exits early

We just added Immersive Reader to the MakeCode Arcade tutorials, and we've noticed that if an error occurs between launching the Immersive Reader and displaying the content, and the user presses the back button, they won't be able to reopen it until they refresh (with an 'Immersive Reader is already launching' error).

For example, if a user loses internet connection while launching, they won't be able to launch again.

I did a local build of the SDK where I added isLoading = false to the exit function --

immersive-reader-sdk/js/src/launchAsync.ts

Line 134 in 60aaec3

reset();

, and it seems to fix it for our case! But let me know if we're calling the SDK incorrectly and this isn't a bug 😊 I am also happy to open a PR to contribute!

Safari webview on iOS is not populated correctly

When starting Immersive Reader through the javascript sdk on a iOS Safari webview it displays a blank page.
Looking inside the iframe added to the body, the div containing the react app is not populated:

When testing on the electron version of our app using the same code, the div is correctly populated:

Add option to turn off Grammar and Translation buttons

In a Learning Management System, some courses are about learning a different language.
In such cases reading an assignment in the original language with ImmersiveReader can be very helpful, but a teacher would not want their students to see the correct translation.

gotOS() returns undefined on iOS 12 and earlier with overridden user-agent

Just like mentioned in #86 in a comment from @robinwassen, the getOS() function returns undefined when using a custom user-agent.

It was fixed for iOS 13 but it still does not work on iOS 12 and earlier.

onPreferencesChanged does not support shouldTranslateDocument

Hi, if a user selects 'translate document', I want to remember that setting and lauch each new instance of ImmersiveReader with autoEnableDocumentTranslation = true until the user switches this off again.

Unfortunately, the onPreferencesChanged callback is not called when changing the 'translate document' button.
And if the onPreferencesChanged callback is called for something else, the translationState does not contain a flag for e.g. shouldTranslateDocument.

Is this a bug, and if so, is a fix on the roadmap?

Tested with immersive-reader-sdk.js version 1.1.0.

Break/pause between line

Hello,
In "old" documentation, i think i read i'ts possible to add some html mark to break/pause the reader ? Actually, i don't find way to stop reader after each line.

Kind regards,

Change Voice Engine || Open Source Immersive Reaader?

Hi there,
Love the project. I would like to request to enable OpenAI's TTS for the voice instead of MSFT's model. Ideally, the voice method would take an openAI key parameter and then allow the developer to use that model and the voice options within.

I'm not sure how much support Immersive Reader is getting right now but if you open-source it then it may take off as engineers are working with text AI more than ever before.

Cheers,
Jeremiah

Fullscreen button doesn't work when using Immersive Reader inside Chrome app

Unable to create Immersive Reader with AAD configuration according to PowerShell script

This resource https://docs.microsoft.com/en-us/azure/applied-ai-services/immersive-reader/how-to-create-immersive-reader documents a PowerShell script that would create the resources and configure them as need be. However, I am encountering troubles trying to execute the script, such as the following:

SecretExpiration '2022-12-31'                        
Exception: 
Line |
  17 |          throw "Error: AADAppIdentifierUri must be a valid URI"
     |          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Error: AADAppIdentifierUri must be a valid URI

Is there any advice or fix for this?

Add preview label to readme

Doesn't work correctly

I can't see the immersive reader. Only this message."The appropriate information could not be obtained to display the text correctly."

Chrome app compatibility mode loads blank iframe

We have immersive-reader into our app which is running as a Chrome app, as mentioned in #8 we are launching with the option useWebview: true. But end up with a blank iframe inside the WebView (from a user perspective the app becomes white).

Code used to launch Immersive Reader:

const content = {
    title: self.question.title,
    chunks: [ {
        content: helperElement.innerText,
    } ],
};

const options = {
    useWebview: true,
}

ImmersiveReader.launchAsync(accessToken, content, options);

Result:

In our Electron app immersive reader works with useWebview: false, if useWebview: true is set in the Electron app it produces the exact same result.

I know that useWebview option is designed for Chrome though.

Need a "Stop" or "Pause"\"Resume" API from IR team. Without this, we can't pause or resume audio talkback.

Scenario : Voiceover Audio does not stop even when we go away from the Android app(using IR web sdk)
Repro Steps

Install App
launch app
capture/select any image which has text.
Done
open immersive reader in save targets screen
play audio
Press Home button and switch to a different app.

Expected

Voiceover Audio should stop.

Actual.:
Audio did not stop if we go to different app

Do you have an example in web forms?

Read aloud stops at certain html content

Hi, reading aloud text works great, for both plain text and html chunks, except in one case.
When I launch with a chunk with mimeType: 'text/html' and contents a shown below:

• read aloud proceeds until the last paragraph, and there it stops
• if I click on text in that last paragraph (starting with 'Romeo and Juliet has been adapted') I see a script error

Removing the <p> tags from the content avoids the problem, but I would rather not do that.
Can anyone confirm that this is a bug in the SDK ?

Sample html chunk

<p style="margin-bottom: 10px;">
	<span style="font-size: 12px;">Romeo and Juliet is a tragedy written by William Shakespeare early in his career about two young star-crossed lovers whose deaths ultimately reconcile their feuding families. It was among Shakespeare's most popular plays during his lifetime and along with Hamlet, is one of his most frequently performed plays. Today, the title characters are regarded as archetypal young lovers.</span>
</p>
<p style="margin-bottom: 10px;">
	<span style="font-size: 12px;">Romeo and Juliet belongs to a tradition of tragic romances stretching back to antiquity. The plot is based on an Italian tale translated into verse as The Tragical History of Romeus and Juliet by Arthur Brooke in 1562 and retold in prose in Palace of Pleasure by William Painter in 1567. Shakespeare borrowed heavily from both but expanded the plot by developing a number of supporting characters, particularly Mercutio and Paris. Believed to have been written between 1591 and 1595, the play was first published in a quarto version in 1597. The text of the first quarto version was of poor quality, however, and later editions corrected the text to conform more closely with Shakespeare's original.</span>
</p>
<p style="margin-bottom: 10px;">
	<span style="font-size: 12px;">Shakespeare's use of his poetic dramatic structure (especially effects such as switching between comedy and tragedy to heighten tension, his expansion of minor characters, and his use of sub-plots to embellish the story) has been praised as an early sign of his dramatic skill. The play ascribes different poetic forms to different characters, sometimes changing the form as the character develops. Romeo, for example, grows more adept at the sonnet over the course of the play.</span>
</p>
<p style="margin-bottom: 10px;">
	<span style="font-size: 12px;">Romeo and Juliet has been adapted numerous times for stage, film, musical, and opera venues. During the English Restoration, it was revived and heavily revised by William Davenant. David Garrick's 18th-century version also modified several scenes, removing material then considered indecent, and Georg Benda's Romeo und Julie omitted much of the action and added a happy ending. Performances in the 19th century, including Charlotte Cushman's, restored the original text and focused on greater realism. John Gielgud's 1935 version kept very close to Shakespeare's text and used Elizabethan costumes and staging to enhance the drama. In the 20th and into the 21st century, the play has been adapted in versions as diverse as George Cukor's 1936 film Romeo and Juliet, Franco Zeffirelli's 1968 version Romeo and Juliet, and Baz Luhrmann's 1996 MTV-inspired Romeo + Juliet.</span>
</p>

Script error

react-dom.production.min.js:101 Uncaught TypeError: Cannot read properties of undefined (reading 'height')
    at t.scrollSelectedTokenIntoView (virtualized-content-pane.tsx:1856:58)
    at t.setSelectedExpressionForTts (virtualized-content-pane.tsx:1639:18)
    at t.setCurrentlyReadToken (virtualized-content-pane.tsx:1762:18)
    at t.selectToken (virtualized-content-pane.tsx:1719:14)
    at n.onClick (virtualized-content-pane.tsx:730:19)
    at Object.We (react-dom.production.min.js:52:317)
    at Je (react-dom.production.min.js:52:471)
    at react-dom.production.min.js:53:35
    at _o (react-dom.production.min.js:100:68)
    at Do (react-dom.production.min.js:101:380)

Tested with https://ircdname.azureedge.net/immersivereadersdk/immersive-reader-sdk.1.3.0.js

BUG: "self is not defined" error when using npm package, but not with global UMD one (<script>)

As soon as an element with the launchAsync function on it renders, I see this error message. Doesn't matter if I don't click, it happens immediately on render.

Using the script tag version works fine.

Overlay doesnt close on route change

Hi,

we use Immersive Reader on a few places in our project with React and React router. Theres a confusing behaviour of the Immersive Reader overlay. If the user goes back a page etc with the browser controls the Immersive Reader overlay is still showing. Can I fix this? Is this expected behaviour on your end?