microsoft / botframework-solutions

Welcome to the Bot Framework Solutions repository which is the home for a set of templates and solutions to help build advanced conversational experiences using Azure Bot Service and Bot Framework. Microsoft Bot Framework is a comprehensive framework for building enterprise-grade conversational AI experiences.

The Virtual Assistant template and samples have been replace by the Enterprise Assistant Template in Bot Framework Composer. All new projects should use this template instead. This repo will be archived and no longer actively supported on March 31, 2022. The existing assets will remain for reference but will no longer be available in the Visual Studio Marketplace or NuGet repository.

Questions regarding the Enterprise Assistant Template and/or migration support can be directed to the BotFramework-Components or BotFramework-Composer repositories.

This repository is the home for a set of templates and solutions to help build conversational experiences using Azure Bot Service and Bot Framework.

Name	Description
Virtual Assistant	Customers and partners have a significant need to deliver a conversational assistant tailored to their brand, personalized to their users, and made available across a broad range of canvases and devices. This brings together all of the supporting components and greatly simplifies the creation of a new bot project including: basic conversational intents, Dispatch integration, QnA Maker, Application Insights and an automated deployment.
Skills (Preview).	Important: Sample Bot Framework Skills have been moved to the Bot Framework Skills repository. Extend your assistant with reusable conversational Skills to add common functionality such as collection of re-usable conversational skills enabling you to add functionality to a Virtual Assistant. Try the sample Skills available like Calendar, Email, To Do, Point of Interest, and more.
Analytics	Gain key insights into your assistant’s health and behavior with the Bot Framework Analytics solutions, which includes: sample Application Insights queries and Power BI dashboards to understand the full breadth of your assistant's conversations with users.

Packages

.NET

Name	Released Package	Daily Build
Microsoft.Bot.Solutions

JS

Name	Released Package	Daily Build
bot-solutions		Link
generator-bot-virtualassistant		Link
botskills (Preview)		Link

Need Help?

If you have any questions please start with Stack Overflow where we're happy to help. Please use this GitHub Repos issue tracking capability to raise issues or feature requests.

Reporting Security Issues

Security issues and bugs should be reported privately, via email, to the Microsoft Security Response Center (MSRC) at [email protected]. You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Further information, including the MSRC PGP key, can be found in the Security TechCenter.

License

Copyright (c) Microsoft Corporation. All rights reserved.

botframework-solutions's Issues

(POI) Convert full street name to abbreviation & vice versa

Azure Maps returns POIs with addresses like "2800 160th Pl NE." When reading aloud and uttered by a user, they will say "2800 160th Place Northeast." The VA should convert the abbreviation to full name and be able to convert back when a user repeats it.

P1: (Automotive) Skill creation

P2: (Email) follow up on device experience

P1: (Email) Proactive new mail from your team

P1: Search/News Skill

P0: Exception handling improvements

Add better exception handling for the different configuration steps (Authentication, updating appsettings with skills, etc.)

PointOfInterestSkill: Parse number entity for navigation

Allow user the option to say things like "direction to option one" based on found locations. Update the LUIS model to support prebuilt number entity and save in state.

P0: Unit tests

• non interactive authentication

P0: (Calendar) Google support

P0: Assistant Handoff - Cortana

Identify initial domains - e.g. weather, general knowledge, flight status, parcel tracking and create a training set (text file) for Dispatcher.

Work with Cortana team on Cloud-Cloud API

PointOfInterestSkill: Text and event-based navigation

A user needs to be able to navigate the POI skill by both UI interaction (events and prompts) and speech (text). Virtual Assistant will show options that user could say out loud to continue conversation.

VA: User Preferences Store for prefered account for skill actions

The Virtual Assistant needs to have through Onboarding / First time use/ and/or Companion App the ability to allow the user to determine which linked account should be used for a determined skill.

Example:

User can chose to store their ToDos in Outlook Tasks / ToDoist / OneNote

User can chose to use work Email in Office365 and Personal Email in GMail

As linked accounts are added, there should be a dialog that verifies the skills that the user wants to leverage with that linked account to allow the user to change accordingly or to leave them alone.

Example:

User adds a GMail account, they are asked if they want to add this as a work or personal email address default.

User adds a O365 account and user is asked if they want to add as default for ToDos / Email / Work Calendar / Contacts / etc.

VA: Use .bot luis config for skill luis config

Use the luis service id in the skill registration to send the luis config to the skill

P1: (Email) Triage experience

P0: Skills to support multiple authentication providers

For each Skill we need a list of authConnections similar to parameters, e.g. MicrosoftAd, Google, Todoist which the skill supports. This should include a type as well as name, e.g. "Name: MyConnection, Type: Google" so we the skill knows how to use a given token (name is arbitary)

When we get a tokenRequest fromn a skill we need to check which of these a user has linked, if there's only one we pass it through otherwise we'll likely need to prompt which account and then pass it through.
​
The GetTokenStatusAsync on the adapter (example in linked accounts) provides a way to check this. When passing the token back to the Skill we'll need to ensure the authConnection type is passed so the skill knows how to get that to work).

Calendar skill has Google capabilities already.

P0: (Email) Google support

some LUIS intent results are not accurate

Description
As designed, the user input "more" should be recognized as "ShowNext" intent. But now it is "Confirm" intent. So users can't see the next page meetings by inputting "more".

To Reproduce
Steps to reproduce the behavior:

1. input "what are my meetings today?"
2. input "no"
3. input "more"

Expected behavior
Bot will show the next page meetings.

(POI) Strategy to handle unmatched entities mid-dialog

The current LU model is not trained on all types of POI entities, so when navigating mid-POI dialog this scenario might take place:
[VA]: Displays 3 POIs (Tully's Coffee, Starbucks, Starbucks) and expects user to ask to route to one if they'd like directions.
[User]: Navigate to Tully's Coffee
[VA]: Dialog depends on LU results to match Tully's Coffee as a keyword entity, but it is never seen before and LUIS has no match.

Currently we are able to navigate with a more basic "Option 1" to get around this, but as the POI skill expands there will be more unique location names that won't be picked up by the skill.

(Contacts) Find A Contact and Take Action

There currently is no skill that will allow a user to find a contact and take action on that Contact.

We would need to add this with the following utterance support:
Find "Contact Name"
Call "Contact Name"
Call "Contact Name" on "Mobile / Home / Cell / Work"
Email "Contact Name" (May already be a function of Email)

P0: (Email) Delete Email

P0: LUIS runtime and BF SDK .NET core Offline Xamarin Wrapper

P0: New adaptive cards for each skill

• Calendar
• Email
• To Do
• Point of interest
• Virtual Assistant

Nice to have: Enable a user to unzip a card "personality pack" into the Virtual Assistant source and have it reflect in the VA's branding. The card assets follow uniform image names & an Adaptive Card host config to enable developers to plug into web chat and show a theme.

P0: (To do) Align fully with provided LUIS model and utilise all entities to avoid prompts

P0: Create automated build to update test environment every day

• Builds
• Tests
• Deployment

P1: Multi-language pre-built domains

• Calendar
• Email
• To Do
• POI

P3: (POI) Multi-stop

User Story

As ...
I want ...
so that ...

Acceptance Criteria

List your requirements.

• ...
• ...
• ...

Resources

Provide any screenshots or conversation mockups to support your request.
Learn how to mock up bot conversations with .chat files.

Notes

Provide any additional information.

P0: (Email) P0: Align fully with provided LUIS model and utilise all entities to avoid prompts

POI Skill Provider model

The PoI skill is likely to have a different data source to use instead of Azure Maps. Investigate a "provider" model enabling plugin of a new data source but use the same skill.

Also Update PoI skill docs to provide guidance

P2: (Calendar) All day meeting refinement

All day meetings are shown as 12:00 AM - 12:00 AM rather than just shown as a "All Day". This should be reflected in any Speak variation too - e.g. you have 2 all day meetings - "Darren's Birthday, etc."

P1: User Profile store

Move User Profile Information (timezone, name, email, news topics, etc.) out of the BF state work. We can likely use the same cosmosdb but a different section

Developer Defined List Type in To Do Skill

Give the developer the ability to add custom list types to the skill and the language model.

P0: Create a Skill Template for developers to quickly get a Skill working the SkillMode capabilities already wired in

The DemoSkill is a simple example of a Skill to get people started. We should take this one step further and create a template enabling people to get a skill working right out of the box using a template and enable them to purely focus on their skill code.

Components:

• Configuration/Parameters
• Authentication
• Events
• Responses/Localization
• API integration
• Skill mode/Local mode (General and Task LU)

VA: LU strategy for skill mode and local mode

Requirements:

• In local mode, Skill should be able to handle task intents (book a meeting, send an email, etc.) as well as general intents (greeting, help, cancel, etc.)
• In skill mode, the dispatch intent should only be triggered by the task intents, while the general intents should be handled normally, or forwarded to the active skill
• Cancel and Help should be handled contextually by the skill

P1: (Calendar) "I'm running late" intent

Check the next meeting and offer to let people know. Detect if online meeting if not offer to add skype/teams so the user can dial in

P0: (To Do) Add outlook tasks support

PointOfInterestSkill: Limit "What's nearby" results to 3

P1: (To Do) Todoist support

P2: (To do) Proactive task triage

P0: (Calendar) Align fully with provided LUIS model and utilise all entities to avoid prompts

P1: Translation

VA: Script LUDown to LUIS on post build step to remove orphaned recipe files

P0: Localization

Test localized responses based on emulator culture & add responses for EFIGS+Chinese to skills and base virtual assistant skills

• Logic to capture locale
• English - .lu and responses
• Mandarin - .lu and responses
• French - .lu and responses
• Italian - .lu and responses
• German - .lu and responses
• Spanish - .lu and responses

Enable deployment of a Chinese simplified deployment and ensure chinese responses are returned when the emulator locale is set to chinese.

[POI] Call POI Event Action

User Story

As ...
I want ...
so that ...

Acceptance Criteria

List your requirements.

• ...
• ...
• ...

Resources

Provide any screenshots or conversation mockups to support your request.
Learn how to mock up bot conversations with .chat files.

Notes

Provide any additional information.

POI Skill today offers Event for Find a Route, however we should also add the Event for Call the POI. This is a common need in many devices that have ability to call.

Adaptive Cards will need to be updated as well as LUIS and Dialog to allow you to do actions as "Navigate to the first one" or "Call the second one"

P1: (Calendar) Proactive "Upcoming meeting"

PointOfInterestSkill: If single results found, offer prompt rather than expect text/event input

If multiple options are found, VA offers card with action buttons to send events and suggested text user can say to select one and move to the next step. If a single option, VA asks users to confirm with a prompt.

VA: Enable deployment of a FIGS or zh-Hans locale VA and ensure localised responses are automatically selected based on activity locale

Enable deployment of a Chinese simplified deployment and ensure chinese responses are returned when the emulator locale is set to chinese.

(PlayMusic) Identify Artist / Song / Album / Genre of Music Play Request

There is currently not a skill that will allow a user to request to play a song / artist / album / or Genre of Music. This is a typically needed use case for many device types.

Solution should allow for a user to state:
Play "Genre" music
Play "Song" by "Artist"
Play "Song"
Play "Album"
Play "Album" by "Artist"

The solution will need to provide disambiguation of multiple and return an Event much like the POI Find skill that would allow the skill to launch with context the music service / app of choice

Additional context
This skill is not to stream the music, but to find the appropriate information to then send to the host to allow music the correct music to be passed to a backend service API.

VA: Publishing to azure (within VS) completes but skill response files aren't found

Description

• Publish VA to Azure App Service
• Bot works - introduction card shows
• Call a skill that relies on JSON files for bot responses and the responses cannot be found.

Skill Error: The given key 'en' was not present in the dictionary. |    at System.Collections.Generic.Dictionary`2.get_Item(TKey key)\r\n   at PointOfInterestSkill.Dialogs.Shared.Resources.PointOfInterestBotResponses.GetBotResponse(String propertyName) in C:\\Users\\darrenj\\source\\repos\\ai\\solutions\\Virtual-Assistant\\src\\csharp\\skills\\pointofinterestskill\\Dialogs\\Shared\\Resources\\PointOfInterestBotResponses.cs:line 118\r\n   at PointOfInterestSkill.Dialogs.Shared.Resources.PointOfInterestBotResponses.get_PointOfInterestErrorMessage() in C:\\Users\\darrenj\\source\\repos\\ai\\solutions\\Virtual-Assistant\\src\\csharp\\skills\\pointofinterestskill\\Dialogs\\Shared\\Resources\\PointOfInterestBotResponses.cs:line 52\r\n   at PointOfInterestSkill.PointOfInterestSkillDialog.CheckIfActiveRouteExists(WaterfallStepContext sc, CancellationToken cancellationToken) in C:\\Users\\darrenj\\source\\repos\\ai\\solutions\\Virtual-Assistant\\src\\csharp\\skills\\pointofinterestskill\\Dialogs\\Shared\\PointOfInterestSkillDialog.cs:line 59\r\n   at Microsoft.Bot.Builder.Dialogs.WaterfallDialog.OnStepAsync(WaterfallStepContext stepContext, CancellationToken cancellationToken)\r\n   at 

P1: (Calendar) Proactive "You're not going to make your meeting"

"Automated Deployment" link broken

Description
"Automated Deployment" link in Feature-Description table in README.md is broken.

To Reproduce
Steps to reproduce the behavior:

1. Go to 'https://github.com/Microsoft/AI/blob/master/solutions/Virtual-Assistant/docs/README.md'
2. Click on 'Automated Deployment'
3. See error

Expected behavior
404 Github page appears, url:
https://github.com/Microsoft/AI/blob/master/solutions/Virtual-Assistant/docs/,/virtualassistant-createvirtualassistant.md

Additional context
There is probably an extra "," in the file path; https://github.com/Microsoft/AI/blob/master/solutions/Virtual-Assistant/docs/virtualassistant-createvirtualassistant.md works :)