itp-xstory / uniflow

Full Docs -->

Dialogflow V2: Unity Plugin Proposal

• Project Description
• Team Members
• Define the problem
• Address Greater Landscape
• Deliverables
• Timeline
• Documentation
• Accessibility
• Mentoring
• More about team
  • Ilana Pecis Bonder
  • Alice Sun
  • Lin Zhang
• References

Project Description

The Dialogflow V2 Unity Plugin is a continuation from Api.ai: Unity Plugin, which made it easy to integrate the Dialogflow(Api.ai) natural language processing API into your Unity project.

Team Members

• Ilana Pecis Bonder
• Alice Sun
• Lin Zhang

Define the problem

• Api.ai: Unity Plugin is no longer working in Dialogflow V2, due to the difference in the way the basic system works (Dialogflow API v2 versus v1).
• Dialogflow API V1 will be shut down in one year – on October 23rd, 2019.
• Our goal will be figuring out ways to build similar plugin that works in Dialogflow V2 system.

Address Greater Landscape

Opening new possibilities to involve voice-based conversational interaction in VR and AR environment.

Deliverables

• Feature: listening and sending voice data from Unity to the Dialogflow V2 service for recognition and processing
• Doc: show and explain how to use it (including STT process, if required)

Timeline

This project will be completed over 5 weeks (Nov 6 - Dec 11). Describe a timeline in detail below.

Week 1

• Defining project schedule and make proposal presentation
• Reaching out mentors
• Meeting with Nicole @1PM, Nov 8
• Dialogflow V2 Unity Plugin Hackathon Nov 11

Week 2

• Analyzing the authentication process in Dialogflow V2
  • Authentication via Google Cloud Service Account JSON key
  • It will give 401 error without adding the generated “client_email” in JSON file into the Dialogflow project authentication panel:

• Adding the “client_email” to "INVITE NEW PEOPLE" will give access to the Dialogflow project:

• JSON Web Token (JWT) for Google Clout Platform in Unity
• How can we automate this process?

Week 3

• Audio Request Support: Passing audio file directly via V2 API call

• Using Audio Source in Unity to capture audio from a microphone

Week 4

• Testing with a text call "hello", which received fulfillment response of "Hiiiii Unity~~~" from "Default Welcome Intent" in Dialogflow

Week 5

• Using Audio Source instead of text query
• Creating example!

Documentation

• Initial documentation including basic intro, proposal and brief process will be updated in README.md, while instructional contents will exist as a GitHub Page.

Accessibility

All documentation will be done with considerations of the W3C Web Content Accessibility Guide.

Mentoring

• Nicole He: Dialogflow Mentor
• Igal Nassima: Unity Mentor

More about team

Ilana Pecis Bonder

• Creative Technologist and UI/UX Specialist
• Interactive Telecommunications Program (ITP)
• Portfolio

Alice Sun

• Graphic & interaction designer
• Interactive Telecommunications Program (ITP)
• Blog

Lin Zhang

• Front-end developer
• Interactive Telecommunications Program (ITP)
• Blog

References

• Api.ai: Unity Plugin
• Dialogflow API v2 versus v1

uniflow's Issues

HTTP/1.1 400 Bad Request Error when executing PostRequest()

Hello,

The error comes after the step 5 of the tutorial uniFlow: Dialogflow V2 Unity Plugin Prototype : https://itp-xstory.github.io/uniFlow/#step5

I changed the post request routine as following : PostRequest("https://dialogflow.googleapis.com/v2/projects/my_project_name/agent/sessions/34563:detectIntent", my_token_from_previous_steps)); and also in the bottom as described in the video. But, when running the example in unity i get the following 03 outputs :
################################################
{
"queryInput": {
"audioConfig": {
"audioEncoding": 0,
"sampleRateHertz": 0,
"languageCode": "",
"phraseHints": []
}
},
"inputAudio": ""
}
UnityEngine.Debug:Log(Object)
d__2:MoveNext() (at Assets/NewScript.cs:38)
UnityEngine.MonoBehaviour:StartCoroutine(IEnumerator)
NewScript:Start() (at Assets/NewScript.cs:18)
################################################
400
UnityEngine.Debug:Log(Object)
d__2:MoveNext() (at Assets/NewScript.cs:51)
UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)
################################################

HTTP/1.1 400 Bad Request
UnityEngine.Debug:Log(Object)
d__2:MoveNext() (at Assets/NewScript.cs:52)
UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)
################################################
The mentionned lines from the NewScript.cs script are :
18 : calling the PostRequest(
38 : Debug.Log(jsonRequestBody);
51 : Debug.Log(postRequest.responseCode);
52 : Debug.Log(postRequest.error);

I'm using unity 2019.3.7f1 in Windows10-x64, and i followed the tutorial step by step, could you help please ?

Thank you,

Push samples

• Push samples

iOS/Android mobile

Does this work with iOS/Android mobile - i notice that the credentials has to be set in environment path

Broken links

Please fix the Doc's link.

the new one should be the following.
https://itp-xstory.github.io/uniFlow/

A Problem

Hi Alice and your teammates! Thank you for sharing such lucid tutorial videos. I'm a designer and I'm doing a project related to audio interaction based on Unity. I have followed your steps, but a problem showed up when I was adding the "NewScript" to the camera, which is an AR camera based on Vuforia. An error "400 Bad Request" was requested in the console. I have checked some answers on StackFlow, but most of those solutions look like javascript rather than CSharp. I wonder if you have ever met the same problem and probably you may share your solutions.

Refresh Access Token

Hey, Thanks for such a detailed DiaglogFlowV2 pipeline. Earlier I was using V1 and it involved a very simple authentication process. But with V2 the authentication has become crazy difficult. Moreover, the Access Token generated gets expired in one hour. I tried searching for this but didn't find a solution. I can see people have solved it here, but I am not sure how to integrate same here.
Please Help!