nvidia-omniverse / kit-extension-sample-airoomgenerator Goto Github PK

Prompt is: This is the reception room where we welcome the customers. Fill it with modern furniture.

If I use the free version of ChatGPT (GPT-3.5) it keeps stuck in producing.

Also, I don't think if this is the problem on ChatGPT side since the ChatGPT response is produced however I assume the "grey boxing" part is stuck. I may be wrong.

Please note that Grey Boxing works fine if the prompt is empty.

Any tips?

I assume the missing ngsearch dependency in the screenshot is the deep search module available for enterprise customers? The docs indicate you can turn off the deep search checkbox, allowing others to give it a try, but as a non-enterprise customer I cannot install it. (I am going to work around it by cloning the repo and removing the dependency - just noting in case you thought worth looking at.)

I have an enterprise version of omniverse with deepsearch and I want to use the paid GPT-3.5 Turbo we have access to through our company's paid OpenAI key. However, I am not sure how I can add that to use it? Currently, you only allow for key directly from OpenAI. Any chance you could help with this inquiry?

@StrainFlow @mario-nvidia
Thanks a lot.

P.S. Expected error but just wanted you to see the error message if I enter Enterprise Azure OpenAI key:

MYKEYVALUE. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}

I use the extension but it threw some errors in the console:
https://global.discourse-cdn.com/nvidia/optimized/3X/e/d/ed344a9dd8da4f0f90ee0497eec833b3d13203c0_2_1380x604.jpeg

Items generated by ai will have rotation (0, -90, -90), may of them look good, but some will have wrong orientation.
For example:

The ChatGPT return:

{
    "area_name": "Reception_Area",
    "X": 0.0,
    "Y": 0.0,
    "Z": 0.0,
    "area_size_X": 1055.422119140625,
    "area_size_Z": 1136.84423828125,
    "area_objects_list": [
        {
            "object_name": "Coffee_Table_1",
            "X": -202.7110595703125,
            "Y": 0.0,
            "Z": 315.69482421875, 
            "Length": 203.2,
            "Width": 93.04,
            "Height": 62.23,
            "Material": "Mahogany"
        }
    ]
}

In this extension, DeepSearch maybe found ('Coffee_Table_1', '/NVIDIA/Assets/Extensions/Samples/ActionGraph/samples/Desk/Desk.usda'),
then I will get a fallen desk with orientation(0, -90, -90).
But if I search in DeepSearch panel, and then drag the desk.usda to 3d scene panel, I will get a upright desk with orientation (0, 0, 0).

It is not very clear to me how I can get to the point that I can get to the extension inside the code. I ran the following commands and the app folder was created however not sure how to get to the GUI. Further, I was not able to find the extension inside the Omniverse Code Extension search.

Could you please tell how I can get to the window GUI that shows the "Content Generation with ChatGPT"?

https://www.youtube.com/live/4H9fJrcWK1c?app=desktop&feature=share

(base) mona@ard-gpu-01:~/3D_generational_ai/kit-extension-sample-airoomgenerator$ ./link_app.sh
bash: ./link_app.sh: Permission denied
(base) mona@ard-gpu-01:~/3D_generational_ai/kit-extension-sample-airoomgenerator$ chmod +x ./link_app.sh
(base) mona@ard-gpu-01:~/3D_generational_ai/kit-extension-sample-airoomgenerator$ ./link_app.sh
./link_app.sh: line 8: /home/mona/3D_generational_ai/kit-extension-sample-airoomgenerator/tools/packman/python.sh: Permission denied
(base) mona@ard-gpu-01:~/3D_generational_ai/kit-extension-sample-airoomgenerator$ chmod +x /home/mona/3D_generational_ai/kit-extension-sample-airoomgenerator/tools/packman/python.sh
(base) mona@ard-gpu-01:~/3D_generational_ai/kit-extension-sample-airoomgenerator$ ./link_app.sh
Adjusting /home/mona/.profile
Adjusting /home/mona/.bashrc
Creating packman packages repository at /home/mona/packman-repo
Fetching [email protected]_64.tar.gz from bootstrap.packman.nvidia.com ...
Unpacking python
Fetching [email protected] from bootstrap.packman.nvidia.com ...
Unpacking ...
Processing project file '/home/mona/packman-repo/packman-common/6.33.2/deps.packman.xml'
Package '7za' at version '16.02.4' is missing from local storage.
Downloading from http://bootstrap.packman.nvidia.com/7za%4016.02.4.zip (4.33 MiB)
100.00% (speed 17.9 MiB/s)                 
Total of 0.24 seconds
Unzipping: [email protected] (9.87 MiB)
100.00% (speed 101 MiB/s)                  
Total of 0.10 seconds
Package successfully installed to /home/mona/packman-repo/chk/7za/16.02.4
Path is not specified, looking for Omniverse Apps...

Found following Omniverse Apps:
0: Isaac Sim (isaac_sim) at: '/home/mona/.local/share/ov/pkg/isaac_sim-2022.2.0'
1: Nucleus Workstation (nucleus-workstation) at: '/home/mona/.local/share/ov/pkg/nucleus-workstation-2022.4.1'
2: Code (code) at: '/home/mona/.local/share/ov/pkg/code-2022.3.3'
3: Create (create) at: '/home/mona/.local/share/ov/pkg/create-2022.3.1'

Selected app: code
Creating a link '/home/mona/3D_generational_ai/kit-extension-sample-airoomgenerator/tools/scripts/../../app' -> '/home/mona/.local/share/ov/pkg/code-2022.3.3'
Success!

Also, when I tried the following command, I got error:

(base) mona@ard-gpu-01:~/3D_generational_ai/kit-extension-sample-airoomgenerator$ ./link_app.sh  --path '/home/mona/.local/share/ov/pkg/create-2022.3.1'
Creating a link '/home/mona/3D_generational_ai/kit-extension-sample-airoomgenerator/tools/scripts/../../app' -> '/home/mona/.local/share/ov/pkg/create-2022.3.1'
packman(WARNING): Path '/home/mona/3D_generational_ai/kit-extension-sample-airoomgenerator/tools/scripts/../../app' exists but is incorrect. Removing ...
Success!

I tried to follow along the following

However, it's rather hard to follow. I am not sure where to enter them. Would you please be able to show a video how you run your extension?

I also added the link you have in your README.md in the Extensions --> Extension Search Paths and I am not sure what I should expect from it here or how to start using the extension.

Further, when I click on Open below, it gives read error while the file is readable.

(base) mona@ard-gpu-01:~$ ls  /home/mona/.cache/ov/gitexts/BRMORqhbNYcUJw/exts/
total 16K
drwxrwxr-x 7 mona mona 4.0K Apr 10 14:49 omni.sample.deepsearchpicker
drwxrwxr-x 7 mona mona 4.0K Apr 10 14:49 omni.example.airoomgenerator
drwxrwxr-x 6 mona mona 4.0K Apr 10 14:49 ..
drwxrwxrwx 4 mona mona 4.0K Apr 10 14:49 .

When trying to import this extension into CODE 2022.3.3 I get the following error message:

How can I resolve this error?

I am having a use case to utilize spatial awareness of GPT 4.

I try to send the new OpenAI request with the previous history of generated items to spawn a new asset on an existing asset

e.g. Place a cup on there coffee table

GPT4 can successfully send coordinates to place the asset on top of it and DeepSearch can spawn it on the coffee table. However, the entire assets set disappears and gets re-spawned every time we call a new prompt.

What is the best way to spawn only the new item without resetting the whole asset set?

Automatic behavior with startup, can you explain?

When hitting the "Generate" button in

I get the following error messages in the Composer Console:

2023-09-19 06:33:20 [Error] [omni.ui.python] ArgumentError: Python argument types in
2023-09-19 06:33:20 [Error] [omni.ui.python] Stage.GetPrimAtPath(Stage, NoneType)
2023-09-19 06:33:20 [Error] [omni.ui.python] did not match C++ signature:
2023-09-19 06:33:20 [Error] [omni.ui.python] GetPrimAtPath(class pxrInternal_v0_22__pxrReserved__::UsdStage {lvalue}, class pxrInternal_v0_22__pxrReserved__::SdfPath path)
2023-09-19 06:33:20 [Error] [omni.ui.python]
2023-09-19 06:33:20 [Error] [omni.ui.python] At:
2023-09-19 06:33:20 [Error] [omni.ui.python] c:/work/ov_exts/omni.example.airoomgenerator/omni/example/airoomgenerator/window.py(212): get_prim
2023-09-19 06:33:20 [Error] [omni.ui.python] c:/work/ov_exts/omni.example.airoomgenerator/omni/example/airoomgenerator/window.py(157): _generate
2023-09-19 06:33:20 [Error] [omni.ui.python] c:/work/ov_exts/omni.example.airoomgenerator/omni/example/airoomgenerator/window.py(120):
2023-09-19 06:33:20 [Error] [omni.ui.python]

Any idea what went wrong? Do you need any other information from my machine?

When I assign and create a area, enter the prompt and click Generate, the tool gets stuck on Processing for hours.

Using USD Composer 2023.2.5

Hello,

Would you please make a step-by-step video on how to install it... I believe people are interested in trying and using this tool but stuck on installation. I can't seem to find any more information on installation other than the README. But it's not detailed enough for some new users to even get started....

Thanks,