yanchxx / mopa Goto Github PK

I tried to use it with simple-romp but it did not work.
I already created an issue at the ROMP repository and described my problem here and here in detail.

The author of the ROMP repository answered there:

About live blender driving, please refer to this repo. https://github.com/yanch2116/CharacterDriven-BlenderAddon My colleague is responsible for maintaining this funciton now. Best regard.

and closed the issue.

@yanch2116 So how to solve it?

My goal is to send the positions and quaternions of ROMP (or any other SMPL based solution) over the VMC protocol to other application (not Blender).
Any ideas, whats the best way to do it?

it now parallel with ground?

where did u download the background?

thx

I am a beginner. I want to know about the part of 3D bone deshaking, where can I see it?

Hello, Thank you for your great job. I am not familiar with blender. Can you help me? I can not find the Panel. Thank you.

之前问过一个问题，为什么mixamo 无法被直接驱动，后来通过改骨骼可以驱动了。

但是现在有个问题阿，我想驱动更多的模型，大部分都是右边的形式，不可能每个模型都取修改骨骼，有些地方也不能修改iafbx，例如UE里面。

此时，要怎么通过修改 输出的Pose数据，来是配更通用的骨骼?

Hello. I have successfully run the demo of ROMP, which exported .fbx file. And currently I want to use the .fbx to control the character.
Can you provide steps for the video demo? I can only see the camera one

Also, I would like to know if there is a way to replace the 3D Character before I run the script and process the video so that I don't have to manually change it every time?

大佬您好，看了你这个项目很受震惊，因为让一个动漫角色跳舞，是我一直想做但是没有实现思路的地方。
我没有用过Blender，对她不熟悉，所以有个问题想问一下，假设我有一个动漫角色了，比如龙珠里面的孙悟空，obj或者其他的3D格式，然后我想用B站或者油管的跳舞视频进行驱动，让孙悟空跳舞，得到一个GIF或者MP4文件
因为我不用Blender，所以不能使用您的插件，请问实现上述目标的代码在哪里，可以参考一下吗
请原谅我幼稚的问题，期待您的解惑！

Thanks for your wonderful work! I follow your instruction and run the beta.blender in Blender. But I don't run server.py yet.

首先感谢您分享这个插件，有个问题希望能帮忙解决
我在运行插件时报错，按space没有产生动作，无法解决，请问是什么原因，谢谢

传送是有跑完的

Blender:2.93

I found that the avatar will shake drastically when running the webcam demo. And I set the mode =1 to insert keyframe record the webcam results for playback.

Now, if we set one more condition for inserting keyframe, only the frame_idx % 3 == 0 or frame_idx % 5 == 0. This would allow the avatar to move along these keyframes much smoother.

However, is there a way to let the webcam demo runs in real-time using the keyframe strategy I said? This skip keyframe strategy only seems to work with the recorded playback. The character is still moving on each frame when we are actually running in real-time.

currently seems the hips postion always to zero, how to make it move around origin point then it can looks more natural?

您好！
当我导入自己的已经绑定好骨骼的标准的smpl模型，并点击fix bones时出现了下面的报错：

请问这是什么原因呢？
感谢！

Hello again, and I noticed, in the chat room, there are two characters, which part needs to modify to make it happen, may I ask 👍

你好，
非常感觉开发这个插件，我想请教几个问题，

1. 我注意到你把romp的输出里面的"cam_trans"直接当成世界坐标的位移信息去驱动blender中的人，虽然我也能够用仿照你这样做去给ue中的人物赋予位移信息，但是我并不了解其中的原理。如果想得到真实坐标的物理信息，不是要用到相机内参外参、图像像素点的位移，然后才能去估算世界坐标的位移吗？
2. 在把网络输出"cam"转成相机外参中的位移信息时，我看其他人的做法是

    cam_s, cam_tx, cam_ty = pred_cam
    trans = [cam_tx, cam_ty, 2*FOCAL_LENGTH/(CROP_SIZE*cam_s + 1e-9)]

感觉和 convert_cam_to_3d_trans不太一样，能说说区别吗？

Hi @yanch2116
Thank you for your wonderful work!

With the help of smpl.forward().joints , 'smpl_poses' and 'smpl_trans' conver to 'keypoints3d' .
But How to conver 'keypoints3d' to 'smpl_poses' and 'smpl_trans' ?

Hey, yanch2116, very impressive job done in visualizing 3D characters. I tested and it works, one question to ask, the code romp_server.py has the setting.show_largest=True, I am thinking, what is the condition with multiple people in the webcamera, when there are many people, the blender character keeps on shifting. Are they ways to solve this?
1, keep on the object tracker in single object_ID
2, visualize multiple 3D characters as input in the camera

您好感谢您开发的插件，对blender不是很熟悉，我添加了插件但却没有出现在列表里面，这是正常的吗？

在打开模型后执行ROMP中的 webcam_blender.sh一直停在这个界面，好像没有request请求，ctrl+e也没有反应，请问要如何解决呢？

请问，执行代码
`outputs_all = romp_model(image)
if outputs_all is None:
client.send('none'.encode('utf-8'))
continue

        poses = getAxisAngle(outputs_all['smpl_thetas'])[0]
        if 'cam_trans' not in outputs_all:
            trans = convert_cam_to_3d_trans(outputs_all['cam']).tolist()
        else:
            trans = outputs_all['cam_trans'].tolist()

        outputs = {'poses': poses, 'trans': trans[0]}`

得到的poses信息是24*4的
这是会收到poses是24根骨骼的四元数信息吗？如果是的话，那请问这个四元数的四个数具体代表什么呢？

when I ues my own npz file, it got wrongs as follows, can you tell me why?
in run
self._target(*self._args, **self._kwargs)
File "server.py", line 26, in tcplink
data = getData()
File "server.py", line 16, in getData
temp = np.append(a[key][0]['poses'], a[key][0]['trans'])
KeyError: 'trans'

Hey, Yanchxx,

May I ask, what is the output variable, I think "trans" is the translation between camera and object.

What are the poses, is it the location x, y, z, or rotation x, y, z, degrees?

Thank you 👍

Hi, I'm able to run to demo with blender now, but it turns out that the view is locked while the script is running.

If there's a way to enable rotation?

Hello，I have some question,I can not solve it. Can you help me? Thanks!!!

Python: Traceback (most recent call last): File "/home/guo/Downloads/CharacterDriven-BlenderAddon-master (1)/CharacterDriven-BlenderAddon-master/blender/Beta.blend/Text", line 42, in modal File "/home/guo/Downloads/CharacterDriven-BlenderAddon-master (1)/CharacterDriven-BlenderAddon-master/blender/Beta.blend/Text", line 98, in process_poses KeyError: 'bpy_prop_collection[key]: key "Pelvis" not found' location: <unknown location>:-1

@yanch2116 你好，非常酷的工作！我基本把整个项目跑通了，但我有两个问题想进一步请教一下你。　、
１．导入fbx时骨骼的方向错乱了，参见:https://www.bilibili.com/read/cv2520452 。但是这个文章里的方法没有完全解决方向错乱的问题，所以我想请问一下你，你是怎么解决这个问题的呢？
２．我想请教一下，如何将编辑骨架让其和ｓｍｐｌ的骨架一致呢？
望不吝赐教！万分感谢！

带带我

您好！当我导入mixamo模型，然后点击fix bones，总会报如下错误，请问您应该如何解决

Hey! Thanks for your tool! It's really cool to drive the avator moving in realtime.

I got two further questions:

1. Can I init the 3D space with a pre-build model? For example a room model as of .obj file.
2. How to set the camera's position to a specific point? Let's say one of a corner of the room?

I use ROMP v1.1 on Windows machine.
But CDBA works with version 1.0 of ROMP
I read installation and using documentation of ROMP v1.0 but i can't figured out all
How can i use this addon properly?
I'm not good at programming i'm animator and interested in your project.
Can you write steps how to install and use ROMP with this addon,please?

@yanch2116 I've pressed Ctrl + E, after completing all frames, it's showing this error. Please help me here.

I have ROMP driven SMPL model looks OK, but when using CDBA (I using script locally) driven mixamo model, result looks wrong:

I am using this bone mapper in your repo:

bones_mixamo_smpl_mapper = {
    "Hips": "Pelvis",
    "LeftUpLeg": "L_Hip",
    "RightUpLeg": "R_Hip",
    "Spine2": "Spine3",
    "Spine1": "Spine2",
    "Spine": "Spine1",
    "LeftLeg": "L_Knee",
    "RightLeg": "R_Knee",
    "LeftFoot": "L_Ankle",
    "RightFoot": "R_Ankle",
    "LeftToeBase": "L_Foot",
    "RightToeBase": "R_Foot",
    "Neck": "Neck",
    "LeftShoulder": "L_Collar",
    "RightShoulder": "R_Collar",
    "Head": "Head",
    "LeftArm": "L_Shoulder",
    "RightArm": "R_Shoulder",
    "LeftForeArm": "L_Elbow",
    "RightForeArm": "R_Elbow",
    "LeftHand": "L_Wrist",
    "RightHand": "R_Wrist",
    "LeftHandIndex1": "L_Hand",
    "LeftHandMiddle1": "L_Hand",
    "RightHandMiddle1": "R_Hand",
    "RightHandIndex1": "R_Hand",
}
bones_smpl_mixamo_mapper = {v: k for k, v in bones_mixamo_smpl_mapper.items()}
bone_name_from_index_character = {
    k: bones_smpl_mixamo_mapper[v] for k, v in bone_name_from_index.items()
}

Do u know why?

also the hand look not right.

是不是会跟bone.keyframe_insert插入太多有关？

您好大佬，我跑通了你的这个 demo，在 mixamo 上下载的3d 角色模型也能跑通，我现在用 SMPL 模型（T-pose）的 obj 格式 mesh 网格去 mixamo 绑定骨骼,保存位 T-pose 的 fbx 格式（图 1是pose模式下的样子），但是在点击 fixbone 后，模型变成下图 2 这样，然后
也能运行，骨架是错乱的，老哥可以给点建议吗？感激不尽。

作者您好，我有以下两个问题：
1.我注意到您使用四元数来表示旋转的。请问在driveCharacter(rotations, translation)中的rotation参数是如何从romp的结果中转换而来的呢

2.在

中，每一个bone的旋转向量的原点是什么呢

I met the following error while running Beta.blend. I'm using blender 2.83.9. What's the expected version?

Read blend: E:\Workspace\blender_test\addons\CharacterDriven-BlenderAddon-master\blender\Beta.blend
0 meshes freed
Error: File written by newer Blender binary (290.0), expect loss of data!

Hello. Would your program support run with other characters? And are there any references about building other characters?

请问大佬：我们这个插件只能绑定单人驱动（基于ROMP），能否实现多人驱动呢（基于bev）？

是因为使用smpl模型的骨骼原因吗？

如果在导入fbx的选项里选择“自动骨骼坐标系”，骨骼确实会分布在人体里，但使用脚本驱动时，动作就不对了。
谢谢！