Hi i was interested in this project and wanted to try it out in my windows machine. i am new to python and i followed your manual installation tutorial and reached endless chaos of issues. i think the read me is either outdated or wrong.

Clone this repo and its submodules: git clone --recurse-submodules [email protected]:rlleshi/phar.git and then create and environment with python 3.8+.

Install torch (of course, it is recommended that you have CUDA & CUDNN installed).

i cloned your project
i installed python (Python 3.9.13)
installed cuda and cuDNN from nvidea
installed pytorch (pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121)

Install the correct version of mmcv based on your CUDA & Torch, e.g. pip install mmcv-full==1.3.18 -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.10.0/index.html

i navigated to project main directory (where src etc... are located)
i went below link and installed mmcv mmdet,mmaction etc...
https://mmaction2.readthedocs.io/en/latest/get_started/installation.html#best-practices

Install extra dependencies: pip install -r requirements/extra.txt.

done

now when i run the project like below
python src/demo/multimodial_demo.py video.mp4 demo.json --timestamps

i am getting errors

there is a requirements.txt inside requirements folder . am i supposed to install that as well?
in documentation there is only mention about extra.txt

these are the errors i get do you know why this happen

Traceback (most recent call last):
File "src/demo/multimodial_demo.py", line 20, in
from demo_skeleton import frame_extraction
File "C:\zprojects\test\phar\src\demo\demo_skeleton.py", line 11, in
from mmcv import DictAction
ImportError: cannot import name 'DictAction' from 'mmcv' (C:\Users\admin.conda\envs\openmmlab\lib\site-packages\mmcv_init_.py)

I am interested in the API. How can I contact you?

Hi,

I can't get the docker image to run, it gives me the following error:

docker run rlleshi/phar python src/demo/multimodial_demo.py video.mp4 demo.json --timestamps

Traceback (most recent call last):
  File "__init__.cython-30.pxd", line 984, in numpy.import_array
RuntimeError: module compiled against API version 0x10 but this version of numpy is 0xf . Check the section C-API incompatibility at the Troubleshooting ImportError section at https://numpy.org/devdocs/user/troubleshooting-importerror.html#c-api-incompatibility for indications on how to solve this problem .

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/workspace/phar/mmaction2/demo/demo_skeleton.py", line 23, in <module>
    from mmpose.apis import (inference_top_down_pose_model, init_pose_model,
  File "/workspace/phar/mmpose/mmpose/apis/__init__.py", line 2, in <module>
    from .inference import (inference_bottom_up_pose_model,
  File "/workspace/phar/mmpose/mmpose/apis/inference.py", line 14, in <module>
    from mmpose.datasets.dataset_info import DatasetInfo
  File "/workspace/phar/mmpose/mmpose/datasets/__init__.py", line 7, in <module>
    from .datasets import (  # isort:skip
  File "/workspace/phar/mmpose/mmpose/datasets/datasets/__init__.py", line 2, in <module>
    from ...deprecated import (TopDownFreiHandDataset, TopDownOneHand10KDataset,
  File "/workspace/phar/mmpose/mmpose/deprecated.py", line 5, in <module>
    from .datasets.datasets.base import Kpt2dSviewRgbImgTopDownDataset
  File "/workspace/phar/mmpose/mmpose/datasets/datasets/base/__init__.py", line 2, in <module>
    from .kpt_2d_sview_rgb_img_bottom_up_dataset import \
  File "/workspace/phar/mmpose/mmpose/datasets/datasets/base/kpt_2d_sview_rgb_img_bottom_up_dataset.py", line 8, in <module>
    from xtcocotools.coco import COCO
  File "/opt/conda/lib/python3.8/site-packages/xtcocotools/coco.py", line 58, in <module>
    from . import mask as maskUtils
  File "/opt/conda/lib/python3.8/site-packages/xtcocotools/mask.py", line 3, in <module>
    import xtcocotools._mask as _mask
  File "xtcocotools/_mask.pyx", line 23, in init xtcocotools._mask
  File "__init__.cython-30.pxd", line 986, in numpy.import_array
ImportError: numpy.core.multiarray failed to import

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "src/demo/multimodial_demo.py", line 20, in <module>
    from demo.demo_skeleton import frame_extraction
  File "/workspace/phar/mmaction2/demo/demo_skeleton.py", line 26, in <module>
    raise ImportError('Failed to import `inference_top_down_pose_model`, '
ImportError: Failed to import `inference_top_down_pose_model`, `init_pose_model`, and `vis_pose_result` form `mmpose.apis`. These apis are required in this demo! 

When I compile from source, I get almost the same error:

/ml/phar/mmcv/mmcv/cnn/bricks/transformer.py:28: UserWarning: Fail to import ``MultiScaleDeformableAttention`` from ``mmcv.ops.multi_scale_deform_attn``, You should install ``mmcv-full`` if you need this module. 
  warnings.warn('Fail to import ``MultiScaleDeformableAttention`` from '
Traceback (most recent call last):
  File "/ml/phar/src/demo/demo_skeleton.py", line 15, in <module>
    from mmdet.apis import inference_detector, init_detector
  File "/ml/phar/mmdet/mmdet/apis/__init__.py", line 1, in <module>
    from .inference import (async_inference_detector, inference_detector,
  File "/ml/phar/mmdet/mmdet/apis/inference.py", line 6, in <module>
    from mmcv.ops import RoIPool
  File "/ml/phar/mmcv/mmcv/ops/__init__.py", line 2, in <module>
    from .assign_score_withk import assign_score_withk
  File "/ml/phar/mmcv/mmcv/ops/assign_score_withk.py", line 5, in <module>
    ext_module = ext_loader.load_ext(
  File "/ml/phar/mmcv/mmcv/utils/ext_loader.py", line 13, in load_ext
    ext = importlib.import_module('mmcv.' + name)
  File "/home/firebug/miniconda3/envs/nsfw2/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named 'mmcv._ext'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/ml/phar/src/demo/demo_skeleton.py", line 17, in <module>
    raise ImportError('Failed to import `inference_detector` and '
ImportError: Failed to import `inference_detector` and `init_detector` form `mmdet.apis`. These apis are required in this demo! 

When compiling from source, I made sure I had the right versions of the libraries by compiling from the exact git checkpoints referenced by your code:

mmaction2 0.23.0 /ml/phar/mmaction2
mmcv 1.3.18 /ml/phar/mmcv
mmdet 2.12.0 /ml/phar/mmdet
mmpose 0.22.0 /ml/phar/mmpose

Any ideas? Thanks.

please send code for this one

For example, to determine if a scene would be classified as POV or not.

I saw a repo related text2video and thought about customized porno generation. I made a plan including 3 stages:
1- make classifiers for ethnicity, age, actor attirbutes, position, and scenerio to label videos with timestamps.
The data creation method I used in stage one is download images from google,yandex, and bing and label them with search keyword. For example I searched for cowgirl porno and downloaded images and labeled respectively. (2000 images per class in total without augmentation)
I stuck at stage one because all of my models overfitting no matter what I tried. here is my email [email protected] for detailes.

Hello, I tried to test the app, but i got the same problem as the issue here:
#1

Stack Trace:
File "src/demo/multimodial_demo.py", line 601, in
main()
File "src/demo/multimodial_demo.py", line 562, in main
audio_inference(clip, args.coefficients)
File "src/demo/multimodial_demo.py", line 385, in audio_inference
results = inference_recognizer(AUDIO_MODEL, out_feature)
File "/workspace/phar/mmaction2/mmaction/apis/inference.py", line 99, in inference_recognizer
raise RuntimeError('The type of argument video is not supported: '
RuntimeError: The type of argument video is not supported: <class 'str'>

I checked that the code is up to date, and it's the case:
line 381..:
subprocess.run(
['python', AUDIO_FEATURE_SCRIPT, TEMP, TEMP, '--ext', 'wav'],
capture_output=True)

Any ideas on what I'm doing wrong ?

I've been following the steps to install and spent several hours today trying to get the demo working but I'm getting errors. I've followed the steps in the instructions exactly. When doing a manual install, I get this error:
No CUDA runtime is found, using CUDA_HOME='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.3\bin'
Traceback (most recent call last):
File "c:\users\tyler\source\repos\phar\mmaction2\demo\demo_skeleton.py", line 16, in
from mmdet.apis import inference_detector, init_detector
File "C:\Users\tyler\source\repos\venv\lib\site-packages\mmdet\apis_init_.py", line 2, in
from .inference import (async_inference_detector, inference_detector,
File "C:\Users\tyler\source\repos\venv\lib\site-packages\mmdet\apis\inference.py", line 8, in
from mmcv.ops import RoIPool
File "C:\Users\tyler\source\repos\venv\lib\site-packages\mmcv\ops_init_.py", line 2, in
from .active_rotated_filter import active_rotated_filter
File "C:\Users\tyler\source\repos\venv\lib\site-packages\mmcv\ops\active_rotated_filter.py", line 8, in
ext_module = ext_loader.load_ext(
File "C:\Users\tyler\source\repos\venv\lib\site-packages\mmcv\utils\ext_loader.py", line 13, in load_ext
ext = importlib.import_module('mmcv.' + name)
File "C:\Users\tyler\AppData\Local\Programs\Python\Python38\lib\importlib_init_.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ImportError: DLL load failed while importing _ext: The specified module could not be found.

When trying to use docker, I get an error about not having an nvidia driver (even though I do with cuda, and cudnn setup)

I feel that image processing would be a relatively easy proposition with your library, but before I attempt my own solution, I wanted to ask if you have (or can prepare) a demo that does single images.
Thanks.

Welcome update to OpenMMLab 2.0

I am Vansin, the technical operator of OpenMMLab. In September of last year, we announced the release of OpenMMLab 2.0 at the World Artificial Intelligence Conference in Shanghai. We invite you to upgrade your algorithm library to OpenMMLab 2.0 using MMEngine, which can be used for both research and commercial purposes. If you have any questions, please feel free to join us on the OpenMMLab Discord at https://discord.gg/amFNsyUBvm or add me on WeChat (van-sin) and I will invite you to the OpenMMLab WeChat group.

Here are the OpenMMLab 2.0 repos branches:

Attention: please create a new virtual environment for OpenMMLab 2.0.

I've been looking into this topic for a while as well as similar topics: comprehensive automated tagging, personalized automated rating, automated video scripting for interactive content. I'd be interested in chatting and seeing if we have any insights that might be useful to one another. My discord is Skier23#9916

At first, I got Module Demo not found. I removed the period at demo.demo_skeleton import frame extraction. The multimodial_demo.py is in the same directory as the demo_skeleton.py I thought that was a path problem. Afterward, I get this:

An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ... 

I'm using Windows 10 been trying for a few days can you help a noob out.

Hey,

Is their any chance we can get the dataset videos you used to train the model?

Thanks

I checked the other installation issues and this appears to be different. I installed the phar package dependencies locally and tried to run the demo, but I get this error:

Traceback (most recent call last):
  File "/home/user/projects/test/scripts/phar/src/demo/multimodial_demo.py", line 20, in <module>
    from demo.demo_skeleton import frame_extraction
ModuleNotFoundError: No module named 'demo.demo_skeleton'

I am not an expert with python but it appears that I cannot install the source code as a module without a setup.py file.

I'm trying to test your model, but I ran into an issue with the audio inference, maybe you would have some ideas what could be wrong?

`docker run --gpus all rlleshi/phar python src/demo/multimodial_demo.py /mnt/videos/tr_87505_hd.mp4 /mnt/videos/demo.mp4
Resizing video for faster inference...
Moviepy - Building video temp/tr_87505_hd.mp4.
MoviePy - Writing audio in tr_87505_hdTEMP_MPY_wvf_snd.mp3
MoviePy - Done.
Moviepy - Writing video temp/tr_87505_hd.mp4

Moviepy - Done !
Moviepy - video ready temp/tr_87505_hd.mp4
load checkpoint from local path: checkpoints/har/timeSformer.pth
Performing RGB inference...
100%|██████████| 12/12 [00:17<00:00, 1.44s/it]load checkpoint from local path: checkpoints/har/audio.pth
Performing audio inference...

8%|▊ | 1/12 [00:14<02:43, 14.88s/it]
Traceback (most recent call last):
File "src/demo/multimodial_demo.py", line 601, in
main()
File "src/demo/multimodial_demo.py", line 562, in main
audio_inference(clip, args.coefficients)
File "src/demo/multimodial_demo.py", line 385, in audio_inference
results = inference_recognizer(AUDIO_MODEL, out_feature)
File "/workspace/phar/mmaction2/mmaction/apis/inference.py", line 99, in inference_recognizer
raise RuntimeError('The type of argument video is not supported: '
RuntimeError: The type of argument video is not supported: <class 'str'>`