ComfyUI IPA: Failed to save graph file & Increased Time Consumption about onediff

[strint]

https://github.com/search?q=repo%3Asiliconflow%2Fonediff%20set_model_patch_replace&type=code

@ccssu Is set_model_patch_replace cost a lot as expected?

[xiecon]

Update version information:

• oneflow 0.9.1.dev20240509+cu118
• onediff commit accc466
• ComfyUI_IPAdapter_plus commit d76bdbd2b59cb4164adb662359dc59b5aa213548

Same error message:
Building a graph for <class 'register_comfy.openaimodel.UNetModel'> ...
W20240513 11:36:00.435367 71267 cuda_stream.cpp:49] Runtime version 8.7 of cuDNN incompatible with compiletime version 8.9.
Failed to save graph file: /ComfyUI/input/graphs/anyloraCheckpoint_bakedvaeBlessedFp16.safetensors_BaseModel/UNetModel_304c305b00_3_f1f8a7f1ca61b188044db654a526065b05d40d2524e0d31165e22847fc11c900_0.9.1.dev20240509+cu118.graph! Can't pickle <class 'onediff.infer_compiler.transform.builtin_transform.<class 'register_comfy.CrossAttentionPatch.CrossAttentionPatch'>'>: attribute lookup <class 'register_comfy.CrossAttentionPatch.CrossAttentionPatch'> on onediff.infer_compiler.transform.builtin_transform failed

[xiecon]

在显卡A30上测试发现，在set_model_patch_replace这个函数运行时，onediff重写后的运行时间比原本长，原本的时间为：0.0009s，使用onediff后运行时间为：1.04s，耗时增加了很多。

[strint]

Update version information:

• oneflow 0.9.1.dev20240509+cu118
• onediff commit accc466
• ComfyUI_IPAdapter_plus commit d76bdbd2b59cb4164adb662359dc59b5aa213548

Same error message: Building a graph for <class 'register_comfy.openaimodel.UNetModel'> ... W20240513 11:36:00.435367 71267 cuda_stream.cpp:49] Runtime version 8.7 of cuDNN incompatible with compiletime version 8.9. Failed to save graph file: /ComfyUI/input/graphs/anyloraCheckpoint_bakedvaeBlessedFp16.safetensors_BaseModel/UNetModel_304c305b00_3_f1f8a7f1ca61b188044db654a526065b05d40d2524e0d31165e22847fc11c900_0.9.1.dev20240509+cu118.graph! Can't pickle <class 'onediff.infer_compiler.transform.builtin_transform.<class 'register_comfy.CrossAttentionPatch.CrossAttentionPatch'>'>: attribute lookup <class 'register_comfy.CrossAttentionPatch.CrossAttentionPatch'> on onediff.infer_compiler.transform.builtin_transform failed

Save and load is fixed in #872

[xiecon]

The issue with ‘Failed to save graph file’ has indeed been fixed, but the problem of runtime duration has not been resolved, which is 1.95s.

[strint]

The issue with ‘Failed to save graph file’ has indeed been fixed, but the problem of runtime duration has not been resolved, which is 1.95s.

Have you tried other device other than A30? We have no A30 at hand.

https://github.com/siliconflow/onediff/wiki/Compatibility-with-Nvidia-GPUs

[xiecon]

Time this code on the A100:

if not is_sdxl:
        start_time = time.perf_counter()
        for id in [1, 2, 4, 5, 7, 8]:  # id of input_blocks that have cross attention
            set_model_patch_replace(model, patch_kwargs, ("input", id))
            patch_kwargs["number"] += 1
        for id in [
            3,
            4,
            5,
            6,
            7,
            8,
            9,
            10,
            11,
        ]:  # id of output_blocks that have cross attention
            set_model_patch_replace(model, patch_kwargs, ("output", id))
            patch_kwargs["number"] += 1
        set_model_patch_replace(model, patch_kwargs, ("middle", 0))
        end_time = time.perf_counter()
        print(f"time:{end_time - start_time} s")

with onediff: 1.2s
without onediff: 0.0009s
Specifically, I am using ip-adapter-faceid-plusv2_sd15.bin, with the insightface module.

[BradVidler]

Hi all. I am running into issues with IPAdapter as well with ComfyUI. I am using the latest onediff v1.1.0. During Ksampler execution it spits out an error on every step:
Unsupported type: <class 'onediff.infer_compiler.transform.builtin_transform.<class 'register_comfy.CrossAttentionPatch.CrossAttentionPatch'>'> e=RuntimeError('An exception occurred during class transformation:\nTraceback (most recent call last):\n File "/home/me/.local/lib/python3.10/site-packages/onediff/infer_compiler/import_tools/dyn_mock_mod.py", line 165, in _get_module\n return importlib.import_module(module_name)\n File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n File "<frozen importlib._bootstrap>", line 1050, in _gcd_import\n File "<frozen importlib._bootstrap>", line 1027, in _find_and_load\n File "<frozen importlib._bootstrap>", line 1001, in _find_and_load_unlocked\nModuleNotFoundError: No module named "onediff.infer_compiler.transform.builtin_transform.<class \'register_comfy"; \'onediff.infer_compiler.transform.builtin_transform\' is not a package\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/home/me/.local/lib/python3.10/site-packages/onediff/infer_compiler/transform/builtin_transform.py", line 59, in proxy_class\n out = transform_mgr.transform_cls(cls)\n File "/home/me/.local/lib/python3.10/site-packages/onediff/infer_compiler/transform/manager.py", line 82, in transform_cls\n mock_cls = self._transform_entity(mock_full_cls_name)\n File "/home/me/.local/lib/python3.10/site-packages/onediff/infer_compiler/transform/manager.py", line 59, in _transform_entity\n result = self.mocker.mock_entity(entity)\n File "/home/me/.local/lib/python3.10/site-packages/onediff/infer_compiler/import_tools/importer.py", line 94, in mock_entity\n return self.load_entity_with_mock(entity)\n File "/home/me/.local/lib/python3.10/site-packages/onediff/infer_compiler/import_tools/importer.py", line 132, in load_entity_with_mock\n mock_pkg = getattr(mock_pkg, name)\n File "/home/me/.local/lib/python3.10/site-packages/onediff/infer_compiler/import_tools/dyn_mock_mod.py", line 180, in __getattr__\n obj_entity = self._get_module(name)\n File "/home/me/.local/lib/python3.10/site-packages/onediff/infer_compiler/import_tools/dyn_mock_mod.py", line 167, in _get_module\n raise RuntimeError(\nRuntimeError: Failed to import onediff.infer_compiler.transform.builtin_transform.<class \'register_comfy because of the following error (look up to see its traceback):\nNo module named "onediff.infer_compiler.transform.builtin_transform.<class \'register_comfy"; \'onediff.infer_compiler.transform.builtin_transform\' is not a package\n\nException: Failed to import onediff.infer_compiler.transform.builtin_transform.<class \'register_comfy because of the following error (look up to see its traceback):\nNo module named "onediff.infer_compiler.transform.builtin_transform.<class \'register_comfy"; \'onediff.infer_compiler.transform.builtin_transform\' is not a package') Unsupported type: <class 'onediff.infer_compiler.transform.builtin_transform.<class 'register_comfy.CrossAttentionPatch.CrossAttentionPatch'>'> e=RuntimeError('An exception occurred during class transformation:\nTraceback (most recent call last):\n File "/home/me/.local/lib/python3.10/site-packages/onediff/infer_compiler/import_tools/dyn_mock_mod.py", line 165, in _get_module\n return importlib.import_module(module_name)\n File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module\n return _bootstrap._gcd_import(name[level:], package, level)\n File "<frozen importlib._bootstrap>", line 1050, in _gcd_import\n File "<frozen importlib._bootstrap>", line 1027, in _find_and_load\n File "<frozen importlib._bootstrap>", line 1001, in _find_and_load_unlocked\nModuleNotFoundError: No module named "onediff.infer_compiler.transform.builtin_transform.<class \'register_comfy"; \'onediff.infer_compiler.transform.builtin_transform\' is not a package\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/home/me/.local/lib/python3.10/site-packages/onediff/infer_compiler/transform/builtin_transform.py", line 59, in proxy_class\n out = transform_mgr.transform_cls(cls)\n File "/home/me/.local/lib/python3.10/site-packages/onediff/infer_compiler/transform/manager.py", line 82, in transform_cls\n mock_cls = self._transform_entity(mock_full_cls_name)\n File "/home/me/.local/lib/python3.10/site-packages/onediff/infer_compiler/transform/manager.py", line 59, in _transform_entity\n result = self.mocker.mock_entity(entity)\n File "/home/me/.local/lib/python3.10/site-packages/onediff/infer_compiler/import_tools/importer.py", line 94, in mock_entity\n return self.load_entity_with_mock(entity)\n File "/home/me/.local/lib/python3.10/site-packages/onediff/infer_compiler/import_tools/importer.py", line 132, in load_entity_with_mock\n mock_pkg = getattr(mock_pkg, name)\n File "/home/me/.local/lib/python3.10/site-packages/onediff/infer_compiler/import_tools/dyn_mock_mod.py", line 180, in __getattr__\n obj_entity = self._get_module(name)\n File "/home/me/.local/lib/python3.10/site-packages/onediff/infer_compiler/import_tools/dyn_mock_mod.py", line 167, in _get_module\n raise RuntimeError(\nRuntimeError: Failed to import onediff.infer_compiler.transform.builtin_transform.<class \'register_comfy because of the following error (look up to see its traceback):\nNo module named "onediff.infer_compiler.transform.builtin_transform.<class \'register_comfy"; \'onediff.infer_compiler.transform.builtin_transform\' is not a package\n\nException: Failed to import onediff.infer_compiler.transform.builtin_transform.<class \'register_comfy because of the following error (look up to see its traceback):\nNo module named "onediff.infer_compiler.transform.builtin_transform.<class \'register_comfy"; \'onediff.infer_compiler.transform.builtin_transform\' is not a package')

Then once it tries to save the graph we get another error and Comfy crashes.

`Building a graph for <class 'register_comfy.deep_cache_unet.FastDeepCacheUNet'> ...
F20240515 12:30:31.829761 1711 operator.cpp:198] Check failed: 'blob_desc' Must be non NULL
*** Check failure stack trace: ***
@ 0x7f57adedd96a google::LogMessage::Fail()
@ 0x7f57adee08a1 google::LogMessage::SendToLog()
@ 0x7f57adedd499 google::LogMessage::Flush()
@ 0x7f57adee1189 google::LogMessageFatal::~LogMessageFatal()
@ 0x7f57a5f3c355 ZNSt17_Function_handlerIFRKN7oneflow8BlobDescERKSsEZNS0_12_GLOBAL__N_119FillLogicalBlobDescERKSt8functionIFPS1_S5_EERKN6google8protobuf16RepeatedPtrFieldISsEEPSt10unique_ptrISt6vectorISt10shared_ptrIS2_ESaISN_EESt14default_deleteISP_EEEUlS5_E_E9_M_invokeERKSt9_Any_dataS5
@ 0x7f57a5f49fcd oneflow::(anonymous namespace)::FillLogicalBlobDesc()
@ 0x7f57a5f50e8c oneflow::(anonymous namespace)::FillLogicalBlobDesc()
@ 0x7f57a5f51146 oneflow::Operator::FillLogicalInBlobDesc()
@ 0x7f57a5119df8 oneflow::JobBuildAndInferCtx::AddAndInferOp()
@ 0x7f57a511e227 oneflow::JobBuildAndInferCtx::AddAndInferGlobalOp()
@ 0x7f57a4501a73 oneflow::one::LazyInterpreter::ApplyImpl()
@ 0x7f57a450eda6 oneflow::one::LazyInterpreter::Apply()
@ 0x7f57a450d711 oneflow::one::AutogradInterpreter::Apply()
@ 0x7f57a4515fa0 oneflow::one::OpInterpUtil::Dispatch()
@ 0x7f57a45168d9 oneflow::one::OpInterpUtil::Dispatch<>()
@ 0x7f57a4516bfe oneflow::one::OpInterpUtil::Dispatch<>()
@ 0x7f57a46afb7f oneflow::one::OpInterpUtil::Dispatch<>()
@ 0x7f57a4b62fd1 oneflow::one::functional::impl::MatMulFunctor::operator()()
@ 0x7f57a4b64db5 ZNSt17_Function_handlerIFN7oneflow5MaybeINS0_3one6TensorEvEERKSt10shared_ptrIS3_ES8_RKbSA_RKdEZNS2_10functional18PackedFunctorMakerISD_E4makeINSE_4impl13MatMulFunctorELi0EEENSE_13PackedFunctorISD_EERKSsRKT_EUlS8_S8_SA_SA_SC_E_E9_M_invokeERKSt9_Any_dataS8_S8_SA_SA_SC
@ 0x7f579eba58d6 oneflow::one::functional::MatMul()
@ 0x7f58a16047ef oneflow::one::functional::matmul()
@ 0x561085d6010e (unknown)
@ 0x561085d56a7b _PyObject_MakeTpCall
@ 0x561085d50150 _PyEval_EvalFrameDefault
@ 0x561085d6e93e (unknown)
@ 0x561085d4b5d7 _PyEval_EvalFrameDefault
@ 0x561085d55c14 _PyObject_FastCallDictTstate
@ 0x561085d6b86c _PyObject_Call_Prepend
@ 0x561085e86700 (unknown)
@ 0x561085d56a7b _PyObject_MakeTpCall
@ 0x561085d4f096 _PyEval_EvalFrameDefault
@ 0x561085d6e93e (unknown)
Stack trace (most recent call last) in thread 1711:
Object "/home/me/.local/lib/python3.10/site-packages/oneflow/_oneflow_internal.cpython-310-x86_64-linux-gnu.so", at 0x7f58a16047ee, in functional::matmul(_object*, _object*, _object*)
Object "/home/me/.local/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-a92a2d92.so", at 0x7f579eba58d5, in functional::MatMul(std::shared_ptr const&, std::shared_ptr const&, bool, bool, double)
Object "/home/me/.local/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-a92a2d92.so", at 0x7f57a4b64db4, in std::_Function_handler<Maybe (std::shared_ptr const&, std::shared_ptr const&, bool const&, bool const&, double const&), functional::PackedFunctorMaker<Maybe (std::shared_ptr const&, std::shared_ptr const&, bool const&, bool const&, double const&)>::make<functional::impl::MatMulFunctor, 0>(std::string const&, functional::impl::MatMulFunctor const&)::{lambda(std::shared_ptr const&, std::shared_ptr const&, bool const&, bool const&, double const&)#1}>::_M_invoke(std::_Any_data const&, std::shared_ptr const&, std::shared_ptr const&, bool const&, bool const&, double const&)
Object "/home/me/.local/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-a92a2d92.so", at 0x7f57a4b62fd0, in functional::impl::MatMulFunctor::operator()(std::shared_ptr const&, std::shared_ptr const&, bool const&, bool const&, double const&) const
Object "/home/me/.local/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-a92a2d92.so", at 0x7f57a46afb7e, in Maybe OpInterpUtil::Dispatch(OpExpr const&, TensorTuple const&, AttrMap const&)
Object "/home/me/.local/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-a92a2d92.so", at 0x7f57a4516bfd, in Maybe OpInterpUtil::Dispatch(OpExpr const&, TensorTuple const&, OpExprInterpContext const&)
Object "/home/me/.local/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-a92a2d92.so", at 0x7f57a45168d8, in Maybe OpInterpUtil::Dispatch(OpExpr const&, TensorTuple const&, OpExprInterpContext const&)
Object "/home/me/.local/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-a92a2d92.so", at 0x7f57a4515f9f, in OpInterpUtil::Dispatch(OpExpr const&, TensorTuple const&, TensorTuple*, OpExprInterpContext const&)
Object "/home/me/.local/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-a92a2d92.so", at 0x7f57a450d710, in AutogradInterpreter::Apply(OpExpr const&, TensorTuple const&, TensorTuple*, OpExprInterpContext const&) const
Object "/home/me/.local/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-a92a2d92.so", at 0x7f57a450eda5, in LazyInterpreter::Apply(OpExpr const&, TensorTuple const&, TensorTuple*, OpExprInterpContext const&) const
Object "/home/me/.local/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-a92a2d92.so", at 0x7f57a4501a72, in LazyInterpreter::ApplyImpl(UserOpExpr const&, TensorTuple const&, TensorTuple*, OpExprInterpContext const&) const
Object "/home/me/.local/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-a92a2d92.so", at 0x7f57a511e226, in JobBuildAndInferCtx::AddAndInferGlobalOp(OperatorConf const&)
Object "/home/me/.local/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-a92a2d92.so", at 0x7f57a5119df7, in JobBuildAndInferCtx::AddAndInferOp(OperatorConf const&, ParallelConf const&, JobDesc const*, bool)
Object "/home/me/.local/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-a92a2d92.so", at 0x7f57a5f51145, in Operator::FillLogicalInBlobDesc(std::function<BlobDesc* (std::string const&)> const&)
Object "/home/me/.local/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-a92a2d92.so", at 0x7f57a5f50e8b, in
Object "/home/me/.local/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-a92a2d92.so", at 0x7f57a5f49fcc, in
Object "/home/me/.local/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-a92a2d92.so", at 0x7f57a5f3c354, in
Object "/home/me/.local/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-a92a2d92.so", at 0x7f57adee1188, in
Object "/home/me/.local/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-a92a2d92.so", at 0x7f57adedd498, in
Object "/home/me/.local/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-a92a2d92.so", at 0x7f57adee08a0, in
Object "/home/me/.local/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-a92a2d92.so", at 0x7f57adedd969, in
Object "/home/me/.local/lib/python3.10/site-packages/oneflow/../oneflow.libs/liboneflow-a92a2d92.so", at 0x7f579e0f644e, in

Aborted (Signal sent by tkill() 1576 1000)
Aborted`

Additionally, IPAdapter FaceID hangs indefinitely while trying to load Insightface with CUDA, but continues on CPU. EDIT: The hang appears to be unrelated to OneDiff.

[strint]

I am using the latest onediff v1.1.0

Are you using the main branch of onediff?

And there is a new update on ipadaptor is on the way: #879