lazyagi / lazyllm Goto Github PK

import lazyllm
t = lazyllm.TrainableModule('internlm2-chat-7b')
Traceback (most recent call last):
File "", line 1, in
File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/lazyllm/module/module.py", line 382, in init
self.base_model = ModelDownloader(lazyllm.config['model_source']).download(base_model)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/py11/lib/python3.11/site-packages/lazyllm/components/utils/downloader/model_downloader.py", line 68, in download
if model in model_name_mapping.keys() and self.model_source in model_name_mapping[model][['source']].keys():
~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
TypeError: unhashable type: 'list'

Description:

Hi team,

I noticed that the CI workflow defined in main.yml for pull requests shows significant time differences between the BasicTests and AdvancedTests stages. Specifically:

• BasicTests takes approximately 24m 54s
• AdvancedTests takes approximately 28m 8s

Could someone help clarify why there is such a notable time difference between these two test stages? Are there specific tests or steps within AdvancedTests that inherently require more time, or is there potential for optimization?

Additionally, any insights into how we might reduce the overall testing time for both stages would be greatly appreciated.

Thank you!

hi,
I am reaching out to invite you to participate in our upcoming AGI2B streaming session on WeChat, which focuses on the AI agent business. We are planning to discuss the AI agent business landscape in America, and your insights would be incredibly valuable.

Your expertise in demoing AI products and engaging local customers in China would add great depth to our discussion. We believe that your participation would be beneficial not only for our audience but also for expanding the reach and impact of AI agent technologies.

Could you please let me know if you are available to join us? We are flexible with the scheduling and can accommodate your availability.
my wechat: jachenms

Hi, I found that the input and output of toolManager in lazyLLM need to be passed in the input and output results of LLM. I am confused, why does toolManager rely on these contents? Shouldn't it only accept the tool name and input parameters?

1. Refactor Creation and Usage of TrainableModule and ServerModule:

   • Enhance the creation and usage of TrainableModule and ServerModule to support accessing deployed services via URL.
   • Users should not need to be concerned with the pre-set inference parameters.
   • TrainableModule should be able to share a module that uses the same service but has an independent prompt.

2. Add Global Configuration Pool:

   • Implement a global configuration pool, accessed using session-id (default to thread ID if not provided).
   • Use HTTP headers instead of LazyLLMRequest to pass global parameters.
   • Sub-services should store received parameters in the global resource pool and read them when invoking the next level of services.

3. Refactor Trace/Exception Mechanism:

   • Refactor the trace and exception mechanism to utilize the aforementioned global resource pool.
   • Remove the ModuleResponse class.

4. Support Streaming of Intermediate Results:

   • Implement support for streaming intermediate results.
   • Any module should be able to temporarily store results in a cross-process message queue.
   • The interface service will read from this queue and output results to the user.

Hi, i have a problem now. When i use with to create a pipeline object in a function and use the switch object in the pipeline, an error is reported that the switch object is not defined, as follows:

The error is as follows:

But there is no problem when the pipeline is not defined in a function, as follows:

Model support must accommodate both inference and fine-tuning, with a higher priority on inference. Even if only inference is available, it should still be added to the TrainableModule.

1. Text-to-Image Model
   Stable Diffusion 3 has seen significant improvements in image quality, text rendering, and multi-theme generation.
   Stable Diffusion 3 Medium

2. Image-Text Understanding Model
   InternVL 1.5 supports image description and image Q&A. While it performs slightly worse than Gemini Ultra in TextQA (more focused on image description), it is the best open-source multimodal solution for comprehensive scenarios.
   InternVL-Chat V1-5 (inference using lmdeploy)

3. Text-to-Speech Model
   A new highly praised open-source speech synthesis project trained on 10 million hours of data.
   ChatTTS
   The open-source model generates 90-second music clips. The non-open-source Stable Audio 2.0 can produce up to 3-minute-long music, songs, melodies, or vocals.
   Stable Audio Open 1.0

4. Speech-to-Text Model
   OpenAI's Whisper is significantly outperformed by DAMO Academy's FunASR in Chinese. The best results are achieved by combining a recognition model with endpoint detection and punctuation prediction. Three models are needed, listed from top to bottom: speech recognition, endpoint detection, and punctuation prediction.
   Speech Recognition
   Endpoint Detection
   Punctuation Prediction

why am getting the above error when am trying to run the code in colab

During inference, when the available number of GPUs exceeds the demand, the tasks are distributed across multiple computing nodes. This is achieved through a load balancing mechanism to manage the task execution.

Hi, I found that the input and output of toolManager in lazyLLM need to be passed in the input and output results of LLM. I am confused, why does toolManager rely on these contents? Shouldn't it only accept the tool name and input parameters?

Develop a simple version is enough currently.

Assigned to @yewentao256

1. Legal Assistant
2. Agent
3. Data Analysis and Chart Drawing
4. Long Text Generation (Writing Novels)
5. Arxiv Paper Reading and Q&A
6. PPT Outline and Content Generation
7. Recipe Query and Generation (Check Calories)
8. Industry Morning Report
9. English Speaking Practice (+ Vocabulary Memorization)
10. Logo Generation

1. order: openai, sensenova, glm, kimi, qianwen, doubao

```
if source.lower() == "openai":
    return OpenAIModule(**params)
elif source.lower() == "glm":
    return GLMModule(**params)
elif source.lower() == "kimi":
    return MoonshotAIModule(**params)
elif source.lower() == "sensenova":
    return SenseNovaModule(**params)
elif source.lower() == "qwen":
    return QwenModule(**params)
elif source.lower() == "doubao":  
```

maybe you can use locals()[f'{source}Module'], source shoule be 'OpenAI', 'GLM', 'SenseNova' and so on, and config check should use lower()
3. move argument-check of doubao to DoubaoModule.init()

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.

Describe the bug

I traced the request processing, it seems in lazyllm/components/deploy/relay/server.py generate function

if getattr(getattr(func, '_meta', func.__class__), '__enable_request__', False):
    output = func(h.make_request(input, **kw))
else:
    output = func(input, **kw)

As the TrainableModule's __enable_request__ is indeed False, the keyword extraction 'h.make_request' won't be executed. Consequently, the context set by 'llm_chat_history' won't take effect.

Reproduction

# coding:utf-8
import lazyllm
from lazyllm import launchers, deploy, LazyLlmRequest

# ====================================== Setting parameters:

base_model = "xxxx/internlm2-chat-20b"
deploy_config = {"launcher": launchers.slurm(partition="pat_rd", nnode=1, nproc=1, ngpus=1, sync=False), "port": 22332}


prompter = lazyllm.ChatPrompter(instruction=" 模仿孔子的说话方式进行对话")
tm = lazyllm.TrainableModule(base_model, "").prompt(prompter).deploy_method(deploy.vllm, **deploy_config)
m = lazyllm.ServerModule(tm)
m.start()


params = {
    tm._module_id: {"llm_chat_history": [["**的首都是哪个城市", "**的首都是北京"]]},
}


req = LazyLlmRequest(
    input="它有哪些著名的景点",
    global_parameters=params,
)

# llm_chat_history takes effect if it's sent to tm driectly
print(tm("它有哪些著名的景点", llm_chat_history=[["**的首都是哪个城市", "**的首都是北京"]]))

# BUG: if tm._module_id's history will not take effect 
print(m(req))

Environment

slurm 20.11.8-7.7
vllm 0.4.3

Issue Description

I encountered an error while trying to import the module lazyllm.components.deploy.relay.server using importlib.import_module.

Steps to Reproduce

1. Use the following code to attempt the import:

   import importlib
   importlib.import_module('lazyllm.components.deploy.relay.server')

Error Traceback

Error                                     Traceback (most recent call last)
Cell In[1], line 2
      1 import importlib
----> 2 importlib.import_module('lazyllm.components.deploy.relay.server')

File ~/.conda/envs/lazyllm_docs/lib/python3.10/importlib/__init__.py:126, in import_module(name, package)
    124             break
    125         level += 1
--> 126 return _bootstrap._gcd_import(name[level:], package, level)

File <frozen importlib._bootstrap>:1050, in _gcd_import(name, package, level)

File <frozen importlib._bootstrap>:1027, in _find_and_load(name, import_)

File <frozen importlib._bootstrap>:1006, in _find_and_load_unlocked(name, import_)

File <frozen importlib._bootstrap>:688, in _load_unlocked(spec)

File <frozen importlib._bootstrap_external>:883, in exec_module(self, module)

File <frozen importlib._bootstrap>:241, in _call_with_frames_removed(f, *args, **kwds)

File ~/project/lazyllm/lazyllm/components/deploy/relay/server.py:38
     35 def load_func(f):
     36     return cloudpickle.loads(base64.b64decode(f.encode('utf-8')))
---> 38 func = load_func(args.function)
     39 if args.before_function:
     40     before_func = load_func(args.before_function)

File ~/project/lazyllm/lazyllm/components/deploy/relay/server.py:36, in load_func(f)
     35 def load_func(f):
---> 36     return cloudpickle.loads(base64.b64decode(f.encode('utf-8')))

File ~/.conda/envs/lazyllm_docs/lib/python3.10/base64.py:87, in b64decode(s, altchars, validate)
     85 if validate and not re.fullmatch(b'[A-Za-z0-9+/]*={0,2}', s):
     86     raise binascii.Error('Non-base64 digit found')
---> 87 return binascii.a2b_base64(s)

Error: Incorrect padding

Additional Context

• Python Version: 3.10
• Environment: conda environment (lazyllm_docs)
• Related File: ~/project/lazyllm/lazyllm/components/deploy/relay/server.py

The error is related to incorrect padding in a base64 encoded string used in the load_func function.
This occurs at the following line in server.py:

func = load_func(args.function)