multi-on / cookbook Goto Github PK
View Code? Open in Web Editor NEWMultiOn API
License: MIT License
MultiOn API
License: MIT License
I have a page that I use often called products, and I was using multion to delete 20k products. I had to slow it down to doing one product delete at a time as it would get way off if I let it run for very long. Obviously it would work ok for a few and I wasn't trying to do 20k at once, but after about 4 correct iterations, it would suddenly be on the wrong page etc. The only thing I could surmise is that because it was a tab INside the browser window it would refresh the product count/numbers and the way it was labeling the page elements was being resorted so then it was trying to click the exact same element each time even though the page was being reordered or something? I am not much of a programmer so Im sorry if this doesnt make sense.... I can take a short video or something if you would like to see what task I am using it for and whatnot. Thanks for all your work!
Getting "Server Disconnected. Please press connect in the Multion extension popup" when trying to use the client:
import multion
api_key = 'xxxx'
multion.login(use_api=True, multion_api_key=api_key)
multion.set_remote(False)
Logged in using API key.
response = multion.create_session({"url": "https://www.linkedin.com"})
session_id = response['session_id']
session_id
Request failed with status code: 404
Response text: {"detail":"Could not connect to the extension"}
Add observability to MultiON sessions
Even when specifically asking for navigating to the exact url chrome://history, it'll instead try to access chrome//history.
(I hope I ended up in the right repository, I just felt like at least generally reporting it and wanting to be helpful.. 😃)
code:
import base64
import os
from io import BytesIO
from typing import Optional
import multion.client
multion.login()
from langchain.llms import OpenAI
from langchain.agents import initialize_agent, Tool
from langchain.agents import AgentType
from langchain.chat_models import ChatOpenAI
os.environ["LANGCHAIN_TRACING"] = "true"
os.environ['OPENAI_API_KEY'] = "sk-***************"
from langchain.tools import StructuredTool
from human import HumanInputRun
#import multion
import multion
from multion.client import MultiOn
#multion_toolkit = Multion(use_api=True, mode="auto")
multion.CreateSessionInputBrowserParams.
def agent(query: str):
#from multion.client import MultiOn
multion_toolkit = MultiOn(api_key="50***************")
# multion.set_remote(True)
tool = StructuredTool.from_function(multion_toolkit.browse(cmd="search flights in google"),description="This function performs a browsing operation based on the given query.")
human_input = HumanInputRun()
llm = OpenAI(temperature=0)
# Structured tools are compatible with the STRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION agent type.
agent_executor = initialize_agent(
[tool, human_input],
llm,
agent=AgentType.STRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION,
verbose=True,
)
return agent_executor.run(query)
PROMPT = f"""
You are an expert AI Agent whose job is to find and get a gpu on a given service HYPERSTACK (https://www.hyperstack.cloud). Pass in the full detailed instructions exactly as below to MultiOn for browsing.
Here are the high-level steps:
1. Login and go to reach the gpu selection page
2. Click on the Load Provising profile and then select the multion-profile and load it
3. After the profile has been correctly selected and loaded continue
4. If an on-demand A100 GPU is available with 80GB RAM, then reserve it
5. If reservation is successful, then enable ssh access and return public ip to access the machine
"""
agent(query=PROMPT)
error:
yping.cast(InternalServerErrorResponse, construct_type(type_=InternalServerErrorResponse, object_=response.json())) # type: ignore
File "C:\Users\tfd250\AppData\Local\miniconda3\envs\albert\lib\site-packages\httpx_models.py", line 764, in json
return jsonlib.loads(self.content, **kwargs)
File "C:\Users\tfd250\AppData\Local\miniconda3\envs\albert\lib\json_init.py", line 346, in loads
return _default_decoder.decode(s)
File "C:\Users\tfd250\AppData\Local\miniconda3\envs\albert\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\tfd250\AppData\Local\miniconda3\envs\albert\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
please help
or do a video on using multion with langchain.
i am trying tom build assistant which can book movie ticket flight ticket everything automatically and chat with the user also..
i am trying to implement the this
https://github.com/MULTI-ON/api/tree/main/examples/gpu_agent
The example has lots of deprecation warnings and an error:
C:Workspace\multion-api\.venv\Lib\site-packages\langchain\llms\__init__.py:548:` LangChainDeprecationWarning: Importing LLMs from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.llms import OpenAI`.
To install langchain-community run `pip install -U langchain-community`.
warnings.warn(
C:Workspace\multion-api\.venv\Lib\site-packages\langchain\chat_models\__init__.py:31: LangChainDeprecationWarning: Importing chat models from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:
`from langchain_community.chat_models import ChatOpenAI`.
To install langchain-community run `pip install -U langchain-community`.
warnings.warn(
C:Workspace\multion-api\.venv\Lib\site-packages\langchain_core\_api\deprecation.py:117: LangChainDeprecationWarning: The class `langchain_community.llms.openai.OpenAI` was deprecated in langchain-community 0.0.10 and will be removed in 0.2.0. An updated version of the class exists in the langchain-openai package and should be used instead. To use it run `pip install -U langchain-openai` and import as `from langchain_openai import OpenAI`.
warn_deprecated(
C:Workspace\multion-api\.venv\Lib\site-packages\langchain_core\_api\deprecation.py:117: LangChainDeprecationWarning: The function `initialize_agent` was deprecated in LangChain 0.1.0 and will be removed in 0.2.0. Use Use new agent constructor methods like create_react_agent, create_json_agent, create_structured_chat_agent, etc. instead.
warn_deprecated(
C:Workspace\multion-api\.venv\Lib\site-packages\langchain_core\_api\deprecation.py:117: LangChainDeprecationWarning: The function `run` was deprecated in LangChain 0.1.0 and will be removed in 0.2.0. Use invoke instead.
warn_deprecated(
Traceback (most recent call last):
File "C:Workspace\multion-api\examples\gpu_agent\agent_example.py", line 51, in <module>
agent(query=PROMPT)
File "C:Workspace\multion-api\examples\gpu_agent\agent_example.py", line 37, in agent
return agent_executor.run(query)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:Workspace\multion-api\.venv\Lib\site-packages\langchain_core\_api\deprecation.py", line 145, in warning_emitting_wrapper
return wrapped(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:Workspace\multion-api\.venv\Lib\site-packages\langchain\chains\base.py", line 538, in run
return self(args[0], callbacks=callbacks, tags=tags, metadata=metadata)[
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:Workspace\multion-api\.venv\Lib\site-packages\langchain_core\_api\deprecation.py", line 145, in warning_emitting_wrapper
return wrapped(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:Workspace\multion-api\.venv\Lib\site-packages\langchain\chains\base.py", line 363, in __call__
return self.invoke(
^^^^^^^^^^^^
File "C:Workspace\multion-api\.venv\Lib\site-packages\langchain\chains\base.py", line 138, in invoke
inputs = self.prep_inputs(input)
^^^^^^^^^^^^^^^^^^^^^^^
File "C:Workspace\multion-api\.venv\Lib\site-packages\langchain\chains\base.py", line 465, in prep_inputs
raise ValueError(
ValueError: A single string input was passed in, but this chain expects multiple inputs ({'', 'input'}). When a chain expects multiple inputs, please call it by passing in a dictionary, eg `chain({'foo': 1, 'bar': 2})```
If an older version of langchain is installed, error ModuleNotFoundError: No module named 'langchain.agents.agent_toolkits.multion'
is raised.
After setting up task scheluder, i can't login to multion.
task-scheduler-py3.10) C:\Users\hu\Documents\multion_exampels\examples\task_scheduler>python scheduler_application.py
Getting OpenAI API Key...
Submitted API key: SECRET
Set OPENAI_API_KEY: SECRET
Exception in thread Thread-2 (login):
Traceback (most recent call last):
File "C:\Users\hu\.pyenv\pyenv-win\versions\3.10.11\lib\threading.py", line 1016, in _bootstrap_inner
self.run()
File "C:\Users\hu\.pyenv\pyenv-win\versions\3.10.11\lib\threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\hu\AppData\Local\pypoetry\Cache\virtualenvs\task-scheduler-KfTrUDer-py3.10\lib\site-packages\multion\multion.py", line 300, in login
_multion_instance.login()
File "C:\Users\hu\AppData\Local\pypoetry\Cache\virtualenvs\task-scheduler-KfTrUDer-py3.10\lib\site-packages\multion\multion.py", line 72, in login
self.client_id = self.register_client()
File "C:\Users\hu\AppData\Local\pypoetry\Cache\virtualenvs\task-scheduler-KfTrUDer-py3.10\lib\site-packages\multion\multion.py", line 105, in register_client
raise Exception("Failed to register client")
Exception: Failed to register client
After adding multion_token.txt with multion token i have errors:
Thread is still running. Closing the thread.
An error occurred when calling message handler
Traceback (most recent call last):
File "C:\Users\ha\AppData\Local\pypoetry\Cache\virtualenvs\task-scheduler-KfTrUDer-py3.10\lib\site-packages\pystray\_win32.py", line 401, in _dispatcher
return int(icon._message_handlers.get(
File "C:\Users\ha\AppData\Local\pypoetry\Cache\virtualenvs\task-scheduler-KfTrUDer-py3.10\lib\site-packages\pystray\_win32.py", line 213, in _on_notify
descriptors[index - 1](self)
File "C:\Users\ha\AppData\Local\pypoetry\Cache\virtualenvs\task-scheduler-KfTrUDer-py3.10\lib\site-packages\pystray\_base.py", line 324, in inner
callback(self)
File "C:\Users\ha\AppData\Local\pypoetry\Cache\virtualenvs\task-scheduler-KfTrUDer-py3.10\lib\site-packages\pystray\_base.py", line 449, in __call__
return self._action(icon, self)
File "C:\Users\ha\Documents\multion_exampels\examples\task_scheduler\scheduler_application.py", line 590, in after_click
login_multion()
File "C:\Users\ha\Documents\multion_exampels\examples\task_scheduler\scheduler_application.py", line 444, in login_multion
multion_login()
File "C:\Users\ha\Documents\multion_exampels\examples\task_scheduler\scheduler_application.py", line 97, in multion_login
t1.cancel()
AttributeError: 'Thread' object has no attribute 'cancel'
Exception in thread Thread-2 (login):
Traceback (most recent call last):
File "C:\Users\ha\.pyenv\pyenv-win\versions\3.10.11\lib\threading.py", line 1016, in _bootstrap_inner
self.run()
File "C:\Users\ha\.pyenv\pyenv-win\versions\3.10.11\lib\threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\ha\AppData\Local\pypoetry\Cache\virtualenvs\task-scheduler-KfTrUDer-py3.10\lib\site-packages\multion\multion.py", line 300, in login
_multion_instance.login()
File "C:\Users\ha\AppData\Local\pypoetry\Cache\virtualenvs\task-scheduler-KfTrUDer-py3.10\lib\site-packages\multion\multion.py", line 72, in login
self.client_id = self.register_client()
File "C:\Users\ha\AppData\Local\pypoetry\Cache\virtualenvs\task-scheduler-KfTrUDer-py3.10\lib\site-packages\multion\multion.py", line 105, in register_client
raise Exception("Failed to register client")
Exception: Failed to register client
When i use poetry and .env file with keys - i also have error:
PS C:\Users\hu\Documents\multion_exampels\examples\task_scheduler> c:; cd 'c:\Users\hu\Documents\multion_exampels\examples\task_scheduler'; & 'c:\Users\hu\AppData\Local\pypoetry\Cache\virtualenvs\task-scheduler-KfTrUDer-py3.10\Scripts\python.exe' 'c:\Users\hu\.vscode\extensions\ms-python.debugpy-2024.1.10371006-win32-x64\bundled\libs\debugpy\adapter/../..\debugpy\launcher' '54935' '--' 'C:\Users\hu\Documents\multion_exampels\examples\task_scheduler\scheduler_application.py'
Set OPENAI_API_KEY:
Exception in thread Thread-7 (login):
Traceback (most recent call last):
File "C:\Users\hu\.pyenv\pyenv-win\versions\3.10.11\lib\threading.py", line 1016, in _bootstrap_inner
self.run()
File "C:\Users\hu\.pyenv\pyenv-win\versions\3.10.11\lib\threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "c:\Users\hu\AppData\Local\pypoetry\Cache\virtualenvs\task-scheduler-KfTrUDer-py3.10\lib\site-packages\multion\multion.py", line 300, in login
_multion_instance.login()
File "c:\Users\hu\AppData\Local\pypoetry\Cache\virtualenvs\task-scheduler-KfTrUDer-py3.10\lib\site-packages\multion\multion.py", line 72, in login
self.client_id = self.register_client()
File "c:\Users\hu\AppData\Local\pypoetry\Cache\virtualenvs\task-scheduler-KfTrUDer-py3.10\lib\site-packages\multion\multion.py", line 105, in register_client
raise Exception("Failed to register client")
Exception: Failed to register client
Hi there,
Heard good things about your tech and approach
Just wanted to come and say I hope you'll consider developing MultiON AI openly
Soon there will be a plethora of ai projects like hyperwrite and yourselves and I can't wait to contribute to the ones that open
And in general I think most revenue will come from B2B/B2C business operations, open source would give you a super dev boost
All the best!
venv adds a env/ directory that should be ignored
Hi,
I'm unable to execute example given here: https://docs.multion.ai/EXAMPLES/event_rsvps_using_langchain with latest langchain version.
I've tried running it with AgentExecutor but couldn't get it to work. Also tried to downgrade langchain to version <0.1.0
and tried a couple of 0.0.2xx and 0.0.3xx versions but couldn't find a version which runs this example like this.
Could anyone please tell which is the pinned langchain version to use which has stable multion support?
Going through "Add API Billing" and entering credit card details results in "No Stripe customer found" and "Billing Portal couldn't be created"
Jam recording sent privately to Div
response:
{
...
screenshot = ''
}
screenshot = response["screenshot"]
get_screenshot(screenshot)
When running Multion_api_example.ipynb with set_remote(True), browse() comes back with an empty screenshot - which might also be a bug, it works fine when set_remote(False). This empty screenshot causes get_screenshot() to throw an exception. May want to handle it elegantly so that the notebook continues.
sessionId/session_id in response are inconsistent. Noticed when running the Multion_api_example.ipynb which failed when setting session = response["session_id"] after running browse
browse:
{
'result': 'ANSWER: The weather today in San Francisco is mostly cloudy with a temperature of 57°F (13°C). The precipitation is 0%, humidity is at 90%, and the wind speed is 5 mph.\n\nSTATUS: DONE',
'status': 'DONE',
'lastUrl': 'https://www.google.com/search?q=weather+today+in+san+francisco',
'content': '',
'screenshot': '',
'sessionId': '0c8dcab0-9a9b-46fc-96b4-02544d5c8d6e'
}
create_session:
{
...
'session_id': ...
}
I can get a real sample result for create_session in a bit, having trouble connecting at the moment
I am having an error while trying to integrate Multion with langchain.
here is the code I am executing:
def multion_agent(input):
toolkit = MultionToolkit()
tools = toolkit.get_tools()
multion_api_key = os.getenv("MULTION_API_KEY")
multion.login(use_api=True, multion_api_key=multion_api_key)
agent = initialize_agent(
tools,
llm=model,
agent=AgentType.STRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION,
verbose=True,
)
result = agent.run(input)
print(f"{result}")
multion_agent("What are top best restaruants in Ghana based on their ranks?")
The agent kicks off successfully and then there is an error:
$ python main.py
Logged in using API key.
> Entering new AgentExecutor chain...
Action:
{
"action": "create_multion_session",
"action_input": {
"url": "https://www.google.com/",
"query": "What are top best restaruants in Ghana based on their ranks?"
}
}
Request failed with status code: 405
Response text: {"detail":"Method Not Allowed"}
Request failed with status code: 405
Response text: {"detail":"Method Not Allowed"}
Request failed with status code: 405
Response text: {"detail":"Method Not Allowed"}
Request failed with status code: 405
Response text: {"detail":"Method Not Allowed"}
Traceback (most recent call last):
File "C:\Users\ERASMUS\Documents\projects\inboxgenius\.venv\Lib\site-packages\langchain_community\tools\multion\create_session.py", line 61, in _run
response = multion.new_session({"input": query, "url": url})
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ERASMUS\Documents\projects\inboxgenius\.venv\Lib\site-packages\deprecated\classic.py", line 285, in wrapper_function
return wrapped_(*args_, **kwargs_)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ERASMUS\Documents\projects\inboxgenius\.venv\Lib\site-packages\multion\multion.py", line 600, in new_session
return _multion_instance.new_session(data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ERASMUS\Documents\projects\inboxgenius\.venv\Lib\site-packages\multion\multion.py", line 358, in new_session
return self.post(url, data)
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ERASMUS\Documents\projects\inboxgenius\.venv\Lib\site-packages\multion\multion.py", line 289, in post
raise Exception("Failed to get a valid response after 5 attempts")
Exception: Failed to get a valid response after 5 attempts
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\ERASMUS\Documents\projects\inboxgenius\main.py", line 15, in <module>
multion_agent("What are top best restaruants in Ghana based on their ranks?")
File "C:\Users\ERASMUS\Documents\projects\inboxgenius\langchain_lab\lab.py", line 216, in multion_agent
result = agent.run(input)
^^^^^^^^^^^^^^^^
File "C:\Users\ERASMUS\Documents\projects\inboxgenius\.venv\Lib\site-packages\langchain_core\_api\deprecation.py", line 145, in warning_emitting_wrapper
return wrapped(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ERASMUS\Documents\projects\inboxgenius\.venv\Lib\site-packages\langchain\chains\base.py", line 538, in run
return self(args[0], callbacks=callbacks, tags=tags, metadata=metadata)[
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ERASMUS\Documents\projects\inboxgenius\.venv\Lib\site-packages\langchain_core\_api\deprecation.py", line 145, in warning_emitting_wrapper
return wrapped(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ERASMUS\Documents\projects\inboxgenius\.venv\Lib\site-packages\langchain\chains\base.py", line 363, in __call__
return self.invoke(
^^^^^^^^^^^^
File "C:\Users\ERASMUS\Documents\projects\inboxgenius\.venv\Lib\site-packages\langchain\chains\base.py", line 162, in invoke
raise e
File "C:\Users\ERASMUS\Documents\projects\inboxgenius\.venv\Lib\site-packages\langchain\chains\base.py", line 156, in invoke
self._call(inputs, run_manager=run_manager)
File "C:\Users\ERASMUS\Documents\projects\inboxgenius\.venv\Lib\site-packages\langchain\agents\agent.py", line 1329, in _call
next_step_output = self._take_next_step(
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ERASMUS\Documents\projects\inboxgenius\.venv\Lib\site-packages\langchain\agents\agent.py", line 1055, in _take_next_step
[
File "C:\Users\ERASMUS\Documents\projects\inboxgenius\.venv\Lib\site-packages\langchain\agents\agent.py", line 1055, in <listcomp>
[
File "C:\Users\ERASMUS\Documents\projects\inboxgenius\.venv\Lib\site-packages\langchain\agents\agent.py", line 1151, in _iter_next_step
observation = tool.run(
^^^^^^^^^
File "C:\Users\ERASMUS\Documents\projects\inboxgenius\.venv\Lib\site-packages\langchain_core\tools.py", line 401, in run
raise e
File "C:\Users\ERASMUS\Documents\projects\inboxgenius\.venv\Lib\site-packages\langchain_core\tools.py", line 358, in run
self._run(*tool_args, run_manager=run_manager, **tool_kwargs)
File "C:\Users\ERASMUS\Documents\projects\inboxgenius\.venv\Lib\site-packages\langchain_community\tools\multion\create_session.py", line 67, in _run
raise Exception(f"An error occurred: {e}")
Exception: An error occurred: Failed to get a valid response after 5 attempts
latest pip package only works with python=3.11
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.