ddupont808 / gpt-4v-act Goto Github PK
View Code? Open in Web Editor NEWAI agent using GPT-4V(ision) capable of using a mouse/keyboard to interact with web UI
gpt-4v-act's People
Contributors
Stargazers
Watchers
Forkers
twelvearrays zeroxclem thzhfei mivanovitch michaelzona xbmcnerds maifeeulasad ruijian-zha ashbuilds edyuan bortus-ai lilyliao aristoddle davidsonggithub 2132660698 gaocode james4ever0 zianzhangzz alvin-4550 wanggrun tab-dev-ac a1364533187 wuxiaoxrj nifty0x theariaki eureca2017 jxqlovejava cafeii tomchapin wulino zlapp er-hgy deluair y1y2u3u4 genexis-ai chuciqin tominwu akhil4rajan mbyase dgo2dance llwcy8801 ricardrao ljl215186 jeffara lyc-agi contropist 0xtaos happykkai timothybao mattpang michael-jennings wxqianggo lplzyp zebrajack mrg7 piandpower bennyyu79 sreeshaks gitrjaa localserviceapp holen-zhang cskrishna liz-starfield dnuzzolillo everblake sarthak42 zhuoooo youweideng80 neuromod0 nnwhisperer bellyfat aureliuspatiens lincunlv georgel gmh5225 haiandaxia cellinlab godzillaas tomoyukiorita tbergman welkang panzergpt-4v-act's Issues
Not able to run the copilot
I cloned your repo and tried to run it with npm start command, however I'm getting following error:
[email protected] start
electron .
2023-12-02 16:36:21.723 Electron[6233:316840] WARNING: Secure coding is not enabled for restorable state! Enable secure coding by implementing NSApplicationDelegate.applicationSupportsSecureRestorableState: and returning YES.
Acquired webviewId 2
(node:6233) UnhandledPromiseRejectionWarning: Error: ENOENT: no such file or directory, open 'tmp/screenshot.png'
(Use Electron --trace-warnings ... to show where the warning was created)
(node:6233) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag --unhandled-rejections=strict (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
How can I fix this problem?
Use random ids instead of numbers
Or prefixes
Ideas on improving GPT-4V-Act
I am working on a similar project called the Cybergod, which is using AI for controlling the computer to do any task, to eventually replace all human computer operators.
I am harvesting ideas, including yours and this one at: https://github.com/abhiprojectz/SingularGPT
I have discussed with the author at: Significant-Gravitas/AutoGPT#346
My repo is at: https://github.com/James4Ever0/agi_computer_control
If you are interested, you can take a look and get inspiration. Hope it helps.
如何使用GPT-4v-act
作者你好，在使用时遇到了无法正常调用gpt4v的情况，右侧框输入对应命令，没有相应的响应，命令行界面报如下错误：
想问下这个可以怎么解决呢？
还有个疑问，在使用过程中未能找到chatgpt4 的账户密码设置地方，是和这个有关吗？
Error: ENOENT: no such file or directory
@ddupont808 I tried to replicate what you showed in your demo but keep getting errors/rejections.
The response I get in the UI "I don't have the capability to interact with a search bar or perform live actions such as typing or browsing the internet...."
Error:
(node:30692) UnhandledPromiseRejectionWarning: Error: ENOENT: no such file or directory, open 'C:\Users\Admin\ai-browser\tmp\screenshot.png'
Can you help with that?
Does it capture the full page?
Not sure if you have implemented it already, but you should capture the whole page so you could automate things outside the viewport
How to specify the OpenAI API key ?
following the quick start guide when executing npm start, electron is loaded but then error like the one below appear.
Most likely the lack of the GPT-4 API Key is making it not work.
2024-03-24 21:26:20.512 Electron[3538:12247424] WARNING: Secure coding is not enabled for restorable state! Enable secure coding by implementing NSApplicationDelegate.applicationSupportsSecureRestorableState: and returning YES.
[3538:0324/212620.956073:ERROR:trust_store_mac.cc(414)] Error parsing certificate:
ERROR: Unexpected extensions (must be V3 certificate)
ERROR: Failed parsing TBSCertificate
[3538:0324/212620.957650:ERROR:trust_store_mac.cc(846)] Error parsing certificate:
ERROR: Unexpected extensions (must be V3 certificate)
ERROR: Failed parsing TBSCertificate
Acquired webviewId 2
(node:3538) UnhandledPromiseRejectionWarning: TargetCloseError: Waiting for selector `input[type="file"]` failed: Protocol error (Runtime.callFunctionOn): Target closed
at CallbackRegistry.clear (/Users/gp/Desktop/ai-browser/node_modules/puppeteer-core/lib/cjs/puppeteer/cdp/Connection.js:139:36)
at CdpCDPSession._onClosed (/Users/gp/Desktop/ai-browser/node_modules/puppeteer-core/lib/cjs/puppeteer/cdp/CDPSession.js:97:25)
at Connection.onMessage (/Users/gp/Desktop/ai-browser/node_modules/puppeteer-core/lib/cjs/puppeteer/cdp/Connection.js:249:25)
at WebSocket.<anonymous> (/Users/gp/Desktop/ai-browser/node_modules/puppeteer-core/lib/cjs/puppeteer/node/NodeWebSocketTransport.js:52:32)
at callListener (/Users/gp/Desktop/ai-browser/node_modules/ws/lib/event-target.js:290:14)
at WebSocket.onMessage (/Users/gp/Desktop/ai-browser/node_modules/ws/lib/event-target.js:209:9)
at WebSocket.emit (node:events:514:28)
at Receiver.receiverOnMessage (/Users/gp/Desktop/ai-browser/node_modules/ws/lib/websocket.js:1192:20)
at Receiver.emit (node:events:514:28)
at Receiver.dataMessage (/Users/gp/Desktop/ai-browser/node_modules/ws/lib/receiver.js:560:14)
(Use `Electron --trace-warnings ...` to show where the warning was created)
(node:3538) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 5)
^C/Users/gp/Desktop/ai-browser/node_modules/electron/dist/Electron.app/Contents/MacOS/Electron exited with signal SIGINT
Great work! And here is the SoM codebase
It is great to see set-of-mark prompting can be used for webui navigation!
Please find the code for SoM prompting here: https://github.com/microsoft/SoM. It can now support GPT-4V API. Try it out!
Thanks,
Jianwei
Use Vision API
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.