go-rod / rod Goto Github PK

I'm using this function to get xpath of the element in DOM in Chrome:

function getXPathForElement(element) {
    const idx = (sib, name) => sib 
        ? idx(sib.previousElementSibling, name||sib.localName) + (sib.localName == name)
        : 1;
    const segs = elm => !elm || elm.nodeType !== 1 
        ? ['']
        : elm.id && document.getElementById(elm.id) === elm
            ? [`id("${elm.id}")`]
            : [...segs(elm.parentNode), `${elm.localName.toLowerCase()}[${idx(elm)}]`];
    return segs(element).join('/');
}

Been trying to use page.Eval with Rod but with no luck. Any pointer on how should I proceed?

觉得这个项目很好，想做个 url资产收集的爬虫。 想请教下 怎么获取一个网页里全部的 标签呢，目前我只能一次获取一个
el := browser.Page("https://movie.douban.com/top250").Element("a")
fmt.Println(el.Text())

感谢大佬的指导！

网站： http://demo.aisec.cn/demo/aisec/
请问怎么获取网站里的，js、ajax渲染的链接呢？就是： click_link.php?id=1 ; ajax_link.php?id=1&t= ; if_link.php?id=2
感谢啊，从下班回来折腾了3小时，还没搞定。就来求助大佬了，感谢！

I am trying to get text of element via: page.Element(selector).Text(). It returns empty string in my case.
If in browser I use document.getElementById("receipt-result").textContent I will get full text of element.
What method should I use in this case to get similar result?

I am following the installation steps. While trying to create binary I am getting the below error:-

standard_init_linux.go:211: exec user process caused "no such file or directory"

Below is the output of Docker version command

I am running it on Ubuntu. I have installed Docker and cloned the repo. Please let me know if I am missing anything.

Rod v0.28.0 used.
Hi I am trying to use WaitRequestIdle in following way:

package main

import (
	"fmt"
	"time"

	"github.com/ysmood/rod"
	"github.com/ysmood/rod/lib/launcher"
)

func main() {
	url := launcher.New().
		Headless(false).
		Launch()

	b := rod.New().ControlURL(url).Connect().Timeout(30 * time.Second)
	defer b.Close()

	page := b.Page("https://google.com")
	w := page.WaitRequestIdle()

	page.Navigate("https://wikipedia.org")
	fmt.Println(time.Now())
	w()
	fmt.Println(time.Now())

	page.Navigate("https://gmail.com")
	fmt.Println(time.Now())
	w()
	fmt.Println(time.Now())
}

It seems w() really waits only after first navigation attempt but not on second. If I create second instance before second attempt it will wait twice. But I am not sure that this is intended behaviour.
Am I using it correctly?

Does rod support setting http/socks5 proxy? If so, with or without authentication?

I want to use an existing browser instance to create a new context,
In puppeteer：
browser.createIncognitoBrowserContext() // Target.createBrowserContext

To help users who are familiar with chromedp to understand rod better.

Reference: https://github.com/chromedp/examples

When running headless mode, it's better to have a way to see what's going on with the browser.

We can start an HTTP server to serve a debug port to let the user use a browser to watch the screenshots of the headless browser.

The monitor page will have a dropdown list to choose which tab to watch.

hello,

I'm struggling on getting cookies from the page. I've seen on how rod sets the cookie using *rod.Page and see this doc; but I cant manage to pull all the cookie from given url.

Here's the snippet:

func RodGetCookies(p *rod.Page, urls []string) {
	result := p.Call("Network.getCookies", cdp.Object{
		"urls": urls,
	})

	fmt.Printf("%+v", result)
}

...
    page, err := browser.PageE("https://github.com")
	if err != nil {
		panic(err)
	}

	RodGetCookies(page, []string{"*.github.com"})
...

the output:

{"cookies":[]}

thank you

We don't need full type support, a better way to do it is to add optional type helper to help us type the json objects that chrome uses.

By this optional way, the user can still interact with the latest chrome API even if the type helper is outdated.

The schema to use is on this page: GET /json/protocol/

The way https://github.com/mafredri/cdp or https://github.com/chromedp/chromedp did it is not good enough. Their implementations are tightly coupled with their high-level lib. Therefore their type is hard to be used as a standalone lib when someone only wants to use the type to generate a request JSON payload.

Thanks for your program. It's very easy to use.
Now I don't know how to pass cookies in array JSON format to page.call. page.Call("Network.setCookies",cdp.Array{})

arrayJson=`[
    {
        "domain": ".test.com",
        "expirationDate": 158255720,
        "hostOnly": true,
        "httpOnly": true,
        "name": "datr",
        "path": "/",
        "sameSite": "no_restriction",
        "secure": false,
        "session": false,
        "storeId": "0",
        "value": "5798798Q2B-",
        "id": 1
    },
    {
        "domain": ".test.com",
        "expirationDate": 15985720,
        "hostOnly": true,
        "httpOnly": true,
        "name": "bs",
        "path": "/",
        "sameSite": "no_restriction",
        "secure": false,
        "session": false,
        "storeId": "0",
        "value": "8DfXXcNRyf9zIM57nG4Maa1c",
        "id": 2
    }
]`

Put a launch service inside the docker image ysmood/rod to help launch chrome programmatically via a port.

The service will proxy this port to the real browser devtools port.

Good morning, sir,

I would like to make a system that allows the user to solve ReCaptcha himself. I've already taken the iFrame, but I'd like to know if it's possible to display only that (e.g. embed it in an electron app, or just display the captcha alone).

Here is my current code:

frame01 := page.Timeout(3 * time.Minute).ElementX("/html/body/div[3]/div[2]/iframe").Frame()
// Doesn't work
frame01.WindowFullscreen()

Thanks for your help!

Traditionally we can use a dedicated HTTP proxy server to proxy all requests and hijack whatever we want. But when it comes to HTTPs, it will be painful to get around it. This helper will make it totally transparent for the user to deal with any type of request.

The result will be something likes this:

go browser.Hijack("/user/:id", func(ctx rod.ProxyContext) (stop bool) {
    ctx.JSON(my_mock_user)
    return false
})

We will use Fetch domain. For now Fetch.takeResponseBodyAsStream can only handle the response as a stream, also websocket is not supported.

Content of your question.
I see the download example. How can I upload the file?

Previous commits accidentally removed the slow-motion helpers

For example, the overlay domain

https://chromedevtools.github.io/devtools-protocol/tot/Overlay/

or the legacy methods it replaces, like

https://chromedevtools.github.io/devtools-protocol/1-3/DOM/#method-highlightRect

I saw a Header() function at launcher, but I do't known how to use it.
I want to add a custom request header x-lpm-country: us in it.
May you add a example for it.
Thank you.

Could you give an example code of "PageAddScriptToEvaluateOnNewDocument"？
I want to change the value of "window.navigator.webdriver"！

The next line will cause automation to stop
page.Eval(() => Object.defineProperties(navigator, 'webdriver', {get: () => undefined}))

It's painful to debug headless mode mouse interaction if we can't visually see it from the screenshot.

Related to #46

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
I used this code for fake browser fingerprint ,

proto.PageEnable{}.Call(page)

proto.PageAddScriptToEvaluateOnNewDocument{
	Source: fingerprint,
}.Call(page)

the js code from https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin-stealth
I don't known why need proto.PageEnable ,but if I don't write it the proto.PageAddScriptToEvaluateOnNewDocument not works.
proto.PageAddScriptToEvaluateOnNewDocument only works on current page/tab not working on a link click new pages/tabs.

Describe the solution you'd like
A clear and concise description of what you want to happen.

Can you add a function to add JS script for all browser's pages/tabs something like hook browser's page or some network.
Thanks.

I want call addScriptToEvaluateOnNewDocument method when i open a new page,But it not working,
`out := page.Call("Page.addScriptToEvaluateOnNewDocument", cdp.Object{

"source" :
"window.navigator.chrome = window.chrome = {...window.chrome, runtime: {},};",
})`

how can i do it like puppeteer

`
const page = await browser.newPage()

await page.evaluateOnNewDocument(
"window.navigator.chrome = window.chrome = {...window.chrome, runtime: {},};"
);
`

Hello, i want to use proxy with auth, but i can't login.
When i add header in chromium with proxy, after i have an alert with username and password for auth, but if i user your method HandleAuth nothing happens, i need to click submit button to auth.
How can i do it with code?

P.S. Sorry for my English)

To prevent the race condition of using domains, we need a central hub to handle the status of domains.

Events, mouse, keyboard, etc all belong to this class.

Cannot get element inside iFrame. When last code line is invoked browser just hangs though i see element in it. When printing FrameID and isFrame I get some ID and true values so I assume that iFrame instance was created. How can I debug it to know the reason?

`url := launcher.New().
Headless(false).
Launch()

browser := rod.New().ControlURL(url).Connect()
defer browser.Close()

page := browser.Page("test_url")
fr := page.Element("#iframe_selector").Frame()

fmt.Println(fr.FrameID)
fmt.Println(fr.IsIframe())

fmt.Println(fr.Element("#element_selector").Describe().Type)`

For example, we can run this server on the cloud, so that users can use any language like Java or C++ to drive a chrome instance without the installation of any dependency.

The main problem of browserless is that they still require js, and you cannot share states between two scripts.

wayang is one try.

It may cause an unexcepted order or event stream.

https://github.com/ysmood/rod/blob/f9774865a0ddedd0914e2cacd7b83678ae398e3d/browser.go#L280-L282

Describe the bug
Can not download chrome at Launch()

To Reproduce
Do not set launcher Bin , then it will download chrome.I am from china, the download url will forward taobao.org , but it is not download success.

Expected behavior
opening zip archive for reading: creating reader: zip: not a valid zip file[rod/lib/launcher] Download chromium from: https://npm.taobao.org/mirrors/chromium-browser-snapshots/Win/757680/chrome-win.zip [rod/lib/launcher]
I saw the dictionary https://npm.taobao.org/mirrors/chromium-browser-snapshots/Win/, does not have the version 757680,the latest is 737027.

Rod Version: v0.26.1

OS: Windows

It'd be nice to also stream these to the monitor page. Perhaps as webp streams?

Use runtime.Stack(buf, true) after calling browser.Close() to exam if any orphan goroutine exists.

Currently, the WebSocket read will hang.

请教一下 FindByURL 这个方法怎么用呢，看了文档，新手没搞懂具体咋用，大佬方便时候加个简单实际案例吗，谢谢啦！

This package would feel more idiomatic if it didn't use this variadic panicing helper.

Describe the bug

Error when calling ScreenshotFullPage or ScreenshotE with FullPage=true

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1718b81]

goroutine 1 [running]:
github.com/ysmood/rod.(*Page).ScreenshotE(0xc001e16090, 0x18ce801, 0xc0004c9c18, 0x0, 0x0, 0x0, 0x0, 0x0)
	/Users/arturkondas/go/pkg/mod/github.com/ysmood/[email protected]/page.go:293 +0x281

To Reproduce

This is the current code:

body, err := page.ScreenshotE(true, &proto.PageCaptureScreenshot{})
if err != nil {
	log.Fatal(err)
}

Expected behavior
The action shouldn't fail.

Rod Version: v0.35.0

OS: Mac

May I propose an improvement? Can method to clear field be added?
Or if there is way to do it now - please tell me.

I tried to solve a drag captcha with rod:

The captcha will be loaded in an iframe, I can click buttons in the iframe but cannot drag.

	el:=page.Element(`#iframe`)
	iframe := el.WaitVisible().Frame()

        drag := frame.Element(`#drag`).WaitVisible()

        // copy from the Click 
	box, err := drag.ScrollIntoView().BoxE()
	if err != nil {
		panic(err)
	}

	x := box.Left + box.Width/2
	y := box.Top + box.Height/2

	err = iframe.Mouse.MoveE(x, y, 1)
	if err != nil {
		panic(err)
	}

       iframe.Mouse.Down(proto.InputMouseButtonLeft)
       iframe.Mouse.Move(1, 0)
       iframe.Mouse.Move(1, 0)
       iframe.Mouse.Move(2, 0)
       ...
       iframe.Mouse.Up(proto.InputMouseButtonLeft)

       

It seemed that only the Down and Up success, but the Move do nothing with no error.

Any idea? Thanks!

Browser.Close is not enough to stop chrome inside docker.

Is there a way in which we could get a screenshot of the whole page full size, not a screenshot of the window size?

Describe the bug
Some times Chrome will magically freeze to response rpc. It might be related to this ticket. Not sure if we can get around it in Rod.

To Reproduce
Not yet be able to stably reproduce it.

Expected behavior
Chrome should crash or always response a rpc.

The FAQ states:

Q: Does it support other browsers like Firefox or Edge

Rod should work with any browser that supports Chrome DevTools Protocol. For now, Firefox is supporting this protocol, and Edge will adopt chromium as their backend, so it seems like most major browsers will support it in the future except for Safari.

So, I wonder how to make Rod use other browsers such as Firefox? Or, Tor Browser (based on Firefox).

Why I need this? I need different IPs for each instance. It is possible to run multiple Tor Browsers and set the proxy but that requires extra memory (Rod chrome + Tor Browser). So, it would be nice to directly run Rod on top of Tor Browser.

I checked that chrome doesn't support Socks5 proxy with auth, but I use http proxy it is still not working.

proxyString:="http://username:password@ip:port"
Set("proxy-server", proxyString) 

If I remove username and pasword, the page show a alert to input username and password,is there any simple method to use proxy with auth.

Why Describe().Value return empty, but Attributes[value] return real value?

The test is not stable. I guess when CPU is slow, the scroll input cannot scroll the page to the place we want?