andrewvy / chrome-remote-interface Goto Github PK
View Code? Open in Web Editor NEWElixir Client for the Chrome Debugger Protocol
Home Page: https://hexdocs.pm/chrome_remote_interface
Elixir Client for the Chrome Debugger Protocol
Home Page: https://hexdocs.pm/chrome_remote_interface
If usage of a deprecated API function call is detected, we should log out the deprecation message:
{
"name": "addScriptToEvaluateOnLoad",
"parameters": [
{
"name": "scriptSource",
"type": "string"
}
],
"returns": [
{
"name": "identifier",
"$ref": "ScriptIdentifier",
"description": "Identifier of the added script."
}
],
"deprecated": true,
"description": "Deprecated, please use addScriptToEvaluateOnNewDocument instead.",
"experimental": true
}
Here, we can parse out the deprecated
boolean attribute and log out the description
.
Right now, you can run Page.enable
to subscribe to events from the Page
domain, but that won't forward any events to the requesting process.
We should add a wrapping function that will allow consumers to subscribe to different domain events.
Building on top of #6.
We could also provide a blocking call, more akin to GenServer.call
with timeout functionality if an event hasn't fired after X
seconds, for synchronous use-cases. (This is what we do for regular RPC calls anyways.)
{:ok, response} = ChromeRemoteInterface.Page.await(page_pid, "Page.frameStoppedLoading", timeout // 5000)
While I like the composability of HTTPipe, it does require an adapter specified in the config to be set on the consuming application:
config :httpipe, :adapter, HTTPipe.Adapters.Hackney
So I am using Chroxy and ChromeRemoteInterface to fetch and further parse HTML with Floki. If I get the outer HTML inside the iex I am able to fetch and then find the elements I desire, however, outside of iex, Floki is not able to find anything.
Here it goes a sample of the code I currently have:
ws_addr = Chroxy.connection()
{:ok, page} = ChromeRemoteInterface.PageSession.start_link(ws_addr)
ChromeRemoteInterface.RPC.Page.enable(page)
ChromeRemoteInterface.PageSession.subscribe(page, "Page.loadEventFired", self())
ChromeRemoteInterface.RPC.Page.navigate(page, %{url: url})
{:ok, dom} = ChromeRemoteInterface.RPC.DOM.getDocument(page)
nodeId = dom["result"]["root"]["backendNodeId"]
{:ok, %{"result" => result}} = ChromeRemoteInterface.RPC.DOM.getOuterHTML(page, %{backendNodeId: nodeId})
pre_selected_content = Floki.find(result["outerHTML"], "div.productBoxTop")
Any suggestions?
Steps to reproduce:
ChromeLauncher.launch()
server = ChromeRemoteInterface.Session.new()
{:ok, page} = ChromeRemoteInterface.Session.new_page(server)
At this point you can navigate to http://localhost:9222 and confirm that there are two pages
ChromeRemoteInterface.Session.close_page(server, page["id"])
This last command returns false
and by refreshing the Headless page in your browser you can confirm that there are still two pages open.
This is running against Canary. (Chrome 63)
In Chrome Beta (Chrome 62) for close_page/2
I get:
{:error, {:invalid, "T", 0}}
However, the page does close.
I cannot find documentation for the JSON API that this library is using to issue the close command. Maybe the API changed between 62 and 63?
https://github.com/andrewvy/chrome-remote-interface/blob/master/lib/page_session.ex#L102
This should be changed from having just a default 5s timeout to additionally allowing a user-customizable option.
This is a band-aid solution in which we should recommend users to use the asynchronous version for more expensive RPC operations.
Hi all, great library. I'm trying to subscribe to the Page.lifecycleEvent in the below snippet:
with {:ok, _data} <- RPC.Page.enable(pid),
# :ok <- PageSession.subscribe(pid, "Page.loadEventFired", self()),
:ok <- PageSession.subscribe(pid, "Page.lifecycleEvent", self()),
{:ok, data} <- RPC.Page.navigate(pid, %{url: url}) do
IO.inspect(data)
end
From my understanding on navigation the lifecycle event should fire, I'm also trying to listen for network idle events which the lifecycleEvent covers. But in both cases it never does anything. The Page.loadEventFired works properly but that doesn't wait for network idle.
Should we use structs to represent the object types in the https://chromedevtools.github.io/devtools-protocol ?
Should benefit by providing some common use-case examples.
These examples are built on unstable version and is not guaranteed to be working in the near future!
Taking a screenshot, quick and dirty, error-prone
defmodule CriExample do
require Logger
def init() do
server = ChromeRemoteInterface.Session.new()
{:ok, pages} = ChromeRemoteInterface.Session.list_pages(server)
List.first(pages)
end
def take_screenshot(page, url, file_path) do
start_time = System.monotonic_time()
{:ok, page_pid} = ChromeRemoteInterface.PageSession.start_link(page)
ChromeRemoteInterface.PageSession.subscribe(page_pid, "Page.frameStoppedLoading")
ChromeRemoteInterface.RPC.Page.enable(page_pid)
ChromeRemoteInterface.RPC.Page.navigate(page_pid, %{url: url})
receive do
{:chrome_remote_interface, "Page.frameStoppedLoading", _payload} ->
{:ok, %{"result" => %{"data" => data}}} = ChromeRemoteInterface.RPC.Page.captureScreenshot(page_pid)
binary_data = Base.decode64!(data)
{:ok, file} = File.open(file_path, [:write])
IO.binwrite(file, binary_data)
File.close(file)
stop_time = System.monotonic_time()
diff = System.convert_time_unit(stop_time - start_time, :native, :micro_seconds)
Logger.info("url=#{url} file_path=#{file_path} total_time=#{formatted_diff(diff)}")
ChromeRemoteInterface.PageSession.stop(page_pid)
after
10_000 ->
:error
end
end
defp formatted_diff(diff) when diff > 1000, do: [diff |> div(1000) |> Integer.to_string, "ms"]
defp formatted_diff(diff), do: [Integer.to_string(diff), "µs"]
end
CriExample.init()
|> CriExample.take_screenshot("https://google.com", "test.png")
It would be nice to implement an API for launching Chrome, like chrome-launcher.
But, should this library have the ability to create new headless chrome processes? Or should a new library manage the actual headless process with something like Porcelain?
Hi,
I have chrome running as docker image with docker-compose.
If I try to connect to exposed to local machine port - it works fine, however if I try to connect using "compose" URL as host, getting just false
.
Any hints?
I'm connecting like this:
server = ChromeRemoteInterface.Session.new(host: "chrome")
{:ok, pages} = ChromeRemoteInterface.Session.list_pages(server)
Getting this:
im_1 | ** (exit) an exception was raised:
im_1 | ** (MatchError) no match of right hand side value: false
Docker image with chrome correctly expose needed default port. As I mentioned above, using host machine works fine.
{:ok, pages} = ChromeRemoteInterface.Session.list_page(server) should be changed to {:ok, pages} = ChromeRemoteInterface.Session.list_pages(server) i.e list_page to list_pages
Chrome currently logs everything to stderr. Which means that the current logger is very noisy. I'd like to propose a parser that will determine if Logger.info/2
, Logger.warn/2
, Logger.debug/2
, or Logger.error/2
should be used.
There does seem to be some useful information in the log:
"[1029/202012.957526:WARNING:dns_config_service_posix.cc(336)] Failed to read DnsConfig.\n"
in that example we can see WARNING
which leads me to believe we can delegate this message to Logger.warn/2
"[1029/184906.750058:INFO:CONSOLE(25315)]
This could be sent to Logger.info/2
or up for debate. It appears INFO:CONSOLE
is anything used for console.log
in the browser. Not sure if this should be in the server log.
Following on from the discussion of System Environment Variable versus App Config for configuration of the Compile Time protocol target: #27
@andrewvy has expressed concern regarding the use of environment variable as a compile-time configuration option to signal which Chrome DevTool protocol version to be selected. There is merit in this objection so I am opening this issue for further discussion and resolution.
I am open to the use of config.ex (as @andrewvy points out this is what Logger uses to configure its macros) and would be happy enough with this to change as long as a sensible default remains when the configuration is not present on the inclusion of the CRI dependency 👍 .
There should be a minimal pool implementation to manage checking out / checking in Pages.
Looking at either a poolboy or sbroker implementation.
A Session should be able to specify pool options for pages:
page_pool_options: [
initial_size: 20, # Number of Pages to initially start the pool with
max_size: 100, # Max Number of Pages to the pool
blocking: true # Should block the requesting process from
]
Right now, the auto-generated documentation at the moment is very lack-luster. I think we should be able to generate cleaner docs with nicer param formatting to make this even more useful.
https://hexdocs.pm/chrome_remote_interface/ChromeRemoteInterface.RPC.Page.html
https://github.com/andrewvy/chrome-remote-interface/blob/master/lib/page_session.ex#L180
This doesn't remove the callback when a message is received for a callback. This leaves resolved callbacks in the state, when we should actually remove it!
For instance
page.setRequestInterception(true)
page.on('request', request => {
if (request.resourceType() === 'image' ||
request.resourceType() === 'stylesheet' ||
request.resourceType() === 'stylesheet' ||
)
request.abort();
else {
request.continue();
}
});
return page
After this basic API abstraction for wrapping the chrome-remote-interface
, I'd like to tackle a new library that builds on top of these foundations.
And naming a library is hard. Something about managing groups of "headless workers"..
It would be great if the RPC commands could take an option of ref
or something like that to be used as the from
param. Currently I do not see a way to override the from
default value of self()
other than calling the GenServer directly and bypassing the RPC API.
Currently there are no unit/integration tests which makes me sad. 😢
It would be nice to setup simple integration tests with several chrome versions to catch regressions.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.