andrewvy / chrome-remote-interface Goto Github PK

If usage of a deprecated API function call is detected, we should log out the deprecation message:

{
    "name": "addScriptToEvaluateOnLoad",
    "parameters": [
        {
            "name": "scriptSource",
            "type": "string"
        }
    ],
    "returns": [
        {
            "name": "identifier",
            "$ref": "ScriptIdentifier",
            "description": "Identifier of the added script."
        }
    ],
    "deprecated": true,
    "description": "Deprecated, please use addScriptToEvaluateOnNewDocument instead.",
    "experimental": true
}

Here, we can parse out the deprecated boolean attribute and log out the description.

Right now, you can run Page.enable to subscribe to events from the Page domain, but that won't forward any events to the requesting process.

We should add a wrapping function that will allow consumers to subscribe to different domain events.

Building on top of #6.

We could also provide a blocking call, more akin to GenServer.call with timeout functionality if an event hasn't fired after X seconds, for synchronous use-cases. (This is what we do for regular RPC calls anyways.)

{:ok, response} = ChromeRemoteInterface.Page.await(page_pid, "Page.frameStoppedLoading", timeout // 5000)

While I like the composability of HTTPipe, it does require an adapter specified in the config to be set on the consuming application:

config :httpipe, :adapter, HTTPipe.Adapters.Hackney

So I am using Chroxy and ChromeRemoteInterface to fetch and further parse HTML with Floki. If I get the outer HTML inside the iex I am able to fetch and then find the elements I desire, however, outside of iex, Floki is not able to find anything.

Here it goes a sample of the code I currently have:

ws_addr = Chroxy.connection()
{:ok, page} = ChromeRemoteInterface.PageSession.start_link(ws_addr)
ChromeRemoteInterface.RPC.Page.enable(page)
ChromeRemoteInterface.PageSession.subscribe(page, "Page.loadEventFired", self())
ChromeRemoteInterface.RPC.Page.navigate(page, %{url: url})
{:ok, dom} = ChromeRemoteInterface.RPC.DOM.getDocument(page)
nodeId  = dom["result"]["root"]["backendNodeId"]
{:ok, %{"result" => result}} = ChromeRemoteInterface.RPC.DOM.getOuterHTML(page, %{backendNodeId: nodeId})
pre_selected_content = Floki.find(result["outerHTML"], "div.productBoxTop")

Any suggestions?

Steps to reproduce:

ChromeLauncher.launch()
server = ChromeRemoteInterface.Session.new()
{:ok, page} = ChromeRemoteInterface.Session.new_page(server)

At this point you can navigate to http://localhost:9222 and confirm that there are two pages

ChromeRemoteInterface.Session.close_page(server, page["id"])

This last command returns false and by refreshing the Headless page in your browser you can confirm that there are still two pages open.

This is running against Canary. (Chrome 63)

In Chrome Beta (Chrome 62) for close_page/2 I get:

{:error, {:invalid, "T", 0}}

However, the page does close.

I cannot find documentation for the JSON API that this library is using to issue the close command. Maybe the API changed between 62 and 63?

https://github.com/andrewvy/chrome-remote-interface/blob/master/lib/page_session.ex#L102

This should be changed from having just a default 5s timeout to additionally allowing a user-customizable option.

This is a band-aid solution in which we should recommend users to use the asynchronous version for more expensive RPC operations.

Hi all, great library. I'm trying to subscribe to the Page.lifecycleEvent in the below snippet:

with {:ok, _data} <- RPC.Page.enable(pid),
     #  :ok <- PageSession.subscribe(pid, "Page.loadEventFired", self()),
     :ok <- PageSession.subscribe(pid, "Page.lifecycleEvent", self()),
     {:ok, data} <- RPC.Page.navigate(pid, %{url: url}) do
    IO.inspect(data)
end

From my understanding on navigation the lifecycle event should fire, I'm also trying to listen for network idle events which the lifecycleEvent covers. But in both cases it never does anything. The Page.loadEventFired works properly but that doesn't wait for network idle.

Should we use structs to represent the object types in the https://chromedevtools.github.io/devtools-protocol ?

Should benefit by providing some common use-case examples.

These examples are built on unstable version and is not guaranteed to be working in the near future!

Taking a screenshot, quick and dirty, error-prone

defmodule CriExample do
  require Logger

  def init() do
    server = ChromeRemoteInterface.Session.new()
    {:ok, pages} = ChromeRemoteInterface.Session.list_pages(server)
    List.first(pages)
  end

  def take_screenshot(page, url, file_path) do
    start_time = System.monotonic_time()
    {:ok, page_pid} = ChromeRemoteInterface.PageSession.start_link(page)

    ChromeRemoteInterface.PageSession.subscribe(page_pid, "Page.frameStoppedLoading")
    ChromeRemoteInterface.RPC.Page.enable(page_pid)
    ChromeRemoteInterface.RPC.Page.navigate(page_pid, %{url: url})

    receive do
      {:chrome_remote_interface, "Page.frameStoppedLoading", _payload} ->
        {:ok, %{"result" => %{"data" => data}}} = ChromeRemoteInterface.RPC.Page.captureScreenshot(page_pid)
        binary_data = Base.decode64!(data)
        {:ok, file} = File.open(file_path, [:write])
        IO.binwrite(file, binary_data)
        File.close(file)

        stop_time = System.monotonic_time()
        diff = System.convert_time_unit(stop_time - start_time, :native, :micro_seconds)

        Logger.info("url=#{url} file_path=#{file_path} total_time=#{formatted_diff(diff)}")
        ChromeRemoteInterface.PageSession.stop(page_pid)
    after
      10_000 ->
        :error
    end
  end

  defp formatted_diff(diff) when diff > 1000, do: [diff |> div(1000) |> Integer.to_string, "ms"]
  defp formatted_diff(diff), do: [Integer.to_string(diff), "µs"]
end

CriExample.init()
|> CriExample.take_screenshot("https://google.com", "test.png")

It would be nice to implement an API for launching Chrome, like chrome-launcher.

But, should this library have the ability to create new headless chrome processes? Or should a new library manage the actual headless process with something like Porcelain?

Hi,

I have chrome running as docker image with docker-compose.

If I try to connect to exposed to local machine port - it works fine, however if I try to connect using "compose" URL as host, getting just false.

Any hints?

I'm connecting like this:

    server = ChromeRemoteInterface.Session.new(host: "chrome")
    {:ok, pages} = ChromeRemoteInterface.Session.list_pages(server)

Getting this:

im_1  | ** (exit) an exception was raised:
im_1  |     ** (MatchError) no match of right hand side value: false

Docker image with chrome correctly expose needed default port. As I mentioned above, using host machine works fine.

{:ok, pages} = ChromeRemoteInterface.Session.list_page(server) should be changed to {:ok, pages} = ChromeRemoteInterface.Session.list_pages(server) i.e list_page to list_pages

Because of https://bugs.chromium.org/p/chromium/issues/detail?id=773572&can=1&q=stderr&colspec=ID%20Pri%20M%20Stars%20ReleaseBlock%20Component%20Status%20Owner%20Summary%20OS%20Modified

Chrome currently logs everything to stderr. Which means that the current logger is very noisy. I'd like to propose a parser that will determine if Logger.info/2, Logger.warn/2, Logger.debug/2, or Logger.error/2 should be used.

There does seem to be some useful information in the log:

"[1029/202012.957526:WARNING:dns_config_service_posix.cc(336)] Failed to read DnsConfig.\n"

in that example we can see WARNING which leads me to believe we can delegate this message to Logger.warn/2

"[1029/184906.750058:INFO:CONSOLE(25315)]

This could be sent to Logger.info/2 or up for debate. It appears INFO:CONSOLE is anything used for console.log in the browser. Not sure if this should be in the server log.

Following on from the discussion of System Environment Variable versus App Config for configuration of the Compile Time protocol target: #27

@andrewvy has expressed concern regarding the use of environment variable as a compile-time configuration option to signal which Chrome DevTool protocol version to be selected. There is merit in this objection so I am opening this issue for further discussion and resolution.

I am open to the use of config.ex (as @andrewvy points out this is what Logger uses to configure its macros) and would be happy enough with this to change as long as a sensible default remains when the configuration is not present on the inclusion of the CRI dependency 👍 .

Poison version 0.3.1 is referenced which is showing compiler warnings, and is now 1 major version behind.

I would recommend moving to using Jason as it is much commonly used (probably due to Phoenix using it now) ~ based on stats from Hex.pm

There should be a minimal pool implementation to manage checking out / checking in Pages.

Looking at either a poolboy or sbroker implementation.

A Session should be able to specify pool options for pages:

page_pool_options: [
  initial_size: 20, # Number of Pages to initially start the pool with 
  max_size: 100, # Max Number of Pages to the pool
  blocking: true # Should block the requesting process from 
]

Right now, the auto-generated documentation at the moment is very lack-luster. I think we should be able to generate cleaner docs with nicer param formatting to make this even more useful.

https://hexdocs.pm/chrome_remote_interface/ChromeRemoteInterface.RPC.Page.html

https://github.com/andrewvy/chrome-remote-interface/blob/master/lib/page_session.ex#L180

This doesn't remove the callback when a message is received for a callback. This leaves resolved callbacks in the state, when we should actually remove it!

For instance

page.setRequestInterception(true)

page.on('request', request => {
    if (request.resourceType() === 'image' ||
        request.resourceType() === 'stylesheet' ||
        request.resourceType() === 'stylesheet' ||
      )
      request.abort();
    else {
      request.continue();
    }
  });
  return page

After this basic API abstraction for wrapping the chrome-remote-interface, I'd like to tackle a new library that builds on top of these foundations.

• GenStage for funneling jobs to chrome headless workers
• Multiple chrome headless instances (Not sure if this is necessary if we have multiple pages)
• Distributed chrome headless job processing?

And naming a library is hard. Something about managing groups of "headless workers"..

It would be great if the RPC commands could take an option of ref or something like that to be used as the from param. Currently I do not see a way to override the from default value of self() other than calling the GenServer directly and bypassing the RPC API.

Currently there are no unit/integration tests which makes me sad. 😢

It would be nice to setup simple integration tests with several chrome versions to catch regressions.