Use layerPainted event to update _last_active_time may lead to timeout. about chrome-prerender

[messense]

Thank you for reporting this, can you give an example page so I can test it?

[knuhol]

Is it the same? Let's try https://google.com:

2017-06-12 14:41:28,750 INFO prerender.chromerdp.navigate:205   Page 08a8d058-66e4-4bd6-ae57-6a5942381f6f [1] navigating to https://google.com
2017-06-12 14:41:58,765 INFO websockets.protocol.fail_connection:618   Failing the WebSocket connection: 1006 
2017-06-12 14:41:58,767 WARNING prerender.app._render:95    Temporary browser failure: , retry rendering https://google.com in 1s
2017-06-12 14:41:58,786 ERROR asyncio.serve:450   Task exception was never retrieved
future: <Task finished coro=<Page._evaluate_prerender_ready() done, defined at /Users/idolezalova/.pyenv/versions/3.6.1/lib/python3.6/site-packages/prerender/chromerdp.py:219> exception=AttributeError("'NoneType' object has no attribute 'send'",)>
Traceback (most recent call last):
  File "/Users/idolezalova/.pyenv/versions/3.6.1/lib/python3.6/site-packages/prerender/chromerdp.py", line 221, in _evaluate_prerender_ready
    res = await self.evaluate('window.prerenderReady == true')
  File "/Users/idolezalova/.pyenv/versions/3.6.1/lib/python3.6/site-packages/prerender/chromerdp.py", line 215, in evaluate
    'params': {'expression': expr}
  File "/Users/idolezalova/.pyenv/versions/3.6.1/lib/python3.6/site-packages/prerender/chromerdp.py", line 171, in send
    await self.websocket.send(json.dumps(payload))
AttributeError: 'NoneType' object has no attribute 'send'
2017-06-12 14:41:59,773 INFO prerender.chromerdp.navigate:205   Page 5702afd5-b794-4c0c-85f3-5817f2bcc59f [1] navigating to https://google.com
2017-06-12 14:42:28 - (sanic)[ERROR]: NoneType: None

2017-06-12 14:42:28,742 ERROR sanic.log:104   NoneType: None

2017-06-12 14:42:28,751 INFO websockets.protocol.fail_connection:618   Failing the WebSocket connection: 1006 
2017-06-12 14:42:28,751 WARNING prerender.app.handle_request:172   Got 504 for https://google.com in 60006ms
2017-06-12 14:42:28 - (sanic)[ERROR]: Connection lost before response written @ ('127.0.0.1', 62967)
2017-06-12 14:42:28,752 ERROR sanic.write_response:217   Connection lost before response written @ ('127.0.0.1', 62967)

[myrfy001]

Sorry for a late reply. I missed the mail sent to me 11 days ago.
Yes, the search engine is a good example.

[messense]

For https://google.com I can't reproduce the exception locally. What's your OS and Chrome version? @akarienta

[knuhol]

OS: macOS Sierra, version 10.12.5 (16F73)
Chrome: Version 59.0.3071.86 (Official) (64bit)

[messense]

Oh, I haven't try Chrome stable yet. I am using Chrome Canary for development. Chrome stable might miss some features that unstable/canary version has.

[knuhol]

@messense ok makes sense, I will try Canary and let you know, thanks

[myrfy001]

You can also try www.baidu.com， the biggest search engine in China.
I ran on Sierra too.

[messense]

@myrfy001 I have no problem rendering www.baidu.com too.

2017-06-12 21:35:00,284 INFO prerender.chromerdp.navigate:205   Page d1945e18-ff0b-4111-95cf-96611fa58975 [1] navigating to https://www.baidu.com
2017-06-12 21:35:01,017 WARNING prerender.chromerdp._on_log_entry_added:316   https://www.baidu.com/:705 console javascript log warning: A Parser-blocking, cross site (i.e. different eTLD+1) script, https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/static/protocol/https/global/js/all_async_search_0596852.js, is invoked via document.write. The network request for this script MAY be blocked by the browser in this or a future page load due to poor network connectivity. If blocked in this page load, it will be confirmed in a subsequent console message.See https://www.chromestatus.com/feature/5718547946799104 for more details.
2017-06-12 21:35:01,018 WARNING prerender.chromerdp._on_log_entry_added:316   https://www.baidu.com/:705 console javascript log warning: A Parser-blocking, cross site (i.e. different eTLD+1) script, https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/static/protocol/https/global/js/all_async_search_0596852.js, is invoked via document.write. The network request for this script MAY be blocked by the browser in this or a future page load due to poor network connectivity. If blocked in this page load, it will be confirmed in a subsequent console message.See https://www.chromestatus.com/feature/5718547946799104 for more details.
2017-06-12 21:35:02,634 INFO prerender.app.handle_request:161   Got 200 for https://www.baidu.com in 2355ms

[knuhol]

@messense It's absolutely the same for me with Version 61.0.3128.0 (Official) canary (64bit)

[knuhol]

@messense I have also the same problem to render www.baidu.com:

2017-06-12 15:38:33,258 INFO prerender.chromerdp.navigate:205   Page 4f5a3630-5014-4d85-b5ef-249096334e24 [1] navigating to https://www.baidu.com
2017-06-12 15:38:33,932 WARNING prerender.chromerdp._on_log_entry_added:316   https://www.baidu.com/:705 console javascript log warning: A Parser-blocking, cross site (i.e. different eTLD+1) script, https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/static/protocol/https/global/js/all_async_search_0596852.js, is invoked via document.write. The network request for this script MAY be blocked by the browser in this or a future page load due to poor network connectivity. If blocked in this page load, it will be confirmed in a subsequent console message.See https://www.chromestatus.com/feature/5718547946799104 for more details.
2017-06-12 15:38:33,933 WARNING prerender.chromerdp._on_log_entry_added:316   https://www.baidu.com/:705 console javascript log warning: A Parser-blocking, cross site (i.e. different eTLD+1) script, https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/static/protocol/https/global/js/all_async_search_0596852.js, is invoked via document.write. The network request for this script MAY be blocked by the browser in this or a future page load due to poor network connectivity. If blocked in this page load, it will be confirmed in a subsequent console message.See https://www.chromestatus.com/feature/5718547946799104 for more details.
2017-06-12 15:39:02 - (sanic)[ERROR]: NoneType: None

2017-06-12 15:39:02,242 ERROR sanic.log:104   NoneType: None

2017-06-12 15:39:02,250 INFO websockets.protocol.fail_connection:618   Failing the WebSocket connection: 1006 
2017-06-12 15:39:02,251 WARNING prerender.app.handle_request:172   Got 504 for https://www.baidu.com in 60011ms
2017-06-12 15:39:02 - (sanic)[ERROR]: Connection lost before response written @ ('127.0.0.1', 64642)
2017-06-12 15:39:02,252 ERROR sanic.write_response:217   Connection lost before response written @ ('127.0.0.1', 64642)
2017-06-12 15:39:02,398 ERROR asyncio.serve:450   Task exception was never retrieved
future: <Task finished coro=<Page._evaluate_prerender_ready() done, defined at /Users/idolezalova/.pyenv/versions/3.6.1/lib/python3.6/site-packages/prerender/chromerdp.py:219> exception=AttributeError("'NoneType' object has no attribute 'send'",)>
Traceback (most recent call last):
  File "/Users/idolezalova/.pyenv/versions/3.6.1/lib/python3.6/site-packages/prerender/chromerdp.py", line 221, in _evaluate_prerender_ready
    res = await self.evaluate('window.prerenderReady == true')
  File "/Users/idolezalova/.pyenv/versions/3.6.1/lib/python3.6/site-packages/prerender/chromerdp.py", line 215, in evaluate
    'params': {'expression': expr}
  File "/Users/idolezalova/.pyenv/versions/3.6.1/lib/python3.6/site-packages/prerender/chromerdp.py", line 171, in send
    await self.websocket.send(json.dumps(payload))
AttributeError: 'NoneType' object has no attribute 'send'

@myrfy001 Do you have the same error?

[messense]

@akarienta How did you launch Chrome Canary? I am using:

/Applications/Google\ Chrome\ Canary.app/Contents/MacOS/Google\ Chrome\ Canary --remote-debugging-port=9222

Make sure you didn't already opened a Chrome Canary before running the command in terminal.

[messense]

To debug it, start it with DEBUG=1 prerender

[knuhol]

@messense I go with command from your README:

/Applications/Google Chrome Canary.app/Contents/MacOS/Google\ Chrome\ Canary --headless --remote-debugging-port=9222 --disable-gpu "about:blank"

[knuhol]

@messense Ahahah, it's ok with your command:

/Applications/Google\ Chrome\ Canary.app/Contents/MacOS/Google\ Chrome\ Canary --remote-debugging-port=9222

So maybe just let's update the README file 😄

BTW it works in the last stable Chrome as well!

[messense]

Good to know. The readme command is for Linux server.

[knuhol]

@messense Ah I see. Very well then, thank you very much for your help!

[messense]

BTW, I would expect Chrome Canary to work with --headless, I guess It's broken for now. But it's unstable version anyway, breakages are kind of normal.

[knuhol]

I have the same problem again (with both google.com and baidu.com), even when using the command

/Applications/Google\ Chrome\ Canary.app/Contents/MacOS/Google\ Chrome\ Canary --remote-debugging-port=9222

I tried also the --headless flag. It seems like a problem with Chrome, it works on some runs, on others it doesn't.

[messense]

Removed layerPainted events in 280ca79