Comments (53)
WARNING prerender.app._render:99 Temporary browser failure: , retry rendering https://www.apple.com in 1s
Seems like that prerender can't connect to your headless Chrome instance.
What's your Chrome version? How did you start it?
from chrome-prerender.
wow, fast response đ
it works fine with http based URL, for example if I do
curl -v http://myserver.com:8000/http://www.somesite.com
then it works fine, just doesn't like https
chrome is Chromium 59.0.3071.115, it's started like this chrome --headless --remote-debugging-port=9222 --disable-gpu "about:blank"
as per your instructions in README
from chrome-prerender.
Might related to #13
from chrome-prerender.
Can you try prerender https://www.apple.com/mac/
?
from chrome-prerender.
unfortantely this does not help, we need it to run with --headless as this is in server environment for rendering our SPA's
from chrome-prerender.
The problem is with how we use layerPainted
event, not --headless
, I tried prerender https://www.apple.com/mac/
and it worked.
https://www.apple.com/
has something producing layerPainted
event constantly causing prerender timeout.
from chrome-prerender.
ok I inferred it was --headless switch from someone mentioning it on the ticket link you sent. so what you are saying its layerPainted?, but how come I can do this chrome --headless --disable-gpu --dump-dom https://www.apple.com
it works fine.
also other https targets have same problem, for example if I do
curl -v http://myserver.com:8000/https://www.google.com
same problem
it seems something else is problematic with https and how prerender works with this possibly
from chrome-prerender.
Because in prerender it tries to infer when the page has done rendering. There are several methods:
- evaluate
window.prerenderReady == true
- count finished network requests
- wait for DOM tree has no changes in 1s
The last one can be problematic when the page constantly modify DOM like Apple homepage do. (If you SPA don't do that it should work fine)
from chrome-prerender.
That said, I would recommend you to try to prerender your SPA instead of Apple/Google page.
from chrome-prerender.
our SPA does not work as well, we were hoping we would be able to try it out instead of using our existing PhantomJS + Prerender.IO setup, which works fine.
from chrome-prerender.
So you SPA has the same issue?
from chrome-prerender.
We may have to look at puppeteer instead, https://github.com/GoogleChrome/puppeteer#readme
but we will likely need to re-implement some of functionality which you already have here
from chrome-prerender.
Yes our SPA has same issue
from chrome-prerender.
@beyondcreed Try comment out https://github.com/bosondata/chrome-prerender/blob/master/prerender/chromerdp.py#L152 to see if it helps. Thanks!
I got it work for Apple homepage.
from chrome-prerender.
Since layerPainted
events are problematic, I removed it in master. Try:
pip install -U https://github.com/bosondata/chrome-prerender/archive/master.zip
to test it.
from chrome-prerender.
If I comment line 152 in chromerdp.py same problem. Will try your new version now as well.
from chrome-prerender.
If you encountered the same issue again, please post the logs here. Thanks!
from chrome-prerender.
ok trying now
Collecting https://github.com/bosondata/chrome-prerender/archive/master.zip
Downloading https://github.com/bosondata/chrome-prerender/archive/master.zip
\ 20kB 1.5MB/s
Requirement already up-to-date: websockets in /usr/local/lib/python3.6/site-packages (from prerender==0.8.1)
Requirement already up-to-date: aiohttp in /usr/local/lib/python3.6/site-packages (from prerender==0.8.1)
Requirement already up-to-date: ujson in /usr/local/lib/python3.6/site-packages (from prerender==0.8.1)
Requirement already up-to-date: sanic>=0.5.4 in /usr/local/lib/python3.6/site-packages (from prerender==0.8.1)
Requirement already up-to-date: raven in /usr/local/lib/python3.6/site-packages (from prerender==0.8.1)
Requirement already up-to-date: raven-aiohttp in /usr/local/lib/python3.6/site-packages (from prerender==0.8.1)
Requirement already up-to-date: diskcache in /usr/local/lib/python3.6/site-packages (from prerender==0.8.1)
Requirement already up-to-date: chardet in /usr/local/lib/python3.6/site-packages (from aiohttp->prerender==0.8.1)
Requirement already up-to-date: multidict>=2.1.4 in /usr/local/lib/python3.6/site-packages (from aiohttp->prerender==0.8.1)
Requirement already up-to-date: async_timeout>=1.2.0 in /usr/local/lib/python3.6/site-packages (from aiohttp->prerender==0.8.1)
Requirement already up-to-date: yarl>=0.11 in /usr/local/lib/python3.6/site-packages (from aiohttp->prerender==0.8.1)
Requirement already up-to-date: httptools>=0.0.9 in /usr/local/lib/python3.6/site-packages (from sanic>=0.5.4->prerender==0.8.1)
Requirement already up-to-date: uvloop>=0.5.3 in /usr/local/lib/python3.6/site-packages (from sanic>=0.5.4->prerender==0.8.1)
Requirement already up-to-date: aiofiles>=0.3.0 in /usr/local/lib/python3.6/site-packages (from sanic>=0.5.4->prerender==0.8.1)
Requirement already up-to-date: contextlib2 in /usr/local/lib/python3.6/site-packages (from raven->prerender==0.8.1)
Installing collected packages: prerender
Found existing installation: prerender 0.8.1
Uninstalling prerender-0.8.1:
Successfully uninstalled prerender-0.8.1
Running setup.py install for prerender ... done
Successfully installed prerender-0.8.1
from chrome-prerender.
prerender DEBUG=1
2017-08-16 19:59:22 - (sanic)[DEBUG]:
\u2584\u2584\u2584\u2584\u2584
\u2580\u2580\u2580\u2588\u2588\u2588\u2588\u2588\u2588\u2584\u2584\u2584 _______________
\u2584\u2584\u2584\u2584\u2584 \u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2584 /
\u2580\u2580\u2580\u2580\u2588\u2588\u2588\u2588\u2588\u258c \u2580\u2590\u2584 \u2580\u2590\u2588 | Gotta go fast! |
\u2580\u2580\u2588\u2588\u2588\u2588\u2588\u2584\u2584 \u2580\u2588\u2588\u2588\u2588\u2588\u2588\u2584\u2588\u2588 | _________________/
\u2580\u2584\u2584\u2584\u2584\u2584 \u2580\u2580\u2588\u2584\u2580\u2588\u2550\u2550\u2550\u2550\u2588\u2580 |/
\u2580\u2580\u2580\u2584 \u2580\u2580\u2588\u2588\u2588 \u2580 \u2584\u2584
\u2584\u2588\u2588\u2588\u2580\u2580\u2588\u2588\u2584\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2584 \u2584\u2580\u2580\u2580\u2580\u2580\u2580\u2588\u258c
\u2588\u2588\u2580\u2584\u2584\u2584\u2588\u2588\u2580\u2584\u2588\u2588\u2588\u2580 \u2580\u2580\u2588\u2588\u2588\u2588 \u2584\u2588\u2588
\u2584\u2580\u2580\u2580\u2584\u2588\u2588\u2584\u2580\u2580\u258c\u2588\u2588\u2588\u2588\u2592\u2592\u2592\u2592\u2592\u2592\u2588\u2588\u2588 \u258c\u2584\u2584\u2580
\u258c \u2590\u2580\u2588\u2588\u2588\u2588\u2590\u2588\u2588\u2588\u2592\u2592\u2592\u2592\u2592\u2590\u2588\u2588\u258c
\u2580\u2584\u2584\u2584\u2584\u2580 \u2580\u2580\u2588\u2588\u2588\u2588\u2592\u2592\u2592\u2592\u2584\u2588\u2588\u2580
\u2580\u2580\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2580
\u2584\u2584\u2588\u2588\u2580\u2588\u2588\u2588\u2588\u2588\u2588\u2580\u2588
\u2584\u2588\u2588\u2580 \u2580\u2580\u2580 \u2588
\u2584\u2588 \u2590\u258c
\u2584\u2584\u2584\u2584\u2588\u258c \u2580\u2588\u2584\u2584\u2584\u2584\u2580\u2580\u2584
\u258c \u2590 \u2580\u2580\u2584\u2584\u2584\u2580
\u2580\u2580\u2584\u2584\u2580
2017-08-16 19:59:22 - (sanic)[INFO]: Goin' Fast @ http://0.0.0.0:8000
2017-08-16 19:59:22,751 INFO prerender.chromerdp.new_page:49 Created new page fc6548e1-041e-4501-b99b-645c8a871d19
2017-08-16 19:59:22,760 INFO prerender.chromerdp.new_page:49 Created new page cd1a4b9b-2283-456e-b8af-f2e4c6729000
2017-08-16 19:59:22,767 INFO prerender.chromerdp.new_page:49 Created new page f1ccfb0b-0f85-4f18-8f16-6fd915fb6867
2017-08-16 19:59:22,775 INFO prerender.chromerdp.new_page:49 Created new page 29b4a610-e7d4-4110-b3fb-a6aa7c564814
2017-08-16 19:59:22,784 INFO prerender.chromerdp.new_page:49 Created new page 87cd694a-ebd7-43d3-9b5e-b54222692228
2017-08-16 19:59:22,787 INFO prerender.chromerdp.new_page:49 Created new page 64d444f4-744b-4085-a7d5-fa5ff4e9d1d9
2017-08-16 19:59:22,916 INFO prerender.chromerdp.new_page:49 Created new page 733978e8-6978-4cfb-ab49-d954322e9214
2017-08-16 19:59:23,050 INFO prerender.chromerdp.new_page:49 Created new page 9474f707-e168-494f-a21e-5121cd111cb4
2017-08-16 19:59:23,059 INFO prerender.chromerdp.new_page:49 Created new page b2e2de74-88c4-4461-807c-39087af49586
2017-08-16 19:59:23,065 INFO prerender.chromerdp.new_page:49 Created new page 7d2f682c-29e7-4f28-b424-5f0dcf4b85c6
2017-08-16 19:59:23,107 INFO prerender.chromerdp.new_page:49 Created new page 0f0ef699-a9af-4b6d-b350-7535dfca66b5
2017-08-16 19:59:23,116 INFO prerender.chromerdp.new_page:49 Created new page b2bd0362-fe01-4469-9d87-5df557037c75
2017-08-16 19:59:23,143 INFO prerender.chromerdp.new_page:49 Created new page dc69a01d-b80f-4768-8073-1c6450df8e84
2017-08-16 19:59:23,242 INFO prerender.chromerdp.new_page:49 Created new page 0a4a3701-5c11-4a17-a029-7abac531581c
2017-08-16 19:59:23,316 INFO prerender.chromerdp.new_page:49 Created new page de021820-7009-4608-a142-05d7b373992c
2017-08-16 19:59:23,325 INFO prerender.chromerdp.new_page:49 Created new page cb288c6c-3184-4e1b-a4f3-fbc54a956261
2017-08-16 19:59:23 - (sanic)[INFO]: Starting worker [95687]
2017-08-16 19:59:23,326 INFO sanic.serve:526 Starting worker [95687]
2017-08-16 19:59:30,409 INFO prerender.chromerdp.navigate:216 Page fc6548e1-041e-4501-b99b-645c8a871d19 [1] navigating to https://origin.thrinacia.com
2017-08-16 20:00:00,418 INFO websockets.protocol.fail_connection:618 Failing the WebSocket connection: 1006
2017-08-16 20:00:00,419 WARNING prerender.app._render:99 Temporary browser failure: , retry rendering https://origin.thrinacia.com in 1s
2017-08-16 20:00:00,447 ERROR asyncio.serve:527 Task exception was never retrieved
future: <Task finished coro=<Page._evaluate_prerender_ready() done, defined at /usr/local/lib/python3.6/site-packages/prerender/chromerdp.py:230> exception=AttributeError("'NoneType' object has no attribute 'send'",)>
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/prerender/chromerdp.py", line 232, in _evaluate_prerender_ready
res = await self.evaluate('window.prerenderReady == true')
File "/usr/local/lib/python3.6/site-packages/prerender/chromerdp.py", line 226, in evaluate
'params': {'expression': expr}
File "/usr/local/lib/python3.6/site-packages/prerender/chromerdp.py", line 176, in send
await self.websocket.send(json.dumps(payload))
AttributeError: 'NoneType' object has no attribute 'send'
2017-08-16 20:00:01,423 INFO prerender.chromerdp.navigate:216 Page cd1a4b9b-2283-456e-b8af-f2e4c6729000 [1] navigating to https://origin.thrinacia.com
2017-08-16 20:00:30 - (sanic)[ERROR]: Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sanic/server.py", line 143, in connection_timeout
raise RequestTimeout('Request Timeout')
sanic.exceptions.RequestTimeout: Request Timeout
2017-08-16 20:00:30,415 ERROR sanic.log:104 Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sanic/server.py", line 143, in connection_timeout
raise RequestTimeout('Request Timeout')
sanic.exceptions.RequestTimeout: Request Timeout
2017-08-16 20:00:30,419 INFO websockets.protocol.fail_connection:618 Failing the WebSocket connection: 1006
2017-08-16 20:00:30,420 WARNING prerender.app.handle_request:201 Got 504 for https://origin.thrinacia.com in 60012ms
2017-08-16 20:00:30 - (sanic)[ERROR]: Connection lost before response written @ ('96.50.156.110', 15034)
2017-08-16 20:00:30,420 ERROR sanic.write_response:267 Connection lost before response written @ ('96.50.156.110', 15034)
2017-08-16 20:00:30,423 ERROR asyncio.serve:527 Task exception was never retrieved
future: <Task finished coro=<Page._evaluate_prerender_ready() done, defined at /usr/local/lib/python3.6/site-packages/prerender/chromerdp.py:230> exception=AttributeError("'NoneType' object has no attribute 'send'",)>
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/prerender/chromerdp.py", line 232, in _evaluate_prerender_ready
res = await self.evaluate('window.prerenderReady == true')
File "/usr/local/lib/python3.6/site-packages/prerender/chromerdp.py", line 226, in evaluate
'params': {'expression': expr}
File "/usr/local/lib/python3.6/site-packages/prerender/chromerdp.py", line 176, in send
await self.websocket.send(json.dumps(payload))
AttributeError: 'NoneType' object has no attribute 'send'
from chrome-prerender.
still a problem
from chrome-prerender.
You might want to upgrade your Chromium to latest version to avoid any issue with devtools protocol the old version might have.
from chrome-prerender.
I believe that is latest I have in my packages, let me check
from chrome-prerender.
Chromium 59.0.3071.115 is latest I have http://www.freshports.org/www/chromium/
from chrome-prerender.
Thanks! I have reproduced the problem locally, I'll try to find out what is causing it.
from chrome-prerender.
I see your website set prerenderReady
to false
but never set it to true
which might caused the issue because we prefer to read window.prerenderReady
over other methods.
https://github.com/bosondata/chrome-prerender/blob/master/prerender/chromerdp.py#L242
from chrome-prerender.
ok, btw your current version is working with https://www.apple.com
from chrome-prerender.
I will let front-end developer know about prerenderReady. If there is any other issues they will follow up on this thread. thanks.
from chrome-prerender.
@beyondcreed Try the current master, it should work with your website
from chrome-prerender.
very nice, it worked.
from chrome-prerender.
thanks for fixing that quickly
from chrome-prerender.
For more accurate result, you should set prerenderReady
to true in JS when you know it's done rendering.
from chrome-prerender.
hm, just tried it on our main site, https://www.thrinacia.com/blog/post/express-checkout-feature
but it waits very long, then returns with error, debug is:
2017-08-16 20:43:29,951 INFO prerender.chromerdp.navigate:216 Page 135db0bc-8e9d-495b-b50e-10f9f4905de8 [1] navigating to https://www.thrinacia.com/blog/post/express-checkout-feature
2017-08-16 20:44:29 - (sanic)[ERROR]: Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sanic/server.py", line 143, in connection_timeout
raise RequestTimeout('Request Timeout')
sanic.exceptions.RequestTimeout: Request Timeout
2017-08-16 20:44:29,971 ERROR sanic.log:104 Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sanic/server.py", line 143, in connection_timeout
raise RequestTimeout('Request Timeout')
sanic.exceptions.RequestTimeout: Request Timeout
2017-08-16 20:44:29,971 WARNING prerender.app.handle_request:201 Got 504 for https://www.thrinacia.com/blog/post/express-checkout-feature in 60028ms
2017-08-16 20:44:29 - (network)[INFO][96.50.156.110:57234]:
from chrome-prerender.
Because counting network requests isn't very reliable when you have some external service like disqus loading lots of stuffs.
Turn on DEBUG=1
you will see that the page sends over 160 network requests. Manually set prerenderReady
to true
should resolve this issue.
from chrome-prerender.
yeah, I didn't write this code, talking to front-end devs to see if we can resolve, thanks for info
from chrome-prerender.
it works ok in our existing phantomJS setup with prerender.io, but here it hangs, likely due to what you are saying.
from chrome-prerender.
@beyondcreed I tried it locally, it works sometimes, hangs sometimes, depends on how long it takes to finish all the network requests. prerender has default timeout set to 30s, you can set envvar PRERENDER_TIMEOUT=xx
to increase/decrease it
from chrome-prerender.
I think prerender.io can render it is because they disabled image loading while we didn't. #20
from chrome-prerender.
the problem is some sites like linkedin, facebook, etc. don't wait long, max out at 10s when crawling, so output generally needs to be pre-rendered and served relatively fast. I tried it on linkedin and it times out. going to look at #20
from chrome-prerender.
I looked at your website, I think it has too many javascript and css files to load, might be worth to use tools like webpack to pack vendor js/css together to reduce number of requests.
from chrome-prerender.
we already pack it (minify and build checksums), also static resources are served off CDN.
I am trying chrome with --blink-settings=imagesEnabled=false
from chrome-prerender.
still times out
from chrome-prerender.
we already pack it (minify and build checksums),
You might be doing something wrong, this is a lot of scripts to load, ideally they should be lazy loaded on demand.
from chrome-prerender.
On my laptop, it takes Chrome at least 10+s to load the page already. (with cache disabled)
from chrome-prerender.
hi, I didn't write the code, but front-end developers who did, pack it using gulp process and generate checksums so we can bust cache (you can see code is already minified if you curl the URL, e.g. spaces are removed, etc.). Front-end dev mentioned due to legacy angular framework being used code could not be easily lazily loaded, we tried it before quite a few times and there were many issues with this.
from chrome-prerender.
phantomjs with prerender.io renders it quite fast, so I am just surprised to see this timing out, that is all
from chrome-prerender.
also if you are loading resources from overseas (china, etc.) it may take longer to load due to cdn nodes not located there, on my chrome load is at 2.51s
from chrome-prerender.
I understand.
I have done lazy loading legacy angular framework (1.x) with webpack before, so if you guys have issues I might be able to help. :-)
with --blink-settings=imagesEnabled=false
I can render it within 12s so I think it should be better for your network?
from chrome-prerender.
22233d4#diff-b987f64b4374fdef9d36193a6f11b3ddR252
Becasue we sleep 0.2s there, manually setting prerenderReady
to true
might save you 2s (10 iterations).
from chrome-prerender.
ok, I can message you privately on this, may need to discuss things further. I'll email you on how we can connect
from chrome-prerender.
ok sent you email and slack invite, thanks
from chrome-prerender.
Hi. I upgraded to Chromium 60.0.3112.101. However same problem still persists. I run the chrome as follows: chrome --blink-settings=imagesEnabled=false --headless --remote-debugging-port=9222 --disable-gpu "about:blank"
Then, I run prerender as: prerender DEBUG=1
. After 2 requests, the prerender will stop working. I may be wrong, but it does not seem like this is chrome issue, instead it looks like something is wrong with prerender code as multiple chrome versions are not working but when I dump DOM with chrome directly it works without a problem (I can do it more then 2 times).
On console error log I am seeing
2017-08-19 15:01:53,544 INFO prerender.chromerdp.navigate:216 Page 77d8643b-7129-420f-9fd2-3000b52648f8 [1] navigating to https://www.thrinacia.com/blog/post/express-checkout-feature
2017-08-19 15:02:53 - (sanic)[ERROR]: Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sanic/server.py", line 143, in connection_timeout
raise RequestTimeout('Request Timeout')
sanic.exceptions.RequestTimeout: Request Timeout
2017-08-19 15:02:53,829 ERROR sanic.log:104 Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sanic/server.py", line 143, in connection_timeout
raise RequestTimeout('Request Timeout')
sanic.exceptions.RequestTimeout: Request Timeout
2017-08-19 15:02:53,829 WARNING prerender.app.handle_request:201 Got 504 for https://www.thrinacia.com/blog/post/express-checkout-feature in 60286ms
from chrome-prerender.
I am also on slack if you need to discuss more. Thanks.
from chrome-prerender.
@beyondcreed I left message on slack.
from chrome-prerender.
Related Issues (20)
- compare to https://github.com/prerender/prerender, which features are supported? HOT 3
- Use layerPainted event to update _last_active_time may lead to timeout. HOT 20
- Reset/renew cache on POST request
- Useragent compatability with nodejs prerender HOT 2
- beforeSend plugin HOT 3
- 诡éŽćŞćĽćŻĺŚäźćŻćéĺśćŞĺžčĺ´ HOT 1
- Invalid syntax error HOT 2
- removeScriptTags plugin HOT 1
- option to disable images HOT 5
- /browser/enable endpoint is not working if CONCURRENCY OS env is not set
- Close page on PRERENDER_TIMEOUT HOT 2
- Subdirs in S3 storage is always same HOT 2
- Remove <meta name="fragment"... from prerendered pages HOT 5
- Prerender stops rendering after some time under high load HOT 8
- chromeless proxy support
- Install fail on Centos 7 using pip HOT 1
- Is it possible to use chrome-prerender as a squid parent in a proxy sandwich setup? HOT 5
- Can't view generated mhtml in chrome HOT 1
- Chrome healthcheck
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
đ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. đđđ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google â¤ď¸ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from chrome-prerender.