manivannanmurugavel / selenium-python-aws-lambda Goto Github PK

I'm having problems running test events. In my test event I have the following

{
 "url":"www.google.com"
}

when I run test it gives me the following message

{
  "errorMessage": "Message: unknown error: unhandled inspector error: {\"code\":-32000,\"message\":\"Cannot navigate to invalid URL\"}\n  (Session info: headless chrome=62.0.3202.94)\n  (Driver info: chromedriver=2.35.528139 (47ead77cb35ad2a9a83248b292151462a66cd881),platform=Linux 4.14.88-90.76.amzn2.x86_64 x86_64)\n",
  "errorType": "WebDriverException",
  "stackTrace": [
    [
      "/var/task/lambda_function.py",
      49,
      "lambda_handler",
      "driver.get(event['url'])"
    ],
    [
      "/var/task/selenium/webdriver/remote/webdriver.py",
      332,
      "get",
      "self.execute(Command.GET, {'url': url})"
    ],
    [
      "/var/task/selenium/webdriver/remote/webdriver.py",
      320,
      "execute",
      "self.error_handler.check_response(response)"
    ],
    [
      "/var/task/selenium/webdriver/remote/errorhandler.py",
      242,
      "check_response",
      "raise exception_class(message, screen, stacktrace)"
    ]
  ]
}

along with

START RequestId: ce5f4ee0-0263-484e-95cd-3909c4390241 Version: $LATEST
Starting google.com
Message: unknown error: unhandled inspector error: {"code":-32000,"message":"Cannot navigate to invalid URL"}
  (Session info: headless chrome=62.0.3202.94)
  (Driver info: chromedriver=2.35.528139 (47ead77cb35ad2a9a83248b292151462a66cd881),platform=Linux 4.14.88-90.76.amzn2.x86_64 x86_64)
: WebDriverException
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 49, in lambda_handler
    driver.get(event['url'])
  File "/var/task/selenium/webdriver/remote/webdriver.py", line 332, in get
    self.execute(Command.GET, {'url': url})
  File "/var/task/selenium/webdriver/remote/webdriver.py", line 320, in execute
    self.error_handler.check_response(response)
  File "/var/task/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: unhandled inspector error: {"code":-32000,"message":"Cannot navigate to invalid URL"}
  (Session info: headless chrome=62.0.3202.94)
  (Driver info: chromedriver=2.35.528139 (47ead77cb35ad2a9a83248b292151462a66cd881),platform=Linux 4.14.88-90.76.amzn2.x86_64 x86_64)


END RequestId: ce5f4ee0-0263-484e-95cd-3909c4390241
REPORT RequestId: ce5f4ee0-0263-484e-95cd-3909c4390241	Duration: 5548.74 ms	Billed Duration: 5600 ms 	Memory Size: 256 MB	Max Memory Used: 198 MB	

Any idea what could be happening here? I followed all the steps properly as far as I can. Even tried updating the bin files and changing the permission rights.

sir, How can i run this code, can you tell me step by step because every time this code shows me error.

Hello,

I am testing the code on my mac and it returns this information.
Is the code made for pc ?

Hey there. I tried to make it work. If I just upload your ZIP to lambda and run the function, it works.
So I decided to change a bit of code in lambda_function.py, and that's only to edit line 49,
driver.get('http://www.geneva.info/')
after this change, I pack it back to ZIP, upload and run test. I get this error:

START RequestId: f6bd2ac2-e78c-11e8-86d6-19ee2b8cd09b Version: $LATEST
Starting google.com
Message: 'chromedriver' executable may have wrong permissions. Please see https://sites.google.com/a/chromium.org/chromedriver/home
: WebDriverException
Traceback (most recent call last):
  File "/var/task/lambda_function.py", line 46, in lambda_handler
    driver = webdriver.Chrome(chrome_options=chrome_options)
  File "/var/task/selenium/webdriver/chrome/webdriver.py", line 68, in __init__
    self.service.start()
  File "/var/task/selenium/webdriver/common/service.py", line 88, in start
    os.path.basename(self.path), self.start_error_message)
selenium.common.exceptions.WebDriverException: Message: 'chromedriver' executable may have wrong permissions. Please see https://sites.google.com/a/chromium.org/chromedriver/home

I use Selenium on Windows, but I am fairly new to AWS. What is a problem here and how to make it work?
Thank you.

Hello Manivannan,
Thanks for sharing the code, in the first instance I had no problems, but to the following code fragment of "lambda_function.py":

Original:
    driver = webdriver.Chrome (chrome_options = chrome_options)
    page_data = ""
    if 'url' en event.keys ():
        driver.get (evento ['url'])
        page_data = driver.page_source
        print (page_data)
    driver.close ()
    volver page_data

New:

    driver = webdriver.Chrome (chrome_options = chrome_options)
    page_data = ""
    if 'url' en event.keys ():
        driver.get (evento ['url'])
        page_data = driver.title
        print (page_data)
    driver.close ()
    volver page_data

When testing the lambda I am getting the following error:

{
"errorMessage": "Message: 'chromedriver' executable may have wrong permissions. Please see https://sites.google.com/a/chromium.org/chromedriver/home\n",
"errorType": "WebDriverException",
"stackTrace": [
[
"/var/task/lambda_function.py",
46,
"lambda_handler",
"driver = webdriver.Chrome(chrome_options=chrome_options)"
],
[
"/var/task/selenium/webdriver/chrome/webdriver.py",
68,
"init",
"self.service.start()"
],
[
"/var/task/selenium/webdriver/common/service.py",
88,
"start",
"os.path.basename(self.path), self.start_error_message)"
]
]
}

Any ideas? Please your help.

Hi 🙂

I'm trying to run this file on lambda, but this error showed up

{ "errorMessage": "2021-02-16T01:48:40.456Z c9ee96ed-c1dd-46c5-bbed-fd12d512d186 Task timed out after 20.01 seconds" }

This is the log
START RequestId: c9ee96ed-c1dd-46c5-bbed-fd12d512d186 Version: $LATEST
Starting google.com
END RequestId: c9ee96ed-c1dd-46c5-bbed-fd12d512d186
REPORT RequestId: c9ee96ed-c1dd-46c5-bbed-fd12d512d186 Duration: 20009.43 ms Billed Duration: 20000 ms Memory Size: 256 MB Max Memory Used: 256 MB Init Duration: 56.57 ms
2021-02-16T01:48:40.456Z c9ee96ed-c1dd-46c5-bbed-fd12d512d186 Task timed out after 20.01 seconds

I didn't even unzip and re-zip the file, I uploaded the very file I downloaded here.
The runtime is Python 3.6 and I also created the PATHs.

If you have any clue, please do not hesitate to let me know. Thanks!

I downloaded the ZIP, uploaded to AWS, and the lambda ran successfully. If I simply download the zip > extract all > send to compressed file, it no longer works when I upload it. I tracked back to this after trying upload edited code and having it fail.

Is there some step/procedure that I am missing? Some other way of extracting/zipping? I tried through direct upload and S3.

After unzipping and rezipping I get the unable to import lambda_function error (when this python file is still in the folder and unchanged)

{
"errorMessage": "Unable to import module 'lambda_function'"
}

Thanks for any help!

Hello,

I am able to return the html for the webpage that I am trying to scrape. However, the div that contains all of the information that is shown on the webpage is empty. When I run the code on my local machine with a different chromedriver, it returns all of the information from the page. Would you happen to know what the issue might be?

Thanks

I would like to copy the downloaded file into S3. How to add boto3?

I tried uploading your ZIP in Python Lambda
Python version 3.7, 3.8, and 3.6 gives following error

{
"errorMessage": "Message: 'chromedriver' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home\n",
"errorType": "WebDriverException",
"stackTrace": [
" File "/var/task/lambda_function.py", line 46, in lambda_handler\n driver = webdriver.Chrome(chrome_options=chrome_options)\n",
" File "/var/task/selenium/webdriver/chrome/webdriver.py", line 68, in init\n self.service.start()\n",
" File "/var/task/selenium/webdriver/common/service.py", line 81, in start\n raise WebDriverException(\n"
]
}

Hello!

When i run the functions, the following error appears:

_

selenium.common.exceptions.WebDriverException: Message: 'chromedriver' executable may have wrong permissions.

_