kailu3 / airbnb-scraper Goto Github PK

Hi,
I am trying to rebuild your scraper. I was very happy to find something like your approach combining scrapy and splash as it seems to be the only way to get hold of the JSON database. Many thanks for this!! Have you run your code recently? I don't see any bad HTML errors coming back. Docker is up and running but my output is empty. Maybe Airbnb has adapted the code recently.
Cheers
marcello

2020-04-17 23:51:22 [scrapy.utils.log] INFO: Scrapy 1.6.0 started (bot: airbnb_scraper) 2020-04-17 23:51:22 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0, libxml2 2.9.9, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 20.3.0, Python 3.7.6 (default, Jan 8 2020, 13:42:34) - [Clang 4.0.1 (tags/RELEASE_401/final)], pyOpenSSL 19.1.0 (OpenSSL 1.1.1d 10 Sep 2019), cryptography 2.8, Platform Darwin-19.0.0-x86_64-i386-64bit 2020-04-17 23:51:22 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'airbnb_scraper', 'CONCURRENT_REQUESTS_PER_DOMAIN': 16, 'CONCURRENT_REQUESTS_PER_IP': 16, 'DOWNLOAD_DELAY': 20, 'FEED_FORMAT': 'json', 'FEED_URI': 'Berlin.json', 'NEWSPIDER_MODULE': 'airbnb_scraper.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['airbnb_scraper.spiders'], 'USER_AGENT': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1'} 2020-04-17 23:51:22 [scrapy.extensions.telnet] INFO: Telnet Password: e654a180743fce0d 2020-04-17 23:51:22 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.memusage.MemoryUsage', 'scrapy.extensions.feedexport.FeedExporter', 'scrapy.extensions.logstats.LogStats'] 2020-04-17 23:51:22 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy_splash.SplashCookiesMiddleware', 'scrapy_splash.SplashMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2020-04-17 23:51:22 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy_splash.SplashDeduplicateArgsMiddleware', 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2020-04-17 23:51:22 [scrapy.middleware] INFO: Enabled item pipelines: [] 2020-04-17 23:51:22 [scrapy.core.engine] INFO: Spider opened 2020-04-17 23:51:22 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2020-04-17 23:51:22 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023 2020-04-17 23:51:23 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.airbnb.com/robots.txt> (referer: None) 2020-04-17 23:51:23 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.airbnb.com/api/v2/explore_tabs?_format=for_explore_search_web&_intents=p1&allow_override%5B%5D=&auto_ib=false&client_session_id=621cf853-d03e-4108-b717-c14962b6ab8b&currency=CAD&experiences_per_grid=20&fetch_filters=true&guidebooks_per_grid=20&has_zero_guest_treatment=true&is_guided_search=true&is_new_cards_experiment=true&is_standard_search=true&items_per_grid=18&key=d306zoyjsyarp7ifhu67rjxn52tv0t20&locale=en&luxury_pre_launch=false&metadata_only=false&query=Berlin&query_understanding_enabled=true&refinement_paths%5B%5D=%2Fhomes&s_tag=QLb9RB7g&search_type=FILTER_CHANGE&selected_tab_id=home_tab&show_groupings=true&supports_for_you_v3=true&timezone_offset=-240&version=1.5.6&price_min=100&price_max=200> (referer: None) 2020-04-17 23:51:24 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.airbnb.com/api/v2/explore_tabs?_format=for_explore_search_web&_intents=p1&allow_override%5B%5D=&auto_ib=false&client_session_id=621cf853-d03e-4108-b717-c14962b6ab8b&currency=CAD&experiences_per_grid=20&fetch_filters=true&guidebooks_per_grid=20&has_zero_guest_treatment=true&is_guided_search=true&is_new_cards_experiment=true&is_standard_search=true&items_per_grid=18&key=d306zoyjsyarp7ifhu67rjxn52tv0t20&locale=en&luxury_pre_launch=false&metadata_only=false&query=Berlin&query_understanding_enabled=true&refinement_paths%5B%5D=%2Fhomes&s_tag=QLb9RB7g&search_type=FILTER_CHANGE&selected_tab_id=home_tab&show_groupings=true&supports_for_you_v3=true&timezone_offset=-240&version=1.5.6&price_min=100&price_max=200> (referer: None) Traceback (most recent call last): File "/Users/marcelwieting/opt/anaconda3/lib/python3.7/site-packages/scrapy/utils/defer.py", line 102, in iter_errback yield next(it) File "/Users/marcelwieting/opt/anaconda3/lib/python3.7/site-packages/scrapy_splash/middleware.py", line 156, in process_spider_output for el in result: File "/Users/marcelwieting/opt/anaconda3/lib/python3.7/site-packages/scrapy/spidermiddlewares/offsite.py", line 29, in process_spider_output for x in result: File "/Users/marcelwieting/opt/anaconda3/lib/python3.7/site-packages/scrapy/spidermiddlewares/referer.py", line 339, in <genexpr> return (_set_referer(r) for r in result or ()) File "/Users/marcelwieting/opt/anaconda3/lib/python3.7/site-packages/scrapy/spidermiddlewares/urllength.py", line 37, in <genexpr> return (r for r in result or () if _filter(r)) File "/Users/marcelwieting/opt/anaconda3/lib/python3.7/site-packages/scrapy/spidermiddlewares/depth.py", line 58, in <genexpr> return (r for r in result or () if _filter(r)) File "/Volumes/Elements Marcel Wieting/airbnb3/airbnb_scraper/spiders/airbnb.py", line 105, in parse_id data_dict[room_id]['price'] = home.get('pricing_quote').get('rate').get('amount') AttributeError: 'NoneType' object has no attribute 'get' 2020-04-17 23:51:24 [scrapy.core.engine] INFO: Closing spider (finished) 2020-04-17 23:51:24 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 1356, 'downloader/request_count': 2, 'downloader/request_method_count/GET': 2, 'downloader/response_bytes': 34990, 'downloader/response_count': 2, 'downloader/response_status_count/200': 2, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2020, 4, 17, 21, 51, 24, 9263), 'log_count/DEBUG': 2, 'log_count/ERROR': 1, 'log_count/INFO': 9, 'memusage/max': 66531328, 'memusage/startup': 66531328, 'response_received_count': 2, 'robotstxt/request_count': 1, 'robotstxt/response_count': 1, 'robotstxt/response_status_count/200': 1, 'scheduler/dequeued': 1, 'scheduler/dequeued/memory': 1, 'scheduler/enqueued': 1, 'scheduler/enqueued/memory': 1, 'spider_exceptions/AttributeError': 1, 'start_time': datetime.datetime(2020, 4, 17, 21, 51, 22, 989936)} 2020-04-17 23:51:24 [scrapy.core.engine] INFO: Spider closed (finished) (base) MacBook-Pro:airbnb_scraper marcelwieting$