nicolas-gervais / predicting-car-price-from-scraped-data Goto Github PK
View Code? Open in Web Editor NEWPicture and specifications scraper
Picture and specifications scraper
Hello Nicolas, I saw your repo and I think it is really cool! Right now I tried to run the picture scraper using python 3.7, and it gives me some error. May I know which python version are you using? Thanks!
hello, thanks for your codes, when i run the main.py, i met a problem like this:
scrape.py started running.
Traceback (most recent call last):
File "F:/BONC/vehicle/predicting-car-price-from-scraped-data/picture-scraper/scrape.py", line 92, in
run(sys.argv[1])
File "F:/BONC/vehicle/predicting-car-price-from-scraped-data/picture-scraper/scrape.py", line 78, in run
a = all_makes()
File "F:/BONC/vehicle/predicting-car-price-from-scraped-data/picture-scraper/scrape.py", line 20, in all_makes
for a in fetch(website, "/new-cars").find_all("a", {"class": "add-zip"}):
File "F:/BONC/vehicle/predicting-car-price-from-scraped-data/picture-scraper/scrape.py", line 15, in fetch
'lxml')
File "D:\python3.6\lib\site-packages\bs4_init_.py", line 245, in init
% ",".join(features))
bs4.FeatureNotFound: Couldn't find a tree builder with the features you requested: lxml. Do you need to install a parser library?
tag.py started running.
Traceback (most recent call last):
File "F:/BONC/vehicle/predicting-car-price-from-scraped-data/picture-scraper/tag.py", line 69, in
run()
File "F:/BONC/vehicle/predicting-car-price-from-scraped-data/picture-scraper/tag.py", line 11, in run
df = pd.read_csv('specs-and-pics.csv', dtype=str, index_col=0).T
File "D:\python3.6\lib\site-packages\pandas\io\parsers.py", line 676, in parser_f
return _read(filepath_or_buffer, kwds)
File "D:\python3.6\lib\site-packages\pandas\io\parsers.py", line 448, in _read
parser = TextFileReader(fp_or_buf, **kwds)
File "D:\python3.6\lib\site-packages\pandas\io\parsers.py", line 880, in init
self._make_engine(self.engine)
File "D:\python3.6\lib\site-packages\pandas\io\parsers.py", line 1114, in _make_engine
self._engine = CParserWrapper(self.f, **self.options)
File "D:\python3.6\lib\site-packages\pandas\io\parsers.py", line 1891, in init
self._reader = parsers.TextReader(src, **kwds)
File "pandas_libs\parsers.pyx", line 374, in pandas._libs.parsers.TextReader.cinit
File "pandas_libs\parsers.pyx", line 674, in pandas._libs.parsers.TextReader._setup_parser_source
FileNotFoundError: [Errno 2] File specs-and-pics.csv does not exist: 'specs-and-pics.csv'
Traceback (most recent call last):
File "F:/BONC/vehicle/predicting-car-price-from-scraped-data/picture-scraper/save.py", line 15, in
df = pd.read_csv('id_and_pic_url.csv')
File "D:\python3.6\lib\site-packages\pandas\io\parsers.py", line 676, in parser_f
return _read(filepath_or_buffer, kwds)
File "D:\python3.6\lib\site-packages\pandas\io\parsers.py", line 448, in _read
parser = TextFileReader(fp_or_buf, **kwds)
File "D:\python3.6\lib\site-packages\pandas\io\parsers.py", line 880, in init
self._make_engine(self.engine)
File "D:\python3.6\lib\site-packages\pandas\io\parsers.py", line 1114, in _make_engine
self._engine = CParserWrapper(self.f, **self.options)
File "D:\python3.6\lib\site-packages\pandas\io\parsers.py", line 1891, in init
self._reader = parsers.TextReader(src, **kwds)
File "pandas_libs\parsers.pyx", line 374, in pandas._libs.parsers.TextReader.cinit
File "pandas_libs\parsers.pyx", line 674, in pandas._libs.parsers.TextReader._setup_parser_source
FileNotFoundError: [Errno 2] File id_and_pic_url.csv does not exist: 'id_and_pic_url.csv'
and i have a question about the srape.py:
def fetch(page, addition=''):
return bs.BeautifulSoup(urlopen(Request(page + addition,
headers={'User-Agent': 'Opera/9.80 (X11; Linux i686; Ub'
'untu/14.10) Presto/2.12.388 Version/12.16'})).read(),
'lxml')
can you help me? thank you
I had to install lxml to get main.py to run, but it seems to run now. However, I get tons of errors saying *Problem with https://www.thecarconnection.com/specifications/" for each car and no images download. Do you have an idea as to what is going on?
Howdy; thanks for the dataset. You mentioned that there are a lot of duplicates. I used the imagehash library https://github.com/JohannesBuchner/imagehash to try to find dupes. There are about 32k (a little less than half the dataset) depending on what counts as a dupe. For example Acura_ILX_2014_28_16_110_15_4_70_55_179_39_FWD_5_4_4dr_Fdd.jpg
and Acura_ILX_2013_28_16_110_15_4_70_55_179_39_FWD_5_4_4dr_onJ.jpg
are exact (found by md5) duplicates, while Acura_MDX_2017_44_18_290_35_6_77_67_196_19_FWD_7_4_SUV_Jss.jpg
and Acura_MDX_2017_44_18_290_35_6_77_67_196_19_FWD_7_4_SUV_CgX.jpg
are resized (found by dhash) duplicates.
I made a little spreadsheet with the hashes/duplicate information in case anyone downloads the dataset and wants to clean it quickly (imagehashes.txt) -- the fastest way to use it would be something like this:
import csv
import os
records = []
with open('imagehashes.txt', newline='') as csv_file:
reader = csv.DictReader(csv_file)
for record in reader:
record['duplicate'] = record['duplicate'] == 'True'
records.append(record)
for record in records:
if record['duplicate']: # alternately, if record['matches'] == 'md5':
filename = record['filename']
print(f"deleting {filename}")
os.unlink(filename)
Hello Nicolas, I saw your work on Reddit a few weeks ago, and it's really cool, and I like your approach.
I finished writing an article on a similar approach to collect data. Still, I used the website Turo as a data source.
I want to share it with you because Turo could be an excellent addition to your data sources. For me, I will definitely add The Car Connection to my data sources ๐ธ .
Regards
Hello,
Firstly, thanks for sharing such a comprehensive dataset on car models!
You have mentioned on the reddit post on scraping high res photos is possible by modifying the scraper. Would you mind telling about the element id or get request that is related with the high res photos, so that I can modify the scraper script accordingly?
Bests
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.