tmurakam / googleplay_dev_scraper

Google Play and Google Checkout scraper for developers

👋 Hi, I'm tmurakam

Profile

Programming language

• Java
• Kotlin
• Swift
• Objective-C
• C/C++
• C#
• Go
• JavaScript
• Ruby
• Python

Framework

• Spring Framework / Spring Boot

OS

• Linux
• Android
• iOS

Cloud

• AWS
• Azure
• Kubernetes

Other technologies

• Biometrics authentication
• Video analytics
• TCP/IP

Fun

• Piano

googleplay_dev_scraper's Issues

Optional STDERR URL printing?

Can you make STDERR echo of URL optional, or only done at a 'verbose' logging level?

It is causing our tests to print the url for every request. It adds a lot of noise to our tests.

I am happy to fix the problem and open a pull request. The problem is line 113 of lib/googleplay_dev_scraper/scraper.rb

Thanks!

Cant seek estimated Sales Report

Hi, I cant "scrape" the estimated Sales Report. The Url cant be found.
I'd guess the URL has changed. If I try to access the url directly in the browser I see a Google Error Page saying:
400. That’s an error.

The requested URL was not found on this server. That’s all we know.

Please find the Console Output of the scraper below:

fetch': 400 => Net::HTTPBadRequest for https://play.google.com/apps/publish/salesreport/download?report_date=2013_08&report_type=sales_report&dev_acc=00158313850196121289 -- unhandled response (Mechanize::ResponseCodeError) from /var/lib/gems/1.9.1/gems/mechanize-2.7.1/lib/mechanize/http/agent.rb:974:inresponse_redirect'
from /var/lib/gems/1.9.1/gems/mechanize-2.7.1/lib/mechanize/http/agent.rb:298:in fetch' from /var/lib/gems/1.9.1/gems/mechanize-2.7.1/lib/mechanize/http/agent.rb:974:inresponse_redirect'
from /var/lib/gems/1.9.1/gems/mechanize-2.7.1/lib/mechanize/http/agent.rb:298:in fetch' from /var/lib/gems/1.9.1/gems/mechanize-2.7.1/lib/mechanize/http/agent.rb:974:inresponse_redirect'
from /var/lib/gems/1.9.1/gems/mechanize-2.7.1/lib/mechanize/http/agent.rb:298:in fetch' from /var/lib/gems/1.9.1/gems/mechanize-2.7.1/lib/mechanize.rb:1272:inpost_form'
from /var/lib/gems/1.9.1/gems/mechanize-2.7.1/lib/mechanize.rb:539:in submit' from /var/lib/gems/1.9.1/gems/mechanize-2.7.1/lib/mechanize/form.rb:223:insubmit'
from /var/lib/gems/1.9.1/gems/mechanize-2.7.1/lib/mechanize/form.rb:229:in click_button' from /var/lib/gems/1.9.1/gems/googleplay_dev_scraper-1.0.2/lib/googleplay_dev_scraper/scraper_base.rb:55:intry_get'
from /var/lib/gems/1.9.1/gems/googleplay_dev_scraper-1.0.2/lib/googleplay_dev_scraper/scraper.rb:55:in get_estimated_sales_report' from /var/lib/gems/1.9.1/gems/googleplay_dev_scraper-1.0.2/bin/googleplay_dev_scraper:93:inestimated'
from /var/lib/gems/1.9.1/gems/googleplay_dev_scraper-1.0.2/bin/googleplay_dev_scraper:33:in main' from /var/lib/gems/1.9.1/gems/googleplay_dev_scraper-1.0.2/bin/googleplay_dev_scraper:122:in<top (required)>'
from /usr/local/bin/googleplay_dev_scraper:23:in load' from /usr/local/bin/googleplay_dev_scraper:23:in

'

Getting errors as of last week - cant login?

/usr/local/rvm/gems/ruby-2.1.1/gems/googleplay_dev_scraper-1.0.3/lib/googleplay_dev_scraper/scraper_base.rb:54:in try_get': undefined methodvalue=' for nil:NilClass (NoMethodError)

appstats csv download

When I try to run the appstats command, it creates me a corrupted zip file. Do you know what may be the problem?

License missing from gemspec

Some companies will only use gems with a certain license.
The canonical and easy way to check is via the gemspec
via e.g.

spec.license = 'MIT'
# or
spec.licenses = ['MIT', 'GPL-2']

Bundler now generates gems with a default 'MIT' license. There is even a License Finder
to help companies ensure all gems they use meet their licensing needs. This tool depends on license information being available in the gemspec.
Including a license in your gemspec is a good practice, in any case.

If you need help choosing a license, github has created a license picker tool

How did I find you?

I'm using a script to collect stats on gems, originally looking for download data, but decided to collect licenses too,
and make issues for gemspecs not specifying a license as a public service :)
So far it's going pretty well.
I've written a blog post about it

fetch: 403 => Net::HTTPForbidden

Any idea what would cause this? I've entered the details correctly.

The only thing I can think of is that I have multiple developer profiles tied to my Google login?

Thanks,

Steve

Appstats not working anymore

As of yesterday the appstats download is not working anymore. I'd guess the URL has changed. If I try to access the url directly in the browser I see a Google Error Page saying:
400. That’s an error.

The requested URL was not found on this server. That’s all we know.

Please finde the Console Output of the scraper below:

URL = https://play.google.com/apps/publish/v2/statistics/download?package=my.package&sd=20130804&ed=20130804&dim=overall,country,language,os_version,device,app_version,carrier&met=active_device_installs,daily_device_installs,daily_device_uninstalls,daily_device_upgrades,active_user_installs,total_user_installs,daily_user_installs,daily_user_uninstalls,daily_avg_rating,total_avg_rating
/var/lib/gems/1.9.1/gems/mechanize-2.7.1/lib/mechanize/http/agent.rb:306:in fetch': 400 => Net::HTTPBadRequest for https://play.google.com/apps/publish/statistics/download?package=my.package&sd=20130804&ed=20130804&dim=overall,country,language,os_version,device,app_version,carrier&met=active_device_installs,daily_device_installs,daily_device_uninstalls,daily_device_upgrades,active_user_installs,total_user_installs,daily_user_installs,daily_user_uninstalls,daily_avg_rating,total_avg_rating -- unhandled response (Mechanize::ResponseCodeError) from /var/lib/gems/1.9.1/gems/mechanize-2.7.1/lib/mechanize/http/agent.rb:974:inresponse_redirect'
from /var/lib/gems/1.9.1/gems/mechanize-2.7.1/lib/mechanize/http/agent.rb:298:in fetch' from /var/lib/gems/1.9.1/gems/mechanize-2.7.1/lib/mechanize/http/agent.rb:974:inresponse_redirect'
from /var/lib/gems/1.9.1/gems/mechanize-2.7.1/lib/mechanize/http/agent.rb:298:in fetch' from /var/lib/gems/1.9.1/gems/mechanize-2.7.1/lib/mechanize/http/agent.rb:974:inresponse_redirect'
from /var/lib/gems/1.9.1/gems/mechanize-2.7.1/lib/mechanize/http/agent.rb:298:in fetch' from /var/lib/gems/1.9.1/gems/mechanize-2.7.1/lib/mechanize.rb:1272:inpost_form'
from /var/lib/gems/1.9.1/gems/mechanize-2.7.1/lib/mechanize.rb:539:in submit' from /var/lib/gems/1.9.1/gems/mechanize-2.7.1/lib/mechanize/form.rb:223:insubmit'
from /var/lib/gems/1.9.1/gems/mechanize-2.7.1/lib/mechanize/form.rb:229:in click_button' from /var/lib/gems/1.9.1/gems/googleplay_scraper-0.2.2/lib/googleplay_scraper/scraper_base.rb:55:intry_get'
from /var/lib/gems/1.9.1/gems/googleplay_scraper-0.2.2/lib/googleplay_scraper/scraper.rb:156:in get_appstats' from /var/lib/gems/1.9.1/gems/googleplay_scraper-0.2.2/bin/googleplay_scraper:136:inappstats'
from /var/lib/gems/1.9.1/gems/googleplay_scraper-0.2.2/bin/googleplay_scraper:41:in main' from /var/lib/gems/1.9.1/gems/googleplay_scraper-0.2.2/bin/googleplay_scraper:146:in<top (required)>'
from /usr/local/bin/googleplay_scraper:23:in load' from /usr/local/bin/googleplay_scraper:23:in

'

Thanks for your effort.
Kind regards
Nicow

Configuration File

Hi, I'm a super beginner with ruby and so I have a really silly question. What are the specific steps of creating the configuration file? Thank you!

Add CRASHES & ANRS

Is it possible to add CRASHES & ANRS to the scraper? Was looking for a download link in the web interface but couldn't find one. Any ideas?

Support new Google Wallet Merchant Center