Request to add features about google-play-crawler

[Akdeniz]

Hi there,

First I want to apologize that I havent been able to look what is going on issues of this project for SO long because of my daily job and other things :/

Anyway here is my findings about your issue:

https://android.clients.google.com/market/api/ApiRequest

Android sends information about installed applications within base64 encoded request to this url. To be precise; request field is base64 encoded RequestProto structure as you can observe below.

RequestPropertiesProto contains information about device, to be used for some kind of authentication i think. Here "logging_id" is hexedecimal of random long number which is provided while device is checked in! (I ignored this value in my code because of not caring about logging until now, but it can easily be added.)

This call request is being made at random times and according to response structure (ResponseProto>ContentSyncResponseProto), it only contains update information about each applications. So I dont think this call is used for updating download count at server side. But I may be wrong about this, so further testing is required.

https://android.clients.google.com/fdfe/log

This url is called at every download operation and likely to be used for download count. It contains LogRequest structure.

I tried to generate some calls to these urls to see if download count of my purposely published application on google play is changed.

I had a diffuculty on authenticating ApiRequest url which always returns me HTTP-403 error. I need to find out what is wrong there..

I successfully generated a request to second url but Google updates statistics about an application at daily basis. So I have to wait until tomorrow to see if it has any affect on download count.

Proto structures:

message ContentSyncRequestProto {
  optional bool incremental = 1;
  repeated group AssetInstallState = 2 {
    optional string assetId = 3;
    optional int32 assetState = 4;
    optional int64 installTime = 5;
    optional int64 uninstallTime = 6;
    optional string packageName = 7;
    optional int32 versionCode = 8;
    optional string assetReferrer = 9;
  }
  repeated group SystemApp = 10 {
    optional string packageName = 11;
    optional int32 versionCode = 12;
    repeated string certificateHash = 13;
  }
  optional int32 sideloadedAppCount = 14;
}
message ContentSyncResponseProto {
  optional int32 numUpdatesAvailable = 1;
}

message RequestPropertiesProto {
  optional string userAuthToken = 1;
  optional bool userAuthTokenSecure = 2;
  optional int32 softwareVersion = 3;
  optional string aid = 4;
  optional string productNameAndVersion = 5;
  optional string userLanguage = 6;
  optional string userCountry = 7;
  optional string operatorName = 8;
  optional string simOperatorName = 9;
  optional string operatorNumericName = 10;
  optional string simOperatorNumericName = 11;
  optional string clientId = 12;
  optional string loggingId = 13;
}
message RequestProto {
  optional RequestPropertiesProto requestProperties = 1;
  repeated group Request = 2 {
    optional ContentSyncRequestProto contentSyncRequest = 9;
  }
}

message ResponsePropertiesProto {
  optional int32 result = 1;
  optional int32 maxAge = 2;
  optional string etag = 3;
  optional int32 serverVersion = 4;
  optional int32 maxAgeConsumable = 6;
  optional string errorMessage = 7;
  repeated InputValidationError errorInputField = 8;
}
message ResponseProto {
  repeated group Response = 1 {
    optional ResponsePropertiesProto responseProperties = 2;
    optional ContentSyncResponseProto contentSyncResponse = 8;
  }
}
//////////////////////////////////////////
message ClickLogEvent {
  optional int64 eventTime = 1;
  optional string url = 2;
  optional string listId = 3;
  optional string referrerUrl = 4;
  optional string referrerListId = 5;
}
message LogRequest {
  repeated ClickLogEvent clickEvent = 1;
}

[NaomiLi]

Sorry to ask for help !!! im confused with how to make a "ContentSyncRequestProto" request, always returns 403, forbidden. what's the rule of SystemApp .certificateHash . but i replicated a ContentSyncRequestProto request sent by a Device, it returns 200. thanks.

[ouyangzhanzhu]

Sorry to ask for help !!! How to successfully call this interface (https://android.clients.google.com/fdfe/log), what parameters can be passed to share the solution? thanks. @Akdeniz

[azaleski9]

That's quite easy

private static final String FDFE_URL = "https://android.clients.google.com/fdfe/";
public static final String LOG_URI = FDFE_URL + "log";
`public Log.LogResponse log(Log.LogRequest log) {
        try {
        ResponseWrapper responseWrapper = executePOSTRequest(LOG_URI,
                log.toByteArray(), "application/x-protobuf");
            return responseWrapper.getPayload().getLogResponse();
        } catch (Exception ex) {

        }
        return null;
    }

public static Log.ClickLogEvent getConfirmFreeDownloadEvent(String app) {
        return Log.ClickLogEvent.newBuilder().setEventTime(System.currentTimeMillis()).setUrl("confirmFreeDownload?doc=" + app).build();
    }

//Can be added in CheckinWorker for instance
    Log.LogRequest log = Log.LogRequest.newBuilder().addClickEvent(getConfirmFreeDownloadEvent(pkg)).build();
            api.Log(log); //api is your GooglePlayAPI

`

However, does anybody know how to make ContentSyncRequest, because it gives me always error 403 code. That's my protobuf code:

@Akdeniz @ouyangzhanzhu

requestProperties {
  userAuthToken: "lwMPdRVONlT7b0IZp5ujsoUTPY_fEKClTAjuoWM6A5J3ldX8bWIiZcCinOxyg92aPjJlbw."
  userAuthTokenSecure: false
  softwareVersion: 2009011
  aid: "32F7967C1BDF09DD"
  productNameAndVersion: "shamu:22"
  userLanguage: "en"
  userCountry: "US"
  operatorName: "Plus"
  simOperatorName: "Plus"
  operatorNumericName: "26001"
  simOperatorNumericName: "26001"
  clientId: "am-android-google"
loggingId: "" //random here
}
request {
  contentSyncRequest {
    AssetInstallState {
      assetId: "v2:com.google.android.gms:1:9452230"
      installTime: 2016073043
      packageName: "com.google.android.gms"
    }
    sideloadedAppCount: 0
  }
}`