flipkart-incubator / batchman Goto Github PK

This library for Android will take any set of events and batch them up before sending it to the server. It also supports persisting the events on disk so that no event gets lost because of an app crash. Typically used for developing any in-house analytics sdk where you have to make a single api call to push events to the server but you want to optimize the calls so that the api call happens only once per x events, or say once per x minutes. It also supports exponential backoff in case of network failures

Java 100.00%

analytics android-library batch-manager batch-processing batching java job-scheduler networking persistence persistence-strategies retry serialization strategies

batchman's People

Contributors

Stargazers

Watchers

Forkers

flipkart matrixy myyhc morristech harshsharma94 vinayasathyanarayana manojv0001 dopentest isabella232 premacck

batchman's Issues

Easier interface for simple use case

Refactor Test Cases

resume persitence callbacks on network connectivity change

move duplicacy check from persistence to strategy

Clean up and refactor code

Test case for TapePersistenceStrategy

NullPointerException

CrashLytics: 4.6 Production app for Batching Lib

Fatal Exception: java.lang.NullPointerException
at com.flipkart.batching.persistence.TapePersistenceStrategy.getAllDataFromTapeQueue(TapePersistenceStrategy.java:105)
at com.flipkart.batching.persistence.TapePersistenceStrategy.syncData(TapePersistenceStrategy.java:1095)
at com.flipkart.batching.persistence.TagBasedPersistenceStrategy.onInitialized(TagBasedPersistenceStrategy.java:55)
at com.flipkart.batching.strategy.BaseBatchingStrategy.onInitialized(BaseBatchingStrategy.java:57)
at com.flipkart.batching.strategy.SizeBatchingStrategy.onInitialized(SizeBatchingStrategy.java:60)
at com.flipkart.batching.strategy.ComboBatchingStrategy.onInitialized(ComboBatchingStrategy.java:51)
at com.flipkart.batching.strategy.TagBatchingStrategy.onInitialized(TagBatchingStrategy.java:63)
at com.flipkart.batching.BatchManager.initialize$175fd481(BatchManager.java:1151)
at com.flipkart.batching.BatchManager$1.run(BatchManager.java:63)
at android.os.Handler.handleCallback(Handler.java:808)
at android.os.Handler.dispatchMessage(Handler.java:103)
at android.os.Looper.loop(Looper.java:193)
at android.os.HandlerThread.run(HandlerThread.java:61)

SLF4J integration

change trimming to work on basis of number of data, not batches

removing data twice while trimming

In TrimPersistedBatchReadyListener.java , we are calling remove() twice in trimQueue() method because of which trimming is removing data twice when invoked.

Performance Improvements

Converted all the custom deserializers to TypeAdapters.

Velocity based reconfiguration of size strategy

Batch to be an interface

Removing gson dependency from the library

Support for protobufs

protocol-buffers

Documentation part

remove data-object from TagData

issue in serialization and deserialization

IncompatibleClassChangeError

Fatal Exception: java.lang.IncompatibleClassChangeError: Couldn't find com.google.gson.annotations.SerializedName.value
       at libcore.reflect.AnnotationAccess.toAnnotationInstance(AnnotationAccess.java:659)
       at libcore.reflect.AnnotationAccess.toAnnotationInstance(AnnotationAccess.java:641)
       at libcore.reflect.AnnotationAccess.getDeclaredAnnotation(AnnotationAccess.java:170)
       at java.lang.reflect.Field.getAnnotation(Field.java:242)
       at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.getFieldName(ReflectiveTypeAdapterFactory.java:1076)
       at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.getBoundFields(ReflectiveTypeAdapterFactory.java:156)
       at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.create(ReflectiveTypeAdapterFactory.java:97)
       at com.google.gson.Gson.getDelegateAdapter(Gson.java:442)
       at com.flipkart.batching.toolbox.RuntimeTypeAdapterFactory.create(RuntimeTypeAdapterFactory.java:194)
       at com.google.gson.Gson.getAdapter(Gson.java:360)
       at com.google.gson.Gson.fromJson(Gson.java:813)
       at com.google.gson.Gson.fromJson(Gson.java:779)
       at com.google.gson.Gson.fromJson(Gson.java:728)
       at com.google.gson.Gson.fromJson(Gson.java:700)
       at com.flipkart.batching.persistence.GsonSerializationStrategy.deserializeData(GsonSerializationStrategy.java:180)
       at com.flipkart.batching.persistence.TapePersistenceStrategy.getAllDataFromTapeQueue(TapePersistenceStrategy.java:121)
       at com.flipkart.batching.persistence.TapePersistenceStrategy.syncData(TapePersistenceStrategy.java:1106)
       at com.flipkart.batching.persistence.TagBasedPersistenceStrategy.onInitialized(TagBasedPersistenceStrategy.java:55)
       at com.flipkart.batching.strategy.BaseBatchingStrategy.onInitialized(BaseBatchingStrategy.java:57)
       at com.flipkart.batching.strategy.SizeBatchingStrategy.onInitialized(SizeBatchingStrategy.java:60)
       at com.flipkart.batching.strategy.ComboBatchingStrategy.onInitialized(ComboBatchingStrategy.java:51)
       at com.flipkart.batching.strategy.TagBatchingStrategy.onInitialized(TagBatchingStrategy.java:63)
       at com.flipkart.batching.BatchManager.initialize$175fd481(BatchManager.java:1151)
       at com.flipkart.batching.BatchManager$1.run(BatchManager.java:63)
       at android.os.Handler.handleCallback(Handler.java:739)
       at android.os.Handler.dispatchMessage(Handler.java:95)
       at android.os.Looper.loop(Looper.java:145)
       at android.os.HandlerThread.run(HandlerThread.java:61)
 Show all 35 Threads

Remove dependency on CONNECTIVITY_ACTION as in Android N this is going to go away

corrupt file handling if looper is slow

Hint : look at NetworkPersistedBatchReadyTest#testReinitialize

file path API from library

Since library is the consumer of file path so either we can have it implicitly consumed and not expose to be set by client or we can provide a api to get a valid path from the library itself.

Gson labels must be unique

registerSubtype is causing IllegalArgumentException("types and labels must be unique");

Reproducible by Sharath Pandeswar

onTrimmed implementation

NullPointerException on RuntimeTypeAdapterFactory

Fatal Exception: java.lang.NullPointerException: Attempt to invoke virtual method 'java.lang.Class java.lang.Object.getClass()' on a null object reference
       at com.flipkart.batching.toolbox.RuntimeTypeAdapterFactory$1.write(RuntimeTypeAdapterFactory.java:221)
       at com.google.gson.Gson.toJson(Gson.java:605)
       at com.google.gson.Gson.toJson(Gson.java:584)
       at com.google.gson.Gson.toJson(Gson.java:539)
       at com.flipkart.batching.persistence.GsonSerializationStrategy.serializeData(GsonSerializationStrategy.java:148)
       at com.flipkart.batching.persistence.TapePersistenceStrategy.getAllDataFromTapeQueue(TapePersistenceStrategy.java:111)
       at com.flipkart.batching.persistence.TapePersistenceStrategy.syncData(TapePersistenceStrategy.java:1095)
       at com.flipkart.batching.persistence.TagBasedPersistenceStrategy.onInitialized(TagBasedPersistenceStrategy.java:55)
       at com.flipkart.batching.strategy.BaseBatchingStrategy.onInitialized(BaseBatchingStrategy.java:57)
       at com.flipkart.batching.strategy.SizeBatchingStrategy.onInitialized(SizeBatchingStrategy.java:60)
       at com.flipkart.batching.strategy.ComboBatchingStrategy.onInitialized(ComboBatchingStrategy.java:51)
       at com.flipkart.batching.strategy.TagBatchingStrategy.onInitialized(TagBatchingStrategy.java:63)
       at com.flipkart.batching.BatchManager.initialize$175fd481(BatchManager.java:1151)
       at com.flipkart.batching.BatchManager$1.run(BatchManager.java:63)
       at android.os.Handler.handleCallback(Handler.java:810)
       at android.os.Handler.dispatchMessage(Handler.java:99)
       at android.os.Looper.loop(Looper.java:189)
       at android.os.HandlerThread.run(HandlerThread.java:61)

Path for event persistence and batch persistence should not be same

NullPointerException: Attempt to invoke virtual method tagData.getTag() on a null object reference

Fatal Exception: java.lang.NullPointerException: Attempt to invoke virtual method 'com.flipkart.batching.data.Tag com.flipkart.batching.data.TagData.getTag()' on a null object reference 
       at com.flipkart.batching.persistence.TagBasedPersistenceStrategy.filterByTag(TagBasedPersistenceStrategy.java:68) 
       at com.flipkart.batching.persistence.TagBasedPersistenceStrategy.getData(TagBasedPersistenceStrategy.java:43) 
       at com.flipkart.batching.strategy.SizeBatchingStrategy.flush(SizeBatchingStrategy.java:49) 
       at com.flipkart.batching.strategy.ComboBatchingStrategy.flush(ComboBatchingStrategy.java:65) 
       at com.flipkart.batching.strategy.TagBatchingStrategy.flush(TagBatchingStrategy.java:81) 
       at com.flipkart.batching.BatchManager$2.run(BatchManager.java:96) 
       at android.os.Handler.handleCallback(Handler.java:739) 
       at android.os.Handler.dispatchMessage(Handler.java:95) 
       at android.os.Looper.loop(Looper.java:135) 
       at android.os.HandlerThread.run(HandlerThread.java:61)

FlushByTag implementation

OutOfMemory Exception due to QueueFile

java.lang.OutOfMemoryError: Failed to allocate a 1936010286 byte allocation with 4194304 free bytes and 56MB until OOM
at 
com.squareup.tape.QueueFile.peek(QueueFile.java:414)
at 
com.flipkart.batching.toolbox.TapeQueueFile.peek(TapeQueueFile.java:32)
at com.flipkart.batching.persistence.TapePersistenceStrategy.getAllDataFromTapeQueue(TapePersistenceStrategy.java:111)
at com.flipkart.batching.persistence.TapePersistenceStrategy.syncData(TapePersistenceStrategy.java:98)
at com.flipkart.batching.persistence.TapePersistenceStrategy.onInitialized(TapePersistenceStrategy.java:92)
at com.flipkart.batching.persistence.TagBasedPersistenceStrategy.onInitialized(TagBasedPersistenceStrategy.java:55)
at com.flipkart.batching.strategy.BaseBatchingStrategy.onInitialized(BaseBatchingStrategy.java:57)
at com.flipkart.batching.strategy.SizeBatchingStrategy.onInitialized(SizeBatchingStrategy.java:60)
at com.flipkart.batching.strategy.ComboBatchingStrategy.onInitialized(ComboBatchingStrategy.java:51)
at com.flipkart.batching.strategy.TagBatchingStrategy.onInitialized(TagBatchingStrategy.java:63)
at com.flipkart.batching.TagBatchManager.initialize(TagBatchManager.java:77)
at com.flipkart.batching.TagBatchManager.access$300(TagBatchManager.java:27)
at com.flipkart.batching.TagBatchManager$1.run(TagBatchManager.java:61)
at android.os.Handler.handleCallback(Handler.java:739)
at android.os.Handler.dispatchMessage(Handler.java:95)
at android.os.Looper.loop(Looper.java:148)
at android.os.HandlerThread.run(HandlerThread.java:61)

IllegalStateException when finish() called

[PersistedBatchReadyListener.java line 134]
Fatal Exception: java.lang.IllegalStateException: Finish was called with a different batch, expected com.flipkart.batching.strategy.TagBatchingStrategy$TagBatch@2d0d81bb was com.flipkart.batching.strategy.TagBatchingStrategy$TagBatch@1de966d8
at com.flipkart.batching.listener.PersistedBatchReadyListener$2.run(PersistedBatchReadyListener.java:134)
at android.os.Handler.handleCallback(Handler.java:815)
at android.os.Handler.dispatchMessage(Handler.java:104)
at android.os.Looper.loop(Looper.java:194)
at android.os.HandlerThread.run(HandlerThread.java:61)

Batch Event turns to null

Hi,
We are facing an issue with the a custom implementation of a Strategy wherein an EventData turns to null.
Attached below is the crashlog. Can you please go through it and suggest some measure.
As this repeatedly causes an exception whenever the data is flushed.

Caused by java.lang.NullPointerException: Attempt to invoke virtual method 'boolean com.flipkart.batching.core.data.Tag.equals(java.lang.Object)' on a null object reference
at com.flipkart.batching.core.data.TagData.equals(TagData.java:52)
at java.util.ArrayList.indexOf(ArrayList.java:305)
at java.util.ArrayList.contains(ArrayList.java:288)
at com.flipkart.batching.persistence.TapePersistenceStrategy.removeData(TapePersistenceStrategy.java:101)
at com.example.analytics.HybridBatchingStrategy.flush(HybridBatchingStrategy.java:63)
at com.example.analytics..HybridBatchingStrategy$1.run(HybridBatchingStrategy.java:30)
at android.os.Handler.handleCallback(Handler.java:751)
at android.os.Handler.dispatchMessage(Handler.java:95)
at android.os.Looper.TinnoLoop(Looper.java:138)
at android.os.Looper.loop(Looper.java:208)
at android.os.HandlerThread.run(HandlerThread.java:61)

The HybridBatchingStrategy is similar to SizeTimeBatchingStrategy with the timer called every N seconds.
The exception is thrown everytime it tries to checks using .equals() method to remove data from strategy.

What we have observed is that the first event data gets turned into a reference with null event and null tag, which looks like below.

Any help is appreciated.
Finally, thanks for this brilliant library. It saved a lot of our efforts.

Exposing `forced` from BatchController

Currently, in BatchController.addToBatch, BatchingStrategy.flush(false) is being called, and the forced argument in the flush method is not configurable.

Requirement:
Expose the flush argument from BatchController in order for it to be configurable by callers

Invalid json format in collector call

    {
        "ct": "productCard",
        "p": "{value=21}",
        "iid": "30a0bbe2-44a3-4146-b612-7b9ddfeb0983.SEARCH",
        "en": "DCI",
        "t": "{value=1459751849340}"
    }, {
        "ct": "productCard",
        "p": "{value=22}",
        "iid": "17dc7c76-98f7-4a6c-a4db-60c160f4da81.SEARCH",
        "en": "DCI",
        "t": "{value=1459751849343}"
    }, {
        "ct": "productCard",
        "p": "{value=23}",
        "iid": "aaabf9cc-7ee9-4e9e-9ec4-1333409499c2.SEARCH",
        "en": "DCI",
        "t": "{value=1459751849600}"
    }, {
        "ct": "productCard",
        "p": "{value=24}",
        "iid": "ff5f12b4-406e-4406-aaa9-4fa161d76926.SEARCH",
        "en": "DCI",
        "t": "{value=1459751849606}"
    }, {
        "ct": "productCard",
        "p": "{value=25}",
        "iid": "be6eb820-4bd6-4e10-9ec8-0b8a74d97d80.SEARCH",
        "en": "DCI",
        "t": "{value=1459751849802}"
    }, {
        "ct": "productCard",
        "p": "{value=26}",
        "iid": "c2c2a86b-2feb-4536-b632-e2abce5d3096.SEARCH",
        "en": "DCI",
        "t": "{value=1459751849809}"
    }

expose flush method to batch manager

expose the flush method to batch manager, so that it can be called when the app opens

java.lang.VerifyError

Fatal Exception: java.lang.VerifyError: com/flipkart/android/analytics/f
       at com.flipkart.android.analytics.BatchManagerHelper.init(BatchManagerHelper.java:1205)
       at com.flipkart.android.init.FlipkartApplication.onCreate(FlipkartApplication.java:175)
       at android.app.Instrumentation.callApplicationOnCreate(Instrumentation.java:1014)
       at android.app.ActivityThread.handleBindApplication(ActivityThread.java:4205)
       at android.app.ActivityThread.access$1300(ActivityThread.java:133)
       at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1263)
       at android.os.Handler.dispatchMessage(Handler.java:99)
       at android.os.Looper.loop(Looper.java:137)
       at android.app.ActivityThread.main(ActivityThread.java:4812)
       at java.lang.reflect.Method.invokeNative(Method.java)
       at java.lang.reflect.Method.invoke(Method.java:511)
       at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:792)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:559)
       at dalvik.system.NativeStart.main(NativeStart.java)

Caused by: java.lang.IllegalStateException: Not a JSON Object

Repro Unit Test

GsonSerializationStrategy<Data, Batch<Data>> serializationStrategy;
serializationStrategy = new GsonSerializationStrategy<>();
registerBuiltInTypes(serializationStrategy);
serializationStrategy.build();

JSONObject jsonObject = new JSONObject();
jsonObject.put("name", "Anirudh");
Batch<Data> originalBatch = new BatchImpl(Collections.singleton(jsonObject.toString()));

byte[] bytes = serializationStrategy.serializeBatch(originalBatch);
Batch<Data> deserializedBatch = serializationStrategy.deserializeBatch(bytes);

Assert.assertEquals(deserializedBatch.getDataCollection().size(), originalBatch.getDataCollection().size());

While deserializing the JsonElement is of type JsonPrimitive instead of JsonObject