Store Signals Locally if no Internet Connection about swiftclient

[winsmith]

So far we can. Let's see how the open beta changes that 😂

[outcoldman]

Also, consider using JSON streaming format (https://en.wikipedia.org/wiki/JSON_streaming).

This is how I would see the implementation.

The client should collect and store all signals on the disk, appending JSONs line by line to a temporary file to a known unique location of the app. In case of Apple apps could be

try FileManager.default.url(
    for: .libraryDirectory,
    in: .userDomainMask,
    appropriateFor: nil,
    create: true
)
.appendingPathComponent(Bundle.main.bundleIdentifier!)
.appendingPathComponent("apptelemetry.noindex")
.appendingPathComponent("data.json")

As you can see I store it in the library folder, so if the app will crash or Mac will reboot, we still have access to the old telemetry.

There are should be a background Timer (like fired once an hour) that will check on the disk if there is anything to send, will pick up those files and send them to your Telemetry server.

Preferable to send with the JSON streaming format, so there will be just one large HTTP call, and not bunch of small ones. Also, when sending, please consider using gzip-compression (to be honest I am not sure if URLSession does it automatically).

[winsmith]

JSON streaming looks super cool, I haven't seen that before. Thanks for that. GZIP should be enabled by default but it can't hurt to check that.

[nik6018]

Storage Choices:

• If we had a day's worth Signals using Defaults would load the entire thing into the memory before it would be transmitted to the server, so I don't know if this trade off can be made
• Core Data would bring lots of goodies for storage management but it too has it's own can of worms (mainly migrations)
• Haven't worked that much on storing data directly to disk, so can't comment on that.

Live Transmission

• Uploading signals in bunch will truly optimise the process, this is big up side
• We can have Live signals for Debug Mode only and then bundle up signals while in Productions (Firebase does this ... stream view can only be enabled with a flag in Debug builds and disable them in Production) (Sorry to repeat if you already knew)
• I don't know if it's your product goal to display Live Signals but IMO they should be restricted to Debugging only (But I like them for my app though 😄)

[winsmith]

Why do you want to restrict live signals to debug only? I think live signals is a super neat thing, and I'd really like to keep them at least live-ish (I wouldn't mind a bit of a lag)

[nik6018]

Well I thought it's a load thing on the server .... (Firebase tutorials given this reason for not enabling their continuous stream even view for production). But if we can have that without degrading server performances than it's a win :)

[nik6018]

Cool. So were you able to give any thought on the storage part ?

[winsmith]

Probably needs to be a JSON file written to a temp location. I can't see any other way to make that work on all supported platforms easily.

[nik6018]

Cool. Can I help in anyway possible ?

[winsmith]

I'm open to pull requests, if you feel like tackling this. Otherwise, you can help by finding out by thinking about when and how often the client should try and send data to the server, and without sending data multiple times, and posting your thoughts into this ticket. (This is an open invitation to all users of the client of course)

[nik6018]

I think we can use a session based approach, where we gather all the signals for the current session and then when applicationDidEnterBackground(_:) gets called, upload all the signals using a BackgroundTask (we'll have at least 5 - 10 seconds to complete the operation once the app enters background)

[winsmith]

Grouping by session is a good idea. We shouldn't forget that the device might have no internet connection at backgrounding time though. Also, I'd like to send signals more often than just when the device backgrounds, so a periodic periodic uploader should be a good addition here.

[nik6018]

We can implement something that checks for signal count. Uploading 10 signals in a bunch seems like a good conservative approach at least for the time being. So the utility will save 10 signals and once we have more than that we upload the last 10 and save the new signals.

[winsmith]

I like that!

[nik6018]

Nice I can come up with a solution for this, for time being I'll start with Defaults as storage solution .... this will reduce the storage complexity initially

[winsmith]

Please do not use Defaults as storage. Using defaults as storage will cause the client to not work under Linux and Windows any more, which are supported platforms.

[nik6018]

Aaah this is downfall of me coding for Apple Platforms only .... can you suggest any other method ? (Saving to file is the best way to go ?)

[winsmith]

I'd like this to be in a temporary JSON file. How about I get started with the general structure and then you can help out with PRs later?

[nik6018]

Yeah cool no issues :)

…

On 16-Apr-2021, at 4:46 PM, Daniel Jilg ***@***.***> wrote: I'd like this to be in a temporary JSON file. How about I get started with the general structure and then you can help out with PRs later? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#19 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AD3HBNS5TJVNTGXECXVUP4DTJAMARANCNFSM42OMR2UA>.

[winsmith]

This is now done, and I completely forgot to close the ticket. Oops.