Importing Historic Share Data about chompy

[sbsmith86]

Let's automatically accept these since the export itself already filters for completed shares. One thing to note though -- the event.name was going to be renamed completed_facebook_share as it is in the provided file, but this hasn't been done on Rocket's end yet. What it actually is named for the Analytic Data Model is share action completed

Sounds good - it really doesn't matter what the value is if we don't need to do any logic on it. So we can just automatically accept!

Definitely, if not from Keen (the data source), we can add it in manually. You'd need the campaign_id and not campaign_run or both? Either/both are doable.

Both if you can!

In the file that I attached to the Pivotal card, I hid a lot of columns that didn't seem necessarily when I first looked at the file, but now I'm thinking we might want to send the FB share timestamp (keen.created_at in Column B) over in the post's source_details for reporting purposes since the post created_at and updated_at are from the importer-client right?

We could import posts and use the keen timestamps instead of the automatically generated ones. I sorta like this idea since in the future we want these shares going directly to rogue and the timestamps would align. So we wouldn't get in a situation where you have to look at different fields depending on when something was entered into Rogue.

Additionally, if you unhide all of the columns, here's some other stuff Data could find useful in the post source_details: keen.id , meta.id ! NOTE: These are the requested fields so far, but Data is taking a final review of this so it might change, will keep you updated.

Let's do what we did for the TurboVote imports and store the csv row with extra data in the post details column so we have it.

[sbsmith86]

I updated the JSON examples above to reflect what @ngjo mentioned

I will note in the code/wiki/etc that we are ignoring records without NS IDs

[ngjo]

For post status, I think we should automatically accept these. But, if there are other event.name values that could be parsed we could do something more sophisticated like- set the post status to accepted only if the event.name includes "completed".

Let's automatically accept these since the export itself already filters for completed shares. One thing to note though -- the event.name was going to be renamed completed_facebook_share as it is in the provided file, but this hasn't been done on Rocket's end yet. What it actually is named for the Analytic Data Model is share action completed

I think we are going to need a campaign_id column to associate signups/posts to. Is that possible?

Definitely, if not from Keen (the data source), we can add it in manually. You'd need the campaign_id and not campaign_run or both? Either/both are doable.

I think type should be prefixed with share- and then the platform it was shared on. Like share-facebook, share-twitter. Thoughts?

I like this too!

Would be good to pass the final write-up to data and rocket for a 👍 on it too before starting in on the work.

My questions/additions (5/31/18)

• In the file that I attached to the Pivotal card, I hid a lot of columns that didn't seem necessarily when I first looked at the file, but now I'm thinking we might want to send the FB share timestamp (keen.created_at in Column B) over in the post's source_details for reporting purposes since the post created_at and updated_at are from the importer-client right?
• Additionally, if you unhide all of the columns, here's some other stuff Data could find useful in the post source_details: keen.id , meta.id ! NOTE: These are the requested fields so far, but Data is taking a final review of this so it might change, will keep you updated.
• Let's document in the wiki that we're ignoring anything w/o a NS ID (as we've been doing so far) and we haven't built in any logic to create accounts.

Let me know if you wanna talk IRL on any of this! I can also send you additional exports for you to take a look at for more examples.

[sohaibhasan]

Would you all be against having type be social - share and the action be facebook - quiz?

[sbsmith86]

@sohaibhasan Interesting- Can you explain these a little more? Not sure I understand the need for this distinction.

[sohaibhasan]

@sbsmith86 It's really just to try to get ahead of a proliferation of types for us to account for. social can subsume all the other sharing platforms, with additional details being in the action column.

[sbsmith86]

@sohaibhasan I agree that we want to get ahead of this. This is actually why I think what we have proposed makes sense.

type
Should indicate the type of action taken i.e
photo - a photo submission,
text - a text submission.
share-{platform} - content was shared, and the platform it was shared on.

The reason i like share better than social is because I thing "share" describes the type of thing done and is not specific to something that happened on a social platform. I think this follows the pattern of our other types better.

Action
Used to describe what and when we asked for the action. So we don't want this to necessarily describe the type in more detail, we want to be able to use the action column to say

"X number of people shareed content on platform and it should be counted towards when we asked members to take a specific action (i.e share the quiz)"

If we need more details, that is what the post details column is for.

[sohaibhasan]

@sbsmith86 First off, everything you said makes perfect sense.

The main reason I want to keep the platform out is that I want the concatenation of type and action to be sufficient to get reportback counts. My understanding is that (@ngjo , please confirm), a user who shares on twitter and facebook should only get credit for one action. If the platform is in that string, then we'd first have to scrub it out before being able to do the count distinct.

Maybe we can leave the platform in the post details?

[ngjo]

Hahahaha it gets a little complicated - it's a yes/no kinda thing. Instead of trying to explain it, I'm gonna just write-up some examples instead so I don't butcher anything in trying to translate it:

• If a campaign has a Facebook, Twitter, Instagram (or whatever other social channel) that's sharing the same resource and the member shares it on any or all of them, that would count as one social "reportback"
• If a campaign has a Facebook, Twitter, Instagram (or whatever social channel) that's sharing different resources (FB = resource A, Twitter = resource B, Instagram = resource C) then each share would count as a social "reportback" (three, in this case)
• A campaign like Grab the Mic which is open all year but has different actions month to month: January FB share, February FB share -- each, because they are sharing a different resource, should count as a separate social "reportback"
• If a user shares the same resource 500 times on FB, we only want to count it once

A lot of ^ has been solved w/ how we wanna use the type and action columns, but just wanted to reiterate just in case! Let me know if more examples would be useful or if you wanna talk about it IRL.

[sbsmith86]

totally down to add platform in the post details to make this easier. That makes sense to me!

[sohaibhasan]

Got it. Given the above, I would like to have social share in post.type and resource in post.action. I would also like to make sure future actions are set up so the info required to determine whether it is a distinct reportback is contained entirely in the post and action columns, with anything else going in post details.

Is that doable?

[sbsmith86]

@sohaibhasan not sure what you mean by resource in the action field? I also want to keep in mind that these fields need to make some sort of sense to other apps using the data via the API, so we can't optimize only for data analysis.

This is what I propose:

{
    "id" : 3,
    "signup_id" : 1234,
    "campaign_id" : 34567,
    "northstar_id" : "59ba9xxxxxxxxe564f",
    "type" : "share-social",
    "action" : "quiz-share",
    "quantity" : null,
    "url" : null,
    "text" : null,
    "details": {"platform": "facebook"},
    "status" : "accepted",
    "source" : "importer-client",
    "source_detail" : { "original-source" : "phoenix-next"},
    "created_at" : keen.created_at,
    "updated_at" : "timestamp",
    "deleted_at" : null
}

[sohaibhasan]

Sorry, by resource I basically meant the thing that was shared, so "quiz" in the example above.

The structure above makes sense, but I'd like to also include the ID of the event in Keen in the source detail.

This should be easy, but I need to do some work here, because the ID in Keen is the not the ID that we are receiving in quasar. I think this is because Puck is created its own ID, which is being passed over instead of the keen ID. I need to verify this though.

[ngjo]

yup, @sbsmith86 mentioned that she's gonna include all the info "Let's do what we did for the TurboVote imports and store the csv row with extra data in the post details column so we have it."

[sohaibhasan]

@sbsmith86 @ngjo

Ashley and I talked and realized it isn't a good idea to use a Keen export because the records that get sent to rogue will never touch keen. So the right thing to use is the puck ID and the puck TS since that is what will be sending the events and that is also what data is using to get phoenix data.

For you all, this literally just amounts to a find and replace swapping puck for keen. I can send over a puck export for chompy testing.

[sbsmith86]

Sorry I think I am a little confused here because I thought puck sent data to keen? So i'm not sure what the difference is.

I think I might need more context about how shares are set up, how we are currently counting these share actions as "reportbacks", and where they are being tracked and how. And then it might be helpful for us to sit down and talk through how this looks in rogue. Or if you can propose what a share post should look like, we can talk through it.

I would also love some insight into how this data is planned to be used by not only data, but phoenix, and gambit.

@sohaibhasan @ngjo

[sohaibhasan]

Puck sends data to keen but keen doesn't send it onward to anything else, nor can it.

Hence, it makes sense to scope chompy work using an export from puck rather than keen.

The share post above is perfect, except change the word keen to puck and add puck ID to post details.

Happy to chat through it if it is easier than all this written back and forth.

[ngjo]

@sohaibhasan ok, yea a little unclear why the puck and keen export aren't the same, but in terms of next steps -- are you saying that we will get a puck export and use that instead of keen and everything else outlined will be the same?

[sohaibhasan]

Yes, the reason is the puck points to keen and quasar and puck will also point to rogue. So it makes sense to use a puck export. Keen's ID only exists within Keen and so will never be sent anywhere.

Where/how should I share the puck export?

[ngjo]

Can you slack it to both of us for now? Maybe just a subset of the data too for testing, now there's been tons of shares

[ngjo]

Update: we've received the puck export but are blocked because we don't know which column shows what was actually shared (this is important for the action column). Ashley will be helping with this when she's back in the office.

[chloealee]

@sbsmith86 @ngjo @sohaibhasan I just copy and pasted a lot from this convo to create a new page and added to the wiki!