RFC: Metaphysics PRs to include data generating code about readme

[dleve123]

Thanks for your thoughts here @dleve123 you sound like you've got a really good sense of this problem space and I'm happy to defer to your proposed solution.

To be super transparent, I've done some research on this topic, but haven't implemented a solution for a distributed test suite in the past, so don't have the wisdom of experience here. My "sniff test" suggests that a test suite / some automation seems a missing ingredient of our service approach.

PACT sounds dope - I just looked briefly so I hope this understanding is correct, but basically it acts as a fake service boundary, providing stubbed data to e.g. metaphysics so that gravity stops being an explicit dev dependency but also verifies that stub data against gravity separately?

Yes, this is my understanding!

---

My sense is that resolving this RFC isn't going to be a workflow change, but a research project? Some priority for a team? Either way, I think it's good that you are bringing up this problem @mbilokonsky.

[ashfurrow]

So, the requirement would be to include Ruby code in every Metaphysics schema change PR that one could run in the Gravity staging console to get back a result from the new schema? At first glance, it seems kind of like overkill, but that might be the nature of the simple example at the top. I might not be understanding the need for this RFC.

Are there any PRs or other instances where this would have helped us? I'm having a hard time balancing the benefits and costs, and some concrete examples would be helpful 👍

[mbilokonsky]

I'm working on one right now, actually, and I'll link it when it's ready. It's reasonably complex - intersection of artists and partner galleries on a given fair with the artists and galleries that a user follows, demonstrating both places where they overlap and where they diverge. This means I need to create multiple artists, multiple artworks, a gallery partner, a bunch of follow relations, some non-partner-but-still-have-a-profile entities (one to add to the fair, one to add to follows) etc.

It's sufficiently complex that I have to look up how all the parts fit together on Monday morning when I'm resuming work on the ticket and spend some time in the console reconstructing, figuring out caches, etc - a turnkey solution would be a massive force accelerator in a case like this.

[orta]

One way that @ashkan18 handles this kind of problem is to set up that HTTP request library everyone uses to re-create the environment so that certain objects exist in the db on the next week

I know @dleve123 has been trying to think a lot about how some of these issues that occur due to our implementations of how we do a staging env

Maybe there could be a scripts section in MP that can make some of these types of setups feasible? e.g. it pipes a list of commands into hokusai staging run I've had to solve a problem a bit like this before for CocoaPods that could give some ideas too?

[mbilokonsky]

Here is a PR that includes the sort of thing I'm referring to - as you can see, it's got a bit more going on than the toy example in the RFC above.

artsy/metaphysics#1479

[mbilokonsky]

@orta I've thought about scripts but unless someone is actively maintaining those you run the risk of missing migrations and it seems more trouble than its worth. The specific data I'm talking about is not really of historical interest outside the context of a specific PR, so I think I'm happy to just let it live on the PR?

I think ad hoc persistence of specific datasets is fine from week to week but also not quite what I mean - we don't want these things persisted forever, they're just relevant to a given dev effort that may span a weekend or two. This isn't a migration - this is just a quick and dirty way to populate the database, almost no different from a Fabricator except it crosses service boundaries.

What I really want is a fabricator for cross-service data models, I guess.

[mzikherman]

It sort of sounds like you're expressing that there's a general need for having easily accessible and correct/full seed data for certain projects + features (ie- a BNMO feature might require partners/users/artworks, an auction feature might require sales/bidders/artworks). That need also seems to intertwine with our 'production -> staging' sync and discussions we've had around that, since that's sort of the purpose of the whole sync: to have realistic data on staging, which is where most development + testing occurs.

In terms of having better organization for migrations + seed data....I don't think there's a great solution to be had that won't involve a bunch of 'work'. For instance, if the production/staging syncs and the current state of affairs across various services aren't sufficient to test a particular feature (such as the follows + fair interactions), due to weekly wiping or whatever other reason, then someone basically has to bite the bullet and, possibly manually/laboriously, come up with some seed data and scripts that get everyone what they need. It's definitely valuable work since it will unlock other engineers working on the same project to not have to spend that setup time. The next project might similarly need a work item to have someone generate data/scripts to be used for the life of the project.

Other than just doing that manually on a per-project basis when needed, I'm not sure how else to solve that issue. The production -> staging syncs hopefully lessen the amount of times you need to be creating seed data from week to week, but I think we're all finding that as projects and features evolve and micro-services grow, we are needing to spend more time on getting setup with scripts and seed data.

So, this all being said, I think you raise some great points, but I'm just wondering if that really ties into Metaphysics at all. Or, if that's just where you're actually exposing backend data since that's our canonical orchestrated Artsy schema, and so that's where there's difficulty in getting all the data + caches 'correct' so that your Metaphysics query works.

[mbilokonsky]

Exactly agreed on all points. I think it's hard to solve this generally, or in verifiable code, exactly. You're right that it's not exactly unique to MP but your last line is why MP feels like the place where maybe this is most important to get right.

You're right about just the laborious manual work, too, but I don't see another way to do it. I'm more concerned with where that work lives. For instance, I'm going to get this PR merged soon and then someone else is going to do front-end work exposing it. By including all of my data generation work in the PR, I give the next person everything they need right out of the gate to start using that new feature.

I have to wonder how many times it's happened that way - someone figures out how to configure staging data, massages it into shape, satisfies the constraints of the JIRA ticket, merges the PR and moves on. Then someone else picks up the feature, and then THEY spend a bunch of time massaging the data etc.

So this notion have having like a 'named release' of some bunch of database state just seems really useful to me. Not something you'd drop into production seeds, not even anything you'd maintain on staging over time - just a library of datasets available when you need them.

[ashfurrow]

intertwine with our 'production -> staging' sync and discussions

Yes exactly. This came up in our MO retrospective, and we've got a ticket to explore better tooling and documentation: https://artsyproduct.atlassian.net/browse/PURCHASE-838

I have to wonder how many times it's happened that way - someone figures out how to configure staging data, massages it into shape, satisfies the constraints of the JIRA ticket, merges the PR and moves on. Then someone else picks up the feature, and then THEY spend a bunch of time massaging the data etc.

I think this tends to go in waves, where we do this a lot for a project, but then the project finishes and we don't need to anymore. So I worry about having to maintain the code attached to a Metaphysics PR.

I think an alternative to requiring code within Metaphysics PRs (a process change) is to encourage more upfront discussions of data requirements for projects (a culture change). If Make Offer had tackled this head-on at the outset, iterating on scripts to setup testing data, we would've saved a tonne of time and confusion on Monday mornings. Ideally, we would script this through a k8s cron job, as @dleve123 and I have been discussing (but we could also use Jenkins).

So I'm a soft 👎 on this RFC. I like the idea, I like the spirit, but I don't think it's quite the right solution to our problem.

[mbilokonsky]

The point is not to maintain the code for longer than the feature. It becomes documentation after that.

But I do hear your larger point. If I'm honest what I'd really like to see is this data attached to any JIRA ticket defining a new task: this data shouldn't come at the end of a unit of work but at the beginning, and defining the task can involve having a dev sit down with the PM and scope out the required models to ensure the feature works? But that feels like throwing too much sand into the gears to actually be adopted, so this is kinda where I landed.

But, I am open to other approaches to this problem! :)

[dleve123]

In terms of having better organization for migrations + seed data....I don't
think there's a great solution to be had that won't involve a bunch of
'work'.

I agree with this point.

For context, a lot (search for "Dependency Disorder")
of literature out there point to the fact that
"things go bad when microservices depend upon each other a lot to
function". Other language that's often associated with microservices is the desire for "high
cohesion and loose coupling". I think a lot of Artsy's services aren't isolated services
-- each has their emphasis, but data in one service is often a dependency of
the function of another service. For example:

1. Client-exposing dependencies: i.e. Exchange and MP to provide Exchange data
   to the client in client-friendly way
2. Service-Service dependencies: i.e. between Pulse, Causality and Gravity to
   send auction information to bidders and sync state
3. Frontend-Backend dependencies: i.e. CMS and Ohm both depend upon Gravity.

The integration of each service boundary introduces risk and overhead to ensure
functionality is correct -- which is exactly what we're discussing in this thread.

MP is a service with the express goal of integrating with other services, so,
unless we significantly re-architect how Artsy's clients read and write data,
this is a service boundary we need to address head on (and not remove).

I think the solution to the core problem here isn't ad-hoc data creation, which
requires humans to do repetitive work, but automated testing that spans service
boundaries. I haven't implemented them myself, but I think either:

1. An end-to-end test suite that calls across services, or
2. Contract test suites which stub other services (and verify those stubs), maybe with
   a tool like PACT

is how we solve this problem in a scalable and repeatable way.

I know that we've tried (1) with limited success, so maybe we ought to attempt (2).

I guess my TL;DR is:

1. I think should try to minimize services that depend on one another b/c of costs like this one.
2. If we need a service to depend on another service, we should have an
   automated test suite that verifies that services play well with each other.
3. We should set the expectation that this is going to be a hard work!
4. I'm also 👎 on this approach.

I think production-like data on staging will be very helpful for ad-hoc QA and
product acceptance testing, but don't really think that's the true solution to
this problem. I think it's a good crutch while we work on a way to ensure
correctness across services.

[dleve123]

I particularly eager to hear from members of the team on how they've seen "code boundaries" tested / otherwise verified at other organizations. cc @SamRozen

[mbilokonsky]

Thanks for your thoughts here @dleve123 you sound like you've got a really good sense of this problem space and I'm happy to defer to your proposed solution.

PACT sounds dope - I just looked briefly so I hope this understanding is correct, but basically it acts as a fake service boundary, providing stubbed data to e.g. metaphysics so that gravity stops being an explicit dev dependency but also verifies that stub data against gravity separately? That sounds like a solution worth exploring.

[mbilokonsky]

Yeah I'm increasingly realizing that the RFC was maybe not the right way to bring this up because low key I wasn't 100% sold on the actual RFC action item myself - I just wanted to get people who care about it talking about it.

I'd be totally interested in taking on a project like this if it's something we want to allocate resources to!

[sweir27]

@mbilokonsky this sounds like something that could be talk about in the platform practice as a first step! Maybe worth pinging @joeyAghion to add it to the agenda for an upcoming week.

[joeyAghion]

• I'd rather not increase the "code" that's uncommitted or untested or targets only test/dev scenarios.
• On the other hand, I think that traditional "seed data" is a practice we haven't exploited much, and could serve (via largely existing tooling) to fill gaps when we're doing significant development without there being representative data easily reference-able in staging environments. Even when a system (like Gravity or Positron) performs a periodic copy of data from production, it could supplement with a standing seed dataset.
• When applicable, though, I think source systems should be the ones responsible for maintaining their own seed or minimal data, as opposed to Metaphysics. E.g., Gravity could supply a few partners and auctions, while Exchange supplies some orders in the various interesting states. Presumably Metaphysics development could target that data locally or in staging to confirm the expected shape of results.
• It does seem like this aims to address some of the same needs as "contract testing," which is a worthy research project. It's hard to say if one or both are worthwhile without demonstrations of either.
• Seed data might also be a useful artifact of coordination between teams. E.g., if a new product need depends on auctions or fairs in a particular state, it could serve as a very concrete deliverable between them and ease development across those boundaries.

[SamRozen]

@dleve123 thanks for the in person ping to check this discussion. One way Spotify tackled this was by relying heavily on data schema (mainly protobuf). Schemas were in their own separate repo and versioned so that each service could reference precisely what there were using. Then APIs were versioned too and backward compatibily has to be maintained for a while. Not sure if that really tranlates to our ecosystem here though.

[sweir27]

It sounds like we want to keep this conversation going, but perhaps not in the exact format that this RFC proposes.

Going to close this for now but will follow up with @mbilokonsky to see what the next steps should be!