openbeta / openbeta-graphql Goto Github PK

use case

I want to pan, zoom the map and be able to ask the api to return crags/climbs within the map bounding box.

Use something like https://github.com/pinojs/pino for logging

FYI @ygingras

Need to figure out how to add new areas and routes to the database

https://github.com/OpenBeta/opentacos-content/tree/develop/content/USA/Oregon/Mt.%20Hood/Petes%20Pile

we need unzip and wget in base image

https://gitlab.com/openbeta/openbeta-nodejs-docker

image: registry.gitlab.com/openbeta/openbeta-nodejs-docker:16

Climbs are now children of Area.

• Call Mongo $unwind Area.children to get all climbs and insert to algolia
• Use Area.id instead of metadta.uuid

Story: Crag Finder OpenBeta/open-tacos#178

As a user I want to find areas and climbs near a latitude, longitude.

Input:

• latitude, longitude
• Optional filters:
  • radius (default to x km)
  • discipline: sport, trad, bouldering (default to all)

Todo:

• Update metadata.lat and metada.lng to Geojson type https://mongoosejs.com/docs/geojson.html
• GraphQL type: is there any benefits of switching raw lat,lng to geojson type?
• Create new query or new a new input filter

Climb job should only delete climb index

https://github.com/OpenBeta/openbeta-graphql/blob/staging/src/db/export/Typesense/Typesense.ts#L141

Use case

As a user I want to get all climbs in Colorado.

The following query failed:

query AllClimbsInColorado {
  areas(filter: {path_tokens: {tokens: "Colorado"}, leaf_status: {isLeaf: true}}) {
    areaName
    climbs {
      name
      yds
      id
    }
  }
}

Error:

{
  "errors": [
    {
      "message": "Cannot return null for non-nullable field Climb.name.",
      "locations": [
        {
          "line": 5,
          "column": 7
        }
      ],
      "path": [
        "areas",
        0,
        "climbs",
        0,
        "name"
      ],
      "extensions": {
        "code": "INTERNAL_SERVER_ERROR"
      }
    },

• #91
• New staging deployment https://stg-api.openbeta.io
• Seed staging data

Use case

As a user I want to add a new climb or edit an existing field.

Investigate JWT/role-based access control middleware for Apollo GraphQL

A starting point: https://www.apollographql.com/docs/apollo-server/security/authentication/

Right now there is a lot of grades with different formats
ex. V2/3, V2-3, V2+

• We should find one way to represent these grades, and update them in the data base. Then only allow grades that fit the new format.
• We should also think about how we can allow switching between grade systems.

Problem

The db seeding function loops through all available data files and insert them all at once causing heavy load on mongo db.

Solution

Limit the number of async calls to a few states at a time. Use p-limit or a similar package.

https://www.npmjs.com/package/p-limit

Whenever we add a new area we should also update Typesense Area collection.

Return total climbs, crags in the database.

Parent issue: #97

To do:

• Include an optional desc field

New build should be deployed automatically to stg-api.openbeta.io when we merge PRs to staging branch.

• Auto publish Docker image to docker.io
• Config ArgoCD to watch staging branch for changes

We now have bbox for all areas. It's good to calculate the area center lat,lng (metadata.lat and metadata.lng)

https://turfjs.org/docs/#centroid

Use case:

I want to search for an area by name, for example, Black Velvet Canyon and be able to click on the result to jump to the page.

Todo: Include area.uuid in Typesense index so that search widget can generate URL to the area.

The current schema is hard-coded to YDS.

• Update the Climb schema to support other systems (we can prioritize the French system for now)
• Update GQL queries
• Migrate frontend to new fields (PR TBD)
• Update USA import script #140
• Update statistics script #141
• Delete old yds fields from schema and GQL

• run clean up script and add raw .md files to opentacos-content repo
• run import

Create a GQL query to show recent edit history

• Add prevHistoryID field to link to previous changeset
• Store stream event.updateFields (describing updated fields) in db
• Expose updatedFields in GQL query

The readme is lacking set up instructions for seeding the database with test data.

• 1. Use MUUID instead of Mongo ObjectID for _id (see ClimbSchema.tsx for example)
• 2. Refactor code to use Area._id instead of Area.metadata.area_id
• 3. Retire metadata.area_id in both mongo schema and GQL typedef
• 4. Make sure frontend code uses either Area.uuid or Area.id`

This task impacts multiple layers (DB schemas, DB queries, GQL/frontend). I'd highly recommend tackling this in multiple PRs and gradually deprecate out-dated fields.

We'll want to be able to search climbs by

• name
• grade
• discipline

Leave room in the future to filter by

• stars
• description - technical, overhung

To do

• Add all countries (by ISO alpha3 codes) to the database.

Example: https://github.com/OpenBeta/openbeta-graphql/blob/staging/src/model/__tests__/AreaHistoryDataSource.ts#L52

The graphql api has rate limiting enabled to mitigate DDoS.

yarn build fails because rate limiting is too restricted

nginx-ingress log:

2022/01/03 18:26:19 [error] 4682#4682: *58324847 limiting requests, excess: 150.950 by zone "openbeta_graphql-api_0f0e359b-87be-4802-a75e-be0d7456f75b_rpm", client: 70.123.62.81, server: api.openbeta.io, request: "POST / HTTP/1.1", host: "api.openbeta.io"

Create a query to return all leaf areas, ie areas with climbs, not intermediate parent areas. Since climbs are embedded documents in an Area, this will be useful to get climbs.

1. Create a isLeaf: boolean on Area object
2. Add the new field to graphql schema, update type definition and mongo schema
3. Add isLeaf param to data source
4. Add isLeaf param to areas resolver

I removed the old code.

• get area density function https://github.com/OpenBeta/openbeta-graphql/blob/develop/src/geo-utils.ts#L36
• add it here https://github.com/OpenBeta/openbeta-graphql/blob/develop/src/db/utils/AreaUpdates.ts#L66

Make it easier to add new area from the frontend by requiring only a country code

TBD How to define a UUID for climbs and crags

Proposal

Create filter type so that we can filter areas by multiple criteria.

// find all crags at Smith Rock where grade < 5.10 
query Example1 {
  areas(filter: {name: "Smith Rock", type: "sport", yds: {'lt': "5.10"} }) {
    area_name 
  }
}

Input type tutorial: https://www.yld.io/blog/query-by-2-or-more-fields-on-graphql/

Assuming area page has the following slug: areas/<uuid>/optional-area-name

The breadcrumb component will need all ancestor uuids + display names

      {
        "area_name": "Smith Rock",
        "children": [
          {
            "area_name": "Aggro Gully",
            "metadata": {
              "lat": 44.36724,
              "lng": -121.14238
            }
          },
          _path: {
              [ <uuid of great grandparent>, "Oregon"],
              [<grandparent uuid>, "Central Oregon"],
              [<parent uuid>,"Smith Rock"],
           }
       ...
        ]
      }

we import json file directly which requires Node --experimental-json-modules flag, available in node v16.14

https://github.com/OpenBeta/openbeta-graphql/blob/staging/src/model/MutableAreaDataSource.ts#L14

To do:

• Specify 16.14.0 in package.json

Learn more https://simonplend.com/import-json-in-es-modules/

Sample climb md file

At the moment we only process data in the frontmatter section (data loader code), ignoring content below --- marker.

TO DO:

• Return the markdown body.
• Optionally, do additional parsing and return the data in 3 pieces: 'Description', 'Protection', 'Location'

• Create a change log collection to record app-level changes
• Create newArea/Country mutation
• #158
• Add description field to New Area mutation #123
• Show changes by area id
• Update children pathTokens and ancestors???
• Test rollback

Typesense tentant on https://cloud.typesense.org was created under my personal namespace. We should move it to OpenBeta project so other team members can access it.

• Update backend config (.env and K8s configmap)
• Update frontend config (.env and Vercel.com)

Create a search index on Typesense containing only area data including climb count, density

#45

Use case

I want to move an area to be under a different parent.

To do:

• Update parent id
• Update path tokens (look at area renaming code for ways to recursively update children's paths)
• Update ancestors
• Unit test

East coast

• ny
• wv
• ky

Mountain

• Utah
• Colorado
• Montana
• Idaho
• Arizona
• New Mexico
• Nevada
• Wyoming

West coast

• Oregon
• Washington
• California
• ak

Add a new query to help getting photo URLs by bounding box or area id/climb id.

#88 requires Mongo to run with as a replica set.

How do we track edit history?

For this prototype

• Use Mongo watch to save changes to Media collection to media_history collection
• Implement 'soft' delete
• Implement undo-last-change operation

https://www.mongodb.com/docs/manual/changeStreams/

Calculate area statistics, number of climbs, trad/sport breakdown, etx during db load time. Fyi -parent-child linking

Make db connection str more flexible to support "mini" database setup (using docker) and managed cluster in production