tofu-tf / typed-schema Goto Github PK

Today we have a cross-reference between typed-schema artifacts and derevo-core, which introduces some of inconvenience when an binary incompatible version of typed-schema comes out. I think we should migrate derevo-tschema into this repository (and release them simultaneously). What do you think, @Odomontois ?

OpenAPI allows descriptions for http status-codes of responces
https://swagger.io/docs/specification/describing-responses/

But at the moment there is no syntax to specify them via the properties file:
https://github.com/TinkoffCreditSystems/typed-schema/blob/master/modules/swagger/src/main/scala/ru/tinkoff/tschema/swagger/PathDescription.scala

typo in https://tinkoffcreditsystems.github.io/typed-schema/getting_started.html

   [error] /MkSwagger-FatalWarnings/src/main/scala/pkg/inst/package.scala:26:9: discarded non-Unit value
   [error]     def schema: SwaggerBuilder = MkSwagger(api)

Extra small project with represented problem
MkSwagger-FatalErrors.zip

SwaggerMapper[FormField[name, T]] is inferred with an empty types field
https://scastie.scala-lang.org/aNaAiIpzTK2nyL3TuWJwvQ

Try @eld0727 's idea to get rid of aborts in macro. This should speed up compilation in case of errors a lot

Example from https://swagger.io/docs/specification/data-models/oneof-anyof-allof-not/:

paths:
  /pets:
    patch:
      requestBody:
        content:
          application/json:
            schema:
              oneOf:
                - $ref: '#/components/schemas/Cat'
                - $ref: '#/components/schemas/Dog'
              discriminator:
                propertyName: pet_type
      responses:
        '200':
          description: Updated
components:
  schemas:
    Pet:
      type: object
      required:
        - pet_type
      properties:
        pet_type:
          type: string
      discriminator:
        propertyName: pet_type
    Dog:     # "Dog" is a value for the pet_type property (the discriminator value)
      allOf: # Combines the main `Pet` schema with `Dog`-specific properties 
        - $ref: '#/components/schemas/Pet'
        - type: object
          # all other properties specific to a `Dog`
          properties:
            bark:
              type: boolean
            breed:
              type: string
              enum: [Dingo, Husky, Retriever, Shepherd]
    Cat:     # "Cat" is a value for the pet_type property (the discriminator value)
      allOf: # Combines the main `Pet` schema with `Cat`-specific properties 
        - $ref: '#/components/schemas/Pet'
        - type: object
          # all other properties specific to a `Cat`
          properties:
            hunts:
              type: boolean
            age:
              type: integer

Replace all mediaTypes with strings and Status Codes with Ints or custom enum

Currently type complexity prevents easy explicit type declaration of Serve instances for custom directives, forcing sometimes to use derived types, which is discouraged for implicits

Get rid of shitty english, cover all aspects. Probably translate internal wiki to english

Support anyOf

Reference: https://swagger.io/docs/specification/data-models/oneof-anyof-allof-not/

Sometimes maybe usefull to return part of query params as Map[String, String].
For example in case of proxying request when you want to handle few params and send rest(or all) to a different system.
Smth like

operation('proxyRequest) :>
      queryParam[Long]('system_id) :>
      queryParam[String]('user_id) :>
      rest[Map[String, String] :>
      get[SystemResponse]

It may be useful to implement inheritance using allOf. Some openApi generators check this field in ancestors, and doesn't generate inheritance if this field is not presented.

Problem

Error while running example projects

sbt finagleEnv/run

or

sbt finagleZio/run

[error]     in parameter 'child' of product type ru.tinkoff.tschema.example.MultiParams.User
[error]   @derive(Swagger, HttpParam, AsOpenApiParam, show)
[error]    ^
[error] one error found

At the moment Rejection.Handler is a pure function from Rejection to Finagle Response and
is used as ZIO.succeed(handler(..)).

We could rework Rejection.Handler into Rejection => URIO[R, Response] in order to enable effectful logic embedding into rejection handling, e.g. logging.

Make a tut microsite

There is only one action allowed per unique prefix now. But it violates REST API best practices.

https://habr.com/post/351890/
https://docs.microsoft.com/en-us/azure/architecture/best-practices/api-design
https://hackernoon.com/restful-api-designing-guidelines-the-best-practices-60e1d954e7c9

So in order to support REST we must be able to create these routes:
/organizations
/organizations/{id}
/organizations/{id}/accounts
/organizations/{orgId}/accounts/{accountId}

In case

operation('request) :>
	queryParam[Option[Double]]('lat) :>
	queryParam[Option[Double]]('lon) :>
        queryParam[Option[Long]]('accuracy) :>
        get[Response]

with GET /request?lat=0.0&lon=0.0&accuracy=5.3
typed schema will not raise error, but just skip accuracy.

Hi,
I am gonna (miss)use your issue tracker to start this conversation as I am not able to find any mail address or Gitter channel. So, mea culpa :).

Do you know Typedapi? It is basically the same idea, bringing Servant to Scala. But so far (at least in its release branch 0.1.0-release) it supports the server and client side and the following frameworks/libs: akka-http, http4s and Scalaj-Http. Furthermore, it compiles to JavaScript with ScalaJS.

It seems to me that your project is at the beginning (?). Maybe a cooperation between the two projects makes sense to not develop the same thing twice. And especially running in the same problems twice :D.

Let me know what you think.

Add SwaggerMapper instance for QueryParams atom

Move akka-http and swagger to submodules
Also get rid of akka http domain types in the swagger

current if send a request with a broken body on existing endpoint - typed-schema returns 404

"ru.tinkoff" %% "typed-schema-finagle-zio" % "0.12.0"

Currently it return 200 with null text in the response in case of None.

Problem

1. Path matching behavior depends on the order, in which the endpoints are defined
2. capture[String]('id) matches empty string in paths like /resource/{id}, but capture[Int] does not

How to reproduce

Typed Schema version: v0.12.0

Consider having two API endpoints:

• GET /resource/{id}
• GET /resource/

Currently, the api endpoints definition using typed-schema would be:

  def listBooks =
    prefix('books) :>
      PathEnd :>
      get :>
      key('listBooks) :>
      $$[List[String]]

  def getBook =
    prefix('books) :>
      capture[String]('bookId) :>
      PathEnd :>
      get :>
      key('getBook) :>
      $$[String]

and a handler for that could look like this:

class TestController[F[_]: Sync: RoutedPlus] { self =>
  private val books = Map(
    "1" -> "Book1",
    "2" -> "Book2",
    "3" -> "Book3",
    "4" -> "Book4",
  )

  def routes: H[Response] = MkService[H](api)(self)
  def listBooks(): List[String] = books.values.toList
  def getBook(bookId: String): String = books.getOrElse(bookId, "Not found")

}

where PathEnd checks that all symbols in the request path have been consumed.

Case 1: Definition order

1.1

Final definition:

  def api = getBook <|> listBooks

Actual:

> curl http://localhost:8080/books
< "Not found"

Expected:

> curl http://localhost:8080/books
< ["Book1","Book2","Book3","Book4"]

1.2

Final definition:

  def api = listBooks  <|> getBook

Actual matches expected

> curl http://localhost:8080/books
< ["Book1","Book2","Book3","Book4"]

Case 2: Matching inconsistency

Final definition:

  def api = getBook <|> listBooks

Let's modify getBook to accept Int:

  def getBook =
    prefix('books) :>
      capture[Int]('bookId) :>
      PathEnd :>
      get :>
      key('getBook) :>
      $$[String]

And call the API again (actual matches expected):

> curl http://localhost:8080/books
< ["Book1","Book2","Book3","Book4"]

Method results are now strict and evaluated before Routable has a chance to verify path is fully matched.

Publish derevo library and update examples to correct version

groupPrefix is convenient for the purpose of ordering handler code. However it doesn't have much flexibility (i.e. declaring prefix that differs from group name in case would be manual operators chaining and look like prefix("my_group") |> group("myGroup")).
It would be nice to have some flexible DSL combinator for these cases.

Add special directive allowing to automatically apply CORS rules according to implicit config

Currently, there's a cyclic dependency between derevo and tschema.

Swagger have build-in formats https://swagger.io/specification/
But it`s writing in typed-schema is incorrect: dateTime instead of date-time

See attached image - first field (form one) is red and have correct writing date-time, second is not highlighted because of incorrect writing

Used java version: 1.8.0_191
Step to reproduce: sbt "project examples" +clean ++2.11.12 run
Error description:

[error] Exception Details:
[error]   Location:
[error]     ru/tinkoff/tschema/swagger/MkSwagger$macroInterface$ServePA$.equals$extension(Lscala/runtime/BoxedUnit;Ljava/lang/Object;)Z @38: pop
[error]   Reason:
[error]     Attempt to pop empty stack.
[error]   Current Frame:
[error]     bci: @38
[error]     flags: { }
[error]     locals: { 'ru/tinkoff/tschema/swagger/MkSwagger$macroInterface$ServePA$', 'scala/runtime/BoxedUnit', 'java/lang/Object', 'java/lang/Object', integer }
[error]     stack: { }
[error]   Bytecode:
[error]     0x0000000: 2c4e 2dc1 0032 9900 0904 3604 a700 0603
[error]     0x0000010: 3604 1504 9900 4b2c c700 0701 a700 102c
[error]     0x0000020: c000 32b6 0035 57bb 0037 59bf 3a05 b200
[error]     0x0000030: 1457 b200 14b2 0014 57b2 0014 3a06 59c7
[error]     0x0000040: 000c 5719 06c6 000e a700 0f19 06b6 003b
[error]     0x0000050: 9900 0704 a700 0403 9900 0704 a700 0403
[error]     0x0000060: ac                                     
[error]   Stackmap Table:
[error]     append_frame(@15,Object[#4])
[error]     append_frame(@18,Integer)
[error]     same_frame(@31)
[error]     same_locals_1_stack_item_frame(@44,Null)
[error]     full_frame(@75,{Object[#2],Object[#16],Object[#4],Object[#4],Integer,Null,Object[#16]},{Object[#16]})
[error]     same_frame(@83)
[error]     same_frame(@87)
[error]     same_locals_1_stack_item_frame(@88,Integer)
[error]     chop_frame(@95,2)
[error]     same_locals_1_stack_item_frame(@96,Integer)
[error] java.lang.VerifyError: Operand stack underflow
[error] Exception Details:
[error]   Location:
[error]     ru/tinkoff/tschema/swagger/MkSwagger$macroInterface$ServePA$.equals$extension(Lscala/runtime/BoxedUnit;Ljava/lang/Object;)Z @38: pop
[error]   Reason:
[error]     Attempt to pop empty stack.
[error]   Current Frame:
[error]     bci: @38
[error]     flags: { }
[error]     locals: { 'ru/tinkoff/tschema/swagger/MkSwagger$macroInterface$ServePA$', 'scala/runtime/BoxedUnit', 'java/lang/Object', 'java/lang/Object', integer }
[error]     stack: { }
[error]   Bytecode:
[error]     0x0000000: 2c4e 2dc1 0032 9900 0904 3604 a700 0603
[error]     0x0000010: 3604 1504 9900 4b2c c700 0701 a700 102c
[error]     0x0000020: c000 32b6 0035 57bb 0037 59bf 3a05 b200
[error]     0x0000030: 1457 b200 14b2 0014 57b2 0014 3a06 59c7
[error]     0x0000040: 000c 5719 06c6 000e a700 0f19 06b6 003b
[error]     0x0000050: 9900 0704 a700 0403 9900 0704 a700 0403
[error]     0x0000060: ac                                     
[error]   Stackmap Table:
[error]     append_frame(@15,Object[#4])
[error]     append_frame(@18,Integer)
[error]     same_frame(@31)
[error]     same_locals_1_stack_item_frame(@44,Null)
[error]     full_frame(@75,{Object[#2],Object[#16],Object[#4],Object[#4],Integer,Null,Object[#16]},{Object[#16]})
[error]     same_frame(@83)
[error]     same_frame(@87)
[error]     same_locals_1_stack_item_frame(@88,Integer)
[error]     chop_frame(@95,2)
[error]     same_locals_1_stack_item_frame(@96,Integer)

EXAMPLE:

val upload = post |>
  operation('download) |>
  formField[(FileInfo, Source[ByteString, Any])]('file) |>
  $$[String]

val download = get |>
  operation(Symbol("download")) |>
  $$[File]

AKKA HTTP DOCUMENTATION:

fileUpload
https://doc.akka.io/docs/akka-http/current/routing-dsl/directives/file-upload-directives/fileUpload.html

getFromFile
https://doc.akka.io/docs/akka-http/current/routing-dsl/directives/file-and-resource-directives/getFromFile.html

I'm getting notable improvement of compilation speed in latest milestone 2.4.0-M1, also they've fixed nasty duplication bug in RemoveAll.
@Odomontois WDYT, should we consider merging milestones, if they contain quality of life improvements and bugfixes?

It would be nice to support at least http4s beyond current Akka http.

Swagger creation fails with NullPointerException if one class has reference to itself, for example:

  case class ClassA(name: String, parent: Option[ClassA])

Full test case:

import derevo.circe.{decoder, encoder}
import derevo.derive
import ru.tinkoff.tschema.swagger.{MkSwagger, Swagger}
import ru.tinkoff.tschema.syntax._

object definitions {

  @derive(encoder, decoder, Swagger)
  case class ClassA(name: String, parent: Option[ClassA])

  def api = tagPrefix("module") |> (operation("test") |> get |> complete[ClassA])
}

object TestModule {

  import definitions._

  trait ModuleApi {
    def test: ClassA
  }

  object ModuleApiImpl extends ModuleApi {
    override def test: ClassA = {
      ClassA(
        name = "John Doe",
        parent = Some(ClassA(
          name = "John Doe Sr.",
          parent = None)
        )
      )
    }
  }

  def swagger = MkSwagger(api)
}

object Test {

  def main(args: Array[String]): Unit = {
    TestModule.swagger // throw java.lang.NullPointerException: Cannot invoke "ru.tinkoff.tschema.swagger.SwaggerTypeable.typ()" because the return value of "shapeless.Lazy.value()" is null
  }
}

The problem occurred when moving from older version 0.10.6 to latest 0.15.2 (reproduces on 0.15.2, didn't reproduce on 0.10.6)

Java 17.0.1, Scala 2.13.10

Let define some typed-schema endpoint with prefix atom.

def test =
    get :>
    prefix("test") :>
    key("testMethod") :>
    someAtom :>
    $$[Unit]

def someAtom: SomeAtom = null
class SomeAtom extends DSLAtom

And define serving type class implementations for SomeAtom

implicit def serveAkka[In <: HList]: akkaHttp.Serve.Check[SomeAtom, In] =
  akkaHttp.Serve.serveCheck {
    println("gotcha")
    akka.http.scaladsl.server.Directives.pass
  }

implicit def serveFinagle[F[_]: Sync, In <: HList]: finagle.Serve[SomeAtom, F, In, In] =
  (in, f) => 
    for {
      _    <- Sync[F].delay(println("gotcha"))
      next <- f(in)
    } yield next

Then, if we use our endpoint with URL = "testkek" or with other string with "test" prefix, all atoms will be served and "gotcha" will be printed, but request will be rejected due checkPathEnd in finagle or pathEnd in akka.

Also, if we use combination of prefix atoms, for example

prefix("test") :> prefix("kek")

Request with "testkek" also will be rejected.

Maybe it makes sense to change behavior of serving prefix atom to match till '/' or string end to avoid extra atoms serving

Given piece of code:

implicit final val RedirectSwagger: MkSwagger[Complete[Redirect]] = MkSwagger.single[Complete[Redirect]](
    OpenApiOp(
      responses = OpenApiResponses(
        codes = Map(Found.intValue -> OpenApiResponse(
          description = "Редирект на страницу".some,
          headers = Map(Location.name -> SwaggerStringValue(pattern = """https:\/\/(?:.+)?""".some))
        ))
      )
    ),
    TreeMap.empty
  )

... generates invalid format:

  "302": {
    "description": "Редирект на страницу",
    "content": {},
    "headers": {
      "Location": {
        "pattern": "https:\\/\\/(?:.+)?",
        "type": "string"
      }
    }
  }

Correct example in spec:

"headers": {
    "X-Rate-Limit-Limit": {
      "description": "The number of allowed requests in the current period",
      "schema": {
        "type": "integer"
      }
    }
}

Отметить для людей момент с особенностями компиляции проекта examples (или вообще выпилить BuildInfo)

Serve definition for Patch
uses scala.tools.nsc.ast.parser.Patch from Scala compiler instead of
ru.tinkoff.tschema.typeDSL.Patch.

Current example shows only usage with derevo library for derivation here
One may not want to use it, so it would be great to show examples of using queryParam and derivation without derevo (or maybe some docs).

Currently SwaggerPrimitive is defined as

class SwaggerPrimitive[Typ <: SwaggerValue](
    val format: Option[OpenApiFormat[Typ]] = None,
    ...
) 

where OpenApiFormat is a sealed trait. This means that one cannot define custom primitives with arbitrary formats whilst the OpenAPI Data Types spec states:

However, format is an open value, so you can use any formats, even not those defined by the OpenAPI Specification,
...
Tools can use the format to validate the input or to map the value to a specific type in the chosen programming language. Tools that do not support a specific format may default back to the type alone, as if the format is not specified.