Two MIME type issues in the Amazon BedrockRuntime client about paws

[DyfanJones]

Sorry about that @alex23lemm and thanks for raising this issue. It looks like we will need a custom handler function for bedrockruntime. I will have a little look at botocore to see if they have something there :)

[DyfanJones]

My guess is we are missing these default content-type and accept header:

RestJSONSerializer:
https://github.com/boto/botocore/blob/48c1bcfdd752f5fa965451e58103ccebad3e165b/botocore/serialize.py#L671-L700

add default accept header:
https://github.com/boto/botocore/blob/48c1bcfdd752f5fa965451e58103ccebad3e165b/botocore/handlers.py#L673-L676

[DyfanJones]

Hi @alex23lemm,

I believe I have an initial implementation.

remotes::install_github("DyfanJones/paws/paws.common", ref = "application/json")

library(paws)
client <- bedrockruntime(config(credentials(profile = "paws")))

body <- list(prompt = "\n\nHuman: Tell me a funny joke about outer space\n\nAssistant:") |>
  jsonlite::toJSON(auto_unbox = TRUE)

response <- client$invoke_model(
  body = body,
  modelId = "ai21.j2-ultra"
)

jsonlite::fromJSON(rawToChar(response$body))
#> $id
#> [1] 1234
#> 
#> $prompt
#> $prompt$text
#> [1] "\n\nHuman: Tell me a funny joke about outer space\n\nAssistant:"
#> 
#> $prompt$tokens
#>    generatedToken.token generatedToken.logprob generatedToken.raw_logprob
#> 1           <|newline|>            -4.71495771                -4.71495771
#> 2           <|newline|>            -0.06756511                -0.06756511
#> 3                ▁Human            -9.61753464                -9.61753464
#> 4                     :            -5.56536293                -5.56536293
#> 5              ▁Tell▁me            -7.38631439                -7.38631439
#> 6              ▁a▁funny            -6.16015053                -6.16015053
#> 7           ▁joke▁about            -4.98311758                -4.98311758
#> 8          ▁outer▁space            -8.84491539                -8.84491539
#> 9           <|newline|>            -0.02789355                -0.02789355
#> 10          <|newline|>            -2.34469557                -2.34469557
#> 11           ▁Assistant            -5.89183331                -5.89183331
#> 12                    :            -0.01577717                -0.01577717
#>    topTokens textRange.start textRange.end
#> 1         NA               0             1
#> 2         NA               1             2
#> 3         NA               2             7
#> 4         NA               7             8
#> 5         NA               8            16
#> 6         NA              16            24
#> 7         NA              24            35
#> 8         NA              35            47
#> 9         NA              47            48
#> 10        NA              48            49
#> 11        NA              49            58
#> 12        NA              58            59
#> 
#> 
#> $completions
#>                                                                   data.text
#> 1  They say the earth is round,\n\nBut it's square,\nThey think it's flat\n
#>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                data.tokens
#> 1 ▁They▁say, ▁the▁earth, ▁is, ▁round, ,, <|newline|>, <|newline|>, ▁But, ▁it's, ▁square, ,, <|newline|>, ▁They▁think, ▁it's, ▁flat, <|newline|>, -7.92391395568848, -1.7487964630127, -0.306633919477463, -0.0392366088926792, -0.931648254394531, -1.87512421607971, -0.141698807477951, -0.738928914070129, -2.99068760871887, -0.54417484998703, -2.74556589126587, -1.28033459186554, -9.45336151123047, -0.948258876800537, -0.110951364040375, -2.02394270896912, -7.92391395568848, -1.7487964630127, -0.306633919477463, -0.0392366088926792, -0.931648254394531, -1.87512421607971, -0.141698807477951, -0.738928914070129, -2.99068760871887, -0.54417484998703, -2.74556589126587, -1.28033459186554, -9.45336151123047, -0.948258876800537, -0.110951364040375, -2.02394270896912, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 0, 9, 19, 22, 28, 29, 30, 31, 34, 39, 46, 47, 48, 58, 63, 68, 9, 19, 22, 28, 29, 30, 31, 34, 39, 46, 47, 48, 58, 63, 68, 69
#>   finishReason.reason finishReason.length
#> 1              length                  16

^{Created on 2024-02-20 with reprex v2.1.0}

[DyfanJones]

Are you sure the returning object should be a json? When you say it should return json do you mean a Json string? Or a json but in bytes similar to what Boto3 is doing.

import boto3
import json

bedrock = boto3.Session(profile_name="paws").client(service_name="bedrock-runtime")
body = json.dumps(
    {
        "prompt": "\n\nHuman: Tell me a funny joke about outer space\n\nAssistant:"
    }
)

response = bedrock.invoke_model(body=body, modelId="ai21.j2-ultra")
obj = response.get("body").read()

type(obj)
#> bytes

This is basically what paws is returning. It is returning a raw vector that is in json format.

library(paws)
client <- bedrockruntime(config(credentials(profile = "paws")))

body <- list(prompt = "\n\nHuman: Tell me a funny joke about outer space\n\nAssistant:") |>
  jsonlite::toJSON(auto_unbox = TRUE)

response <- client$invoke_model(
  body = body,
  modelId = "ai21.j2-ultra"
)

yyjsonr::read_json_raw(response$body)
#> $id
#> [1] 1234
#> 
#> $prompt
#> $prompt$text
#> [1] "\n\nHuman: Tell me a funny joke about outer space\n\nAssistant:"
#> 
#> $prompt$tokens
#>                                           generatedToken topTokens textRange
#> 1      <|newline|>, -4.71495771408081, -4.71495771408081      NULL      0, 1
#> 2  <|newline|>, -0.0675651058554649, -0.0675651058554649      NULL      1, 2
#> 3           ▁Human, -9.61753463745117, -9.61753463745117      NULL      2, 7
#> 4                :, -5.56536293029785, -5.56536293029785      NULL      7, 8
#> 5         ▁Tell▁me, -7.38631439208984, -7.38631439208984      NULL     8, 16
#> 6           ▁a▁funny, -6.1601505279541, -6.1601505279541      NULL    16, 24
#> 7      ▁joke▁about, -4.98311758041382, -4.98311758041382      NULL    24, 35
#> 8     ▁outer▁space, -8.84491539001465, -8.84491539001465      NULL    35, 47
#> 9  <|newline|>, -0.0278935506939888, -0.0278935506939888      NULL    47, 48
#> 10     <|newline|>, -2.34469556808472, -2.34469556808472      NULL    48, 49
#> 11      ▁Assistant, -5.89183330535889, -5.89183330535889      NULL    49, 58
#> 12           :, -0.0157771650701761, -0.0157771650701761      NULL    58, 59
#> 
#> 
#> $completions
#>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               data
#> 1  Mars is red\n\nWhy can't trees grow on Mars?\nBecause it's only fit for is, ▁Mars, -7.97078895568848, -7.97078895568848, ▁is, -1.78697526454926, -1.78697526454926, ▁red, -0.91144585609436, -0.91144585609436, <|newline|>, -0.178860738873482, -0.178860738873482, <|newline|>, -0.128698348999023, -0.128698348999023, ▁Why, -2.22174859046936, -2.22174859046936, ▁can't, -1.63458228111267, -1.63458228111267, ▁trees, -4.07934379577637, -4.07934379577637, ▁grow, -0.0818066596984863, -0.0818066596984863, ▁on▁Mars, -0.264529824256897, -0.264529824256897, ?, -0.0579109378159046, -0.0579109378159046, <|newline|>, -0.163965195417404, -0.163965195417404, ▁Because, -0.254661977291107, -0.254661977291107, ▁it's▁only, -5.48099517822266, -5.48099517822266, ▁fit▁for, -1.99698293209076, -1.99698293209076, ▁is, -3.21487283706665, -3.21487283706665, 0, 5, 5, 8, 8, 12, 12, 13, 13, 14, 14, 17, 17, 23, 23, 29, 29, 34, 34, 42, 42, 43, 43, 44, 44, 51, 51, 61, 61, 69, 69, 72
#>   finishReason
#> 1   length, 16

^{Created on 2024-02-20 with reprex v2.1.0}

[DyfanJones]

Closing as paws.common 0.7.1 is now on the cran

[alex23lemm]

Hi @DyfanJones ,
thanks for the fix and apologies for the delayed response! I was on a lengthy business trip with limited hands-on-keyboard time. You were correct with your remark regarding the return object, which should and now is a json in raw bytes. Your fixes solved the issues described above.

After the latest release of paws.common-v0.7.1, I only found one minor difference in paws compared to boto3 for invoke_model.

The default contentType value is application/json (see boto3 documentation here), as shown below, re-using the Python example from above.

response = bedrock.invoke_model(body=body, modelId="anthropic.claude-v2:1")
print(response.get("contentType"))`

application/json

However, the R equivalent returns the following:

response <- bedrock$invoke_model(body = body, modelId = "anthropic.claude-v2:1")
response$contentType

logical(0)

The behavior stays the same even when setting the contentType explicitly:

response <- bedrock$invoke_model(body = body, contentType = "application/json", modelId = "anthropic.claude-v2:1")
response$contentType

logical(0)

[DyfanJones]

Interesting, I will raise another ticket so we don't lose that bug

[alex23lemm]

Thanks!