Add Authentication Bearer token about restler-fuzzer

[llaumegui27]

Update, after searching for a long time, I've succeded to make it work, but in an unconventional way. I'll explain, when I was looking in files generated by compiling of my api spec (swagger.json), particularly in grammar.json and grammar.py.

For my grammar.json I was surprised not to see an Authorization field in headers part :

{
  "Requests": [
    {
      "id": {
        "endpoint": "",
        "method": "Get"
      },
      "method": "Get",
      "basePath": "",
      "path": [],
      "queryParameters": [
        [
          "Schema",
          {
            "ParameterList": []
          }
        ]
      ],
      "bodyParameters": [
        [
          "Schema",
          {
            "ParameterList": []
          }
        ]
      ],
      "headerParameters": [
        [
          "Schema",
          {
            "ParameterList": []
          }
        ],
        [
          "DictionaryCustomPayload",
          {
            "ParameterList": []
          }
        ]
      ],
      "token": "Refreshable",
      "headers": [
        [
          "Accept",
          "application/json"
        ],
        [
          "Host",
          "localhost:8080"
        ],
      ],
      "httpVersion": "1.1",
      "requestMetadata": {
        "isLongRunningOperation": false
      }
    },
[...]

Then I went to check de grammar.py and I had the idea to add my Authorization token here, like this :

# Endpoint: , method: Get
request = requests.Request([
    primitives.restler_static_string("GET "),
    primitives.restler_basepath(""),
    primitives.restler_static_string("/"),
    primitives.restler_static_string(" HTTP/1.1\r\n"),
    primitives.restler_static_string("Accept: application/json\r\n"),
    primitives.restler_static_string("Host: localhost:8080\r\n"),
    primitives.restler_static_string("Authorization: Bearer <MY_JWT_TOKEN>\r\n"), # I've just add this line 
    primitives.restler_refreshable_authentication_token("authentication_token_tag"),
    primitives.restler_static_string("\r\n"),

],
requestId=""
)
req_collection.add_request(request)

And after launching the Test command : ../restler/Restler test --grammar_file Compile/grammar.py --dictionary_file Compile/dict.json --settings Compile/engine_settings.json --no_ssl :

Starting task Test...
Using python: 'python3' (Python 3.11.7)
Request coverage (successful / total): 1 / 6
Attempted requests: 5 / 6
No bugs were found.
See 'coverage_failures_to_investigate.txt' to investigate API coverage.
Task Test succeeded.
Collecting logs...

It worked, here I only modified one route (the root of the "/" api) on my other routes, which is why you see 1/6.
Now I'd like to know why the compilation of my swagger.json doesn't take into account the authorization part that should appear in grammar.json and grammar.py. (You can see a trailer of my specification file in my first post where I specify that certain API routes require a JWT bearer authorization token)

I'd also like to know how it's displayed in these two files when the specification is compiled correctly, so that we can use the location method or other with the engine_settings.json configuration, because my engine_settings.json contains only this:

{
  "max_combinations": 20,
  "exclude_requests": [
    {
      "endpoint": "/login",
      "methods": ["POST"]
    }
  ]
}

Any help is welcome !

[marcindulak]

The expected contents of authentication_token.txt is not intuitive, and not illustrated well in the docs https://github.com/microsoft/restler-fuzzer/blob/main/docs/user-guide/Authentication.md, but mentioned in several issues in this repo. I hope someone with a definitive understanding of the code will make a PR to make the documentation more explicit, since there are many issues about tokens.
The format below may work for you

{"id":{}}
atokenhere

If you solve the token issue, it may be better to open a separate issue as a follow up with other questions.

[llaumegui27]

Thanks for your answer @marcindulak but I have two questions :

1. What does it correspond to : { "id":{}}
   In my txt file where there is my token, do I have to modify it, is it just used because we only read the second line of the file, can you tell me more about this?

2. I've tried to make this, my txt file :

{"id":{}}
<MY_JWT_TOKEN>

My engine_settings.json :

{
  "authentication": {
    "token": {
      "location": "/home/guillaume/fuzzing-tools/restler/token_test/authentication_token.txt",
      "token_refresh_interval":  21000
    }
  },
  "exclude_requests": [
    {
      "endpoint": "/login",
      "methods": ["POST"]
    }
  ]
}

It doesn't work, the result :

Starting task Test...
Using python: 'python3' (Python 3.11.7)
Request coverage (successful / total): 0 / 6
Attempted requests: 5 / 6
No bugs were found.
See 'coverage_failures_to_investigate.txt' to investigate API coverage.
Task Test succeeded.
Collecting logs...

But I use wireshark to debug and, I've saw this :

GET / HTTP/1.1
Accept: application/json
Host: localhost:8080
eyJhbGciOiJIUzI9NiIsInR5cCI6IkpXVCJ9.eyJ1c2VybmFtZSI6Imd1aWxsYXVtYRIsImlhdCI6MTcwNzkwMjMwOSwiZXhwIjoxNzA3OTIzOTA5fQ.A9Zg9Q6rjv4MrQgyFAPm9-YfPcybCrDqRbneIbQNAW0
Content-Length: 0
User-Agent: restler/9.2.3
x-restler-sequence-id: eb07e22e-6424-47e2-bba0-3eafc3b0eba8

HTTP/1.1 400 Bad Request
Connection: close

And now you can see the token but no prefix like : Authorization: Bearer
So i think the problem is my API spec before the generation doesn't take into account the authorization field in the Header, though I tried to refer to the Open Api doc for that.
You can see a trailer of my swagger.json on my first post, something wrong with this file ? And a trailer of my grammar.json and grammar.py on my second post.
Thx for the help !

[marcindulak]

1. please refer to https://github.com/microsoft/restler-fuzzer/blob/main/docs/user-guide/Authentication.md and various issues in this repo, e.g. #460 (comment) or #168 (comment)
2. this suggests the expected contents of authentication_token.txt may be

{"id":{}}
Authorization: Bearer atokenhere