rawhat / mist Goto Github PK

Doesn't have to be super thorough benchmarking, but it would be cool to see if we're in the same ballpark as the most popular web server.

Hello!

The README example on how to use websockets doesn't compile currently. Seems to be some duplicate modules

Thanks,
Louis

This would be useful for unit testing various pieces of the stack.

Hello! I tried Mist as a replacement for Elli in my application and got this error:

{"Error in process ~p with exit value:~n~p~n",
 [<0.172.0>,
  {#{function => <<"handler">>,gleam_error => assert,line => 245,
     message => <<"Assertion pattern match failed">>,
     module => <<"mist/http">>,
     value => {error,unknown_header}},
   [{mist@http,'-handler/1-fun-0-',3,
               [{file,"build/dev/erlang/mist/build/[email protected]"},
                {line,333}]},
    {glisten@tcp,'-start_handler/3-fun-0-',4,
                 [{file,"build/dev/erlang/glisten/build/[email protected]"},
                  {line,211}]},
    {gleam@otp@actor,loop,1,
                     [{file,"build/dev/erlang/gleam_otp/build/gleam@[email protected]"},
                      {line,92}]}]}]}

Here's the request headers from firefox

GET / undefined
Host: localhost:3000
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:100.0) Gecko/20100101 Firefox/100.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Language: en-GB,en;q=0.5
Accept-Encoding: gzip, deflate, br
DNT: 1
Connection: keep-alive
Cookie: csrftoken=redacted; firstVisit=redacted; sessionid=redacted; ph_GO532nkfIyRbVh8r-ts579S0ibtS4N7F8q1u7qy9FyY_posthog=%7B%22distinct_id%22%3A%22development-309%22%2C%22%24device_id%22%3A%2217f2233bb99e1f-0c5659974edb81-455a69-7e9000-17f2233bb9a8d6%22%2C%22%24user_id%22%3A%22development-309%22%2C%22%24initial_referrer%22%3A%22http%3A%2F%2Flocalhost%3A8000%2F%22%2C%22%24initial_referring_domain%22%3A%22localhost%3A8000%22%2C%22%24referrer%22%3A%22http%3A%2F%2Flocalhost%3A8000%2Fin-house-legal%2Fquery%2Fagreements%2F%22%2C%22%24referring_domain%22%3A%22localhost%3A8000%22%2C%22%24sesid%22%3A%5B1653655256592%2C%221810588be10b6a-070d87da3b86a88-402e2c34-4b9600-1810588be11e5a%22%5D%2C%22%24session_recording_enabled_server_side%22%3Afalse%2C%22%24active_feature_flags%22%3A%5B%5D%2C%22%24enabled_feature_flags%22%3A%7B%7D%7D; uid=SFMyNTY.MQ.3UIahins3fngCF2xLC2znGYe_xSbmG_bRdx0YSTwt_c; sessionid-3GDYO=redacted; CSRF-Token-3GDYO=tJoio2hbqKggK7fHZgFZyWVPnA7wySLf; CSRF-Token-CKIKR=Xc75SKFZS9qDDgLKD26rLYJtgtiH7Gja; sessionid-CKIKR=eikQox9V6M9sQwRnLHLj9HE5b2zRtopN; sessionid-NHOUY=wV9rHjEGqHzSpTfVrtMw3SkSUfV99YWq; CSRF-Token-NHOUY=ePu9N5NuRQSS6bwRfvrRTiSNdKjJV9Ro
Upgrade-Insecure-Requests: 1
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Pragma: no-cache
Cache-Control: no-cache

My first thought from the code was that it's due to the Cookie header being large?

Hello!

Is there a way to the the IP address of the requester?

Thanks,
Louis

Hello!

The current websocket example shows how to route based upon method and path, and also how to include a pre-existing websocket handler in those routes. I think it would be useful to instead omit the routing and show how to make a custom websocket handler, the routing can be taken from the routing examples.

Thanks,
Louis

Minimum requirement here is an upstream change in glisten to return the supervisor Subject, and then having the start_http(s) methods also return that

Hi,

I'm the author of Howdy, which wraps a simple API around any webserver the conforms the Geam's HTTP server functions. On lpil's recommendation I moved it from Cowboy to Mist, and all works great when I compile the project on my Mac or Linux, but I'm having an issue with Windows.

If I do the following:

gleam new webtest
cd webtest
gleam add mist
gleam run

It returns this error (Only in Windows):

  Resolving versions
  Compiling mist_server
   Compiled in 0.38s
    Running mist_server.main
=INFO REPORT==== 14-Jun-2022::09:34:31.298000 ===
    application: gleam_http
    exited: stopped
    type: temporary

=INFO REPORT==== 14-Jun-2022::09:34:31.319000 ===
    application: gleeunit
    exited: stopped
    type: temporary

=INFO REPORT==== 14-Jun-2022::09:34:31.319000 ===
    application: gleam_stdlib
    exited: stopped
    type: temporary

exception error: no match of right hand side value
                 {error,{hackney,{"no such file or directory","hackney.app"}}}

I have tried added hackney as a dependency to the project gleam add hackney but it doesn't solve the issue, have you come across this before, and/or do you know a solution to this?

Thanks

Mike

These could be sent over as chunked encoding for HTTP/1.1.

I'm not sure how feasible this would be, but adding the JavaScript target support to mist would be amazing and very appreciated. Other libraries (like wisp) depend on mist, and since mist does not support a JavaScript target, they do not support a JavaScript target either. Anyway, not sure how feasible this would be, but it is something to think about! I am willing to help with this, though I don't know much about Erlang.

I am writing a simple server using websocket.
There is no problem with normal data transfer after making a connection, but the following error may occur in some cases when more data is transferred.

=ERROR REPORT==== 7-Nov-2023::01:35:07.483288 ===
Error in process <0.124.0> with exit value:
{function_clause,
    [{prim_inet,recv0,[#Port<0.13>,-6,-1],[]},
     {mist@internal@websocket,frame_from_message,3,
         [{file,
              "/Users/akiomi/src/github.com/akiomik/mist_playground/build/dev/erlang/mist/_gleam_artefacts/mist@[email protected]"},
          {line,161}]},
     {mist@internal@websocket,'-initialize_connection/5-fun-11-',4,
         [{file,
              "/Users/akiomi/src/github.com/akiomik/mist_playground/build/dev/erlang/mist/_gleam_artefacts/mist@[email protected]"},
          {line,254}]},
     {gleam_erlang_ffi,'-map_selector/2-fun-0-',3,
         [{file,
              "/Users/akiomi/src/github.com/akiomik/mist_playground/build/dev/erlang/gleam_erlang/_gleam_artefacts/gleam_erlang_ffi.erl"},
          {line,169}]},
     {gleam_erlang_ffi,select,2,
         [{file,
              "/Users/akiomi/src/github.com/akiomik/mist_playground/build/dev/erlang/gleam_erlang/_gleam_artefacts/gleam_erlang_ffi.erl"},
          {line,195}]},
     {gleam_erlang_ffi,select,1,
         [{file,
              "/Users/akiomi/src/github.com/akiomik/mist_playground/build/dev/erlang/gleam_erlang/_gleam_artefacts/gleam_erlang_ffi.erl"},
          {line,180}]},
     {gleam@otp@actor,loop,1,
         [{file,
              "/Users/akiomi/src/github.com/akiomik/mist_playground/build/dev/erlang/gleam_otp/_gleam_artefacts/gleam@[email protected]"},
          {line,120}]}]}

Any ideas?

Hello! Just wanted to open an issue as Pedro reported on discord the they were unable to get their Mist websocket program to run on IPv6. I have not investigated this myself.

I have a small server that allows anyone to POST an image so that this is saved to the disk of the server.

I can use mist to easily get the bitstring representation of the image in the request's body, and in theory I could parse this to extract the various metadata attributes about the file (e.g. filename, content-type, etc), as well as the actual bytes of the image.

But I was wondering if there is already a built-in way to do this, or you can think of an easier way than parsing the request body manually 🤗 thanks!

Hello!

This is the current API for websocket handlers.

fn websocket_handler(
  message: websocket.Message,
  subject: Subject(HandlerMessage),
) -> Result(Nil, Nil) {
  Ok(Nil)
}

It's good but I think there's some limitations with it currently:

It cannot hold state as unlike the actor and gen_server abstractions there's no state parameters. Statefulness is a key feature of websocket sessions, for example, without it we could not implement LiveView in Gleam.

There is no way for the handler process to handle other messages beyond the ones from the websocket. Without this we could not implement things like pub-sub where we would want to send a message to many websocket processes so that they can each relay the message to their client.

The subject with with to reply is sent with each message, modelling a request-response relationship between messages. This means the process cannot send a message before receiving one. It's also slightly unusual as it implies that a different subject is to be used after each received message, but I think all the messages will be going to the same place. (Or could they directly write to the socket rather than using a subject?).

I think the solution here may be to use more-or-less the actor API for handlers. A handler is a normal actor which an init function (in which the mechanism for sending messages is injected, either a subject or some kind of socket writer type, and multiple messages can be subscribed to via a receiver), and a message handler function is invoked per message, giving the actor the chance to perform behaviour and also update state.

Thanks,
Louis

Hello! I'm getting started with Gleam, and I'm trying to build a web app with it. So far, I just have the code from the web server tutorial (plus logic to determine the port based on environment variables), and it's working perfectly well locally. When I try to deploy it to Railway, however, it gives this error whenever I try to access it:

=ERROR REPORT==== 20-Feb-2023::17:59:47.884721 ===
Error in process <0.155.0> with exit value:
{{case_clause,{tcp,fun 'glisten@tcp':accept/1,
fun 'glisten@tcp':accept_timeout/2,
fun 'glisten@tcp':close/1,
fun 'glisten@tcp':controlling_process/2,
fun 'glisten@tcp':handshake/1,fun 'glisten@tcp':listen/2,
#Fun<glisten@[email protected]>,
fun 'glisten@tcp':receive/2,
fun 'glisten@tcp':receive_timeout/3,
fun 'glisten@tcp':send/2,fun 'glisten@tcp':set_opts/2,
fun 'glisten@tcp':shutdown/1,fun socket:info/1}},
[{mist@http,parse_request,3,
[{file,"build/prod/erlang/mist/build/[email protected]"},{line,299}]},
{mist@handler,'-with_func/1-fun-6-',3,
[{file,"build/prod/erlang/mist/build/[email protected]"},
{line,42}]},
{glisten@handler,'-start/1-fun-3-',3,
[{file,"build/prod/erlang/glisten/build/[email protected]"},
{line,169}]},
{gleam@otp@actor,loop,1,
[{file,"build/prod/erlang/gleam_otp/build/gleam@[email protected]"},
{line,113}]}]}

I'm quite new to both Erlang and Gleam, but I don't see a message, and the stack trace isn't of much help as it refers to the intermediate Erlang files.

Mist currently supports binary and string messages. It'd make sense to support StringBuilder and BitBuilder outgoing messages.

Hi Rawhat,

I hit an issue when I set the HTTP method to PATCH, Mist crashes, with this error:

Error in process <0.164.0> with exit value:
{badarg,[{erlang,atom_to_binary,
                 [<<"PATCH">>,utf8],
                 [{error_info,#{module => erl_erts_errors}}]},
         {gleam_erlang_ffi,atom_to_string,1,
                           [{file,"build/dev/erlang/gleam_erlang/build/gleam_erlang_ffi.erl"},
                            {line,40}]},
         {mist@http,parse_request,2,
                    [{file,"build/dev/erlang/mist/build/[email protected]"},
                     {line,153}]},
         {mist@http,'-handler_func/1-fun-4-',3,
                    [{file,"build/dev/erlang/mist/build/[email protected]"},
                     {line,291}]},
         {glisten@tcp,'-start_handler/3-fun-0-',4,
                      [{file,"build/dev/erlang/glisten/build/[email protected]"},
                       {line,231}]},
         {gleam@otp@actor,loop,1,
                          [{file,"build/dev/erlang/gleam_otp/build/gleam@[email protected]"},
                           {line,92}]}]}

It looks like it's happening in http.gleam on line 124 when it calls |> atom.to_string. To me it seems like it's expecting an Atom but is getting passed a bitstring, although I am very new to Erlang so the errors do not make massive sense to me yet. I tried these other HTTP verbs Get,Post, Put, Delete and those all work fine, I haven't tried header, option etc.. yet.

Let me know if I can help to identify the issue and fix.

I'm getting the following errors with a long running websocket connection.

=ERROR REPORT==== 7-Jun-2023::09:13:17.031402 ===
Error in process <0.123.0> with exit value:
{{case_clause,9},
 [{mist@internal@websocket,frame_from_message,3,
                           [{file,"/home/arnar/Code/gliew/build/dev/erlang/mist/_gleam_artefacts/mist@[email protected]"},
                            {line,154}]},
  {mist@internal@handler,handle_websocket_message,3,
                         [{file,"/home/arnar/Code/gliew/build/dev/erlang/mist/_gleam_artefacts/mist@[email protected]"},
                          {line,28}]},
  {glisten@handler,'-start/1-fun-3-',3,
                   [{file,"/home/arnar/Code/gliew/build/dev/erlang/glisten/_gleam_artefacts/[email protected]"},
                    {line,169}]},
  {gleam@otp@actor,loop,1,
                   [{file,"/home/arnar/Code/gliew/build/dev/erlang/gleam_otp/_gleam_artefacts/gleam@[email protected]"},
                    {line,137}]}]}

Which I have traced to https://github.com/rawhat/mist/blob/v0.11.0/src/mist/internal/websocket.gleam#L63 as it's not matching opcode 9 afaict.

Here it's expecting it as a possible output from frame_from_message but if I'm not mistaken this will never happen.

I was looking to have / load up 'index.html but not sure how to achieve it without going into the internals. Also I am sort of new to gleam so I couldn't get this to work properly just yet.

Here is my attempt at trying to get "/" to load "index.html" in the serve_file from the README

import mist.{Connection, ResponseData}
import mist/internal/file
import gleam/http/request.{type Request}
import gleam/http/response.{type Response}

// Having to move over internals to get access to this function

fn convert_file_errors(err: file.FileError) -> FileError {
  case err {
    file.IsDir -> IsDir
    file.NoAccess -> NoAccess
    file.NoEntry -> NoEntry
    file.UnknownFileError -> UnknownFileError
  }
}

fn serve_file(
  _req: Request(Connection),
  path: List(String),
) -> Response(ResponseData) {
  let file_path = string.join(path, "/")
  // Omitting validation for brevity
  mist.send_file(file_path, offset: 0, limit: None)
  |> result.map(fn(file) {
    case file_path {
      // Loading "/"
      "" -> {
        let content_type = "text/html"
        // Load up "index.html"
        let file_resp =
          "index.html"
          |> bit_array.from_string
          |> file.stat
          |> result.map_error(convert_file_errors)
          |> result.map(fn(stat) {
            mist.File(
              descriptor: stat.descriptor,
              offset: 0,
              length: option.unwrap(None, stat.file_size),
            )
          })

        // Ideally here we return 404 if the file isn't found otherwise 200 with the index.html file.
        response.new(200)
        |> response.prepend_header("content-type", content_type)
        |> response.set_body(
          file_resp
          |> result.or(mist.Bytes(bytes_builder.new())), // Return 404 response
        )
      }
      _ -> {
        let content_type = guess_content_type(file_path)
        response.new(200)
        |> response.prepend_header("content-type", content_type)
        |> response.set_body(file)
      }
    }
  })
  |> result.lazy_unwrap(fn() {
    response.new(404)
    |> response.set_body(mist.Bytes(bytes_builder.new()))
  })
}

Additionally I thought of doing it the gleam way of defining the pattern but couldn't get it working (not sure if it works in gleam?).

fn serve_file(
  _req: Request(Connection),
  [""]: List(String),
) -> Response(ResponseData)   { ...

Thank you for this library!

This is currently in-progress, just keeping it visible.

It's going... okay right now 😅

Header host: 127.0.0.1:8888
request host will be localhost (default value?) instead of 127.0.0.1, port None instead of Some(8888).

This makes it impossible to let mist serve multiple hosts / DNS names, as it is impossible to pattern match on the host, like it is for path_segments.

mist 0.17.0

When trying to use mist websockets across page reloads, guided by the readme example, mist seems to crash due to some issue related to connection.transport.send, but I am not sure why.

Removing the assertion on line mist/internal/websocket.gleam:232:

          let assert Ok(_) =
            connection.transport.send(
              connection.socket,
              frame_to_bit_builder(frame),
            )

changing to:

          let _ =
            connection.transport.send(
              connection.socket,
              frame_to_bit_builder(frame),
            )

fixes the issue, however I am not sure that is the correct fix or not. But by removing this assertion, the code is allowed to proceed with gracefully stopping the actor and prevents it from crashing the entire process, putting the website in an unrecoverable state.

This issue is reproducible in the following project:
https://github.com/bitbldr/mist_ws_crash

Hi Rawhat,

We have an issue with how Mist is parsing query strings. You can see this with the sample below:

import gleam/io
import mist
import gleam/http/response
import gleam/bit_builder
import gleam/erlang

pub fn main() {
  assert Ok(_) =
    mist.run_service(
      8080,
      fn(req) {
        io.debug(req.path)
        io.debug(req.query)

        response.new(200)
        |> response.set_body(bit_builder.from_bit_string(<<
          "hello, world!":utf8,
        >>))
      },
    )
  erlang.sleep_forever()
}

Calling the server with http://localhost:8080/?test=text

It will print in the command line:

"/?test=text" // <- path
None // <- query

The Request should have the following:
path = "/"
query = Some("test=text")

When a handler crashes, this can be somewhat ideal at times. But ideally not regularly.

For example errors report as:

<<"Caught error in websocket handler">>

But maybe we can put the reason in the error string. This would help a lot in figuring out what is causing the error.

fn handle_ws_message(state: Socket, conn, message) {
  case message {
    mist.Binary(json) -> {
      case decode_type(message) {
         Ok(RatingType("rating")) -> {
              let assert Ok(_) = mist.send_binary_frame(conn, <<image:utf8>>)
          }
      }
    }
  ...
}

import gleam/erlang

logger.error(string.concat(["Caught error in websocket handler: ", erlang.format(reason)]))

<<"Caught error in websocket handler: {errored,{case_clause,{ok,{rating_type,<<\"rating\">>}}}}">>

Would make it a lot easier to find these errors, even if it requires translating the erlang response. I would also love the ability to turn on stack traces but I did some poking around and couldn't find support in gleam for them.

Thank you for reading.

Edit:

Another example. Maybe it could be cleaned up a little more or handled differently.

<<"Caught error in websocket handler: {errored,#{function => <<\"handle_json_message\">>,line => 123,\n           message => <<\"Assertion pattern match failed\">>,\n           module => <<\"image_scorer\">>,\n           value => {error,{bit_decode,<<\"Could not decode base64\">>}},\n           gleam_error => let_assert}}">>

It's not very ergonomic right now, though technically possible.

There were a few approaches discussed in the Discord. Just going to copy them for visibility.

Messages

• did a little thinking on potential APIs for this... i think what i'm going to be able to provide would be pretty minimal, i.e. you're kind of on your own. i think it's unfortunately a bit necessary due to how it works. but my initial thought was something like:

let assert Ok(sse_connection) =
  mist.start_sse(conn, response.new() |> response.add_header(...)) // dunno if headers are valid
// this is where you'd have to do all the heavy lifting to get whatever data you need...
// but i'm not sure there's another way
let resp = mist.send_event(sse_connection, data) // -> Result(Nil, ..)
// if resp is Error there was some issue with the socket, so you shouldn't try to send any more
mist.end_events(sse_connection)

• i slept like shit so my brain is not firing on all cylinders rn... i just kinda skimmed some documentation about SSEs. there seems to be stuff like some "retry window" on the response, and probably some other stuff. but i think this would work?

• i could try to do something like the websocket api like you mentioned, but right now that's annoying because i have to spawn a whole separate process for that... which kinda sucks
  that is likely a better api though?

Hello!

Currently when the web handler crashes the connection is closed. I think we would want to instead return 500 and a body, perhaps Internal server error.

We would also want to ensure that the error is logged using the Erlang logger.

Hey there!
I've set up a little todo rest api example. I have an IdGen type for generating my ID's, and I'd like to be able to use the updated verison for each new request. It's currently structured like

pub fn main() {
  use conn <- sqlight.with_connection("file:db.sqlite3")
  let todo_id_gen: IdGen(Todo) = IdGen(0)
  let state = State(conn, todo_id_gen)
  let init_service = fn(req) {
    let #(resp, state) = router(req, state)
    resp
  }

  let assert Ok(_) =
    mist.run_service(8080, init_service, max_body_limit: 4_000_000)
  process.sleep_forever()
}

so the updated state isn't getting used. How would you recommend passing updated state to request handlers?