Coder Social home page Coder Social logo

graphql-multipart-request-spec's Introduction

GraphQL multipart request specification

An interoperable multipart form field structure for GraphQL requests, used by various file upload client/server implementations.

It’s possible to implement:

  • Nesting files anywhere within operations (usually in variables).
  • Operation batching.
  • File deduplication.
  • File upload streams in resolvers.
  • Aborting file uploads in resolvers.

Sync vs async GraphQL multipart request middleware

Multipart form field structure

An “operations object” is an Apollo GraphQL POST request (or array of requests if batching). An “operations path” is an object-path string to locate a file within an operations object.

So operations can be resolved while the files are still uploading, the fields are ordered:

  1. operations: A JSON encoded operations object with files replaced with null.
  2. map: A JSON encoded map of where files occurred in the operations. For each file, the key is the file multipart form field name and the value is an array of operations paths.
  3. File fields: Each file extracted from the operations object with a unique, arbitrary field name.

Examples

Single file

Operations

{
  query: `
    mutation($file: Upload!) {
      singleUpload(file: $file) {
        id
      }
    }
  `,
  variables: {
    file: File // a.txt
  }
}

cURL request

curl localhost:3001/graphql \
  -F operations='{ "query": "mutation ($file: Upload!) { singleUpload(file: $file) { id } }", "variables": { "file": null } }' \
  -F map='{ "0": ["variables.file"] }' \
  -F [email protected]

Request payload

--------------------------cec8e8123c05ba25
Content-Disposition: form-data; name="operations"

{ "query": "mutation ($file: Upload!) { singleUpload(file: $file) { id } }", "variables": { "file": null } }
--------------------------cec8e8123c05ba25
Content-Disposition: form-data; name="map"

{ "0": ["variables.file"] }
--------------------------cec8e8123c05ba25
Content-Disposition: form-data; name="0"; filename="a.txt"
Content-Type: text/plain

Alpha file content.

--------------------------cec8e8123c05ba25--

File list

Operations

{
  query: `
    mutation($files: [Upload!]!) {
      multipleUpload(files: $files) {
        id
      }
    }
  `,
  variables: {
    files: [
      File, // b.txt
      File // c.txt
    ]
  }
}

cURL request

curl localhost:3001/graphql \
  -F operations='{ "query": "mutation($files: [Upload!]!) { multipleUpload(files: $files) { id } }", "variables": { "files": [null, null] } }' \
  -F map='{ "0": ["variables.files.0"], "1": ["variables.files.1"] }' \
  -F [email protected] \
  -F [email protected]

Request payload

--------------------------ec62457de6331cad
Content-Disposition: form-data; name="operations"

{ "query": "mutation($files: [Upload!]!) { multipleUpload(files: $files) { id } }", "variables": { "files": [null, null] } }
--------------------------ec62457de6331cad
Content-Disposition: form-data; name="map"

{ "0": ["variables.files.0"], "1": ["variables.files.1"] }
--------------------------ec62457de6331cad
Content-Disposition: form-data; name="0"; filename="b.txt"
Content-Type: text/plain

Bravo file content.

--------------------------ec62457de6331cad
Content-Disposition: form-data; name="1"; filename="c.txt"
Content-Type: text/plain

Charlie file content.

--------------------------ec62457de6331cad--

Batching

Operations

[
  {
    query: `
      mutation($file: Upload!) {
        singleUpload(file: $file) {
          id
        }
      }
    `,
    variables: {
      file: File, // a.txt
    },
  },
  {
    query: `
      mutation($files: [Upload!]!) {
        multipleUpload(files: $files) {
          id
        }
      }
    `,
    variables: {
      files: [
        File, // b.txt
        File, // c.txt
      ],
    },
  },
];

cURL request

curl localhost:3001/graphql \
  -F operations='[{ "query": "mutation ($file: Upload!) { singleUpload(file: $file) { id } }", "variables": { "file": null } }, { "query": "mutation($files: [Upload!]!) { multipleUpload(files: $files) { id } }", "variables": { "files": [null, null] } }]' \
  -F map='{ "0": ["0.variables.file"], "1": ["1.variables.files.0"], "2": ["1.variables.files.1"] }' \
  -F [email protected] \
  -F [email protected] \
  -F [email protected]

Request payload

--------------------------627436eaefdbc285
Content-Disposition: form-data; name="operations"

[{ "query": "mutation ($file: Upload!) { singleUpload(file: $file) { id } }", "variables": { "file": null } }, { "query": "mutation($files: [Upload!]!) { multipleUpload(files: $files) { id } }", "variables": { "files": [null, null] } }]
--------------------------627436eaefdbc285
Content-Disposition: form-data; name="map"

{ "0": ["0.variables.file"], "1": ["1.variables.files.0"], "2": ["1.variables.files.1"] }
--------------------------627436eaefdbc285
Content-Disposition: form-data; name="0"; filename="a.txt"
Content-Type: text/plain

Alpha file content.

--------------------------627436eaefdbc285
Content-Disposition: form-data; name="1"; filename="b.txt"
Content-Type: text/plain

Bravo file content.

--------------------------627436eaefdbc285
Content-Disposition: form-data; name="2"; filename="c.txt"
Content-Type: text/plain

Charlie file content.

--------------------------627436eaefdbc285--

Security

GraphQL server authentication and security mechanisms are beyond the scope of this specification, which only covers a multipart form field structure for GraphQL requests.

Note that a GraphQL multipart request has the Content-Type multipart/form-data; if a browser making such a request determines it meets the criteria for a “simple request” as defined in the Fetch specification for the Cross-Origin Resource Sharing (CORS) protocol, it won’t cause a CORS preflight request. GraphQL server authentication and security mechanisms must consider this to prevent Cross-Site Request Forgery (CSRF) attacks.

Implementations

Pull requests adding either experimental or mature implementations to these lists are welcome! Strikethrough means the project was renamed, deprecated, or no longer supports this spec out of the box (but might via an optional integration).

Client

Server

graphql-multipart-request-spec's People

Contributors

alberthaff avatar andrew-demb avatar ardatan avatar doctorjohn avatar imolorhe avatar jaydenseric avatar jeroenvisser101 avatar jpascal avatar klis87 avatar koresar avatar leszekhanusz avatar lmcgartland avatar mcg-web avatar peldax avatar powerkiki avatar quentincaffeino avatar samirelanduk avatar sunli829 avatar tobias-tengler avatar truongsinh avatar vojtapol avatar yasudatoshiyuki avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

graphql-multipart-request-spec's Issues

Batched mutations

Hi @jaydenseric,

I'm looking to implement this spec into Absinthe's upload handling. I'm just confused by the following:

{
  "1": [
    "0.variables.image",
    "variables.images.0"
  ],
  "2": [
    "variables.images.1"
  ],
  "3": [
    "variables.images.2"
  ]
}

Either there's something else going on, or some of these mappings don't have the operation index prefixed in their key. I think it should be like this instead:

{
  "1": [
    "0.variables.image",
    "1.variables.images.0"
  ],
  "2": [
    "1.variables.images.1"
  ],
  "3": [
    "1.variables.images.2"
  ]
}

Can you let me know what is the correct way? Thanks for all your work on the client, server and this spec 👍

Issues with multiple file list example?

{
  query: `
    mutation($files: [Upload!]!) {
      multipleUpload(files: $files) {
        id
      }
    }
  `,
  variables: {
    files: [
      File, // b.txt
      File // c.txt
    ]
  }
}

For above mutation, you this example-
curl localhost:3001/graphql
-F operations='{ "query": "mutation($files: [Upload!]!) { multipleUpload(files: $files) { id } }", "variables": { "files": [null, null] } }'
-F map='{ "0": ["variables.files.0"], "1": ["variables.files.1"] }'
-F [email protected]
-F [email protected]

But Here User can upload 1 or 2 or 3 or 5 or any numbers of file. How can I understand user only upload 2 files? If I write above code User only can upload 2 files, If user want to upload 3 or 5 files or any number of files then?

Switch to the JSON Pointer standard for `map` field operations paths

The IETF RFC 6901 “JSON Pointer” spec defines a string syntax for identifying a specific value within a JSON document.

Ideally, the GraphQL multipart request spec would use JSON pointer strings instead of less standard object-path strings for operations paths in the map field array.

Here is an example of the difference for the map field JSON for the batching example:

  {
    "0": [
-     "0.variables.file"
+     "/0/variables/file"
    ],
    "1": [
-     "1.variables.files.0"
+     "/1/variables/files/0"
    ],
    "2": [
-     "1.variables.files.1"
+     "/1/variables/files/1"
    ]
  }

Benefits of switching to the JSON Pointer format:

  • It has a detailed specification.
  • It’s a proposed web standard.
  • It should be more familiar to programers across languages.
  • There should be more tools and libraries available across languages to work with it, making implementations easier.

Cons:

  • It’s 1 character longer per path, leading to slightly larger request content lengths.
  • The / separators are “uglier” to people who would intuitively use . as that's what's used to reference properties within objects in languages such as JS.
  • A spec change like this would be a massive breaking change and would require simultaneous client and server implementation updates.

The benefits don't enable new features or capability and are probably not worth the pain. Until now there have been few, if any, complaints about the format for operations paths. It might make sense to leave this spec as it is until it moves to an official GraphQL Foundation GraphQL over HTTP spec (see graphql/graphql-over-http#7).

Payload modification

Hi, I am making a graphql mutation
{
query: mutation($file: Upload!) { singleUpload(file: $file) { id } },
variables: {
file: File // a.txt
}
}

and the payload is:
...
{ "0": ["variables.file"] }
--------------------------cec8e8123c05ba25
Content-Disposition: form-data; name="0"; filename="a.txt"
Content-Type: text/plain

Is there a way for the payload name="0" property to equal the one stated in the mutation for example
{
query: mutation($file: Upload!) { singleUpload(file: $file) { id } },
variables: {
someCustomName: File // a.txt
}
}

and the payload to be :
...
{ "0": ["variables.someCustomName"] }
--------------------------cec8e8123c05ba25
Content-Disposition: form-data; name="someCustomName"; filename="a.txt"
Content-Type: text/plain

Thank you!

GraphQlRequest.ToInputs throws an InvalidCastException for nested types.

Hallo,

i'm testing the lib with my GraphQL schema and using a Type where the file property is on the second level in a GraphQL Input object.
Because the Inputs type contains internally the Dictionary and not anothe Inputs, the code throws an InvalidCastException.
Proposal: change the GetInputs function and cast to the Dictionary<string,object>

    public Inputs GetInputs()
    {
        var variables = Variables?.ToInputs();


        // the following implementation seems brittle because of a lot of casting
        // and it depends on the types that ToInputs() creates.

        foreach (var info in TokensToReplace)
        {
            int i = 0;
            object o = variables;

            foreach (var p in info.Parts)
            {
                var isLast = i++ == info.Parts.Count - 1;

                if (p is string s)
                {
                    if (!(o is IDictionary<string, object> dict))
                        break;
                    if (isLast)
                    {
                        dict[s] = info.File;
                    }
                    else
                    {
                        o = dict[s];
                    }
                }
                else if (p is int index)
                {
                    if (!(o is List<object> list))
                        break;

                    if (isLast)
                    {
                        list[index] = info.File;
                    }
                    else
                    {
                        o = list[index];
                    }
                }
            }
        }

        return variables;
    }

Spec Improvement for the broader GraphQL Ecosystem

I'd like to reopen the discussion on @enjoylife's suggested simplification of the spec #50 . I believe their suggestion can greatly simplify this specification and ease the implementation for typesafe frameworks while still achieving the original goals for JS GraphQL libraries.

Just for clarity, this is a rough draft of how I imagine the new version of the spec could look building off of what @enjoylife has defined:

  1. Requests are multipart/form-data
  2. There is exactly 1 non-file form field. This contains the GraphQL request. The GraphQL server has a Scalar backed by a String that references the name of the related file form field.
    1. It doesn't need to be named operations anymore but it might be better to reserve this name for further extension
    2. It doesn't need to be before the file form fields but it might be best to enforce this for performance
    3. We might not need to specify the name of our scalar but Upload seems good.
  3. Every other form field should be a file with a unique name which will be referenced by the Upload scalar.

ex:

curl http://localhost:4000/graphql \
  -F gql='{ "query": "mutation { upload(files: [\"file_id\"]) }", "variables": null }' \
  -F [email protected]

I've implemented a prototype of this in my own Scala server and I have a very rough implementation of this for Apollo:

Prototype Apollo Attachments Gist

Regarding the comments in #11 (describing how the map field is necessary for performance), this is an implementation detail of the server and something we can solve for Apollo. If our Apollo plugin finds an Upload(file_id) we grab a promise from our shared Uploads object which we will resolve once we parse our file or reject after we finish parsing the entire request. This lets us execute our GraphQL request as soon as we find it in our form fields.

This is a trace from running my gist:

curl http://localhost:4000/graphql \
  -F gql='{ "query": "mutation { upload(files: [\"file1\", \"file2\", \"file1\"]) }", "variables": null }' \
  -F file1=@Untitled \
  -F file2=@API\ Wizard.mp4

2021-04-attachments_timing

You can see that we've achieved the same async waterfall where our GraphQL request execution starts immediately.


The first thing that comes to mind (although it's a pretty exotic) is that the current spec allows files to be used anywhere in the GraphQL operations objects, not just in variables

Yes, each file is always referenced by its uid so your server can choose to arrange its json however it desires without any issues.

An added benefit of this proposal over the current spec is the ability to define file references outside of variables. Right now you're required to always have a "variables" section to reference via your map form-field. It's not possible to send something like:

curl http://localhost:4000/graphql \
  -F gql='{ "query": "mutation { upload(files: [\"file_id\", \"file_id\"]) }", "variables": null }' \
  -F [email protected]

With your proposal that doesn't include a map of where files are used in the operations, it's not clear to me how one file variable used as an argument in multiple mutations in the same request could be implemented on the server. Is that something you have considered?

This doesn't really change between the current spec and this proposal. You're always looking up in your context for the file based on its uid. There's no reason you can't repeatedly query the same file based on its uid.

ex:

curl http://localhost:4000/graphql \
  -F gql='{ "query": "mutation { a: upload(files: [\"file_id\"]) b: upload(files: [\"file_id\"]) }", "variables": null }' \
  -F [email protected]

Performance wise the map allows the server to cheaply see up front how many files there are, and where they are expected to be used in the operation without having to parse the GraphQL query looking for certain upload scalars in an AST, etc. For example, the map makes implementing the maxFiles setting in graphql-upload trivial.

Although this is true of the new spec change, we'll always be parsing GraphQL requests in GraphQL servers, it's a matter of leveraging the server libraries to facilitate this. This is something that could maybe be handled by an Apollo validationRule or definitely by an Apollo plugin. We're writing a spec for GraphQL, we should be using the tools our GraphQL servers provide to us.

Even if we're writing an implementation of this spec for a framework that gives 0 options to validate our GraphQL request, the current JS spec implementation has already defined code that would catch maxFiles as they were streaming through via Busboy: https://github.com/jaydenseric/graphql-upload/blob/2ee7685bd990260ee0981378496a8a5b90347fff/public/processRequest.js#L67


The point of this spec is to create a standard for interoperability between all GraphQL clients and servers, regardless of the languages or ecosystems

Exactly, this spec appears to be designed in order to run as a JS Server middleware. There is a good amount of indirection, implementation specific solutions, and dependencies on the implementing language/framework. This all creates more work for server implementers.

I did an audit of the various server implementations and all of the ones I looked at either depend on:

  • Their language being dynamic
  • Their language having a top Object type and casting (ignoring type safety)
  • Or they basically implement the proposed spec change internally

There doesn't seem to be a good way to add this specification to a typesafe language/framework without it devolving into the proposed spec change.

For Example async-graphql in Rust

https://github.com/async-graphql/async-graphql/blob/f62843cbd34ef9bf28f70f8df07d4f61f8038e0a/src/request.rs#L115

*variable = Value::String(format!("#__graphql_file__:{}", self.uploads.len() - 1));

we can see that internally, after the map is parsed, we replace the null inside the variables definition with a uid to reference the specific file.

https://github.com/async-graphql/async-graphql/blob/f62843cbd34ef9bf28f70f8df07d4f61f8038e0a/src/types/upload.rs#L99

/// Get the upload value.
pub fn value(&self, ctx: &Context<'_>) -> std::io::Result<UploadValue> {
   ctx.query_env.uploads[self.0].try_clone()
}

When we get the value out of the Scalar, we pull the actual file stream out of the context via that same uid.

For Example caliban in Scala

https://github.com/ghostdogpr/caliban/blob/660beeae538768d817a73cb4535e0e3bd1a8cb82/adapters/play/src/main/scala/caliban/uploads/Upload.scala#L89

// If we are out of values then we are at the end of the path, so we need to replace this current node
// with a string node containing the file name
StringValue(name)

We're setting our variable value to the filename in order to pull it out of the context later.

For Example sangria in Scala

(This isn't actually in the library but this gist describes how to implement it).

https://gist.github.com/dashared/474dc77beb67e00ed9da82ec653a6b05#file-graphqlaction-scala-L54

(GraphQLRequest(gql = gqlData, upload = Upload(mfd.file(mappedFile._1)), request = request))

we store our uploaded file separately from our GraphQL request.

https://gist.github.com/dashared/474dc77beb67e00ed9da82ec653a6b05#file-controller-scala-L68

userContext = SangriaContext(upload, maybeUser),

we pass our file through via the context.

https://gist.github.com/dashared/474dc77beb67e00ed9da82ec653a6b05#file-exampleschema-scala-L15

val maybeEggFile = sangriaContext.ctx.maybeUpload.file.map(file => Files.readAllBytes(file.ref))

Inside of our resolver we lookup the file in the context to actually use it.

All of these examples have implemented @enjoylife's proposal under the covers in order to preserve some form of type safety.

Note:

We can use these libraries as a guide to show us how to implement supporting both the current version of the spec and the proposed change in the same server with plenty of code sharing.

For Reference graphql-upload in JavaScript

This JavaScript implementation depends on the language being dynamic so that we can overwrite our variables with an Upload instance.

https://github.com/jaydenseric/graphql-upload/blob/60f428bafd85b93bc36524d1893aa39501c50da1/public/processRequest.js#L232

operationsPath.set(path, map.get(fieldName));

We assign the Upload instance to the specified location in our GraphQL Json Request

https://github.com/jaydenseric/graphql-upload/blob/60f428bafd85b93bc36524d1893aa39501c50da1/public/GraphQLUpload.js#L81

if (value instanceof Upload) return value.promise;

When parsing our Scalar value, we check and cast to make sure we found an Upload instance.

For Reference graphql-java-servlet in Java

This Java implementation depends on the top level Object type so that we can check and cast our variables on the fly.

https://github.com/graphql-java-kickstart/graphql-java-servlet/blob/eb4dfdb5c0198adc1b4d4466c3b4ea4a77def5d1/graphql-java-servlet/src/main/java/graphql/kickstart/servlet/GraphQLMultipartInvocationInputParser.java#L138

objectPaths.forEach(objectPath -> VariableMapper.mapVariable(objectPath, variables, part));

We set each http.Part in our variable map

https://github.com/graphql-java-kickstart/graphql-java-servlet/blob/eb4dfdb5c0198adc1b4d4466c3b4ea4a77def5d1/graphql-java-servlet/src/main/java/graphql/kickstart/servlet/apollo/ApolloScalars.java#L28

if (input instanceof Part) {
  return (Part) input;

When parsing our Scalar, we check and cast to make sure we found a http.Part instance.


This spec and the graphql-upload JS server-side implementation are not tied in any way to Apollo, or a "heavy js graphql abstraction"

I can't speak for @enjoylife but I don't believe the proposed changes to this spec are implying the code for graphql-upload is heavy. graphql-upload is quite elegant in its implementation. In fact, for my Apollo prototype I borrowed heavily from graphql-upload. The heavy parts are that:

  • We have multiple implementation specific details baked into the specification
  • Using null as a placeholder is really another server implementation detail, it doesn't make sense from the client perspective
  • There is a lot of indirection in the variables in order to support implementing GraphQL Upload libraries inside of JS middleware

In Summary

"The point of this spec is to create a standard for interoperability between all GraphQL clients and servers, regardless of the languages or ecosystems" and the current iteration of this specification constrains non-dynamic languages in order to be written inside of a JS Server Middleware. Evolving this specification will better fit the growing GraphQL ecosystem and make this specification future proof so that everybody can benefit from the work you've done here.

README graphic is *somewhat* misleading regarding buffering

Hello there!
I was reading the various implementations to see how people where dealing the streaming of files without writing to temp disk first.
In the rust implementation, we always buffer on disk or memory first and it looks like the other implementations also do that since multipart is sequential in nature so by default if you need the second file you need to buffer the first file somewhere.

So I feel like it the graphic presented is somewhat misleading, the stream as in the data from the client is never really passed to the resolver. Some implementations might have an optimization for a single file upload, but that seems like the exception to the general rule. Unless I am mistaken even the JS implementation uses fs-capacitor to buffer on disk. I think it should be made clearer that this is a limitation of multipart, you just can't get concurrent streams of files.

That being said, I am wondering if there should be a spec for single file upload where we could get the stream of data without buffer into the resolver?

You have to solve my problem. Please help me.

Where I am wrong. Why getting again again 400 bad request. Please help. Here is the error image-

Screenshot 2021-10-18 132322

export const signUp = (data) => async (dispatch) => {
    console.log(data.picture[0].name);
    dispatch({type: SIGNUP_LOADING})
    var formData = new FormData();
    formData.append('operations', '{ "query": "mutation($file: Upload!) {signUp(avatar: $file, input: {name: "Siam Ahnaf", email: "[email protected]", password: "siam1980"}){message, token}}", "variables": { "file": null } }');
    formData.append('map', '{ "0": ["variables.file"] }');
    formData.append('0', Image);
    await axios({
        url: "http://localhost:3001/graphql",
        method: "post",
        data: formData,
        Headers: {
            Accept: "application/json"
        }
    })
        .then(response => console.log(response))
        .catch(error => console.log(error));
}

I can't anything. Please, jaydenseric, I need your help. Please help me. Please.
The mutation is worked in playground. Here is image-
Screenshot 2021-10-18 131758

You have to help me. Please. Where I am wrong. 😌 Please please please.

How does it compare to express-graphql way ?

In a discussion about whether to implement this spec or something else, it came up that express-graphql, apollo-server-express and relay seems to all do something similar, which seems to be different than graphql-multipart-request-spec is doing.

Could you explain why you did differently ? Whether you were aware of those pseudo-standard ? And in what way graphql-multipart-request-spec is better ? or the limitations of the "express way" ?

I feel the answers to those question could be part of the spec itself, as an introduction or meta document. It would certainly help to promote this spec.

Is it possible to not use variables?

All examples in the spec use variables. Is it possible to upload file(s) without using variables (pass file(s) directly as mutation argument value)?

Curl example

The readme could be improved by adding an example of usage with CURL. I myself am having difficulty figuring out exactly how it would look.

Maybe you should remove the "map" key?

I think that the "map" in the request is unnecessary. In my opinion it's much easier to do this:

cURL request

curl localhost:3001/graphql \
  -F operations='{ "query": "mutation ($file: Upload!) { singleUpload(file: $file) { id } }", "variables": { "file": "0" } }' \
  -F [email protected]

Request payload

--------------------------cec8e8123c05ba25
Content-Disposition: form-data; name="operations"

{ "query": "mutation ($file: Upload!) { singleUpload(file: $file) { id } }", "variables": { "file": "0" } }
--------------------------cec8e8123c05ba25
Content-Disposition: form-data; name="0"; filename="a.txt"
Content-Type: text/plain

Alpha file content.

--------------------------cec8e8123c05ba25--

Thus, it is easier to organize server-side support:

Example for webonyx/graphql-php

class UploadType extends ScalarType
{
    public $name = 'Upload';

    public function parseValue($value)
    {
        return $_FILES[$value] ?? null;
    }
}

Example for nodejs

import { GraphQLScalarType } from 'graphql'
import requestContext from from 'global-request-context'

export const GraphQLUpload = new GraphQLScalarType({
  name: 'Upload',
  parseValue: value => {
    const { req } = requestContext
    return req.files[value] || null
  }
})

What do you think about this?

Some missing points

Hi,
We are implementing a middleway to this php lib for some backward compatibility reasons we can't use php middleway lib... but It seems specifications don't explains:

  • how server deal with optional files? Can you provide an example of request payload and server behavior in this case?
  • how server answer in case of batch request?

Can you please help me to get all this?

Variable * got invalid value {}; String cannot represent a non string value: {}

I got error: Variable \"$uid\" got invalid value {}; String cannot represent a non string value: {}

My form-data of Body:

operations:{"query":"mutation($uid: String!, $photos: [Upload!]!) {\n  uploadPhotos(uid: $uid, photos: $photos)\n}", "variables": { "uid": null, "photos": [null, null, null]}}↵
map:{"0":["variables.uid"], "1":["variables.photos.0"], "2":["variables.photos.1"], "3":["variables.photos.2"]}
0:test

My Graphql mutation:

mutation UploadPhotos($uid: String!, $photos: [Upload!]!) {
  uploadPhotos(uid: $uid, photos: $photos)
}

File downloads via GraphQL

Everything is clear about file uploads via GraphQL using mutations. But what about file downloads via GraphQL using queries?

Operations and map content type ?

Is there a reason why the operations and map parts of the spec do not have a content-type application/json set?

This could help frameworks the content automatically.

Ordered fields?

Why is there the requirement to order fields in the request? Trying to build something that would handle this in react native with mostly native code handling the requests to ensure execution and completion in the background, but when passing a dictionary of parameters over the bridge to the native code, it's getting ordered and added to the body alphabetically.

Just wanted to know if there were technical limitations/requirements to the field ordering before spending time trying to re-work my native implementation. Thanks!

Conside using JsonPath for defining fiile field path.

From my understanding field path for the case when we have an array of files:

"SomeStructure":  {
"files" : ["file0", "file1"] 
}

will look like this:
variables.SomeStructure.files.0 and variables.SomeStructure.files.1

Why not consider using JsonPath for field path definition? Cause at the moment we can have a situation when fields name that represents file can be literal or a number in resulting JSON with the same behavior as an array.
Switching to JsonPath would slightly change path definition, but it would be parsable out of the box:
variables.SomeStructure.files.[0] and variables.SomeStructure.files.[1]

Thank you @jaydenseric.
Look forward to your response.

Request payload has no file content

Hi!

I have an issue when posting a file using apollo-client (I am posting to apollo-server-lambda). The POST request looks like this, in other words, it's empty where the file content should be (I think?)

{"1":["variables.file"]}
------WebKitFormBoundaryms9IIBb8QTb29vwl
Content-Disposition: form-data; name="1"; filename="74b233b0.jpg"
Content-Type: image/jpeg


------WebKitFormBoundaryms9IIBb8QTb29vwl--

The file sent to the server looks like this:

File {
  lastModified: 1598943888000,
  lastModifiedDate: Tue Sep 01 2020 09:04:48 GMT+0200 (Central European Summer Time){},
  name: "IMG_2439.JPG",
  size: 3955743,
  type: "image/jpeg",
  webkitRelativePath: ",
  __proto__: File
}

I'm using apollo-client-server like this:

  ....
  return new ApolloClient({
    ssrMode: typeof window === "undefined",
    link: createUploadLink({
      uri: "my-uri",
      headers: {
        authorization: `Bearer ${token}`,
      },
    }),
    cache: new InMemoryCache(),
  });

And here's my typedefs

gql`
  mutation createPerson($person: PersonInput!, $file: Upload!) : String
`

And the mutation query looks like this:

  createPerson({
      variables: { person: person, file: image },
      update: updateCache,
    });

I don't think I can google my problem more than I have. I would really appreciate som help <3

Always batch operations to make JSON decoding easier in strongly-typed languages

Hi, I try to implement this spec with Go lang. But I face issue is: operations can be object or array (in the batch query). Method json.Unmarshal in Go lang required to knows the root object is array or map.

You may consider changing operations to an array, with a single item for the non-batch query?

PS: I have a way to workaround is check the first byte of JSON is [ or not. It not, I can add [ to the begin, and ] to the end, then decode bytes to array.

‘operations’ multipart field

  • Resolver :
import { Resolver, Args, Mutation } from '@nestjs/graphql';
import { GraphQLUpload } from 'apollo-server-express';
import { createWriteStream } from 'fs';
import { FileUpload } from 'graphql-upload';


@Mutation(() => Boolean)
  async uploadFile(
    @Args({ name: 'file', type: () => GraphQLUpload })
    { createReadStream, filename }: FileUpload,
  ): Promise<boolean> {
    return new Promise(async (resolve, reject) =>
      createReadStream()
        .pipe(createWriteStream(`./uploads/${filename}`))
        .on('finish', () => resolve(true))
        .on('error', () => reject(false)),
    );
  }
  • Postman :
  - operations: {“query”:“mutation UploadFile($file:Upload!) {\n uploadFile(file:$file)\n}”, “variables”: { “file”: null }}
  - map : [{"key":"map","value":"{ “0”: [“variables.file”] ","description":"","type":"text","enabled":false}]
  - 0 : file.jpg
  • Result
    get this error message:
   [Nest] 29328   - 2020-10-26 12:45:33   [ExceptionsHandler] Object:                           
[                                                                                            
  {                                                                                          
    "message": "Invalid JSON in the ‘operations’ multipart field (https://github.com/jaydense
/graphql-multipart-request-spec).",                                                          
    "extensions": {                                                                          
      "code": "INTERNAL_SERVER_ERROR"                                                        
    }                                                                                        
  }                                                                                          
]       

any idea ??

`map` field in context of backward compatibility

Hi, I am upgrading old GraphQL dependencies on my project. For file uploads [email protected] is used (I understand that now there is version 11 :)). We have a lot of users using mobile client, so we have no ability to update all our clients.
How should I deal with map property which is required now, but is not sent from most of the clients?

I am using:
Fastify - as a web framework
Mercurius - as fastify gql plugin
MercuriusUpload - as gql upload plugin

Content-Length or arbitrary headers

What do you think about adding a new sizes map which corresponds to the map implementation which includes information on the expected size of the uploaded payload. Or, overhaul map to contain arbitrary headers for each file. Off the top of my head I'm thinking of Last-Modified (advanced caching possibilities) Range (resumable uploads), Content-Encoding (compression for large text payloads) would be valuable.

Content-Length in particular in my case is required because I want to stream the request directly from the client right into S3 without needing to buffer the entire thing in memory or disk. S3 requires the length of the payload upfront, so right now I also need to send the payload size along, something like input UploadWithSize { file: Upload!, size: Int! }. It's useful in other cases as well, for example terminating the request early if the size will be too large, allocating an ArrayBuffer to avoid costly memcpys, or filesystem extents. Of course, the stream should error out if the content continues past the given Content-Length, or if it closes before.

It doesn't seem to be possible to set additional headers via multipart requests over FormData so implementing it in a blessed side-channel would be required. When uploading a single file via fetch or XHR you get the Content-Length for free, but once you go through FormData you lose the information.

Multipart File Upload

Hi,

I have my graphql server implemented in apollo. I have my client built using react. I would like to upload files in chunks ie. When a user uploads 3 files to upload, I would like to upload file one by one and in parts. So the first file gets uploaded first in part something similar to first n bytes and next n bytes an so on.

I have followed the above-mentioned format for sending the request. But I facing this issue

BadRequestError: Misordered multipart fields; files should follow ‘map’ (https://github.com/jaydenseric/graphql-multipart-request-spec).

I am not able to debug it. Can you please help me with this?

Thank you. Much appreciated!!!!

createReadStream is not a function

Hi I'm trying to upload a file using GraphQL and I'm getting the following error

Error: GraphQL error: createReadStream is not a function
    at new ApolloError (bundle.esm.js:63)
    at Object.next (bundle.esm.js:1004)
    at notifySubscription (Observable.js:135)
    at onNotify (Observable.js:179)
    at SubscriptionObserver.next (Observable.js:235)
    at bundle.esm.js:866
    at Set.forEach (<anonymous>)
    at Object.next (bundle.esm.js:866)
    at notifySubscription (Observable.js:135)
    at onNotify (Observable.js:179)

The following is my mutation

mutation UploadProfile($file: Upload!) {
  upload(file: $file) {
    id
  }
}

And the following is how I post my image

client.mutate<UploadProfile, UploadProfileVariables>({
        mutation: UploadProfileMutation,
        variables: {
          file: profileProperties.file,
        },
        context: {
          headers: {
            Authorization: `Bearer ${token}`,
          },
        },
      })

I don't know why it's not working and I'm not finding the solution on the internet. Please help

I cant send one file and other data, like a string.

Always I receive this error: Variable "$name" got invalid value { resolve: [function], reject: [function], promise: {} }; Boolean cannot represent a non boolean value: { resolve: [function], reject: [function], promise: {} }

Where I am wrong when I am using axios to upload file, Please help me.

Hello, I am uploading file from next js application. I am getting error like this images-
Screenshot 2021-10-14 124425

But I can't understand where I am wrong. Here is my code-

import Image from "assets/Hero-bg.jpg"
var formData = new FormData();
    formData.append('operations', '{ "query": "mutation($file: Upload!) {signUp(avatar: $file, input: {name: "Siam Ahnaf", email: "[email protected]", password: "siam1980"}){message, token}}", "variables": { "file": null } }');
    formData.append('map', '{ "0": ["variables.file"] }');
    formData.append('0', Image);
    await axios({
        url: "http://localhost:3001/graphql",
        method: "post",
        data: formData,
        Headers: {
            'Accept': 'application/json'
        }
    })
        .then(response => console.log(response))
        .catch(error => console.log(error));

The query mutation is-

mutation signUp($avatar: Upload!) {
  signUp(
    avatar: $avatar
    input: { 
      name: "Siam Ahnaf"
      email: "[email protected]"
      password: "12345678" }
  ) {
    message
    token
  }
}

Please help me.

Need some help please - how to upload files

Hi , I created a SO question here, can you please help me out with this ? https://stackoverflow.com/questions/57541447/upload-files-using-apollo

Please note that I can upload files using Insomnia, so I guess my graphql server is ready and is working fine. I just need some help uploading some files from the UI.

Any help will be well appreciated.

If you would like me to create a detailed post here, please let me know. Apologies in advance if this post is not OK .

Thanks
Gagan

Simple alternative if you are not tied to the JS graphql ecosystem.

TL;DR If you are not using a server side graphql implementation with limitations on how scalars and other types should be "resolved", (such limitations arise in the apollo's js server see #11 and here) and your not tied to graphql-upload. Then I would not recommend using this spec verbatim. Instead just use uuid's to map between the schema and each body part.

# 1. Define a scalar which will be the type and identifier we use when reading off the multipart data.
scalar MultiPartUUID

# 2. Define an input type using said scalar
input FileUpload {
     uuid: MultiPartUUID
    # other data which your app requires for upload
    # e.g... `fileName` if you want to trust a users provided file extension, ... etc.
} 

The environment I'm assuming is on client side, we collect files from a <input type="file" /> element providing File objects. We ultimately want to use these File objects as variables within our request. Now instead of doing the funky null replacement which this spec defines, which is semantically questionable, just use the scalar to hold a uuid and use it when to constructing requests. High level steps:

  1. Set a uuid for each file
  2. Use the uuids within the variables to satisfy the schema {variables: files:[ {uuid: 'xxx'} ]
  3. Fire off the request appending on the file(s) using the uuid for each multipart name.

Following such an approach creates a curl request along the lines of:

curl localhost:3001/graphql \
  -F operations='{ "query": "mutation ($files: [FileUpload!]!) { filesUpload(files: $files) { id } }", "variables": { "files": [{uuid: ‘xxx’ }, {uuid: ‘yyy' }] }' \
  -F [email protected] \
  -F [email protected]

Then server side you simply read off the first part of the body which holds the query. Pretty much every graphql implementation gives you access when parsing the ast, and you can execute the schema as desired having read that first part. Now there you could either block and consume all the remaining parts, setting them aside in memory or temp files, which the resolvers would use as a lookup per each file variable uuid. Or you can read the parts as needed, streaming in the data, and have the resolvers, waiting in parallel for other resolvers to consume the body up to the needed part.

For additional clarity here’s an outline for given a Js client and Go server setup.

1 Set uuid's on the files to upload

// form component gets the `uploadedFiles`
function onChange(files) {
  return files.map(f => {
    // make sure we have our uuid ready
    if (!f.key) {
      f.key = uuidV4();
    }
    return f;
  })
}
// => uploadedFiles

Map the variables to satisfy the schema {variables: files:[ {uuid: 'xxx'} ]

const variables = {
  files: uploadedFiles.map(({uuid}) => {
    return { uuid, /* fileName, etc. */}
  }),
}

Fire off the request appending on the file(s) using the uuid as the multipart name

var data = new FormData();
data.append('operations', JSON.stringify(graphQLParams));
// look ma,  no map included. 

for (const file of uploadedFiles) {
  data.append(file.uuid, file, file.name);
}
let xhr = new XMLHttpRequest();
xhr.send(data);

Server Side rough example:

var MultiPartKey = "ctx-key"
func handler(w http.ResponseWriter, r *http.Request)  {
	w.Header().Set("Content-Type", "application/json")
	reader,_ := r.MultipartReader()
	// Step 1: Get the operations part
	// e.g. Content-Disposition: form-data; name="operations"
	operationsPart, _ := reader.NextPart()
	

	// Insert the multipart reader into the request context for use in resolvers
	r = r.WithContext(context.WithValue(r.Context(), MultiPartKey, reader))

	//  Execute your schema using operationsPart
	// ...
}

In the resolvers....

func utilToWaitForPart(ctx context.Context, uuid string) *multipart.Part {
	// if run in parallel this could wait until other resolvers have finished reading up the uuid needed

	reader := ctx.Value(MultiPartKey).(*multipart.Reader)
	// would need to smartly wait until part is ready
	part, _ := reader.NextPart()
	for part.FormName() != uuid {
		// *sync.Cond.Wait() this is only a rough sketch,
                // you will have to use more sync primitives for a real implementation
	}
	return part
}
func Resolver(ctx context.Context, input FileUploadInput) (Response, error) {
	for _, file := range input.Files {
		//
		// use the uuid to correctly get the part data
		part := utilToWaitForPart(ctx, file.uuid)
		bytes, _ := ioutil.ReadAll(part)
		// leverage the files...
	}
}

If you made it thus far, I just want to remind you that if you are using apollo or any of the heavy js graphql abstractions, this projects spec is good and correctly works within the limitations of node and the rest of the node based graphql libraries. But if your outside of that world, this simple uuid mapping like I have outlined above is a simple alternative.

Null means the opposite of Null

Hey folks,

I'm very happy to see a standardized spec surrounding file uploads, particularly one that lets you treat them as arguments. We've been doing something like this for a while with Absinthe https://hexdocs.pm/absinthe_plug/Absinthe.Plug.Types.html#content but would be happy to change to a more standardized format.

My trouble with this particular setup though is the use of null as a placeholder. This issue is both conceptual and technical. From a conceptual perspective we're using null to mean that something is there, which is the precise opposite of what null communicates. From a technical perspective, I'm not entirely sure how this would work with arguments that are marked non null. It seems counterintuitive to permit null to be a valid argument to a non null field.

Alternatives: If we aren't trying to use the values to mean much of anything then simply an empty string would suffice, and help avoid the problems I've outlined. In the Absinthe implementation the strings point to the HTTP part that contains the file they want, but the mapping solution you have already handles that sort of issue.

Thoughts?

In your example you only show not nullable file upload system?

If there is a optional file upload system then how can I send data. It is very sad that in your example I can't see any example. If user want to upload file then he upload otherwise not, for this case you do not show any example. Please throw an example for the following oparations-

{
  query: `
    mutation($file: Upload) {
      singleUpload(file: $file) {
        id
      }
    }
  `,
  variables: {
    file: null(No file)
  }
}

Here file upload is optional. Thank you very much.

For above operations If I do this-

curl localhost:3001/graphql \
  -F operations='{ "query": "mutation ($file: Upload) { singleUpload(file: $file) { id } }", "variables": { "file": null } }' \
  -F map='{ "0": ["variables.file"] }' \
  -F 0=@null

It throw a error message like image-
Screenshot 2021-10-26 133513

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.