Sync async behavior about apollo-upload-examples

[jaydenseric]

I don't fully understand the sync/async part.
There is 2 differents behaviors but where can I change it ?

graphql-upload is meant to be async, and you can't opt in to the sync scenario. The purpose of explaining the two scenarios is to help people understand why the architecture and API design (streams in resolvers, etc.) is the way it is.

In the past, before I could work out how to make the ideal async scenario possible, apollo-upload-server was sync – formidable was used to store all the uploads in temp files, then run the resolvers once they have been fully received. Most other attempts at GraphQL file uploads are a suboptimal sync scenario.

I tried this example and the behavior surprises me. It seems it's neither sync nor async.

It's meant to be, if it's not then it might be some sort of bug.

Thanks for including some details, but TBH I am a little confused trying to understand them.

CC @mike-marcacci

[nikonhub]

Sorry the screenshot is certainly not obvious.

There are 3 screens.

• Left : npm run dev
  Output of all console.log
• Right Top : watch -n 1 ls -alh /tmp
  Prints files in /tmp folder every second with human readable size
• Right Bottom : watch -n 1 ls -alh ~/apollo-upload-examples/uploads
  Prints files in apollo-upload-examples/uploads folder every second with human readable size

I think that the capacitor-cba.....cba0.tmp file is the origin temp file from where createReadStream is created.
It appears at the same time as my mutation and grows until the full size of uploaded file then is deleted.

The modified storeFS function looks like this.

const storeFS = ({ stream, filename }) => {
  const id = shortid.generate()
  const path = ${UPLOAD_DIR}/${id}-${filename}
  return new Promise((resolve, reject) =>
    stream
      .on('error', error => {
        console.log(error)
        if (stream.truncated)
          // Delete the truncated file.
          fs.unlinkSync(path)
        reject(error)
      })
      .on('end', () => {
        console.log('end')
      })
      .on('data', (chunk) => {
        console.log('stream on data event')
        console.log('chunk size is : ', chunk.length)
      })
      .pipe(fs.createWriteStream(path))
      .on('error', error => reject(error))
      .on('finish', () => resolve({ id, path }))
  )

So what I observe is this scenario (All numbers are just for example)

1. capacitor file appears : 1KB
2. stream.on('data') is called
3. then I see my file in uploads folder : 1KB
4. capacitor file grows : 2KB
5. stream.on('data') is called
6. then I see my file in uploads folder : 2KB
7. then capacitor file grows : 3KB => 50MB => 1GB => 5GB => 7GB but stream.on('data') is not called. It waits until the full file is uploaded (in my case it was an iso file of 7GB). And it's not just a matter of console.log which are delayed because of javascript single thread.
8. the rest of stream.on('data') are called and the write of my file in uploads folder ends

Then I tried the same code with API running on my local machine and I remarked that my machine started to freeze. So I think maybe it's just a lack of memory allocated for a node process and it can't upload and write at the same time. I continue my research.

Edit :

I found where the problem seems to come from.
After reading an article on backpressure and _read() method override which handles pipe() backpressure automatically.

I tried to remove the overriden _read() method in fs-capacitor => ReasStream. And the pipe write behaves differently. Sometimes it goes beyond 1GB. So I think there is something to do with it.

I created another issue here more related to fs-capacitor I think.

[mike-marcacci]

Hi @nikosmonaut, thanks for the great explanation here. This sounds like there may be an issue with fs-capacitor, which is my project. What version of node is this on?

[nikonhub]

Hello.
I just edited my comment. I indeed created another issue (maybe solution) on your repository.

I use v 12.0.0

[mike-marcacci]

This was caused by a bug in fs-capacitor, which has been fixed in v2.0.3. @jaydenseric feel free to close this issue.