download cancelled event? about streamsaver.js

[allengordon011]

Is it possible to access the abort() to do more than just console.log('user aborted')?

[guest271314]

For Chrome, Canary version 99.0.4828.0 I was able to handle user-level cancellation by simply checking if write would throw (not the best of the handlings but it does work).

One way to check when close() is called on writer, for example, in a different thread

try {
  if (writable.locked) {
    await writer.ready;
    await writer.write(value);
  }
} catch (e) {
  console.warn(e.message);
}

if necessary handle cannot write to a closing writable stream error(s).

[Spongman]

yeah, thanks. that's what i figured...

i reported it here: w3c/ServiceWorker#957 (comment)
(cc'd, here: https://bugs.chromium.org/p/chromium/issues/detail?id=638494)

[jimmywarting]

Now with transferable stream there is a way to detect when the bucket strategy is full (Meaning the client paused the stream) you can also detect if user aborted the request.

[gwdp]

Chrome issue (#638494) got merged feel days ago into upstream (https://chromium-review.googlesource.com/c/chromium/src/+/3347484)..
Still, long road to stable/end-users, but it's something.

Getting a different behaviour on Canary, write(xx) is throwing undefined; digging into it to see if can squeeze something out of it although no console.log('user aborted') being fired :/

[jimmywarting]

There should be right about here

StreamSaver.js/sw.js

Line 67 in 2f6c53e

console.log('user aborted')

But it doesn't get triggered... Think it's a bug or missing
Someone should report this to chromium

[jimmywarting]

Thanks for that 👍

[TexKiller]

@jimmywarting

Just a heads up: Firefox already does notify the Stream passed to respondWith when the download is cancelled... This line is executed:

StreamSaver.js/sw.js

Line 65 in da6218e

console.log('user aborted')

[jimmywarting]

I consider the abort event as a minor issue and it would automatically be resolved once all browser start supporting transferable streams.

However it would be nice to solve this abort event in Firefox
Will push this missing abort event for a later release

[eschaefer]

Hey all, this was a "must" for me in Firefox, so here's my solution: #105

Would love feedback.

[M0aB0z]

Now with transferable stream there is a way to detect when the bucket strategy is full (Meaning the client paused the stream) you can also detect if user aborted the request.

Hello Guys,
Thanks a lot jimmywarting for the amazing work you did on this project, it's very appreciated.
Unfortunately the user's events (Pause & Cancel) seem critical features in order to stop the write operations accordingly to the user intent.

Is there any news about this point ?

Thks

[jimmywarting]

sry, got some bad news.

transferable streams is still only supported in Blink with an experimental flag. https://chromestatus.com/feature/5298733486964736

2nd issue is about cancelation... chrome never emits the cancel event (here) but it can fill up the bucket to the point where it stop stops calling pull(ctrl) {...} (asking for more data)
Here is the (now old) chromium bug about cancelation: https://bugs.chromium.org/p/chromium/issues/detail?id=638494 - pls star it to make it important
Only FF emits this cancel event

3th issue is that streamsaver lacks the concept about buckets when talking to the service worker over MessageChannel it don't use the pull system and just eagerly enqueues more data without any respect to a bucket or the pull request. - which can lead to memory issue if enqueue data faster than what you are able to write it to the disk.

---

I have written a 2nd stream saving library based on native file system access that kind of acts like an adapter for different storages (such as writing to sandboxed fs, IndexedDB, cache storage and the memory) it too also comes with a adoption for writing data to the disk using same technique as StreamSaver with service worker. However my adapter dose it i slightly different and behaves more like a .pipe should with respect to cancel and only ask for more data when it needs it. it also properly reports back with a promise when data has been sent from main thread over to the service worker streams (which streamsaver totally lacks - it just resolves writer.write(data) directly)
and using service worker is optional too in which case it will build up a blob in the memory and later download it using a[download] instead. I have made it optional since a few ppl wants to host the service worker themself so there is more manual work to set it up properly.

I think that in the feature native file system access will supersede FileSaver and my own StreamSaver lib when it gets more adoptions at which point i will maybe deprecate StreamSaver in favor of my 2nd file system adapter - but not yet

Maybe you would like to try it out instead?

One thing that native file system dose differently is that it can enqueue other types of data such as string, blobs and any typed array or arraybuffer - so saving a large blob/file is more beneficial since the browser don't need to read the blob.

Oh, and give this issue a 👍 as well ;)

[M0aB0z]

sry, got some bad news.

transferable streams is still only supported in Blink with an experimental flag. https://chromestatus.com/feature/5298733486964736

2nd issue is about cancelation... chrome never emits the cancel event (here) but it can fill up the bucket to the point where it stop stops calling pull(ctrl) {...} (asking for more data)
Here is the (now old) chromium bug about cancelation: https://bugs.chromium.org/p/chromium/issues/detail?id=638494 - pls star it to make it important
Only FF emits this cancel event

3th issue is that streamsaver lacks the concept about buckets when talking to the service worker over MessageChannel it don't use the pull system and just eagerly enqueues more data without any respect to a bucket or the pull request. - which can lead to memory issue if enqueue data faster than what you are able to write it to the disk.

I have written a 2nd stream saving library based on native file system access that kind of acts like an adapter for different storages (such as writing to sandboxed fs, IndexedDB, cache storage and the memory) it too also comes with a adoption for writing data to the disk using same technique as StreamSaver with service worker. However my adapter dose it i slightly different and behaves more like a .pipe should with respect to cancel and only ask for more data when it needs it. it also properly reports back with a promise when data has been sent from main thread over to the service worker streams (which streamsaver totally lacks - it just resolves writer.write(data) directly)
and using service worker is optional too in which case it will build up a blob in the memory and later download it using a[download] instead. I have made it optional since a few ppl wants to host the service worker themself so there is more manual work to set it up properly.

I think that in the feature native file system access will supersede FileSaver and my own StreamSaver lib when it gets more adoptions at which point i will maybe deprecate StreamSaver in favor of my 2nd file system adapter - but not yet

Maybe you would like to try it out instead?

One thing that native file system dose differently is that it can enqueue other types of data such as string, blobs and any typed array or arraybuffer - so saving a large blob/file is more beneficial since the browser don't need to read the blob.

Oh, and give this issue a 👍 as well ;)

Thanks for your detailed answer, I'll have a look on your file system lib, looks very interesting and may solve my problem.
Thanks again for all your quality work.

[guest271314]

@jimmywarting Is there a minimal, verifiable, complete example of this issue?

[jimmywarting]

Hmm, i tried to create a minimal plunkr example here: https://plnkr.co/edit/I27Dl0chuMCuaoHD?open=lib%2Fscript.js&preview

basically wait 2s until the iframe pops up and save the never ending file download. then cancel the download from the browser UI and expect the cancel event to be called but never happens.

I'm 100% sure that this used to work in firefox but i can't get the cancel event to fire anymore in firefox. 😕
also tried my examples but i didn't get the "user aborted" console message there either.

[guest271314]

cancel is not an event. cancel() method is called after cancel(reason) is executed if the stream is not locked. The stream becomes locked momentarily after respondWith() is excuted. You can step through this with placement of rs.cancel()

self.addEventListener('activate', (event) => {
  event.waitUntil(clients.claim());
});

var _;
onfetch = async (evt) => {
  console.log(evt.request.url);
  if (evt.request.url.endsWith('ping')) {
    try {
      var rs = new ReadableStream({
        async start(ctrl) {
          return (_ = ctrl);
        },
        async pull() {
          _.enqueue(new Uint8Array([97]));
          await new Promise((r) => setTimeout(r, 250));
        },
        cancel(reason) {
          console.log('user aborted the download', reason);
        },
      });

      const headers = {
        'content-disposition': 'attachment; filename="filename.txt"',
      };
      var res = new Response(rs, { headers });
      // rs.cancel(0);
      evt.respondWith(res);
      // rs.cancel(0);
      setTimeout(() => {
        // rs.cancel(0);
        console.log(rs, res, _);
      }, 3000);
    } catch (e) {
      console.error(e);
    }
  }
};

console.log('que?');

sw.js:11 Uncaught (in promise) TypeError: Failed to execute 'cancel' on 'ReadableStream': Cannot cancel a locked stream
at sw.js:11

sw.js:30 Uncaught (in promise) DOMException: Failed to execute 'fetch' on 'WorkerGlobalScope': The user aborted a request.
    at onfetch (https://run.plnkr.co/preview/ckh2vkij700082z6y9i3qqrz3/sw.js:30:21)

The FetchEvent for "https://run.plnkr.co/preview/ckh2vkij700082z6y9i3qqrz3/ping" resulted in a network error response: the promise was rejected.
Promise.then (async)
onfetch @ VM4 sw.js:27
VM4 sw.js:1 Uncaught (in promise) DOMException: The user aborted a request.

user aborted the download 0
run.plnkr.co/preview/ckh2y3vu2000a2z6ym2r13z5o/sw.js:35 TypeError: Failed to construct 'Response': Response body object should not be disturbed or locked
    at onfetch (run.plnkr.co/preview/ckh2y3vu2000a2z6ym2r13z5o/sw.js:27)
onfetch @ run.plnkr.co/preview/ckh2y3vu2000a2z6ym2r13z5o/sw.js:35
VM2582 script.js:6 GET https://run.plnkr.co/preview/ckh2y3vu2000a2z6ym2r13z5o/ping 404
(anonymous) @ VM2582 script.js:6
setTimeout (async)

user aborted the download 0
The FetchEvent for "https://run.plnkr.co/preview/ckh2y9o8r000d2z6yucmc7bz4/ping" resulted in a network error response: a Response whose "bodyUsed" is "true" cannot be used to respond to a request.
Promise.then (async)
onfetch @ sw.js:28
script.js:6 GET https://run.plnkr.co/preview/ckh2y9o8r000d2z6yucmc7bz4/ping net::ERR_FAILED
(anonymous) @ script.js:6
setTimeout (async)
(anonymous) @ script.js:3
Promise.then (async)
(anonymous) @ script.js:2
TypeError: Failed to fetch
(anonymous) @ VM2582 script.js:3


sw.js:31 Uncaught (in promise) TypeError: Failed to execute 'cancel' on 'ReadableStream': Cannot cancel a locked stream
    at sw.js:31
Promise.then (async)
(anonymous) @ VM2582 script.js:2
VM2582 script.js:14 <link rel="stylesheet" href="//fonts.googleapis.com/css?family=Roboto:300,300italic,700,700italic">
<link rel="stylesheet" href="//unpkg.com/normalize.css/normalize.css">
<link rel="stylesheet" href="//unpkg.com/milligram/dist/milligram.min.css">
<h1>Oh dear, something didn't go quite right</h1>
<h2>Not Found</h2>

At client side AbortController can be used (see logged messaged above)

var controller, signal;
navigator.serviceWorker.register('sw.js', {scope: './'}).then(reg => {
  setTimeout(() => {
    controller = new AbortController();
    signal = controller.signal;
    fetch('./ping', {signal})
    .then(r => {
      var reader = r.body.getReader();
      reader.read().then(function process({value, done}) {
          if (done) {
            console.log(done);
            return reader.closed;
          }
          console.log(new TextDecoder().decode(value));
          return reader.read().then(process)
      })
    })
    .catch(console.error)

    document.querySelector('h1')
    .onclick = e => controller.abort();

  }, 2000)
})

See also the code at Is it possible to write to WebAssembly.Memory in PHP that is exported to and read in JavaScript in parallel? at background.js in an extension, where we stream raw PCM audio (without a definitive end) via fetch() from php passthru() and stop the stream using abort(), which can be achieved at Chromium using QuicTransport https://github.com/guest271314/quictransport without Native Messaging.

const id = 'native_messaging_stream';
let externalPort, controller, signal;

chrome.runtime.onConnectExternal.addListener(port => {
  console.log(port);
  externalPort = port;
  externalPort.onMessage.addListener(message => {
    if (message === 'start') {
      chrome.runtime.sendNativeMessage(id, {}, async _ => {
        console.log(_);
        if (chrome.runtime.lastError) {
          console.warn(chrome.runtime.lastError.message);
        }
        controller = new AbortController();
        signal = controller.signal;
        // wait until bash script completes, server starts
        for await (const _ of (async function* stream() {
          while (true) {
            try {
              if ((await fetch('http://localhost:8000', { method: 'HEAD' })).ok)
                break;
            } catch (e) {
              console.warn(e.message);
              yield;
            }
          }
        })());
        try {
          const response = await fetch('http://localhost:8000?start=true', {
            cache: 'no-store',
            mode: 'cors',
            method: 'get',
            signal
          });
          console.log(...response.headers);
          const readable = response.body;
          readable
            .pipeTo(
              new WritableStream({
                write: async value => {
                  // value is a Uint8Array, postMessage() here only supports cloning, not transfer
                  externalPort.postMessage(JSON.stringify(value));
                },
              })
            )
            .catch(err => {
              console.warn(err);
              externalPort.postMessage('done');
            });
        } catch (err) {
          console.error(err);
        }
      });
    }
    if (message === 'stop') {
      controller.abort();
      chrome.runtime.sendNativeMessage(id, {}, _ => {
        if (chrome.runtime.lastError) {
          console.warn(chrome.runtime.lastError.message);
        }
        console.log('everything should be done');
      });
    }
  });
});

ServiceWorker does not appear to be well-suited for the task. We can stream the file without ServiceWorker using fetch() see this answer at How to solve Uncaught RangeError when download large size json where successfully streamed and downloaded a 189MB file, and, as you indicated, use File System Access, something like

(async() =>  {
  const dir = await showDirectoryPicker();
  const status = await dir.requestPermission({mode: 'readwrite'});
  const url = 'https://fetch-stream-audio.anthum.com/72kbps/opus/house--64kbs.opus?cacheBust=1';
  const handle = await dir.getFile('house--64kbs.opus', { create: true });
  const wfs = await handle.createWritable();
  const response = await fetch(url);
  const body = await response.body;
  console.log("starting write");
  await body.pipeTo(wfs, { preventCancel: true });
  const file = await (await dir.getFile('house--64kbs.opus')).getFile();
  console.log(file);
})();

(BTW, created several screenshot workarounds, two of which are published at the linked repository https://gist.github.com/guest271314/13739f7b0343d6403058c3dbca4f5580)

[jimmywarting]

cancel is not an event.

didn't know what to call it, it is kind of like an event that happens when it get aborted by the user... but whatever

ServiceWorker does not appear to be well-suited for the task. We can stream the file without ServiceWorker using fetch() see this answer at How to solve Uncaught RangeError when download large size json where successfully streamed and downloaded a 189MB file, and, as you indicated, use File System Access, something like

I know service worker isn't the best solution but it's currently the only/best client side solution at the moment until native file system access becomes more wildly adapted in more browser without a experimental flag. it too comes with its drawback

• it lacks support for suggested filename, so you are required to ask for directory and write the file yourself - or you can let the user choose the name.
• it isn't associated with any native browser UI element where you can see the progress and cancel the download

I'm using service worker to mimic a normal download that occur from downloading something from a server so that i don't have to build a Blob in the memory and later download the hole file at once as it's wasteful use of memory when downloading large files. + there is better ways to solve that 4y old issue if he just did response.blob() and hope that browser offload large blob to the disk instead. ( see Chrome's Blob Storage System Design )
or if he really needed a json response call response.json() it seems to be much more performant to just do
new Response(str).json().then(...) instead of JSON.parse(str)

And as always use the server to download the file if it comes from the cloud if you can
or download the file without fetch and download it directly without blob and fetch

var a = document.createElement("a")
a.download = "citylots.json"
// mostly only work for same origin
a.href = "/citylots.json"
document.body.appendChild(a)
a.click()

[guest271314]

For this specific issue you can rearrage the placement of cancel(reason) to get the reason 0, at cancel (reason) {} method.

AbortController client side a MessagePort can be utilized to send the messsage to ServiceWorker to cancel the stream - before the stream is locked. I'm not seeing releaseLock() defined at Chromium 88. Either way the data is read into memory. If you fetch() client side you can precisely count progress and abort the request - then just doenload the file. Using Native Messaging, which is available at Chromium and Firefox you can write the file or files directly to disk at a shell, or combination of browser and shell.

I'm sure we could build a custom HTML element and implement progress events, either by estimation https://stackoverflow.com/a/41215449 or counting every byte How to read and echo file size of uploaded file being written at server in real time without blocking at both server and client?.

[guest271314]

OS froze at plnkr during tests. Registering and un-registering ServiceWorkers. We should be able to achieve something similar to what is described here. Will continue testing.

[guest271314]

The plnkr throws error at Nightly 84

Failed to register/update a ServiceWorker for scope ‘https://run.plnkr.co/preview/ckh4cy0qq00071w6pnxnhpr3v/’: 
Storage access is restricted in this context due to user settings or private browsing mode. 
script.js:1:24
Uncaught (in promise) DOMException: The operation is insecure.

Some observations running the below code https://plnkr.co/edit/P2op0uo5YBA5eEEm?open=lib%2Fscript.js at Chromium 88, which might not be the exact requirement, though is a start and extensible.

• AbortController does not cancel the ReadableStream passed to Response
• Download Cancel UI does not communicate messages to ServiceWorker or vice versa
• Once Response is passed to respondWith() the ReadableStream is locked - AFAICT there does not appear to be any way to unlock the stream for the purpose of calling cancel() without an error being thrown

Utilizing clinet-side code we can get progress of bytes enqueed, post messages containing download status to main thread using MessageChannel or BroadcastChannel, and call ReadableStreamDefaultController.close() and AbortController.abort() when the appropirate message is received from client document.

index.html

<!DOCTYPE html>

<html>
  <head>
    <script src="lib/script.js"></script>
  </head>

  <body>
    <button id="start">Start download</button>

    <button id="abort">Abort download</button>
  </body>
</html>

lib/script.js

const unregisterServiceWorkers = async (_) => {
  const registrations = await navigator.serviceWorker.getRegistrations();
  for (const registration of registrations) {
    console.log(registration);
    try {
      await registration.unregister();
    } catch (e) {
      throw e;
    }
  }
  return `ServiceWorker's unregistered`;
};

const bc = new BroadcastChannel('downloads');

bc.onmessage = (e) => {
  console.log(e.data);
  if (e.data.aborted) {
    unregisterServiceWorkers()
      .then((_) => {
        console.log(_);
        bc.close();
      })
      .catch(console.error);
  }
};

onload = (_) => {
  document.querySelector('#abort').onclick = (_) =>
    bc.postMessage({ abort: true });

  document.querySelector('#start').onclick = (_) => {
    const iframe = document.createElement('iframe');
    iframe.src = './ping';
    document.body.append(iframe);
  };
};

navigator.serviceWorker.register('sw.js', { scope: './' }).then((reg) => {});

sw.js

self.addEventListener('activate', (event) => {
  event.waitUntil(clients.claim());
});

let rs;

let bytes = 0;

let n = 0;

let abort = false;

let aborted = false;

const controller = new AbortController();

const signal = controller.signal;

signal.onabort = (e) => {
  try {
    console.log(e);
    console.log(source, controller, rs);
    ({ aborted } = e.currentTarget);
    bc.postMessage({ aborted });
  } catch (e) {
    console.error(e);
  }
};

const bc = new BroadcastChannel('downloads');

bc.onmessage = (e) => {
  if (e.data.abort) {
    abort = true;
  }
};

const source = {
  controller: new AbortController(),
  start: async (ctrl) => {
    console.log('starting download');
    return;
  },
  pull: async (ctrl) => {
    ++n;
    if (abort) {
      ctrl.close();
      controller.abort();
    } else {
      const data = new TextEncoder().encode(n + '\n');
      bytes += data.buffer.byteLength;
      ctrl.enqueue(data);
      bc.postMessage({ bytes, aborted });
      await new Promise((r) => setTimeout(r, 50));
    }
  },
  cancel: (reason) => {
    console.log('user aborted the download', reason);
  },
};

onfetch = (evt) => {
  console.log(evt.request);

  if (evt.request.url.endsWith('ping')) {
    rs = new ReadableStream(source);
    const headers = {
      'content-disposition': 'attachment; filename="filename.txt"',
    };

    const res = new Response(rs, { headers, signal });
    console.log(controller, res);

    evt.respondWith(res);
  }
};

console.log('que?');

[guest271314]

Once the ReadableStream is passed to Response the stream is locked and AFAICT cannot be cancelled, thus await cancel(reason) will throw error and cancel(reason) {console.log(reason)} will not be executed.

Response actually does not expect a signal property per https://bugs.chromium.org/p/chromium/issues/detail?id=823697#c14

You may have included a signal attribute in your Response constructor options dictionary, but its not read. The spec only supports adding a signal to the Request.

Also, its not clear what the signal on a Response would accomplish. If you want to abort the Response body you can just error the body stream, no?

pipeTo() and pipeThrough() do expect optional signal properties https://streams.spec.whatwg.org/#ref-for-rs-pipe-to%E2%91%A1.

Firefox does not support pipeTo() and pipeThrough(). We need to adjust the code to branch at a condition, e.g., 'pipeTo' in readable, then utilize only getReader() and read() instead of AbortController with WritableStream, which is still behind a flag at Nightly 84.

We tee() a ReadableStream to read bytes and wait for abort signal or message to cancel the download by cancelling or closing all streams, initial and derived tee'd pairs. If the paired stream is not aborted the unlocked pair is passed to Response

Tested several hundred runs at Chromium 88 to derive the current working example that still requires independent verification. The main issue that encountered when testing is ServiceWorker "life-cycle", or ServiceWorkers that remain after page reload, and re-run code that has changed; determining exactly when all service workers are unregistered; storage messages; inconsistent behaviour between reloads of the tab.

Running the code at Firefox or Nightly at localhost logs exception

Failed to get service worker registration(s): 
Storage access is restricted in this context 
due to user settings or private browsing mode. script.js:2:54

Uncaught (in promise) DOMException: The operation is insecure. script.js:2

Have not yet successfully run the code at Mozilla browsers. The working Chromium version provides a template of how the code can work at Firefox, given similar implementations and support.

From what can gather from the entirety of the issue this is resulting interpretation of a potential solution to handle both aborting the download and notifying the client of the state of the download. Kindly verify the code produces the expected output and handles the use cases described based on own interpreation of the issue, above.

index.html

<!DOCTYPE html>

<html>
  <head>
    <script src="lib/script.js"></script>
  </head>

  <body>
    <button id="start">Start download</button>

    <button id="abort">Abort download</button>
  </body>
</html>

lib/script.js

const unregisterServiceWorkers = async (_) => {
  const registrations = await navigator.serviceWorker.getRegistrations();
  for (const registration of registrations) {
    console.log(registration);
    try {
      await registration.unregister();
    } catch (e) {
      throw e;
    }
  }
  return `ServiceWorker's unregistered`;
};

const bc = new BroadcastChannel('downloads');

bc.onmessage = (e) => {
  console.log(e.data);
};

onload = async (_) => {
  console.log(await unregisterServiceWorkers());

  document.querySelector('#abort').onclick = (_) =>
    bc.postMessage({ abort: true });

  document.querySelector('#start').onclick = async (_) => {
    console.log(await unregisterServiceWorkers());
    console.log(
      await navigator.serviceWorker.register('sw.js', { scope: './' })
    );
    let node = document.querySelector('iframe');
    if (node) document.body.removeChild(node);
    const iframe = document.createElement('iframe');
    iframe.onload = async (e) => {
      console.log(e);
    };
    document.body.append(iframe);
    iframe.src = './ping';
  };
};

sw.js

// https://stackoverflow.com/a/34046299
self.addEventListener('install', (event) => {
  // Bypass the waiting lifecycle stage,
  // just in case there's an older version of this SW registration.
  event.waitUntil(self.skipWaiting());
});

self.addEventListener('activate', (event) => {
  // Take control of all pages under this SW's scope immediately,
  // instead of waiting for reload/navigation.
  event.waitUntil(self.clients.claim());
});

self.addEventListener('fetch', (event) => {
  console.log(event.request);

  if (event.request.url.endsWith('ping')) {
    var encoder = new TextEncoder();

    var bytes = 0;

    var n = 0;

    var abort = false;

    let aborted = false;

    var res;

    const bc = new BroadcastChannel('downloads');

    bc.onmessage = (e) => {
      console.log(e.data);
      if (e.data.abort) {
        abort = true;
      }
    };

    var controller = new AbortController();
    var signal = controller.signal;
    console.log(controller, signal);
    signal.onabort = (e) => {
      console.log(
        `Event type:${e.type}\nEvent target:${e.target.constructor.name}`
      );
    };
    var readable = new ReadableStream({
      async pull(c) {
        if (n === 10 && !abort) {
          c.close();
          return;
        }
        const data = encoder.encode(n + '\n');
        bytes += data.buffer.byteLength;
        c.enqueue(data);
        bc.postMessage({ bytes, aborted });
        await new Promise((r) => setTimeout(r, 1000));
        ++n;
      },
      cancel(reason) {
        console.log(
          `readable cancel(reason):${reason.join(
            '\n'
          )}\nreadable ReadableStream.locked:${readable.locked}\na locked:${
            a.locked
          }\nb.locked:${b.locked}`
        );
      },
    });

    var [a, b] = readable.tee();
    console.log({ readable, a, b });

    async function cancelable() {
      if ('pipeTo' in b) {
        var writeable = new WritableStream({
          async write(v, c) {
            console.log(v);
            if (abort) {
              controller.abort();
              try {
                console.log(await a.cancel('Download aborted!'));
              } catch (e) {
                console.error(e);
              }
            }
          },
          abort(reason) {
            console.log(
              `abort(reason):${reason}\nWritableStream.locked:${writeable.locked}`
            );
          },
        });
        return b
          .pipeTo(writeable, { preventCancel: false, signal })
          .catch((e) => {
            console.log(
              `catch(e):${e}\nReadableStream.locked:${readable.locked}\nWritableStream.locked:${writeable.locked}`
            );
            bc.postMessage({ aborted: true });
            return 'Download aborted.';
          });
      } else {
        var reader = b.getReader();
        return reader.read().then(async function process({ value, done }) {
          if (done) {
            if (abort) {
              reader.releaseLock();
              reader.cancel();
              console.log(await a.cancel('Download aborted!'));
              bc.postMessage({ aborted: true });
            }
            return reader.closed.then((_) => 'Download aborted.');
          }

          return reader.read().then(process).catch(console.error);
        });
      }
    }

    var downloadable = cancelable().then((result) => {
      console.log({ result });
      const headers = {
        'content-disposition': 'attachment; filename="filename.txt"',
      };
      try {
        bc.postMessage({ done: true });
        bc.close();
        res = new Response(a, { headers, cache: 'no-store' });
        console.log(res);
        return res;
      } catch (e) {
        console.error(e);
      } finally {
        console.assert(res, { res });
      }
    });

    evt.respondWith(downloadable);
  }
});

console.log('que?');

Updated plnkr https://plnkr.co/edit/P2op0uo5YBA5eEEm

[jimmywarting]

I have another older (original) idea in mind also that didn't work earlier. Blink (v85) have recently gotten support for streaming upload. Example:

await fetch('https://httpbin.org/post', {
  method: 'POST',
  body: new ReadableStream({
    start(ctrl) {
      ctrl.enqueue(new Uint8Array([97])) // a
      ctrl.close()
    }
  })
}).then(r => r.json()).then(j=>j.data) // "a"

None of the other browser supports it yet. but it can simplify stuff quite a bit.
You can just echo back everything and pipe the the download iframe and the ajax request to eachother

// oversimplified (you need two fetch events for this to work)
// canceling the ajax with a AbortSignal or interrupt the readableStream from the main thread can abort the download (aka: iframeEvent).
// canceling the downloading iframe body (from browser UI) can abort the ajax (aka: ajaxEvent)

iframeEvent.respondWith(new Response(ajaxEvent.request.body, { headers: ajaxEvent.request.headers }))
ajaxEvent.respondWith(new Response(iframeEvent.request.body))

• You don't need any MessageChannel to transfer chunks (means less overhead)
• It's tighter coupled as a writable stream pipeline should be (with the bucket) (StreamSaver currently lacks any backpressure algorithm) writable.write(chunk) just resolves directly
• you don't need to ping the service worker to keep it alive since it don't have to do any more work.

[jimmywarting]

if ('pipeTo' in b) {

is insufficient, should check for both pipeTo and WirtableStream support
(Safari have pipeTo but no WritableStream)

Other thing I don't like is that you use BroadCast channel.
one download should have one own dedicated MessageChannel - think of it as downloading two files at the same time...
aborting one of them should not abort both or the other
(but i get it if it's just for testing (PoC) purpose)

[guest271314]

Yes, the code is proof-of-concept. Does the code embody what you are trying to achieve? I first tried messaging with navigator.serverWorker.controller yet could not get the messages to signal back and forth, used BroadcastChannel as a template for MessageChannel usage.

Firefox has some form of bug that might have to do with ServiceWorker not recognizing iframe as a destination, therefore fetch event is not caught for iframe.src = 'destination_that_does_not_exist', instead 404 response is returned which could lead to BroadcastChannel surviving page reload, just narrowed that down and have a prospective fix for the page-reloaded BroadcastChannel messages surviving https://bugzilla.mozilla.org/show_bug.cgi?id=1676043 if (n === 10 || abort) at the reader that is the writer for the download content; just started reading whatwg/fetch#948, which might provide some useful information; though do not have a solution for Firefox loading the same page at localhost or 404 at plnkr for the request ./ping.

Will check how useful streaming upload is for this case. That has been a limitation for expansion of use cases.

From perspective here Chromium version achieves what is described here, using both WritableStream with pipeTo() and AbortController and only ReadableStreams with tee().

Does the example code achieve the expected result for you at Chromium re notifications of aborted request and download at this issue?

[guest271314]

Question: Why not just write to a Blob or File at main thread, instead of using ServiceWorker at all, then download?

[guest271314]

Just tried the plnkr again at Firefox 82. The download was successful initially, and aborted the download, then began throwing error. The issue could be relaed to storage.

[Spongman]

I’m not sure if this is still the case but this issue’s use-case originally had the ability to navigate away from the page while the download continued. Also the downloads I was doing were significantly larger than RAM.

[guest271314]

I’m not sure if this is still the case but this issue’s use-case originally had the ability to navigate away from the page while the download continued. Also the downloads I was doing were significantly larger than RAM.

Navigating away from the page while download continues appears to be different from knowing if the user cancels a download.

If the download is larger than RAM what is the expected result once RAM capacity is reached by either direct or peripheral correlation with the download process?

Drag and drop the completed zipped folder?

[guest271314]

@jimmywarting The fetch() with ReadableStream example requires a HTTP/3 server. Tested downloading using QuicTransport briefly yesterday; works to an appreciable degree, yet because we are sending streams there was an issue of the Python code that used with a bash script

#!/bin/bash
echo "$1" | cat > written
echo 'ok'

only writing the last part of a larger input stream to the file. Still need to test more with that API.

[Spongman]

I think you misunderstand. The use case worked - navigating away AND huge downloads. The only thing that didn’t was the worker wasn’t notified if the download was cancelled.

[guest271314]

Is cancelation of download at browser UI specified? Specifications tend to stay away from mandating exact browser UI.

If the code is in control of the input ReadableStream then when close() is called from inside start() or pull() then you can notify the user at the next step. Once the stream is passed to Response the stream is locked, await cancel(reason) will throw error and cancel(reason) {} will not be called. If you supply buttons or a programmic means to the user to communicate with the ServiceWorker and tee() the download stream, you can cancel the paired stream and return the unlocked stream, which will either be the content to download of a disturbed or closed stream if you also close that paired stream. That works at Chromium due to pipeTo() with AbortController signal support. At Firefo the download is still canceled, you just need to keep the BroadcastChannel or other means of communication active long enough to signal the main thread before unregistering the ServiceWorker which is the most fragile part of the process, as we do not want ServiceWorkers remaining active after the process is complete.

[guest271314]

One question I have is what is the expected result of canceling the download as to downloaded content? Download partial content or do not download anything?

[Spongman]

I would expect that if the download is in the browser’s download window then it should behave the same as canceling a regular download.

[guest271314]

That does not really answer the question. Both Chromium and Firefox have the ability to download partial content when automatic downloads is set, the default folder for downloads is set where no prompts occur for where to download, or when prompted. That means that canceling a download can result in partial downloads, however, intuitively, when I cancel a download by any means, whether browser UI or an application, I did not expect nor do I want partially downloaded content to be saved to local filesystem anyway.

[guest271314]

A general working solution for Chromium, Chrome

sw.js

let readable;

self.addEventListener('message', async (event) => {
  readable = event.data;
  console.log(event);
  event.source.postMessage('ServiceWorker ready to serve download');
});

self.addEventListener('fetch', (event) => {
  console.log(event);
  let url = new URL(event.request.url);
  if (url.pathname.includes('get-file')) {
    console.log({ readable });
    const headers = {
      'content-disposition': 'attachment; filename="filename.txt"',
    };
    event.respondWith(
      new Response(readable, {
        headers,
      })
    );
  }
});

self.addEventListener('install', (event) => {
  console.log(event);
  event.waitUntil(self.skipWaiting());
});

self.addEventListener('activate', (event) => {
  console.log(event);
  event.waitUntil(self.clients.claim());
});

index.html

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <title>ServiceWorker - Response.body cancellation test</title>
  </head>
  <body>
    <h3>Test progress is outputted to console</h3>
    Related chromium issue:
    <a
      href="https://bugs.chromium.org/p/chromium/issues/detail?id=638494"
      target="_blank"
      >638494</a
    >
    - Response.body cancellation should be notified to the service worker.
    <hr />
    <a
      href="https://github.com/norstbox/trial/tree/master/sw-stream-cancellation"
      target="_blank"
      >View source</a
    >
    on GitHub
    <br /><br />
    <button id="start-download">Start download</button
    ><button id="abort-download">Abort download</button>
    <script>
      let readable, resolve, promise;

      const unregisterServiceWorkers = async (_) => {
        const registrations = await navigator.serviceWorker.getRegistrations();
        for (const registration of registrations) {
          console.log(registration);
          try {
            await registration.unregister();
          } catch (e) {
            throw e;
          }
        }
        return `ServiceWorker's unregistered`;
      };
      unregisterServiceWorkers().then(console.log, console.error);
      const start = document.getElementById('start-download');

      const abort = document.getElementById('abort-download');

      abort.onclick = async (e) => {
        readable.cancel('Download aborted');
      };

      start.onclick = async (e) => {
        if (document.getElementById('download')) {
          document.body.removeChild(document.getElementById('download'));
        }
        promise = new Promise((_) => (resolve = _));
        const unregisterable = await unregisterServiceWorkers();

        console.log(unregisterable);

        const reg = await navigator.serviceWorker.register('sw.js', {
          scope: './',
        });

        console.log(reg);

        let controller;

        readable = new ReadableStream({
          start(_) {
            return (controller = _);
          },
          cancel(reason) {
            console.log(`cancel(reason): ${reason}`);
            resolve(reason);
          },
        });
        const encoder = new TextEncoder();
        downloadable: for (let i = 0; i < 100; i++) {
          console.log(i, readable, controller);
          try {
            await new Promise((r) => setTimeout(r, 100));
            controller.enqueue(encoder.encode(i + '\n'));
          } catch (err) {
            console.warn(err.message);
            break downloadable;
          }
        }
        resolve('Download ready');

        if ((await promise) === 'Download aborted') {
          console.log(`${await promise}, ${await unregisterServiceWorkers()}`);
          readable = promise = resolve = void 0;
        } else {
          console.log(reg);
          controller.close();

          navigator.serviceWorker.onmessage = async (event) => {
            console.log(event);
            if (event.data === 'ServiceWorker ready to serve download') {
              const iframe = document.createElement('iframe');

              document.body.appendChild(iframe);
              iframe.width = iframe.height = 0;
              iframe.id = 'download';
              iframe.src = 'get-file';
            }
          };
          navigator.serviceWorker.controller.postMessage(readable, [readable]);
        }
      };
    </script>
  </body>
</html>

Explanation

Use enqueue() and cancel() on the main thread.

Since ServiceWorker's are, from perspective here, unreliable and inconsistent, we use ServiceWorker as little as possible. And since we are at Chromium where Transferable Streams are implemented we can transfer the ReadableStream to ServiceWorker for the sole purpose of downloading the resource. We do not need ServiceWorker for any other purpose.

To use the same code at Firefox we can utilize MessageChannel for signaling, though since Firefox does not have this bug we can branch the code for Chromium or Chrome and Firefox.

Tested several hundred times at Chromium 89. The download can be started and cancelled without reloading the page.

plnkr https://plnkr.co/edit/Mdni4JTTOI0ywmyM

GitHub https://github.com/guest271314/trial/tree/guest271314-patch-1

[Spongman]

does the download continue if the page is closed?

[guest271314]

@Spongman I do not know. I did not test that variance. What is expected?

[Spongman]

yes. in this bug's original use-case the download continues even after the page is closed. the bug is that the download cannot be cancelled by the user interacting with the browser's 'Downloads' window.

[guest271314]

Where is that language in this issue or the Chromium bug? I used ServiceWorker as little as possible and did what could to unregister the ServiceWorker(s) that happen to survive. If I navigate away from a page I do not want residual effects of that page to continue.

The 'Downloads' window, as I mentioned above, is UI, and I am not sure how closely Chromium UI is tied to ServiceWorkers. The goal of the code that I posted is to notify user of download cancellation in the HTML document itself, not to attempt to tie what occurs there to browser UI.

[Spongman]

please see the linked issue here: w3c/ServiceWorker#957

Is there any way for the ServiceWorker to know if/when the user cancels the download (in the chrome://downloads/ window)?

[guest271314]

does the download continue if the page is closed?

No. Just tested at plnkr.

We can probably do that using WebTransport, without ServiceWorker at all. I just completed a proof-of-concept using that API to download content to local filesystem.

The title of this issue and the linked Chromium bug and the OP of both do not specifically mention continuing download after navigating away from the page.

Why is the chrome://downloads window important here? If that is equally part of the requirement that should probably be its own bug/issue.

If that is part of the goal then we can just launch the page with window.open() and set the window width and height to 1 to create a background download application, then close the window when the download completes or is cancelled.

[Spongman]

do not specifically mention continuing download after navigating away from the page.

ok, i'm sorry that it wasn't clear from the original report. but that was specifically my use case. i would have thought that the fact that this library causes downloads to appear in the 'Downloads' window would have made that clear.

[guest271314]

Depending on how far you want to go with assigning importance to chrome://downloads we can, as I stated, launch the download HTML and JavaScript in a new window, and then close the window at any point in time. Technically, we can also launch an entirely new instance of Chromium or Chrome just for the purpose of downloading and keeping chrome://downlaods open during the procedure.

I have actually not used this library. I am focused on solving the stated problem.

[guest271314]

If you review the recently posted code we do not even transfer the ReadableStream to ServiceWorker until the stream is closed, so, there is no on-going download to cancel, until we are sure the user really want to download the resource(s).

That brings us back to the important question of whether or not partially dowloaded content is expected or not, which is really an individual decision.

[Spongman]

yes, i see the code and it looks good, but it does not address this issue.
again, in w3c/ServiceWorker#957:

I would have expected the cancel(reason) method of the Response stream's underlyingSource to be called. But in Chrome v52.0.2743.116, it's not.

[guest271314]

If you really want to download partial content, without waiting for the process to complete, at Chromium or Chrome you can create a FileSystemWritableFileStream for each buffer of data and be sure to call close() after write(), or utilize WebTransport to write() data, at server append to the file created at first bytes received as stdin, and then close() the transport. That does not address chrome://downloads at all.

[Spongman]

If you really want to download partial content

i'm not sure where you're getting this. i would expect cancelling the download in the downloads window to do the same as it does for normal downlaods. i haven't tried this since i originally files this bug, but i believe that chrome does delete the .crdownload file when the download is cancelled. however, it doesn't notify the ServiceWorker, and continues to accept enqueued bytes after the download was cancelled and deleted. this is the bug.

[guest271314]

I do not wait on any individual or institution to solve problems. Chromium authors are obviously aware of the problem in the bug, not the specification, yet have not fixed the problem, while origin trialing and deploying multiple API's in the interim. So, let us see if we can fix the problem given the API's we have implemented.

The code does address the issue. We create the ReadableStream in main thread. That same stream is transfered to the ServiceWorker if the user does not cancel() the stream before the stream completes.

What that means as a practical matter is that the approach that you have taken in the original code is, from perspective here, not necessary. That is, there is no reason to perform the entire procedure in the ServiceWorker when at Chromium we have the option of utilizing transferable Streams. That is, if the goal is to actually achieve the requirement of notifying the user when the download is cancelled (excluding chrome://downloads window, which the code does not address) then we write in main thread, where are certain of the writes and the stream not being cancelled and having to communicate with unreliable and inconsistent ServiceWorker's. If the goal is to actually get Chromium authors to fix the bug filed at Chromium bugs, well, that can takes years, if is done at all, e.g., in brief, see https://bugs.chromium.org/p/chromium/issues/detail?id=992235 (https://github.com/guest271314/MediaFragmentRecorder); https://bugs.chromium.org/p/webrtc/issues/detail?id=2276; https://bugs.chromium.org/p/chromium/issues/detail?id=952700; https://bugs.chromium.org/p/chromium/issues/detail?id=931749; et al.

Using a workaround for a bug does not constitute the bug being fixed. Your Chromium issue is still valid. I would emphasize the chrome://downloads window part in the bug itself.

[Spongman]

I do not wait on any individual or institution to solve problems

i think at this point it may be worth moving this discussion to another issue, as whatever problem you're solving here isn't the one that was originally reported.

[guest271314]

I suggest breaking the issue into parts, as I have done.

The entire problem statement should be in the OP of each and every issue or bug filed. Instead of appending additional core requirements in subsequent posts.

Yes, the problem is solved, here. At Chromium you have the ability to transfer the stream to the service worker, for the sole purpose of initiating a download via iframe. it is not necessary to commence the stream in the service worker.

Again, your Chromium issue is valid, but this repository cannot solve that problem or fix that bug, either chrome://downloads aspect or cancel(){} being fired in the service worker. That is the purpose of the Chromium bug. My sole intent is to Fix WontFix, to provide workarounds until implementers actually fix the bug. If that is not viable for you, right now, in order to implement your own application, then I suggest the proper venue is the Chrimium bug - as again, whatever fixes for this issue, in this repository, AFAIK, are not binding on Chromium authors to implement.

[JounQin]

I meet similar situation today.

And I try to change streamsaver:

diff --git a/node_modules/streamsaver/StreamSaver.js b/node_modules/streamsaver/StreamSaver.js
index 018ddc3..acc9288 100644
--- a/node_modules/streamsaver/StreamSaver.js
+++ b/node_modules/streamsaver/StreamSaver.js
@@ -154,6 +154,9 @@
     } else {
       opts = options || {}
     }
+
+	let stream
+	
     if (!useBlobFallback) {
       loadTransporter()
 
@@ -210,7 +213,23 @@
         channel.port1.postMessage({ readableStream }, [ readableStream ])
       }
 
+	  let aborted
+
       channel.port1.onmessage = evt => {
+		if (aborted) {
+			return
+		}
+
+		if (evt.data.aborted) {
+			channel.port1.onmessage = null
+			aborted = true
+			if (stream._writer) {
+				stream._writer.abort()
+				stream._writer = undefined
+			}
+			return stream.abort()
+		}
+
         // Service worker sent us a link that we should open.
         if (evt.data.download) {
           // Special treatment for popup...
@@ -249,7 +268,7 @@
 
     let chunks = []
 
-    return (!useBlobFallback && ts && ts.writable) || new streamSaver.WritableStream({
+    stream = (!useBlobFallback && ts && ts.writable) || new streamSaver.WritableStream({
       write (chunk) {
         if (!(chunk instanceof Uint8Array)) {
           throw new TypeError('Can only wirte Uint8Arrays')
@@ -302,6 +321,8 @@
         channel = null
       }
     }, opts.writableStrategy)
+
+	return stream
   }
 
   return streamSaver

And add the following into sw.js:

cancel() {
  port.postMessage({ aborted: true })
  console.log('user aborted')
}

It seems working on Firefox for most cases to me, while there are still two problems:

1. When the download dialog is not ready, cancel will not be fired at all sometimes

2. I want to cancel the stream request by res.body.cancel(), but an error is thrown TypeError: 'cancel' can't be called on a locked stream., but try/catch will just work (the stream requests in main thread and iframe are aborted correctly, the file is about 600MB).

[JounQin]

My related source codes:

import * as streamSaver from 'streamsaver'
import { WritableStream } from 'web-streams-polyfill/ponyfill'

export const pipeStream = async <T = unknown>(
  reader: ReadableStreamDefaultReader<T>,
  writer: WritableStreamDefaultWriter<T>,
  signal?: AbortSignal,
) => {
  let chunkResult: ReadableStreamDefaultReadResult<T>

  let aborted: boolean | undefined

  while (!signal?.aborted && !(chunkResult = await reader.read()).done) {
    try {
      await writer.write(chunkResult.value)
    } catch (err) {
      if (signal?.aborted) {
        break
      }

      if (!err) {
        aborted = true
        break
      }

      throw err
    }
  }

  if (signal?.aborted || aborted) {
    await Promise.all([reader.cancel(), writer.abort()])
    throw new DOMException('aborted', 'AbortError')
  }

  return writer.close()
}

export const downloadFile = async <T = unknown>(
  readStream: ReadableStream<T>,
  fileName: string,
  signal?: AbortSignal,
) => {
  if (
    (__DEV__ || location.protocol === 'https:') &&
    window.showSaveFilePicker
  ) {
    const handle = await window.showSaveFilePicker({
      suggestedName: fileName,
    })
    return pipeStream(
      readStream.getReader(),
      await handle.createWritable<T>(),
      signal,
    )
  }

  if (streamSaver.mitm !== '/streamsaver/mitm.html') {
    Object.assign(streamSaver, {
      // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition
      WritableStream: streamSaver.WritableStream || WritableStream,
      mitm: '/streamsaver/mitm.html',
    })
  }

  const writeStream = streamSaver.createWriteStream(fileName)

  // Safari
  if (typeof readStream.pipeTo === 'function') {
    return readStream.pipeTo(writeStream, { signal })
  }

  // Firefox
  return pipeStream(readStream.getReader(), writeStream.getWriter(), signal)
}

[gwdp]

Okay, below are my findings:

For Chrome, Canary version 99.0.4828.0 I was able to handle user-level cancellation by simply checking if write would throw (not the best of the handlings but it does work).
Example:

      const streamSaver = StreamSaver.createWriteStream(`abc.zip`);
      this.fileStream = streamSaver.getWriter();
      readable.on('data', (d) => {
          if (this.cancelled) return;
          this.fileStream.write(d)
                                   .catch((e) => { this.cancelled = true; this.abort(); });
    });

However, testing on Firefox, webworker does print user aborted but nothing was implemented there.
To replicate the possible chrome implementation on firefox and possibly other browsers I had to send to the main thread the request to abort and then abort it as manual abort would do it.

@jimmywarting do you believe this commit could be merged or there are any other cases I'm not handling properly?
gwdp@f9e375e

[jimmywarting]

@gwdp hmm, yea maybe.

However I would like to rewrite the hole main-thread <-> service-worker communication.
When I wrote my native-file-system-adapter then i took what I learned from StreamSaver and fixed the problems that I have had in StreamSaver.

When readable streams are not transferable...

...Then we use MessageChannel to write each chunk to send it of to the service worker where i have created a ReadableStream in order to save it. The current problem is that when we do: this.fileStream.write(d) then it's immediately sent of to the service worker via PostMessage and the fileStream.write(d) is resolved right away. This cause a problem cuz you do not know when/if it have been written so it can't handle the back pressure and you will start to have a memory problem if you write data faster then what it is able to write to the disk

So rather than pushing data from main thread to the service worker I instead built a ReadableStream that pulls data from service worker and ask main thread for more data. That way you will know for sure that data has been written to the disk and the service worker is asking for more data when fileStream.write(d) have been resolved.

The source is inspired by @MattiasBuelens remote-web-streams
I would like to use this 👆 it is a tighter glue between two worker threads and much more similar to a transferable stream is suppose to work and communicate back and forth with backpressure

edit: i wish he would have named it transferable-stream-shim or polyfill so it could be easier to google :P

[gwdp]

Wow 🤯. If I understood properly, that would involve a major refactor on the code (simplification as well), sounds like the next release of StreamSaver? 🤓

I have been using StreamSaver for a while now and I found about native-file-system-adapter only yesterday; After a good read on the code and its usage, I still believe stream saver has its own growing use case that is not entirely replaceable by native fs adapter.
In my case, I have been using it for compressing stuff before sending it to the fs, however on my to-do list is to customize the worker to make the compression there and not in the client, which is causing me a double backpressure problem when bandwidth spikes and disk/CPU are busy.
If your proposal is to use https://github.com/MattiasBuelens/remote-web-streams to refactor the communication, I might be able to draft something out in the upcoming week since I need to do something about this compression problem anyways..

For the download cancelled event issue I believe this will need to be handled anyways; Opening a PR for that so other folks can have this fixed in the current version :)

[MattiasBuelens]

edit: i wish he would have named it transferable-stream-shim or polyfill so it could be easier to google :P

I know, I know. 😛 Granted, I made that library before transferable streams were defined in the spec, so I didn't know what the spec would end up calling it.

Also, I'm hesitant to call it a "polyfill". For it to be a proper polyfill, it would need to patch postMessage() and the message event, and then transform the message object to find and replace any streams that also appear in the transferList. That means partially re-implementing StructuredSerializeInternal, so you can replace a ReadableStream in a message like { deeply: { nested: { stream: readableStream } } }. And I really couldn't be bothered with that. 😅

[jimmywarting]

One hacky thing i'm doing in StreamSaver is transferering the MessagePort via another message channel to avoid the so called "Double transfer problem"

More context here: whatwg/streams#276 (comment)

It's essential for streamsaver to work. don't think remote-web-stream taking this into account?

[MattiasBuelens]

You can re-transfer the returned MessagePort as many times as you want, and then construct the ReadableStream in the final realm.

// main.js
const readable = new ReadableStream({ /* ... */ });
const { writable, readablePort } = new RemoteWritableStream();
readable.pipeTo(writable).catch(() => {});
const worker1 = new Worker('./worker1.js');
worker1.postMessage({ readablePort }, [readablePort]);

// worker1.js
const worker2 = new Worker('./worker2.js');
self.onmessage = (event) => {
  const { readablePort } = event.data;
  worker2.postMessage({ readablePort }, [readablePort]);
}

// worker2.js
self.onmessage = (event) => {
  const { readablePort } = event.data;
  const readable = RemoteWebStreams.fromReadablePort(readablePort);
  const reader = readable.getReader();
  // ...
}

As long as you only call fromReadablePort() once, it should be fine.

[jimmywarting]

Ty for the instruction and the use of RemoteWebStreams.fromReadablePort
but isn't that👆 vulnerable to that "double transfer problem"?
i don't see any use of MessageChannel in that example, what happens when worker1 is terminated/closed?

[jimmywarting]

@MattiasBuelens wouldn't it kind of be like this instead?

// To avoid double transfer problem
// 1. creates a `readablePort` that is posted to a MessageChannel
// 2. then send that MessageChannel to worker1 and then forward it to worker2 
// 3. worker #2 listen for one message of the MessageChannel that includes the `readablePort`
// 4. worker #2 sets up a readable with `RemoteWebStreams.fromReadablePort(readablePort)`
// 5. worker1 can now be terminated

const readable = new ReadableStream({ /* ... */ })
const { writable, readablePort } = new RemoteWritableStream()
readable.pipeTo(writable).catch(console.error)

const bridge = new MessageChannel()
bridge.port1.postMessage({}, readablePort)

const worker1 = new Worker('./worker1.js')
worker1.postMessage({}, [bridge.port2])

// --------------------------------------------------------------------------------

// worker1.js (only forwards the MessageChannel bridge to worker2)
const worker2 = new Worker('./worker2.js')
globalThis.onmessage = evt => worker2.postMessage({}, [evt.ports[0]])

// --------------------------------------------------------------------------------

// worker2.js
globalThis.onmessage = evt => {
  const [bridgePort] = evt.ports
  // listen for one message on the BridgePort that will include the readablePort
  bridgePort.onmessage = evt => {
    // readablePort is now directly transferred from main thread to worker2 without
    // being piped through worker1 
    const [readablePort] = evt.ports
    const readable = RemoteWebStreams.fromReadablePort(readablePort)
    const reader = readable.getReader()

    // clean up
    bridgePort.onmessage = null
    bridgePort.close()
  }
}

[MattiasBuelens]

but isn't that👆 vulnerable to that "double transfer problem"?
i don't see any use of MessageChannel in that example, what happens when worker1 is terminated/closed?

Actually, MessagePorts can safely be re-transferred. When you transfer a port, you stop receiving messages in your original realm and the new realm can start receiving its messages instead. Even if the intermediate realm gets terminated, the port still works.

It doesn't really work with the example of worker1 spawning a new worker2, because worker2 cannot outlive worker1 (terminating worker1 also terminates worker2). But you can easily demonstrate this with passing a port back and forth between main and worker:

// main.js
const readable = new ReadableStream({
  _i: 0,
  pull(c) {
    c.enqueue(++this._i);
    if (this._i === 10) {
      c.close();
    }
  }
});
const { writable, readablePort } = new RemoteWebStreams.RemoteWritableStream();
readable.pipeTo(writable).catch(console.error);

// Send the port to the worker
const worker = new Worker("./worker.js");
worker.postMessage({ readablePort }, [readablePort]);

// Receive the port back from the worker
worker.addEventListener('message', evt => {
  // Terminate the worker
  // This shows that the transferred port continues to work, and no longer "goes through" the worker
  worker.terminate();
  // Construct the transferred stream and consume it
  const { readablePort } = evt.data;
  const readable = RemoteWebStreams.fromReadablePort(readablePort);
  readable.pipeTo(new WritableStream({
    write(chunk) {
      console.log('received', chunk);
    }
  }));
});

// worker.js
globalThis.onmessage = evt => {
  // Send the port back to main thread.
  const { readablePort } = evt.data;
  globalThis.postMessage({ readablePort }, [readablePort]);
};

The "double transfer problem" with streams is slightly different. We always construct a new MessageChannel when transferring a ReadableStream, even if that stream was already transferred before. You can simulate this behavior with:

// worker.js
globalThis.onmessage = evt => {
  // Construct a ReadableStream from the given port
  const readable = RemoteWebStreams.fromReadablePort(evt.data.readablePort);
  // Transfer the ReadableStream again
  // This creates a new MessageChannel, independent from the original port
  const { writable, readablePort } = new RemoteWebStreams.RemoteWritableStream();
  // Pipe the two streams together. This is what "connects" the original port and the newly created port.
  readable.pipeTo(writable).catch(console.error);
  // Send the new message port back to main thread
  globalThis.postMessage({ readablePort }, [readablePort]);
};

Here, it's not possible to terminate the worker, since its pipeTo() is passing chunks received from the first port (evt.data.readablePort) to the second port (readablePort). That's the root cause of the double transfer problem.

In order to solve this, the streams specification needs to be changed such that, if a ReadableStream was constructed through the "transfer-receiving steps", then the "transfer steps" should not set up a new MessageChannel and pipeTo(). Instead, it should re-use the original MessagePort that we used to construct the stream. That way, we get the same behavior as in the first example, where the worker simply re-transfers the port. 😉

Of course, the streams specification has to deal with lots of other annoying details, such as "what if the worker first reads a couple of chunks from the stream and then re-transfers it?". remote-web-streams doesn't attempt to solve that right now.

[jimmywarting]

well, i'm not actually using two web workers... I have:

1. The main thread
2. An 3th party hidden iframe or a visible popup, it's sole purpose is to install a service worker (only the popup gets destroyed as soon as the port have been sent to the service worker) (can call this iframe or popup for worker1)
3. A service worker (we could call this worker2)

[guest271314]

@gwdp For completeness when closing the writable side in the pattern release the lock as well for the if condition to be true at that instance

    await writer.close();
    await writer.closed;
    console.log('Writer closed.');
    writer.releaseLock();

If a handler is attached where write() is called multiple errors can be handled in the interim between handler being dispatched multiple times in the same span of time, though will still, in my case, enqueue all necessary chunks.

[MattiasBuelens]

well, i'm not actually using two web workers...

I believe it should work regardless of whether you're using a worker, a service worker or an iframe. MessagePort and postMessage() should work the same in all of them.

But yes, with your current approach, you get the benefits of using native transferable streams, while also avoiding the double transfer problem. With remote-web-streams, you wouldn't be able to use native transferable streams, you'd be manually passing message ports around instead.

Hopefully we can one day fix this double transfer problem in the spec, and then you can finally ditch your "message channel containing a single message with a readable stream" workaround too. 🙏

[guest271314]

once that is done i terminate the 2nd window by closing/removing it and then the stream gets closed or something. i'm not sure if it's a bug or if it's something that isn't spec:ed.

Yes, it is specified, to an appreciable degree w3c/ServiceWorker#980. The ServiceWorker has no WindowClient or Worker client at that point.

Depending on when the ServiceWorker is constructed and how long the download takes persistent data can be lost, if not cached or stored somewhere outside of the ServiceWorker. The fact that a MessageChannel handler is used in ServiceWorker and a ReadableStream is being read from/written to in the ServiceWorker has no impact on ServiceWorker 5 minute inactivity of events. If a variable is declared, the ServiceWorker effectively deletes the variable and restarts again, disentangling all MessagePorts. That is particularly problematic for extension code in Chromium/Chrome which uses MV3 ServiceWorker and Native Messaging (https://bugs.chromium.org/p/chromium/issues/detail?id=1152255; https://bugs.chromium.org/p/chromium/issues/detail?id=1189678). If construction of ServiceWorker and download of all content does not occur within 5 minutes data can be lost. If pausing and resuming a download is a requirement, still needs to be within the 5 minute span, else, again, without storing the data in some temporary container, the data can be lost. To avoid that you can

• Serve a never ending fetch() request (e.g., to a media element) with user-defined ReadableStream constructed by the client and transferred to ServiceWorker with navigator.serviceWorker.active.postMessage(), see w3c/ServiceWorker#882
• Intercept EventSource GET request from client which will keep FetchEvent handler active.

See https://github.com/guest271314/persistent-serviceworker for solutions persist the same ServiceWorker on both Firefox and Chrome.

[guest271314]

once that is done i terminate the 2nd window by closing/removing it and then the stream gets closed or something. i'm not sure if it's a bug or if it's something that isn't spec:ed.

The simplest solution for that case, without changing too much of your existing code pattern, is to keep the <iframe> in the document that created it and post a message to the ServiceWorker every 15 seconds, something like

function closeMessageChannel() {
  parent.postMessage('close', name);
}
navigator.serviceWorker.oncontrollerchange = async (e) => {
  console.log(e);
  closeMessageChannel();
};
navigator.serviceWorker.onmessage = async (e) => {
  parent.postMessage(e.data, name, e.ports);
  try {
    while ((await navigator.serviceWorker.ready).active) {
      (await navigator.serviceWorker.ready).active.postMessage(null);
      await new Promise((resolve) => setTimeout(resolve, 1000 * 15));
    }
    closeMessageChannel();
  } catch (err) {
    console.error(err);
    closeMessageChannel();
  }
};

[jimmywarting]

my hidden iframe (in the cases of when a webpage uses secure https) then the iframe is kept alive and pinging the service worker every now and then... but only if it dose not support transferable streams.

in that case a "real" transferable ReadableStream is sent to the service worker and responded with
evt.respondWith(new Response(transfered_readableStream)), then we don't need any ping/keep-alive hack and the 5 Minutes lifetime dose not matter.

[guest271314]

Then what issue are you having?

Is this

once that is done i terminate the 2nd window by closing/removing it and then the stream gets closed or something. i'm not sure if it's a bug or if it's something that isn't spec:ed.

not the case anymore?

[jimmywarting]

@guest271314 Can we please stop talking about the double transfer problem now? it feels a bit of topic

in StreamSaver case the "double transfer problem" is only a issue when the site is insecure and the browser don't support transferable streams...
when the site is insecure it will open up a secure site in a popup for a brief second only to transfer a transferable stream in best case scenario, worst case it must transfer a MessagePort to the service worker. after that the popup is closed automatically.
This is the only time when we can't use postMessage hack to keep the service worker alive for a longer duration.

MattiasBuelens reproduced the double transfer problem quite well before in a minimal working example here: whatwg/streams#276 (comment)

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Re-transferring streams</title>
</head>
<body>
<script id="worker-script">
    if (self.importScripts) {
        self.onmessage = (ev) => {
            const rs = ev.data;
            self.postMessage(rs, [rs]);
        };
    }
</script>
<script>
    var rs = new ReadableStream({
        start(c) {
            c.enqueue('a');
            c.enqueue('b');
            c.enqueue('c');
        }
    });
    var workerCode = document.getElementById('worker-script').innerHTML;
    var worker = new Worker(URL.createObjectURL(new Blob([workerCode], { type: 'application/javascript' })));
    worker.onmessage = async (ev) => {
        var transferred = ev.data;
        worker.terminate(); // comment this line to "fix" the pipe
        transferred.pipeTo(new WritableStream({
            write(x) {
                console.log(x);
            },
            close() {
                console.log('closed');
            }
        }));
    };
    worker.postMessage(rs, [rs]);
</script>
</body>
</html>

• Only in this example Mattias uses web workers instead of a popup and a service worker
• The different is that the service worker can live on for a longer duration after killing the popup

which leads to this chain:

original ReadableStream <--> MessageChannel <--> worker's ReadableStream <--> MessageChannel <--> re-transferred ReadableStream

I have gotten around this problem by sending the message channel using something like i demonstrated above with MessageChannel here: #13 (comment) in order to get a direct pipeline from point A to B and getting a shorter chain

i go a great length to get this StreamSaver to work in both secure and insecure pages and not having to force the developer to have to install a service worker them self.

That is why StreamSaver uses such a ugly hack of getting things to work. I wanted things to be as easy as just installing a dependency on StreamSaver and expect it to work with a simple import from npm or a cdn

• I use transferable streams when it is possible so i don't have to keep sending messages to keep the service worker alive...
• if it's not possible then i must keep send messages to keep it alive inside of a hidden iframe that is never removed. so it never breaks the chain from main thread <-> iframe <-> service worker
• but if the site is insecure then i can't install the service worker and communicate with it in any possible way
  • so that is why I have to fallback to using a popup to get around the secure mixing warning and be able to successfully install a service worker
  • it also tries to transfer a transferable ReadableStream and falling back to using a message channel - except that this popup is never open all the time, it's just a blank page that installs a service worker and forwards the MessagePort onwards and closed directly. when i closed this page that is when i stumble upon the double transfer problem.

---

StreamSaver never installs a service worker on the developers domain and interferes with with existing service worker that they might have set up already. that's why i have a so called "man in the middle" iframe/page that makes it easy to use StreamSaver from insecure pages and sites like jsfiddle where you can't install a service worker or don't want to have to go to the trouble of having to install a service worker yourself.

[guest271314]

but if the site is insecure then i can't install the service worker and communicate with it in any possible way

Well, yes, you can. The most secure route if for the user to run code on their own machine, which is possible with Native Messaging.

Can we please stop talking about the double transfer problem now? it feels a bit of topic

I'm just trying to provide a perspective from outside looking in. A ServiceWorker is not necessary at all to achieve downloading files locally.

Good luck.

[jimmywarting]

Native Messaging isn't really a good option as it requires installing a extension and using native application and don't work in all browsers

I'm just trying to provide a perspective from outside looking in. A ServiceWorker is not necessary at all to achieve downloading files locally.

Yup,

• you can use the new File System Access (only available in secure context, and newest Blink engines)
• and the <a href="blob:url" download="filename"> but this is what we are trying to avoid...

[guest271314]

Native Messaging isn't really a good option as it requires installing a extension and using native application and don't work in all browsers

Native Messaging works in Firefox and Chrome.

The user installs the extension locally. On Chromium simply select "Developer mode" at chrome://extensions then click "Load unpacked". No external resources or requests or Chrome Web Store are required.

Then you can use the same loading of <iframe> approach with a local file listed in "web_accessible_resources" and postMessage() to the parent, without using ServiceWorker (or CDN) at all. If you do want to use ServiceWorker for some reason, e.g., for user to click on the UI and commence download, you can still do that.

That allows you to use ServiceWorker on any origin, securely, without necessarily being on an HTTPS page online.

If you are talking about insecure pages, and requesting CDN's, all of those concerns go away when you are only running code already on your own machine.

Alternatively, use the Native Messaging host to download directly to the users' local file system.

I already filed a PR for this capability. You appeared to think unpacking a local extension involves external code or requests, it does not, and is at least as secure as the MITM code and requesting CDN's you are already using.

[jimmywarting]

The user installs the extension locally. On Chromium simply select "Developer mode" at chrome://extensions then click "Load unpacked". No external resources or requests or Chrome Web Store are required.

No visitor on your web site is ever going to do this... and you shouldn't force the user do this either

[guest271314]

I think you misunderstand what I am conveying. I am not talking about visiting websites, or StreamSaver used by websites. I am talking about the end user that wants to use your StreamSaver code on their own machine. Native Messaging is useful for me and other users to, for example, stream speech synthesis engine output from local speech synthesis engine as a ReadableStream piped to MediaStreamTrackGenerator, or stream live system audio output to the user on any origin, which Chrome prevents both of the former by not outputting audio of speechSynthesis.speak() on the tab, and not capturing monitor devices on *nix OS's.

From my perspetive StreamSaver concept is based on streaming download of files on any browser. So you can basically do the opposite of what I do here https://github.com/guest271314/captureSystemAudio/tree/master/native_messaging/capture_system_audio (using postMessage(transfer, [transfer]) for Firefox)

onload = () => {
  const { readable, writable } = new TransformStream();
  const writer = writable.getWriter();
  const id = 'capture_system_audio';
  let port = chrome.runtime.connectNative(id);
  let handleMessage = async (value) => {
    try {
      if (writable.locked) {
        await writer.ready;
        await writer.write(new Uint8Array(JSON.parse(value)));
      }
    } catch (e) {
      // handle cannot write to a closing stream 
      console.warn(e.message);
    }
  };
  port.onDisconnect.addListener(async (e) => {
    console.warn(e.message);
  });
  port.onMessage.addListener(handleMessage);
  onmessage = async (e) => {
    const { type, message } = e.data;
    if (type === 'start') {
      port.postMessage({
        message,
      });
      parent.postMessage(readable, name, [readable]);
    }
    if (type === 'stop') {
      try {
        await writer.close();
        await writer.closed;
        console.log('Writer closed.');
        writer.releaseLock();
        port.onMessage.removeListener(handleMessage);
        port.disconnect(id);
        port = null;
        parent.postMessage(0, name);
        onmessage = null;
        await chrome.storage.local.clear();
      } catch (err) {
        console.warn(err.message);
      }
    }
  };
  parent.postMessage(1, name);
};

• stream from the browser to a local file, with option to save or delete the file, stream to native messaging host locally on any origin while browsing the web, or offline. If you want to cancel the download you can do so, both in the browser and at the built-in level.

Native Messaging allows you to deploy your concept locally, to your precise specification, without reliance on any extrnal code - for end users.

Perhaps I misunderstand and your code is written primarily for web site developers exlusively

StreamSaver.js is the solution to saving streams on the client-side. It is perfect for webapps

not individual users that want to run your gear locally on any site they happen to be visting.

Again, good luck.