Error fetching and displaying characters from other languages about axios-fetch

[mdlavin]

It sounds like something about the way that createUrqlClient is calling the fetch method is triggering the problem. If I had to guess, I'd say it's something either with the headers provided, or with the way that createUrqlClient is handling the response body. Since you have a reproducible case, can you try the next level of debugging to find an axios-fetch only testcase?

[lukadriel7]

Hey @mdlavin ,
Just did an axios-fetch only test case, same error

const axios = inject<AxiosInstance>('axios');
const myFetch = buildAxiosFetch(axios);
const data = {
  query: `
  query ingredients(
  $cursor: IngredientWhereUniqueInput
  $distinct: [IngredientScalarFieldEnum!]
  $orderBy: [IngredientOrderByWithRelationAndSearchRelevanceInput!]
  $skip: Int
  $take: Int
  $where: IngredientWhereInput
) {
  ingredients(
    cursor: $cursor
    distinct: $distinct
    orderBy: $orderBy
    skip: $skip
    take: $take
    where: $where
  ) {
    id
    name
  }
  ingredientsCount(
    cursor: $cursor
    distinct: $distinct
    orderBy: $orderBy
    where: $where
  )
}
  `
}
const endpoint = process.env.API_URL + '/graphql';
const headers = {
	'content-type': 'application/json'
};
const graphqlQuery = {
    'operationName': 'ingredients',
    'query': data.query,
    'variables': {}
};

const options = {
    'method': 'POST',
    'headers': headers,
    'body': JSON.stringify(graphqlQuery)
};

const response = await myFetch(endpoint, options);
const myFetchResult = await response.json();

[mdlavin]

Thanks for following up. Is it possible to include the headers, and ideally the exact body, from the response? Since the failure has to do with decoding non-ascii characters, it seems related to headers to body encoding.

[lukadriel7]

Headers:

HTTP/1.1 200 OK
X-Powered-By: Express
Access-Control-Allow-Origin: http://localhost:9000
Vary: Origin
Access-Control-Allow-Credentials: true
Content-Type: application/json; charset=utf-8
Content-Length: 120
ETag: W/"78-AYBikrrtlHMhCBQZAFVfBdXEm00"
Date: Tue, 20 Sep 2022 21:30:52 GMT
Connection: keep-alive
Keep-Alive: timeout=5

Body:

{"data":{"ingredients":[{"id":"cl7l1blyy0080q0imzc0t4z4e","name":{"fr":"Blé","en-US":"Wheat"}}],"ingredientsCount":1}}

The "Blé" appear as "Blé" on the page

[mdlavin]

I tried to reproduce this many times in a testcase, both with utf-8 and with other character encodings. I can't reproduce a difference between Axios and standard fetch. The only thing I can guess is that something about the bytes in the body that's causing trouble. When I test with your body exactly, it renders correctly for me.

Are you reproducing this in a browser or on a server? Either way, can you provide the exact details like Browser version or Node version

[lukadriel7]

I use latest versions of chrome and firefox to access the page.
I also use a nestjs version 9.1.1 and A postgres database.
Node version is 16.16.0

[mdlavin]

Wow, you've been so helpful. Can you do me another favor to help debug? Can you:

1. Change the server to send back a response with 25 é
2. Can you send the full headers and response body for that response?

I have a new theory about what might be happening

[lukadriel7]

Response Header

HTTP/1.1 200 OK
X-Powered-By: Express
Access-Control-Allow-Origin: http://localhost:9000
Vary: Origin
Access-Control-Allow-Credentials: true
Content-Type: application/json; charset=utf-8
Content-Length: 166
ETag: W/"a6-bnb4urLrClKuVnqkVNu3NQVR/gU"
Date: Fri, 23 Sep 2022 18:30:48 GMT
Connection: keep-alive
Keep-Alive: timeout=5

Response Body

{"data":{"ingredients":[{"id":"cl7l1blyy0080q0imzc0t4z4e","name":{"fr":"ééééééééééééééééééééééééé","en-US":"Wheat"}}],"ingredientsCount":1}}

[mdlavin]

I really want to figure this bug out. I ended up creating a Next.JS app to mock the server and browser side of things. When I test it, it all works ok for me. Here is my reproduction - https://github.com/mdlavin/axios-fetch-issue-118

Can you try that and suggest a change to demonstrate the problem?

[lukadriel7]

Hey, sorry it took so long.
I created a repo containing a server, a working client and a client with the problem I described here.
I don't know how to reproduce it but you can check it here https://github.com/lukadriel7/encoding-axios-fetch

[mdlavin]

Thanks for putting your repo up. I reproduced it and then debugged the problem. Here is a fix for your problem (well, it avoids the issue on modern browsers). The problem is that the polyfill from cross-fetch is not handling utf-8 encoded characters correctly when the body of the request is an arraybuffer. It appears that the bug is here. That code is not treating the arraybuffer as utf-8 encoded like it does if a Blob is used for the body. Because removing the polyfill fixes the issue, I'm 95% that the polyfill is implemented incorrectly.

[lukadriel7]

Thank you very much