`sec-ch-ua` header is weirdly inconsistent about fingerprint-suite

[mvolfik]

Also, this header quite reliably causes blocking by Zillow (PerimeterX). This code:

import { gotScraping } from 'got-scraping';

console.log(
    await gotScraping({
        headers: { 'Content-Type': 'application/json', 'sec-ch-ua': '' },
        body: '{"searchQueryState":{"isMapVisible":true,"filterState":{"sortSelection":{"value":"globalrelevanceex"},"isAllHomes":{"value":true}},"mapBounds":{"north":36.18115,"east":-86.666132,"south":35.807142,"west":-86.891423},"isListVisible":true,"mapZoom":12,"regionSelection":[{"regionId":72192,"regionType":7}],"pagination":{}},"wants":{"cat1":["listResults","mapResults"],"cat2":["total"]},"requestId":1,"isDebugRequest":false}',
        url: 'https://www.zillow.com/async-create-search-page-state',
        method: 'PUT',
    }),
);

quite reliably works, but if you remove the forced sec-ch-ua="", you get almost always blocked

[mvolfik]

huh, the code above doesn't seem to work either anymore, is PerimeterX learning our fingerprints? (for context, I pushed a fix for ZIP Search scraper that used the fix above, started a run of ~3000 ZIP codes, which was producing results for a while, but now everything is 403 again)

this still works though:

curl -i -X PUT "https://www.zillow.com/async-create-search-page-state" \
  --data '{"searchQueryState":{"isMapVisible":true,"filterState":{"sortSelection":{"value":"globalrelevanceex"},"isAllHomes":{"value":true}},"mapBounds":{"north":36.18115,"east":-86.666132,"south":35.807142,"west":-86.891423},"isListVisible":true,"mapZoom":12,"regionSelection":[{"regionId":72192,"regionType":7}],"pagination":{}},"wants":{"cat1":["listResults","mapResults"],"cat2":["total"]},"requestId":1,"isDebugRequest":false}' \
  -H "User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:122.0) Gecko/20100102 Firefox/122.0" \
  -H "Accept: */*" \
  -H "Content-Type: application/json"

[B4nan]

Looks like the value is supposed to be random:

https://stackoverflow.com/questions/64413275/why-does-chrome-use-sec-ch-ua-not-abrandv-99

Maybe we should really just strip that from our data (or maybe just the generated fingerprints).

[barjin]

It seems to me that this issue has forked into two different things:

• The "weird" user agent client hint: This is expected behavior (see Martin's linked post, but also e.g. this draft here). The user agent string is purposefully random to make servers not depend on the actual value of the ua string too closely. This originally stems from the GREASE principle in TLS.

  • I'm still standing on my hill - we shouldn't manipulate this, as that would differentiate us from the actual browsers.

• Zillow blocking got-scraping: it blocked my Chrome (and curl-impersonate impersonating Chrome, too). Passed with Firefox (and curl-impersonate impersonating Firefox, too). Blocked got-scraping with (probably) Chrome fingerprint. Passed with @mvolfik 's curl with Firefox UA.

Guess what happens when I run got-scraping with this config:

await gotScraping({
    headers: { 'Content-Type': 'application/json' },
    body: '{"searchQueryState":{"isMapVisible":true,"filterState":{"sortSelection":{"value":"globalrelevanceex"},"isAllHomes":{"value":true}},"mapBounds":{"north":36.18115,"east":-86.666132,"south":35.807142,"west":-86.891423},"isListVisible":true,"mapZoom":12,"regionSelection":[{"regionId":72192,"regionType":7}],"pagination":{}},"wants":{"cat1":["listResults","mapResults"],"cat2":["total"]},"requestId":1,"isDebugRequest":false}',
    url: 'https://www.zillow.com/async-create-search-page-state',
    method: 'PUT',
    headerGeneratorOptions: {
        browsers: ['firefox'],
        devices: ['mobile'], // originally not needed, but I had better probability of not getting blocked with this
    }
})

If I had to guess the root of this, my bet would be on a skewed prior distribution between Chrome and Firefox fingerprints in the PerimeterX database - they just have more Chrome examples, so they can be more specific with the checks. And there, we're still losing with the non-genuine TLS stack etc., see my message from the other thread below:

---

TLDR: I understand that defeating the antiblocking scripts can frustrating, but looking into an automatically generated JSON file and trying to point out "weird" looking data is not the way forward. Creating confirmation bias without testing things properly is not the way forward either.

Let's be systematic about this, base our decisions on actual specifications and data, experiment, take notes, and see what works best.