At RTB House, we are aware that higher latency of ads in Fledge compared to Classic RTB system could have significant impact on final metrics and overall success of transition to Fledge. This topic was previously discussed in these issues: LINK#215 and LINK#385.
Today, due to the disabling of third-party cookies for a portion of Chrome browsers (Mode B testing), event-level reporting, and the fact that we managed to integrate our Fledge implementation with various SSPs, we are able to compare Fledge and Classic end-to-end bidding latency. In this issue, we would like to share our internal findings.
Scope:
- On-device only
- Bidding and Auction services is out of scope
- We're interested in various environments, various devices and various network types
We define end-to-end bidding latency as the time elapsed from the bid (start of bid request processing) to the impression (start of ad rendering). This means that we compare:
- Fledge: rendering request ts (server-side) - contextual bid request ts (server-side)
- Classic: rendering request ts (server-side) - classic RTB bid request ts (server-side)
We conduct measurements on the following segment of Internet traffic:
- Fledge: treatment traffic with third-party cookies disabled - which means bidding for users with treatment_1.X (X=1,2,3) labels from Mode B
- Classic: legacy traffic with third-party cookies enabled - which means bidding for users without Mode A and Mode B labels
Each of the following diagrams consists of:
- A histogram showing the distribution of latencies
- A table of latency percentiles (p50, p80, p90)
Comparison between Fledge (treatment traffic) and Classic (legacy traffic)
The dataset used to generate the diagram includes impressions (from both Fledge and Classic), meaning winning auctions that resulted in ad rendering, for various SSPs and ad slots.
![all-ssps](https://private-user-images.githubusercontent.com/5963708/340353816-e487fc00-ba6d-4716-933b-a491b457b7c9.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM2OTE0NzUsIm5iZiI6MTcyMzY5MTE3NSwicGF0aCI6Ii81OTYzNzA4LzM0MDM1MzgxNi1lNDg3ZmMwMC1iYTZkLTQ3MTYtOTMzYi1hNDkxYjQ1N2I3YzkucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgxNSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MTVUMDMwNjE1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9NDk1N2UxYWJhMTk3MDUyNWYxYzA5NzM3OWY5NmE1ZWQ2YTRmMzFhNWQ2NzE5ZjIzNjYzNGNlMjAzNWUwZDFlOCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.ae4hZnNZ_vP6sfOxJteGKtaYUsHzfUd_E5I--IlIs6A)
. |
fledge_imps.bid_to_rendering_time |
legacy_imps.bid_to_rendering_time |
p50 |
2999 ms |
993 ms |
p80 |
5453 ms |
2254 ms |
p90 |
6630 ms |
4140 ms |
Comparison between Fledge and Classic: split by device type
The same dataset, segmented by device type (PC and PHONE).
![all-ssps-pc](https://private-user-images.githubusercontent.com/5963708/340353914-f6e18d32-ecb2-4641-bf2b-8b46e1908409.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM2OTE0NzUsIm5iZiI6MTcyMzY5MTE3NSwicGF0aCI6Ii81OTYzNzA4LzM0MDM1MzkxNC1mNmUxOGQzMi1lY2IyLTQ2NDEtYmYyYi04YjQ2ZTE5MDg0MDkucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgxNSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MTVUMDMwNjE1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ZGM0Yzc2MTU0MWRlZmJkZDRiNWRiZGYxNTc3ZTdmNDM3MjEzZDdjMWQ2MTY0NGQwZmY4NDcwOThlYWE4NmQyOSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.REhjKGj_NHHJUQujVlereX-obqlHN8c35eaAWZd_9tg)
. |
fledge_imps.bid_to_rendering_time |
legacy_imps.bid_to_rendering_time |
p50 |
1919 ms |
688 ms |
p80 |
4721 ms |
1683 ms |
p90 |
5937 ms |
3375 ms |
![all-ssps-phone](https://private-user-images.githubusercontent.com/5963708/340354005-f2d7d59c-2964-46e5-a8d8-666858adaf3a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM2OTE0NzUsIm5iZiI6MTcyMzY5MTE3NSwicGF0aCI6Ii81OTYzNzA4LzM0MDM1NDAwNS1mMmQ3ZDU5Yy0yOTY0LTQ2ZTUtYThkOC02NjY4NThhZGFmM2EucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgxNSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MTVUMDMwNjE1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YWFmMzgxZjZmNWFmZjljN2I2MGQ0ZTQyZDU5YTJhODQyMDIwODg0YzFmMGE5Yzc4YWYyYjRhMmQwYzJmYTBjNiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.KSLf-xRmNn61VHAICnE1hg91QFUk8Iz0mR1bWao2B8o)
. |
fledge_imps.bid_to_rendering_time |
legacy_imps.bid_to_rendering_time |
p50 |
3833 ms |
1053 ms |
p80 |
5745 ms |
2336 ms |
p90 |
6954 ms |
4224 ms |
Comparison between Fledge-over-RTB and Prebid-over-RTB
In this part, we use results from our own tests. For Fledge-over-RTB and Prebid-over-RTB, we buy impressions via direct integration or Classic RTB on real publishers' pages. Then, depending on the test scenario, we perform one of two actions:
- run our own Fledge auction (Fledge-over-RTB)
- run a Classic auction using Prebid (Prebid-over-RTB)
These tests allow us to compare the latency of Fledge (Fledge-over-RTB) and Classic (Prebid-over-RTB) impressions independently from Fledge integration with SSPs and other buyers, as there are no other buyers participating in the auction in both scenarios.
![fledge-over-rtb](https://private-user-images.githubusercontent.com/5963708/340354087-d614248a-6f66-4c00-9aed-f863be69f23a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM2OTE0NzUsIm5iZiI6MTcyMzY5MTE3NSwicGF0aCI6Ii81OTYzNzA4LzM0MDM1NDA4Ny1kNjE0MjQ4YS02ZjY2LTRjMDAtOWFlZC1mODYzYmU2OWYyM2EucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgxNSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MTVUMDMwNjE1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ZWY5NGYzODQxNzFhYWI2MzM0MTczMWNhYmEyOThkYmMzNWQ2MTQxNTRlZTk0ZDBmNTFlODBlOThhMmQ0M2JjMiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.6KJhUaaD5vCIxkaIq96UJ_9kFRZwg9iptXgxB14s6X4)
. |
fledge_over_rtb.bid_to_rendering_time |
prebid_over_rtb.bid_to_rendering_time |
p50 |
1562 ms |
189 ms |
p80 |
2961 ms |
336 ms |
p90 |
4499 ms |
531 ms |
To eliminate the possibility that our implementation of bid logic in Fledge is the source of the problem, we repeated the Fledge-over-RTB experiment, this time completely removing the part responsible for model evaluation, both server-side (in contextual and TBS requests processing) and client-side (in the bidding function). In our case, this means reducing the size of contextual and TBS responses by over half, as well as decreasing server-side and client-side computations by over half. After such intervention, the histogram latency did not decrease.
![dummy-fledge-over-rtb](https://private-user-images.githubusercontent.com/5963708/340354185-c078ec83-638c-4060-ac67-b02c6fbbfcd0.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjM2OTE0NzUsIm5iZiI6MTcyMzY5MTE3NSwicGF0aCI6Ii81OTYzNzA4LzM0MDM1NDE4NS1jMDc4ZWM4My02MzhjLTQwNjAtYWM2Ny1iMDJjNmZiYmZjZDAucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgxNSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MTVUMDMwNjE1WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9NGRlY2VjYTY3NmJkNzVmMDcxMDkxNmQzOWUxYzJkZjgyYjEwNjk1ZWE4NDZjODczYTAxNTQ2MzAxMTY4ZDNlNCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.SEmTdb1mFEEIISw_Wx8mOdFMrp_aVj76VQlQdbxtHyI)
. |
fledge_over_rtb_dummy.bid_to_rendering_time |
fledge_over_rtb_regular.bid_to_rendering_time |
p50 |
1576 ms |
1562 ms |
p80 |
3056 ms |
2961 ms |
p90 |
4928 ms |
4499 ms |
Conclusions
To sum up our results, we can see that in the Classic, 50% of auctions last less than 1 second, while in the Fledge, which usually lasts three times longer, more than 50% of auctions for mode B users are taking over 3 seconds. The situation is even worse in the case of mobile devices because the difference in latency, when we limit it to auctions on phones, increases from 3x to 4x.
Results from our internal tests comparing Fledge-over-RTB and Prebid-over-RTB indicate the overhead on the Fledge stack. In a very simple setup, where we are the only buyer in the Classic system, 50% of actions take less than 200ms. In the case of Fledge with the same setup, these auctions could last up to 8 times longer. Additionally, the fact that after reducing computations and the size of contextual and TBS responses significantly, latency did not decrease, along with the observation (as we verified) that our bidding function is fetched from the cache in 95% of auctions, suggests that neither processing nor fetching the bidding function in this case is the bottleneck.
At RTB House, we still believe that migrating to the Protected Audience API is feasible without losing retargeting potential. Although remaining concern is the current on-device implementation, where resources dedicated to Fledge auctions are limited and shared by multiple buyers and SSPs altogether significantly impacting e2e latency. It is important to resolve such fundamental concerns before removing support for third-party cookies. Additionally, we believe that transitioning early to Bidding and Auction Services could be a solution although we couldn’t perform similar measures yet due insufficient traffic.