Hashtag ordering kinda sucks about mastodon

[neatchee]

I think this is an easy fix; it looks like a change was made in 2019 that changed the sort order, but then it was changed back, and the field name to sort on was changed, and it was never updated again to sort with the new field name. Testing a fix now...

[neatchee]

Well, this is rather complicated...

The sorting function is set up to sort from values in the tags sql table, which does not store the actual occurrence count. The "uses this week" value is an aggregate value generated at runtime from recent tag history. Sorting was set up to use a "score" which is built over time, and is now recorded as "max_score"

So we could easily redo the sorting to use the max_score but that does NOT produce a sorted list by "uses this week"; for example "mastodogs" appears above "mastodon" because at some point mastodogs got super popular and bursted above the average usage of 'mastodon'.

I've got a version deployed that feels a bit better but still isn't great. It'd be a substantive refactor to actually sort by "uses this week" unfortunately.

[neatchee]

This is the tags table schema:

---------------------+-----------------------------
 id                  | bigint
 name                | character varying
 created_at          | timestamp without time zone 
 updated_at          | timestamp without time zone 
 usable              | boolean 
 trendable           | boolean
 listable            | boolean
 reviewed_at         | timestamp without time zone
 requested_review_at | timestamp without time zone
 last_status_at      | timestamp without time zone
 max_score           | double precision
 max_score_at        | timestamp without time zone
 display_name        | character varying

And this is the current deployed sorting code for Urusai as of now:

def search_for(term, limit = 5, offset = 0, options = {})
    stripped_term = term.strip

    query = Tag.listable.matches_name(stripped_term)
    query = query.merge(matching_name(stripped_term).or(where.not(reviewed_at: nil))) if options[:exclude_unreviewed]

    query.order(Arel.sql('max_score ASC, name ASC'))
         .limit(limit)
         .offset(offset)
  end

Currently checked in upstream is:

query.order(Arel.sql(length(name) ASC, name ASC'))
         .limit(limit)
         .offset(offset)

[neatchee]

Well, the real answer here is to just do what's intended to get better autosuggestion/search: set up elasticsearch integration.

So that's what I'm doing :D

We can try to make this better too, if we want, but IMO it's low prio since there's a preferred solution

[deanveloper]

Think this is worth reopening, doesn't seem fixed, even with elasticsearch :/

[neatchee]

Yeah I don't know what the deal is here at this point. Making changes to that 'order by' snippet changes behavior but in an unpredictable way. It's bizarre

…

On Thu, Jan 12, 2023, 9:02 PM Dean Bassett ***@***.***> wrote: Think this is worth reopening, doesn't seem fixed :/ [image: image] <https://user-images.githubusercontent.com/3196327/212241381-d4e745c7-cb4f-4fde-9222-91602e0c591a.png> — Reply to this email directly, view it on GitHub <#35 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABA3WH2MJ2O7IUDOFNZZJYLWSDOXPANCNFSM6AAAAAATUX7LHY> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[deanveloper]

Oh just looked at the code snippet, you probably want to order by score DESC right? ASC would sort it least-to-most

[neatchee]

You would think so but like I said, the result is unpredictable. I've tried both ways, and with adding where conditions to filter them. I haven't been able to grok whatever is actually happening

…

On Thu, Jan 12, 2023, 9:13 PM Dean Bassett ***@***.***> wrote: Oh just looked at the code snippet, you probably want to sort by score DESC right? ASC would sort it least-to-most — Reply to this email directly, view it on GitHub <#35 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABA3WH3L6IR7TBGCQP3HKTDWSDQBJANCNFSM6AAAAAATUX7LHY> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[deanveloper]

Hmm... I'll see if I can mess around with it and see if I can figure something out. Seems weird because the current behavior seems correct on other hashtags, like this is exactly what I would expect:

[neatchee]

Like I said, absolutely bizarre. :D

…

On Thu, Jan 12, 2023, 10:01 PM Dean Bassett ***@***.***> wrote: Hmm... I'll see if I can mess around with it and see if I can figure something out. Seems weird because the current behavior seems correct on other hashtags, like this is exactly what I would expect: [image: image] <https://user-images.githubusercontent.com/3196327/212248404-e7d625b0-0c54-4193-b6d7-9c818e6240a5.png> [image: image] <https://user-images.githubusercontent.com/3196327/212248373-c34e750c-37f2-4326-aa7f-049dacf00cfd.png> — Reply to this email directly, view it on GitHub <#35 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABA3WH5LWT7CPCNOKXK6AADWSDVT3ANCNFSM6AAAAAATUX7LHY> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[deanveloper]

Hashtag ordering seems to work fine on Megalodon... I wonder if this is just some weird edge case in the browser

[neatchee]

Or megalodon is doing their own ordering

…

On Fri, Feb 10, 2023, 7:05 PM Dean Bassett ***@***.***> wrote: Hashtag ordering seems to work fine on Megalodon... I wonder if this is just some weird edge case in the browser — Reply to this email directly, view it on GitHub <#35 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABA3WH26NGZCB3HHTDDQHXDWW36XZANCNFSM6AAAAAATUX7LHY> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[deanveloper]

Nope, it looks like the frontend is doing something weird... The request seems to return the hashtags in the right order. Wonder what's happening.

However... when redux gets updated, looks like they're out of order. Weird...

[neatchee]

I'm assuming it's the drop-down component in react but I cannot fathom why it would do that

…

On Fri, Feb 10, 2023, 7:22 PM Dean Bassett ***@***.***> wrote: Nope, it looks like the frontend is doing something weird... The request seems to return the hashtags in the right order. Wonder what's happening. [image: image] <https://user-images.githubusercontent.com/3196327/218236496-b95dc60c-f86d-4a26-bf17-a3a6fe8ebf9c.png> — Reply to this email directly, view it on GitHub <#35 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABA3WH5R57PYY6D33CR6DKTWW4AWZANCNFSM6AAAAAATUX7LHY> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[deanveloper]

It looks like some kind of transformation gets applied before it gets put in redux, trying to figure out where this happens (which seems a bit weird, not like suggestions really need to be stored in global state...)

EDIT - Guessing it has something to do with this line. (app/javascript/flavours/glitch/reducers/compose.js)

return mergeLocalHashtagResults(sortHashtagsByUse(state, tags.map(item => ({ ...item, type: 'hashtag' }))), token.slice(1), state.get('tagHistory'));

[deanveloper]

This seems so weird... I can't seem to replicate the behavior in Node, even if I copy the behavior nearly line-for-line

Browser

Node

The browser one starts correctly, but then ends incorrectly, and I have no clue why

[neatchee]

.... This is going to be another JavaScript 'WAT' moment, I'm sure if it. Does this repro in all browsers?

…

On Fri, Feb 10, 2023, 9:31 PM Dean Bassett ***@***.***> wrote: This seems so weird... I can't seem to replicate the behavior in Node, even if I copy the behavior nearly line-for-line Browser [image: image] <https://user-images.githubusercontent.com/3196327/218242087-5485470a-f50d-467b-a248-aae763b6e49a.png> Node [image: image] <https://user-images.githubusercontent.com/3196327/218242073-ac36bc2f-33cb-44be-b3d9-ab79c00890b4.png> — Reply to this email directly, view it on GitHub <#35 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABA3WHZHQL6RUPWAGWMC3UDWW4P4RANCNFSM6AAAAAATUX7LHY> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[deanveloper]

Yup, reproduces in all browsers. Although I've got it reproducing in node now, so that's good! Maybe I was just copying it over incorrectly.

[deanveloper]

Oh man, I figured it out. I'll file this bug with mastodon as well.

It only occurs if:

1. You have used the hashtag in this browser, which puts it in your tagHistory.
2. The tag in your tagHistory matches (case sensitive) one of the suggested hashtags.

If both of these cases happen, it's supposed to promote it to the top, as it's a suggested hashtag which you have used before. However, the sorting algorithm is reversed, and case-sensitive, making it (inconsistently) demote it to the bottom.

[neatchee]

Nice catch! That's an epic find

…

On Fri, Feb 10, 2023 at 10:08 PM Dean Bassett ***@***.***> wrote: Oh man, I figured it out. I'll file this bug with mastodon as well. It only occurs if: 1. You have used the hashtag *in this browser*, which puts it in your tagHistory. 2. The tag in your tagHistory matches (case sensitive) one of the suggested hashtags. If both of these cases happen, it's *supposed* to promote it to the top, as it's a suggested hashtag which you have used before. However, the sorting algorithm is reversed, and case-sensitive, making it (inconsistently) demote it to the bottom. — Reply to this email directly, view it on GitHub <#35 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABA3WH7ZRKSRPDGTDSOG6FDWW4UGPANCNFSM6AAAAAATUX7LHY> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

-- Brian "Neatchee" Resnik

[deanveloper]

Ahh, the bug appears to have been introduced in glitch-soc, I assume mastodon fixed it but glitch-soc never brought the fix over?

[neatchee]

Or just a bad merge

…

On Fri, Feb 10, 2023, 10:25 PM Dean Bassett ***@***.***> wrote: Ahh, the bug appears to have been introduced in glitch-soc, I assume mastodon fixed it but glitch-soc never brought the fix over? — Reply to this email directly, view it on GitHub <#35 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABA3WH5PCX3URFNALBDUWFDWW4WDZANCNFSM6AAAAAATUX7LHY> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>