GitHub changelog links no longer found in certain situations about gem_updater

[MaximeD]

Got it, github changed their css (again) between the moment I tested, and the moment you did 😅

This is fixed now. Let me know if you encounter other issues.

[mattmenefee]

@MaximeD thank you so much for all of these changes!! The gem is working great now!

And regarding the anchor links, it seems like that's just going to be an unfortunate, unavoidable situation, but at least it will be an easy fix if/when they do change their markup.

[mattmenefee]

I think I have a better understanding of why it intermittently fails. As best that I can tell, the new markup that I suggested above only appears to be applicable when the changelog is not listed in the top list of files/folders and is instead nested within a folder, such as for actioncable.

Other GitHub changelogs listed at the top level, such as rubocop-rails, appear to still work as expected.

[MaximeD]

Hello, thank you for reporting this.

I think the issue is that the page is not rendered yet when nokogiri parses it:

irb(#<GemUpdater::SourcePageParser::GitHubParser:0x00007fc54f70bf90>):011:0> doc
=>
#(Document:0x4a1c8 {
  name = "document",
  children = [
    #(DTD:0x4a2f4 { name = "html" }),
    #(Element:0x4a3f8 {
      name = "html",
      children = [
        #(Element:0x4a524 {
          name = "body",
          children = [
            #(Element:0x4a650 {
              name = "p",
              children = [
                #(Text "{\"payload\":{\"allShortcutsEnabled\":false,\"path\":\"actioncable\",\"repo\":{\"id\":8514,\"defaultBranch\":\"main\",\"name\":\"rails\",\"ownerLogin\":\"rails\",\"currentUserCanPush\":false,\"isFork\":false,\"isEmpty\":false,\"createdAt\":\"2008-04-11T02:19:47.000Z\",\"ownerAvatar\":\"https://avatars.githubusercontent.com/u/4223?v=4\",\"public\":true,\"private\":false,\"isOrgOwned\":true},\"currentUser\":null,\"refInfo\":{\"name\":\"v7.0.8\",\"listCacheKey\":\"v0:1694447045.0\",\"canEdit\":false,\"refType\":\"tag\",\"currentOid\":\"fc734f28e65ef8829a1a939ee6702c1f349a1d5a\"},\"tree\":{\"items\":[{\"name\":\"app\",\"path\":\"actioncable/app\",\"contentType\":\"directory\"},{\"name\":\"bin\",\"path\":\"actioncable/bin\",\"contentType\":\"directory\"},{\"name\":\"lib\",\"path\":\"actioncable/lib\",\"contentType\":\"directory\"},{\"name\":\"test\",\"path\":\"actioncable/test\",\"contentType\":\"directory\"},{\"name\":\".babelrc\",\"path\":\"actioncable/.babelrc\",\"contentType\":\"file\"},{\"name\":\".eslintrc\",\"path\":\"actioncable/.eslintrc\",\"contentType\":\"file\"},{\"name\":\".gitignore\",\"path\":\"actioncable/.gitignore\",\"contentType\":\"file\"},{\"name\":\"CHANGELOG.md\",\"path\":\"actioncable/CHANGELOG.md\",\"contentType\":\"file\"},{\"name\":\"MIT-LICENSE\",\"path\":\"actioncable/MIT-LICENSE\",\"contentType\":\"file\"},{\"name\":\"README.md\",\"path\":\"actioncable/README.md\",\"contentType\":\"file\"},{\"name\":\"Rakefile\",\"path\":\"actioncable/Rakefile\",\"contentType\":\"file\"},{\"name\":\"actioncable.gemspec\",\"path\":\"actioncable/actioncable.gemspec\",\"contentType\":\"file\"},{\"name\":\"karma.conf.js\",\"path\":\"actioncable/karma.conf.js\",\"contentType\":\"file\"},{\"name\":\"package.json\",\"path\":\"actioncable/package.json\",\"contentType\":\"file\"},{\"name\":\"rollup.config.js\",\"path\":\"actioncable/rollup.config.js\",\"contentType\":\"file\"},{\"name\":\"rollup.config.test.js\",\"path\":\"actioncable/rollup.config.test.js\",\"contentType\":\"file\"}],\"templateDirectorySuggestionUrl\":null,\"readme\":{\"displayName\":\"README.md\",\"richText\":\""),
…

Looking at the last line, we can see the changelog entry is there. The solution may be to wait for the page to be loaded before trying to scrape its content. Something like watir could help.

[mattmenefee]

@MaximeD you're welcome! And yea, I agree, not waiting until the page is fully loaded seems like one of the problems.

However it also seems like GitHub uses different markup for when the changelog isn't in the root of the directory (identified by aria-labelledby="folders-and-files") versus when it is within the top level directory aria-labelledby="files"). For example, searching for a nested changelog for something like the Rails gems (actioncable, etc.) would look for:

<table aria-labelledby="folders-and-files">

versus when searching for a non-nested changelog such as for RuboCop, it would search within:

<div role="grid" aria-labelledby="files">

for the relevant changelog file to parse.

[MaximeD]

I may have found an alternative: rubygems’ metada allows to set the changelog uri. I don’t know how widely used it is, but at least actioncable and other popular gems like puma or rubocop are using it.

Then getting the changelog url from the metadata is just a matter of:

require 'gems'

Gems.info("actioncable")["changelog_uri"]

In the end I believe it would be better to use this rather than the whole logic from GemUpdater::SourcePageParser, which implies a bunch of rewrite, but that would be for the best.

[MaximeD]

Hello @mattmenefee, I changed how this gem looks for the changelog in #171.

Do you mind giving it a test? That will fix the problem with actioncable.

Let me know what you think.

[mattmenefee]

@MaximeD Just tested it and it didn't find the changelogs for any of the Rails gems. Let me dig deeper into this tomorrow though to make sure that it's not something on my end.

[MaximeD]

Hum strange, here is the output for me on some personal old project, with a rails upgrade:

Here are your changes:
------------------------
* actioncable 7.0.3.1 → 7.0.8
[changelog](https://github.com/rails/rails/blob/v7.0.8/actioncable/CHANGELOG.md#rails-708-september-09-2023)

* actionmailbox 7.0.3.1 → 7.0.8
[changelog](https://github.com/rails/rails/blob/v7.0.8/actionmailbox/CHANGELOG.md#rails-708-september-09-2023)

* actionmailer 7.0.3.1 → 7.0.8
[changelog](https://github.com/rails/rails/blob/v7.0.8/actionmailer/CHANGELOG.md#rails-708-september-09-2023)

* actionpack 7.0.3.1 → 7.0.8
[changelog](https://github.com/rails/rails/blob/v7.0.8/actionpack/CHANGELOG.md#rails-708-september-09-2023)

* actiontext 7.0.3.1 → 7.0.8
[changelog](https://github.com/rails/rails/blob/v7.0.8/actiontext/CHANGELOG.md#rails-708-september-09-2023)

* actionview 7.0.3.1 → 7.0.8
[changelog](https://github.com/rails/rails/blob/v7.0.8/actionview/CHANGELOG.md#rails-708-september-09-2023)

* activejob 7.0.3.1 → 7.0.8
[changelog](https://github.com/rails/rails/blob/v7.0.8/activejob/CHANGELOG.md#rails-708-september-09-2023)

* activemodel 7.0.3.1 → 7.0.8
[changelog](https://github.com/rails/rails/blob/v7.0.8/activemodel/CHANGELOG.md#rails-708-september-09-2023)

* activerecord 7.0.3.1 → 7.0.8
[changelog](https://github.com/rails/rails/blob/v7.0.8/activerecord/CHANGELOG.md#rails-708-september-09-2023)

* activestorage 7.0.3.1 → 7.0.8
[changelog](https://github.com/rails/rails/blob/v7.0.8/activestorage/CHANGELOG.md#rails-708-september-09-2023)

* activesupport 7.0.3.1 → 7.0.8
[changelog](https://github.com/rails/rails/blob/v7.0.8/activesupport/CHANGELOG.md#rails-708-september-09-2023)

* addressable 2.8.1 → 2.8.5
[changelog](https://github.com/sporkmonger/addressable/blob/main/CHANGELOG.md#addressable-285)

* autoprefixer-rails 10.4.7.0 → 10.4.15.0
[changelog](https://github.com/ai/autoprefixer-rails/blob/master/CHANGELOG.md#104150)

* bcrypt 3.1.18 → 3.1.19

* bindata 2.4.13 → 2.4.15

* bootsnap 1.13.0 → 1.16.0
[changelog](https://github.com/Shopify/bootsnap/blob/main/CHANGELOG.md#1160)

* bugsnag 6.24.2 → 6.26.0
[changelog](https://github.com/bugsnag/bugsnag-ruby/blob/HEAD/CHANGELOG.md#v6260-19-july-2023)

* cancancan 3.4.0 → 3.5.0

* capybara 3.38.0 → 3.40.0

* concurrent-ruby 1.1.10 → 1.2.2
[changelog](https://github.com/ruby-concurrency/concurrent-ruby/blob/master/CHANGELOG.md#release-v122-24-feb-2023)

* database_cleaner 2.0.1 → 2.0.2
[changelog](https://github.com/DatabaseCleaner/database_cleaner/blob/master/History.rdoc#201-2021-02-04-)

* database_cleaner-active_record 2.0.1 → 2.1.0

* erubi 1.11.0 → 1.12.0
[changelog](https://github.com/jeremyevans/erubi/blob/master/CHANGELOG)

* execjs 2.8.1 → 2.9.1

* faraday 2.6.0 → 2.7.11
[changelog](https://github.com/lostisland/faraday/releases/tag/v2.7.11)

* faraday-net_http 3.0.1 → 3.0.2
[changelog](https://github.com/lostisland/faraday-net_http/releases/tag/v3.0.2)

* forest_liana 7.7.1 → 8.0.12

* geocoder 1.8.1 → 1.8.2
[changelog](https://github.com/alexreisner/geocoder/blob/master/CHANGELOG.md#182-2023-jul--4)

* globalid 1.0.0 → 1.2.1

* groupdate 6.1.0 → 6.4.0

* httparty 0.20.0 → 0.21.0

* i18n 1.12.0 → 1.14.1
[changelog](https://github.com/ruby-i18n/i18n/releases)

* json 2.6.2 → 2.6.3
[changelog](https://github.com/flori/json/blob/master/CHANGES.md)

* json-schema 3.0.0 → 4.1.1

* jwt 2.5.0 → 2.7.1
[changelog](https://github.com/jwt/ruby-jwt/blob/v2.7.1/CHANGELOG.md#v271-2023-06-09)

* launchy 2.5.0 → 2.5.2
[changelog](https://github.com/copiousfreetime/launchy/blob/master/README.md)

* listen 3.7.1 → 3.8.0
[changelog](https://github.com/guard/listen/releases)

* loofah 2.19.0 → 2.21.3
[changelog](https://github.com/flavorjones/loofah/blob/main/CHANGELOG.md#2213--2023-05-15)

* mail 2.7.1 → 2.8.1

* mini_mime 1.1.2 → 1.1.5

* mini_portile2 2.8.0 → 2.8.4

* minitest 5.16.3 → 5.20.0

* msgpack 1.6.0 → 1.7.2

* net-imap 0.3.1 → 0.3.7

* net-protocol 0.1.3 → 0.2.1

* net-smtp 0.3.2 → 0.3.3

* nio4r 2.5.8 → 2.5.9
[changelog](https://github.com/socketry/nio4r/blob/master/CHANGES.md)

* nokogiri 1.13.9 → 1.15.4
[changelog](https://nokogiri.org/CHANGELOG.html)

* oauth 0.6.2 → 1.1.0
[changelog](https://github.com/oauth-xx/oauth-ruby/blob/v1.1.0/CHANGELOG.md#110-2022-08-29)

* oj 3.13.21 → 3.16.1
[changelog](https://github.com/ohler55/oj/blob/master/CHANGELOG.md#3161---2023-09-01)

* panko_serializer 0.7.7 → 0.8.1
[changelog](https://github.com/yosiat/panko_serializer/releases)

* parallel 1.22.1 → 1.23.0

* parser 3.1.2.1 → 3.2.2.3
[changelog](https://github.com/whitequark/parser/blob/v3.2.2.3/CHANGELOG.md)

* pg 1.4.4 → 1.5.4
[changelog](https://github.com/ged/ruby-pg/blob/master/History.md#v154-2023-09-01-lars-kanis-larsgreiz-reinsdorfde)

* pry 0.14.1 → 0.14.2
[changelog](https://github.com/pry/pry/blob/master/CHANGELOG.md#v0142v0142-january-9-2023)

* public_suffix 5.0.0 → 5.0.3
[changelog](https://github.com/weppos/publicsuffix-ruby/blob/master/CHANGELOG.md#503)

* puma 6.0.0 → 6.3.1
[changelog](https://github.com/puma/puma/blob/master/History.md#631--2023-08-18)

* racc 1.6.0 → 1.7.1

* rack 2.2.4 → 2.2.8
[changelog](https://github.com/rack/rack/blob/main/CHANGELOG.md)

* rack-cors 1.1.1 → 2.0.1

* rack-proxy 0.7.4 → 0.7.7

* rack-test 2.0.2 → 2.1.0
[changelog](https://github.com/rack/rack-test/blob/main/History.md#210--2023-03-14)

* rails 7.0.3.1 → 7.0.8
[changelog](https://github.com/rails/rails/releases/tag/v7.0.8)

* rails-dom-testing 2.0.3 → 2.2.0

* rails-html-sanitizer 1.4.3 → 1.6.0
[changelog](https://github.com/rails/rails-html-sanitizer/blob/v1.6.0/CHANGELOG.md#160--2023-05-26)

* railties 7.0.3.1 → 7.0.8
[changelog](https://github.com/rails/rails/blob/v7.0.8/railties/CHANGELOG.md#rails-708-september-09-2023)

* regexp_parser 2.6.0 → 2.8.1
[changelog](https://github.com/ammar/regexp_parser/blob/master/CHANGELOG.md#281---2023-06-10---janosch-müller)

* rexml 3.2.5 → 3.2.6

* rspec-cells 0.3.7 → 0.3.8
[changelog](https://github.com/trailblazer/rspec-cells/blob/master/CHANGES.md#038)

* rspec-core 3.11.0 → 3.12.2
[changelog](https://github.com/rspec/rspec-core/blob/v3.12.2/Changelog.md#3122--2023-04-18)

* rspec-expectations 3.11.1 → 3.12.3
[changelog](https://github.com/rspec/rspec-expectations/blob/v3.12.3/Changelog.md#3123--2023-04-20)

* rspec-mocks 3.11.1 → 3.12.6
[changelog](https://github.com/rspec/rspec-mocks/blob/v3.12.6/Changelog.md#3126--2023-07-11)

* rspec-rails 5.1.2 → 6.0.3
[changelog](https://github.com/rspec/rspec-rails/blob/v6.0.3/Changelog.md#603--2023-05-31)

* rspec-support 3.11.1 → 3.12.1
[changelog](https://github.com/rspec/rspec-support/blob/v3.12.1/Changelog.md#3121--2023-06-26)

* rubocop 1.37.0 → 1.56.3
[changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md)

* rubocop-ast 1.23.0 → 1.29.0
[changelog](https://github.com/rubocop/rubocop-ast/blob/master/CHANGELOG.md#1290-2023-06-01)

* rubocop-performance 1.15.0 → 1.19.1
[changelog](https://github.com/rubocop/rubocop-performance/blob/master/CHANGELOG.md#1191-2023-09-17)

* rubocop-rails 2.17.0 → 2.21.1
[changelog](https://github.com/rubocop/rubocop-rails/blob/master/CHANGELOG.md#2211-2023-09-14)

* ruby-progressbar 1.11.0 → 1.13.0
[changelog](https://github.com/jfelchner/ruby-progressbar/blob/master/CHANGELOG.md#version-v1130---march-4-2023)

* selenium-webdriver 4.5.0 → 4.12.0
[changelog](https://github.com/SeleniumHQ/selenium/blob/trunk/rb/CHANGES)

* simplecov 0.21.2 → 0.22.0
[changelog](https://github.com/simplecov-ruby/simplecov/blob/main/CHANGELOG.md#0220-2022-12-23)

* skylight 5.3.4 → 6.0.0

* slim 4.1.0 → 5.1.1

* slim-rails 3.5.1 → 3.6.2

* sorcery 0.16.4 → 0.16.5

* sprockets 4.1.1 → 4.2.1

* temple 0.8.2 → 0.10.2

* thor 1.2.1 → 1.2.2
[changelog](https://github.com/rails/thor/releases/tag/v1.2.2)

* tilt 2.0.11 → 2.3.0
[changelog](https://github.com/jeremyevans/tilt/blob/master/CHANGELOG.md#230-2023-09-14)

* timeout 0.3.0 → 0.4.0

* tzinfo 2.0.5 → 2.0.6
[changelog](https://github.com/tzinfo/tzinfo/blob/master/CHANGES.md#version-206---28-jan-2023)

* unicode-display_width 2.3.0 → 2.4.2
[changelog](https://github.com/janlelis/unicode-display_width/blob/main/CHANGELOG.md#242)

* version_gem 1.1.1 → 1.1.3
[changelog](https://gitlab.com/oauth-xx/version_gem/-/blob/v1.1.3/CHANGELOG.md)

* webpacker 5.4.3 → 5.4.4
[changelog](https://github.com/rails/webpacker/blob/v5.4.4/CHANGELOG.md#544---2023-01-31)

* websocket-driver 0.7.5 → 0.7.6

* yard 0.9.28 → 0.9.34

* zeitwerk 2.6.1 → 2.6.11
[changelog](https://github.com/fxn/zeitwerk/blob/master/CHANGELOG.md#2611-2-august-2023)

The rails gems are found, so I’m not sure what your issue could be 🤔

[mattmenefee]

@MaximeD Ahh, my mistake. I saw that you released a new version and mistakingly thought that the changes from #171 were included in it. How would I go about installing a prerelease version of the gem with those changes without using Bundler?

[MaximeD]

Aah now that explains it!

It order to test it without installing with bundler, you can clone the repository, checkout the branch use_changelog_uri and then run the script bin/gem_update from the repository you are testing.

So more or less this:

# Clone the repository in your home (for instance), and checkout the branch
> cd ~
> git clone [email protected]:MaximeD/gem_updater.git
> cd gem_updater/
> git checkout use_changelog_uri

# Change directory to the repository you want to test
> cd <the repository to update>

# Run gem_update executable
> ~/gem_updater/bin/gem_update

[MaximeD]

Hello @mattmenefee, did you have a chance to give it a try?

[mattmenefee]

@MaximeD Sorry! I was on holiday. Yes, just tried it and it works well for the Rails gems! However, now the gems that don't have a changelog listed in Rubygems do not work, such as strong_migrations. But hopefully those will be in the minority? Oh and the anchor links are no longer present since you're not parsing the changelog anymore. What do you think about still grabbing the anchor links?

[MaximeD]

Thank you for your answer!

Regarding the gems that don’t list their changelog uri on rubygems, I believe this is the responsibility of gems owners to declare it. Hopefully, it is quite simple to add it to the gem’s metadata. I opened a MR on strong_migrations to add the changelog uri.

Regarding the anchor links, the code to do this is still there. Do you have an example of gems for which is doesn’t work anymore?