sasquire / idems-sourcing-suite Goto Github PK

Adding the md5 hasher to deviantart would be good.

Some sites give users a fake link to travel to before they leave the site itself. This will then infect the HTML when a user uses the copy description feature.

If there was a way to avoid these redirects and get the final url it should be done.

Really don't know whats going on with twitter. The whole thing should be redone honestly. After you click on an image and then close out, it will leave the image hash on the top of the screen.

The sourcing suite checks the DeviantArt images' large view and social preview MD5s as those are downloaded and cached upon page load. However, the MD5 checker does not consider a simple change that users can make to the DA preview image URL that allows them to get higher-quality versions of such images if the original image were uploaded as JPEGs or PNGs with alpha transparency but isn't available for download.

The path in the direct image URL that determines size and quality for preview images is w_<width #>,h_<height #>,q_<JPEG quality #>,strp , located after [...]/v1/fill and before /<title>_by_<username>_d<six base 36 digits>[...]. The <JPEG quality #> is normally set to 70 for social previews and 75 for large view images. If the JPEG quality is set to 100, then the quality of the image itself increases.

To give an example of this quality improvement in action, here's what each version of this image looks like when viewed in Firefox's page info window, with the URL's quality fragment highlighted:

Navigating to Inkbunny posts with the sourcing suite does not display the script's MD5 checks. Only the native, collapsed Inkbunny "MD5 Hash for Page" section is shown (the source of the confusing title changes).

Example e621 post and IB page:
https://e621.net/posts/2219545
https://inkbunny.net/s/2133766

This occurs on all IB posts I've tested.

This may be due to DA updating their APIs but I can't seem to get the integration to work with DA.

Uncaught (in promise) TypeError: Cannot read property 'parentNode' of null

(anonymous)                        574
Promise.then (async)
add_src                            572
MutationObserver.observe.childList 560

NSFW

ISS inserts an "Upload to e621" link on every FA view page, like https://www.furaffinity.net/view/45134129/ (probably NSFW)
But the description on this FA view page is too long, so clicking on "Upload to e621" leads to an e621 error page:
414 Request-URI Too Large

This can't be fixed: https://e621.net/forum_topics/31783?page=1#forum_post_323866

So I'd like to suggest that ISS also add an "Upload to e621 without description" link to every FA view page.
I can then click the "Copy Description" button and manually paste the description into the e621 upload form.

Or:
There's only an "Upload to e621" link.
But ISS checks if the URL length is ok.
If the URL is too long, it automatically copies the description to the clipboard, and displays a warning message via notification.

Twitter does not have a copy description button because I could not figure out how to cleanly interact with the DOM.

The MD5 checker is missing line breaks on Inkbunny posts, causing text to overflow to the next line.

Example post (safe): https://inkbunny.net/s/1844277

Expected behavior

When I go to NSFW Werewolf riding NSFW (User), the ISS UI should appear to the right, under the tags.

By "ISS UI", I mean:

idem's sourcing suite
Upload to e621
full image ​080c659691f76e48387e29ed50424c02
thumb image​ f9271dc28bd40d1a2ed48246c16b8c94

Actual behavior

The ISS UI is missing.
This is probably because the user's gallery currently contains only this one post, so there is no "See more from Justagoblin" under the tags. Which is probably what ISS looks for when inserting its UI.

Example where the ISS UI is shown

SFW [3D] Esix new years SFW (User)

The order of sources on the upload button should follow the standard of

1. Submission page
2. Direct url to image
3. Artist/gallery profile
4. Alternative submission page of same or inferior quality on another site
5. Same as 4
6. Same as 4
7. ...

The current code lists the sources in the order of

1. Direct url to image
2. Artist/gallery profile
3. Submission page

I'm running latest version of Tampermonkey on Firefox 69 (nice) and the latest Chrome version with both Violentmonkey & Tampermonkey.

On pixiv it used to show bellow each image but now there is no md5 string nor any link to it and when I look into tampermonkey it doesn't show the script running/loaded.

There's a similar issue on at least FA with no md5 link but at least the script is loaded.

It's possible to get PNG versions of "large view" and "social preview" images from DeviantArt if the original file is a PNG without alpha transparency.

Normally, only PNG images with alpha transparency will cause the DA page to grab their "large view" and "social preview" images as PNG images; non-transparent PNGs are converted into JPEGs for display on the site. However, there is a simple URL workaround that allows anyone to grab the "large view" and "social preview" images of such images in their intended PNG format. The Sourcing Suite should be updated to automatically retrieve links to "large view lossless" and "social preview lossless" images, as well as the MD5 hashes for the eSix image hasher function.

Using this deviation (DA submission) as an example, the original file is a PNG without alpha transparency, but the site shows the lower quality "large view" and "social preview" JPEG versions on the page as per normal.

Large view: https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/02c0932e-f858-42ad-babb-c390b692ace1/dbii8qf-245bf065-dc4a-4c50-86a3-d8d95e51abfd.png/v1/fill/w_1024,h_1280,q_80,strp/beloved_by_ly_sany_dbii8qf-fullview.jpg?token=[...]

Simply change the -fullview.jpg to .png to grab a lossless PNG image. (The JPEG quality parameter, q_##, is ignored.)

Large view lossless: https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/02c0932e-f858-42ad-babb-c390b692ace1/dbii8qf-245bf065-dc4a-4c50-86a3-d8d95e51abfd.png/v1/fill/w_1024,h_1280,q_80,strp/beloved_by_ly_sany_dbii8qf.png?token=[...]

Affected page: https://e621.net/extensions

The API key needs to be updated from https://e621.net/user/api_key to https://e621.net/users/USER#/api_key, as the old link leads to a "not found" page.

arrive.js provides events to watch for DOM elements creation and removal.
It makes use of Mutation Observers internally.

This will help with writing clean code to wait for elements and will outsource the hard work so I don't need to do it.

https://github.com/uzairfarooq/arrive

Fur Affinity posts no longer show the MD5 of posts after the site went independent again and changed the image server URL from facdn.net to furaffinity.net. I was able to fix this on my end by changing all instances of facdn.net to furaffinity.net, but the code should be updated in the project proper to reflect that.

Currently, image_compare does not give a crystal-clear indicator as to which files are presently loaded. This may yield confusion if a user is working with multiple similar URL fields to determine any discrepancies in visual data. To remedy this, I suggest adding two lines beneath the existing fields:

Source 1:
Source 2:

The downside of this approach is a reduction in initial visibility of the loaded images. However, I feel that displacing the images slightly to provide a clearer indicator of the specified URLs being successfully loaded is a net positive.

FurAffinity uses em elements with a special class to display site specific icons. Should a work around be built into the generic HTML to dtext function?

I do not know of a way to access the image_compare or upload_bvas pages without referring to the original forum thread or bookmarking the pages in question. If I'm not just missing it completely, I would like an easy method to pull up the aforementioned pages that does not require additional bookmarks.

I currently do not use the on-site-md5-hasher feature. However, using other aspects of the script, such as image_compare and upload_bvas, requires loading md5 hashes for every site supported (as far as I know). If possible, I would like an easy method for disabling this aspect of the script since it occasionally causes brief hangs on supported websites while loading a page (Firefox, 69.0.1).

Firefox 69.0
Linux Mint 19.1

When I installed Idems-Sourcing-Suite in Greasemonkey, https://e621.net/extensions/upload_bvas only opened an empty page.

It works perfectly in Tampermonkey.

Switching to Tampermonkey isn't a problem for me, because eSix Extend (my must-have script for e621) works fine in Tampermonkey.
It might be a problem for other people, though, so I'm reporting it.

The current check for a image element (<ul>) is too fast, and only returns the first image in an album.
Example case: https://twitter.com/Cumbread0/status/1400298464072945667 (NSFW).
By clicking directly on the second image in the above example the ul element doesn't have time to load before the below code runs.

See comments for explanation on the current issues of the code:

// This is the function in plans/twitter/main.js that is the root cause of the issue
async function get_sources () {
	const image_node = await (async () => {
		// This if-statement always returns null unless the album has already been loaded (clicked on another image, or used the arrow to change image)
                // A dead simple fix (which I'm currently using myself) is just to wait like 500ms, however this is less than ideal as its dependent on loading times
		if (document.querySelector('ul') === null) {
                        // This .arrive call only returns the 1 image of the album, regardless of which one is currently visible
			const image_node = await document.getElementById('react-root').arrive('div[role=dialog] img');
			document.getElementById('react-root').forget_arrives();
			return image_node;
		} else {
			const image_id = parseInt((/\d+$/).exec(window.location.href)[0], 10);
			const list_elems = new Array(image_id).fill('li').join(' ~ ');
			const query = `ul[role=list] > ${list_elems} img`;

			const image_node = await Promise.race([
				document.getElementById('react-root').arrive(query),

				document.getElementById('react-root').arrive('div > div > div > div > div > img[alt=Image]')
			]);
			document.getElementById('react-root').forget_arrives();
			return image_node;
		}
	})();
	const all_sources = produce_sources(image_node.src);
	return all_sources;
}

Adding the md5 hasher to patreon would be good.

If the Twitter language is set to something other than English, the script fails to create MD5 hashes and the "Upload to E621" button.
The following error is produced: link.

Presumably, this happens because the script attaches to the page by finding the element that contains the word "Reply".
Needless to say, that does not work if that word is in a different language.

Perhaps, using [data-testid=reply] as a query selector would work better?
Not sure what the testid attribute is, however, or if it's consistently present.

I assume that e621 understands commas between child posts as a separator indicator. However, a new user may misinterpret its default inclusion as an oversight and manually remove it from the "Children" field every time. If possible, I would advise to remove the comma when only one child post is present to prevent requiring an assumption or additional research.

The sourcing script does not display MD5 checks or the "Upload to e621" button when viewing FA posts with the Modern template enabled.

Example URLs tested by toggling between Classic and Modern under Account Settings :
https://www.furaffinity.net/view/24356189/
https://www.furaffinity.net/view/26348050/

Additional info:
Prior to adjustments for compatibility with the e621 update, MD5s were displayed to the left of the Views/Comments/Favorites/Rating section in the top right of a post page, displacing the text. For the purpose of convenience, retaining placement in the top right would prevent unnecessary page scrolls to check MD5s.

Title is self explanatory.

Some sites will serve images differently depending if you are logged in or not.

Document these sites and handle the specific instances.

Pixiv is known to do something with this.

One each of these sites, it would be useful if there was a button to quickly upload a post to e621. It would pre fill all of the fields that it could source, url, possibly tags

An example of this being done with twitter before the update can be found here https://gist.github.com/Sasquire/5629bf6f5b1e04b7fcb848e56e5c0faa

e621 post: https://e621.net/posts/1835094
DeviantArt source: https://www.deviantart.com/mr-shin/art/bin-792223876

Navigating to the DA source to compare MD5 hashes of the downloadable image results in following error message next to the "download" hash:

"Error. Unexpected e621 error"

This error occurs with both the classic DeviantArt theme and the new Eclipse theme. I was logged into DeviantArt when testing.

Other post examples with the same error:
https://www.deviantart.com/mr-shin/art/Fenneko-745446299
https://www.deviantart.com/mr-shin/art/Paltas-705047428

It seems that DeviantArt images that do not have a download option (primarily PNGs but there are JPEGs as well) but still have the full version viewable without any loss of its original quality anyway are not checked by the sourcing suite.

For example, post number 2297756 is a PNG image at 2000 by 2100 pixels with a size of 884.13 KB. This image was taken from DA, where full sized PNG is shown directly to the viewer and can be zoomed into by clicking on it twice, even though there is no download option. The DA URL for the PNG is https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/357d9b70-94c6-4721-ac04-7f2d4030b94e/ddxps4y-da9aef28-d775-4f07-a6c2-50f76501cc24.png?token=<random assortment of capital and lowercase alphanumeric characters and hyphens>. But the sourcing suite doesn't recognize this and only shows the social preview and social preview 100 links and their MD5s, thus preventing it from recognizing and showing users that the original version is viewable on there and it has already been submitted to eSix.

FN e6

These have the same hash, but the hash from this extension does not show FN having the same hash. Downloading these images and computing the hash manually still does not resolve this issue.

When I say these hashes are the same, I am talking about the hash found from FurryNetworks api. Perhaps they do something similar to Inkbunny where they will compress the image, but leave the original hash available?

FA has an option to always display the full date instead of "X days ago".
The script always gets the date's popup which fails in this case https://github.com/Sasquire/Idems-Sourcing-Suite/blob/master/source/plans/furaffinity/modern.js#L11

https://www.pixiv.net/member_illust.php?mode=medium&illust_id=41424535

Pixiv supports images having a spoiler warning. With this feature the script looks for the image and then sees the spoiler image and breaks.

From what I can tell violentmonkey does not seem to be able to load either the /extensions/* tools nor give the md5 on the supported sites while tampermonkey does.

When I view comments on DeviantArt when a submission there has received many of them, the sourcing suite treats it as viewing a new page and checks the MD5s of the image again, causing repeated text to appear. This pushes down the comments and the description, making it hard to read them. This is likely because if you scroll down on the page to view the comments, ?comment=1%3A<random numbers>%3A<random numbers> gets added at the end of the URL. Since the random numbers change depending on what comment you're reading, this means that the URL changes multiple times, triggering multiple checks.