hypothesis / bouncer Goto Github PK

It's hardly ever responding.

For hypothesis/product-backlog#1199

Once we have the following dependencies cleared up:

• #407 - Raven
  • hypothesis/h-pyramid-sentry#19 - h-pyramid-sentry
    • Pylons/pyramid_retry#22 - pyramid-retry
• #408 - pyramid_jinja2

We should be clear to upgrade the version of Python using whatever mechanism was developed for h_periodic in hypothesis/h-periodic#73

The version will likely be decided by pyramid_jinja2, but it should at least be 3.7:

In #48 @robertknight says:

group:{pubid} - No, not yet. I do understand that this would be very useful. It does surface some issues that we'll need to address or have a reasonable idea of what we might do later, such as how to handle the case where the current user is not a member of the group and will we ever support multiple groups. Could you file an issue?

Questions:

What if the current user is not a member of the group

If they follow a hyp.is link that specifies a group they're not a member of, they should be taken to the indicated page, but the target group annotations obviously cannot and presumably should not be shown to them. For now, I don't think they should auto-join the group, though I can imagine that groups which are more open might be able to be configured to allow auto-joining.

In the event that they're not a member, perhaps a card or alert is shown that says "you don't have permission to view annotations in group XXX".

Will we ever support multiple groups (being specified as params)

Possibly-- though I imagine the overwhelming use case here is a single group. For sure multiple groups is not a first requirement. Do we need to determine now?

Hope I'm in the right place:

Let me know if there is somewhere else I should send such alerts.

Summary

If you follow a bouncer link in a browser where the Chrome extension is installed, the extension doesn't activate once the link resolves.

Steps to reproduce

1. Get a Bouncer link for the Public channel for a PDF or web page. Three that we've tested with are:

• https://hyp.is/go?url=https%3A%2F%2Fcontent.govdelivery.com%2Fattachments%2FMIEOG%2F2020%2F04%2F02%2Ffile_attachments%2F1417518%2FEO%25202020-35.pdf&group=__world__
• https://hyp.is/go?url=https%3A%2F%2Fd242fdlp0qlcia.cloudfront.net%2Fuploads%2F2019%2F02%2F08123342%2FHypothesis-VPAT-2019.02.04.pdf&group=__world__
• https://en.wikipedia.org/wiki/Buridan%27s_bridge

2. In a browser where the Chrome extension is installed, follow that link

Expected behavior

The Chrome extension activates and the sidebar opens

Actual behavior

The extension is not activated. You have to manually turn it on:

Notes

Strangely, we don't see this problem with private groups. You can join this private group and it'll work properly. See:

• https://hyp.is/go?url=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FBuridan%2527s_bridge&group=Bd1pN47n
• https://hyp.is/go?url=https%3A%2F%2Fcontent.govdelivery.com%2Fattachments%2FMIEOG%2F2020%2F04%2F02%2Ffile_attachments%2F1417518%2FEO%25202020-35.pdf&group=Bd1pN47n
• https://hyp.is/go?url=https%3A%2F%2Fd242fdlp0qlcia.cloudfront.net%2Fuploads%2F2019%2F02%2F08123342%2FHypothesis-VPAT-2019.02.04.pdf&group=Bd1pN47n

It's also working properly for publisher groups. See:
https://hyp.is/go?url=https%3A%2F%2Fmodernismmodernity.org%2Fforums%2Fposts%2Fresponses-responses-special-issue-weak-theory&group=q9iV3JLd

...so this appears to be isolated to the Public channel.

For details see hypothesis/via#813

See this slack thread for details: https://hypothes-is.slack.com/archives/C4K6M7P5E/p1699367044627499

Use a separate GitHub environment for each Elastic Beanstalk environment, and add URLs to the GitHub environments.

For details see hypothesis/via#820, hypothesis/via#821 and https://hypothes-is.slack.com/archives/C4K6M7P5E/p1666887953922879.

Rob: Looking at the JS code, I do see an issue that we're not checking chrome.runtime.lastError in the sendMessage callback

Go to http://mlawgroup.de/news/publications/detail.php?we_objectID=227

Create an annotation.

Capture the share link, e.g. https://hyp.is/oExZ4PVtEeefHEPs4cEz6w/www.mlawgroup.de/news/publications/detail.php

Visit the share link:

The via link should be:

https://via.hypothes.is/http://mlawgroup.de/news/publications/detail.php?we_objectID=227#annotations:oExZ4PVtEeefHEPs4cEz6w

But instead is:

https://via.hypothes.is/http://mlawgroup.de/news/publications/detail.php#annotations:oExZ4PVtEeefHEPs4cEz6w

Which fails.

When visiting a bouncer link in Safari, I often see a blank white page and a broken image icon, instead of the Hypothesis logo. Example link: http://hyp.is/d0bL8J45Ee2jgp8SioFyUQ/arxiv.org/pdf/2203.02155.pdf. The logo loads correctly in Chrome and Firefox.

If I disable JavaScript in a new tab via Develop -> Disable JavaScript and then visit http://hyp.is/d0bL8J45Ee2jgp8SioFyUQ/arxiv.org/pdf/2203.02155.pdf, the logo and styles load correctly. I'm guessing what is happening is that when the immediate redirect to Via happens after the page loads, it causes the other resources on the page not to be loaded.

We shouldn't proxy docdrop pages, since the client is already embedded there. The recent change to not proxy the youtube domain has exacerbated the issue.

https://twitter.com/ChrisAndrewsEdu/status/1354156722349342722

Bouncer currently uses Python 3.8 which is supported until October 2014. We need to upgrade to a newer version of Python before then.

Background

Praecipio are building an integration between Confluence and Hypothesis. This uses a custom version of the Hypothesis Chrome extension which users log into using their Atlassian ID. Users see groups that correspond to the Spaces they belong to in Confluence. Annotations that are created get saved to Hypothesis and also posted as entries in Confluence pages.

Behind the scenes, this extension communicates with a custom H API endpoint which is a proxy server that wraps the real H API to customize the login process and "decorate" API calls with additional functionality related to the Confluence integration. The user accounts, groups and annotations used with this extension are all associated with the atlassian.hypothes.is authority in h.

See the #atlassian-demo channel in Slack for more details.

Problem

We want to support sharing links for these annotations, similar to the hyp.is links for regular Hypothesis users. These sharing links will be included in the entries posted in Confluence, so users have a convenient way to view the link in context.

The bouncer service currently directs users either to the regular Hypothesis Chrome extension or Via, depending on whether they have the extension installed. The regular Hypothesis Chrome extension assumes first party user accounts, which is not the case for these annotations. Via also won't work for the same reason.

Constraints

We'll need a solution we can quickly deploy for an upcoming demo. We will also need something we can run longer term.

Possible solutions

1. Run a separate instance of the bouncer service specifically for Atlassian users

The hyp.is server allows configuring the Chrome extension and Via URL via environment variables. We could run a new instance of bouncer which modifies this configuration. Sharing links in annotations would somehow need to be modified to reference this new instance. This could be done by the proxy server, although that wouldn't work for any system or client fetching annotations from the API directly rather than going via the proxy server.

Pros:

• Reduces amount of configurability required within the bouncer service

Cons:

• This adds additional infrastructure that we have to run and deploy new updates to
• We have to find a way to change the incontext_link field on annotations to point at the custom service. There is currently nowhere in h to configure authority-wide settings.
• We'll still need to make some changes to bouncer to handle the unavailability of Via for these annotations, and also customize attributes such as the logo

2. Add ability to configure behavior of bouncer depending on annotation authority

Allow some basic configuration of how Bouncer directs users depending on the annotation's authority. The minimal configurability we would need for this project is:

1. Direct users to a different extension if the annotation is associated with the atlassian.hypothes.is authority
2. Provide a fallback if the user does not have the Atlassian extension installed. This could be a redirect to a static URL which tells the user how to install the extension. In future we might want the ability to direct the user to a custom Via-like service.

We don't add new authorities often, so it would be fine if this configuration was stored in a static file.

Pros:

• Makes it easier to support similar use cases in future

Cons:

• Requires more configurability in bouncer

Tracking issue for:

• https://github.com/hypothesis/bouncer/security/code-scanning/1

After internal analysis we are marking this alert as a false positive. Additional information can be found in Slack

The reported issue is:
DOM text reinterpreted as HTML
On this line:
window.location.replace(url);
The error is wrong because the function argument here is interpreted as a URL, not HTML.

We want to Upgrade h to the latest version of Elasticsearch. Since Bouncer talks directly to h's instance of Elasticsearch (in both production and development), Bouncer also needs to support the latest version before we can upgrade h's Elasticsearch.

Bouncer currently talks to h's Elasticsearch directly. If we want to reduce our dependence on Elasticsearch (see hypothesis/h#7975) we might want to remove Bouncer's Elasticsearch dependency (and e.g. have it call an h API instead).

Add a redeploy.yml workflow similar to the one that was recently added to Via.

For: hypothesis/product-backlog#1199

To support the upgrade effort we are attempting to reduce our dependencies particularly where they duplicate other modules or hold back upgrades. Raven looks to duplicate some of the work in our own pyramid_sentry library and should be considered for removal.

This means we need to update h-pyramid-sentry to support multi python: hypothesis/h-pyramid-sentry#19

We want to get away from the frolvlad/alpine-python3 Docker base image just because it's a random third-party image that we don't use anywhere else. The problem is how to get an Alpine Linux with Python 3, including a version of pip that works with Python 3. Nick pointed out how smokey does it: https://github.com/hypothesis/smokey/blob/master/smokey/Dockerfile#L4

Dependabot couldn't authenticate with https://pypi.python.org/simple/.

You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.

View the update logs.

For: hypothesis/product-backlog#1199

It looks like pyramid_jinja2 only support 3.6 according to it's classifiers: https://pypi.org/project/pyramid-jinja2/

The docs mention adding 3.7 support and looking at the repo there is evidence of 3.8 support at least: https://github.com/Pylons/pyramid_jinja2

We should work out what the correct version is and try and work with the maintainers to fix the classifiers if possible.

This is holding up our upgrades to a higher version of Python.

Looking at our production API, some annotated document URIs already have # fragment identifiers in them. Based on a quick look at 100,000 production annotations coming out of the hypothes.is API (not directly out of Elasticsearch, but I think it's the same) these mostly seem to end in # alone with no value, e.g. http://www.thetimes.co.uk/tto/environment/article4702412.ece# but some are #value ones.

Anyway, simply appending #annotations:<id> to that won't work, I think we need to strip these.

We don't delete annotations immediately from the search index but replace it with a {"deleted": true} body. We probably need to detect this at the beginning and then raise an error that should basically show the 404 error.

https://sentry.io/hypothesis/bouncer/issues/323267010/

KeyError: 'group'
(2 additional frame(s) were not displayed)
...
  File "pyramid/view.py", line 541, in _call_view
    response = view_callable(context, request)
  File "pyramid/config/views.py", line 353, in rendered_view
    result = view(context, request)
  File "pyramid/config/views.py", line 483, in _class_requestonly_view
    response = getattr(inst, attr)()
  File "bouncer/views.py", line 43, in annotation
    parsed_document = util.parse_document(document)
  File "bouncer/util.py", line 72, in parse_document
    group = annotation["group"]

KeyError: 'group'

For example given the URL https://blackboard.american.edu/bbcswebdav/pid-3280529-dt-content-rid-11942183_1/courses/WRTG-101-017-034-050-2016S/PerlemanLes_InformationIlliteracy.pdf the redirecting message will be Loading annotations forhttps://blackboard.american....`. The . followed by the … looks like four dots in a row.

Suggested solution: after truncating the string remove any trailing dots before appending the ellipsis?