quru / qis Goto Github PK

View Code? Open in Web Editor NEW

88.0 11.0 7.0 82.96 MB

Dynamic image server for web and print

Home Page: https://quruimageserver.com

License: GNU Affero General Public License v3.0

Makefile 0.07% Shell 0.68% Python 57.55% JavaScript 31.50% CSS 1.72% HTML 7.78% Dockerfile 0.69%

image management image-api responsive-images dynamic-images image-server image-gallery vanilla-js pillow

qis's People

Contributors

Stargazers

Watchers

Forkers

fozcode imfantuan trendingtechnology cloudnepal tromcho pterameta

qis's Issues

How to configure the image max file size?

Hi, when uploading larger images we get an error looking like this.

Firstly it is a terrible UX as the actually error message is missing, so it is not clear what went wrong.
Secondly, how can I configure the max file size for uploading images?

Creation of new user accounts from LDAP auth is broken

The switch to Python 3 and/or the switch from pyldap to python-ldap has caused a regression when creating a new user account via an LDAP login. From the logs:

2018-08-09 12:09:44,514 qis_30863  DEBUG    Authenticating user 'trevor'
2018-08-09 12:09:44,517 qis_30863  DEBUG    Checking LDAP server for unknown user 'trevor'
2018-08-09 12:09:44,545 qis_30863  DEBUG    Identified user 'trevor' on LDAP server, authentication OK, creating new user account
2018-08-09 12:09:44,546 qis_30863  DEBUG    User details: {'gidNumber': [b'1234'], 'cn': [b'Trevor Foo'], 'uid': [b'trevor'], 'givenName': [b'Trevor'], 'uidNumber': [b'1234'], 'sn': [b'Foo'], 'dn': ['uid=trevor,cn=users,cn=accounts,dc=foo,dc=bar,dc=baz']}
2018-08-09 12:09:44,546 qis_30863  ERROR    Error performing login: a bytes-like object is required, not 'str'

The code is trying to process that dictionary as strings whereas the entries now contain bytes (apart from the dn which is a str 🙄). I think python-ldap has a setting for whether it returns strings or bytes so we need to look at that. Alternatively we just need to str() those bytes values in the code.

Delete uploaded image

Hi, How it is possible to delete images via the rest api after they re uploaded by api/v1/upload endpoint ?

Error running docker setup on Ubuntu

Looks like there is a missing configuration in the docker image?

Logs:

$ docker ps | grep as
59de577458a1        quru/qis-as          "/run-qis.sh"         3 hours ago         Up 3 hours (unhealthy)   0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   qis-via-docker_qis_as_1

$ docker logs 59de577458a1
Performing one-time initialization
Waiting 30 seconds for the Postgres database to be created
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.***.***.3. Set the 'ServerName' directive globally to suppress this message

$ docker -v
Docker version 19.03.2, build 6a30dfc

$ curl --insecure -i https://localhost
HTTP/1.1 500 Internal Server Error
Date: Wed, 02 Oct 2019 11:50:37 GMT
Server: Apache/2.4.18 (Ubuntu)
Content-Length: 607
Connection: close
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at 
 [no address given] to inform them of the time this error occurred,
 and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
<hr>
<address>Apache/2.4.18 (Ubuntu) Server at localhost Port 443</address>
</body></html>

README - producing a byte string

The README.md file provides instructions on how to generate a SECRET_KEY as:
python -c 'import os; print(os.urandom(16))'
With python2 installed this becomes unicode characters. To ensure that it works simply run:
python3 -c 'import os; print(os.urandom(16))'

Improve the Docker deployment

At present the Docker files and docker-compose script don't make it easy to install TLS certs or to override the QIS settings. We're also using an env var for the database password from the days before docker-compose had secrets support. I propose:

Change the database password to use a Docker secret
Update the AS Dockerfile, run script, and docker-compose file to mount TLS files from Docker secrets**
Update the AS Dockerfile, run script, and docker-compose file to allow the local_settings.py file to be overridden**
Update the README for the AS image
Update the README for the Postgres image
Update the running.md guide

** This also allows multiple AS containers to use the same certs and session secret for load balancing

.

API inconsistencies

If you delete a file/folder then delete it again you get a code 200, it should be a 404
The API returns HTML errors for "method not allowed" and for missing trailing slashes. Can we intercept these responses (coming from the middleware) and return JSON instead?
The file admin functions returns different fields to the file details function, make them more consistent (specifically the url, download, supported fields are missing from file admin operations)
Update the API docs

I don't think that any of these changes will break compatibility (you only get HTML errors for calls that are broken anyway).

Contrast adjustments

We could easily have contrast adjustment using

convert testimage.jpg -level 0%,75% lighttestimage.jpg
convert testimage.jpg -level 25%,100% darktestimage.jpg

See http://www.imagemagick.org/Usage/color_mods/#level for more details.

API - image object occasionally contains a history field

There is a scenario whereby an API call to /api/v1/details/ or /api/v1/list/ can return a history field in the image object(s). For example:

(qis) qurumatt:qis matt$ curl 'http://localhost:5000/api/v1/details/?src=Mixed/matt-new.tif'
    ...
    "height": 374,
    "history": [
      {
        "action": 1,
        "action_info": "File detected: Mixed/matt-new.tif",
        "action_time": "2018-10-16T15:51:49.579022Z",
        "id": 246,
        "image_id": 146,
        "user": null,
        "user_id": null
      }
    ],
    "id": 146,
    ...

Whereas if you make the same call a second time (and subsequently), there is correctly no history field:

(qis) qurumatt:qis matt$ curl 'http://localhost:5000/api/v1/details/?src=Mixed/matt-new.tif'
    ...
    "height": 374,
    "id": 146,
    ...

This bug is due to the caching of image objects in the SQLAlchemy session. For image files that the application finds on disk for the first time, a new audit trail is saved against the image and the image object is returned from the SQLAlchemy session cache. For images that already existed, the audit trail remains unloaded because it is lazy-loaded and is neither accessed nor changed.

This needs fixing to remove the history field (a) because it makes the return values inconsistent and (b) because it could in theory lead to user details being publicly visible in the same way as #13.

The bug affects APIs that deal with new files on disk (or the discovery of them). This is details, list and upload, though purely by chance the bug does not arise for upload because the internal implementation uses a separate SQLAlchemy session.

Write new tests
Fix for details
Fix for list
Fix for upload (in case of internal implementation changes in future)

API - add a way of retrieving portfolio by human ID

The API provides GET /api/v1/portfolios/[portfolio id]/ but no way of retrieving a portfolio from the unique human ID. It should be easy enough to add a URL for GET /api/v1/portfolios/?human_id=foo that does this.

Itsdangerous - upgrade past v1

Commit 624a468 added a limiter to itsdangerous of being less than v1. This is being upgraded continuously. Has a recent upgrade fixed the issue so that it can be used without large scale code changes?

Remove the user objects from Portfolios API

The Portfolios API "helpfully" includes serialized user objects in its output for the owner and history user fields. Unfortunately portfolios can be public facing, and therefore this can leak the username and email address of anyone that has created or interacted with a portfolio. We need to remove these objects, just leaving the owner_id and history user_id fields in place. Anyone that needs to look up the user information will have to do so with a separate call to ensure they have user admin permission.

The Tasks API also does this but is technically not affected because it should only return the user object of the current user or else only to super users. I think it's best if we delete it from here too though. The current user already knows who they are after all, so it doesn't add much in the common case.

User Management via OpenID Connect

Hi,

in our project we use QIS as our image server, which the users upload images via REST from our React SPA. Currently the app authenticates via a single user which acts as a service account. I am wondering though, if it is possible to connect QIS to our IDM via OpenID Connect. [1]

Ultimately the user would have ownership about his images and we could implement access control bases on permissions managed via the IDM. The user's profile picture for example could be private for example, whereas images the of something like a blog post [2] would be visible to. any logged in user.

Bests,
Martin

[1] We use Keycloak as IDM.
[2] just an example

API - portfolio image reordering returns old value of order_num

The API /api/v1/portfolios/[portfolio id]/images/[image id]/position/ does correctly set the value of order_num in the database, and does return the images in the correct order in the JSON, but the value of order_num in the JSON still contains the old value for the image that was moved.

I suspect this is a case of SQLAlchemy session caching again, we need to refresh the portfolio-image object in the session. The code already has a line for this at data_manager.py line 1582 but it doesn't seem to be working.

Base image detection for tiles sometimes seems to fail

Commit fa583f4 was to fix groups of warnings in the logs like the one below. But the change made was only appropriate for groups of tiles requested very close together in time. These logs show a gap of nearly 40 seconds while still triggering the old warning, which suggests that _get_base_image() was failing to identify the base image for the tile even after it had been generated. The new code will fix this in _get_tile_base_image() but does not address the underlying problem that _get_base_image() should have found the base image in the first place. Therefore this needs more testing using the image parameters shown below.

2018-11-01 23:10:44,256 qis_16219  WARNING  Tile base generation already performed for 24807277-9-1.jpg IMG:33111,W1160,H1740,Inone,Q75,Krgb,X,D72,E9:16
2018-11-01 23:10:53,958 qis_16215  WARNING  Tile base generation already performed for 24807277-9-1.jpg IMG:33111,W1160,H1740,Inone,Q75,Krgb,X,D72,E11:16
2018-11-01 23:10:59,392 qis_16215  WARNING  Tile base generation already performed for 24807277-9-1.jpg IMG:33111,W1160,H1740,Inone,Q75,Krgb,X,D72,E13:16
2018-11-01 23:11:04,100 qis_16215  WARNING  Tile base generation already performed for 24807277-9-1.jpg IMG:33111,W1160,H1740,Inone,Q75,Krgb,X,D72,E14:16
2018-11-01 23:11:08,483 qis_16215  WARNING  Tile base generation already performed for 24807277-9-1.jpg IMG:33111,W1160,H1740,Inone,Q75,Krgb,X,D72,E15:16
2018-11-01 23:11:12,907 qis_16215  WARNING  Tile base generation already performed for 24807277-9-1.jpg IMG:33111,W1160,H1740,Inone,Q75,Krgb,X,D72,E16:16
2018-11-01 23:11:13,575 qis_16219  WARNING  Tile base generation already performed for 24807277-9-1.jpg IMG:33111,W1160,H1740,Inone,Q75,Krgb,X,D72,E1:16
2018-11-01 23:11:19,291 qis_16215  WARNING  Tile base generation already performed for 24807277-9-1.jpg IMG:33111,W1160,H1740,Inone,Q75,Krgb,X,D72,E2:16
2018-11-01 23:11:23,667 qis_16215  WARNING  Tile base generation already performed for 24807277-9-1.jpg IMG:33111,W1160,H1740,Inone,Q75,Krgb,X,D72,E3:16
2018-11-01 23:11:23,849 qis_16219  WARNING  Tile base generation already performed for 24807277-9-1.jpg IMG:33111,W1160,H1740,Inone,Q75,Krgb,X,D72,E4:16

And another group spread out over nearly 2 minutes:

2018-11-01 22:47:47,890 qis_22804  WARNING  Tile base generation already performed for 24807215-10-1.jpg IMG:29843,W3192,H2256,Q75,Krgb,X,D72,E45:64
2018-11-01 22:47:52,282 qis_22803  WARNING  Tile base generation already performed for 24807215-10-1.jpg IMG:29843,W3192,H2256,Q75,Krgb,X,D72,E52:64
2018-11-01 22:47:52,883 qis_22805  WARNING  Tile base generation already performed for 24807215-10-1.jpg IMG:29843,W3192,H2256,Q75,Krgb,X,D72,E53:64
2018-11-01 22:47:57,465 qis_22803  WARNING  Tile base generation already performed for 24807215-10-1.jpg IMG:29843,W3192,H2256,Q75,Krgb,X,D72,E35:64
2018-11-01 22:47:57,880 qis_22805  WARNING  Tile base generation already performed for 24807215-10-1.jpg IMG:29843,W3192,H2256,Q75,Krgb,X,D72,E43:64
2018-11-01 22:48:02,934 qis_22803  WARNING  Tile base generation already performed for 24807215-10-1.jpg IMG:29843,W3192,H2256,Q75,Krgb,X,D72,E51:64
2018-11-01 22:48:05,110 qis_22805  WARNING  Tile base generation already performed for 24807215-10-1.jpg IMG:29843,W3192,H2256,Q75,Krgb,X,D72,E59:64
2018-11-01 22:48:32,599 qis_22805  WARNING  Tile base generation already performed for 24807215-10-1.jpg IMG:29843,W1732,H1224,Q75,Krgb,X,D72,E11:16
2018-11-01 22:48:32,988 qis_22804  WARNING  Tile base generation already performed for 24807215-10-1.jpg IMG:29843,W6328,H4472,Q75,Krgb,X,D72,E38:64
2018-11-01 22:48:34,575 qis_22804  WARNING  Tile base generation already performed for 24807215-10-1.jpg IMG:29843,W3192,H2256,Q75,Krgb,X,D72,E31:64
2018-11-01 22:48:49,628 qis_22804  WARNING  Tile base generation already performed for 24807215-10-1.jpg IMG:29843,W1732,H1224,Q75,Krgb,X,D72,E12:16
2018-11-01 22:48:50,238 qis_22805  WARNING  Tile base generation already performed for 24807215-10-1.jpg IMG:29843,W1732,H1224,Q75,Krgb,X,D72,E14:16
2018-11-01 22:48:53,542 qis_22804  WARNING  Tile base generation already performed for 24807215-10-1.jpg IMG:29843,W6328,H4472,Q75,Krgb,X,D72,E46:64
2018-11-01 22:49:29,973 qis_22803  WARNING  Tile base generation already performed for 24807215-10-1.jpg IMG:29843,W6328,H4472,Q75,Krgb,X,D72,E28:64
2018-11-01 22:49:30,690 qis_22804  WARNING  Tile base generation already performed for 24807215-10-1.jpg IMG:29843,W6328,H4472,Q75,Krgb,X,D72,E29:64

HTTP error 500s sometimes returned after an Apache graceful restart

I'm copying this issue here from Quru's internal bug tracker as it documents some interesting behaviour.

Seen in live today when it ruined a PDF report:
qis_access_log:
109.111.129.21 [08/Dec/2017:03:50:46 +0000] "GET /image?src=Images/live/2016-06/17/9226877-587-1.jpg.tif&top=0&left=0&right=1&bottom=1&fill=&angle=0&tmp=ConditionReport HTTP/1.1" 500 620 - 132606 - "Prince/10 (www.princexml.com)" "-"

No error in qis.log indicates that the error was not in the Python code. But qis.log does indicate that the httpd processes were being restarted at that time. Checking the cron log:

cron:
Dec 8 03:47:01 images7 run-parts(/etc/cron.daily)[8987]: starting logrotate
Dec 8 03:49:54 images7 run-parts(/etc/cron.daily)[11603]: finished logrotate

So yes there was a graceful restart in progress. Checking the Apache error log:
error_log:
[Fri Dec 08 03:50:46 2017] [error] [client 109.111.129.21] Premature end of script headers: runserver.wsgi
[Fri Dec 08 03:50:47 2017] [notice] child pid 11703 exit signal Segmentation fault (11)

We can see that the httpd process crashed at the same time and for the same client IP. Bingo.

It's going to be hard to find out if this is an issue with mod_wsgi on startup or with something the QIS code is doing e.g. when it forks off the auxiliary processes on the first request. The latter would seem more likely and can be tested seeing as this problem is reproducable. We may also need to try using the latest mod_wsgi instead of the old versions shipped in the standard o/s packages.

No module named 'encodings'

I am following the installation guide and have a problem with httpd starting python program. OS is Centos 7, locale is en_US.UTF-8

Here is what I found in httpd error log:

Current thread 0x00007fb1adcf6880 (most recent call first):
[Tue Oct 22 18:01:32.200714 2019] [core:notice] [pid 1951] AH00052: child pid 2170 exit signal Aborted (6)
[Tue Oct 22 18:01:32.201642 2019] [core:notice] [pid 1951] AH00052: child pid 2171 exit signal Aborted (6)
Fatal Python error: Py_Initialize: Unable to get the locale encoding
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ImportErrorImportError: : No module named 'encodings'No module named 'encodings'

I found suggestions to fix python path variables, but I don't have them and manually invoking python commands with UTF-8 encodings works well. Also I tried to install latest python3 from centos repository and compile mod_wsgi from pip, but got the same error.

Try out the app in "Play with Docker"

It would be great if we can spin up a demo environment with this:

https://medium.com/@patternrecognizer/how-to-add-a-try-in-play-with-docker-button-to-your-github-project-41cb65721e94

Needs testing to see whether it is possible to run a web service within it and get/supply the required host name for URL access. Also if so, whether there is any way of deploying HTTPS.

Hide secrets from error messages

Some error conditions leak too much information back to the user, e.g. full operating system paths:

{ "status": 500,
  "message": "Internal error ([Errno 13] Permission denied: '/opt/qis/images/published/2018-10-02/night.jpg')" }

In theory it is possible for worse things to leak, e.g. database connection details when the database is down. We should run error messages through a filtering routine before returning them. This can be initially based on the settings in base_settings.py and any other obvious things to filter out.

Create a "playground" page

Moving here from Quru's issue tracker to record the feature history:

QIS needs a publicly-accessible demo/playground page so that people can explore the imaging features - similar to what the publisher offers plus gallery / zoom viewing and so on. Whether it is enabled should be behind a system setting. Hopefully the URL-generating logic in the publisher page can be re-used.

API - don't return deleted records by default

The folder list admin function returns deleted records (need to check the file admin for this too). This doesn't seem like a sensible default, so change it to filter by status<>0, controlled by a new optional boolean parameter for whether to return soft-deleted data (or perhaps an optional status filter parameter would be better?)

Technically this is a breaking change but I don't think it was ever intended to work this way and should have filtered out deleted records from day 1. Nothing much can be done with the deleted records as the associated physical files are gone, so I don't expect this change to break any existing code.

Add a new optional status filter parameter
Return only "active" folders by default
Add new tests
Update the API docs

Converting colours to transparency

With the mask it is possible to make an image into a circle (have a white mask with a transparent circle in the middle and overlay it over the image) however the circular image remains on a coloured background. Would it be possible to convert the colour of the background to transparent?

To find the colour in a simple mask:
convert mask.gif -format %c -colorspace LAB -colors 5 histogram:info:- | sort -n -r | grep -v "#00000000"
will show the 5 most dominant colours in the image.
convert image.png -fuzz 20% -transparent "#ff00ffff" transparentimage.png
will make all #ff00ffff pixels transparent.

Note that glflib is possibly quicker for getting the colour (need giflib-tools)
gifclrmp mask.gif | grep -r "#00000000"`

Note that 2 colour gif palettes for the mask make this much easier. The fuzz option (ie needed for anti-aliasing etc) might need playing with.

It is possible that someone might want to make a colour in the original image transparent rather than in a mask...

API docs - clarify POST data vs query string parameters

Looking at updating the API docs for #12 it has become obvious that the docs don't properly describe the difference between "parameters" that are GET query string parameters and those that are actually POST/PUT data fields. The difference only becomes apparent when looking at the examples. We should define these separately in the docs and describe the difference in the intro.

Ideally where today we have 1 or 2 URLs and GET/POST/PUT/DELETE methods all described in one section, the docs should describe every URL and method separately. But this would be a complete rewrite of the document and I don't have time to do that right now.

Decide on new terminology for query params vs form data
Add intro section for GET params vs form data
Update the API guide to separate the GET params from the form data

Clarify a few things in the docs

Add a list of contents and section anchors to the overview doc
In the API docs, for 202 responses explicitly say that the task ID is in data {id}
Clarify that the system permissions on a group act as global overrides

Images with GPS - the Google map is broken

For images that have GPS fields in the EXIF data we display a Google map of its location when viewing the image in the admin area. This map is now degraded.

From https://cloud.google.com/maps-platform/user-guide/

In June 2016 we announced that we would stop supporting keyless usage, meaning any request that doesn’t include an API key or Client ID. This went into effect on June 11 2018, and keyless access is no longer supported. Keyless calls to the Maps JavaScript API and Street View API will return low-resolution maps watermarked with “for development purposes only.”

It is now mandatory to use a billing account and a Static Maps API key. This comes with $200 free monthly credits so it should still be free for most use cases, but we are going to have to get users of the image server to apply for and plug in their own API key. That or we change the map to run from a different service.

This one needs a quick review by @qururoland