berlinonline / converjon Goto Github PK
View Code? Open in Web Editor NEWAn advanced image conversion server and command line tool.
Home Page: http://berlinonline.github.io/converjon/
License: MIT License
An advanced image conversion server and command line tool.
Home Page: http://berlinonline.github.io/converjon/
License: MIT License
/status should give an HTML page instead of just a JSON response.
Also, the data shown there should give more information about the actual state of the application.
For debugging/testing/troubleshooting purposes, the original source URL should be included in the responses as a header.
In 1.x.x when you need it to test the origin of an image, you would first have to decode the source URL. Having it in a plaintext header will make this easier.
Hi, I often get this kind of errors(Error during image analysis), and the images are broken.
Is there any way to fix the Error during image analysis ?
The original files that get download are ok, and if I try a few more times I do get a good image.
Apparently there is apossibiliuty that NaN values find their way into command line arguments.
[Mon, 11 Mar 2013 14:41:03 GMT] Process error output convert ./cache/source/www.berlinonline.de/image.php/localnews/asset-25929 -crop NaNxNaN+NaN+NaN -resize 620x250! jpg:./cache/target/www.berlinonline.de/image.php/localnews/asset-25929/0bd13514a1f6933aedf08795e3efece708b81bebcaa58d5a07919f0a0f065068.jpeg convert: invalid argument for option `-crop': NaNxNaN+NaN+NaN @ error/convert.c/ConvertImageCommand/1090.\n"
Some mimetypes are delivered with charset=binary
so that the internal check for the correct mimetype within converjon does not match theese images correctly.
Hot fix for me was to stop delivering the charset information, but I think the right way would be to handle that right in converjon?
information for handling mime-types: http://www.mhonarc.org/~ehood/MIME/2046/rfc2046.html#4.2
Service dies if image url ends with '/'
Example URL: http://www.berlinonline.de/binaries/asset/image_assets/1073880/source/1263907606/667x500/
Check, if Converjon runs on node 0.10.0 without additional problems.
Correct the engine dependency accordingly.
The HTTP agents socket pool fills up to it's configured limit and then stops accepting any more download requests. This results in the Converjon server running into "Processing Timeout" errors on every subsequent request.
Disable agent pooling as a quick fix. A better solution should be found in 1.7.0.
The status page should contain information about the requests that are currently processed or waiting.
It would be great if one could pass an etag or timestamp or something equivalent to converjon when requesting images. When set converjon could then ignore any (default) cache settings and then purge->deliver.
The status information should have a configuration option to disable the public staus page and write the data into a file instead.
The use of the connect middleware framework does not yield any real benefit to converjon.
It should be removed and URL routing should be done directly in the app since it's a really simple routing.
This will also remove the "bouncer" middleware. It was never really used and it's basically out of scope for converjon. A dedicated reverse proxy can probably provide this functionality much better.
Clear invalid cache items when the expire to reduce storage clutter.
converjon should pass all headers from the backend response to the client, except the Content-Length.
We need a stable way to start/stop/respawn converjon on a sysv-init based system (SLE11SP2)
Converjon should be installable from NPM. The package should provide a bin
script to launch the server.
Numbers on all log lines for a request.
URL just on the first entry.
One request must be entirely grepable.
All 3 cache subdirectories ("source", "target", "temp") should be cleared on launch.
converjon should be able to fetch images from HTTPS servers.
Possible explanation:
When internal success/error events are fired, the respective other event handler is not removed. they accumulate until Converjon crashes with an EventEmitter memory leak exception.
For operating behind varnish a simple health check URL would be nice. Answering this URL should not depend on any secondary server.
Due to legal issues (licence agreement with a commercial foto service) we need a max width and height for pictures.
The configuration should be in the global environment and should affect all delivered images.
As an alternative to env variables there should be a way to set the environment for a working copy via a text file in the base directory.
The env variable should still override this setting.
Add some more useful infos to the status page:
Load balancing and clustering directly in node.js
Thanks @aeytom, for the idea:
Converjon should provide a CLI utility that exposes the same features and API as the web server but with local files/IO-Streams.
Example calls could look like this:
cat foo.jpg | conversion-cli --width=140 quality=65 > thumb.jpg
and/or:
converjon-cli --width=140 --quality=65 foo.jpg thumb.jpg
This could enable other applications to use converjon as a wrapper around imagemagick.
We have a GIF image, wich scales from 166x99px to 214x125px, but should be 166x125px. I prepared an example here:
http://test.berlin.de/converjon/?height=125&width=166&url=http%3A%2F%2Ftest.berlin.de%2Ftest%2F_assets%2Flogo_gute_tat.gif&mime=image%2Fgif
http://test.berlin.de/test/_assets/logo_gute_tat.gif
(password protected, contact me for a login)
Public, but might be deleted in near future:
http://www.berlin.de/converjon/?height=125&width=166&url=http%3A%2F%2Fwww.berlin.de%2Fvak%2F_assets%2Faktuelles%2F2014%2Flogo_gute_tat.gif&mime=image%2Fgif
http://www.berlin.de/vak/_assets/aktuelles/2014/logo_gute_tat.gif
Perhaps some problem with the GIF file format?
Versions:
make a /status url that always answeres with "200 OK" and returns some usefuls stats maybe.
The headers should depend on the age of the scaled image and the internal used cache timings.
Service is used via https, and images are coming from https-servers.
If I test on my dev-box I have only self-signed certificates, such that node.js https module is rejecting the image-requests.
I patch it by myself in lib/downloader.js around line 60
--- a/lib/downloader.js
+++ b/lib/downloader.js
@@ -57,6 +57,7 @@ module.exports = function(req) {
var download_options = url.parse(sourceUrl);
download_options.method = "GET";
+ download_options.rejectUnauthorized = false; /* */
var auth_credentials = authentication.getCredentials(sourceUrl);
if (auth_credentials) {
It would be nice to configure it via url-regexp that I permit unauthorized ssl-requests for certain domains.
Images are sometimes distorted when using AOI-Cropping.
When the same source image is requested multiple time simultaneously, sometimes a convert worker starts reading the file before it is downloaded completely.
This happens because imageFetcher.js (line 107) just checks, if the file exists, which is TRUE while the file is being written by the downloadQueue.
This bug results in corrupted target image data.
Just now, we became many: Processing timeout
Thee are no relavant problems on the status page.
A restart of converjon.service solves the problem for now
The status page should include latest errors from the log.
The two-tiered cache storage of 1.x.x should be eliminated. Given the right timing, it can lead to an image being cached almost double the configured time.
The new storage will store every source image in a directory, similar to the source cache of 1.x.x but instead of having the additional target cache directory, the converted images are just stored in the same directory as the source and are identified by their conversion parameters. The storage dir for every item should contain a metadata file (probably JSON) that is used to hold HTTP headers from the origin server, the expiration date and other additional information if necessary.
When a cached item becomes stale, its source file and all conversion targets will be deleted.
When the download of a source URL fails, it is not removed from the list of acrive downloads. Every subsequent request to that URL will result in a "Processing Timeout" error.
Currently, all configs are located in the config
directory. This will no longer be possible, if converjon is installed via NPM (see #47).
Default config should still be packaged with the application but a local config file should be given to the application at launch time via command line argument.
something like converjon --config=foo/bar/conf.json
There needs to be an URL that can be called to determine wether the service is ready to process requests or if waiting queues are full.
I think we should move things like the openSuse service config into a seperate repositiory.
We should also be able to fetch images from HTTPS sources.
For configured URLs (keys from whitelist) converjon needs to pass auth headers to the source server.
The counter for running processes in inreased too soon. It is inreased in the process constructor instead of when the process actually starts running.
This allows the running process count to go beyond the configured limit and prevents aubsequent requests from ever being processed.
For art directional uses it may be cool to increase the fraction the area of interest has when thumbnails get smaller. That is, the smaller the thumbnail the larger the fraction of the image the area of interest takes.
See: http://usecases.responsiveimages.org/#art-direction for a small example. Bullet point 6 in http://blog.cloudfour.com/8-guidelines-and-1-rule-for-responsive-images/ gives another example image.
It would be nice if this feature would be configurable and even allow target images having other ratios than the source image as that is of use in many use cases (huge source images and multiple target teaser sizes and ratios)
The /status page should include an identifier to help distinguish between multiple converjon instances behind a load balancer.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.