letmaik / rawpy

📷 RAW image processing for Python, a wrapper for libraw

rawpy is an easy-to-use Python wrapper for the LibRaw library. It also contains some extra functionality for finding and repairing hot/dead pixels.

API Documentation

Jupyter notebook tutorials

Sample code

Load a RAW file and save the postprocessed image using default parameters:

import rawpy
import imageio

path = 'image.nef'
with rawpy.imread(path) as raw:
    rgb = raw.postprocess()
imageio.imsave('default.tiff', rgb)

Save as 16-bit linear image:

with rawpy.imread(path) as raw:
    rgb = raw.postprocess(gamma=(1,1), no_auto_bright=True, output_bps=16)
imageio.imsave('linear.tiff', rgb)

Extract embedded thumbnail/preview image and save as JPEG:

with rawpy.imread(path) as raw:
    # raises rawpy.LibRawNoThumbnailError if thumbnail missing
    # raises rawpy.LibRawUnsupportedThumbnailError if unsupported format
    thumb = raw.extract_thumb()
if thumb.format == rawpy.ThumbFormat.JPEG:
    # thumb.data is already in JPEG format, save as-is
    with open('thumb.jpeg', 'wb') as f:
        f.write(thumb.data)
elif thumb.format == rawpy.ThumbFormat.BITMAP:
    # thumb.data is an RGB numpy array, convert with imageio
    imageio.imsave('thumb.jpeg', thumb.data)

Find bad pixels using multiple RAW files and repair them:

import rawpy.enhance

paths = ['image1.nef', 'image2.nef', 'image3.nef']
bad_pixels = rawpy.enhance.find_bad_pixels(paths)

for path in paths:
    with rawpy.imread(path) as raw:
        rawpy.enhance.repair_bad_pixels(raw, bad_pixels, method='median')
        rgb = raw.postprocess()
    imageio.imsave(path + '.tiff', rgb)

Installation

Install rawpy by running:

pip install rawpy

64-bit binary wheels are provided for Linux, macOS, and Windows.

Stable vs. pre-release

All stable rawpy releases are always built against a stable LibRaw library release. You can output the LibRaw version with print(rawpy.libraw_version).

rawpy pre-releases have version numbers like 0.15.0a1 and are built against a recent LibRaw snapshot. To install a pre-release, run:

pip install --pre rawpy

Optional features

The underlying LibRaw library supports several optional features. The following table shows which PyPI binary wheels support which features.

Feature	Windows	macOS	Linux
LCMS color engine	yes	yes	yes
RedCine codec	yes	yes	yes
DNG deflate codec	yes	yes	yes
DNG lossy codec	yes	yes	yes
Demosaic Pack GPL2	no	no	no
Demosaic Pack GPL3	no	no	no
OpenMP	yes	no	yes

Tip: You can dynamically query supported features by inspecting the rawpy.flags dictionary.

Note on GPL demosaic packs: The GPL2 and GPL3 demosaic packs are not included as rawpy is licensed under the MIT license which is incompatible with GPL.

Installation from source on Linux/macOS

For macOS, LibRaw is built as part of the rawpy build (see external/). For Linux, you need to install the LibRaw library on your system.

On Ubuntu, you can get (an outdated) version with:

sudo apt-get install libraw-dev

Or install the latest release version from the source repository:

git clone https://github.com/LibRaw/LibRaw.git libraw
git clone https://github.com/LibRaw/LibRaw-cmake.git libraw-cmake
cd libraw
git checkout 0.20.0
cp -R ../libraw-cmake/* .
cmake .
sudo make install

After that, install rawpy using:

git clone https://github.com/letmaik/rawpy
cd rawpy
pip install numpy cython
pip install .

On Linux, if you get the error "ImportError: libraw.so: cannot open shared object file: No such file or directory" when trying to use rawpy, then do the following:

echo "/usr/local/lib" | sudo tee /etc/ld.so.conf.d/99local.conf
sudo ldconfig

The LibRaw library is installed in /usr/local/lib (if installed manually) and apparently this folder is not searched for libraries by default in some Linux distributions.

Installation from source on Windows

These instructions are experimental and support is not provided for them. Typically, there should be no need to build manually since wheels are hosted on PyPI.

You need to have Visual Studio installed to build rawpy.

In a PowerShell window:

$env:USE_CONDA = '1'
$env:PYTHON_VERSION = '3.7'
$env:PYTHON_ARCH = '64'
$env:NUMPY_VERSION = '1.14.*'
git clone https://github.com/letmaik/rawpy
cd rawpy
.github/scripts/build-windows.ps1

The above will download all build dependencies (including a Python installation) and is fully configured through the four environment variables. Set USE_CONDA = '0' to build within an existing Python environment.

rawpy's Issues

lens shading effect after rawpy postprocess

Hi， I tried with this great rawpy lib to preprocess for hdrnet model, but the output got lens shading effect(center is bright and conner is dark)， and the histgram mach even enhanced the shading
I tried different papams but the best param I’ve tried is as below:

      if os.path.splitext(input_path)[1] == ".dng":
        with rawpy.imread(input_path) as raw: 
            im_input = raw.postprocess(use_camera_wb=False,
                                     no_auto_bright=True,
                                     output_color=rawpy.ColorSpace.raw,
                                     user_wb=raw.camera_whitebalance,
                                     demosaic_algorithm=rawpy.DemosaicAlgorithm.AHD,
                                     highlight_mode=rawpy.HighlightMode.Blend,
                                     output_bps=16, half_size=True,)

my output

the original dng test pair
the merged.dng is the input and the final is the expected output， any idea to avoid lens shading while in postprocess?

Test with and without OpenCV

The repair of bad pixels (rawpy.enhance module) will optionally use OpenCV if available for faster processing. Both situations should be explicitly tested in testBadPixelRepair, that is, when OpenCV is available, and when it is not (fallback to scikit-image).

Add bright parameter for postprocessing

Useful when no_auto_bright is set so that brightness can be manually set.

float bright;
dcraw keys: -b
Brightness (default 1.0).

Unsupported file format (support added in libraw 0.19)

Hi there, I am new to rawpy and I tried to import some raw image from IPX and HUAWEI mate 10, but rawpy dont seems to like it.

Traceback (most recent call last):
File "", line 1, in
File "/****/lib/python2.7/site-packages/rawpy/init.py", line 20, in imread
d.open_file(pathOrFile)
File "rawpy/_rawpy.pyx", line 266, in rawpy._rawpy.RawPy.open_file
File "rawpy/_rawpy.pyx", line 662, in rawpy._rawpy.RawPy.handle_error
rawpy._rawpy.LibRawNonFatalError: Unsupported file format or not RAW file

I did check Libraw and Iphone X and Mate 10 are in their supported list. So I am not sure what to do atm, please help.

Best,

LaTieCheng

Build Python 3.6 wheels

Support for reading generic python file objects?

Hi,
I've got a database of about 30Gb of gzip-compressed RAW images that I need to process. I'd like to avoid decompressing the files on disk, and instead just open each file as a file object with python's gzip module and then feed that to rawpy. However, RawPy will only accept a file path, which it then handles opening internally.

From skimming the code I couldn't tell whether or not this was a fundamental limitation of the underlying libraw. It also occurred to me that if libraw operates at a low-level on file pointers, the python-land object file from the gzip module might have no way to communicate w/ the c(++) layer.

Could the author comment on whether this functionality would be feasible to implement? If it's not simple, I will probably just content myself with the somewhat hackish workaround of unzipping each file to a temp location in order to read it with rawpy, but it's not ideal.

Documentation

Hello,

I would like to use this lib, but could you be more precise about the argument to use in the case of:

• postprocess(self, params=None, **kw)
• dcraw_process(self, params=None, **kw)

Thanks.

I am confused with the help of raw_pattern

It saids "The smallest possible Bayer pattern of this image". Just double check is this property the bayer pattern of image, or the smallest possibility indicating it is not?

rawpy/rawpy/_rawpy.pyx

Line 446 in 9a8a32d

The smallest possible Bayer pattern of this image.

Is there a way to modify the raw data

I want to change the content of the raw data that is read from the raw file.
The error I got was an AttributeError: attribute 'raw_image_visible' of 'rawpy._rawpy.RawPy' objects is not writable.

Could you please tell me is there a way to modify the raw data?

raw_pattern attribute returns invalid matrix for Fuji X-trans sensor images

Hi I have used rawpy for a while. I found there is some problem when the Fuji raw data (.RAF)

When I use rawpy to read Sony raw files, everything seems right: I can load the black_level, raw_pattern, etc. However, all these values are almost zeros when I load a Fuji raw file with the format RAF.

The raw_pattern is like:
[ 1 2 1 1 2 1]
[ 0 0 0 0 0 0 ]
...
[ 0 0 0 0 0 0]
And the black_level are also all zeros.

Can anyone fix this problem?

File with no demosaic processing

Hi,

I want to create an array of RGB that is not demosaiced. I tried.....

rgb = raw.postprocess(output_color=rawpy.ColorSpace.raw, gamma=(1, 1), use_camera_wb=False, output_bps=16, user_wb=[1.0, 1.0,1.0, 1.0], no_auto_bright=True, bright=1.0, demosaic_algorithm=None)

But the results are that of the default (AHD) I also tried demosaic_algorithm=False and demosaic_algorithm= 0 but always get same result.

Does this method not have capability to skip demosaic? If so can I just pull the values from raw.raw.image as values and then go use the bayer pattern to pick out the three colors?

Thanks

-Rich

seg fault with raw_image

Hi,

In case I want to access straight to the data contained in a raw file:
dummy = rawpy.imread('myimage.NEF').raw_image
I have a segmentation fault, and python crashes.

I tried without success:
dummy = rawpy.imread('myimage.NEF').raw_image.copy()

Doing it in 2 steps works fine:
dummy = rawpy.imread('myimage.NEF')
dummy = dummy.raw_image

So my solution now is to create a loading function:
def loadraw(im):
dummy = rawpy.imread(im)
return dummy.raw_image.copy()

Not too much of a big issue for me, but I thought you my want to have a look :-)

BTW, is there any way to include reading the EXIF data of a file in rawpy?

Cheers, and thanks a lot for that very nice library !
G

Dark Frame substraction?

Is there any support for it on rawpy?

save back in raw?

is there a way to process in raw and save back to raw format (e.g. ARW). It would be something like raw.save?

Building on Win x64 Anaconda fails

Hi,

Im trying to build rawpy on an Win10 x64 machine. Python is installed with Anaconda. I clone the repository and then run

python setup.py bdist_wheel

from an Visual C++ Command line. I followe the setup.py code and it does everything until

nmake raw_r

which fails when linking raw_r.dll with the error (detailed below)

C:\Program Files\Anaconda3\Library\lib\jpeg.lib : warning LNK4272: library machine type 'x64' conflicts with target machine type 'X86'

I assume it is connected to my python and thus my jpeg.lib being build on x64 and something in rawpy sticking to x86, but I can't find the point where it's stuck. I tried

1. Replacing all machine:X86 hooks
2. Replacing the auto cmake download and install to a new x64 version
   to no avail. Any help?

Cheers

Detailed Error:

(above, its working just fine....)
[ 71%] Building CXX object CMakeFiles/raw_r.dir/src/libraw_c_api.cpp.obj
cl : Command line warning D9025 : overriding '/W3' with '/w'
libraw_c_api.cpp
[ 85%] Building CXX object CMakeFiles/raw_r.dir/src/libraw_datastream.cpp.obj
cl : Command line warning D9025 : overriding '/W3' with '/w'
libraw_datastream.cpp
[100%] Linking CXX shared library raw_r.dll
   Creating library raw_r.lib and object raw_r.exp
dcraw_common.cpp.obj : error LNK2019: unresolved external symbol _jpeg_std_error referenced in function "protected: void __thiscall LibRaw::kodak_jpeg_load_raw(void)" (?kodak_jpeg_load_raw@LibRaw@@IAEXXZ)
libraw_cxx.cpp.obj : error LNK2001: unresolved external symbol _jpeg_std_error
dcraw_common.cpp.obj : error LNK2019: unresolved external symbol _jpeg_CreateDecompress referenced in function "protected: void __thiscall LibRaw::kodak_jpeg_load_raw(void)" (?kodak_jpeg_load_raw@LibRaw@@IAEXXZ)
libraw_cxx.cpp.obj : error LNK2001: unresolved external symbol _jpeg_CreateDecompress
dcraw_common.cpp.obj : error LNK2019: unresolved external symbol _jpeg_destroy_decompress referenced in function "protected: void __thiscall LibRaw::kodak_jpeg_load_raw(void)" (?kodak_jpeg_load_raw@LibRaw@@IAEXXZ)
libraw_cxx.cpp.obj : error LNK2001: unresolved external symbol _jpeg_destroy_decompress
dcraw_common.cpp.obj : error LNK2019: unresolved external symbol _jpeg_mem_src referenced in function "protected: void __thiscall LibRaw::kodak_jpeg_load_raw(void)" (?kodak_jpeg_load_raw@LibRaw@@IAEXXZ)
libraw_cxx.cpp.obj : error LNK2001: unresolved external symbol _jpeg_mem_src
libraw_datastream.cpp.obj : error LNK2001: unresolved external symbol _jpeg_mem_src
dcraw_common.cpp.obj : error LNK2019: unresolved external symbol _jpeg_read_header referenced in function "protected: void __thiscall LibRaw::kodak_jpeg_load_raw(void)" (?kodak_jpeg_load_raw@LibRaw@@IAEXXZ)
libraw_cxx.cpp.obj : error LNK2001: unresolved external symbol _jpeg_read_header
dcraw_common.cpp.obj : error LNK2019: unresolved external symbol _jpeg_start_decompress referenced in function "protected: void __thiscall LibRaw::kodak_jpeg_load_raw(void)" (?kodak_jpeg_load_raw@LibRaw@@IAEXXZ)
dcraw_common.cpp.obj : error LNK2019: unresolved external symbol _jpeg_read_scanlines referenced in function "protected: void __thiscall LibRaw::kodak_jpeg_load_raw(void)" (?kodak_jpeg_load_raw@LibRaw@@IAEXXZ)
dcraw_common.cpp.obj : error LNK2019: unresolved external symbol _jpeg_finish_decompress referenced in function "protected: void __thiscall LibRaw::kodak_jpeg_load_raw(void)" (?kodak_jpeg_load_raw@LibRaw@@IAEXXZ)
dcraw_common.cpp.obj : error LNK2019: unresolved external symbol _jpeg_abort_decompress referenced in function __catch$?lossy_dng_load_raw@LibRaw@@IAEXXZ$0
libraw_datastream.cpp.obj : error LNK2019: unresolved external symbol _jpeg_stdio_src referenced in function "public: virtual int __thiscall LibRaw_bigfile_datastream::jpeg_src(void *)" (?jpeg_src@LibRaw_bigfile_datastream@@UAEHPAX@Z)
C:\Program Files\Anaconda3\Library\lib\jpeg.lib : warning LNK4272: library machine type 'x64' conflicts with target machine type 'X86'
raw_r.dll : fatal error LNK1120: 10 unresolved externals
LINK failed. with 1120
NMAKE : fatal error U1077: 'C:\rawpy\external\cmake-3.7.2-win32-x86\bin\cmake.exe' : return code '0xffffffff'
Stop.
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\nmake.exe"' : return code '0x2'
Stop.
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\nmake.exe"' : return code '0x2'
Stop.
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\BIN\nmake.exe"' : return code '0x2'
Stop.

`.postprocess()` always returns a black image

Using a .nef from a coworker, I'm trying to use rawpy to convert to any sort of raster format. The images load fine, but after .postprocess(), an array full of zeros is returned. When saving, the resulting image is black, of course.

import sys
import numpy
import imageio
import rawpy

with rawpy.imread(sys.argv[1]) as raw:
    rgb = raw.postprocess(gamma=(1, 1), no_auto_bright=True, output_bps=16)
    print 'raw_image is %s' % raw.raw_image
    print 'rgb max is %s' % numpy.max(rgb)
    print 'rgb is %s' % rgb

imageio.imsave('test.tiff', rgb)

Outputs:

raw_image is [[243 260 241 ..., 253 243 253]
 [264 191 258 ..., 189 256 189]
 [243 260 245 ..., 251 241 251]
 ..., 
 [243 181 239 ..., 179 229 179]
 [223 241 225 ..., 231 225 231]
 [241 183 247 ..., 177 231 177]]
rgb max is 6381
rgb is [[[0 0 0]
  [0 0 0]
  [0 0 0]
  ..., 
  [0 0 0]
  [0 0 0]
  [0 0 0]]

 [[0 0 0]
  [0 0 0]
  [0 0 0]
  ..., 
  [0 0 0]
  [0 0 0]
  [0 0 0]]

 [[0 0 0]
  [0 0 0]
  [0 0 0]
  ..., 
  [0 0 0]
  [0 0 0]
  [0 0 0]]

 ..., 
 [[0 0 0]
  [0 0 0]
  [0 0 0]
  ..., 
  [0 0 0]
  [0 0 0]
  [0 0 0]]

 [[0 0 0]
  [0 0 0]
  [0 0 0]
  ..., 
  [0 0 0]
  [0 0 0]
  [0 0 0]]

 [[0 0 0]
  [0 0 0]
  [0 0 0]
  ..., 
  [0 0 0]
  [0 0 0]
  [0 0 0]]]

Question about auto_scale

What did auto_scale did in the postprocessing?

For example,
im2 = raw.postprocess(use_camera_wb=True, half_size=True, no_auto_bright=True,no_auto_scale=False, output_bps=16)
im1 = raw.postprocess(use_camera_wb=True, half_size=True, no_auto_bright=True,no_auto_scale=True, output_bps=16)

The only difference is no_auto_scale. Is there anyway that I can manually scale im1 so that it becomes the same as im2?

Help using LMMSE algorithm...

I struggling to figure out how to tell rawpy to use the LMMSE demosaicing algorithm or any other for that matter. Is it important that I use the dev branch of LibRaw? I just followed the standard build instructions and so have compiled the current 0.17.0 release version of LibRaw:

http://www.libraw.org/docs/Install-LibRaw-eng.html

I have confirmed that both extra demosaic packs are working when using libraw directly from the commandline. I'm on Arch Linux.

So with rawpy I've tried the following in my python script:

rgb = raw.postprocess(gamma=(1,1), DemosaicAlgorithm=9, no_auto_bright=True, output_bps=16)

and

rgb = raw.postprocess(gamma=(1,1), DemosaicAlgorithm=LMMSE, no_auto_bright=True, output_bps=16)

Any pointers would be most welcome.

Missing libraw parameters

Hi,

I'm trying to copy the behavior of my dcraw batch processing pipeline in python but some of the parameters of the output_params_t from libraw (aber[4] # -C; four_color_rgb # -f; highlight # -H) are not available in rawpy.

Are you planning to add them?

Unsufficient memory

I tried

pip install rawpy
pip install imageio
python ./rawpy_demo.py

rawpy_demo.py

import rawpy
import imageio

path = '5.DNG'
with rawpy.imread(path) as raw:
    rgb = raw.postprocess()
imageio.imsave('5.tiff', rgb)

and Unsufficient memory (libraw error) reported.

Traceback (most recent call last):
  File "./rawpy_demo.py", line 6, in <module>
    rgb = raw.postprocess()
  File "rawpy/_rawpy.pyx", line 616, in rawpy._rawpy.RawPy.postprocess
  File "rawpy/_rawpy.pyx", line 583, in rawpy._rawpy.RawPy.dcraw_process
  File "rawpy/_rawpy.pyx", line 660, in rawpy._rawpy.RawPy.handle_error
rawpy._rawpy.LibRawFatalError: Unsufficient memory

rawpy-0.11.0
ubuntu 14.04
Python 2.7.12 |Anaconda 4.2.0 (64-bit)

Could you please kindly help me fix this error? Or should I report the issue at the LibRaw repo instead?

does rawpy.imread(file.dng) close the file?

Working with Python 3.5 on Windows 7, I can open a .dng raw file with rawpy.imread(), and get the data into a numpy array, like this:
raw = rawpy.imread(oldFn))
inImageArray = raw.postprocess(gamma=(1, 1), no_auto_bright=True, output_bps=16)

Then, I can manipulate the data as I wish - all good so far.

Then, I want to rename the original .dng raw file. I've tried:
shutil.move(oldFn, newFn)
os.rename(oldFn, newFn)
and
os.replace(oldFn, newFn)

They all produce the same error:
Traceback (most recent call last): File "C:\WinPython-64bit-3.5.1.1\python-3.5.1.amd64\lib\tkinter\__init__.py", line 1549, in __call__ return self.func(*args) File "C:\Users\Steve\Documents\MyPython\Modules\bwdngv0_2.py", line 139, in doRename print(shutil.move(oldFn, newFn)) File "C:\WinPython-64bit-3.5.1.1\python-3.5.1.amd64\lib\shutil.py", line 553, in move os.unlink(src) PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:/Users/Steve/My pictures/cn3/_DSC2674.dng'
This seems to indicate that the file oldFn.dng is still open, and, if I look at the file with Windows Explorer, and, for example, try to rename it manually, it tells me that the file is open in python.exe and cannot be renamed. When I stop my script, I can then manually rename the file in the usual way.

I've tried opening and closing the file in the normal Python way, like this:
fileID = open(oldFn))
raw = ...
inImageArray = ...
`fileID.close()

but this has no effect on the problem.

How can I ensure that the .dng file is closed after rawpy.imread has finished reading it?

rawpy/_rawpy.c: No such file or directory

I've posted this on AskUbuntu too, but I thought this would be a better place.

I'm rebuilding my machine (14.04 LTS) and rawpy is resisting my efforts to install it (which worked before).

I have the pre-requisites numpy and libraw apparently successfully and no errors. When I sudo pip (or pip3) install rawpy and I get the following error. Any ideas?

# pip install rawpy       
Downloading/unpacking rawpy
  Downloading rawpy-0.10.1.tar.gz (107kB): 107kB downloaded
  Running setup.py (path:/tmp/pip_build_root/rawpy/setup.py) egg_info for package rawpy
Requirement already satisfied (use --upgrade to upgrade): numpy in /usr/local/lib/python2.7/dist-packages (from rawpy)
Downloading/unpacking enum34 (from rawpy)
  Downloading enum34-1.1.6-py2-none-any.whl
Installing collected packages: rawpy, enum34
  Running setup.py install for rawpy
    building 'rawpy._rawpy' extension
    x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -D_HAS_LIBRAW_CONFIG_H=1 -I/usr/local/include/libraw -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c rawpy/_rawpy.c -o build/temp.linux-x86_64-2.7/rawpy/_rawpy.o
    x86_64-linux-gnu-gcc: error: rawpy/_rawpy.c: No such file or directory
    x86_64-linux-gnu-gcc: fatal error: no input files
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 4
    Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip_build_root/rawpy/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-RwIkp2-record/install-record.txt --single-version-externally-managed --compile:
    running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/rawpy
copying rawpy/enhance.py -> build/lib.linux-x86_64-2.7/rawpy
copying rawpy/__init__.py -> build/lib.linux-x86_64-2.7/rawpy
copying rawpy/_version.py -> build/lib.linux-x86_64-2.7/rawpy
running build_ext
building 'rawpy._rawpy' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/rawpy
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -D_HAS_LIBRAW_CONFIG_H=1 -I/usr/local/include/libraw -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c rawpy/_rawpy.c -o build/temp.linux-x86_64-2.7/rawpy/_rawpy.o
x86_64-linux-gnu-gcc: error: rawpy/_rawpy.c: No such file or directory
x86_64-linux-gnu-gcc: fatal error: no input files
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 4
----------------------------------------
Cleaning up...

Change the raw_image and use Postprocess

The bayer output is returned as a numpy array by raw_image. I then make some changes to this numpy array and want to then continue with the postprocess, namely I am interested in applying demosaicing to this changed numpy array. How do I go about doing this?

Thanks.

Sharpness

Hi

Thanks for making such a useful wrapper to LibRaw.

I am noticing that my processed images do not look as sharp as the original raw image, they seem to loose the 'pop'. Is there anything I can do to enhance this?

Thanks

Why it's always raises unsufficient memory?

with this code
img = rawpy.imread(raw_path) img = img.postprocess(use_camera_wb=True, half_size=False, no_auto_bright=True, output_bps=16)
The second line is always raises unsufficient memory!

File "rawpy/_rawpy.pyx", line 616, in rawpy._rawpy.RawPy.postprocess
File "rawpy/_rawpy.pyx", line 583, in rawpy._rawpy.RawPy.dcraw_process
File "rawpy/_rawpy.pyx", line 660, in rawpy._rawpy.RawPy.handle_error
rawpy._rawpy.LibRawFatalError: b'Unsufficient memory'

The raw_path file is about 25.4M.
I use the htop to see the memory, with the 16G memory,.
this code raise error sharply and without much memory used.

why the raw image is rotated

Im using the following code to get the bayer image.
But I dont get why it is rotated?
Any suggestion

raw = rawpy.imread(path)
rgb = raw.raw_image

Access to "LibRaw" datastructures.

Hi,

Follow up on a quick 5 minutes test before shifting to work: I probably overlooked it but didn't find a way to access from a given RawPy instance various LibRaw datastructures of high interest for me such as libraw_colordata_t (http://www.libraw.org/docs/API-datastruct-eng.html#libraw_colordata_t
) and libraw_imgother_t (http://www.libraw.org/docs/API-datastruct-eng.html#libraw_imgother_t)

Any helpers on that would be highly appreciated :)

Cheers,

Assumptions on Mac OS X break the installation via MacPorts

I would like to package rawpy for MacPorts which is very similar to any other Linux distributions, but many assumptions in the setup code break the installation.

One of the problems is

def mac_libraw_compile():

and

def clone_submodules():

I fetch the sources for rawpy from GitHub's tarballs and I would like to use libraw from an external package. The problem is that the following code fails:

    if not os.path.exists('external/LibRaw/README'):
        print('LibRaw git submodule is not cloned yet, will invoke "git submodule update --init" now')
        if os.system('git submodule update --init') != 0:
            raise Exception('git failed')

because I'm not using a git repository. The next problem is

if isWindows or isMac:
    external_dir = os.path.abspath('external')
    ...
    # for Windows and Mac we use cmake, so libraw_config.h will always exist
    libraw_config_found = True

The libraw library that gets installed via the package manager doesn't install libraw with CMake. I'll try to get that fixed, but the logic of "if this is mac, install the package in a completely different way than on linux" (that doesn't allow easy configuration options to proceed as one would on linux) is not the most suitable for a package manager.

rawpy questions (white balance; dcraw -A option)

Maik

I hope you don't mind me contacting you again, but I have some questions about the use of rawpy. I'm used to using dcraw directly, and often use the command line option -A , which calculates the white balance from a selected area of the image, and then returns the white balance multipliers on STDERR, ready for use with the option -r for similar images. Rawpy seems to be able to use the -r option, via the user_wb parameter, but I can't find an equivalent of the -A option, nor a means of obtaining the multipliers for future use.

Have I missed something, or are these facilities not present (yet?) in rawpy?

Best wishes

Steve Morton

Question: if I want to subtract a manual dark image, how?

I have constructed a numpy array of a dark image, from the raw_image values, and want to apply this to the data image before postprocess - however I get:

AttributeError: attribute 'raw_image' of 'rawpy._rawpy.RawPy' objects is not writable

trying to:

    black = raw.black_level_per_channel
    assert black[0] == black[1] == black[2] == black[3]
    help(raw)
    raw.raw_image += black[0]
    raw.raw_image -= dark

here the += black[0] is to ensure value stays positive as all data are type uint16 i.e. keep standard black value...

Any help much appreciated!

Can rawpy be used to change brightness and exposure?

I need to deal with a large number of raw images.

CI Appveyor: Don't swallow download exceptions

Currently, the Appveyor python install script swallows exceptions when downloading miniconda and then only tests for the existence of the file to assert if it was downloaded correctly. This is a bad test though as the file could have been partly downloaded or another IO issue could have happened. The right thing to check is if after all download retries there was still an exception, and then throw that one to make it appear in the console.

Example where the exception was swallowed and a misleading exception down the line was thrown: https://ci.appveyor.com/project/neothemachine/rawpy/build/1.0.194/job/i4ysa7warwtfyify

no_auto_scale implementation?

Good evening,

I'm very interested in using rawpy to convert my Nikon NEF raw files to tiff format with as little image transformations as possible. So far, rawpy has been fantastic for this, but I'm struggling with one issue. I'm not sure if this is a feature request or if the functionality already exists, but here it is:

My Nikon NEF raw files are 14-bits, so 16,384 possible values. When I use rawpy to convert the raw image into a 16-bit tiff, I want the values to be maintained fro 0 to 16384, and not from 0 to 65536.

I think in this context, we are talking about auto-scaling, and I've noticed in the _rawpy.pyx on line 136 there is a no_auto_scale line with the comment "Disable Auto-scale". Is this already implemented in a way that I can use with the raw.postprocess() function or is this something that could be implemented and added into the master branch?

Thank you for your help and response!

--no-binary is not an option

Hi there,
In trying to install using your directions "install the latest release version from the source repository," I successfully get through all steps up until the pip install of rawpy:

k@Calvin:~/libraw$ pip install rawpy --no-binary :all: .
Usage: /usr/bin/pip install [OPTIONS] PACKAGE_NAMES...

/usr/bin/pip install: error: no such option: --no-binary

I am running Ubuntu 12.04 LTS, 32-bit.
Thank you!

Wrong raw_colors / raw_pattern for CR2 file

Hello,

First of all, thank you for sharing with us you useful module !
However, I think I have found a small inconsistency in its results.

With dog.CR2 being this image, I run the following code:

import rawpy
import scipy.misc
import numpy as np

def save_image(image, path):
    image = image.clip(0.0, 1.0)
    image = np.power(image, 1/2.2)
    scipy.misc.toimage(image*255,  high=255, low=0, cmin=0, cmax=255).save(path)

raw_canon = rawpy.imread("dog.CR2")
print(raw_canon.raw_colors_visible)
save_image(raw_canon.raw_image_visible / 16383, "mosaiced_dog.png")

I get the following output:

[[3 2 3 ... 2 3 2]
[0 1 0 ... 1 0 1]
[3 2 3 ... 2 3 2]
...
[3 2 3 ... 2 3 2]
[0 1 0 ... 1 0 1]
[3 2 3 ... 2 3 2]]

And this dumped image: mosaiced_dog.png
If I discard the last (ie. bottom) line, and pack the image into 4 pictures, I get the following subsampled pictures :
- for the pixels at positions (2n, 2n): see here
- for the pixels at positions (2n + 1, 2n): see here
- for the pixels at positions (2n, 2n + 1): see here
- for the pixels at positions (2n + 1, 2n + 1): see here

Looking at the dog's pink nails allows me to say that on the visible part of the RAW, the top-left pixel (0,0) corresponds to the RED filter in the Bayer CFA. This is contradictory with what raw_colors_visible indicates: according to this attribute, the first pixel should correspond to a GREEN (Gb) one ! This seems to happen on all RAW files from this camera body (Canon 5D Mark II).
Is it a misunderstanding on my side, or a bug ?

Thanks a lot for your help !

rawpy.LibRawNonFatalError: Unsupported file format or not RAW file

Hey guys, trying to use rawpy for the first time. I seem to be unable to do imread(myRAWFile) and am getting the error in the title. I have tried to look up the documentation and can't find anything. What file formats are supported?

LibRaw 0.18 released

Is there a version LibRaw 0.18 support?

color artifact in the linux version, reading a raw image

I am reading the raw image (*.nef) via rawpy package.
When I use it in linux it gives me kinda a color artifact.
do you guys have any solution for it?

for loop dir convert problem

Hello, I use the For loop current directory file (all is Cr2), but rawpy error did not find the file, I use print is the normal output file name (such as EXAMPLE.CR2).

The code is as follows:

import rawpy
import imageio
import os

path = './'
files = os.listdir(path)
for file in files:
  with rawpy.imread(file) as raw:
    rgb = raw.postprocess()
  imageio.imsave('d:/images/' + file + '.jpg', rgb)

Thanks~

Compilation error / sdist on PyPI broken

During the build or rawpy 0.10.1 I've this error:

python setup.py install --root=${pkgdir} --prefix=usr --optimize=1

gcc -DNDEBUG -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector --param=ssp-buffer-size=4 -fPIC -D_HAS_LIBRAW_CONFIG_H=0 -I/usr/include/libraw -I/usr/lib/python2.7/site-packages/numpy/core/include -I/usr/include/python2.7 -c rawpy/_rawpy.c -o build/temp.linux-x86_64-2.7/rawpy/_rawpy.o
gcc: error: rawpy/_rawpy.c: File or directory do not exists

Support Python 3.5

There is no miniconda for Python 3.5 yet, but it will come in October. Then it should be more or less easy to support it.

Cannot read .raw image: LibRawNonFatalError - Unsupported file format

I'm trying to use rawpy.imread to load some .raw images

X = rawpy.imread('Run100000001.raw')

But I get an error saying that the format is not supported or it's not RAW file

_rawpy.pyx in _rawpy.RawPy.open_file (_rawpy.cpp:2457)()  
_rawpy.pyx in _rawpy.RawPy.handle_error (_rawpy.cpp:7344)()  
LibRawNonFatalError: b'Unsupported file format or not RAW file'

The images can be processed as RAW file using Matlab

What is the difference between raw_image_visible and raw_image

can someone explain What the difference between raw_image_visible and raw_image is.
I see it say with margin for one of them and without margin for the other one.
What exactly we mean by margin here?
Thanks

Implementing Demosaicing Algorithm

Hi there,

I would like to check on the proper implementation for the demosaicing algorithms.

...
from rawpy import DemosaicAlgorithm

raw = rp.imread('''.\input/raw/1test.dng''')

rgb = raw.postprocess(no_auto_bright=True, user_wb=raw.daylight_whitebalance, demosaic_algorithm=DemosaicAlgorithm.AAHD, output_color=ColorSpace.sRGB)
rgb = raw.postprocess(no_auto_bright=True, user_wb=raw.daylight_whitebalance, demosaic_algorithm=DemosaicAlgorithm.VNG, output_color=ColorSpace.sRGB)

(r, g, b) = cv2.split(rgb)
cv2.imwrite('''.\input/raw/1test_new7.jpg''', cv2.merge([b,g,r]))

Regardless of the type of demosaic algorithm, the results produced were all the same, and some mosaic patterns still visible. Also, some other type, VCD, LMMSE options are not available.

Expected

Current results

Mosaic

Appreciate if anyone cld guide me to the proper implementation of going from current to expected? :))

Exception: git failed / macOS 10.9.5 unsupported

Hello @letmaik, I'm not sure if this is something that you can help me with, but I am trying to pip install rawpy on my Mac OS X 10.9.5 with "pip install --user rawpy," but I run into this error:

Installing collected packages: rawpy
  Running setup.py install for rawpy ... error
    Complete output from command /opt/local/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python -u -c "import setuptools, tokenize;__file__='/private/var/folders/yt/47j3bxc91z7b3_j9pjdq5kb40000gn/T/pip-build-xJgffe/rawpy/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/yt/47j3bxc91z7b3_j9pjdq5kb40000gn/T/pip-4GBXMf-record/install-record.txt --single-version-externally-managed --compile --user --prefix=:
    LibRaw git submodule is not cloned yet, will invoke "git submodule update --init" now
    fatal: Not a git repository (or any of the parent directories): .git
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/yt/47j3bxc91z7b3_j9pjdq5kb40000gn/T/pip-build-xJgffe/rawpy/setup.py", line 322, in <module>
        mac_libraw_compile()
      File "/private/var/folders/yt/47j3bxc91z7b3_j9pjdq5kb40000gn/T/pip-build-xJgffe/rawpy/setup.py", line 283, in mac_libraw_compile
        clone_submodules()
      File "/private/var/folders/yt/47j3bxc91z7b3_j9pjdq5kb40000gn/T/pip-build-xJgffe/rawpy/setup.py", line 127, in clone_submodules
        raise Exception('git failed')
    Exception: git failed

Not sure why this is happening, or if it's on my end or the installer's end. Any thoughts? Thanks.

Build manylinux1 wheels

See PEP 513.

Development happens in manylinux1 branch.

Status:

• Wheels are building (see alpha release at https://pypi.python.org/pypi/rawpy/0.10.0a1)
• docs don't update anymore on python setup.py upload_docs due to pythonhosted.org deprecation -> deployment happens on GitHub Pages now

Support older numpy versions

The numpy version that rawpy is compiled against determines the (more or less) minimum numpy version that the user must have installed. See also Stackoverflow.

The CI environment should be set to compile against the oldest numpy version possible to have as much compatibility as possible. This minimum version should then be included in the requirements of the readme.

Grainy raw images vs jpg / Expose noise reduction postprocessing options from libraw

Hello,

I was not sure where to post this, so I hope you do not mind me posting here. I have noticed there is a lot of 'grainyness' when I load in a raw file vs a jpg. I assume the camera is doing something when it produces a jpg that removes this noise. How would I go about doing this with rawpy?

The reason I need to do it is that I am thresholding my images to remove the background, and if I use images that are grainy I end up with a mess.

many thanks.

pip install fails when installing libraw from source

It happens when you installed libraw from source and there is no package installed.

Error message:

libraw_version.h: No such file or directory

I have fixed it with following patch, but we should find better solution to merge into master.

diff --git a/setup.py b/setup.py
index 63e1a09..ae20b3d 100644
--- a/setup.py
+++ b/setup.py
@@ -103,6 +103,11 @@ if isWindows or isMac:
     libraw_config_found = True
 else:
     use_pkg_config()
+
+    include_dirs += ['/usr/local/include/libraw']
+    library_dirs += ['/usr/local/lib']
+    libraries = ['raw_r']
+    libraw_config_found = True
     
     # check if libraw_config.h exists
     # this header is only installed when using cmake

Include FBDDNoiseReductionMode in docs

See #45 (comment)