fab-jul / torchac Goto Github PK

View Code? Open in Web Editor NEW

228.0 4.0 12.0 1.56 MB

Entropy coding / arithmetic coding for PyTorch

License: GNU General Public License v3.0

Python 46.79% C++ 42.75% Shell 10.46%

arithmetic-coding pytorch

torchac's People

Contributors

Stargazers

Watchers

Forkers

gh4n edmontdants davidalexandre gaurijagatap micmic123 chnvigny abaaac zb12138 swx-tj ddddm-pr aidol tianqi0124

torchac's Issues

Ninja compilation fail: error <glog/logging.h> was not included correctly.

Hello,
do you have any idea why this occurs when the torhac backend is compiled by ninja?
The log is quite long but this is what I get:

Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/2] c++ -MMD -MF torchac_backend.o.d -DTORCH_EXTENSION_NAME=torchac_backend -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1017\" -I/usr/lib/python3.11/site-packages/torch/include -I/usr/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/usr/lib/python3.11/site-packages/torch/include/TH -I/usr/lib/python3.11/site-packages/torch/include/THC -I/usr/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=1 -fPIC -std=c++17 -c /usr/lib/python3.11/site-packages/torchac/backend/torchac_backend.cpp -o torchac_backend.o 
FAILED: torchac_backend.o 
c++ -MMD -MF torchac_backend.o.d -DTORCH_EXTENSION_NAME=torchac_backend -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1017\" -I/usr/lib/python3.11/site-packages/torch/include -I/usr/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/usr/lib/python3.11/site-packages/torch/include/TH -I/usr/lib/python3.11/site-packages/torch/include/THC -I/usr/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=1 -fPIC -std=c++17 -c /usr/lib/python3.11/site-packages/torchac/backend/torchac_backend.cpp -o torchac_backend.o 
In file included from /usr/include/c10/util/logging_is_google_glog.h:50,
                 from /usr/include/c10/util/Logging.h:26,
                 from /usr/include/ATen/core/ivalue_inl.h:25,
                 from /usr/include/ATen/core/ivalue.h:1551,
                 from /usr/include/ATen/core/List_inl.h:4,
                 from /usr/include/ATen/core/List.h:490,
                 from /usr/include/ATen/core/IListRef_inl.h:3,
                 from /usr/include/ATen/core/IListRef.h:631,
                 from /usr/include/ATen/WrapDimUtils.h:3,
                 from /usr/include/ATen/TensorNames.h:3,
                 from /usr/include/ATen/NamedTensorUtils.h:3,
                 from /usr/include/torch/csrc/autograd/variable.h:11,
                 from /usr/include/torch/csrc/autograd/autograd.h:3,
                 from /usr/lib/python3.11/site-packages/torch/include/torch/csrc/api/include/torch/autograd.h:3,
                 from /usr/lib/python3.11/site-packages/torch/include/torch/csrc/api/include/torch/all.h:7,
                 from /usr/include/torch/extension.h:5,
                 from /usr/lib/python3.11/site-packages/torchac/backend/torchac_backend.cpp:8:
/usr/include/glog/logging.h:60:4: error: #error <glog/logging.h> was not included correctly. See the documentation for how to consume the library.
   60 | #  error <glog/logging.h> was not included correctly. See the documentation for how to consume the library.
      |    ^~~~~
In file included from /usr/include/glog/logging.h:63:
/usr/include/glog/flags.h:45:4: error: #error <glog/flags.h> was not included correctly. See the documentation for how to consume the library.
   45 | #  error <glog/flags.h> was not included correctly. See the documentation for how to consume the library.
      |    ^~~~~
...

Thanks for any help.

EDIT:

As a workaround I added
extra_cflags.append("-DGLOG_USE_GLOG_EXPORT")
to
/usr/lib/python3.11/site-packages/torch/utils/cpp_extension.py
here

# NOTE: Emitting a new ninja build file does not cause re-compilation if
    # the sources did not change, so it's ok to re-emit (and it's fast).
    extra_cflags.append("-DGLOG_USE_GLOG_EXPORT")
    _write_ninja_file_to_build_library(

It is a nice job. How can I debug torchac_backend.cpp in VSCODE？

where is the function "encode_logistic_mixture()"

The function "torchac.encode_logistic_mixture() " is used in L3C-Pytorch, but I did not find it in torchac.

A question about symbol range

Thanks for your work. But I have a question, that is whether the symbol must in [0, Lp-2]. For example, I have 3 symbols: 100, 200, 300, and I also know their frequency, can I use torchac to compress them? If I must transform [100, 200, 300] to [0, 1, 2] before compressing, then how can I tranform them back to [100, 200, 300] after decompressing? It really makes me confused. Hope for your reply!

Support for torch 1.5 and above

Compiling failed for torch 1.5.1.
To sovle it, replace AT_CHECK with TORCH_CHECK.
Solution is found here.

Contiguous tensors expected

I faced unexpected behavior that was not described in README file or docstrings.
As i understand from C++ code, it is assumed that the input tensors are presented in the memory contiguously. However big tensors can be stored by torch in different places of memory.
One can easily overcome the issue using torch.Tensor.contiguous.
Please, note this in the README file.

How to use torchac in an autoregressive compression manner?

Thanks for the nice work and sharing!
It seems that the library requires the shapes of output_cdf and sym to be B, C, H, W, Lp and B, C, H, W respectively.
However, when using the autoregressive context probability model, I have to compress each symbol one by one.
Is there any way to use the library in the autoregressive manner?
Thanks.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.