fab-jul / torchac Goto Github PK
View Code? Open in Web Editor NEWEntropy coding / arithmetic coding for PyTorch
License: GNU General Public License v3.0
Entropy coding / arithmetic coding for PyTorch
License: GNU General Public License v3.0
Hello,
do you have any idea why this occurs when the torhac backend is compiled by ninja?
The log is quite long but this is what I get:
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/2] c++ -MMD -MF torchac_backend.o.d -DTORCH_EXTENSION_NAME=torchac_backend -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1017\" -I/usr/lib/python3.11/site-packages/torch/include -I/usr/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/usr/lib/python3.11/site-packages/torch/include/TH -I/usr/lib/python3.11/site-packages/torch/include/THC -I/usr/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=1 -fPIC -std=c++17 -c /usr/lib/python3.11/site-packages/torchac/backend/torchac_backend.cpp -o torchac_backend.o
FAILED: torchac_backend.o
c++ -MMD -MF torchac_backend.o.d -DTORCH_EXTENSION_NAME=torchac_backend -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1017\" -I/usr/lib/python3.11/site-packages/torch/include -I/usr/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/usr/lib/python3.11/site-packages/torch/include/TH -I/usr/lib/python3.11/site-packages/torch/include/THC -I/usr/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=1 -fPIC -std=c++17 -c /usr/lib/python3.11/site-packages/torchac/backend/torchac_backend.cpp -o torchac_backend.o
In file included from /usr/include/c10/util/logging_is_google_glog.h:50,
from /usr/include/c10/util/Logging.h:26,
from /usr/include/ATen/core/ivalue_inl.h:25,
from /usr/include/ATen/core/ivalue.h:1551,
from /usr/include/ATen/core/List_inl.h:4,
from /usr/include/ATen/core/List.h:490,
from /usr/include/ATen/core/IListRef_inl.h:3,
from /usr/include/ATen/core/IListRef.h:631,
from /usr/include/ATen/WrapDimUtils.h:3,
from /usr/include/ATen/TensorNames.h:3,
from /usr/include/ATen/NamedTensorUtils.h:3,
from /usr/include/torch/csrc/autograd/variable.h:11,
from /usr/include/torch/csrc/autograd/autograd.h:3,
from /usr/lib/python3.11/site-packages/torch/include/torch/csrc/api/include/torch/autograd.h:3,
from /usr/lib/python3.11/site-packages/torch/include/torch/csrc/api/include/torch/all.h:7,
from /usr/include/torch/extension.h:5,
from /usr/lib/python3.11/site-packages/torchac/backend/torchac_backend.cpp:8:
/usr/include/glog/logging.h:60:4: error: #error <glog/logging.h> was not included correctly. See the documentation for how to consume the library.
60 | # error <glog/logging.h> was not included correctly. See the documentation for how to consume the library.
| ^~~~~
In file included from /usr/include/glog/logging.h:63:
/usr/include/glog/flags.h:45:4: error: #error <glog/flags.h> was not included correctly. See the documentation for how to consume the library.
45 | # error <glog/flags.h> was not included correctly. See the documentation for how to consume the library.
| ^~~~~
...
Thanks for any help.
EDIT:
As a workaround I added
extra_cflags.append("-DGLOG_USE_GLOG_EXPORT")
to
/usr/lib/python3.11/site-packages/torch/utils/cpp_extension.py
here
# NOTE: Emitting a new ninja build file does not cause re-compilation if
# the sources did not change, so it's ok to re-emit (and it's fast).
extra_cflags.append("-DGLOG_USE_GLOG_EXPORT")
_write_ninja_file_to_build_library(
The function "torchac.encode_logistic_mixture() " is used in L3C-Pytorch, but I did not find it in torchac.
Thanks for your work. But I have a question, that is whether the symbol must in [0, Lp-2]. For example, I have 3 symbols: 100, 200, 300, and I also know their frequency, can I use torchac to compress them? If I must transform [100, 200, 300] to [0, 1, 2] before compressing, then how can I tranform them back to [100, 200, 300] after decompressing? It really makes me confused. Hope for your reply!
Compiling failed for torch 1.5.1.
To sovle it, replace AT_CHECK with TORCH_CHECK.
Solution is found here.
I faced unexpected behavior that was not described in README file or docstrings.
As i understand from C++ code, it is assumed that the input tensors are presented in the memory contiguously. However big tensors can be stored by torch in different places of memory.
One can easily overcome the issue using torch.Tensor.contiguous.
Please, note this in the README file.
Thanks for the nice work and sharing!
It seems that the library requires the shapes of output_cdf
and sym
to be B, C, H, W, Lp
and B, C, H, W
respectively.
However, when using the autoregressive context probability model, I have to compress each symbol one by one.
Is there any way to use the library in the autoregressive manner?
Thanks.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.