Comments (15)
`src/include/fst/float-weight.h:99:2: error: #error "Please compile with -msse -mfpmath=sse, or equivalent." #error "Please compile with -msse -mfpmath=sse, or equivalent.
Actually It's not related with the MinGW.
You are using i686-w64-mingw32 (for 32bit target)
. Did you compile with -msse2 -mfpmath=sse
?
You can see the conditional-macro inside float-weight.h
and It checks __SSE2_MATH__
not __SSE_MATH__
.
openfst/src/include/fst/float-weight.h
Lines 94 to 100 in 3382254
You should specify that compile flags when you compile for the 32-bit target. You can see the exactly same checking in the configure file. X86-64 GCC uses SSE and SSE2 default. You can see this on the GCC x86-options document.
For the x86-32 compiler, you must use -march=cpu-type, -msse or -msse2 switches to enable SSE extensions and make this option effective. For the x86-64 compiler, these extensions are enabled by default.
You can check your macro conditions like below.
# Mingw 64-bit GCC-10
$ /usr/bin/x86_64-w64-mingw32-g++-posix -dM -E - </dev/null | grep SSE
#define __MMX_WITH_SSE__ 1
#define __SSE2_MATH__ 1
#define __SSE_MATH__ 1
#define __SSE__ 1
#define __SSE2__ 1
# Mingw 32-bit GCC-10
$ /usr/bin/i686-w64-mingw32-g++-posix -dM -E - </dev/null | grep SSE
# MinGW 32-bit GCC-10 with -msse2 -mfpmath=sse
$ /usr/bin/i686-w64-mingw32-g++-posix -msse2 -mfpmath=sse -dM -E - </dev/null | grep SSE
#define __SSE2_MATH__ 1
#define __SSE_MATH__ 1
#define __SSE__ 1
#define __SSE2__ 1
Can you show the CMakeList.txt after fixed?
CMakeLists.txt
is not matter in your case. You need to set the compile flags for 32-bit. :)
from openfst.
@hwiorn: Thanks for the recommendation. That's probably the correct way to do it. Let me take care of this.
A question, out of pure curiosity: I never tried MinGW. I am wondering what benefits does it provide? Microsoft cl is one of the best, if not the best optimizing compiler out there, and is available free of charge¹. I understand MinGW is a port of GNU g++ and its glibc to Windows, is this approximately correct? Are you using because you need a library which is compatible with something else compiled with the same compiler, or does it produce object/library files compatible with MSVC?
¹ "Valid license" refers to any SKU EULA, including the free Community Edition. Last time I installed them on a CI server the package did not ask for anything except agreeing with the license. Hope that haven't changed.
from openfst.
I'm sure and agree your opinion that VC(IDE, tools) and cl are the best option on Windows system. I've developed some programs on Windows and I know there is a VS community edition(including VC++ express version) already. I used VC express(now community) version on Windows and I saw VC can optimize better than GCC in some area. So I always recommend it on Windows to my colleagues.
On the other hand, MS provides free version of compiler but Windows is not. And they don't provide a VC(especially cl) for Linux and It means I have to manage a Windows machine(physically or virtually) to compile and maintain the system. MS provides Windows development virtual machine for evaluation and testing. I'm not sure that I can use it for business legally. Last time I saw, I couldn't use it on business.
In my experiences of supporting cross-platform, It's really annoyed to write for different compiler on different systems which I have to compile on both system every time when I changed code. I was trying to make Jenkins to build it for Windows; It didn't work sometimes where I have to maintain the system with Jenkins settings. Eventually I changed from Jenkins to Virtual Box and QEMU/KVM. I know there is a fancy option called WSL. But it is like Linux that needs to cross compile to make windows executables like Cygwin and MSys2(+MinGW) in my experience.
I make programs on Linux using Python, Java, C++ and Go. That languages can cross-compile for Windows using MingGW on Linux. Go have options to build executables for different OSes and architectures with CGO. It means I can handle CI/CD pipeline using Linux containers, just without bring it into other platforms. My purpose is make same(or similar) results using MinGW without MS cl. I guess the libraries compatible with MSVC is next thing which my colleagues may request me.
Recently, I cross-compiled Kaldi for Windows using cmake and mingw successfully to use in a program built by Go . I could have fixed openfst source code just for my purpose privately. But I think the open-source have a new option is always good.
This is just my experience, You can correct me if I'm wrong :)
from openfst.
The patch can be like the below. It needs -Wa,-mbig-obj
with optimization level like -O2
.
I tested many times. I don't know why it makes error when no optimization flags.
diff --git a/CMakeLists.txt b/CMakeLists.txt
index f6189db..dcbd2de 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -17,8 +17,12 @@ set(CMAKE_MACOSX_RPATH 1)
set(CMAKE_CXX_STANDARD 11)
if (WIN32)
- add_definitions(/bigobj)
- set(WHOLEFST "/WHOLEARCHIVE:fst")
+ if (MSVC)
+ add_definitions(/bigobj)
+ set(WHOLEFST "/WHOLEARCHIVE:fst")
+ else()
+ add_compile_options(-Wa,-mbig-obj -O2)
+ endif ()
set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} ${WHOLEFST}")
#set(CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS 1)
#this must be disabled unless the previous option (CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS) is enabled
from openfst.
@hwiorn, thanks for the patch! The -mbig-obj option is required due to differences in the way compiler generates sections in object files. The historical limit on the number of sections in the PE/COFF format is around 2^16, and heavily templated libraries may exceed that limit. /BIGOBJ (msvc) or -mbig-obj (MinGW) enables up to 2^32 sections. It's just a compatibility setting, only one 16-bit word replaced with a 32-bit one in the header, but that is incompatible with very old linkers. I think (but am not sure) this is now the default for C++ applications; it certainly is for UWP apps.
I was not implying that you do something wrong, and I'm sorry if you read my question this way. I was just curious why do people use MinGW, and estimate if its use is common enough for us to support it. I didn't ever realize you could cross-compile programs for Window with it on Linux! I understand you, build tools are free (and you do not even need the Community edition, only Build Tools), but the OS itself isn't!
When I bought mine in 2015 or 16 it was cheap ($200 for a license transferable between computers as many times as you want, so I picked up a Pro edition "just in case"; because Windows 10 is fully supported till 2025, it came to $20/yr TCO which is basically nothing. But now 2025 and EOL is much closer :) ). AFAIK, Windows 10 does not expire or die if you run it without a license, only keeps very mildly nagging you, but generally you need a license. Microsoft had to make a lot of concessions after the Ballmer era has ended—the dude nearly destroyed the internal culture and smeared company public image. Nadella had to do major repairs: the house was falling apart, and he have done a very good job. Thus. I don't expect such a generosity with Windows 11, but we'll see! :)
Thanks again for the feedback and the patch!
from openfst.
heavily templated libraries may exceed that limit
I couldn't think about template optimizations! Thank you! I forgot that openfst is heavily template library. Other template library like fmt uses -mbig-obj
option when it uses mingw.
I was just curious why do people use MinGW, and estimate if its use is common enough for us to support it. I didn't ever realize you could cross-compile programs for Window with it on Linux!
I think cross compiling is used when target system have limited (hardware and software) resource or some reasons which can be philosophy or avoiding license issue.
Even not using dependent code and libraries like MFC, ATL and COM, writing code on both system made me feel like I am in never ending hell sometimes when I met compile error with large code changed(10+ commits) on another system. It means I had to fix errors and logics while switching both system where make mistakes easily. Some compiler makes different template result and some compiler is scolding it.
I use self-hosted gitlab and gitlab CI/CD which can run it on docker containers. Container system like docker and k8s is based on Linux stack and it makes possible to build projects and release the files without switching another system.
Windows 10 does not expire or die if you run it without a license, only keeps very mildly nagging you, but generally you need a license.
I know how I can stop Windows is nagging me because I was a Windows fanboy ;) After I accustomed macOS and Linux, I've used them(especially Linux now) more than Windows. Actually, I have Windows 10 Pro licenses personally, so the license can not make a problem for me. But I experienced some small business company hasn't been affordable for Windows licenses and physical machine powers enough on their business.
Thanks again for the feedback and the patch!
You're welcome. Thank you for your kind response.
from openfst.
@hwiorn hello, I'am tring to use i686-w64-mingw32 to compile this project.
And use your patch, but still causing error. error message:
src/include/fst/float-weight.h:99:2: error: #error "Please compile with -msse -mfpmath=sse, or equivalent." #error "Please compile with -msse -mfpmath=sse, or equivalent." ^~~~~ src\lib\CMakeFiles\fst.dir\build.make:105: recipe for target 'src/lib/CMakeFiles/fst.dir/fst-types.cc.obj' failed make[2]: *** [src/lib/CMakeFiles/fst.dir/fst-types.cc.obj] Error 1 CMakeFiles\Makefile2:1278: recipe for target 'src/lib/CMakeFiles/fst.dir/all' failed make[1]: *** [src/lib/CMakeFiles/fst.dir/all] Error 2 Makefile:144: recipe for target 'all' failed make: *** [all] Error 2
Can you show the CMakeList.txt after fixed? Thanks for your reply!
from openfst.
I experienced some small business company hasn't been affordable for Windows licenses and physical machine powers enough
You need much more of a hardware oomph from the build machine than the absolute minimums for either OS. Take that into account, and the difference disappears. And a business that can't afford an extra few hundred bucks is not anywhere close to being viable anyway.
Tangentially,
I don't expect such a generosity with Windows 11, but we'll see! :)
Yay, I was wrong again!
Srsly, @jtrmal, do you think we want to support cross-compile on Linux for Windows, or not? I can't grasp the idea. You cross-compile binaries for Windows on a free-as-beer-isn't OS, and then... what? Release without testing? God, no. Use them without Windows? But how? I'm lost. Maybe we should, but I currently don't see any benefits. Someone else may fork and do it.
from openfst.
from openfst.
You need much more of a hardware oomph from the build machine than the absolute minimums for either OS. Take that into account, and the difference disappears. And a business that can't afford an extra few hundred bucks is not anywhere close to being viable anyway.
No, For cross compiling use-case and licenses you asked, It were the small companies's issue I saw. In this case, It's not my money issue. I have already valid Windows 10 licenses. I'm not saying poorman's cross-compiling. For me, Developing on Linux is my preference now and I accustomed Linux tools than MSVC now. You know that, using Linux(Unix) tools on Windows is really slower and somtimes pain. So I tried to cross compile for Windows on Linux. Because it works. And It can make CI/CD pipeline docker based easily than Windows VM basded. As I said, I don't like to fix and test core logic many things for different targets on one source code base after reboot to target platform. Because It makes side effects easily I think.
Srsly, @jtrmal, do you think we want to support cross-compile on Linux for Windows, or not? I can't grasp the idea. You cross-compile binaries for Windows on a free-as-beer-isn't OS, and then... what? Release without testing? God, no. Use them without Windows? But how? I'm lost. Maybe we should, but I currently don't see any benefits. Someone else may fork and do it.
On Linux, You can test with wine and wine64 before release on Windows no reboot and VM.
Sometimes, wine can't be enough if you use specific WinAPIs or host DLLs. I tested, kaldi decoder was enough this case.
So When I modify core logic, I can test and see it will work on two different OS right.
I'm more inclined setting up a github actions to make openfst releases
compiled with different compilers
cross-compiling is pain
I'm agreed with you. cross-compiling is REALLY pain.
I think OpenFST doesn't have platform specific features (such as CUDA, GUI, WDK, DXD, Services and etc.). OpenFST side for MinGW needs this CMake patch only I pasted.
diff --git a/CMakeLists.txt b/CMakeLists.txt
index f6189db..dcbd2de 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -17,8 +17,12 @@ set(CMAKE_MACOSX_RPATH 1)
set(CMAKE_CXX_STANDARD 11)
if (WIN32)
- add_definitions(/bigobj)
- set(WHOLEFST "/WHOLEARCHIVE:fst")
+ if (MSVC)
+ add_definitions(/bigobj)
+ set(WHOLEFST "/WHOLEARCHIVE:fst")
+ else()
+ add_compile_options(-Wa,-mbig-obj -O2)
+ endif ()
set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} ${WHOLEFST}")
#set(CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS 1)
#this must be disabled unless the previous option (CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS) is enabled
I use CMake fetchcontent_declare
and PATCH_COMMAND
for MinGW Kaldi patching my own.
fetchcontent_declare(
kaldi
GIT_REPOSITORY "https://github.com/kaldi-asr/kaldi"
GIT_TAG ac29a6ff09823d1cbb4814da60360c966f33cd0d
UPDATE_DISCONNECTED 1
PATCH_COMMAND git apply ${CMAKE_CURRENT_SOURCE_DIR}/patches/kaldi-mingw.patch
)
fetchcontent_makeavailable(kaldi)
I had decided to make MinGW patch for Kaldi because Kaldi supports Cygwin compile and I thought It was possible with a little modification.
OpenFST is impossible to be patch by this way.
its src
is generated by openfst.cmake
of Kaldi.
So I opened this issue.
I'm going to close this issue.
If Someone want trying to cross-compile OpenFST for Windows, you can apply the patch I pasted.
from openfst.
@hwiorn, this may be a good idea, thanks. We can put the patch into contrib/, refer to this discussion and note it's unmaintained.
I'm gonna rid of the /wholearchive
thing. Interesting, the upstream does not compile as an archive (.a), only as an SO, for the same reason this had to be added (an archive member is not even linked in a static build b/c nothing refers to its external symbols, but must be linked into an executable binary, as it registers basic FST kinds in its static initialization, such as vector o_O. CMake has what they call a "unity mode" when it concats a specified group of sources into a temporary single source and compiles it into a single .o/.obj file. I did that for another project already. SO/DLL does not have this problem, as they do the initialization of static BSS variables, and that object is file always linked in.
Often, CMake supports features without the last-resort constructs like "if this compiler, set this, else set this". I'll check if the bigobj thing is one of the properties. It's an MS-only thing, AFAICR, number of entries in a table of some file type (obj? lib?) was limited to 65536 or a few below, so they extended the format; still, it may be exposed by CMake. -O2
is added by selecting a Release build; hard-coding it is superfluous at best, and makes code practically undebuggable.
@jtrmal I have a feeling we'll do good to everyone if we add CMake tooling to our vanilla downstream, too. I'm a fan. It's ugly, naturally, but I think I tried all xplat build systems. Bazel firmly holds the first place as the most impenetrable build system of all. BTW, CMake's support for VS is especially horrible. I've got an impression that they generate build files from templates with very little understanding what they really do. That arguably had been sensible for VS 2010 and earlier (.vcproj); later versions integrated with MSBuild without quirks (.vcxproj). Probably build is just hard.
I didn't grok the idea about binary releases. N(compilers)×M(arch,bitness*,CPU)×2(static,dynamic*)×2(debug,release) is a large number of builds. [*not all platforms]. N(compilers) may be reduced to N'(ABIs), but that's still a buttload of builds.
BTW, on the same note, drop or keep VS 2017 support?
from openfst.
from openfst.
@jtrmal, some algos are pretty intensive. Interesting how SSE vs AVX vs AVX2 affects the perf. May be irrelevant, may be not. They aren't too obsessed with perf.
I had to apply a simple patch to shave 30%(!) from the HCLG build time. Since the graph had to be done preferably while the TTS was still flapping its jaw (if the caller spoke, I buffered, but shut-up-on-VAD was off until the recognizer was ready). You'd never guess where all the time had gone! :) Encode() and Decode() use the default initial capacity of the std::unordered_map, the one that stores weights—as many as there are edges. Starting with the default capacity (16, I think), and doubling it with a realloc as needed did not quite made it too efficient with millions of edges :) It's amazing how much seriously algorithmically complex stuff is going on, and you remember the docs, every op has a specified complexity, including even undecidable. And these two O(1) operations stole the 1/3 of the total runtime!
from openfst.
from openfst.
@jtrmal with this idea in mind, certainly.
Wait, I used to release the whole kaboodle, now doing only .exe files. But VC compiler ABI is much more stable now, so we can do the libs too.
from openfst.
Related Issues (20)
- ERROR: Unknown file read mode when reading VectorFst HOT 6
- Feature request: Publish package on chocolatey HOT 4
- stdout newline conversion of binary data HOT 13
- About the license of this repo HOT 2
- Building openfst and getting 36 errors, Please help me HOT 2
- Build errors on vector-fst.h line 611 when using Visual Studio 14 2015 HOT 3
- Cannot find source file when building from source with Cmake HOT 9
- Error reading Kaldi's GrammarFst HOT 5
- kkm: Bring the upstream up to date HOT 1
- Implement memory-mapped files
- Build ngram and other extensions HOT 4
- Proposed fixes for building with CMake and Clang-cl on Windows with HAVE_BIN=OFF HOT 7
- Access denied: Unable to start the program .../ALL_BUILD HOT 1
- Can't run on windows-xp HOT 1
- Cannot open include file: 'dlfcn.h': No such file or directory HOT 1
- OpenFST 1.8.0 port HOT 2
- openfst DLL LoadLibrary stuck HOT 1
- Please make a release
- ["-Wno-sign-compare", Wno-unused-local-typedefs] dont work in visual studio compiler
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from openfst.