numenta / nupic.core-legacy Goto Github PK

Tests at Travis are presenting errors when we use "make test_everything" instead of "$NTA/bin/testeverything" (or any other test).

This happens in nupic and nupic.core.

Version number is set incorrectly, which results in a crash. Should be set in the constructor.

After running git pull and git submodule update this morning, then removing my old nta dir and running cmake and then make per the README.md I get this error:

[ 1%] Building CXX object CMakeFiles/os.dir/nta/os/FStream.cpp.o
/usr/bin/c++ -mmacosx-version-min=10.7 -std=c++98 -I/Users/iandanforth/nupic -isystem/Users/iandanforth/nupic/external/common/include -isystem/Users/iandanforth/nupic/external/darwin64/include -fPIC -DPIC -m64 -DHAVE_CONFIG_H -DNTA_INTERNAL -DNTA_PLATFORM_darwin64 -DBOOST_NO_WREGEX -DNUPIC2 -fvisibility=hidden -Wall -Wreturn-type -Wunused -Wno-unused-parameter -gfull -DNTA_ASSERTIONS_ON -DNTA_PYTHON_SUPPORT=2.7 -isystem/Library/Python/2.7/site-packages/numpy/core/include -isystem/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -O3 -pipe -DNTA_ASM -o CMakeFiles/os.dir/nta/os/FStream.cpp.o -c /Users/iandanforth/nupic/nta/os/FStream.cpp
/Users/iandanforth/nupic/nta/os/FStream.cpp:186:19: error: cannot initialize a variable of type 'char ' with an rvalue of type 'int'
char retval = ::strerror_r(error, errbuf, 256);

In order for nupic.core to be stand-alone we'll need to reimplement most of encoders from nupic/python to c++.

My motivation for this are large-scale experiments, where python is eating huge amounts of memory.

For starters I suggest base.py and scalar encoder

Related to #7.

• Create base Encoder class
• Create C++ date/time encoder
• Create C++ scalar encoder
• Create C++ category encoder

Create a Coverity project like this for nupic.core: https://scan.coverity.com/projects/979

Depends on #4. Once build is running in travis, run

1. htmtests
2. testeverything

C++ complement for numenta/nupic-legacy#887.

If this file exists, when people file new issues through the issue tracker, there will automatically be a link to this file for people to read before they create a new issue:

Currently Doxygen in nupic.core specified a doxypy filter:

INPUT_FILTER           = "python /usr/local/bin/doxypy.py"

...

FILTER_SOURCE_FILES    = YES

And is also described at Documenting NuPIC with Doxygen .

For the following reason, I intend to remove it from nupic.core:

• it's used in nupic but has no effect in nupic.core, I've run a diff between the document generated with and without filter doxypy, the only difference is that doxypy appended a blank line after source files
• hard-coded platform-specific path isn't cross platform
• it introduced extra dependency to generate the API doc
• with support for python style doc, it might cause the confusion and people might write python style doc in pure C++ nupic.core

I'm not expecting strong objections, so I'll create the PR soon.

I think you should add a LICENSE file to the top-level directory that clarifies the license of the code. From looking at some headers I think your code uses the GPLv3, but a top-level license file is good practice.

Getting errors like this:

/Users/mtaylor/nta/nupic/nta/types/Exception.hpp:183:17: error: implicit instantiation of undefined template 'std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >'
    std::string filename_;
                ^

Seems like the version of clang I'm using is stricter than before. It is an easy fix, just add

#include <string>

to Exception.hpp.

UPDATE

The title has been changed to "Missing dependency zlib or more in external/common/include " to described the issue better, the original title was "Missing dependency checks in CMake such as zlib".

Original issue

While trying to give merged #47 a try in a koding.com VM(which is basically a Ubuntu variant) using gcc version 4.7.3 (Ubuntu/Linaro 4.7.3-1ubuntu1) and numenta/nupic.core's master@HEAD, I got the following compilation error:

utensilsong@vm-1:~/projects/nupic.core/build/scripts$ make -j3

...omitting some output...

[ 15%] Building CXX object CMakeFiles/nupic_core.dir/main/os/FStream.cpp.o
/home/utensilsong/projects/nupic.core/src/main/os/FStream.cpp:46:18: fatal error: zlib.h: No such file or directory
compilation terminated.
make[2]: *** [CMakeFiles/nupic_core.dir/main/os/FStream.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/nupic_core.dir/all] Error 2
make: *** [all] Error 2

It's trivial to fix it by sudo apt-get install zlibc zlib1g zlib1g-dev under Ubuntu, but this got me thinking, should CMake do some extra dependency checks? Don't know if there's any other dependencies, because I tried cleaning the VM environment and no other dependency problems surfaced.

cc @david-ragazzi @rhyolight

As planned in Documentation Priority Listing, nupic.core/src/main/engine/ requires some improvements .

I intend to start the work on this in the coming weeks, disclaimer:

• I have good knowledge about Doxygen and C++, along with some basic understanding of the code in nupic.core
• I'll absorb from NuPIC API A bird's eye view and NuPIC Architecture, and will try to keep the consistencies of wording
• because this work would be done only in spare time, and my job is quite busy, so this would not be quite timely, I might only finish 1 pass for 1 file per day
• any one who has better understanding of nupic.core and more time is welcome to take over and merge the part I've finished and to modify and create separate PRs if I don't finish it in time
• though I've reviewed all tests and docs of engine API, I'm still newbie to the engine API. What I would've done is most likely the basic part, and would rely on seniors to further improve, any help or input would be appreciated

Some plans about this:

• personally, this work would help me better understand nupic.core
• hopefully #76 and #79 could be merged before this PR, I don't know how git merges mv and modifications yet
• after this one, I would like to get to the part regarding adding tests for API or writing samples or encoders in C++ if spare time would not become an issue

Depends on https://github.com/numenta/nupic.tools/issues/64

This is definitely a bug:

/Users/spurdy/nta/nupic/nta/algorithms/spatial_pooler.cpp:59:10: warning: private field 'nrows_' is not used [-Wunused-private-field]
    UInt nrows_;

and this is also an issue:

/Users/spurdy/nta/nupic/qa/testeverything/../../nta/algorithms/unittests/SpatialPoolerTest.cpp:1358:10: warning: unused variable 'numColumnsPerInhArea' [-Wunused-variable]
    UInt numColumnsPerInhArea;

@subutai - Please take a look.

What is the problem we're trying to solve here?

Simply put, file naming inconsistency.

As discusssed in #41 (comment) , there're 4 source file naming styles in nupic.core (50 of "UpperCamelCase", 9 of "snake_case", 2 of "lowerCamelCase", 2 of "Camel_snake_case").

But C-Coding-Guide prefers "snake_case" :

Filenames should be all lower case. Multiple words should be separated by underscore, such as common_defs.hpp or runtime_interfaces.h .

Though this is really a trivial detail, this inconsistency would create "broken window effect" and puzzle new contributors. In the long run, it might lower the quality of nupic.core as an open source project.

What's the solution?

• Rename source files in nupic.core to follow one convention, namely UpperCamelCase since it is the majority.
• Document the new convention in C-Coding-Guide.

Does this solution solve the problem?

I believe this would be a good start.

Does it create new problems?

• whether can all agree on a single convention
• how to deal with Region_io.cpp and Region_parameters.cpp, because they are both implementing Region.hpp, along with Region.cpp.

I could think of no more.

As @breznak suggested at #41 (comment) and http://lists.numenta.org/pipermail/nupic-hackers_lists.numenta.org/2014-April/000521.html :

nupic.core should adopt or implement a lint tool and write rules to validate and ensure conventions .

I realize the input paths will need to be changes as the nupic.core structure changes, but I want to have this process in place.

array_algo.hpp includes <vecLib/vDSP.h> on Mac's. However this does not seem to build on all Mac / Mavericks environments. Possibly they need to install something. The optimizations offered by this in the current use cases are minimal. Given that we want to remove it for now until we can figure out a more portable way to add it in.

Currently boost and apr. Let's discuss how we should deal with removing these.

nupic.core currently only contains source files that used to exist in nupic. The project should have a build process so it can build itself, and it should be easily usable by other projects (like language bindings and/or clients) like nupic.

Once this is done, nupic should be able to call this build script as a part of its build process instead of doing the build within that repo.

profile CLAmodel, encoders, SP, TP, TM on some dataset and see CPU and memory hotspots.
Both for python and c++ version, while C++ is more suitable for speed, python is for better readability.

Depends on #9.

Current C++ unit tests only report to stdout and pass/fail via exit status. These tests need to generate xUnit style reports that can be published publicly. Here is a list of C++ unit testing frameworks.

Before we can properly document and test the API, we need to decide what is a part of the official API. I think this should start as a simple wiki doc that lists the public classes, functions, etc. that users creating clients would need to use to build something like the OPF.

https://github.com/numenta/nupic/wiki/NuPIC-Core-Network-API

Some useful references:

• NuPIC Architecture
• definition of the Network API
• 2014 January Core API Meeting Notes

@subutai I am assigning to you initially because you probably have the most knowledge to fill in the most content the fastest.

Provide examples about how to use nupic.core as in nupic.

Currently only the tests and how nupic uses nupic.core(but not directly, the wrapper make things less clear) could help people understand how to use nupic.core.

A few functions in array_algo.hpp already contain inline asm code for darwin32. This is an issue to support 64bit asm for Linux and Darwin for these functions.

Depends on #4.

A lot of compiler warnings show up. We should either update the code to prevent the warnings, or perhaps turn some of the warnings off with compiler flags. I like the former solution.

Seems this issue depends on numenta/nupic-legacy#584 and numenta/nupic-legacy#583 I would seem. IIRC weren't there others responsible for these tasks?

Secondly I'm aiming at simply extracting the nupic/nta folder into a new git repo as is. Good first step?

Another approach:
Extract then remove python and iterate on the build scripts etc.

Interestingly, I've hit a bug when building nupic in ubuntu 13.04 in Netbeans project. Related just to the Werror.

[ 3%] Building CXX object CMakeFiles/os.dir/nta/os/FStream.cpp.o
/home/nupic/nupic-source/nta/os/FStream.cpp: In static member function ‘static void* nta::ZLib::fopen(const string&, const string&, std::string_)’:
/home/nupic/nupic-source/nta/os/FStream.cpp:186:45: error: ignoring return value of ‘char_ strerror_r(int, char_, size_t)’, declared with attribute warn_unused_result [-Werror=unused-result]
cc1plus: all warnings being treated as errors
make[2]: *_* [CMakeFiles/os.dir/nta/os/FStream.cpp.o] Error 1

strerror_r seems unstandard, is there an alternative? I've thought strerror, but read somewhere it has problems with thread safety or sth...?

I'll provide a workaround that silences the warining, question to judge is the use of strerror_r.

If a user decides to modify this param they would run into a bug where they were not guaranteed to get 100% connected perms.

This is the corresponding CPP issue for numenta/nupic-legacy#679

As previously discussed.

In nupic, the CMakeLists.txt file is in the repo directory, whereas in this repo, the file is in the src directory. This leads to inconsistent build instructions. We should follow a single convention between the two.

As all repositories will build in a independent way very soon, I think a great step would be we uniformize code, naming, etc.

Look, for example, some cpp files names in a same folder which are inconsistent. In nupic.core/algorithms folder, for example, some file names are in uppercase ( SegmentUpdate.cpp) while others are in lowercase (bit_history.cpp). To be consistent, either SegmentUpdate.cpp must be renamed to segment_update.hpp or bit_history.cpp must be renamed to BitHistory.hpp.

These issues and others (variable naming, identing, etc) could be solved if we adopt a global and well accepted coding standard for C++ projects.

This said, I would suggest we discuss some conventions and elaborate a plan to refactoring c++ projects having the chosen convention as guideline. My favorite one is the Google Style Guide convention: http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml Although it contains rules which I don't like (braces indention, for example)

However, one can suggest others...

Depends on #4 and #9.

Once nupic.core exists, builds, and runs tests, create a Travis-CI project association for it that builds and runs all existing C++ tests.

nupic.core contains the core C++ library for NuPIC. We should have some example applications as well that demonstrate how to use it. Some suggestions:

• a simple "hello NuPIC" application demonstrating the simplest case. This might not do anything useful, but developers can use this as a base for building their own applications. (#142)
• a unix style command line interface program. This could take network configuration file as input and then accept data through stdin and write results to stdout. The program could be chained with pipes just like any other unix executable. Could be a very easy app for new users. @sjmackenzie's idea. (#175)
• maybe a more complex demo app that solves a real problem and demonstrates a more real world use case of the API and algorithms. (#176)

Ultimately we might want to split these into their own repo.

C++ complement to numenta/nupic-legacy#878

As discussed in numenta/nupic-legacy#591.

Related to #4.

From NuPIC:

I am seeing these warnings while building nupic. These need to get fixed. Abstract classes must have a destructors that are declared as virtual for proper clean-up. Here is an explanation: http://msdn.microsoft.com/en-us/library/wzxffy8c.aspx

algorithms_py.cpp:59810:7: warning: delete called on 'nta::algorithms::spatial_pooler::SpatialPooler' that has virtual functions but non-virtual destructor [-Wdelete-non-virtual-dtor]
      delete arg1;
      ^

algorithms_py.cpp:71610:7: warning: delete called on 'nta::algorithms::spatial_pooler::FlatSpatialPooler' that has virtual functions but non-virtual destructor [-Wdelete-non-virtual-dtor]
      delete arg1;

Doxygen files, initialized by @rhyolight in #21 , is now broken after #47 due to source structure change.

C++ complement to numenta/nupic-legacy#840

nupic.core-specific ticket for numenta/nupic-legacy#918

Right now, the numenta/nupic and numenta/nupic.core repositories are tightly coupled and have an integrated build process. It'd be nice if nupic.core were installable to a common location and reusable independent of the git repositories. Even better, there were binary releases that didn't require a git checkout.

This presumably means that there is a separate build and install command:

mkdir -p $NUPIC_CORE/build/scripts
cd $NUPIC_CORE/build/scripts
cmake $NUPIC_CORE/src
make -j3
make install

• Travis build that does something, even if no build exists
• IRC notifications for pushes to master and travis build statuses
• tooling server should start monitoring nupic.core
• issue tracker tags

Our build matrices have 4 jobs, but we only need 2: clang and gcc. This is because we're specifying both in compiler and env.matrix:

compiler:
  - clang
  - gcc

env:
  matrix:
    - CC=clang
    - CC=gcc

There must be a better way to do this. We should only have two builds, right?

From CMakeLists.txt:

option(NTA_OPTIMIZATION_ENABLED "--optimization=[ON/OFF] turn on optimization [default=ON]" ON)

It should read:

option(NTA_OPTIMIZATION_ENABLED "-DNTA_OPTIMIZATION_ENABLED=[ON/OFF] turn on optimization [default=ON]" ON)

Since -DNTA_OPTIMIZATION_ENABLED=[ON/OFF] is how the user would actually toggle that flag in the command line.

A C like API is the lowest common denominator, it covers the largest amount of languages.

This is the corresponding CPP issue for numenta/nupic-legacy#709.

It seems to be a regression of numenta/nupic-legacy#669 , reappeared at numenta/nupic.core@HEAD after merging #47, or before it, I don't know.

See also http://stackoverflow.com/a/19774902/200764

Details:

Linking CXX executable /home/utensilsong/projects/nupic.core/build/release/bin/testeverything
c++: error: unrecognized command line option ‘-stdlib=libstdc++’
make[2]: *** [/home/utensilsong/projects/nupic.core/build/release/bin/testeverything] Error 1
make[1]: *** [CMakeFiles/testeverything.dir/all] Error 2
make: *** [all] Error 2

Env:

~/projects/nupic.core/build/scripts$ c++ -v                                                                             
Using built-in specs.
COLLECT_GCC=c++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.7/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.7.3-1ubuntu1' --with-bugurl=file:///usr/share/doc/gcc-4.7/README.
Bugs --enable-languages=c,c++,go,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.7 --enable-shared --enable-linker-build-id --libe
xecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.7 --libdir=/usr/lib --enable-
nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --enable-plugin
 --with-system-zlib --enable-objc-gc --with-cloog --enable-cloog-backend=ppl --disable-cloog-version-check --disable-ppl-version-check --
enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --with-tune=generic --enable-check
ing=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.7.3 (Ubuntu/Linaro 4.7.3-1ubuntu1)

Seems this needs to be fixed before merging numenta/nupic-legacy#751 , cc @david-ragazzi @rhyolight

This super issue plans workflow for speed optimizations by using a specialized library.

Benefits:

• SPEED!
• fewer our-manually optimized (=hacked) code; cleanup
• bugs/improvements delegated to lib's upstream
• better portability
• use of parallel cores (openMP), special CPU instructions (SSE,..), GPGPU backends

Requirements:

• usability
  • suitable licence
  • platform support (Linux/Mac/Win; x86_64)
  • convenient installation/bundling with nupic
• functionality
  • SSE instructions
  • GPGPU backend support (CUDA, openCL)
  • parallelism support (openMP)
  • sparse matrices
• programming
  • clean & lean API
  • active development
  • (opt) bindings to other languages we use (Python)

Workflow:

1. decide on library implementation to use
2. create profiling/benchmark tools
   • I have some python SP/TP profiling tools that can use the c++ implementation and roughly show methods that are bottlenecks (sp_large.py tp_large.py)
   • proper c++ profiling setups
3. hello world usecase using the chosen lib
4. focus on Temporal pooler - the current bottleneck
   • replace ASM code with the library calls
   • find another for-loop candidates for vectorization
   • another function calls that can be sped up using the library?
5. Optimize Connections for Temporal memory
6. Optimize SparseMatrix classes (cleanup, mem reduction)
7. Optimize other (less significant parts)
   • Optimize Spatial pooler
8. Misc
   • enable multi threaded support
   • enable GPGPU backends
   • 2 step compilation for max raw performance - known constants