kxsystems / pyq

PyQ — Python for kdb+

PyQ - Python for kdb+

PyQ brings the Python programming language to the kdb+ database. Part of the Fusion for kdb+ interface collection.

It allows developers to integrate Python and q codes seamlessly in one application. This is achieved by bringing the Python and q interpreters into the same process so that code written in either of the languages operates on the same data. In PyQ, Python and q objects live in the same memory space and share the same data.

Please report issues in this repository.

Installation

pip install pyq

See detailed installation instructions.

Usage

For Python programmers:

$ pyq

>>> from pyq import q
>>> 1 + q.til(10)
k('1 2 3 4 5 6 7 8 9 10')

or run your Python script as

pyq [python options] python-script

For q programmers:

$ q

q)p)from math import hypot  / prefix python code with p)
q)p)q.h = hypot             / import a python function
q)h 3 4                     / call the python function from q
5f

Documentation

Documentation is available on the PyQ homepage.

Testing

Use tox to run tests.

cd path/to/pyq/source
tox

pyq's Issues

Using q.show in Jupyter Notebook

Using q.show in jupyter notebook outputs the value to the command prompt jupyter notebook is started from but doesn't output to the console in the browser, it just shows : :

python_dll not found

q.exe python.q brings an error saying "the specified module could not be found" p.k:3 (`p 2:(`p_init;1))python_dll

Fix documentation for macOS

There is a bug in macOS documentation for PyQ. mkvirtualenv is provided by virtualenvwrapper. When using virtualenv module, virtualenv command should be used instead.

-s argument to enable multithreading

Is there a way to specify the number of slaves prior to executing PyQ? The command

PYQ_BACKTRACE=yes pyq -s 14

does not work.

MacOS 'pip install pyq' shows ' RuntimeError: no python dll'

Questions

• [ Macos 64- bit]
• [ 64-bit kdb+ ]
• [ Install using virtual environmnet ]
• [ Anaconda 4.2.1]

i tried install by pip install pyq and from source code, they all show runtime error?

      File "/private/var/folders/k1/5yp6mmxd1_g2x57_qf_29f980000gn/T/pip-build-6ls07tww/pyq/setup.py", line 264, in get_python_dll
        raise RuntimeError('no python dll')
    RuntimeError: no python dll
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/k1/5yp6mmxd1_g2x57_qf_29f980000gn/T/pip-build-6ls07tww/pyq/

i see someone else have some error? but cannot find how to solve.

install pyq 4.0.2 on Redhat 2.6

Dear support,

We tries to install the latest version (4.0.2) of pyq on Redhat 2.6. But when launch pyq, I got share lib symbol linking issue. Is there a way to get around this?

Also, we tried to compile the pyq source (we did git clone https://github.com/enlnt/pyq ) but we could not find the Makefile. Is there a way to do the build ourselves?

Thanks for your help!

Minor issue in ,: append with pyq

Questions

• Which operating system are you using (if Linux, please provide flavour of it, i.e RedHat, CentOS or Ubuntu), is it 32-bit, or 64-bit?

  RedHat 64 bit
  

• Which version of PyQ are you running? Please provide output of pyq --versions, if PyQ isn't operational, please provide Python interpreter version and PyQ version python -V; python3 -V; pip list | grep pyq:

  python 2.7
  pyq version 4.1.3
  

• Which version of kdb+ are you using, is it 32-bit or 64-bit?

  64bit
  

  • If on 64-bit, is your QLIC set? Please provide output env | grep QLIC on linux/macOS, or set|grep QLIC on Windows.

    QLIC is set, but is NA
    

• Did you use virtual environment to install PyQ? If not, why?

  • no, installing into root environment so everyone can share pyq

• Where is your QHOME? Please provide output env | grep QHOME on linux/macOS, or set|grep QHOME on Windows.

  Qhome is set, but NA
  

• Do you use Conda? If so, what version?

  yes, 4.5.2
  

Steps to reproduce the issue

from pyq import q
def test(x,y):
    return {"x":x, "y":y}

q.set('f',test)
q('g:{d:`x`y!(x+x;y);d,:f (x;y)}')
q('g[4;5]')

Expected result

`x`y!4 5

Actual result

type error

Workaround

two workarounds are known
change function g to
g:{d:xy!(x+x;y);d:d,f (x;y)}
explicitly cast
g:{d:xy!(x+x;y); d,:`long$f(x,y)}

Using PyQ in PyCharm

Questions

• Which operating system are you using (if Linux, please provide flavour of it, i.e RedHat, CentOS or Ubuntu), is it 32-bit, or 64-bit?

  Red Hat Enterprise Linux Server release 6.9 (Santiago)

• Which version of PyQ are you running? You can use pyq --versions or pip list | grep pyq.

PyQ 4.1.3

• Which version of kdb+ are you using, is it 32-bit or 64-bit?

KDB+ 3.4 (2016.10.10) l64

• If on 64-bit, is your QLIC set? Please provide output env | grep QLIC on linux/macOS, or set|grep QLIC on Windows.

 QLIC=/opt/kx/3.4

• Did you use virtual environment to install PyQ? If not, why?

  Yes with Anaconda

Where is your QHOME? Please provide output env | grep QHOME on linux/macOS, or set|grep QHOME on Windows.

QHOME=/opt/kx/3.4

• Do you use Conda? If so, what version?

  Yes. conda 3.14.1

Steps to reproduce the issue

Is there a way to run PyQ as an interpreter in PyCharm? I am trying to run PyQ as an interpreter in PyCharm, but I got the following error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/svc_tpmdb_dev/pycharm/pycharm-community-2018.1.2/helpers/pydev/pydevd.py", line 1664, in <module>
    main()
  File "/home/svc_tpmdb_dev/pycharm/pycharm-community-2018.1.2/helpers/pydev/pydevd.py", line 1658, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/home/svc_tpmdb_dev/pycharm/pycharm-community-2018.1.2/helpers/pydev/pydevd.py", line 1068, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/home/svc_tpmdb_dev/pycharm/pycharm-community-2018.1.2/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 11, in execfile
    stream = tokenize.open(file)  # @UndefinedVariable
  File "/home/svc_tpmdb_dev/.conda/envs/py36an440/lib/python3.6/tokenize.py", line 454, in open
    encoding, lines = detect_encoding(buffer.readline)
  File "/home/svc_tpmdb_dev/.conda/envs/py36an440/lib/python3.6/tokenize.py", line 431, in detect_encoding
    encoding = find_cookie(first)
  File "/home/svc_tpmdb_dev/.conda/envs/py36an440/lib/python3.6/tokenize.py", line 395, in find_cookie
    raise SyntaxError(msg)
SyntaxError: invalid or missing encoding declaration for '/home/svc_tpmdb_dev/.conda/envs/py36an440/bin/pyq'
We've got an error while stopping in post-mortem: <class 'KeyboardInterrupt'>

Expected result

Actual result

Workaround

If you know workaround, please provide it here.

Cannot launch PyQ with 32bit kdb+ on OSX

Questions

• OSX 64-bit
• PyQ version 4.1.3
• kdb+ 32-bit
• Used virtualenv
• QHOME = /home/user/q/.
• Conda version 4.3.30

Steps to reproduce the issue

I was able to install PyQ by cloning the repository and using the workaround in this issue
##32 (comment). I ran

git clone https://github.com/kxsystems/pyq.git

added a setup.cfg file in root
[config] python_dll=libpython2.7m.so

It built fine:

Successfully built pyq
Installing collected packages: pyq
Successfully installed pyq-4.1.3

However when launching I get the error

$:~/q$ pyq
'2018.05.14T00:01:11.050 dlopen(/Users/q/m32/p.so, 2): no suitable image found.  Did find:
	/Users/q/m32/p.so: mach-o, but wrong architecture
  [4]  /Users/q/p.k:3: (`p 2:(`p_init;1))python_dll

Move full PyQ development to Github

Historically, PyQ was developed in-house using internal source-control system and CI (we utilized Gitlab). Our CI pipeline has more than 40 jobs which are run for each push into the repository. The tests are ran using multiple versions of both 32-bit and and 64-bit kdb+ (as of 2018.02.28 we testing with kdb+ 2.8., 3.2, 3.3, 3.4, 3.5 and 3.6t). Internal CI allowed us to simplify bootstrap process and not to worry about users getting access to kdb+ binaries — only internal users have access to the system.
With Travis, we have no secure way of bootstrapping kdb+, and this is main obstacle in moving kdb+ development to Github.

TODO

1. Come up with a secure way of bootstrapping kdb+ in public CI.
2. Migrate all jobs from the internal Gitlab CI to public CI
3. Implement testing of examples in PyQ documentation (KxSystems/docs)

Cannot run pyq with 64bit kdb+

Hi, I am trying to install pyq in a python virtualenv, following the instructions here, with a licensed copy of kdb (64bit).
Installation goes fine, but when I try to run pyq I get the following error:

(krypton_env_64) giovanni@mymachine ~$ pyq
qbinpath = /home/giovanni/workspace/krypton_env_64/q/l64/q
qbinpath = /home/giovanni/workspace/krypton_env_64/q/l64/q
qbinpath = /home/giovanni/workspace/krypton_env_64/q/l64/q
qbinpath = /home/giovanni/q/l64/q
No such file or directory

Although, as you can see, the q binary is there:

(krypton_env_64) giovanni@mymachine ~$ ls -R /home/giovanni/workspace/krypton_env_64/q/
/home/giovanni/workspace/krypton_env_64/q/:
k4.lic  l64  p.k  pyq-config.q  pyq-operators.q  python.q  q.k

/home/giovanni/workspace/krypton_env_64/q/l64:
p.so  q

Could you please advise on how to handle this issue?
I’m using Python 3.6.1 |Continuum Analytics, Inc.| (default, May 11 2017, 13:09:58) [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux

Asynchronous Call

I'm building a Python-based feedhandler. I start in a q session with the following

q) \p 5000
q) .u.upd:insert
q) tbl:([] a:`x`y;b:10 20)
q) num::count tbl
q) num
2

I then run the following script from PyQ

>>> from pyq import *
>>> conn = q.hopen("::5000")
>>> aconn = q.neg(conn) # Asynchronous Call Handle
>>> names = ["a","b","c"]
>>> nums = [1,2,3]
>>> k_names = K._ktn(11,0)
>>> k_nums=K._knt(6,0)
>>> for x in range(3):
        K._ja(k_names, names[x])
        K._ja(k_nums, nums[x])
>>> k_inputs = K._knk(2, k_names, k_nums)
>>> conn(K._knk(3, kp(".u.upd"), K._ks("tbl"), k_inputs))

This successfully add the records into the host environment table "tbl".

q) num
5

If I run the asynchronous call handle, however

>>> aconn(K._knk(3, kp(".u.upd"), K._ks("tbl"), k_inputs))

The table is not updated.

q) num
5

If I run the synchronous call again,

q) num
11

It seems that PyQ is caching my asynchronous call somewhere. Am I doing something wrong?

Importing PyQ in a sub thread

Which operating system are you using (if Linux, please provide flavour of it, i.e RedHat, CentOS or Ubuntu), is it 32-bit, or 64-bit?

Red Hat Enterprise Linux Server release 6.9 (Santiago)

Which version of PyQ are you running? You can use pyq --versions or pip list | grep pyq.

PyQ 4.1.3
Which version of kdb+ are you using, is it 32-bit or 64-bit?
KDB+ 3.4 (2016.10.10) l64
If on 64-bit, is your QLIC set? Please provide output env | grep QLIC on linux/macOS, or set|grep QLIC on Windows.
QLIC=/opt/kx/3.4
Did you use virtual environment to install PyQ? If not, why?

Yes with Anaconda

Where is your QHOME? Please provide output env | grep QHOME on linux/macOS, or set|grep QHOME on Windows.

QHOME=/opt/kx/3.4
Do you use Conda? If so, what version?

Yes. conda 3.14.1

I would like to start PyQ in a sub thread, because I need to free up my main thread to do other tasks. But I got the following errors when I attempted to start PyQ in sub thread.

Traceback (most recent call last):
File "/home/svc_tpmdb_dev/.conda/envs/py36an440/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2963, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 1, in
self.q('\l ' + self._conn)
File "/home/svc_tpmdb_dev/.conda/envs/py36an440/lib/python3.6/site-packages/pyq/init.py", line 646, in call
return K._k(0, m, *map(K, args))
_k.error: sys

Interrupting the pyq-kernel

Versions:
PyQ 4.1.13
Python 2.7
kdb+ 3.5

I have noticed that trying to kill a python call in PyQ is significantly slower than killing it in Python. I am wondering how interrupting a python process running on PyQ kernel differs to interrupting a plain python process on jupyter. I understand that it is a python process running on a kdb+ instance, but trying to stop a running cell either causes the session to hang or remains idle. Does ctrl+C on a python process on PyQ send a SIGINT to the underlying kdb session? If so, is there any way to map ctrl+C to comint-interrupt-subjob so that Python can raise a KeyboardInterrupt exception and leave the kdb process intact?

Error importing SSL in PyQ

Hi,

I am getting a ssl import error when importing _ssl using PyQ. Please see the error below.

import _ssl

ImportError: /lib/python3.6/lib-dynload/_ssl.cpython-36m-x86_64-linux-gnu.so: undefined symbol: SSL_CTX_set_alpn_protos

I don't have the same issue if I were to use Python executable directly to run my code while keeping all environment variable the same. Are you seeing the same issue in the past? Just wondering if any SSL library is overridden in PyQ?

Thanks

Jeffrey

Cannot get QVER

when I import the package pyq into my python program, It throws an error:
"Traceback (most recent call last):
File "", line 1, in
File "C:\Python27\lib\site-packages\pyq-3.8.4-py2.7-win32.egg\pyq_init_.py", line 96, in
_ij = {'i': 'i', 'j': ''} if QVER[0] >= '3' else {'i': '', 'j': 'j'}
TypeError: 'NoneType' object has no attribute 'getitem'"
I execute the file" init.py", and find the parameter "QVER" was none, what's the problem??

adding document for ipc

Please add some document on how to use ipc. I figured it out by seach hopen in the repo

>>> h = q.hopen('::9900')
>>>  h(q('"1 + 1"'))
K('3')
>>>  h((q('+'), 1, 2)))
K('3')

Checking for python_dll makes pip installation impossible if you need to override python_dll

Steps to reproduce the issue

install pyq through pip on a system where python has been built statically

You can attempt to override the python_dll path by using --global-options=configure --global-options="--python-dll=/my/path/to/libs" but it will fail because pip install runs python setup.py egg_info before running python setup.py install

Expected result

Don't throw an error when checking for python_dll in commands that don't require it.

AttributeError: module 'pexpect' has no attribute 'spawn'

When launching ipython notebook on my windows machine with the KDB kernel I get the following error message when the kernel shuts down almost immediately.

Any suggestions what could cause this?

[I 18:31:39.365 NotebookApp] Kernel started: 1d8f3d63-a289-46b9-94ca-0f53521b9a02
Traceback (most recent call last):
  File "C:\anaconda3\envs\databases32\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\anaconda3\envs\databases32\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\anaconda3\envs\databases32\lib\site-packages\kdbq_kernel\__main__.py", line 3, in <module>
    IPKernelApp.launch_instance(kernel_class=KdbQKernel)
  File "C:\anaconda3\envs\databases32\lib\site-packages\traitlets\config\application.py", line 657, in launch_instance
    app.initialize(argv)
  File "<decorator-gen-121>", line 2, in initialize
  File "C:\anaconda3\envs\databases32\lib\site-packages\traitlets\config\application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "C:\anaconda3\envs\databases32\lib\site-packages\ipykernel\kernelapp.py", line 465, in initialize
    self.init_kernel()
  File "C:\anaconda3\envs\databases32\lib\site-packages\ipykernel\kernelapp.py", line 376, in init_kernel
    user_ns=self.user_ns,
  File "C:\anaconda3\envs\databases32\lib\site-packages\traitlets\config\configurable.py", line 412, in instance
    inst = cls(*args, **kwargs)
  File "C:\anaconda3\envs\databases32\lib\site-packages\kdbq_kernel\kernel.py", line 106, in __init__
    self._start_kdbq()
  File "C:\anaconda3\envs\databases32\lib\site-packages\kdbq_kernel\kernel.py", line 127, in _start_kdbq
    child = pexpect.spawn("q", echo=False, encoding='utf-8')
AttributeError: module 'pexpect' has no attribute 'spawn'
[I 18:31:42.327 NotebookApp] KernelRestarter: restarting kernel (1/5), new random ports

failed to import _ssl problem

In my environment, pyq failed to import _ssl, but python succeeded:

$ pyq --versions
PyQ 4.1.1
NumPy 1.12.1
KDB+ 3.5 (2017.05.02) l64
Python 2.7.13 |Anaconda 4.4.0 (64-bit)| (default, Dec 20 2016, 23:09:15)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]

$ python
Python 2.7.13 |Anaconda 4.4.0 (64-bit)| (default, Dec 20 2016, 23:09:15)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import _ssl
>>> _ssl.__file__
'/data/statarb_n/equity_sys/anaconda2/lib/python2.7/lib-dynload/_ssl.so'

$ ls -l /data/statarb_n/equity_sys/anaconda2/lib/python2.7/lib-dynload/_ssl.so
-rwxr-xr-x 2 equity_sys statarb 246877 Dec 21  2016 /data/statarb_n/equity_sys/anaconda2/lib/python2.7/lib-dynload/_ssl.so

$ pyq
Python 2.7.13 |Anaconda 4.4.0 (64-bit)| (default, Dec 20 2016, 23:09:15)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import _ssl
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/statarb_n/equity_sys/anaconda2/lib/python2.7/site-packages/pyq/__init__.py", line 774, in __import__
    m = _imp(name, globals, locals, fromlist, level)
ImportError: /data/statarb_n/equity_sys/anaconda2/lib/python2.7/lib-dynload/_ssl.so: undefined symbol: SSL_get0_next_proto_negotiated
>>>

OrderedDict bug in 4.1.1

from statsmodels.compat.collections import OrderedDict

threw an exception:

/site-packages/pyq/__init__.py", line 777, in <genexpr>
    _c.update((getattr(m, cname), conv) for cname, conv in pairs)
AttributeError: 'module' object has no attribute 'OrderedDict'

I think we used to have this bug in 3.8.x version but it has been fixed. Not sure why it comes back again.

Thanks a lot!

Issues installing PyQ on Win10

I'm trying to install PyQ on a 64-bit Win 10 system, python v 3.6.6. I currently have the 32-bit version of kdb+ installed.

I'd initially run into an error indicating I was missing the VS 14 build tools. The link provided was dead (I'm guessing as it point to the VS2015 build tools and VS2017 had been released) so I DLed and installed the the VS2017 build tools. This didn't fix the issue. I tried installing VS 2017 which had some additional python support. This fixed the original issue, but gave rise to another related unresolved externals, which I was able to resolve by running from developer cmd.

I was able to resolve that issue but am now running into an error stating:
error: command 'cl.exe' failed: No such file or directory

In short, from what I've seen and also following the issues posted here on git, Windows support for PyQ isn't a primary concern nor is it known to play particularly well with Windows natively. As I have the option of dual booting into a Linux system, before I continue down this rabbit hole, am I likely going to be better off by simply attempting a Ubuntu install? I'd considered simply moving all my development activities to Linux anyway as I've run into a number of issues setting up my system on Windows.

Any advice on resolving the error or input on the relative suitability of PyQ (and kdb and python by extension) to Windows versus Linux is appreciated.

Regards,
Chris

Multiple lines of q codes in Jupyter

Questions

Which operating system are you using (if Linux, please provide flavour of it, i.e RedHat, CentOS or Ubuntu), is it 32-bit, or 64-bit?

Red Hat Enterprise Linux Server release 6.9 (Santiago)

Which version of PyQ are you running? You can use pyq --versions or pip list | grep pyq.

PyQ 4.1.3
Which version of kdb+ are you using, is it 32-bit or 64-bit?
KDB+ 3.4 (2016.10.10) l64
If on 64-bit, is your QLIC set? Please provide output env | grep QLIC on linux/macOS, or set|grep QLIC on Windows.
QLIC=/opt/kx/3.4
Did you use virtual environment to install PyQ? If not, why?

Yes with Anaconda

Where is your QHOME? Please provide output env | grep QHOME on linux/macOS, or set|grep QHOME on Windows.

QHOME=/opt/kx/3.4
Do you use Conda? If so, what version?

Yes. conda 3.14.1

Hi,

What would be the correct syntax if I wish to write multiple lines of q codes (e.g. defining a q function that spam multiple lines) in Jupyter? %q seems to parse single line at a time only.

For example the function definition below will not work in Jupyter.

%q .funCust:{[a]
%q b:a+1;
%q :b;
%q };

Thanks

Jeffrey

Symbol is broken when imported into Pandas DataFrame

The following produce different result:

import pyq,pandas
pyq.q("sym:`a`b`c`d")
dict(pyq.q("flip ([]sym:`sym$`a`b`c`d;upd:til 4)"))['sym']

It returns a list of k symbol, which is expected:

k('`sym$`a`b`c`d')

But when imported into DataFrame, it becomes integer:

import pyq,pandas
pyq.q("sym:`a`b`c`d")
pandas.DataFrame(dict(pyq.q("flip ([]sym:`sym$`a`b`c`d;upd:til 4)")))['sym']

0    0
1    1
2    2
3    3

This is not right. I understand that the underlying enum is an integer, but enumerated sym should return as string in pandas, rather than a number.

undefined symbol error installing pyq 4.1 on ubuntu

I got the following errors while installing pyq on ubuntu (Linux ubuntu.fmr.com 4.4.0-96-generic #119-Ubuntu SMP Tue Sep 12 14:59:54 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux). I am upgrading from pyq 4.0.2. I can confirm 4.0.2 does not have this issue.

Thanks,
Ray

(env_py3) root@ubuntu:/opt# pyq
Traceback (most recent call last):
  File "/opt/anaconda3/envs/env_py3/lib/python3.6/site-packages/pyq/__init__.py", line 22, in <module>
    from ._k import K as _K, error as kerr, Q_VERSION, Q_DATE, Q_OS
ImportError: /opt/anaconda3/envs/env_py3/lib/python3.6/site-packages/pyq/_k.cpython-36m-x86_64-linux-gnu.so: undefined symbol: clr

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/anaconda3/envs/env_py3/lib/python3.6/site-packages/pyq/__init__.py", line 31, in <module>
    raise ImportError(message)
ImportError: Importing pyq from stock python is not supported. Use pyq executable.
'2017.09.21T12:42:31.611 python

Allow installation of embedPy alongside PyQ

Currently, both PyQ and embedPy install p.so in QHOME and this name conflict means that users cannot use both in the same kdb+ instance.

Why would users want to have both PyQ and embedPy? While the two projects provide some overlapping functionality, PyQ is primarily focused on accessing kdb+ data from Python while embedPy has more features for q programers to call Python functions.

Implementation plan

• rename p.so in PyQ to pyq.so
• provide means of (optionally?) installing embedPy components into a PyQ environment, e.g, pip install pyq[embedpy]

ModuleNotFoundError: No module named 'pyq._k'

Questions

• Which operating system are you using (if Linux, please provide flavour of it, i.e RedHat, CentOS or Ubuntu), is it 32-bit, or 64-bit?
  Linux (ArchLinux based without systemd - Artix Linux)

  64-bit

• Which version of PyQ are you running? Please provide output of pyq --versions, if PyQ isn't operational, please provide Python interpreter version and PyQ version python -V; python3 -V; pip list | grep pyq:

  Python 3.6.5
  pyq 4.1.3

• Which version of kdb+ are you using, is it 32-bit or 64-bit?

  KDB+ 3.6

  32-bit

  • If on 64-bit, is your QLIC set? Please provide output env | grep QLIC on linux/macOS, or set|grep QLIC on Windows.

• Did you use virtual environment to install PyQ? If not, why?

  • yes

• Where is your QHOME? Please provide output env | grep QHOME on linux/macOS, or set|grep QHOME on Windows.

  $HOME/pyq/q
  nb: my virtualenv is $HOME/pyq

• Do you use Conda? If so, what version?

  no

Steps to reproduce the issue

• compile and install python 3.6 with CFLAGS=-m32 and ldflags=-m32
• install virtualenv with this python 32bits
• create a virtualenv and activate it
• copy the kdb 32bits folder on it and set corresponding QHOME
• do a pip3 install pyq -> no erro
• start pyq -> gives error: ModuleNotFoundError: No module named 'pyq._k'

Expected result

Actual result

Workaround

If you know workaround, please provide it here.

Is it possible to return Python output into KDB?

Hello,

Using PyQ from inside a running q process, is it possible at all to return evaluated data from Python in an anonymous and dynamic way?

For example (from q):

q)show .p.e "6*7"

I know that I can send data from Python into q using q global variables as follows:

q)p)q.my_variable = 6*7
q)show my_variable

But this is a very inelegant solution, and would not scale well for large data science operations.

Ideally, something like .p.pyeval in embedPy but for PyQ. Is there anything like this I could use?

I would use embedPy if I could, but I'm on Windows, and while I can get the p.dll from embedPy to compile, it always crashes whenever I try to use it. PyQ I can successfully compile and operate on Windows.

I asked a similar question to this last year, and at the time, this was the ultimate answer from @abalkin

If you want to use .p.e or p) syntax, then you are restricted to communication via global variables.

Is this restriction still the case for PyQ 4.1.2?

Pandas incompatibility

OSX 64-bit
PyQ version 4.1.2
kdb+ 32-bit
Used virtualenv
QHOME = /home/user/q/.
Conda version 4.3.30

I noticed a bug in the pandas indexing engine when pyq is imported. There seems to be a conflict in how categorical Interval indexes are handled.

import pandas as pd
import numpy as np

t = pd.DataFrame(dict(sym=np.arange(2), y=1., z=-1.))
t.loc[:, 'x'] = pd.Series([pd.Interval(-1., 0.0, closed='right'), pd.Interval(0.0, 1, closed='right')])
t.set_index('x', inplace=True)
t.index = pd.Categorical(t.index)
t.loc[t.index.categories[0], :]

This returns

sym    0.0
y      1.0
z     -1.0
Name: (-1.0, 0.0], dtype: float64

in a Python 2.7 environment. However, in a pyq IPython environment, adding a pyq import statement

import pandas as pd
import numpy as np
from pyq import q

t = pd.DataFrame(dict(sym=np.arange(2), y=1., z=-1.))
t.loc[:, 'x'] = pd.Series([pd.Interval(-1., 0.0, closed='right'), pd.Interval(0.0, 1, closed='right')])
t.set_index('x', inplace=True)
t.index = pd.Categorical(t.index)
t.loc[t.index.categories[0], :]

results in a type error

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()
TypeError: 'slice(0,2,None)' is an invalid key

So I suspect that the keys in a pandas categorical index are coerced into integer slices, which then cannot be used for index slicing.

I am using
pandas 0.22
PyQ 4.1.2
numpy 1.14.2
IPython 5.3.0
kdb+ 3.5 64bit

Installing on Ubuntu 16.0.4

Append /sbin to PATH; otherwise pip3 install pyq fails to find ldconfig.

Create version of PyQ for Windows?

Hi
I am evaluating kdb+/q for use in managing large data volumes for an investment process and was keen on utilising PyQ to help manage data wrangling into a kdb database. I am restricted to operating on Windows and noticed that the pip install or build from source won't work as it is written only for Unix variants OSs. Is there any chance this could be extended to Windows? I'd have a crack at it myself, but my programming skills not that extensive .. Would it require much work to adjust to compile on WIndows? I am running Python 3.4

TypeError: 'NoneType' object has no attribute '__getitem__'

My python enviroments: python27, win32
After I import pyq into my python console, it throws:
Traceback (most recent call last):
File "", line 1, in
File "C:\Python27\lib\site-packages\pyq-3.8.4-py2.7-win32.egg\pyq_init_.py", line 96, in
_ij = {'i': 'i', 'j': ''} if QVER[0] >= '3' else {'i': '', 'j': 'j'}
TypeError: 'NoneType' object has no attribute 'getitem'

I really don''t know why? can anyone helps me ?

Code sample to convert Pandas dataframe to KDB table

Hi,

I am trying to find the best practices for converting pandas dataframe back to a kdb table using pyq. Could you please provide an example or a link to the examples?

Thanks

Jeffrey

numpy matrix

Has anyone seen examples of passing numpy/panda matrices into KDB and vice versa?

Suggestion: separate the package to two

Hello:
We are in a situation that the q is installed in a place that we don't have write access.
I'm wondering if we can split the pyq package into 2 parts: those installed to q, and those into virtual environment.

Regards

Send data to python from q function scope?

Hello,

Is it at all possible to set/get a variable in python from q, when running in a q function, meaning the data has no global q scope? Perhaps using e0 inside p.k? This would be the identical of using Rset/Rget in the Kx rserver package to communicate with R from KDB. So what I want to do is this:

q)myFunc:{[x;y]
         set_variable_in_python[x;"py_var_x"];
         set_variable_in_python[y;"py_var_y"];
         // check vars are in Python
        .p.e["print(py_var_x)"];.p.e["print(py_var_y)"];
        // run some crazy python func
        .p.e["res = some_py_func(py_var_x,py_var_y)"];
        // fetch res from python
        kdb_res:get_var_from_python["res"];
        :kdb_res;
};

Is anything like this possible using PyQ? I know that if I use global scope, it can work by using pyq.q(), even if called from q, like this:

q)myFunc:{[x]
         global_x::x;
         .p.e["q.global_res = some_py_func(q('global_x'))"];
         :global_res;
};

However this is not efficient for a number of reasons, chiefly in that it prohibits using myFunc with peach.

Basically, is there something like the equivalent of pyq.q() but from the KDB side, in order to get and set variables into python? I want to do everything which pyq allows one to do from Python->KDB, but going the other way, KDB->python. Not sure if pyq can do this or not?

How to turn on widgetsnbextension from pyq -m notebook?

How do I do the pyq equivalent from the command line?
jupyter nbextension enable --py --sys-prefix widgetsnbextension

Thanks,
Ray

installing pyq on redhat 2.7 with anaconda 5.0

I tried to install pyq with python 3.6 under anaconda 5.0. I got the following error:

Processing /apps/download/pyq-master
    Complete output from command python setup.py egg_info:
    running egg_info
    creating pip-egg-info/pyq.egg-info
    writing pip-egg-info/pyq.egg-info/PKG-INFO
    writing dependency_links to pip-egg-info/pyq.egg-info/dependency_links.txt
    writing requirements to pip-egg-info/pyq.egg-info/requires.txt
    writing top-level names to pip-egg-info/pyq.egg-info/top_level.txt
    writing manifest file 'pip-egg-info/pyq.egg-info/SOURCES.txt'
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-29sp_qa8-build/setup.py", line 658, in <module>
        run_setup(METADATA)
      File "/tmp/pip-29sp_qa8-build/setup.py", line 654, in run_setup
        setup(**keywords)
      File "/apps/anaconda3/envs/env_py3/lib/python3.6/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/apps/anaconda3/envs/env_py3/lib/python3.6/distutils/dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "/apps/anaconda3/envs/env_py3/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/apps/anaconda3/envs/env_py3/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 280, in run
        self.find_sources()
      File "/apps/anaconda3/envs/env_py3/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 295, in find_sources
        mm.run()
      File "/apps/anaconda3/envs/env_py3/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 526, in run
        self.add_defaults()
      File "/apps/anaconda3/envs/env_py3/lib/python3.6/site-packages/setuptools/command/egg_info.py", line 562, in add_defaults
        sdist.add_defaults(self)
      File "/apps/anaconda3/envs/env_py3/lib/python3.6/site-packages/setuptools/command/py36compat.py", line 36, in add_defaults
        self._add_defaults_ext()
      File "/apps/anaconda3/envs/env_py3/lib/python3.6/site-packages/setuptools/command/py36compat.py", line 119, in _add_defaults_ext
        build_ext = self.get_finalized_command('build_ext')
      File "/apps/anaconda3/envs/env_py3/lib/python3.6/distutils/cmd.py", line 299, in get_finalized_command
        cmd_obj.ensure_finalized()
      File "/apps/anaconda3/envs/env_py3/lib/python3.6/distutils/cmd.py", line 107, in ensure_finalized
        self.finalize_options()
      File "/tmp/pip-29sp_qa8-build/setup.py", line 449, in finalize_options
        ('define', 'define'))
      File "/apps/anaconda3/envs/env_py3/lib/python3.6/distutils/cmd.py", line 287, in set_undefined_options
        src_cmd_obj.ensure_finalized()
      File "/apps/anaconda3/envs/env_py3/lib/python3.6/distutils/cmd.py", line 107, in ensure_finalized
        self.finalize_options()
      File "/tmp/pip-29sp_qa8-build/setup.py", line 384, in finalize_options
        ('q_version', 'q_version'))
      File "/apps/anaconda3/envs/env_py3/lib/python3.6/distutils/cmd.py", line 287, in set_undefined_options
        src_cmd_obj.ensure_finalized()
      File "/apps/anaconda3/envs/env_py3/lib/python3.6/distutils/cmd.py", line 107, in ensure_finalized
        self.finalize_options()
      File "/tmp/pip-29sp_qa8-build/setup.py", line 284, in finalize_options
        self.python_dll = get_python_dll(sys.executable)
      File "/tmp/pip-29sp_qa8-build/setup.py", line 242, in get_python_dll
        raise RuntimeError('no python dll')
    RuntimeError: no python dll
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-29sp_qa8-build/

I took a look at setup.py, looks like pyq installer is looking for python shared library libpython3.6m.so by using ldd But in this case, the share lib has been statically compiled into python3.6 executable. How do I get around this issue?

/apps/anaconda3/lib$ ldd /apps/anaconda3/bin/python
	linux-vdso.so.1 =>  (0x00007ffd2fd37000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fb7daf6d000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fb7dabab000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007fb7da9a7000)
	libutil.so.1 => /lib64/libutil.so.1 (0x00007fb7da7a4000)
	librt.so.1 => /lib64/librt.so.1 (0x00007fb7da59b000)
	libm.so.6 => /lib64/libm.so.6 (0x00007fb7da299000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fb7db6ea000)

pyq does not work with pydash

I found that pyq does not work with pydash, which has its own definition for collections. I don't know how to workaround this problem, should I log an issue for pydash?

$ pyq --versions
PyQ 4.1.1
NumPy 1.12.1
KDB+ 3.5 (2017.05.02) l64
Python 2.7.13 |Anaconda 4.4.0 (64-bit)| (default, Dec 20 2016, 23:09:15)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
[18:00:38]azhang@wplsngen18[env=conda.q35][~]

$ pyq
Python 2.7.13 |Anaconda 4.4.0 (64-bit)| (default, Dec 20 2016, 23:09:15)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import pydash
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/statarb_n/equity_sys/anaconda2/lib/python2.7/site-packages/pyq/__init__.py", line 774, in __import__
    m = _imp(name, globals, locals, fromlist, level)
  File "/data/statarb_n/equity_sys/anaconda2/lib/python2.7/site-packages/pydash/__init__.py", line 114, in <module>
    from .collections import (
  File "/data/statarb_n/equity_sys/anaconda2/lib/python2.7/site-packages/pyq/__init__.py", line 777, in __import__
    _c.update((getattr(m, cname), conv) for cname, conv in pairs)
  File "/data/statarb_n/equity_sys/anaconda2/lib/python2.7/site-packages/pyq/__init__.py", line 777, in <genexpr>
    _c.update((getattr(m, cname), conv) for cname, conv in pairs)
AttributeError: 'module' object has no attribute 'OrderedDict'

Segmentation fault when using threading

• Linux version:
  64-bit Ubuntu

• PyQ version:
  PyQ 4.1.3
  NumPy 1.13.3
  KDB+ 3.5 (2018.02.26) l64
  Python 2.7.12 (default, Dec 4 2017, 14:50:18)
  [GCC 5.4.0 20160609]

• kdb+ version:
  64-bit kdb+
  QLIC=/home/zakariyya/q/q64

• Not install via virtual environment

• QHOME=/home/zakariyya/q/q64

• Not using Conda

Hi guys,

I'm getting a segmentation fault when running a q process that loads in a Python script (.p)
The use-case if fairly simple. I'm exposing Python subscription and unsubscription functions for Redis to q, then I call them so that they can run in a separate thread. That means that q can still work in "parallel"

$ cat redis.q
system "l /home/zakariyya/redis_thread.p";
.z.pc:{ if[x ~ hndl; -2" lost handle to tp"; if[`res in key `.;unsubscribe_redis enlist (res)] ] };
.z.exit:{ if[`res in key `.;unsubscribe_redis enlist (res)]; };
hndl:hopen 6006;
upd:{[t;d] data:d; neg[hndl](`upd;`tbl; data)};
show .z.i
$ q redis.q
KDB+ 3.5 2018.02.26 Copyright (C) 1993-2018 Kx Systems
l64/ 8()core 15999MB zakariyya zakariyya-pc-2193 127.0.1.1 EXPIRE 2018.07.01 zak********* KOD #51578

3645i
q)res:subscribe_redis("quote*";"quote")

And the Python script

$ cat redis_thread.p
import threading
import sys
import time
import redis
import signal
import os
from functools import partial
from pyq import q, K, _k

class PublisherThread(threading.Thread):

    def __init__(self, channel, r, kdb_table):
        super(PublisherThread, self).__init__()
        self._stopper = threading.Event()
        self.channel = channel
        self.kdb_table = kdb_table
        self.redis = r
        self.pubsub = self.redis.pubsub()
        self.redis.client_setname("kdb-feed-" + self.kdb_table)
        self.pubsub.psubscribe(self.channel)

    def stop(self):
        print('Closing Redis connection...')
        self.pubsub.close()
        self._stopper = True

    @property
    def stopped(self):
        return self._stopper.isSet()

    def run(self):
        while not self.stopped:
            try:
                msg = self.pubsub.get_message()
                if msg:
                    if msg['type'] in ('message', 'pmessage'):
                        #print(msg)
                        qmsg = K.string(msg)
                        q('upd', self.kdb_table, qmsg)
                        time.sleep(0.001)
            except _k.error as e:
                print('Caught Q error. Cannot insert data to table')
                self.stop()
            except Exception as e:
                print('Received unhandled exception. Cannot insert data to table')
                self.stop()


class RedisManager(object):

    def __init__(self, subscriber_init):
        self._subscribers_store = {}
        self.subscriber_init = subscriber_init

    def add(self, feed, kdb_table):
       print('Subscribing')
        key = ':'.join([feed, kdb_table])
        self._subscribers_store[key] = self.subscriber_init(feed, kdb_table)
        return key

    def remove(self, key):
        self._subscribers_store[key].stop()
        del self._subscribers_store[key]
        return True


def subscriber_init(feed, kdb_table, redis_client):
    t = PublisherThread(feed, redis_client, kdb_table)
    t.start()
    return t


# Create curried function of RedisManager's init
redis_manager = RedisManager(
    partial(subscriber_init,
         redis_client=redis.StrictRedis(host='XX.XXX.XXX.XXX', port=6399, db=0)))

# Create and expose Python functions a q callables

def q_subscribe_redis(feed, kdb_table):
    return redis_manager.add(str(feed), str(kdb_table))

def q_unsubscribe_redis(key):
    return redis_manager.remove(str(key))


q.subscribe_redis = q_subscribe_redis
q.unsubscribe_redis = q_unsubscribe_redis

And the segmentation fault:

q)Sorry, this application or an associated library has encountered a fatal error and will exit.
If known, please email the steps to reproduce this error to [email protected]
with a copy of the kdb+ startup banner.
Thank you.
/home/zakariyya/q/q64/l64/q() [0x47a8b1]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390) [0x7fc0aa739390]
/home/zakariyya/q/q64/l64/q(r0+0) [0x41bd40]
/home/zakariyya/q/q64/l64/q() [0x408a9c]
/home/zakariyya/q/q64/l64/q() [0x40fb6f]
/home/zakariyya/q/q64/l64/q() [0x4042c8]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7fc0aa37e830]
/home/zakariyya/q/q64/l64/q() [0x4043a1]
rlwrap: warning: q crashed, killed by SIGSEGV (core dumped).
rlwrap itself has not crashed, but for transparency,
it will now kill itself with the same signal


warnings can be silenced by the --no-warnings (-n) option
Segmentation fault (core dumped)

cann't get QVER

when I import the packages "pyq" in python, It throws an error "Traceback (most recent call last):
File "", line 1, in
File "c:\python27\lib\site-packages\pyq-3.8.4-py2.7-win32.egg\pyq_init_.py", line 96, in
_ij = {'i': 'i', 'j': ''} if QVER[0] >= '3' else {'i': '', 'j': 'j'}
TypeError: 'NoneType' object has no attribute 'getitem'"
I find that "QVER = os.environ.get('QVER')" cannot get QVER, what's the problem?

PyQ not picking up CPUS env variable when running as a Jupyter kernel

Questions

• Which operating system are you using (if Linux, please provide flavour of it, i.e RedHat, CentOS or Ubuntu), is it 32-bit, or 64-bit?
  Red Hat Enterprise Linux Server release 6.9 (Santiago)

• Which version of PyQ are you running? You can use pyq --versions or pip list | grep pyq.
  PyQ 4.1.3

• Which version of kdb+ are you using, is it 32-bit or 64-bit?
  KDB+ 3.4 (2016.10.10) l64

• If on 64-bit, is your QLIC set? Please provide output env | grep QLIC on linux/macOS, or set|grep QLIC on Windows.
  QLIC=/opt/kx/3.4

• Did you use virtual environment to install PyQ? If not, why?
  Yes with Anaconda

• Where is your QHOME? Please provide output env | grep QHOME on linux/macOS, or set|grep QHOME on Windows.
  QHOME=/opt/kx/3.4

• Do you use Conda? If so, what version?
  Yes. conda 3.14.1

Steps to reproduce the issue

1. install pyq 3 as a kernel in Jupyter
2. set environment variable CPUS=3 in jupyter kernel.json (i.e. /home/dev/.local/share/jupyter/kernels/pyq_3/kernel.json) See http://jupyter-client.readthedocs.io/en/stable/kernels.html for more detail

This is what I have set

{
    "argv": [
        "/opt/kx/3.4/l64/q",
        "pyq-kernel.p",
        "1024",
        "65535",
        "{connection_file}"
    ],
    "display_name": "PyQ-3",
    "env": {
        "CPUS": "3",
        "QHOME": "/opt/kx/3.4",
        "QLIC": "/opt/kx/3.4"
    },
    "language": "python"
}

Expected result

Confirm that the variable has been picked up by starting a new notebook in Jupyter with PyQ 3 kernel. The new notebook should start successfully with kernel ready

Actual result

kernel error with the following error message from Jupyter

[I 17:25:41.354 NotebookApp] Creating new notebook in
[I 17:25:41.736 NotebookApp] Kernel started: 5c3c1f5f-2124-4507-b1bb-044e9e0af35f
'2018.05.08T17:25:41.745 cores
[I 17:25:44.737 NotebookApp] KernelRestarter: restarting kernel (1/5)
'2018.05.08T17:25:44.752 cores
[I 17:25:47.747 NotebookApp] KernelRestarter: restarting kernel (2/5)
'2018.05.08T17:25:47.761 cores
[I 17:25:50.758 NotebookApp] KernelRestarter: restarting kernel (3/5)
'2018.05.08T17:25:50.773 cores
[W 17:25:51.772 NotebookApp] Timeout waiting for kernel_info reply from 5c3c1f5f-2124-4507-b1bb-044e9e0af35f

Workaround

no workaround
If you know workaround, please provide it here.

symbol error while running "pyq -m notebook "

I am installed pyq 4.1.2 succesfully on redhat 2.7. When I run pyq from the shell, everything works fine.

(env_py36) [a393831@lrtp9101 ~]$ pyq
Python 3.6.3 |Anaconda custom (64-bit)| (default, Oct 13 2017, 12:05:00) 
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyq
>>> print(pyq.__version__)
4.1.2
>>> 

But when I launched the notebook with "pyq -m notebook ", and tried to the above code from notebook. I received the following error (also see the attached screen shot)

---

ImportError                               Traceback (most recent call last)
/apps/anaconda3/envs/env_py36/lib/python3.6/site-packages/pyq/__init__.py in <module>()
     21 try:
---> 22     from ._k import K as _K, error as kerr, Q_VERSION, Q_DATE, Q_OS
     23 except ImportError:

ImportError: /apps/anaconda3/envs/env_py36/lib/python3.6/site-packages/pyq/_k.cpython-36m-x86_64-linux-gnu.so: undefined symbol: sd1

During handling of the above exception, another exception occurred:

ImportError                               Traceback (most recent call last)
<ipython-input-1-7abfe09f4cdb> in <module>()
----> 1 import pyq
      2 print(tf.__version__)
      3 print(pyq.__version__)

/apps/anaconda3/envs/env_py36/lib/python3.6/site-packages/pyq/__init__.py in <module>()
     29         else:
     30             message += "Use pyq executable."
---> 31         raise ImportError(message)
     32     raise
     33 

ImportError: Importing pyq from stock python is not supported. Use pyq executable.

Socket stale when opening port in PyQ

KDB+/Q supports socket connection natively. But when activating the socket listener in Q, it will not response when inside PyQ.

Server Side:

>>> pyq.q("\p 12345")
k('::')

Client Side:

>>> import pyq
>>> pyq.q("hopen `::12345")

The hopen command will wait for a long time, it seems the server side isn't responding.

pyq doesn't pick up 64bit q on OSX

Perhaps I'm doing something wrong, but no matter what I try I can't get pyq to recognize that it should be looking for q in the m64 directory. It seems like setup.py is reading QHOME and doing the right thing, so I'm not quite sure what the problem is:

>> pip install --no-cache-dir pyq
Collecting pyq
  Downloading pyq-4.1.2-cp36-cp36m-macosx_10_12_intel.whl (123kB)
    100% |████████████████████████████████| 133kB 1.7MB/s
Installing collected packages: pyq
Successfully installed pyq-4.1.2
>> pyq
qbinpath = /Users/g/dev/q/m32/q
qbinpath = /Users/g/anaconda/q/m32/q
qbinpath = /Users/g/q/m32/q
No such file or directory
>> ls $QHOME/m64
ffi.so      ffi.so.dSYM q
>> ls $QHOME/m32
ffi.so      ffi.so.dSYM

Installation with Python 3.5 - NameError: name 'pypath' is not defined

Hi,

NameError: name 'pypath' is not defined is thrown when installing pyq with Python 3.5.
On the same machine, it installs fine with Python 2.7.
kdb+ version is 3.4, 64-bit licensed.

Thanks

(venv) adil@adil-OptiPlex-3020:~/workspace/technical-alerts$ pip install -i https://pyq.enlnt.com --no-binary pyq pyq
Collecting pyq
  Downloading https://github.com/enlnt/pyq/archive/pyq-4.0.1.tar.gz (585kB)
    100% |████████████████████████████████| 593kB 1.4MB/s 
Skipping bdist_wheel for pyq, due to binaries being disabled for it.
Installing collected packages: pyq
  Running setup.py install for pyq ... error
    Complete output from command /home/adil/workspace/technical-alerts/venv/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-chld175c/pyq/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-wj5cg4zw-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/adil/workspace/technical-alerts/venv/include/site/python3.5/pyq:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.5
    creating build/lib.linux-x86_64-3.5/pyq
    copying src/pyq/__init__.py -> build/lib.linux-x86_64-3.5/pyq
    copying src/pyq/cmd.py -> build/lib.linux-x86_64-3.5/pyq
    copying src/pyq/magic.py -> build/lib.linux-x86_64-3.5/pyq
    copying src/pyq/version.py -> build/lib.linux-x86_64-3.5/pyq
    copying src/pyq/ptk.py -> build/lib.linux-x86_64-3.5/pyq
    copying src/pyq/conftest.py -> build/lib.linux-x86_64-3.5/pyq
    copying src/pyq/_pt_run.py -> build/lib.linux-x86_64-3.5/pyq
    copying src/pyq/_n.py -> build/lib.linux-x86_64-3.5/pyq
    creating build/lib.linux-x86_64-3.5/pyq/tests
    copying src/pyq/tests/__init__.py -> build/lib.linux-x86_64-3.5/pyq/tests
    copying src/pyq/tests/test_com.py -> build/lib.linux-x86_64-3.5/pyq/tests
    copying src/pyq/tests/test_no_numpy.py -> build/lib.linux-x86_64-3.5/pyq/tests
    copying src/pyq/tests/test_sd.py -> build/lib.linux-x86_64-3.5/pyq/tests
    copying src/pyq/tests/test_q.py -> build/lib.linux-x86_64-3.5/pyq/tests
    copying src/pyq/tests/test_magic.py -> build/lib.linux-x86_64-3.5/pyq/tests
    copying src/pyq/tests/test_numpy.py -> build/lib.linux-x86_64-3.5/pyq/tests
    copying src/pyq/tests/test_version.py -> build/lib.linux-x86_64-3.5/pyq/tests
    copying src/pyq/tests/test_pyq_executable.py -> build/lib.linux-x86_64-3.5/pyq/tests
    copying src/pyq/tests/test_threads.py -> build/lib.linux-x86_64-3.5/pyq/tests
    copying src/pyq/tests/test_pypath.py -> build/lib.linux-x86_64-3.5/pyq/tests
    copying src/pyq/tests/test_ci.py -> build/lib.linux-x86_64-3.5/pyq/tests
    copying src/pyq/tests/test_python_q.py -> build/lib.linux-x86_64-3.5/pyq/tests
    copying src/pyq/tests/test_p.py -> build/lib.linux-x86_64-3.5/pyq/tests
    copying src/pyq/tests/test_k.py -> build/lib.linux-x86_64-3.5/pyq/tests
    copying src/pyq/tests/test_memoryview.py -> build/lib.linux-x86_64-3.5/pyq/tests
    copying src/pyq/tests/test_pathlib.py -> build/lib.linux-x86_64-3.5/pyq/tests
    copying src/pyq/tests/test_n.py -> build/lib.linux-x86_64-3.5/pyq/tests
    running build_ext
    building 'pyq._k' extension
    creating build/temp.linux-x86_64-3.5-kx-3
    creating build/temp.linux-x86_64-3.5-kx-3/src
    creating build/temp.linux-x86_64-3.5-kx-3/src/pyq
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DKXVER=3 -DQVER=3 -DPY3K=35 -I/usr/include/python3.5m -I/home/adil/workspace/technical-alerts/venv/include/python3.5m -c src/pyq/_k.c -o build/temp.linux-x86_64-3.5-kx-3/src/pyq/_k.o -Wpointer-arith -Werror -fno-strict-aliasing
    x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.5-kx-3/src/pyq/_k.o -o build/lib.linux-x86_64-3.5/pyq/_k.cpython-35m-x86_64-linux-gnu.so
    running build_scripts
    creating build/scripts-3.5
    copying src/scripts/pyq-runtests -> build/scripts-3.5
    copying and adjusting src/scripts/pyq-coverage -> build/scripts-3.5
    copying src/scripts/ipyq -> build/scripts-3.5
    copying src/scripts/pq -> build/scripts-3.5
    copying src/scripts/qp -> build/scripts-3.5
    changing mode of build/scripts-3.5/pyq-coverage from 664 to 775
    changing mode of build/scripts-3.5/ipyq from 664 to 775
    changing mode of build/scripts-3.5/pq from 664 to 775
    changing mode of build/scripts-3.5/qp from 664 to 775
    running build_qk
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-chld175c/pyq/setup.py", line 796, in <module>
        **metadata)
      File "/usr/lib/python3.5/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/lib/python3.5/distutils/dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/home/adil/workspace/technical-alerts/venv/lib/python3.5/site-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/usr/lib/python3.5/distutils/command/install.py", line 583, in run
        self.run_command('build')
      File "/usr/lib/python3.5/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/usr/lib/python3.5/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/usr/lib/python3.5/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib/python3.5/distutils/dist.py", line 973, in run_command
        cmd_obj.ensure_finalized()
      File "/usr/lib/python3.5/distutils/cmd.py", line 107, in ensure_finalized
        self.finalize_options()
      File "/tmp/pip-build-chld175c/pyq/setup.py", line 358, in finalize_options
        lib_string = 'lib:"%s\\000"\n' % pypath
    NameError: name 'pypath' is not defined
    ----------------------------------------
Command "/home/adil/workspace/technical-alerts/venv/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-chld175c/pyq/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-wj5cg4zw-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/adil/workspace/technical-alerts/venv/include/site/python3.5/pyq" failed with error code 1 in /tmp/pip-build-chld175c/pyq/

Null value interpreted differently when imported to DataFrame from dictionary

I found that we get different result when constructing a panda Dataframe from dictionary or array:

>>> pandas.DataFrame(dict(pyq.q("flip ([]empty:3#0Ni)")))
        empty
0 -2147483648
1 -2147483648
2 -2147483648

Null integer should be mapped to the None type. This is fine when we import to DataFrame as an array:

>>> pandas.DataFrame(list(dict(pyq.q("flip ([]empty:3#0Ni)"))['empty']))
      0
0  None
1  None
2  None
>>>

32-Bit Installation

Python version 2.7.13 on 64 bit Centos 6.

CFLAGS=-m32 pip install pyq
keeps yielding “src/pyq.o: could not read symbols: File in wrong format” error

null values in enumerated columns causes crash

Below is my test case. Please advise.

Thanks!
Ray

from pyq import q
import numpy as np
import pandas as pd

q('suits: `hearts`clubs`spades`diamonds')
q.good = q('([] c1: 1 2 3 4; c2:`suits$(`hearts`clubs`clubs`diamonds))')
na = np.array(q.good)
# column c2 now has a null value
q.bad = q('([] c1: 1 2 3 4; c2:`suits$(`hearts`clubs``diamonds))')
q.bad_column = q('bad[`c2]')
# q.bad and q.bad_column both  type <class 'pyq.K'>
# each of the following 3 lines will crash due to null values in enumerated column
t = list(q.bad_column)
na = np.array(q.bad)
df = pd.DataFrame(na)

PyQ on Mac OS X error

Using python 3.6. I have 2 versions of kdb installed on my system; the free 32-bit version and the 64-bit personal edition. I've successfully installed PyQ using pip3...

~ master*
❯ pip3 show pyq
Name: pyq
Version: 4.1.2
Summary: PyQ - Python for kdb+
Home-page: http://pyq.enlnt.com
Author: Enlightenment Research, LLC
Author-email: [email protected]
License: PyQ General License
Location: /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages
Requires:

When I enter the pyq command it throws the following error...

~ master*
❯ pyq
Welcome to kdb+ 32bit edition
For support please see http://groups.google.com/d/forum/personal-kdbplus
Tutorials can be found at http://code.kx.com/wiki/Tutorials
To exit, type \\
To remove this startup msg, edit q.q
'2017.11.25T15:31:18.701 dlopen(/Library/Frameworks/Python.framework/Versions/3.6/bin/python3:, 10): image not found
  [4]  /Users/marrowgari/q/p.k:3: (`p 2:(`p_init;1))python_dll

I've also tried using Virtualenv. Following the steps here...

~ master*
❯ pip3 show virtualenv
Name: virtualenv
Version: 15.1.0
Summary: Virtual Python Environment builder
Home-page: https://virtualenv.pypa.io/
Author: Jannis Leidel, Carl Meyer and Brian Rosner
Author-email: [email protected]
License: MIT
Location: /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages
Requires:

I've created the virtual environment and activated it...

~ master*
❯ virtualenv /Users/marrowgari/pykdb
Using base prefix '/Library/Frameworks/Python.framework/Versions/3.6'
New python executable in /Users/marrowgari/pykdb/bin/python3.6
Also creating executable in /Users/marrowgari/pykdb/bin/python
Installing setuptools, pip, wheel...done.

~ master*
❯ source /Users/marrowgari/pykdb/bin/activate

~ master*
(pykdb) ❯

Since I have a license for my 64-bit personal edition I create a directory for it first...

~ master*
(pykdb) ❯ mkdir -p /Users/marrowgari/pykdb/q && unzip /Users/marrowgari/downloads/m64.zip -d /Users/marrowgari/pykdb/q
Archive:  /Users/marrowgari/downloads/m64.zip
  inflating: /Users/marrowgari/pykdb/q/m64/q
  inflating: /Users/marrowgari/pykdb/q/q.k

and check to make sure that it's been unzipped and installed properly...

~ master*
(pykdb) ❯ cd pykdb

~/pykdb master*
(pykdb) ❯ ls
bin                lib                q
include            pip-selfcheck.json

~/pykdb master*
(pykdb) ❯ cd q

~/pykdb/q master*
(pykdb) ❯ ls
m64 q.k

~/pykdb/q master*
(pykdb) ❯ cd m64

~/pykdb/q/m64 master*
(pykdb) ❯ ls
q

I copy my kdb license to the q folder previously created in the virtualenv...
❯ cp /Users/marrowgari/downloads/m64/kc.lic /Users/marrowgari/pykdb/q

and check to make sure it's been added...

~/pykdb/q master*
(pykdb) ❯ ls
kc.lic m64    q.k

I install PyQ in the virtualenv...

~/pykdb/q master*
(pykdb) ❯ pip3 install pyq
Collecting pyq
Installing collected packages: pyq
Successfully installed pyq-4.1.2

However, when I try to start an interactive session it throws the following error...

~/pykdb/q master*
(pykdb) ❯ pyq
Welcome to kdb+ 32bit edition
For support please see http://groups.google.com/d/forum/personal-kdbplus
Tutorials can be found at http://code.kx.com/wiki/Tutorials
To exit, type \\
To remove this startup msg, edit q.q
'2017.11.25T15:56:52.494 dlopen(/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6:, 10): image not found
  [4]  /Users/marrowgari/pykdb/q/p.k:3: (`p 2:(`p_init;1))python_dll

Any help on resolving this would be greatly appreciated. Thank you.