devanshkv / fetch

A set of deep learning models for FRB/RFI binary classification.

FETCH

fetch is Fast Extragalactic Transient Candidate Hunter. It has been detailed in the paper Towards deeper neural networks for Fast Radio Burst detection.

This is the tensorflow>=2 version of the fetch, if you are looking for the older tensorflow version click here.

Install

git clone https://github.com/devanshkv/fetch.git
cd fetch
pip install -r requirements.txt
python setup.py install

The installation will put predict.py and train.py in your PYTHONPATH.

Usage

To use fetch, you would first have to create candidates. Use your for this purpose, this notebook explains the whole process. Your also comes with a command line script your_candmaker.py which allows you to use CPU or single/multiple GPUs.

To predict a candidate h5 files living in the directory /data/candidates/ use predict.py for model a as follows:

predict.py --data_dir /data/candidates/ --model a

To fine-tune the model a, with a bunch of candidates, put them in a pandas readable csv, candidate.csv with headers 'h5' and 'label'. Use

train.py --data_csv candidates.csv --model a --output_path ./

This would train the model a and save the training log, and model weights in the output path.

Example

Test filterbank data can be downloaded from here. The folder contains three filterbanks: 28.fil 29.fil 34.fil. Heimdall results for each of the files are as follows:

for 28.fil

16.8128	1602	2.02888	1	127	475.284	22	1601	1604

for 29.fil

18.6647	1602	2.02888	1	127	475.284	16	1601	1604

for 34.fil

13.9271	1602	2.02888	1	127	475.284	12	1602	1604 

The cand.csv would look like the following:

file,snr,stime,width,dm,label,chan_mask_path,num_files
28.fil,16.8128,2.02888,1,475.284,1,,1
29.fil,18.6647,2.02888,1,475.284,1,,1
34.fil,13.9271,2.02888,1,475.284,1,,1

Running your_candmaker.py will create three files:

cand_tstart_58682.620316710374_tcand_2.0288800_dm_475.28400_snr_13.92710.h5
cand_tstart_58682.620316710374_tcand_2.0288800_dm_475.28400_snr_16.81280.h5
cand_tstart_58682.620316710374_tcand_2.0288800_dm_475.28400_snr_18.66470.h5

Running predict.py with model a will give results_a.csv:

,candidate,probability,label
0,cand_tstart_58682.620316710374_tcand_2.0288800_dm_475.28400_snr_18.66470.h5,1.0,1.0
1,cand_tstart_58682.620316710374_tcand_2.0288800_dm_475.28400_snr_16.81280.h5,1.0,1.0
2,cand_tstart_58682.620316710374_tcand_2.0288800_dm_475.28400_snr_13.92710.h5,1.0,1.0

Training Data

The training data is available at astro.phys.wvu.edu/fetch.

Citating this work

---

If you use this work please cite:

@article{Agarwal2020,
  doi = {10.1093/mnras/staa1856},
  url = {https://doi.org/10.1093/mnras/staa1856},
  year = {2020},
  month = jun,
  publisher = {Oxford University Press ({OUP})},
  author = {Devansh Agarwal and Kshitij Aggarwal and Sarah Burke-Spolaor and Duncan R Lorimer and Nathaniel Garver-Daniels},
  title = {{FETCH}: A deep-learning based classifier for fast transient classification},
  journal = {Monthly Notices of the Royal Astronomical Society}
}
@software{agarwal_aggarwal_2020,
  author       = {Devansh Agarwal and
                  Kshitij Aggarwal},
  title        = {{devanshkv/fetch: Software release with the 
                   manuscript}},
  month        = jun,
  year         = 2020,
  publisher    = {Zenodo},
  version      = {0.1.8},
  doi          = {10.5281/zenodo.3905437},
  url          = {https://doi.org/10.5281/zenodo.3905437}
}

fetch's Issues

predict.py model download links appear to be offline

Hi @devanshkv, @KshitijAggarwal and @nategarver-daniels,

I'm unable to use the models in predict.py. The link to download the models seems to be offline again. For models 'a' to 'h', for example, I get the HTTPS Error 502: Bad Gateway.

Could you make the download links active again?

Many thanks in advance,
Best,
ruiten22

Traning Data Link Broken

The README hyperlink, astro.phys.wvu.edu/fetch, links to https://github.com/devanshkv/fetch/blob/master/astro.phys.wvu.edu/fetch, which does not exist.

astro.phys.wvu.edu/fetch itself responds with 403.

Incompatibility with Tensorflow >=2.5.3

Issue
Since Tensorflow 2.5.3 and above, model_from_yaml has been removed. Would it be possible to update things to make use of model_from_json instead?

Screenshots
A screenshot of the related error:

TensorFlow 2.7 drops YAML support

In the 2.7 version of TensorFlow (see https://github.com/tensorflow/tensorflow/releases) they have dropped the support of yaml:
"The methods Model.to_yaml() and keras.models.model_from_yaml have been replaced to raise a RuntimeError as they can be abused to cause arbitrary code execution. It is recommended to use JSON serialization instead of YAML, or, a better alternative, serialize to H5."

predict.py therefore doesn't work with TensorFlow >=2.7.

Suggested fixes:

Short fix:
In requirements.txt update the tensorflow-gpu>=2.0 line such that it is bound between 2.0 and 2.6.

Long fix:
Update predict.py such that it makes use of JSON or H5 files.

Cann't find json

Describe the bug
The model have JSON when use tf>2.6, but install doesn't copy the JSON file to models, when use it, it can't find any file

To Reproduce
Steps to reproduce the behavior:
just run it using batch tf2

We need to fix the setup.py

package_data={"fetch": ["models/model_list.csv", "models/*/*yaml"]},

to

package_data={"fetch": ["models/model_list.csv", "models/*/*"]},

Issue with tensorflow-gpu requirement

tensorflow2 has done away with the tensorflow-gpu package. It seems like the replacement is tensorflow[and-cuda]

cuda 9.2 dependency problems

I am having trouble getting the dependencies installed at the moment. I am using cuda 9.2, so I tried:
conda install -c anaconda cudatoolkit==9.2 tensorflow-gpu==1.12.0.

However, conda says there are many conflicts in other packages in my environment. This is in a relatively new environment with relatively common scientific python packages.
Have you tried a fresh installation recently for cuda 9.2?

Cannot load model

Describe the bug
Reading a model (using keras) produces an error. It looks like some attribute is a string, but a byte was expected. Some of the dependencies did get upgraded, so there may be some backwards-incompatible change introduced.

> fetch.utils.get_model('a')
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-32e1218fd4f0> in <module>
----> 1 fetch.utils.get_model('a')

~/anaconda/envs/deployment/lib/python3.6/site-packages/fetch-0.1.8-py3.6.egg/fetch/utils.py in get_model(model_idx)
     96 
     97     # dump weights
---> 98     model.load_weights(weights)
     99 
    100     return model

~/anaconda/envs/deployment/lib/python3.6/site-packages/keras/engine/saving.py in load_wrapper(*args, **kwargs)
    490                 os.remove(tmp_filepath)
    491             return res
--> 492         return load_function(*args, **kwargs)
    493 
    494     return load_wrapper

~/anaconda/envs/deployment/lib/python3.6/site-packages/keras/engine/network.py in load_weights(self, filepath, by_name, skip_mismatch, reshape)
   1228             else:
   1229                 saving.load_weights_from_hdf5_group(
-> 1230                     f, self.layers, reshape=reshape)
   1231             if hasattr(f, 'close'):
   1232                 f.close()

~/anaconda/envs/deployment/lib/python3.6/site-packages/keras/engine/saving.py in load_weights_from_hdf5_group(f, layers, reshape)
   1181     """
   1182     if 'keras_version' in f.attrs:
-> 1183         original_keras_version = f.attrs['keras_version'].decode('utf8')
   1184     else:
   1185         original_keras_version = '1'

AttributeError: 'str' object has no attribute 'decode'

To Reproduce
Running fetch 0.1.8 with CUDA 9.2 and the following:

> conda list | egrep 'tensor|keras'
keras                     2.3.1                    py36_0    conda-forge
keras-applications        1.0.8                      py_1    conda-forge
keras-preprocessing       1.1.2              pyhd8ed1ab_0    conda-forge
tensorboard               1.14.0                   py36_0    conda-forge
tensorflow                1.14.0               hc3e5e64_0    conda-forge
tensorflow-base           1.14.0           py36hc3e5e64_0    conda-forge
tensorflow-estimator      1.14.0           py36h5ca1d4c_0    conda-forge

Then:

> import fetch.utils
> fetch.utils.get_model('a')

Problem with example fil files

The three example fil files throw an error when used with heimdall-astro (I tried using heimdall for verification)

Typo in predict.py

Hello!

This is not a bug report, but I didn't know where else to put it. In predict.py, line 36 it says:
"raise ValueError(f'Model only range from a -- j.')", but the models go up to 'k'. So this is a typo that can be changed.

Furthermore lines 49 & 50 are the same as 35 & 36, not sure what the use of that is.

Thank you for making FETCH!

candmaker.py not working

Hi, can you please advise me on the following issue.

$ candmaker.py --frequency_size 256 --time_size 256 --cand_param_file cands.csv --plot --fout /data/candidates/
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/parul/anaconda3/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "/home/parul/anaconda3/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/home/parul/anaconda3/lib/python3.7/site-packages/fetch-0.1.8-py3.7.egg/EGG-INFO/scripts/candmaker.py", line 52, in cand2h5
cand.get_chunk()
File "/home/parul/anaconda3/lib/python3.7/site-packages/pysigproc-0.40-py3.7.egg/candidate.py", line 233, in get_chunk
self.data = self.get_data(nstart=nstart, nsamp=nsamp)[:, 0, :]
File "/home/parul/anaconda3/lib/python3.7/site-packages/pysigproc-0.40-py3.7.egg/pysigproc.py", line 151, in get_data
dtype=self.dtype).reshape((-1,self.nifs,self.nchans))
File "/home/parul/anaconda3/lib/python3.7/site-packages/pysigproc-0.40-py3.7.egg/pysigproc.py", line 130, in dtype
raise RuntimeError('nbits=%d not supported' % self.nbits)
RuntimeError: nbits=1 not supported
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/parul/anaconda3/bin/candmaker.py", line 4, in
import('pkg_resources').run_script('fetch==0.1.8', 'candmaker.py')
File "/home/parul/anaconda3/lib/python3.7/site-packages/pkg_resources/init.py", line 666, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/home/parul/anaconda3/lib/python3.7/site-packages/pkg_resources/init.py", line 1462, in run_script
exec(code, namespace, namespace)
File "/home/parul/anaconda3/lib/python3.7/site-packages/fetch-0.1.8-py3.7.egg/EGG-INFO/scripts/candmaker.py", line 133, in
pool.map(cand2h5, process_list, chunksize=1)
File "/home/parul/anaconda3/lib/python3.7/multiprocessing/pool.py", line 268, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/home/parul/anaconda3/lib/python3.7/multiprocessing/pool.py", line 657, in get
raise self._value
RuntimeError: nbits=1 not supported

installation instructions

I'm having a bit of trouble getting tensorflow installed correctly. One thing that would help is to be explicit that the conda-forge channel does not work. That is my default.
Also, my understanding is that the cuda compiler version (nvcc --version) must be aligned with cudatool version.

why anaconda channel?

The installation instructions suggest conda install -c anaconda .... However, that channel is a mirror of the default channel (see https://docs.anaconda.com/anaconda/user-guide/tasks/using-repositories/).
Have you found that forcing this channel is required? If so, why?
I'm partly asking because I have had problems getting conda to make self-consistent builds after upgrading in the way suggested here.

bad marshal data using models H and J

I've installed FETCH (and your) on a fairly old machine (with no GPU) in a new conda environment. All seems to go well except when I use models H and J.

To Reproduce
Steps to reproduce the behavior:
In the directory where the h5 files are:

(fetchenv2) snelders@dop263:~/testfetch/cand$ ls *h5 -l
-rw-r--r-- 1 snelders 9000 359255 nov 15 17:00 cand_tstart_58682.620316710374_tcand_2.0288800_dm_475.28400_snr_13.92710.h5
-rw-r--r-- 1 snelders 9000 359751 nov 15 17:00 cand_tstart_58682.620316710374_tcand_2.0288800_dm_475.28400_snr_16.81280.h5
-rw-r--r-- 1 snelders 9000 357159 nov 15 17:00 cand_tstart_58682.620316710374_tcand_2.0288800_dm_475.28400_snr_18.66470.h5

predict.py -m h -g -1 -c .

2021-11-16 08:55:37.059175: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
2021-11-16 08:55:40,352 - get_model -fetch.utils - INFO - Getting model h
2021-11-16 08:55:47.978135: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-11-16 08:55:47.978571: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2021-11-16 08:55:47.978598: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)
2021-11-16 08:55:47.978633: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (dop263): /proc/driver/nvidia/version does not exist
2021-11-16 08:55:47.978866: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE4.1 SSE4.2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-11-16 08:55:47.983296: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
Traceback (most recent call last):
  File "/export/astron/snelders/anaconda3/envs/fetchenv2/bin/predict.py", line 4, in <module>
    __import__('pkg_resources').run_script('fetch==0.2.0', 'predict.py')
  File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/pkg_resources/__init__.py", line 651, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/pkg_resources/__init__.py", line 1448, in run_script
    exec(code, namespace, namespace)
  File "/misc/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/fetch-0.2.0-py3.9.egg/EGG-INFO/scripts/predict.py", line 79, in <module>
    model = get_model(args.model)
  File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/fetch-0.2.0-py3.9.egg/fetch/utils.py", line 92, in get_model
    model = model_from_yaml(y.read())
  File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/saving/model_config.py", line 105, in model_from_yaml
    return deserialize(config, custom_objects=custom_objects)
  File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/layers/serialization.py", line 173, in deserialize
    return generic_utils.deserialize_keras_object(
  File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/utils/generic_utils.py", line 354, in deserialize_keras_object
    return cls.from_config(
  File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/engine/training.py", line 2261, in from_config
    return functional.Functional.from_config(
  File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/engine/functional.py", line 668, in from_config
    input_tensors, output_tensors, created_layers = reconstruct_from_config(
  File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/engine/functional.py", line 1275, in reconstruct_from_config
    process_layer(layer_data)
  File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/engine/functional.py", line 1257, in process_layer
    layer = deserialize_layer(layer_data, custom_objects=custom_objects)
  File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/layers/serialization.py", line 173, in deserialize
    return generic_utils.deserialize_keras_object(
  File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/utils/generic_utils.py", line 354, in deserialize_keras_object
    return cls.from_config(
  File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/engine/training.py", line 2261, in from_config
    return functional.Functional.from_config(
  File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/engine/functional.py", line 668, in from_config
    input_tensors, output_tensors, created_layers = reconstruct_from_config(
  File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/engine/functional.py", line 1275, in reconstruct_from_config
    process_layer(layer_data)
  File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/engine/functional.py", line 1257, in process_layer
    layer = deserialize_layer(layer_data, custom_objects=custom_objects)
  File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/layers/serialization.py", line 173, in deserialize
    return generic_utils.deserialize_keras_object(
  File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/utils/generic_utils.py", line 354, in deserialize_keras_object
    return cls.from_config(
  File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/layers/core.py", line 1019, in from_config
    function = cls._parse_function_from_config(
  File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/layers/core.py", line 1071, in _parse_function_from_config
    function = generic_utils.func_load(
  File "/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/utils/generic_utils.py", line 457, in func_load
    code = marshal.loads(raw_code)
ValueError: bad marshal data (unknown type code)

Expected behavior

predict.py -m a -g -1 -c .
`2021-11-16 08:53:16.533682: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
2021-11-16 08:53:21,303 - get_model -fetch.utils - INFO - Getting model a
2021-11-16 08:53:24.693684: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-11-16 08:53:24.695914: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2021-11-16 08:53:24.695944: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)
2021-11-16 08:53:24.695976: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (dop263): /proc/driver/nvidia/version does not exist
2021-11-16 08:53:24.696268: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE4.1 SSE4.2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-11-16 08:53:24.700161: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
/export/astron/snelders/anaconda3/envs/fetchenv2/lib/python3.9/site-packages/tensorflow/python/keras/engine/training.py:1905: UserWarning: `Model.predict_generator` is deprecated and will be removed in a future version. Please use `Model.predict`, which supports generators.
  warnings.warn('`Model.predict_generator` is deprecated and '
2021-11-16 08:53:31.605966: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2021-11-16 08:53:31.607097: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2400190000 Hz
1/1 [==============================] - 5s 5s/step

(fetchenv2) snelders@dop263:~/testfetch/cand$ cat results_a.csv 
,candidate,probability,label
0,./cand_tstart_58682.620316710374_tcand_2.0288800_dm_475.28400_snr_13.92710.h5,1.0,1.0
1,./cand_tstart_58682.620316710374_tcand_2.0288800_dm_475.28400_snr_16.81280.h5,1.0,1.0
2,./cand_tstart_58682.620316710374_tcand_2.0288800_dm_475.28400_snr_18.66470.h5,1.0,1.0

I expected that all models would work or that none of the models would work.
Info about environment:
conda list

Click to collapse.

(fetchenv2) snelders@dop263:~/testfetch/cand$ conda list
# packages in environment at /export/astron/snelders/anaconda3/envs/fetchenv2:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             4.5                       1_gnu  
_tflow_select             2.1.0                       gpu  
absl-py                   0.13.0           py39h06a4308_0  
aiohttp                   3.7.4.post0      py39h7f8727e_2  
astor                     0.8.1            py39h06a4308_0  
astropy                   4.3.1            py39h09021b7_0  
astunparse                1.6.3                      py_0  
async-timeout             3.0.1            py39h06a4308_0  
attrs                     21.2.0             pyhd3eb1b0_0  
blas                      1.0                         mkl  
blinker                   1.4              py39h06a4308_0  
blosc                     1.21.0               h8c45485_0  
bottleneck                1.3.2            py39hdd57654_1  
brotli                    1.0.9                he6710b0_2  
brotlipy                  0.7.0           py39h27cfd23_1003  
brunsli                   0.1                  h2531618_0  
bzip2                     1.0.8                h7b6447c_0  
c-ares                    1.17.1               h27cfd23_0  
ca-certificates           2021.10.26           h06a4308_2  
cachetools                4.2.2              pyhd3eb1b0_0  
certifi                   2021.10.8        py39h06a4308_0  
cffi                      1.14.6           py39h400218f_0  
cfitsio                   3.470                hf0d0db6_6  
chardet                   4.0.0           py39h06a4308_1003  
charls                    2.2.0                h2531618_0  
charset-normalizer        2.0.4              pyhd3eb1b0_0  
click                     8.0.3              pyhd3eb1b0_0  
cloudpickle               2.0.0              pyhd3eb1b0_0  
colorama                  0.4.4              pyhd3eb1b0_0  
commonmark                0.9.1              pyhd3eb1b0_0  
coverage                  5.5              py39h27cfd23_2  
cryptography              3.4.8            py39hd23ed53_0  
cudatoolkit               10.1.243             h6bb024c_0  
cudnn                     7.6.5                cuda10.1_0  
cupti                     10.1.168                      0  
cycler                    0.10.0           py39h06a4308_0  
cython                    0.29.24          py39hdbfa776_0  
cytoolz                   0.11.0           py39h27cfd23_0  
dask-core                 2021.10.0          pyhd3eb1b0_0  
dataclasses               0.8                pyh6d0b6a4_7  
dbus                      1.13.18              hb2f20db_0  
expat                     2.4.1                h2531618_2  
fetch                     0.2.0                    pypi_0    pypi
fontconfig                2.13.1               h6c09931_0  
fonttools                 4.25.0             pyhd3eb1b0_0  
freetype                  2.11.0               h70c0345_0  
fsspec                    2021.10.1          pyhd3eb1b0_0  
future                    0.18.2           py39h06a4308_1  
gast                      0.4.0              pyhd3eb1b0_0  
giflib                    5.2.1                h7b6447c_0  
glib                      2.69.1               h5202010_0  
google-auth               1.33.0             pyhd3eb1b0_0  
google-auth-oauthlib      0.4.4              pyhd3eb1b0_0  
google-pasta              0.2.0              pyhd3eb1b0_0  
grpcio                    1.36.1           py39h2157cd5_1  
gst-plugins-base          1.14.0               h8213a91_2  
gstreamer                 1.14.0               h28cd5cc_2  
h5py                      2.10.0           py39hec9cf62_0  
hdf5                      1.10.6               hb1b8bf9_0  
icu                       58.2                 he6710b0_3  
idna                      3.2                pyhd3eb1b0_0  
imagecodecs               2021.8.26        py39h4cda21f_0  
imageio                   2.9.0              pyhd3eb1b0_0  
importlib-metadata        4.8.1            py39h06a4308_0  
intel-openmp              2021.4.0          h06a4308_3561  
jpeg                      9d                   h7f8727e_0  
jxrlib                    1.1                  h7b6447c_2  
keras-preprocessing       1.1.2              pyhd3eb1b0_0  
kiwisolver                1.3.1            py39h2531618_0  
krb5                      1.19.2               hac12032_0  
lcms2                     2.12                 h3be6417_0  
ld_impl_linux-64          2.35.1               h7274673_9  
lerc                      3.0                  h295c915_0  
libaec                    1.0.4                he6710b0_1  
libcurl                   7.78.0               h0b77cf5_0  
libdeflate                1.8                  h7f8727e_5  
libedit                   3.1.20210910         h7f8727e_0  
libev                     4.33                 h7f8727e_1  
libffi                    3.3                  he6710b0_2  
libgcc-ng                 9.3.0               h5101ec6_17  
libgfortran-ng            7.5.0               ha8ba4b0_17  
libgfortran4              7.5.0               ha8ba4b0_17  
libgomp                   9.3.0               h5101ec6_17  
libllvm11                 11.1.0               h3826bc1_0  
libnghttp2                1.41.0               hf8bcb03_2  
libpng                    1.6.37               hbc83047_0  
libprotobuf               3.17.2               h4ff587b_1  
libssh2                   1.9.0                h1ba5d50_1  
libstdcxx-ng              9.3.0               hd4cf53a_17  
libtiff                   4.2.0                h85742a9_0  
libuuid                   1.0.3                h7f8727e_2  
libwebp                   1.2.0                h89dd481_0  
libwebp-base              1.2.0                h27cfd23_0  
libxcb                    1.14                 h7b6447c_0  
libxml2                   2.9.12               h03d6c58_0  
libzopfli                 1.0.3                he6710b0_0  
llvmlite                  0.37.0           py39h295c915_1  
locket                    0.2.1            py39h06a4308_1  
lz4-c                     1.9.3                h295c915_1  
markdown                  3.3.4            py39h06a4308_0  
matplotlib                3.4.3            py39h06a4308_0  
matplotlib-base           3.4.3            py39hbbc1b5f_0  
mkl                       2021.4.0           h06a4308_640  
mkl-service               2.4.0            py39h7f8727e_0  
mkl_fft                   1.3.1            py39hd3c417c_0  
mkl_random                1.2.2            py39h51133e4_0  
multidict                 5.1.0            py39h27cfd23_2  
munkres                   1.1.4                      py_0  
ncurses                   6.3                  h7f8727e_2  
networkx                  2.6.3              pyhd3eb1b0_0  
numba                     0.54.1           py39h51133e4_0  
numexpr                   2.7.3            py39h22e1b3c_1  
numpy                     1.20.3           py39hf144106_0  
numpy-base                1.20.3           py39h74d4b33_0  
oauthlib                  3.1.1              pyhd3eb1b0_0  
olefile                   0.46               pyhd3eb1b0_0  
openjpeg                  2.4.0                h3ad879b_0  
openssl                   1.1.1l               h7f8727e_0  
opt_einsum                3.3.0              pyhd3eb1b0_1  
packaging                 21.0               pyhd3eb1b0_0  
pandas                    1.3.4            py39h8c16a72_0  
partd                     1.2.0              pyhd3eb1b0_0  
pcre                      8.45                 h295c915_0  
pillow                    8.4.0            py39h5aabda8_0  
pip                       21.2.4           py39h06a4308_0  
protobuf                  3.17.2           py39h295c915_0  
pyasn1                    0.4.8              pyhd3eb1b0_0  
pyasn1-modules            0.2.8                      py_0  
pycparser                 2.21               pyhd3eb1b0_0  
pyerfa                    2.0.0            py39h27cfd23_0  
pygments                  2.10.0             pyhd3eb1b0_0  
pyjwt                     2.1.0            py39h06a4308_0  
pyopenssl                 21.0.0             pyhd3eb1b0_1  
pyparsing                 3.0.4              pyhd3eb1b0_0  
pyqt                      5.9.2            py39h2531618_6  
pysocks                   1.7.1            py39h06a4308_0  
python                    3.9.7                h12debd9_1  
python-dateutil           2.8.2              pyhd3eb1b0_0  
python-flatbuffers        2.0                pyhd3eb1b0_0  
python_abi                3.9                      2_cp39    conda-forge
pytz                      2021.3             pyhd3eb1b0_0  
pywavelets                1.1.1            py39h6323ea4_4  
pyyaml                    6.0              py39h7f8727e_1  
qt                        5.9.7                h5867ecd_1  
readline                  8.1                  h27cfd23_0  
requests                  2.26.0             pyhd3eb1b0_0  
requests-oauthlib         1.3.0                      py_0  
rich                      10.13.0          py39hf3d152e_0    conda-forge
rsa                       4.7.2              pyhd3eb1b0_1  
scikit-image              0.18.3           py39h51133e4_0  
scipy                     1.7.1            py39h292c36d_2  
setuptools                58.0.4           py39h06a4308_0  
sip                       4.19.13          py39h2531618_0  
six                       1.16.0             pyhd3eb1b0_0  
snappy                    1.1.8                he6710b0_0  
sqlite                    3.36.0               hc218d9a_0  
tbb                       2021.4.0             hd09550d_0  
tensorboard               2.4.0              pyhc547734_0  
tensorboard-plugin-wit    1.6.0                      py_0  
tensorflow                2.4.1           gpu_py39h8236f22_0  
tensorflow-base           2.4.1           gpu_py39h29c2da4_0  
tensorflow-estimator      2.6.0              pyh7b7c402_0  
tensorflow-gpu            2.4.1                h30adc30_0  
termcolor                 1.1.0            py39h06a4308_1  
tifffile                  2021.7.2           pyhd3eb1b0_2  
tk                        8.6.11               h1ccaba5_0  
toolz                     0.11.2             pyhd3eb1b0_0  
tornado                   6.1              py39h27cfd23_0  
typing-extensions         3.10.0.2             hd3eb1b0_0  
typing_extensions         3.10.0.2           pyh06a4308_0  
tzdata                    2021e                hda174b7_0  
urllib3                   1.26.7             pyhd3eb1b0_0  
werkzeug                  2.0.2              pyhd3eb1b0_0  
wheel                     0.37.0             pyhd3eb1b0_1  
wrapt                     1.12.1           py39he8ac12f_1  
xz                        5.2.5                h7b6447c_0  
yaml                      0.2.5                h7b6447c_0  
yarl                      1.6.3            py39h27cfd23_0  
your                      0.6.6                    pypi_0    pypi
zfp                       0.5.5                h2531618_6  
zipp                      3.6.0              pyhd3eb1b0_0  
zlib                      1.2.11               h7b6447c_3  
zstd                      1.4.9                haebb681_0

When I run all models I get the same message for models A, B, C, D, E, F, G, I, K. With these models the results_?.csv is also created correctly.
Models H and J produce the same error message and no results_?.csv is made.

Any ideas why this happens? For now I am just skipping models H and J, but I am curious why the other models would work and these wouldn't.

Problem with multiprocessing in predict.py

Describe the bug
predict.py has issues when running with -n greater than 1.

To Reproduce
run predict with -n greater than 1

I suspect the problem is with the following code starting on line 70 of predict.py

 if args.nproc == 1:
        use_multiprocessing = True
        logging.info(f"Using multiprocessing with {args.nproc} workers")
    else:
        use_multiprocessing = False

Shouldn't the logic be:
if args.nproc > 1:

How to make the candidates file?

Hi. I am following your paper 'Towards deeper neural networks for Fast Radio Burst detection'. Could you please direct on how to make the candidate file? The information in README.md is not very clear to me.

Website hosting models weight offline

Hi Devansh and Kshitij,

It appears that the website hosting the model weights is offline.

I was trying to use FETCH on a new machine and I got the following when I wanted to use predict.py.

(fetchenv2) m@rk:~/temp$ predict.py -c /home/m/temp/ -m a

Traceback (most recent call last):
  File "/home/m/anaconda3/envs/fetchenv2/bin/predict.py", line 4, in <module>
    __import__('pkg_resources').run_script('fetch==0.2.0', 'predict.py')
  File "/home/m/anaconda3/envs/fetchenv2/lib/python3.7/site-packages/pkg_resources/__init__.py", line 665, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/home/m/anaconda3/envs/fetchenv2/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1463, in run_script
    exec(code, namespace, namespace)
  File "/home/m/anaconda3/envs/fetchenv2/lib/python3.7/site-packages/fetch-0.2.0-py3.7.egg/EGG-INFO/scripts/predict.py", line 79, in <module>
    model = get_model(args.model)
  File "/home/m/anaconda3/envs/fetchenv2/lib/python3.7/site-packages/fetch-0.2.0-py3.7.egg/fetch/utils.py", line 103, in get_model
    hash_algorithm="md5",
  File "/home/m/anaconda3/envs/fetchenv2/lib/python3.7/site-packages/tensorflow/python/keras/utils/data_utils.py", line 250, in get_file
    raise Exception(error_msg.format(origin, e.errno, e.reason))
Exception: URL fetch failure on http://psrpop.phys.wvu.edu/download.php?val=ft_DenseNet121_2_dt_Xception_13_256.h5: None -- Bad Gateway

For me this happens with all the models, not only model a.
Do you have an estimate on when things will be back online? Thank you in advance.

Run over many directories

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

I am currently running the predict program over many directories. however it takes a substantial time for the model to load for each directory that will be processed.

Describe the solution you'd like
A clear and concise description of what you want to happen.

I would like the predict to be able to iterate through directories when i supply it with many directories through the -c (--data_dir) option. Basically i supply predict.py -c dir1 -c dir2 and it runs predict over each directory saving the results in that directory without unloading the model until if finishes all directories.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

The only alternative I have considered to get a feature like this is to move all files into a single directory and keep track of them, have predict run on them, then split the results.csv for each set of files, then move the files and results back. Other than that i have considered reducing the number of independent files in need of processing, however this is not completely possible.

Additional context
Add any other context or screenshots about the feature request here.

Migration from keras 2 to 3

The updated requirements.txt invokes the installation of keras version 3. The version 3 API has changed and requires some code changes, see Migrating Keras 2 code to multi-backend Keras 3. Please migrate to keras 3.

FYI: tensorflow[and-cuda]==2.15.0.post1 still loads keras 2 version

Note: also the keras model construct has changed, which does not support deep nesting. Not sure if this is impacting fetch.

Could you help me check if this is a fast radio burst

The 2017-04-28-10:03:02_01.fil i download from https://data.csiro.au/collection/csiro:34437v3 This file is in cutouts/FRB1704. Then i run with heimdall generate 2017-04-28-10:03:02_01.cand file.
The parameters in the file is below:
14.869 10956 13.8754 10 68 225.311 2817 8046 11714
Then i use the your tutorial https://thepetabyteproject.github.io/your/ipynb/Candidate/ with the parameters below:
cand = Candidate(
fp=fil_file,
dm=225.311,
tcand=13.8754,
width=3,
snr=14.869,
device=0,
)
And generate dispersed frequency-time, DM-Time and dedispersed frequency-time images like this:

Is there something wrong with my operation. I can't find the characteristics of a fast radio burst.

URL failure

Hi,

When I ran predict.py, I got the following error.
This is the same error as the case of #23.

Traceback (most recent call last):
File "/home/shisano/anaconda3/envs/fetch_env/bin/predict.py", line 4, in
import('pkg_resources').run_script('fetch==0.2.0', 'predict.py')
File "/home/shisano/anaconda3/envs/fetch_env/lib/python3.7/site-packages/pkg_resources/init.py", line 651, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/home/shisano/anaconda3/envs/fetch_env/lib/python3.7/site-packages/pkg_resources/init.py", line 1448, in run_script
exec(code, namespace, namespace)
File "/KUMA/kuma1/shisano/anaconda3/envs/fetch_env/lib/python3.7/site-packages/fetch-0.2.0-py3.7.egg/EGG-INFO/scripts/predict.py", line 79, in
model = get_model(args.model)
File "/home/shisano/anaconda3/envs/fetch_env/lib/python3.7/site-packages/fetch-0.2.0-py3.7.egg/fetch/utils.py", line 103, in get_model
hash_algorithm="md5",
File "/home/shisano/anaconda3/envs/fetch_env/lib/python3.7/site-packages/tensorflow/python/keras/utils/data_utils.py", line 260, in get_file
raise Exception(error_msg.format(origin, e.code, e.msg))
Exception: URL fetch failure on http://psrpop.phys.wvu.edu/download.php?val=ft_DenseNet121_2_dt_Xception_13_256.h5: 502 -- Bad Gateway

Is the link offline again now?

unsupported operand type(s) for ** or pow(): 'str' and 'int'

Hi @devanshkv @KshitijAggarwal
Hey I just tried to run the code but there are some problems happening whenever I run:

candmaker.py --frequency_size 256 --time_size 256 --cand_param_file cands.csv --plot --fout /data/candidates/

I am getting:

Traceback (most recent call last):
File "/home/sanat/anaconda3/envs/testenv/bin/candmaker.py", line 4, in
import('pkg_resources').run_script('fetch==0.1.8', 'candmaker.py')
File "/home/sanat/anaconda3/envs/testenv/lib/python3.6/site-packages/pkg_resources/init.py", line 666, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/home/sanat/anaconda3/envs/testenv/lib/python3.6/site-packages/pkg_resources/init.py", line 1462, in run_script
exec(code, namespace, namespace)
File "/home/sanat/anaconda3/envs/testenv/lib/python3.6/site-packages/fetch-0.1.8-py3.6.egg/EGG-INFO/scripts/candmaker.py", line 130, in
[row['fil_file'], row['snr'], row['width'] **2, row['dm'], row['label'], row['stime'],
TypeError: unsupported operand type(s) for ** or pow(): 'str' and 'int'

And can you please provide the cands.csv file so I can test the code? And please can you please help me with running the code?

Thanks, in advance !!!!

hello , how can i generate the train file like train_data.hd5?

Hello,Thank you for your contribution and open source spirit， I'm a beginner in deep learning and astronomy FRB from China three gorges University. I want to run your experiment , how can i generate the train file like train_data.hd5? Do you use fits format data to conversion and generate hd5？

GPU memory not being released

Describe the bug
A clear and concise description of what the bug is.

When predict.py is run, sometimes it will release the gpu memory allocated and sometimes it wont release the memory from the GPU. This causes some problems when trying to run multiple models in serial execution.

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

This error is not easily reproducible, and I also do not know if the files you guys have to work with can reproduce the error.

The error seems to happen the most when trying to run model e with the gpu specified. As in:
predict.py -g 0 -c Temp_Data/Candidates/ -m e

Expected behavior
A clear and concise description of what you expected to happen.

The expected behavior is for the program to run and then to close.

Screenshots
If applicable, add screenshots to help explain your problem.

It gets stuck on the las line of the command line.

Additional context
Add any other context about the problem here.

I always like to find solutions for these problems before posting any bug. My best guess as to whats happening is that the Tensorflow module does not release the memory allocated in the get_model call in line 79.

By searching for solutions online, i found this:
https://stackoverflow.com/questions/39758094/clearing-tensorflow-gpu-memory-after-model-execution

Therefore, by splitting the program into two different processes, then asking them to join, it should release the GPU memory. I have not yet tried to implement this, however, I could possibly do a pull request after trying it and fixing it (if it is the right solution).

getting model without internet connection

The fetch/utils.py defines get_model as a call to download the weights from a server at WVU. It would be nice to be able to set this to a model on disk locally.
I am interested in using fetch on a machine that has no internet connection (but I can put files there manually).

Using predict.py

I have a .fil file, from the Parkes FRB010125 data. The file size is a few MB and contains a single burst. Can I use a model of FETCH to find if the filterbank file contains an FRB or not? If yes, how?

Models h, j fail

Models h,j fail.

When running all the models with

for model in ("a", "b", "c", "d", "e", "f", "g", "h", "i", "j", "k"):
    cmd = f"timeout 125m predict.py -m {model} -g 2 -c ."
     with open(f"{start_dir}/{base}_predict_{model}.out", "a") as f:
            f.write(f"Processing file: {f}")
            call(cmd, shell=True, stdout=f, stderr=f, executable='/bin/bash')

Models h and j fail with the following, the other models run fine.

pciBusID: 0000:18:00.0 name: Quadro P2000 computeCapability: 6.1
coreClock: 1.4805GHz coreCount: 8 deviceMemorySize: 4.93GiB deviceMemoryBandwidth: 130.53GiB/s
2023-09-25 05:12:09.291609: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
2023-09-25 05:12:09.291640: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.10
2023-09-25 05:12:09.291658: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.10
2023-09-25 05:12:09.291676: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2023-09-25 05:12:09.291693: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2023-09-25 05:12:09.291711: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2023-09-25 05:12:09.291728: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.10
2023-09-25 05:12:09.291746: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.7
2023-09-25 05:12:09.292371: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2023-09-25 05:12:09.292433: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.10.1
2023-09-25 05:12:10.011767: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2023-09-25 05:12:10.011830: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0 
2023-09-25 05:12:10.011842: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N 
2023-09-25 05:12:10.013939: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4582 MB memory) -> physical GPU (device: 0, name: Quadro P2000, pci bus id: 0000:18:00.0, compute capability: 6.1)
Traceback (most recent call last):
  File "~/python/miniconda3/envs/kpe/bin/predict.py", line 79, in <module>
    model = get_model(args.model)
  File "~/python/miniconda3/envs/kpe/lib/python3.9/site-packages/fetch/utils.py", line 92, in get_model
    model = model_from_yaml(y.read())
  File "~/python/miniconda3/envs/kpe/lib/python3.9/site-packages/tensorflow/python/keras/saving/model_config.py", line 105, in model_from_yaml
    return deserialize(config, custom_objects=custom_objects)
  File "~/python/miniconda3/envs/kpe/lib/python3.9/site-packages/tensorflow/python/keras/layers/serialization.py", line 173, in deserialize
    return generic_utils.deserialize_keras_object(
  File "~/python/miniconda3/envs/kpe/lib/python3.9/site-packages/tensorflow/python/keras/utils/generic_utils.py", line 354, in deserialize_keras_object
    return cls.from_config(
  File "~/python/miniconda3/envs/kpe/lib/python3.9/site-packages/tensorflow/python/keras/engine/training.py", line 2261, in from_config
    return functional.Functional.from_config(
  File "~/python/miniconda3/envs/kpe/lib/python3.9/site-packages/tensorflow/python/keras/engine/functional.py", line 668, in from_config
    input_tensors, output_tensors, created_layers = reconstruct_from_config(
  File "~/python/miniconda3/envs/kpe/lib/python3.9/site-packages/tensorflow/python/keras/engine/functional.py", line 1275, in reconstruct_from_config
    process_layer(layer_data)
  File "~/python/miniconda3/envs/kpe/lib/python3.9/site-packages/tensorflow/python/keras/engine/functional.py", line 1257, in process_layer
    layer = deserialize_layer(layer_data, custom_objects=custom_objects)
  File "~/python/miniconda3/envs/kpe/lib/python3.9/site-packages/tensorflow/python/keras/layers/serialization.py", line 173, in deserialize
    return generic_utils.deserialize_keras_object(
  File "~/python/miniconda3/envs/kpe/lib/python3.9/site-packages/tensorflow/python/keras/utils/generic_utils.py", line 354, in deserialize_keras_object
    return cls.from_config(
  File "~/python/miniconda3/envs/kpe/lib/python3.9/site-packages/tensorflow/python/keras/engine/training.py", line 2261, in from_config
    return functional.Functional.from_config(
  File "~/python/miniconda3/envs/kpe/lib/python3.9/site-packages/tensorflow/python/keras/engine/functional.py", line 668, in from_config
    input_tensors, output_tensors, created_layers = reconstruct_from_config(
  File "~/python/miniconda3/envs/kpe/lib/python3.9/site-packages/tensorflow/python/keras/engine/functional.py", line 1275, in reconstruct_from_config
    process_layer(layer_data)
  File "~/python/miniconda3/envs/kpe/lib/python3.9/site-packages/tensorflow/python/keras/engine/functional.py", line 1257, in process_layer
    layer = deserialize_layer(layer_data, custom_objects=custom_objects)
  File "~/python/miniconda3/envs/kpe/lib/python3.9/site-packages/tensorflow/python/keras/layers/serialization.py", line 173, in deserialize
    return generic_utils.deserialize_keras_object(
  File "~/python/miniconda3/envs/kpe/lib/python3.9/site-packages/tensorflow/python/keras/utils/generic_utils.py", line 354, in deserialize_keras_object
    return cls.from_config(
  File "~/python/miniconda3/envs/kpe/lib/python3.9/site-packages/tensorflow/python/keras/layers/core.py", line 1019, in from_config
    function = cls._parse_function_from_config(
  File "~/python/miniconda3/envs/kpe/lib/python3.9/site-packages/tensorflow/python/keras/layers/core.py", line 1071, in _parse_function_from_config
    function = generic_utils.func_load(
  File "~/python/miniconda3/envs/kpe/lib/python3.9/site-packages/tensorflow/python/keras/utils/generic_utils.py", line 457, in func_load
    code = marshal.loads(raw_code)
ValueError: bad marshal data (unknown type code)