ronaldokun / rfpye

This repo parses and preprocesses the binary files created by the CFRS RFeye Node Logger Script

RFPYE

Este módulo tem como objetivo o processamento e extração otimizada de dados dos arquivos .bin de monitoramento do espectro provenientes do script Logger executados nas estações de Monitoramento CRFS RFeye Node. Para tal utilizamos as várias funcionalidades da biblioteca fastcore, que expande e otimiza as estruturas de dados da linguagem python.

Instalação

Como parte dessa lib utiliza código c compilado com Cython, é preciso que um compilador C esteja instalado. É recomendado a criação de um ambiente virtual para que a instalação das dependências não interfira com o a instalação base do python. Para tal é recomendamos o uso do conda. A seguir é mostrado instruções para a criação do ambiente virtual, com todas as dependências utilizando o conda.

Instale o miniconda. Com o conda instalado e disponível no seu PATH ou através do Anaconda Prompt, execute os comando:

Linux:

Em Linux normalmente o sistema já possui o compilador gcc instalado.

conda create -n rfpye pip python=3.7 gcc -c intel -c conda-forge -y
conda activate rfpye
python -m pip install rfpye

Windows

É preciso ter o compilador Microsoft Visual C++ 2015-2019 Redistributable x64 Versão 14.x instalado.

conda create -n rfpye pip python=3.7 libpython m2w64-toolchain -c intel -y
conda activate rfpye
python -m pip install rfpye

O comando acima cria um ambiente virtual com o mesmo nome da biblioteca rfpye, instala as dependências básicas necessárias para a compilação, em seguida ativa o ambiente virtual e instala o módulo.

Depois disso basta instalar normalmente a lib: python -m pip install rfpye

Como utilizar

Abaixo mostramos as funcionalidades principais dos módulos, utilizando-os dentro de algum outro script ou REPL

Precisamos necessariamente de um diretório de entrada, contendo um ou mais arquivos .bin

Mude os caminhos abaixo para suas pastas locais

from fastcore.xtras import Path
from rfpye.utils import get_files
from rich import print

A função abaixo baixa alguns arquivos de exemplo:

path = Path(r'binfiles')
if not path.exists() or not len(get_files(path, extensions=['.bin'])):
    path = Path('.')
    !wget --header 'Host: raw.githubusercontent.com' --user-agent 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0' --header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' --header 'Accept-Language: pt-BR,pt;q=0.8,en-US;q=0.5,en;q=0.3' --referer 'https://github.com/EricMagalhaesDelgado/SpecFiles/blob/main/Combo3%20(CRFS%20Bin%20-%20DataTypes%204%2C%207%2C%208%2C%2060-65%20e%2067-69)/rfeye002092_210208_T202310_CRFSBINv.5.bin' --header 'DNT: 1' --header 'Upgrade-Insecure-Requests: 1' 'https://raw.githubusercontent.com/EricMagalhaesDelgado/SpecFiles/main/Combo3%20(CRFS%20Bin%20-%20DataTypes%204%2C%207%2C%208%2C%2060-65%20e%2067-69)/rfeye002092_210208_T202310_CRFSBINv.5.bin' --output-document 'rfeye002092_210208_T202310_CRFSBINv.5.bin'
    !wget --header 'Host: raw.githubusercontent.com' --user-agent 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0' --header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' --header 'Accept-Language: pt-BR,pt;q=0.8,en-US;q=0.5,en;q=0.3' --referer 'https://github.com/EricMagalhaesDelgado/SpecFiles/blob/main/Combo3%20(CRFS%20Bin%20-%20DataTypes%204%2C%207%2C%208%2C%2060-65%20e%2067-69)/rfeye002092_210208_T203131_CRFSBINv.2.bin' --header 'DNT: 1' --header 'Upgrade-Insecure-Requests: 1' 'https://raw.githubusercontent.com/EricMagalhaesDelgado/SpecFiles/main/Combo3%20(CRFS%20Bin%20-%20DataTypes%204%2C%207%2C%208%2C%2060-65%20e%2067-69)/rfeye002092_210208_T203131_CRFSBINv.2.bin' --output-document 'rfeye002092_210208_T203131_CRFSBINv.2.bin'
    !wget --header 'Host: raw.githubusercontent.com' --user-agent 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0' --header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' --header 'Accept-Language: pt-BR,pt;q=0.8,en-US;q=0.5,en;q=0.3' --referer 'https://github.com/EricMagalhaesDelgado/SpecFiles/blob/main/Combo3%20(CRFS%20Bin%20-%20DataTypes%204%2C%207%2C%208%2C%2060-65%20e%2067-69)/rfeye002292_210208_T202215_CRFSBINv.4.bin' --header 'DNT: 1' --header 'Upgrade-Insecure-Requests: 1' 'https://raw.githubusercontent.com/EricMagalhaesDelgado/SpecFiles/main/Combo3%20(CRFS%20Bin%20-%20DataTypes%204%2C%207%2C%208%2C%2060-65%20e%2067-69)/rfeye002292_210208_T202215_CRFSBINv.4.bin' --output-document 'rfeye002292_210208_T202215_CRFSBINv.4.bin'
    !wget --header 'Host: raw.githubusercontent.com' --user-agent 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0' --header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' --header 'Accept-Language: pt-BR,pt;q=0.8,en-US;q=0.5,en;q=0.3' --referer 'https://github.com/EricMagalhaesDelgado/SpecFiles/blob/main/Combo3%20(CRFS%20Bin%20-%20DataTypes%204%2C%207%2C%208%2C%2060-65%20e%2067-69)/rfeye002292_210208_T203238_CRFSBINv.3.bin' --header 'DNT: 1' --header 'Upgrade-Insecure-Requests: 1' 'https://raw.githubusercontent.com/EricMagalhaesDelgado/SpecFiles/main/Combo3%20(CRFS%20Bin%20-%20DataTypes%204%2C%207%2C%208%2C%2060-65%20e%2067-69)/rfeye002292_210208_T203238_CRFSBINv.3.bin' --output-document 'rfeye002292_210208_T203238_CRFSBINv.3.bin'

A função parse_bin é a função principal que encapsula o processamento dos arquivos bin.

parse_bin[source]

parse_bin(bin_file:Union[str, Path], precision=float32)

Receives a CRFS binfile and returns a dictionary with the file metadata, a GPS Class and a list with the different Spectrum Classes A block is a piece of the .bin file with a known start and end and that contains different types of information. It has several fields: file_type, header, data and footer. Each field has lengths and information defined in the documentation. Args: bin_file (Union[str, Path]): path to the bin file

Returns: Dictionary with the file metadata, file_version, string info, gps and spectrum blocks.

Extração de Dados

Vamos listar arquivos da última versão do script Logger, CRFS Bin - Versão 5

files = get_files(r'binfiles/v5', extensions=['.bin'])
file = files.shuffle()[0]

%%time
dados = parse_bin(file)

Wall time: 8.45 s

print(dados)

{
    'filename': 'rfeye002092_210630_T094705.bin',
    'file_version': 23,
    'string': 'CRFS DATA FILE V023',
    'hostname': 'rfeye002092',
    'method': 'ScriptRFeye2021_v2.cfg',
    'unit_info': 'Stationary',
    'file_number': 0,
    'identifier': 'INFO',
    'gps': GPS Data - Median of Coordinates: -12.82684:-38.47805 Altitude: 120.80 
#Satellites: 12.0,
    'spectrum': [Spectrum(type=67, thread_id=300, description='PMEC 2021 (Faixa 1 de 10).', 
start_mega=105, stop_mega=140, dtype='dBm', ndata=3584, bw=18457, processing='peak', 
antuid=0), Spectrum(type=67, thread_id=310, description='PMEC 2021 (Faixa 2 de 10).', 
start_mega=155, stop_mega=165, dtype='dBm', ndata=1024, bw=18457, processing='peak', 
antuid=0), Spectrum(type=67, thread_id=100, description='PRD 2021 (Faixa principal 1 de 4).',
start_mega=50, stop_mega=90, dtype='dBμV/m', ndata=1024, bw=73828, processing='peak', 
antuid=0), Spectrum(type=67, thread_id=110, description='PRD 2021 (Faixa principal 2 de 4).',
start_mega=70, stop_mega=110, dtype='dBμV/m', ndata=1024, bw=73828, processing='peak', 
antuid=0), Spectrum(type=67, thread_id=120, description='PRD 2021 (Faixa principal 3 de 4).',
start_mega=170, stop_mega=220, dtype='dBμV/m', ndata=1280, bw=73828, processing='peak', 
antuid=0), Spectrum(type=67, thread_id=130, description='PRD 2021 (Faixa principal 4 de 4).',
start_mega=470, stop_mega=700, dtype='dBμV/m', ndata=5888, bw=73828, processing='peak', 
antuid=0), Spectrum(type=67, thread_id=200, description='PMEF 2021 (Faixa 1 de 6).', 
start_mega=700, stop_mega=960, dtype='dBm', ndata=6656, bw=73828, processing='peak', 
antuid=0), Spectrum(type=67, thread_id=210, description='PMEF 2021 (Faixa 2 de 6).', 
start_mega=1710, stop_mega=1980, dtype='dBm', ndata=6912, bw=73828, processing='peak', 
antuid=0), Spectrum(type=67, thread_id=220, description='PMEF 2021 (Faixa 3 de 6).', 
start_mega=2100, stop_mega=2169, dtype='dBm', ndata=1792, bw=73828, processing='peak', 
antuid=0), Spectrum(type=67, thread_id=230, description='PMEF 2021 (Faixa 4 de 6).', 
start_mega=2290, stop_mega=2390, dtype='dBm', ndata=2560, bw=73828, processing='peak', 
antuid=0), Spectrum(type=67, thread_id=240, description='PMEF 2021 (Faixa 5 de 6).', 
start_mega=2500, stop_mega=2690, dtype='dBm', ndata=4864, bw=73828, processing='peak', 
antuid=0), Spectrum(type=67, thread_id=250, description='PMEF 2021 (Faixa 6 de 6).', 
start_mega=3290, stop_mega=3700, dtype='dBm', ndata=10496, bw=73828, processing='peak', 
antuid=0), Spectrum(type=67, thread_id=320, description='PMEC 2021 (Faixa 3 de 10).', 
start_mega=320, stop_mega=340, dtype='dBm', ndata=512, bw=73828, processing='peak', 
antuid=0), Spectrum(type=67, thread_id=330, description='PMEC 2021 (Faixa 4 de 10).', 
start_mega=400, stop_mega=410, dtype='dBm', ndata=256, bw=73828, processing='peak', 
antuid=0), Spectrum(type=67, thread_id=340, description='PMEC 2021 (Faixa 5 de 10).', 
start_mega=960, stop_mega=1219, dtype='dBm', ndata=6656, bw=73828, processing='peak', 
antuid=0), Spectrum(type=67, thread_id=350, description='PMEC 2021 (Faixa 6 de 10).', 
start_mega=1389, stop_mega=1429, dtype='dBm', ndata=1280, bw=73828, processing='peak', 
antuid=0), Spectrum(type=67, thread_id=360, description='PMEC 2021 (Faixa 7 de 10).', 
start_mega=1530, stop_mega=1649, dtype='dBm', ndata=3072, bw=73828, processing='peak', 
antuid=0), Spectrum(type=67, thread_id=370, description='PMEC 2021 (Faixa 8 de 10).', 
start_mega=2690, stop_mega=2899, dtype='dBm', ndata=5376, bw=73828, processing='peak', 
antuid=0), Spectrum(type=67, thread_id=380, description='PMEC 2021 (Faixa 9 de 10).', 
start_mega=5000, stop_mega=5160, dtype='dBm', ndata=4096, bw=73828, processing='peak', 
antuid=0), Spectrum(type=67, thread_id=390, description='PMEC 2021 (Faixa 10 de 10).', 
start_mega=5339, stop_mega=5459, dtype='dBm', ndata=3328, bw=73828, processing='peak', 
antuid=0)]
}

A saída da função é um dicionário, com os metadados do arquivo.

GPS

No entanto as duas chaves mais importantes do dicionário retornado são gps e spectrum

Se você imprimir a classe retornada pela chave gps é retornado um resumo dos seus atributos:

print(dados['gps'])

GPS Data - Median of Coordinates: -12.82684:-38.47805 Altitude: 120.80 #Satellites: 12.0

Para extrair os atributos em si de dado objeto e retorná-los todos num dicionário, o módulo utils tem a função auxiliar getattrs

getattrs[source]

getattrs(obj:Any, attrs:Iterable=None, as_tuple=False)

Receives an object and return the atributes listed in attrs, if attrs is None return its public attributes

print(getattrs(dados['gps']))

{'altitude': 120.8, 'latitude': -12.826842, 'longitude': -38.478047, 'num_satellites': 12.0}

Os atributos listados são os valores consolidados por meio da mediana dos diversos blocos de GPS do arquivo.

Dados Brutos de GPS

Caso desejar a lista original de valores, a classe é iterável num loop normalmente e também é possível selecionar os índices individualmente.

dados['gps'][0] , dados['gps'][-1]

((-12.826869, -38.478055, 119.9, 9), (-12.826869, -38.478037, 114.4, 12))

for coords in dados['gps']:
    lat, long, alt, num = coords
    print(f'{lat:.6f} {long:.6f} {alt:.6f} {num}')
    break

-12.826869 -38.478055 119.900000 9

Para saber quantos dados brutos existem, basta utilizar a função len:

len(dados['gps'])

9060

Dados de Nível Espectral

Cada arquivo bin normalmente possui vários fluxos de espectro distintos, cada fluxo espectral é uma classe Python, na chave spectrum é retornado uma lista com todos os fluxos de espectro.

fluxos = dados['spectrum']
print(len(fluxos))

20

Vamos investigar um deles:

fluxo = fluxos[0]

Ao imprimir um fluxo é mostrado informações mínimas sobre o seu conteúdo:

print(fluxo)

Blocks of Type: 67, Thread_id: 300, Start: 105 MHz, Stop: 140 MHz

A função repr retorna uma representação com todos os metadados do fluxo:

print(repr(fluxo))

Spectrum(type=67, thread_id=300, description='PMEC 2021 (Faixa 1 de 10).', start_mega=105, 
stop_mega=140, dtype='dBm', ndata=3584, bw=18457, processing='peak', antuid=0)

Qualquer um dos atributos listados podem ser acessados diretamente:

print(fluxo.description) , print(fluxo.bw)

PMEC 2021 (Faixa 1 de 10).

18457

(None, None)

No entanto o principal atributo de um fluxo de espectro são os valores de nível medidos, os valores medidos são retornados por meio do atributo levels:

print(fluxo.levels)

[[ -88.5  -86.   -84.  ... -100.  -101.  -107.5]
 [ -85.   -84.   -85.  ...  -97.  -101.5 -103. ]
 [ -83.   -83.   -84.5 ... -103.5 -100.   -99. ]
 ...
 [ -89.5  -88.   -87.  ...  -99.5 -102.5 -114. ]
 [ -87.5  -86.5  -87.5 ...  -99.5 -103.5 -105. ]
 [ -92.5  -91.5  -90.  ... -104.5  -99.5  -99. ]]

print(f'Formato da matriz com os níveis: {fluxo.levels.shape}')

Formato da matriz com os níveis: (9060, 3584)

O nº de linhas da matriz nos dá o número de pontos medidos naquele dado fluxo e as colunas o número de traços no qual o Span ( Stop - Start ) foi dividido. O número de traços pode ser retornada também por meio da função len

print(len(fluxo))

9060

A classe Spectrum é iterável, ou seja, pode ser acessada como uma lista, é retornada uma tupla com o timestamp e os pontos daquele traço:

for time, traço in fluxo:
    print(time)
    print(traço)
    break

2021-06-30T09:46:11.447522

[ -88.5  -86.   -84.  ... -100.  -101.  -107.5]

O atributo anterior retorna uma numpy.ndarray, que é um formato eficiente para processamento.

Medidas de nível como pandas dataframe

No entanto temos adicionalmente o método .matrix() que retorna a matriz de dados como um Pandas Dataframe formatada com o tempo da medição de cada traço como índice das linhas e as frequências de cada traço como coluna.

Vamos mostrar as cinco primeiras e cinco últimas linhas e colunas.

fluxo.matrix().iloc[:5, :5]

<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

Frequencies	105.000000	105.009768	105.019537	105.029305	105.039073
Time
2021-06-30 09:46:11.447522	-88.5	-86.0	-84.0	-80.5	-82.5
2021-06-30 09:47:00.736878	-85.0	-84.0	-85.0	-88.0	-90.5
2021-06-30 09:48:00.736849	-83.0	-83.0	-84.5	-92.0	-87.0
2021-06-30 09:49:00.736763	-90.5	-96.5	-90.5	-85.5	-87.5
2021-06-30 09:50:00.736788	-86.5	-86.0	-86.5	-84.5	-85.0

fluxo.matrix().iloc[-5:, -5:]

<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

Frequencies	139.960927	139.970695	139.980463	139.990232	140.000000
Time
2021-07-06 16:41:00.741214	-96.5	-96.5	-97.0	-97.0	-101.0
2021-07-06 16:42:00.781447	-111.0	-103.0	-102.0	-102.0	-105.0
2021-07-06 16:43:00.751170	-95.0	-98.0	-99.5	-102.5	-114.0
2021-07-06 16:44:00.761445	-98.0	-100.0	-99.5	-103.5	-105.0
2021-07-06 16:45:00.862489	-96.5	-101.0	-104.5	-99.5	-99.0

Novamente, caso desejado acessar todos os atributos de um fluxo no formato de dicionário, basta utilizar a função getattrs

print(getattrs(fluxo))

{
    'antuid': 0,
    'bw': 18457,
    'description': 'PMEC 2021 (Faixa 1 de 10).',
    'dtype': 'dBm',
    'ndata': 3584,
    'precision': <class 'numpy.float32'>,
    'processing': 'peak',
    'start_dateidx': datetime.datetime(2021, 6, 30, 9, 46, 11, 447522),
    'start_mega': 105,
    'stop_dateidx': datetime.datetime(2021, 7, 6, 16, 45, 0, 862489),
    'stop_mega': 140,
    'thread_id': 300,
    'type': 67
}

CRFS Bin - Versão 5 - Arquivos Comprimidos

Vamos listar arquivos da última versão do script Logger, Versão 5, arquivos comprimidos onde o piso de ruído é suprimido.

file = r'binfiles\Comprimidos\rfeye002290_210922_T204046_MaskBroken.bin'

%%time
compressed = parse_bin(file)

Wall time: 5.91 s

print(compressed)

{
    'filename': 'rfeye002290_210922_T204046_MaskBroken.bin',
    'file_version': 23,
    'string': 'CRFS DATA FILE V023',
    'hostname': 'rfeye002290',
    'method': 'ScriptRFeye2021_v2.cfg',
    'unit_info': 'Stationary',
    'file_number': 0,
    'identifier': 'INFO',
    'gps': GPS Data - Median of Coordinates: -23.95765:-46.37637 Altitude: 19.70 #Satellites:
11.0,
    'spectrum': [Spectrum(type=68, thread_id=321, thresh=-100, description='PMEC 2021 (Faixa 
3 de 10). @ -80dBm, 100kHz.', start_mega=320, stop_mega=340, minimum=-147.5, dtype='dBm', 
ndata=512, bw=73828, processing='peak', antuid=0), Spectrum(type=68, thread_id=301, 
thresh=-100, description='PMEC 2021 (Faixa 1 de 10). @ -80dBm, 10kHz.', start_mega=108, 
stop_mega=137, minimum=-147.5, dtype='dBm', ndata=14848, bw=3690, processing='peak', 
antuid=0), Spectrum(type=68, thread_id=341, thresh=-100, description='PMEC 2021 (Faixa 5 de 
10). @ -80dBm, 100kHz.', start_mega=960, stop_mega=1219, minimum=-147.5, dtype='dBm', 
ndata=6656, bw=73828, processing='peak', antuid=0), Spectrum(type=68, thread_id=311, 
thresh=-100, description='PMEC 2021 (Faixa 2 de 10). @ -80dBm, 10kHz.', start_mega=156, 
stop_mega=163, minimum=-147.5, dtype='dBm', ndata=3584, bw=3690, processing='peak', 
antuid=0), Spectrum(type=68, thread_id=371, thresh=-100, description='PMEC 2021 (Faixa 8 de 
10). @ -80dBm, 100kHz.', start_mega=2690, stop_mega=2899, minimum=-147.5, dtype='dBm', 
ndata=5376, bw=73828, processing='peak', antuid=0), Spectrum(type=68, thread_id=351, 
thresh=-100, description='PMEC 2021 (Faixa 6 de 10). @ -80dBm, 100kHz.', start_mega=1389, 
stop_mega=1429, minimum=-147.5, dtype='dBm', ndata=1280, bw=73828, processing='peak', 
antuid=0), Spectrum(type=68, thread_id=331, thresh=-100, description='PMEC 2021 (Faixa 4 de 
10). @ -80dBm, 100kHz.', start_mega=400, stop_mega=410, minimum=-147.5, dtype='dBm', 
ndata=256, bw=73828, processing='peak', antuid=0), Spectrum(type=68, thread_id=361, 
thresh=-100, description='PMEC 2021 (Faixa 7 de 10). @ -80dBm, 100kHz.', start_mega=1530, 
stop_mega=1649, minimum=-147.5, dtype='dBm', ndata=3072, bw=73828, processing='peak', 
antuid=0)]
}

fluxo = compressed['spectrum'] ; fluxos

(#20) [Spectrum(type=67, thread_id=300, description='PMEC 2021 (Faixa 1 de 10).', start_mega=105, stop_mega=140, dtype='dBm', ndata=3584, bw=18457, processing='peak', antuid=0),Spectrum(type=67, thread_id=310, description='PMEC 2021 (Faixa 2 de 10).', start_mega=155, stop_mega=165, dtype='dBm', ndata=1024, bw=18457, processing='peak', antuid=0),Spectrum(type=67, thread_id=100, description='PRD 2021 (Faixa principal 1 de 4).', start_mega=50, stop_mega=90, dtype='dBμV/m', ndata=1024, bw=73828, processing='peak', antuid=0),Spectrum(type=67, thread_id=110, description='PRD 2021 (Faixa principal 2 de 4).', start_mega=70, stop_mega=110, dtype='dBμV/m', ndata=1024, bw=73828, processing='peak', antuid=0),Spectrum(type=67, thread_id=120, description='PRD 2021 (Faixa principal 3 de 4).', start_mega=170, stop_mega=220, dtype='dBμV/m', ndata=1280, bw=73828, processing='peak', antuid=0),Spectrum(type=67, thread_id=130, description='PRD 2021 (Faixa principal 4 de 4).', start_mega=470, stop_mega=700, dtype='dBμV/m', ndata=5888, bw=73828, processing='peak', antuid=0),Spectrum(type=67, thread_id=200, description='PMEF 2021 (Faixa 1 de 6).', start_mega=700, stop_mega=960, dtype='dBm', ndata=6656, bw=73828, processing='peak', antuid=0),Spectrum(type=67, thread_id=210, description='PMEF 2021 (Faixa 2 de 6).', start_mega=1710, stop_mega=1980, dtype='dBm', ndata=6912, bw=73828, processing='peak', antuid=0),Spectrum(type=67, thread_id=220, description='PMEF 2021 (Faixa 3 de 6).', start_mega=2100, stop_mega=2169, dtype='dBm', ndata=1792, bw=73828, processing='peak', antuid=0),Spectrum(type=67, thread_id=230, description='PMEF 2021 (Faixa 4 de 6).', start_mega=2290, stop_mega=2390, dtype='dBm', ndata=2560, bw=73828, processing='peak', antuid=0)...]

fluxo = fluxos[0]
fluxo.matrix().iloc[:5, [0, 1, 2, -3, -2, -1]]

<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

Frequencies	105.000000	105.009768	105.019537	139.980463	139.990232	140.000000
Time
2021-06-30 09:46:11.447522	-88.5	-86.0	-84.0	-100.0	-101.0	-107.5
2021-06-30 09:47:00.736878	-85.0	-84.0	-85.0	-97.0	-101.5	-103.0
2021-06-30 09:48:00.736849	-83.0	-83.0	-84.5	-103.5	-100.0	-99.0
2021-06-30 09:49:00.736763	-90.5	-96.5	-90.5	-103.5	-105.0	-101.5
2021-06-30 09:50:00.736788	-86.5	-86.0	-86.5	-104.5	-101.5	-99.5

print(len(fluxo))

9060

CRFS Bin - Versão 4

file = r'binfiles\v4\rfeye002292_210208_T202215_CRFSBINv.4.bin'
blocks = parse_bin(file)
print(blocks)

{
    'filename': 'rfeye002292_210208_T202215_CRFSBINv.4.bin',
    'file_version': 22,
    'string': 'CRFS DATA FILE V022',
    'hostname': 'rfeye002292',
    'method': 'Script_CRFSBINv4',
    'unit_info': 'Stationary',
    'file_number': 0,
    'identifier': 'LOGGER_VERSION',
    'description': 'ClearWrite. Peak.',
    'gps': GPS Data - Median of Coordinates: -12.97163:-38.48149 Altitude: 151.65 
#Satellites: 12.0,
    'spectrum': [Spectrum(type=63, thread_id=20, description='Peak', start_mega=76, 
stop_mega=108, dtype='dBm', ndata=8192, processing='peak', antuid=0), Spectrum(type=63, 
thread_id=30, description='Peak', start_mega=70, stop_mega=110, dtype='dBm', ndata=1024, 
processing='peak', antuid=0), Spectrum(type=63, thread_id=10, description='Peak', 
start_mega=105, stop_mega=140, dtype='dBm', ndata=3584, processing='peak', antuid=0), 
Spectrum(type=63, thread_id=11, description='Peak', start_mega=105, stop_mega=140, 
dtype='dBm', ndata=3584, processing='average', antuid=0), Spectrum(type=63, thread_id=12, 
description='Peak', start_mega=105, stop_mega=140, dtype='dBm', ndata=3584, 
processing='peak', antuid=0), Spectrum(type=63, thread_id=13, description='Peak', 
start_mega=105, stop_mega=140, dtype='dBm', ndata=3584, processing='average', antuid=0), 
Spectrum(type=64, thread_id=14, thresh=-90, minimum=-147.5, description='Peak', 
start_mega=105, stop_mega=140, dtype='dBm', ndata=3584, processing='average', antuid=0), 
Spectrum(type=65, thread_id=15, start_mega=105, stop_mega=140, dtype='dBm', ndata=3584, 
processing='peak', antuid=0)]
}

blocks['spectrum'][0].matrix().iloc[:5, [0, 1, 2, -3, -2, -1]]

<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

Frequencies	76.000000	76.003907	76.007813	107.992187	107.996093	108.000000
Time
2021-02-08 20:22:15.500658	-110.0	-100.5	-99.0	-103.0	-99.0	-96.0
2021-02-08 20:22:16.142770	-105.5	-100.0	-97.5	-94.5	-95.0	-98.0
2021-02-08 20:22:16.500750	-104.0	-102.5	-105.5	-95.5	-98.5	-93.0
2021-02-08 20:22:17.132990	-105.0	-107.0	-103.0	-99.5	-99.5	-102.5
2021-02-08 20:22:17.501352	-97.5	-101.5	-97.0	-104.5	-102.0	-99.5

CRFS Bin - Versão 3

file = r'binfiles\v3\rfeye002292_210208_T203238_CRFSBINv.3.bin'
blocks = parse_bin(file)
print(blocks)

{
    'filename': 'rfeye002292_210208_T203238_CRFSBINv.3.bin',
    'file_version': 21,
    'string': 'CRFS DATA FILE V021',
    'hostname': 'rfeye002292',
    'method': 'Script_CRFSBINv3',
    'unit_info': 'Stationary',
    'file_number': 0,
    'identifier': 'LOGGER_VERSION',
    'description': 'ClearWrite. Peak.',
    'gps': GPS Data - Median of Coordinates: -12.97163:-38.48149 Altitude: 150.60 
#Satellites: 12.0,
    'spectrum': [Spectrum(type=60, thread_id=10, start_mega=105, stop_mega=140, ndata=3584, 
nloops=1, processing='peak', antuid=0), Spectrum(type=60, thread_id=11, start_mega=105, 
stop_mega=140, ndata=3584, nloops=1, processing='average', antuid=0), Spectrum(type=60, 
thread_id=20, start_mega=76, stop_mega=108, ndata=8192, nloops=1, processing='peak', 
antuid=0), Spectrum(type=60, thread_id=30, start_mega=70, stop_mega=110, ndata=1024, 
nloops=4, processing='peak', antuid=0), Spectrum(type=60, thread_id=12, start_mega=105, 
stop_mega=140, ndata=3584, nloops=1, processing='peak', antuid=0), Spectrum(type=60, 
thread_id=13, start_mega=105, stop_mega=140, ndata=3584, nloops=1, processing='average', 
antuid=0), Spectrum(type=61, thread_id=14, thresh=-90, minimum=-147.5, start_mega=105, 
stop_mega=140, ndata=3584, nloops=1, processing='average', antuid=0), Spectrum(type=62, 
thread_id=15, start_mega=105, stop_mega=140, thresh=-90, sampling=9, ndata=3584, antuid=0), 
Spectrum(type=62, thread_id=15, start_mega=105, stop_mega=140, thresh=-90, sampling=10, 
ndata=3584, antuid=0)]
}

blocks['spectrum'][0].matrix().iloc[:5, [0, 1, 2, -3, -2, -1]]

<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

Frequencies	105.000000	105.009768	105.019537	139.980463	139.990232	140.000000
Time
2021-02-08 20:32:39.548000	-76.5	-76.0	-76.5	-94.5	-91.0	-90.0
2021-02-08 20:32:40.133600	-79.5	-80.5	-79.5	-99.0	-94.5	-92.5
2021-02-08 20:32:41.858000	-69.0	-69.0	-69.0	-97.5	-92.5	-90.0
2021-02-08 20:32:42.137500	-70.5	-71.0	-71.5	-97.0	-98.0	-94.5
2021-02-08 20:32:43.716000	-71.0	-69.5	-70.0	-97.0	-89.0	-87.0

CRFS Bin Versão 2

from rfpye.parser import parse_bin
file = r'binfiles\v2\rfeye002092_210208_T203131_CRFSBINv.2.bin'
blocks = parse_bin(file)
print(blocks)

{
    'filename': 'rfeye002092_210208_T203131_CRFSBINv.2.bin',
    'file_version': 21,
    'string': 'CRFS DATA FILE V021',
    'description': 'ClearWrite. Peak.',
    'gps': GPS Data - Median of Coordinates: -1.00000:-1.00000 Altitude: -1.00 #Satellites: 
0.0,
    'spectrum': [Spectrum(type=4, thread_id=20, start_mega=76, stop_mega=108, ndata=8192, 
processing='Peak'), Spectrum(type=4, thread_id=30, start_mega=70, stop_mega=110, ndata=1024, 
processing='Peak'), Spectrum(type=4, thread_id=10, start_mega=105, stop_mega=140, ndata=3584,
processing='Peak'), Spectrum(type=4, thread_id=11, start_mega=105, stop_mega=140, ndata=3584,
processing='Average'), Spectrum(type=4, thread_id=12, start_mega=105, stop_mega=140, 
ndata=3584, processing='Peak'), Spectrum(type=4, thread_id=13, start_mega=105, stop_mega=140,
ndata=3584, processing='Average'), Spectrum(type=7, thread_id=14, thresh=-90, minimum=-147.5,
start_mega=105, stop_mega=140, ndata=3584), Spectrum(type=8, thread_id=15, start_mega=105, 
stop_mega=140, sampling=9, ndata=3584), Spectrum(type=8, thread_id=15, start_mega=105, 
stop_mega=140, sampling=10, ndata=3584)]
}

blocks['spectrum'][0].matrix().iloc[:5, [0, 1, 2, -3, -2, -1]]

<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

Frequencies	76.000000	76.003907	76.007813	107.992187	107.996093	108.000000
Time
NaN	10.5	11.5	13.0	31.0	26.0	22.5
NaN	18.0	19.5	16.5	27.5	29.5	25.5
NaN	11.5	5.0	8.0	30.5	28.5	30.0
NaN	4.5	4.5	13.5	27.5	31.0	31.0
NaN	17.5	15.5	11.0	33.5	29.5	30.0

Fluxo de Ocupação

from rfpye.parser import parse_bin
file = r'binfiles\occ\rfeye002090-VCP_FM_occ15min_191221_085803.bin'
blocks = parse_bin(file)
print(blocks)

{
    'filename': 'rfeye002090-VCP_FM_occ15min_191221_085803.bin',
    'file_version': 22,
    'string': 'CRFS DATA FILE V022',
    'hostname': 'rfeye002090-VCP',
    'method': 'CRFS default method',
    'unit_info': 'RFeye002090',
    'file_number': 0,
    'identifier': 'LOGGER_VERSION',
    'description': 'Ocupacao em 15 minutos na faixa FM',
    'gps': GPS Data - Median of Coordinates: -1.00000:-1.00000 Altitude: -1.00 #Satellites: 
0.0,
    'spectrum': [Spectrum(type=65, thread_id=121, start_mega=80, stop_mega=110, dtype='dBm', 
ndata=1536, processing='peak', antuid=0)]
}

blocks['spectrum'][-1].matrix().iloc[:5, 1003:1010]

<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

Frequencies	99.602606	99.622150	99.641694	99.661238	99.680782	99.700326	99.719870
Time
2019-12-21 09:00:01.367337	17.5	36.0	53.5	62.5	76.5	80.0	72.5
2019-12-21 09:15:01.357259	15.0	29.5	48.0	61.0	76.5	78.5	71.0
2019-12-21 09:30:01.357357	16.0	28.5	46.5	61.0	76.5	77.5	70.0
2019-12-21 09:45:01.357273	15.0	33.0	49.0	66.0	76.0	78.0	70.0
2019-12-21 10:00:01.419225	15.5	35.0	50.0	62.0	74.0	77.0	67.5

rfpye's Issues

Parsing incompatible with version 4

Detected several incompatibilities with CRFS BIN Version 4.

Main issue is the creation of empty dictionary for the spectrum data.

File examples available at: https://anatel365.sharepoint.com/:u:/s/Estruturaodosdadosdemonitorao/ETJu7zJ4WuNBpnJQAM_t0JgBKrD0rKZpJvbIOJ98plzuFA?e=IQ3cfg

https://anatel365.sharepoint.com/:u:/s/Estruturaodosdadosdemonitorao/EVoXNV9tlvZLm9h_0BkVKWIBJfS5BQ6CCj2Q2JLzVEGKqg?e=2l7CXh

parallelize block creation in parse_bin

For a typical ~100MB bin data, the function parse_bin is taking approximately 20s to return the parsed dictionary.

That's more than 5 times the time taken compared with the previous API.

It seems that the main culprit is the block creation, as expected, but also the attributes extraction.

Possible solution:

• Parallelize the block creation passing the list of byte_blocks
• At each parallel block creation also extract its public attributes, since they are cached afterwards

ValueError buffer size must be a multiple of element size

Error processing file: rfeye002263_210618_T210255.bin (veja link)

Code executed
rfpyeDict = extract_bin_data(path = fileFullPath)

Error reported

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File ".\.conda\envs\rfpy\lib\site-packages\rfpye\main.py", line 343, in extract_bin_data
    parsed_bin = parse_bin(file, btypes=BTYPES)
  File ".\.conda\envs\rfpy\lib\site-packages\rfpye\parser.py", line 61, in parse_bin
    "blocks": classify_blocks(body.split(marker), btypes=btypes),
  File ".\.conda\envs\rfpy\lib\site-packages\rfpye\parser.py", line 77, in classify_blocks
    checksum = np.frombuffer(block[-4:], dtype=np.uint32).item()
ValueError: buffer size must be a multiple of element size