Support for Spinsolve Files? about nmrglue

[LCageman]

I wrote something that can use the function of @kaustubhmote (thanks!) and export the processed spectrum with one the 4 possible files given by the Spinsolve software (data.1d, fid.1d, spectrum.1d and spectrum_processed.1d). Note that data.1d and fid.1d are the raw FIDs, so they need a Fourier transform. Also the plotting depends on whether the raw data or processed data is read.

import nmrglue as ng
import matplotlib.pyplot as plt
import numpy as np

dic,data = read(fpath,fname)

#Definition of udic parameters
udic = ng.fileiobase.create_blank_udic(1)
udic[0]['sw'] = float(dic["acqu"]["bandwidth"]) * 1000 # Spectral width in Hz - or width of the whole spectrum
udic[0]['obs'] = float(dic["acqu"]["b1Freq"]) # Magnetic field strenght in MHz (is correctly given for carbon spectra)
udic[0]['size'] = len(data) # Number of points - from acqu (float(dic["acqu"]["nrPnts"]), NB This is different from the data object when zerofilling is applied and will then create a ppm_scale with to little points
udic[0]['car'] = float(dic["acqu"]["lowestFrequency"]) + (udic[0]['sw'] / 2) # Carrier frequency in Hz - or center of spectrum

#For completion, but not important for the ppm scale
udic[0]['label'] = dic["acqu"]["rxChannel"].strip('"')
udic[0]['complex'] = False
udic[0]['time'] = False
udic[0]['freq'] = True

#Create PPM scale object
uc = ng.fileiobase.uc_from_udic(udic)
ppm_scale = uc.ppm_scale()

## For data.1d  and fid.1d processing is needed
if fname == "data.1d" or fname == "fid.1d": 
    data = ng.proc_base.zf_size(data, 2*len(data)) # Zerofill, now 2x of total amount of points
    udic[0]['size'] = len(data)
    uc = ng.fileiobase.uc_from_udic(udic)
    ppm_scale = uc.ppm_scale() #ppm_scale needs to be redefined due to zerofilling
    data = ng.proc_base.fft(data) # Fourier transformation
    # data = ng.proc_base.ps(data, p0=2.0, p1=0.0) # Phasing - can be taken from dic["proc"]["p0Phase"], for "p1" I'm not sure as there are multiple values
    data = ng.proc_base.di(data) # Removal of imaginairy part

#Plot
fig = plt.figure()
ax = fig.add_subplot(111)
if fname == "data.1d" or fname == "fid.1d": 
    ax.plot(ppm_scale, data)
elif fname == "spectrum.1d" or fname == "spectrum_processed.1d" :
    ax.plot(dic["spectrum"]["xaxis"], data.real)
else:
    ax.plot(data)
plt.xlim((10,0))
fig.savefig(os.path.join(fpath, "Spectrum.png"))

Also, there are some differences in the files between the expert software and normal software. Different parameters are stored in "acqu.par" and the expert software has the .pt1 files and a proc.par. All of the acqu parameters used in the script above are present in both versions of "acqu.par".

@kaustubhmote, As proc.par is only present in the expert software, I suggest to make this part optional in the read function:

   with open(os.path.join(fpath, "proc.par"), "r") as f:
        info = f.readlines()

    for line in info:
        line = line.replace("\n", "")
        k, v = line.split("=")
        dic["proc"][k.strip()] = v.strip()

I am new to NMRGlue and Github, but would love to see a spinsolve module. Let me know if I can be of help.

[kaustubhmote]

@mobecks and @NichVC , thanks for sharing the documentation. Based on these, I think we will need functions specific to spinsolve to properly do read in the dictionary and data. As far as I can see, there are three places where the acquisition parameters and data formats for the are stored: (1) The first 32 bytes of the spectrum.1d file (2) acqu.par and (3) proc.par. The simplest case would be to read in these files and manually extract these data out:

def read(fpath, fname):

    with open(os.path.join(fpath, fname), "rb") as f:
        data_raw = f.read()   

    dic = {"spectrum": {}, "acqu": {}, "proc":{}} 

    keys = ["owner", "format", "version", "dataType", "xDim", "yDim", "zDim", "qDim"]

    for i, k in enumerate(keys):
        start = i * 4
        end = start + 4
        value = int.from_bytes( data_raw[start:end], "little")
        dic["spectrum"][k] = value

    data = np.frombuffer(data_raw[end:], "<f")

    split = data.shape[-1] // 3
    xscale = data[0 : split]
    dic["spectrum"]["xaxis"] = xscale

    data = data[split : : 2] + 1j * data[split + 1 : : 2]

    with open(os.path.join(fpath, "acqu.par"), "r") as f:
        info = f.readlines()

    for line in info:
        line = line.replace("\n", "")
        k, v = line.split("=")
        dic["acqu"][k.strip()] = v.strip()
        
        
    with open(os.path.join(fpath, "proc.par"), "r") as f:
        info = f.readlines()

    for line in info:
        line = line.replace("\n", "")
        k, v = line.split("=")
        dic["proc"][k.strip()] = v.strip()

        
    return dic, data

I suggest that this function be used instead of the one described in #117 (since that was just a hack, without knowing the exact file structure). With this, you can read the "Expert" files in the following manner:

dic, data = read(path, "spectrum.1d") # or, read(path, "fid.1d")

fig, ax = plt.subplots()
ax.plot(dic["spectrum"]["xaxis"], data.real)

Similar functions can be made to read in the ".pt" files as well.

Before we put the above function in nmrglue, some additional things need to be done: (1) Most of the values in dic are strings. The function will need to refactored to cast to appropriate values. (2) some refactoring for file paths (3) Multidimensional datasets also need to be separately handled. If anyone would like to do this, please feel free to copy the above function into a "spinsolve.py" file in ng.fileio folder, and we can try and make spinsolve into a separate module just like bruker or pipe.

[jjhelmus]

Are there example Spinsolve files and/or documentation on the format available.

[NichVC]

I've attached some of the documentation provided by the software as well as two data examples, one from Spinsolve and one from Spinsolve Expert (although I believe they are very similar in structure, if not completely the same).
Initially I tried to use the nmrglue.jcampdx function to load the nmr_fid.dx file from the Spinsolve data, but it gave rise to a mirrored spectrum. The developers said that "you will need to do a bit of extra processing here since the Spinsolve complex data is collected differently from the standard. This results in a reflection of the frequencies about the center" - although maybe it'll just be more ideal to work with either the data.1d or spectrum.1d files?

If you want the full documentation or have questions about the data format I suggest writing to Magritek (the company that distributes the software as well as the spectrometers).

Spinsolve Data + Documentation.zip

[LCageman]

I am working this out at the moment as I was stumbling on the same problem for our spinsolve device. I found the answer in this thread. The issue is that reading the FID (dic,data = ng.jcampdx.read...) creates an array containing 2 arrays with either the real or imaginary data instead of a normal 'data' object. Therefore, doing the normal things like fourier transformation or listing all the real points with 'data.real', are not working properly.

See here the solution for Spinsolve data:

import nmrglue as ng
import matplotlib.pyplot as plt
import numpy as np

#Import
dataFolder = "Drive:/folder/"
dic, raw_data = ng.jcampdx.read(dataFolder + "nmr_fid.dx")

#Create proper data object for ng scripts to understand
npoints = int(dic["$TD"][0])
data = np.empty((npoints, ), dtype='complex128')
data.real = raw_data[0][:]
data.imag = raw_data[1][:]

#Processing
data = ng.proc_base.zf_size(data, int(dic["$TD"][0])*2) # Zerofill, now 2x of total amount of points
data = ng.proc_base.fft(data) # Fourier transformation
data = ng.proc_base.ps(data, p0=float(dic["$PHC0"][0]), p1=float(dic["$PHC1"][0])) # Phasing, values taken from dx file
data = ng.proc_base.di(data) # Removal of imaginairy part

# Set correct PPM scaling
udic = ng.jcampdx.guess_udic(dic, data)
udic[0]['car'] = (float(dic["$BF1"][0]) - float(dic["$SF"][0])) * 1000000 # center of spectrum, set manually by using "udic[0]['car'] = float(dic["$SF"][0]) * x", where x is a ppm value
udic[0]['sw'] = float(dic["$SW"][0]) * float(dic["$BF1"][0])
uc = ng.fileiobase.uc_from_udic(udic)
ppm_scale = uc.ppm_scale()

# Plot spectrum
fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(ppm_scale, data)
plt.xlim((8,0)) # plot as we are used to, from positive to negative
fig.savefig(dataFolder + "Spectrum.png")

[LCageman]

This question is actually a duplicate of this closed issue.

[mobecks]

@LCageman's solution works perfectly if there is a nmr_fid.dx file. For the SpinSolve expert data example by @NichVC (and my data as well), this file is missing.
I think the data can be extracted (as in the closed issue) by
data = np.fromfile("spectrum.1d", "<f")[::-1]
data = data[1:131072:2] + 1j*data[0:131072:2]
but you cannot create a dic to extract the header information. Any suggestions?

[kaustubhmote]

@LCageman , I'll be happy to review and add to any PRs you submit. My suggestion would be to start with a simple read function that handles the most basic case in a new spinsolve.py file under nmrglue/fileio, similar to what is there in bruker.py or pipe.py. It should ideally return a dictionary and a np.ndarray similar to all other read functions in nmrglue. Maybe then we can extend it to multi-dimensional datasets as well. Once this is set, one can start with things like the guess_udic and read_pdata functions as well.

[daisyzhu1997]

Hello, is it possible to add a function writing NMR files of spinsolve or convert it to other file format?

[kaustubhmote]

You can convert spinsolve files to other formats via the universal format:

dic, data = ng.spinsolve.read(".")
udic = ng.spinsolve.guess_udic(dic, data)

C = ng.convert.converter()
C.from_universal(udic, data)
dic_pipe, data_pipe = C.to_pipe()
ng.pipe.write(dic_pipe, dic_data)

# dic_bruker, data_bruker = C.to_bruker()
# ng.bruker.write(dic_bruker, dic_bruker)

You cannot convert other formats to spinsolve as of now, but this should not be a hard thing to add.