meaningful-data / sdmxthon Goto Github PK

Add Changelog file and github link using Read The Docs theme

Implement methods to write XML from this classes and then submit them to FMR.

• 'code' parameter of WebService should be chaged to 'agency_code'

Hi, thank you for this library.

I am trying to read an sdmx file (at this url: 'https://www.i14y.admin.ch/api/CodeLists/CL_HGDE_KT/exports/SDMX-ML/2.1?annotations=false') and I get errors due to the date string format. The library only allows the format %Y-%m-%d but I have date time format of the type 1978-12-31T23:00:00 in my file. Would it be possible to update the library to allow this format ?

Something like this in 'set_date_from_string' in model/utils.py

def set_date_from_string(value: str, format_: str = "%Y-%m-%dT%H:%M:%S"):
    """Generic function to format a string to datetime

    Args: value: The value to be validated.
    format_: A regex pattern to validate if the string has a specific format

    Returns:
        A datetime object

    Raises:
        ValueError: If the value violates the format constraint.
    """

    if value is None:
        return None
    
    for fmt in (format_, "%Y-%m-%d", "%Y-%m-%dT%H:%M:%S"):
        try:
            return datetime.strptime(value, fmt)
        except ValueError:
            pass
        
    raise ValueError(f"Wrong date string format. The formats {format_} "
                     f"or %Y-%m-%d or %Y-%m-%dT%H:%M:%S "
                     f"should be followed. {str(value)} passed")

Thank you :)

Fix bug on get_codes as it raises an unhandled exception if the attribute descriptor is None.

message = read_sdmx('https://stats.bis.org/api/v1/data/BIS,WS_CBPOL_D,1.0/all/all?lastNObservations=1&detail=full')
dataflow_code = list(message.content.keys())[0]
dataset = message.content[dataflow_code]
print(dataset.dataflow)

returns None, shouldn't it return the dataflow?

Implement method on DataSet class to validate data using SDMX-CSV from the Pandas Dataframe.

Signature of method should allow the definition of the FMR host (domain and port, on separated arguments. Port must be an integer between 1 and 65535). Even on remote it is asynchronous, the process should wait until the validation has status completed and returns the response from FMR (in the future we may change this)

Endpoints to be used:
https://fmrwiki.sdmxcloud.org/Asynchronous_Data_Validation_and_Transformation_Web_Service

Process:

1. Ensure the host is up and available -> Exception if not.
2. Generate SDMX-CSV in memory (for now, expect to use disk on large files in the future) and send request
3. Get UID from response and query status (until status is Completed, interval of 0.5s (default value with parameter)
4. Parse response as JSON when status is completed and check for errors.

https://fmrwiki.sdmxcloud.org/Data_Validation_Web_Service#Dataset_with_Errors

Add the webservices docstrings in functions and a brief description with .rst files.

Now, it is mandatory to have parameters (for instance, for BIS, Agency Code), when most of the times we want to query a specific option (e.g., for BIS, if nothing is passed, BIS agency should be selected)

Add support for SDMX-CSV in get_pandas_df and get_datasets. Make sure the limitations of these methods are still present (data file checking and same output)

Add tests for the following items:

• Reading data file with Dataflow (readingValidations)
• Writing more than one dataset (dataWriting)
• Read from URL using query builder (queryBuilder)
• Test get_pandas_df with SDMX-CSV (APImethods)
• Test get_datasets with SDMX-CSV (APImethods)
• Test metadata download from data (APImethods)

Adapt read_sdmx to digest files and use convenience method to read SDMX-ML, SDMX-CSV and SDMX-JSON

Running the following code returns a NoneType. Is this expected?

import sdmxthon

url = "https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/dataflow/ESTAT/MET_EDAT_LFSE4/1.0?detail=full"

message = sdmxthon.read_sdmx(url)
resource = message.payload[list(message.payload)[0]]
df = resource[list(resource)[0]]

df.structure

I would expect it returns <s:Structure>, for example:
<Ref id="MET_EDAT_LFSE4" version="18.0" agencyID="ESTAT" package="datastructure" class="DataStructure"/>

Is this correct or is there another way to handle it?

Thank you very much in advance.

Change message payload to retrieve the sole object if only one of one kind is found:

• If a DataSet is present and it is only one, return the Dataset object
• If any ItemScheme or Definition is present, return the sole object
• If more than one is present, but of the same kind, return a list
• If more than one is present from different kinds, return same type as content

Add flag use_dataset_id on get_pandas_df

Check error sdmxthon/parsers/data_read.py", line 135, in create_dataset
df = pd.DataFrame(dataset[OBS]).replace(np.nan, '')

Should be able to go through when Dataframe has only 1 record.

Add functionality to validate a dataset column with the codes of a Codelist, passed as argument to the function.

It must support unique_id as argument and download of the codelist from the WebService, if available, as well as passing the Codelist object.

For reader, use lxml.etree to iterate over data and get pandas dataframe. Add Streaming option.

For writer, use this: https://stackoverflow.com/a/43567758

Explore how to handle streaming reading and writing.

Running the following does not return any code

import sdmxthon message = sdmxthon.read_sdmx("https://www.ilo.org/sdmx/rest/codelist/ILO/CL_AREA/1.0?detail=full") message.content["Codelists"]['ILO:CL_AREA(1.0)'].items

However, running the same code for Eurostat seems to work properly:
message = sdmxthon.read_sdmx("https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/codelist/ESTAT/GEO/1.0?detail=full") message.content["Codelists"]['ESTAT:GEO(1.0)'].items

Could you kindly help with this issue? Thank you

message = read_sdmx('https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data/BOP_C6_A/?lastNObservations=1', validate=True)

If a dataset does not have a structure associated, calling the semantic_validation method should raised a specific exception

• Read_sdmx file that points to a dataflow.
• Use dataset.dataflow
• Perform structural validation.
• Read csv file and use pd.assert_equal to compare two dataframes.

Revision dataset attributes bug on _check_DA_keys when removing a key in the Dataset attributes.

Revision SDMX-CSV v2 to include action column (default value I -> Inform)

Implement method in Message class that allows a Structure message to be submitted into FMR.

Documentation on FMR endpoint: https://fmrwiki.sdmxcloud.org/Submit_Structures_Web_Service

Requires basic authentication and an admin account.

Change read_sdmx to use webservices by downloading any possible metadata we may find using the query builder.

I'm following the notebook for SDMX comprehensive example, but I keep getting an error while initializing the Dataset object. (See attached)

Any thoughts on how this can be resolved?

Add support for XML files as string on all methods, focusing on read_sdmx