delphix / dxm-toolkit Goto Github PK

The DXM-Toolkit is a collection of compiled Python command line interface (CLI) executables to assist in automating or extending the Delphix Masking Engine functionality via the APIs.

License: Apache License 2.0

Python 99.85% Dockerfile 0.07% Shell 0.02% Jinja 0.06%

dxm-toolkit's Introduction

Dxm - DxToolkit for Masking

What is it

Dxtoolkit for masking is a command line script to manipulate Delphix Masking Engine and it is delivered by Delphix professional services team. Dxmc script is a single command with parameters and arguments delivered as executable file with libraries. Dxm is written in Python, but no knowledge of Python is equired unless you want to extend it. In fact, no programming experience whatsoever is required to use the Dxtoolkit for masking.

Supperted Delphix Engine version: >= 5.3.X and 6.0.X

What's new

Please check a change log for list of changes.

How to get started

Compiled version

Download a compiled version of Dxm for required platform from a releases page. Dxm keep a configuration using SQLLite database and it's creating a configuration file when a first engine is added. When Dxm is started it will execute an action against Masking engine specified in option or against engine configured as default one.

Compiled version is distributed as single command on Linux and OSX and a zipped folder on Windows platform. Single command distribution on Windows is limited by PyInstaller issue which require to disable some security features on Windows 10. To start a DXM on the Linux or OSX, please run dxmc command from place where it was downloaded. On Windows please run dxmc.exe from unzipped folder.

Configure a masking engine using the following steps:

add engine

dxmc engine add --engine testeng --ip testeng.foo.com --username delphix_admin --default Y
Password:
Repeat for confirmation:
Engine added to configuration

list all engines from configuration

dxmc engine list

Engine name                     IP                              username                        protocol  port   default
==============================  ==============================  ==============================  ========  =====  =======
testeng                         testeng.foo.com                 delphix_admin                   http      8282   Y

check if configuration is OK and list all applications from Masking Engine

dxmc application list

Engine name                     Application name
==============================  ==============================
testeng                         test app 2

Check a documentation for more details

Known issues

N/A

Source version

Python 3.7.X or higher

Required packages

Check setup.py for list standard Python packages

python setup.py install

Contributing

All contributors are required to sign the Delphix Contributor Agreement prior to contributing code to an open source repository. This process is handled automatically by cla-assistant. Simply open a pull request and a bot will automatically check to see if you have signed the latest agreement. If not, you will be prompted to do so as part of the pull request process.

This project operates under the Delphix Code of Conduct. By participating in this project you agree to abide by its terms.

Statement of Support

This software is provided as-is, without warranty of any kind or commercial support through Delphix. See the associated license for additional details. Questions, issues, feature requests, and contributions should be directed to the community as outlined in the Delphix Community Guidelines.

License

This is code is licensed under the Apache License 2.0. Full license is available here.

dxm-toolkit's People

Contributors

Stargazers

Watchers

Forkers

rakesh-roshan tinopi pioro sleep1ng arunbsar pmaluf yogi-learns yogi-delphix experiortec jose-prieto-dlpx dcstolf mathvbarone soluixivanlay jonathan-wilcox stefan-hmsa

dxm-toolkit's Issues

Add ability to update "ID Method" in DXMC Column function

"05/14/2020: Ability to export environments"

The new version of Delphix allows the sync export of environments we should add that capability.

DXMC user list

need to populate the userStatus into the "isLocked" column, remove isLocked

v0.42 & v0.53 - sync list/export failing for object_type PROFILE_EXPRESSION

Same issue is listed here: #8

Masking engine is v5.3.2

Tested a sync list and export with both v0.42 and the latest release (v0.53) and the following error stack is given:

~
└─ $ ▶ dxmc --version
dxmc, version 0.53
~
└─ $ ▶ dxmc sync list --engine lab
EXCEPTION: Traceback (most recent call last):
  File "dxmc.py", line 22, in <module>
  File "click-6.7-py2.7.egg/click/core.py", line 722, in __call__
  File "click-6.7-py2.7.egg/click/core.py", line 697, in main
  File "click-6.7-py2.7.egg/click/core.py", line 1066, in invoke
  File "click-6.7-py2.7.egg/click/core.py", line 1066, in invoke
  File "click-6.7-py2.7.egg/click/core.py", line 895, in invoke
  File "click-6.7-py2.7.egg/click/core.py", line 535, in invoke
  File "click-6.7-py2.7.egg/click/decorators.py", line 64, in new_func
  File "click-6.7-py2.7.egg/click/core.py", line 535, in invoke
  File "dxm/dxm.py", line 2152, in list
  File "dxm/lib/DxSync/sync_worker.py", line 89, in sync_list
  File "dxm/lib/DxSync/sync_worker.py", line 151, in sync_worker
  File "dxm/lib/DxSync/DxSyncList.py", line 47, in __init__
  File "dxm/lib/DxSync/DxSyncList.py", line 90, in LoadSync
  File "dxm/lib/DxTools/DxTools.py", line 38, in paginator
  File "masking_apis/apis/sync_api.py", line 396, in get_all_syncable_objects
  File "masking_apis/apis/sync_api.py", line 477, in get_all_syncable_objects_with_http_info
  File "masking_apis/api_client.py", line 323, in call_api
  File "masking_apis/api_client.py", line 158, in __call_api
  File "masking_apis/api_client.py", line 236, in deserialize
  File "masking_apis/api_client.py", line 276, in __deserialize
  File "masking_apis/api_client.py", line 627, in __deserialize_model
  File "masking_apis/api_client.py", line 254, in __deserialize
  File "masking_apis/api_client.py", line 276, in __deserialize
  File "masking_apis/api_client.py", line 627, in __deserialize_model
  File "masking_apis/models/export_object_metadata.py", line 96, in object_type
ValueError: Invalid value for `object_type` (PROFILE_EXPRESSION), must be one of ['ALGORITHM_REFERENCE', 'BINARYLOOKUP', 'CLEANSING', 'DATABASE_CONNECTOR', 'DATABASE_RULESET', 'DATE_SHIFT', 'DOMAIN', 'DOMAIN_REFERENCE', 'FILE_CONNECTOR', 'FILE_FORMAT', 'FILE_RULESET', 'GLOBAL_OBJECT', 'KEY', 'LOOKUP', 'MAPPLET', 'MASKING_JOB', 'MIN_MAX', 'SEGMENT', 'SOURCE_DATABASE_CONNECTOR', 'SOURCE_FILE_CONNECTOR', 'TOKENIZATION']

[10979] Failed to execute script dxmc
~
└─ $ ▶

clone ruleset > ruleset can't be executed

Whenever I clone ruleset > the action gives no error but if I later try to run a job using the ruleset it does not work. It tells"success" but the rows are not masked. In monitor the rows / masked rows are empty. Tried simple clone ruleset as well by doing additional metadata add ...

add additional column information to dxmc

column save should add PK, column data type, etc to export - probably as last column to import it using column batch

Is this tool compatible for IBM AIX?

Hi Team,

Is this rool compatible with working on IBM AIX platform? Is there a document for this?

Regards..

masking_apis not installing

while installing the Github it is throwing the below error.

Error: Could not find a version that satisfies the requirement masking_apis (from dxm==0.0.0) (from versions: none)
Error: No matching distribution found for masking_apis (from dxm==0.0.0)

21-Oct-2020: ALGORITHM LIST fails against masking engine 6.0.3+

6.0.3 added Environment Syncable objects https://maskingdocs.delphix.com/What%27s%20New%20for%20Masking/#ui-based-environment-sync

This call to DxSyncList() dxm/lib/DxAlgorithm/DxAlgorithmList.py#L71
should specify "ALGORITHM", so that the SyncList is restricted to Algorithms. This would fix the issue (though not add support for Environment Syncing) and should also reduce a few rest calls as Algorithms are only a small percentage of syncable objects.

This can also be fixed by recompiling with the latest 6.x API, which is needed for syncing Environments.

I'm going to build from source, but this should be fixed for new clients.

** Expected Behavior**
kevin$ ./dxmc algorithms list
< formatted list output of algorithms >

** Actual Behavior**
kevin$ ./dxmc algorithms list
EXCEPTION: ValueError: Invalid value for object_type (ENVIRONMENT), must be one of <....>

Steps To Reproduce the Problem
dxmc algorithms list

Screenshots
N/A

Version
Tried latest compiled v0.8.2

Additional Context
Latest commit here https://github.com/pioro/masking_api_60 is for 6.0.2.

"January 7th 2021: To add JOBID when listing Masking jobs and profiling jobs"

Is your feature request related to a problem? Please describe.
While working with a customer, we wanted to create a quick report on all their masking and profiling jobs but one important information that was not there was the JOBID - this was needed for them to keep track but also to make a list of the JOBIDs they would be calling when running APIs.

Describe the solution you'd like
It would be just to print out the JOBID in the "list" option for both dxmc job and dxmc profilejob

Right now, outputs are as follows:

 ~/Documents/Delphix Dynamic Data Platform/DXM Toolkit  ./dxmc job list --engine MaskingEngine2

Engine name Job name Ruleset name Connector name Environment name Completed Status Runtime
============================== ============================== ============================== ============================== ============================== ==================== ==================== ====================
MaskingEngine2 mask1 rs1 testing env1 2020-12-04 20:57:15 SUCCEEDED 0:00:09.274000
MaskingEngine2 mask2 rs2 testing2 env1 2020-11-17 19:52:46 SUCCEEDED 0:00:10.416000
MaskingEngine2 maskjob1 idbrs1 idbtest env1 2020-12-23 01:39:57 SUCCEEDED 0:00:11.774000

 ~/Documents/Delphix Dynamic Data Platform/DXM Toolkit  ./dxmc profilejob list --engine MaskingEngine2

Engine name Job name Ruleset name Connector name Environment name Completed Status Runtime
============================== ============================== ============================== ============================== ============================== ==================== ==================== ====================
MaskingEngine2 profjob1 idbrs1 idbtest env1 2020-12-23 01:37:51 SUCCEEDED 0:00:00.547000

This was tested using latest dxmc -->
 ~/Documents/Delphix Dynamic Data Platform/DXM Toolkit  ./dxmc --version
dxmc, version 0.8.2.1

Would it be possible to add this detail as well? thanks in advance and regards

column save does not export to specified format (JSON)

dxmc v0.53 does not allow for JSON export format when calling column save.

~
└─ $ ▶ dxmc column save --engine lab --rulesetname my_mask --format json --outputfile inventory-ruleset-my_mask.json
Columns saved to file inventory-ruleset-my_mask.json
~
└─ $ ▶ head inventory-ruleset-my_mask.json
#Table Name,Type,Parent Column Name,Column Name,Data Type,Domain,Algorithm,Is Masked,ID Method,Row Type,Date Format
ABCD,FK IX,-,FOO_ID,NUMBER(20),,,N,Auto,All Row,-
ABCD,FK IX,-,BAR_ID,NUMBER(20),,,N,Auto,All Row,-
ABCD,,-,SOME_WORD,VARCHAR2(30),Secrets - CA,1234 Column Value,Y,User,All Row,-
ABCD,FK IX,-,XYZ_ID,NUMBER(20),,,N,Auto,All Row,-

01-DEC-2020 - Invoking dxmc from Jenkins fails

Executing this command directly from the OS works fine :

/home/delphix/dxm-toolkit/dxmc/dxmc job start --engine DelphixDE --envname Dev --jobname job_mask

But If put that same string and use it in Jenkins, it fails with:

Engine name DelphixDE not found in configuration

In order to make it work, I have to specify in my Jenkins pipeline a Change Directory:

sh "cd /home/delphix/dxm-toolkit/dxmc;./dxmc job start --engine DelphixDE --envname Dev --jobname job_mask"

Support for advanced connectors

dxmc algorithms list gives error

dxmc algorithms list --algname EU_COUNTRY --debug --logfile .\log.txt

Error log file contains:
[2018-10-15 14:57:42,095:ERROR:DxAlgorithmList:89 - LoadAlgorithms()] {"errorMessage":"Encountered internal server error, please see the logs for more details"}

CRUD support for domain

"[DATE]: DXM user disable"

New api option to disable a user (instead deleting it)

dxmc fails to launch multi-tenant job

delphix@labserver:~/dxmc$ ./dxmc job start --engine ME --jobname "mask-ora-vdb" --envname "masking" --tgt_connector "new-connector" --tgt_connector_env "masking"
Input error: unable to convert input to com.dmsuite.swagger.model.Execution

dxm toolkit throws an error when endtime is not set for a job during execution

** Expected Behavior**
Describe what should have happened.
if a job is running - Execution end time is not set and code is expecting it to be there.

** Actual Behavior**
Describe what actually happened.
/opt/nifi/Delphix/scripts/Engine_Sync/dxmtoolkit/dxmc/dxmc job report --engine edsdmnp1001 --startdate "2021-03-01 01:01:01" --enddate "2021-03-30 01:01:01" --format csv > edsdmnp1001_Mar2021_mem.csv

EXCEPTION: TypeError: '<=' not supported between instances of 'NoneType' and 'datetime.datetime'

Steps To Reproduce the Problem
Describe in as much detail as possible how to reproduce the problem.

Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error
[2021-04-01 03:22:28,714:ERROR:DxLogging:107 - print_exception()] EXCEPTION: Traceback (most recent call last):
File "dxmc.py", line 22, in
File "click/core.py", line 829, in call
File "click/core.py", line 782, in main
File "click/core.py", line 1259, in invoke
File "click/core.py", line 1259, in invoke
File "click/core.py", line 1066, in invoke
File "click/core.py", line 610, in invoke
File "click/decorators.py", line 73, in new_func
File "click/core.py", line 610, in invoke
File "dxm/dxm.py", line 1099, in report
File "dxm/lib/DxJobs/jobs_worker.py", line 785, in jobs_report
File "dxm/lib/DxJobs/jobs_worker.py", line 1032, in jobs_report_worker
File "dxm/lib/DxJobs/DxJob.py", line 566, in filter_executions
File "dxm/lib/DxJobs/DxJob.py", line 566, in
TypeError: '<=' not supported between instances of 'NoneType' and 'datetime.datetime'

If applicable, add screenshots to help explain your problem.

Version
Indicate the relevant versions of components. This may include the package version, operating system version, or other components relevant to the project.

Additional Context
Add any other context about the problem here, including additional logs or debugging information if appropriate.

Export all rulesets in an Environment

Customer wants export all rulesets from an environment using a single command.

Add ability to delete tables from rule set with table list in an input file (ruleset deletemeta)

Ruleset deletemeta function does not support deletion of tables from rule sets via an input file

The dxmc connector fetch_meta doesn't support the --format json

$ dxmc connector fetch_meta --envname ${ENV} --connectorname ${CNAME} --engine ${DE} --format json

Engine name Connector name Table name
============================== ============================== ========================================
de2 Conn3 BITT_MV
de2 Conn3 EMPLOYEES
de2 Conn3 EMPLOYEES_1810

advanced jdbc string support

adding a support for advanced jdbc connector string

Ability to export / import profile set including profile set name, expression name, expression text

Several clients have requested ability to export (JSON) for profile set and expressions (including expression text) to enable profile set copy/changes for all expressions included in the profile set. This will simplify creating copies of existing profile set to enable changes to expressions in source profile set. This capability will also be required to implement Accelerator Packs when the Masking Initializer is deprecated.

"[DATE] [BUG]" DXMC export of ruleset as csv/fixed results in json output

** Expected Behavior**
Tried to export a ruleset as csv/fixed

** Actual Behavior**
I get json format

Steps To Reproduce the Problem

#Csv export
./dxmc ruleset exportrule --exportmeta N --rulesetname 1_diarietAlwaysRs --format csv --outputfile /tmp/1_diarietAlwaysRs.csv
Ruleset exported to file /tmp/1_diarietAlwaysRs.csv

head /tmp/1_diarietAlwaysRs.csv
[
{
"Ruleset name": "1_diarietAlwaysRs",
"Connector name": "diarietTarget",
"Environent name": "diarietEnv",
"Metadata": [],
"Columns": []
}

#Fixed export
./dxmc ruleset exportrule --rulesetname 1_diarietAlwaysRs --format fixed --outputfile /tmp/1_diarietAlwaysRs.txt
Ruleset exported to file /tmp/1_diarietAlwaysRs.txt

head /tmp/1_diarietAlwaysRs.ctxt
[
{
"Ruleset name": "1_diarietAlwaysRs",
"Connector name": "diarietTarget",
"Environent name": "diarietEnv",
"Metadata": [
{
"table": true,
"meta_name": "FIL",
"key_column": "ROWID",
Version
6.0.5.0

Job history report

Masking users have a common requirement for job execution reporting. Options for reporting by job name and also options for date ranges are desired.

"[DATE] [BUG]"Ruleset clone with advancde oracle db connector results in error

** Expected Behavior**
Cloning of ruleset creates a new ruleset

** Actual Behavior**
EXCEPTION: AttributeError: 'DxDatabaseRuleset' object has no attribute 'database_connector_id'

Steps To Reproduce the Problem
./dxmc ruleset clone --rulesetname 2_aishprAddressRs --newrulesetname 2_aishprAddressRsClone --debug
EXCEPTION: Traceback (most recent call last):
File "dxmc.py", line 22, in
File "click/core.py", line 829, in call
File "click/core.py", line 782, in main
File "click/core.py", line 1259, in invoke
File "click/core.py", line 1259, in invoke
File "click/core.py", line 1066, in invoke
File "click/core.py", line 610, in invoke
File "click/decorators.py", line 73, in new_func
File "click/core.py", line 610, in invoke
File "dxm/dxm.py", line 856, in clone
File "dxm/lib/DxRuleset/rule_worker.py", line 327, in ruleset_clone
File "dxm/lib/DxRuleset/rule_worker.py", line 364, in ruleset_worker
File "dxm/lib/DxRuleset/rule_worker.py", line 387, in do_clone
File "dxm/lib/DxRuleset/DxRulesetList.py", line 312, in copy
File "dxm/lib/DxRuleset/DxDatabaseRuleset.py", line 129, in add
AttributeError: 'DxDatabaseRuleset' object has no attribute 'database_connector_id'

Version
6.0.5.0

Additional Context
We use an oracle advanced connector:
jdbc:oracle:thin:@(DESCRIPTION=(ENABLE=broken)(ADDRESS=(PROTOCOL=tcp)(HOST=xxx)(PORT=yyyy))(CONNECT_DATA=(UR=A)(SERVICE_NAME=vpdbzzz)))

"05-06-2020 Delete file ruleset"

Expected Behavior
file set to be deleted

Actual Behavior

dxmc ruleset delete --rulesetname ZZZ --envname ENC1 --format fixed --engine XXX

EXCEPTION: Traceback (most recent call last):
  File "C:\Users\pioro\Documents\oss_delphix\dxmwindows\lib\site-packages\cx_Freeze\initscripts\__startup__.py", line 14, in run

  File "C:\Users\pioro\Documents\oss_delphix\dxmwindows\lib\site-packages\cx_Freeze\initscripts\Console.py", line 26, in run

  File "dxmc.py", line 22, in <module>

  File "C:\Users\pioro\Documents\oss_delphix\dxmwindows\lib\site-packages\click-6.7-py2.7.egg\click\core.py", line 722, in __call__

  File "C:\Users\pioro\Documents\oss_delphix\dxmwindows\lib\site-packages\click-6.7-py2.7.egg\click\core.py", line 697, in main

  File "C:\Users\pioro\Documents\oss_delphix\dxmwindows\lib\site-packages\click-6.7-py2.7.egg\click\core.py", line 1066, in invoke

  File "C:\Users\pioro\Documents\oss_delphix\dxmwindows\lib\site-packages\click-6.7-py2.7.egg\click\core.py", line 1066, in invoke

  File "C:\Users\pioro\Documents\oss_delphix\dxmwindows\lib\site-packages\click-6.7-py2.7.egg\click\core.py", line 895, in invoke

  File "C:\Users\pioro\Documents\oss_delphix\dxmwindows\lib\site-packages\click-6.7-py2.7.egg\click\core.py", line 535, in invoke

  File "C:\Users\pioro\Documents\oss_delphix\dxmwindows\lib\site-packages\click-6.7-py2.7.egg\click\decorators.py", line 64, in new_func

  File "C:\Users\pioro\Documents\oss_delphix\dxmwindows\lib\site-packages\click-6.7-py2.7.egg\click\core.py", line 535, in invoke

  File "C:\Users\pioro\Documents\oss_delphix\dxmwindows\dxm\dxm.py", line 793, in delete

  File "C:\Users\pioro\Documents\oss_delphix\dxmwindows\dxm\lib\DxRuleset\rule_worker.py", line 299, in ruleset_delete

  File "C:\Users\pioro\Documents\oss_delphix\dxmwindows\dxm\lib\DxRuleset\rule_worker.py", line 361, in ruleset_worker

  File "C:\Users\pioro\Documents\oss_delphix\dxmwindows\dxm\lib\DxRuleset\rule_worker.py", line 372, in do_delete

  File "C:\Users\pioro\Documents\oss_delphix\dxmwindows\dxm\lib\DxRuleset\DxRulesetList.py", line 262, in delete

  File "C:\Users\pioro\Documents\oss_delphix\dxmwindows\dxm\lib\DxRuleset\DxFileRuleset.py", line 108, in delete

AttributeError: 'DxFileRuleset' object has no attribute 'rulesetId'

Version

0.6

fails to create masking job for "on the fly" option

Will return below error
This OnTheFly MaskingJob needs a source File Connector specified.

Example: ./dxmc job add --jobname API-BC_TACCOUNT --envname AwsMask-Target --rulesetname API-BC_TACCOUNT --on_the_fly_source SOURCE-BC_TACCOUNT --max_memory 8192 --min_memory 4096 --num_streams 20 --on_the_fly Y

To start profile job we need to specify target connector

Hello,

Today doing some testings, noticed that to run a profile job, we need to specify a target connector in order to launch the job properly.

PS C:\Users\delphix\Documents\dxmc-0.42-5.3.2> .\dxmc.exe profilejob start --jobname profilejob --envname carloscenv
Connector None not found
Target connector not found
Missing target connector information for multi-tenant job

Can you please help me check it?

Thanks and regards

dxmc job list returns error

issue occurs in .40 and .41 release

No OSX compiled versions provided for 0.7 release

Hello!

I do not see an OSX compiled binary for version 0.7. Can that be built and provided? Alternatively, can you provide build instructions?

Thanks!

"05.26.21: Add connector for Extended JDBC"

its currently not possible to use dxmc to add a new connector when this connector is based on an extended JDBC driver that was loaded. In our test case, we loaded an S4HANA JDBC driver to the masking engine and not able to specify (for example) what JDBC driver you want this connector to be mapped to...

"05/23/2021 Unable to create OTF job

** Expected Behavior**
Create OTF job

** Actual Behavior**
EXCEPTION: ValueError: Invalid value for connector_id, must not be None

Steps To Reproduce the Problem

./dxmc connector test --engine mymskengine --connectorname orasrc_conn                                                                                                         
Connector test orasrc_conn succeeded
./dxmc job add --jobname maskjob7 --engine mymskengine --envname mskdevenv --rulesetname myruleset2 --max_memory 2048 --min_memory 2048 --num_streams 1 --on_the_fly Y --on_the_fly_src_connector orasrc_conn --on_the_fly_src_envname mskprdenv

EXCEPTION: ValueError: Invalid value for connector_id, must not be None
[13101] Failed to execute script dxmc

Version
version 0.8.2.1

"05/10/2020 Unable to call pre and post script with script Path"

** Expected Behavior**
Pre or postscript should be called from the path where the script resides when adding a masking job.

** Actual Behavior**
Unable to call pre or post script with path to script

Jatinders-MBP-2:dxmc jatinder.luthra$ ./dxmc job add --envname ELASTIC_ENV --rulesetname empty_rs --jobname empty_job --prescript /Users/jatinder.luthra/Desktop/AgileMasking/ElasticMasking/scripts/elastic-mask-log-05182020-132933/preScript.sql --engine primary
Invalid character '/' in field 'preScriptPathTx'
Jatinders-MBP-2:dxmc jatinder.luthra$ ./dxmc job add --envname ELASTIC_ENV --rulesetname empty_rs --jobname empty_job --prescript "/Users/jatinder.luthra/Desktop/AgileMasking/ElasticMasking/scripts/elastic-mask-log-05182020-132933/preScript.sql" --engine primary
Invalid character '/' in field 'preScriptPathTx'

Jatinders-MBP-2:dxmc jatinder.luthra$ ./dxmc job add --envname ELASTIC_ENV --rulesetname empty_rs --jobname empty_job --prescript "//Users//jatinder.luthra//Desktop//AgileMasking//ElasticMasking//scripts//elastic-mask-log-05182020-132933//preScript.sql" --engine primary
Invalid character '/' in field 'preScriptPathTx'
Jatinders-MBP-2:dxmc jatinder.luthra$ ./dxmc job add --envname ELASTIC_ENV --rulesetname empty_rs --jobname empty_job --prescript "\/Users\/jatinder.luthra\/Desktop\/AgileMasking\/ElasticMasking\/scripts\/elastic-mask-log-05182020-132933\/preScript.sql" --engine primary
Usage: dxmc job add [OPTIONS]
Error: Invalid value for "--prescript": Could not open file: \/Users\/jatinder.luthra\/Desktop\/AgileMasking\/ElasticMasking\/scripts\/elastic-mask-log-05182020-132933\/preScript.sql: No such file or directory

Steps To Reproduce the Problem
Just call masking job with pre or post script path

Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Screenshots
If applicable, add screenshots to help explain your problem.

Version
Jatinders-MBP-2:dxmc jatinder.luthra$ ./dxmc --version
dxmc, version 0.7
MacOS

Additional Context
Add any other context about the problem here, including additional logs or debugging information if appropriate.

"November 18th 2020: Not able to Import a ruleset from a JSON file created by exportrule command

** Expected Behavior**
When running "dxmc ruleset importrule" and by selecting a previously exported ruleset in json format, we should be able to import it passing all the required parameters

** Actual Behavior**
"dxmc ruleset importrule" did not work as expected and throw this message:

carlos.cuellar@CarlosCllarsMBP  ~/Documents/Delphix Dynamic Data Platform/CAEtest2/dxmc  ./dxmc ruleset importrule --rulesetname rs3 --connectorname testing --envname env1 --inputfile carlos.json --format json --engine dlpxCAEmsk01 --debug
Ruleset rs3 added
EXCEPTION: Traceback (most recent call last):
File "dxmc.py", line 22, in
File "click/core.py", line 829, in call
File "click/core.py", line 782, in main
File "click/core.py", line 1259, in invoke
File "click/core.py", line 1259, in invoke
File "click/core.py", line 1066, in invoke
File "click/core.py", line 610, in invoke
File "click/decorators.py", line 73, in new_func
File "click/core.py", line 610, in invoke
File "dxm/dxm.py", line 906, in importrule
File "dxm/lib/DxRuleset/rule_worker.py", line 742, in ruleset_import
TypeError: ruleset_addmeta() missing 2 required positional arguments: 'fromconnector' and 'bulk'

[26936] Failed to execute script dxmc

This command did created a new ruleset but did not import the inventory.

Steps To Reproduce the Problem
1.- Add a Delphix Masking engine (version 6.0.4.1) to dxmtoolkit.
2.-Export an existing ruleset --> dxmc ruleset exportrule --rulesetname rs1 --envname env1 --outputfile carlos.json --format json --engine dlpxCAEmsk01
3.-Reuse the json file created from step 2 and try to create a new ruleset--> dxmc ruleset importrule --rulesetname rs3 --connectorname testing --envname env1 --inputfile carlos.json --format json --engine dlpxCAEmsk01

Version
Tested with latest dxmc 0.8.2.1 and with a Delphix Masking engine 6.0.4.1

need ability to create a profile job, don't see a profile job option in the "dxmc job add --help" command

Need the ability to create a profile job.

add source_connector / source_env to start multitenant OTF jobs

Require source_connector / source_env argument to create / start OTF multitenant jobs

dxmc column setmasking needs additional parameter support

For adding algorithms ...
$ dxmc column setmasking --rulesetname ${RSNAME} --envname ${ENV} --columnname ${NAM2} --algname ${VAL3} --domainname ${VAL2}
... need ability to specify user/auto profile writable type and for date related algorithms, such as DOB, need to specify date formats.
i.e. JSON parameters
"isProfilerWritable": true|false
"dateFormat": "YYYY-MM-DD"

Add automation functionality export/import

Add functionality export/import (based on Json format) that allows to use result of "export" function as definition for "import" function. Should work for all functions:

algorithms
application
column
connector
environment
fileformat
job
meta
ruleset

If the function is "connector export" --connectorname X .. and JSON result has variable "username = peter " by adding into call --username Derek we can overwrite a variable from the Json using the command line.

Non-Default Algorithms

When running the inventory batch for columns, it does not account for non-default algorithms/domains. I added an SSN algorithm to the inputfile for inventory, but it received an error on "Wrong algoritm name in input file. Not an inventory file ?". This is most likely due to the code not handling the ability to assign algorithms if they don't exist in the default conversion lookup process.

Algorithm functions fail as of masking engine version 6.0.5

Algorithms list and replace fail due to long description fields in new OOB CM algorithms

C:\Toolkit\dxmc821\dxmc>dxmc algorithms list
EXCEPTION: ValueError: Invalid value for description, length must be less than or equal to 255

[18700] Failed to execute script dxmc

Read Timeout Error

Cloning a huge ruleset, but dxmc crashes for timeout

maxfratini-MacbookPro:mfratini:Masking $./dxmc ruleset clone --rulesetname VISORA --newrulesetname VISORA_P1
EXCEPTION: Traceback (most recent call last):
  File "dxmc.py", line 22, in <module>
  File "site-packages/click-6.7-py2.7.egg/click/core.py", line 722, in __call__
  File "site-packages/click-6.7-py2.7.egg/click/core.py", line 697, in main
  File "site-packages/click-6.7-py2.7.egg/click/core.py", line 1066, in invoke
  File "site-packages/click-6.7-py2.7.egg/click/core.py", line 1066, in invoke
  File "site-packages/click-6.7-py2.7.egg/click/core.py", line 895, in invoke
  File "site-packages/click-6.7-py2.7.egg/click/core.py", line 535, in invoke
  File "site-packages/click-6.7-py2.7.egg/click/decorators.py", line 64, in new_func
  File "site-packages/click-6.7-py2.7.egg/click/core.py", line 535, in invoke
  File "dxm/dxm.py", line 825, in clone
  File "dxm/lib/DxRuleset/rule_worker.py", line 324, in ruleset_clone
  File "dxm/lib/DxRuleset/rule_worker.py", line 361, in ruleset_worker
  File "dxm/lib/DxRuleset/rule_worker.py", line 384, in do_clone
  File "dxm/lib/DxRuleset/DxRulesetList.py", line 290, in copy
  File "dxm/lib/DxRuleset/DxDatabaseRuleset.py", line 89, in add
  File "site-packages/masking_apis/apis/database_ruleset_api.py", line 173, in create_database_ruleset
  File "site-packages/masking_apis/apis/database_ruleset_api.py", line 250, in create_database_ruleset_with_http_info
  File "site-packages/masking_apis/api_client.py", line 323, in call_api
  File "site-packages/masking_apis/api_client.py", line 150, in __call_api
  File "site-packages/masking_apis/api_client.py", line 368, in request
  File "site-packages/masking_apis/rest.py", line 248, in POST
  File "site-packages/masking_apis/rest.py", line 148, in request
  File "site-packages/urllib3/request.py", line 72, in request
  File "site-packages/urllib3/request.py", line 150, in request_encode_body
  File "site-packages/urllib3/poolmanager.py", line 322, in urlopen
  File "site-packages/urllib3/connectionpool.py", line 638, in urlopen
  File "site-packages/urllib3/util/retry.py", line 367, in increment
  File "site-packages/urllib3/connectionpool.py", line 600, in urlopen
  File "site-packages/urllib3/connectionpool.py", line 386, in _make_request
  File "site-packages/urllib3/connectionpool.py", line 306, in _raise_timeout
ReadTimeoutError: HTTPConnectionPool(host=u'172.16.87.149', port=80): Read timed out. (read timeout=15)

the problem is in DxMaskingEngine.py that returns always 15 seconds

   @classmethod
    def get_timeout(self):
        """
        Return timeout for query
        Tuple (connect_timeout, read_timeout)
        """
        *return (5, 15)*
        """    enginelist = get_list_of_engines(p_engine)

    if enginelist is None:
        return 1

    data = DataFormatter()
    data_header = [
                    ("Engine name", 30),
                    ("Application name", 30),
                  ]
    data.create_header(data_header)
    data.format_type = format
    for engine_tuple in enginelist:
        engine_obj = DxMaskingEngine(engine_tuple)
        if engine_obj.get_session():
            continue
        applist = DxApplicationList()
        # load all objects
        applist.LoadApplications()
      """

"05-06-2020 adding file ruleset"

Expected Behavior

New file ruleset should be added

Actual Behavior

DXMC is expecting to see --file_enclosure parameter, even if this is not mandatory

Version

all

Additional Context

01/26/2021 Delimited file cannot display columns list

Hi,

When we specify a ruleset with Delimited File type, the dxmc command cannot extract the list of columns of this ruleset. I receive this error message:

-bash-4.2$ ./dxmc column list --rulesetname HCM_ALL --is_masked --engine prod
EXCEPTION: AttributeError: 'DxFileField' object has no attribute 'date_format'

[32377] Failed to execute script dxmc
-bash-4.2$

A+

"14/02/2021 'dxmc run job' needs updating to handle jobs that are queued or cancelled"

Is your feature request related to a problem? Please describe.

In 6.0.4.0 the masking application was enhanced as follows

_"Masking Job Memory Improvements

Memory management has been dramatically improved. Not only can jobs run with less memory, but the Masking Engine will also now ensure that jobs can only run if enough memory is available and that the engine cannot run out of memory.

Along with these changes, there are two new execution statuses: CANCELLED and QUEUED."_

dxm-toolkit needs updating to handle this. Currently if the command is used to execute a jobs and that job is queued dxmc exits 1 as if the job failed (although at this staging the status of the job is now known)

Example output;

./dxmc job start --envname Env --jobname job1
Waiting for job job1 to start processing rows
Problem with masking job job1
0 rows masked

echo $?
1

However the actual job status is not known at this time as it's queued.

Describe the solution you'd like

'dxmc run job' should return 0 if and only if the job has completed successfully. Return non zero if the job does not complete successfully. If the job is cancelled then it should return non zero, in the event it's queued the command should continue checking the status until a result can be obtained.

column type is not displayed

Not able to create mssql connector

Hello,

Yesterday I was trying to create a new mssql connector in a masking engine but it was complaining about the following parameters:

Property username can't be empty
Property databaseName can't be empty
Property schemaName can't be empty
Property instanceName can't be empty
Property connectorName can't be empty

The commands I was trying to execute were:

./dxmc connector add --connectorname carloscconn --envname carloscenv --connectortype mssql --host 172.16.126.135 --port 1433 --schemaname dbo --username delphix --databasename maskingvdb --engine delphix5320 --instancename mssqlserver

Can you please help me check it?

Thanks in advance and regards.

"[05262020] DateFormat not extracted with column list"

Is your feature request related to a problem? Please describe.
While extracting the column list, dateFormat is not extracted for columns which has date algorithm applied.

Jatinders-MBP-2:dxmc jatinder.luthra$ ./dxmc column list --envname 4T_50M_Eeach --rulesetname RS_4T_50M_EACH --engine primary --metaname customer1

Engine name                     Environment name                Ruleset name                    Metadata name                     Column name                       Type      Data type                       Domain name                       Alg name                          
==============================  ==============================  ==============================  ================================  ================================  ========  ==============================  ================================  ================================  
primary                         4T_50M_Eeach                    RS_4T_50M_EACH                  customer1                         DOB                                         date(13)                        DOB                               DateShiftDiscrete                 
primary                         4T_50M_Eeach                    RS_4T_50M_EACH                  customer1                         EMAIL                                       varchar(100)                    EMAIL                             EmailLookup                       
primary                         4T_50M_Eeach                    RS_4T_50M_EACH                  customer1                         FILLER1                                     varchar(50)

Describe the solution you'd like
Provide date format for columns masked with date algorithm

Describe alternatives you've considered
N/A

Additional context
N/A

add ruleset to database - v0.22

Connector has a d<id> instead of <id>

Sync list/export fails with "Invalid value for `object_type" error

Error occurred in dxmc v 0.4 and with Delphix Masking Engine version 5.3.2.0

I was trying to use the sync export and sync list on one of our projects and am getting the same error. It appears that the list function is returning Profile_Expression objects that it cannot consume

EXCEPTION: Traceback (most recent call last):
File "dxmc.py", line 22, in
File "click/core.py", line 764, in call
File "click/core.py", line 717, in main
File "click/core.py", line 1137, in invoke
File "click/core.py", line 1137, in invoke
File "click/core.py", line 956, in invoke
File "click/core.py", line 555, in invoke
File "click/decorators.py", line 64, in new_func
File "click/core.py", line 555, in invoke
File "dxm/dxm.py", line 2046, in list
File "dxm/lib/DxSync/sync_worker.py", line 85, in sync_list
File "dxm/lib/DxSync/sync_worker.py", line 148, in sync_worker
File "dxm/lib/DxSync/DxSyncList.py", line 47, in init
File "dxm/lib/DxSync/DxSyncList.py", line 89, in LoadSync
File "dxm/lib/DxTools/DxTools.py", line 38, in paginator
File "lib/python2.7/site-packages/masking_apis/apis/sync_api.py", line 396, in get_all_syncable_objects
File "lib/python2.7/site-packages/masking_apis/apis/sync_api.py", line 477, in get_all_syncable_objects_with_http_info
File "lib/python2.7/site-packages/masking_apis/api_client.py", line 323, in call_api
File "lib/python2.7/site-packages/masking_apis/api_client.py", line 158, in __call_api
File "lib/python2.7/site-packages/masking_apis/api_client.py", line 236, in deserialize
File "lib/python2.7/site-packages/masking_apis/api_client.py", line 276, in __deserialize
File "lib/python2.7/site-packages/masking_apis/api_client.py", line 627, in __deserialize_model
File "lib/python2.7/site-packages/masking_apis/api_client.py", line 254, in __deserialize
File "lib/python2.7/site-packages/masking_apis/api_client.py", line 276, in __deserialize
File "lib/python2.7/site-packages/masking_apis/api_client.py", line 627, in __deserialize_model
File "lib/python2.7/site-packages/masking_apis/models/export_object_metadata.py", line 96, in object_type
ValueError: Invalid value for object_type (PROFILE_EXPRESSION), must be one of ['ALGORITHM_REFERENCE', 'BINARYLOOKUP', 'CLEANSING', 'DATABASE_CONNECTOR', 'DATABASE_RULESET', 'DATE_SHIFT', 'DOMAIN', 'DOMAIN_REFERENCE', 'FILE_CONNECTOR', 'FILE_FORMAT', 'FILE_RULESET', 'GLOBAL_OBJECT', 'KEY', 'LOOKUP', 'MAPPLET', 'MASKING_JOB', 'MIN_MAX', 'SEGMENT', 'SOURCE_DATABASE_CONNECTOR', 'SOURCE_FILE_CONNECTOR', 'TOKENIZATION']