phdata / pipeforge Goto Github PK

I was thinking that it would be useful if the .yml file input to Pipeforge would be able to handle something like this...

schema:

• myschema1
  tables:
  • table1
  • table2
• yourschema2
  tables:
  • table3
  • table4

So I think we'd have to add a loop on schema surrounding our current code that loops through the tables.
Then in the tables.yml we'd just want it to output
source:
name: table1
schema: myschema1

Right now, when we are sourcing from a single database but multiple schemas, we're having to maintain the files and the outputs in separate directories. The above could cause some issues if the same table existed in multiple schemas, but we already would have similar issues in the target database if that's the case.

In order to simplify the process if a client wants to return both views and tables with a single tables.yml we would need to enhance the listTables method to allow a 3rd option.

This is not critical, but I also don't think it would necessarily be super difficult