dalenewman / transformalize

Configurable Extract, Transform, and Load

Hi there 👋

transformalize's Issues

Question about documentation

Hello, @dalenewman !
Transformalize looks great, but I faced one problem: I just can't find information about functionality. Are you going to publish some more documentation? It would be great if you could publish detailed information about API (for using Transformalize as Nuget package) or at least about configurating through XML.
Thanks a lot!

quick installs for mac os, linux, and windows wsl2

Write up install script like this tool has.

Some questions about features

Hi Dale. You solution seems to be most close to my ideas about how ETL should be looked like :)

I have couple of questions:

1. Are star-schema warehouses supported by Transformalize? When I have many sources loading data into single table?
2. Is is posstible to handle incremental updates by multiple columns? Is it ok to specify primary-key attribute on multiple columns?
3. Is it possible to handle time-based incremental updates? For example we can have lot of historical data we'd like to exclude from update, because we're sure it can't be changed. It would be great to have option to add some 'date-of-origin' attribute and ability to specify that we not going to update data before some date?

Thank you.

Transforming data to GeoJSON format

I have two fields (long,lat) in SQL Server and I have tried to transform those two fields to GeoJSON format, because I would like to use the bettermap panel with Kibana. I cannot figure out how to create such a transform from two fields and send it to the ElasticSearch.

http://www.elasticsearch.org/guide/en/kibana/current/_bettermap.html

Multiple unrelated entities

Is there a way to have multiple unrelated entities in the same config file? I have several tables that don't have relationships and when I run the init command it gives this error: You have 2 entities so you need 1 relationships. You have 0 relationships.

Can I to use tfl in Visual Code in Ubuntu ?

Hi

Can I to use tfl in Visual Code in Ubuntu ?

mapping

How to mapping input table field to output table field?

Logo contribution

Hello @dalenewman , I am a graphic designer and i like to contribute to open source software. So i designed a logo for transformalize. I tried to represent the function of the application. I used a load, a conversion symbol, and the letter t. Any other ideas? What do you think? I will wait for feedback.

Have a nice day.

Linux?

Is it possible to run this tool on Linux? Perhaps with .net core? I'm interested in gathering data from a local Linux MySQL server, doing some transformations and de-normalization, then pushing it to another Linux server back into MySQL. Thanks!

How to use multiple inputs and multiple outputs

Transformalize looks great, but I faced one problem: I have two files orders1.csv and orders2.csv, how to union them and output them to one file.

Getting data out from SQLServer table that has a different schema than dbo

I cannot figure out how to write the right config for getting data out from SQLServer when there is a different schema than dbo.

Here is my current config that I'm trying to use for getting the metadata created. I have no problems getting the metadata created when the table is created in the default "dbo" schema.

<transformalize>
<processes>
    <add name="MyData">
        <connections>
            <add name="input" 
            provider="sqlserver" 
            enabled="true" 
            schema="schemaname" 
            connection-string="Data Source=server; Initial Catalog=databasename; Trusted_Connection=True;" />                
            <!--<add name="output" provider="elasticsearch" server="localhost" port="9200" />-->
        </connections>
        <search-types>
            <add name="default" analyzer="keyword" />
        </search-types>
        <entities>
            <add name="EntityName" />
        </entities> 
        <relationships/>
    </add>
</processes>

How to compile

Hi,
does this project compile? On VS2015 U2 I'm unable to get quite a few dependencies via NuGet as it's not able to find them online:

• Cfg-NET (required 0.7.4 - found 0.6.7)
• Cfg-NET.Environment (req. 0.0.2 - found 0.0.1)
• Cfg-NET.Reader (0.1.1 vs 0.1.0)
• Cfg-NET.Shorthand (0.0.6 vs 0.0.4)
• Common.Logging (3.3.1 vs 3.3.0 / Pre 3.3.2-Alpha3)
• Quartz (2.3.3 vs 2.3.2)
• System.Data.SQLite.Core (1.0.101 vs not found at all)

Any suggestion / clarification?

Autofac not resolving IEntityDeleteHandler

When trying to set deletes to true for an entity, I get the following error:

Unhandled Exception: Autofac.Core.DependencyResolutionException: An exception was thrown while activating ?:Transformalize.Contracts.IProcessController. ---> Autofac.Core.Registration.ComponentNotRegisteredException: The requested service 'horizon-replicationvw_cds_rdw_mbr_acct_grp_sys (Transformalize.Contracts.IEntityDeleteHandler)' has not been registered. To avoid this exception, either register a component to provide the service, check for service registration using IsRegistered(), or use the ResolveOptional() method to resolve an optional dependency.
   at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters)
   at Autofac.ResolutionExtensions.ResolveNamed[TService](IComponentContext context, String serviceName, IEnumerable`1 parameters)
   at Transformalize.Ioc.Autofac.Modules.ProcessControlModule.<Load>b__3_0(IComponentContext ctx)
   at Autofac.Builder.RegistrationBuilder.<>c__DisplayClass0_0`1.<ForDelegate>b__0(IComponentContext c, IEnumerable`1 p)
   at Autofac.Core.Activators.Delegate.DelegateActivator.ActivateInstance(IComponentContext context, IEnumerable`1 parameters)
   at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters, Object& decoratorTarget)
   --- End of inner exception stack trace ---
   at Autofac.Core.Resolving.InstanceLookup.Activate(IEnumerable`1 parameters, Object& decoratorTarget)
   at Autofac.Core.Resolving.InstanceLookup.Execute()
   at Autofac.Core.Resolving.ResolveOperation.GetOrCreateInstance(ISharingLifetimeScope currentOperationScope, IComponentRegistration registration, IEnumerable`1 parameters)
   at Autofac.Core.Resolving.ResolveOperation.Execute(IComponentRegistration registration, IEnumerable`1 parameters)
   at Autofac.ResolutionExtensions.TryResolveService(IComponentContext context, Service service, IEnumerable`1 parameters, Object& instance)
   at Autofac.ResolutionExtensions.ResolveService(IComponentContext context, Service service, IEnumerable`1 parameters)
   at Autofac.ResolutionExtensions.Resolve[TService](IComponentContext context, IEnumerable`1 parameters)
   at Transformalize.Command.NowScheduler.Start()
   at Transformalize.Command.Program.Main(String[] args)

Can't find TFL.exe

Hi,

Great project, but where is TFL.exe located?
Also, building Pipeline.Command crashes the CSC due to IOException.

Thanks.

How to detect Header DataTypes during File Parsing

in my case, the source CSV data file and header content/types change often -- basically unknown, while parsing or before parsing I need to first Discover the Data Types of the data in the cols.

I just need to load & save the data into a Sql table based on whatever data & DataTypes there are in the CSV file. Can I auto load the CSV data and datatypes into the DB automatically with ETL box. I saw your expando example but was a little lost how to achieve a high level task

For e.g high level algorithm, how would I achieve this in your lib.

// Scan first 9 rows to discover the datatypes in the CSV file cols
var etlPrasedFile = DaleNewMan.Trans.Open/parseCSV(someUnknownData.csv).ScanRows(9)
foreach (var col in etlPrasedFile )
new dataTable.addColumn(etlParsedFile.GetNextHeaderCol)
CreateNewSqlServerTable
LoadDatatoSqlTable??

thanks