choplin / embulk-output-parquet Goto Github PK

It will be failed when use url format such as s3a://AWS_ACCESS_KEY_ID@AWS_SECRET_ACCESS_KEY:bucket/keys
and the AWS_SECRET_ACCESS_KEY include the character "/".
Is it possible to add params such like S3 file output plugin with parameters "access_key_id","secret_access_key"?
or we wait for hadoop 2.8's launch?

I try to import data and export to hdfs by using embulk-output-parquet library.
But when I run embulk by below settings, "Error: java.lang.RuntimeException: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS] " exception occurred.

in:
  type: mysql
  host: 192.168.65.2
  port: 3306
  user: user
  password: "pwd"
  database: testdb
  table: testtable
  options: { useLegacyDatetimeCode: true }
out:
  type: parquet
  path_prefix: hdfs://{url}/tmp/embulk
  extra_configurations:
    hadoop.security.authentication: 'KERBEROS'
  config_files:
    - {HADOOP_CONFIG_HOME}/hdfs-site.xml
    - {HADOOP_CONFIG_HOME}/core-site.xml

kerberos setting is ok all, and I inserted code for checking 'hadoop.security.authentication' is really set by 'SIMPLE'. But 'KEBEROS' value was overwrited correctly. I guess even though set by any string, it is overwrited by 'SIMPLE' somewhere.

os : hadoop
embulk : v0.97
haddop : hadoop-2.6.0-cdh5.14.2

related issue I found : civitaspo/embulk-output-hdfs#17

Looking at the docs, it seems like there is no topic about GCS.
Is it not compatible with GCS?

If I run:

embulk run --log-level error my_config.yml

I still get info and warning messages printing. I expect to only see error messages.

Thanks.

Hi, I'm trying to Parquet into S3 via your plugin. It looks like the values in the data are somehow being encoded in a way that seems to make the Parquet file unusable with AWS Glue / Athena (Presto). When I use the stdout plugin, the data looks correct.

When I try to open the Parquet file with Sublime (which renders it like JSON), I see data that looks like the following:

{"event_id":"MjI0Yzg4ODc0ZjEzYWJjM2Q4OGI3M2NiYWE5NTcwODQ=","event_timestamp":"MjAxOC0wOS0wNCAxNTozMjoxOC4wMDEwMDAgKzAwMDA="}
{"event_id":"ZjQzNmQxMmNkNmFlNGM5ZmJkMTc3OTExOTJmZGY2MmY=","event_timestamp":"MjAxOC0wOS0wNCAxNTozMjoxNi4xNzIwMDAgKzAwMDA="}
…

Here's the relevant parts of the Embulk file:

in:
  type: command
  command: lib/splunk export …
      
  parser:
    type: jsonl
    columns:
      - {name: "event_id", type: string}
      - {name: "event_timestamp", type: timestamp, format: "%Q"}
      …
out:
  type: parquet
  path_prefix: s3a://…
  extra_configurations:
    fs.s3a.access.key: …
    fs.s3a.secret.key: …