seratch / bigquery4s

A handy Scala wrapper of Google BigQuery API 's Java Client Library.

👋 I am a maintainer of all the @SlackAPI open-source projects! Also, I've been sharing small pieces of Slack app examples:

• ChatGPT Integration App
• Cloudflare AI Integration App
• Notion Integration App
• DeepL API Integration App
• New Relic Integration App

... and so on

bigquery4s's Issues

Connecting to BigQuery from within google cloud machine

Hey,

Trying to connect to BigQuery when running my app on google cloud VM (I can't use the client_json.secret).

the function fromServiceAccount is in the code but doesnt functioning.

Can that be fixed?

Connection to BQ from inside DataPRoc

Hi,
I'm trying to run my scala code from a dataproc cluster and get this error:
Exception in thread "main" java.io.FileNotFoundException: /root/.bigquery/client_secret.json (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.<init>(FileInputStream.java:138) at java.io.FileInputStream.<init>(FileInputStream.java:93) at bigquery4s.BigQuery$.fromClientSecretJson(BigQuery.scala:137) at bigquery4s.BigQuery$.apply(BigQuery.scala:125)
So that means its looking for fromClientSecretJson instead of fromServiceAccount, can someone suggest how to redirect it to the latter?
Thanks,

Default useLegacy to false

The Google BigQuery UI no longer uses legacy by default.

[Discussion] BigQuery API

I would like to know if you are willing to switch to use com.google.cloud.google-cloud-bigquery api instead of the current com.google.apis.google-api-services-bigquery

There is interesting functions available there using objects like FieldValue containing better typing than TableValue.

It looks like these APIs are mostly for rest apis versus the first mentioned one is a java wrapper for those, we might be able to have something closer to scalikejdbc more easily.

I have a few examples in my code that I can show that I have issues reproducing with TableCell:

protected[this] def string(columnName: String)(implicit row: FieldValueList): String = row.get(columnName).getStringValue

protected[this] def int(columnName: String)(implicit row: FieldValueList): Int = row.get(columnName).getLongValue.toInt

protected[this] def double(columnName: String)(implicit row: FieldValueList): Double = row.get(columnName).getDoubleValue

I wanted to add these to WrappedTableCell and would be a lot easier if the underlying was a FieldValue

I am willing to make a relatively big refactoring if you want to go with it @seratch I could make bigquery4s.BigQuery extend the com.google.cloud.bigquery.BigQuery (or be some kind of wrapper to it like you did) and add the credential utilities to it. We won't need pagination anymore I think. Let me know what you think!

Also another example of some of my code using FieldValues:

  def apply(implicit row: FieldValueList): Member = {
    new Member(
      stringOpt("id"),
      string("name"),
      double("x_local"),
      double("y_local"),
      timestamp("time_created")
    )
  }

 def fetch(): Seq[Member] = bqs"select * from $datasetName.$tableName".map(Member(_))

Issue on auth

Hi,

I've done the readme but it doesn't work.

WHEN I TRY THE AUTH :
val bq = BigQuery() // expects $HOME/.bigquery/client_secret.json

I've got this stacktrace :

java.lang.IllegalArgumentException
	at com.google.api.client.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java:111)
	at com.google.api.client.util.Preconditions.checkArgument(Preconditions.java:37)
	at com.google.api.client.googleapis.auth.oauth2.GoogleClientSecrets.getDetails(GoogleClientSecrets.java:82)
	at com.google.api.client.googleapis.auth.oauth2.GoogleAuthorizationCodeFlow$Builder.<init>(GoogleAuthorizationCodeFlow.java:195)
	at bigquery4s.BigQuery$.fromClientSecretJson(BigQuery.scala:120)
	at bigquery4s.BigQuery$.apply(BigQuery.scala:102)

It seem's that we never setInstalled or setWeb because web and installed are null.

But in debug mode, client_id / ... are well initialized in clientSecrets.
Can you give me some support ?

Best regards,