prologuetech / laravel-big

Google BigQuery for Laravel

Google BigQuery for Laravel

This package aims to wrap laravel functionality around Google's BigQuery.

Install

Via Composer

$ composer require prologuetech/big

Setup

Publish our config file into your application:

php artisan vendor:publish --provider="Prologuetech\Big\BigServiceProvider"

You should have a config/prologue-big.php file to configure defaults.

Laravel 5.4.x

Older versions of Laravel require you to add our big service provider to your application providers array in config/app.php:

Prologuetech\Big\BigServiceProvider::class,

You now have access to a familiar laravel experience, enjoy!

Google Authentication

The Google SDK supports Application Default Credentials (ADC) and thus this package does as well. You may leave your auth_file field inside of your config file null to use ADC. Credentials fetcher is not currently supported but may be added in the future.

For more information see the adc docs.

How to use

Configuration

By default we use the following global config options with BigQuery.

$this->options = [
    'useLegacySql' => false,
    'useQueryCache' => false,
];

Tables

When creating tables in BQ we automatically flip a Eloquent model schema for you. Let's cover an example of archiving data from our events table into BQ using laravel's chunk method.

$datasetId = 'test';
$tableId = 'events';

// Create our BQ helper
$big = new Big();

// Create table, we will pass in a mocked model to mutate into BQ schema
// Note: create table will only make a new table if it does not exist

/** @var Google\Cloud\BigQuery\Table $table */
$table = $big->createFromModel($datasetId, $tableId, new Event());

// Let's stream our events into BQ in large chunks
// Note: notArchived() is a simple scope, use whatever scopes you have on your model
Event::notArchived()->chunk(1000, function ($events) use ($big, $table) {
    // Prepare our rows
    $rows = $big->prepareData($events);

    // Stream into BQ, you may also pass in any options with a 3rd param.
    // Note: By default we use: 'ignoreUnknownValues' => true
    $big->insert($table, $rows);

    // Get our current id's
    /** @var Illuminate\Support\Collection $events */
    $ids = $events->pluck('id')->toArray();

    // Update these event's as processed
    Event::whereIn('id', $ids)->update([
        'system_processed' => 1
    ]);
});

That's it! You now have a replica of your events table in BigQuery, enjoy!

Queries

Instantiating Big will automatically setup a Google ServiceBuilder and give us direct access to BigQuery through our internals via $big->query. However there are many helpers built into Big that make interacting with BigQuery a piece of cake (or a tasty carrot if you're into that kind of thing).

For example when running a query on BigQuery we must use the reload method in a loop to poll results. Big comes with a useful method run so all you need to do is this:

$query = 'SELECT count(id) FROM test.events';

$big = new Big();
$results = $big->run($query);

When using run we automatically poll BigQuery and return all results as a laravel collection object for you so you can enjoy your results as a refreshing cup of Laravel.

Change log

Please see CHANGELOG for more information on what has changed recently.

License

The MIT License (MIT). Please see License File for more information.

laravel-big's Issues

Cannot run drop table command

Hi,

Whenever I do something like: $big->run("DROP TABLE IF EXISTS myproject_id.my_dataset_id.my_table_name"); it will give an error message like this: ErrorException: Undefined index: schema in /var/www/html/laravel/vendor/google/cloud/BigQuery/src/QueryResults.php:162

All I want is to be able to drop a table if it exists or using 'CREATE OR REPLACE TABLE' DDL statement for the "createFromModel" method. Any explaination / guide is much appreciated, thank you!

Add support for Application Default Credentials (ADC)

Currently the library expects an auth_file to be specified. Instead it could use the Application Default Credentials (ADC) mechanics other libraries and applications are using.

See more details here: https://cloud.google.com/docs/authentication/production

Unclear documentation

Hi documentation should be updated .. // Create our BQ helper
$big = new Big(); instantiating this into the controller is not allowed. do we need to add use on top?

Handling of insertId's is causing false-positives

The following line of code is checking in_array() to check for existence of an ID. This is actually checking the values rather than the keys for the value. Would prefer an array_key_exists use here which would resolve the issue and allow a record with non-ID to work as intended.

laravel-big/src/Big.php

Line 172 in ee56839

if (in_array('id', $item)) {

Tag latest release

Is it possible to tag your latest changes that update the Google Client library version? We need access to runJob on the table and currently the dependancies don't pull it through. I'm grabbing dev-master for now but I'd prefer to grab a tag rather than the master branch.

Issue in loading config values

In vendor/prologuetech/big/src/Big.php , env values are not getting loaded .. !

$googleService = new ServiceBuilder([
				'keyFilePath' => config('prologue-big.big.auth_file'),
				'projectId'   => config('prologue-big.big.project_id'),
			]);

resulting in this error

Google \ Cloud \ Core \ Exception \ GoogleException
No project ID was provided, and we were unable to detect a default project ID.

It was able to execute properly after we copied the '/vendor/prologuetech/big/config/prologue-big.php' file to '/config/prologue-big.php'

is this an issue or any other mistakes in installing/loading the package causing this ? Please guide