yatish27 / salesforce_bulk_api Goto Github PK

https://github.com/yatish27/salesforce_bulk_api/blob/master/lib/salesforce_bulk_api/job.rb#L182-L185
comes from this PR
@batch_ids.delete(batch_id) does not work very well with the each, since Array#each use indices to iterate.

I'm wondering since batches_ready is true, why not remove the break if @batch_ids.empty? and @batch_ids.delete(batch_id) logic.

I'm trying to use the gem, but I can't because it is inserting NIL values where I don't want them.

The basic problem is this:

data = [ { id: '1', Field_1__c: nil}, { id: '2', Field_2__c: nil] } ]

bulk_api.update(:contact, data, true, true)

What I expect is for record 1 to have Field_1__c set to nil, and record 2 to have Field_2__c set to nil. What I get is both records having both fields blanked out.

@yatish27 I see the last release being made on Jan 10, 2015 however master has some new commits with bug fixes. Are you going to make a new release?

Rather than using custom retry logic:
https://github.com/yatish27/salesforce_bulk_api/blob/master/lib/salesforce_bulk_api/connection.rb#L46

Link to gem: https://github.com/kamui/retriable

The example for queries only shows a single query being submitted. How do you run multiple queries in one bulk call? When I follow the example in the README it runs immediately and synchronously returns the values for only the one call.

bulk = SalesforceBulkApi::Api.new(restforce_client)
response = bulk.query('Contact', "SELECT Id, AccountId FROM Contact WHERE Email = '[email protected]'")

response['batches'][0]['response'] returns:

[
    [0] {
         "xsi:type" => "sObject",
             "type" => [
            [0] "Contact"
        ],
               "Id" => [
            [0] "00336000004AVvHAAW",
            [1] "00336000004AVvHAAW"
        ],
        "AccountId" => [
            [0] "001360000027qGJAAY"
        ]
    }
]

What I would like to do is something like the example below, but it's unclear how:

array = []
array << "SELECT Id, AccountId FROM Contact WHERE Email = '[email protected]'"
array << "SELECT Id, AccountId FROM Contact WHERE Email = '[email protected]'"
bulk.query('Contact', array)

Thanks!

*1) With the following in my gemfile:
gem "databasedotcom", "> 1.3.2"
gem "salesforce_bulk_api", "> 0.0.4

I get an error: Bundler could not find compatible versions for gem "multipart-post":
In Gemfile:
salesforce_bulk_api (> 0.0.4) ruby depends on
multipart-post (> 1.1.0) ruby

databasedotcom (~> 1.3.2) ruby depends on
  multipart-post (1.2.0)

*2) With the following in my gemfile:
gem "databasedotcom", "~> 1.3.2"
gem "salesforce_bulk_api"

I get this error:
Bundler could not find compatible versions for gem "addressable":
In Gemfile:
salesforce_bulk_api (>= 0) ruby depends on
addressable (~> 2.2.4) ruby

launchy (>= 0) ruby depends on
  addressable (2.3.2)

The require hits an error because of the missing gem dependency in the gemspec.

From the examples in the homepage README, it looks like any bulk query you submit must fit into memory as a Ruby array of hashes. Would it be possible to stream the results to a file for queries that pull back a lot of data?

Hi,

when I user Restforce client with username and password + client_id and client_secret the @session_id is is nil. Instead if i use a token from Salesforce OAuth2 authentication with no username and password the @session_id is correctly set.

Is there any workaround for this?

I set the version to '32.0' in my own branch and it seems to work for the most part.

https://github.com/gostrc/salesforce_bulk_api/tree/bump-api

I did this because there are some fields that require this newer version bump.

e.g. I was getting this error before bumping the api:

ERROR at Row:1:Column:427 No such column 'CleanStatus' on entity 'Account'. If you are attempting to use a custom field, be sure to append the '__c' after the custom field name. Please reference your WSDL or the describe call for the appropriate names.

I didn't create a pull request in case you wanted to do some extra testing around the branch.
I noticed that null values came up with the following value:

"field"=>[{"xsi:nil"=>"true"}],

Leads to wasted hours investigating the below:

I'm able to push records to SF using the Databasedotcom gem itself, but I can't get bulk to work. Something like this:

records = [{"Ticket__c" => "100", "Subject__c" => "Foo"}, ...]
bulk.upsert("Redmine_Tickets__c", records, "Ticket__c")

prints out this, and no records are created or updated in SF:

na4
na4.salesforce.com
<?xml version="1.0" encoding="UTF-8"?><jobInfo xmlns="http://www.force.com/2009/06/asyncapi/dataload">
 <id>750600000011S5oAAE</id>
 <operation>upsert</operation>
 <object>Redmine_Tickets__c</object>
 <createdById>00560000001y43UAAQ</createdById>
 <createdDate>2013-10-03T16:07:34.000Z</createdDate>
 <systemModstamp>2013-10-03T16:07:34.000Z</systemModstamp>
 <state>Open</state>
 <externalIdFieldName>Ticket__c</externalIdFieldName>
 <concurrencyMode>Parallel</concurrencyMode>
 <contentType>XML</contentType>
 <numberBatchesQueued>0</numberBatchesQueued>
 <numberBatchesInProgress>0</numberBatchesInProgress>
 <numberBatchesCompleted>0</numberBatchesCompleted>
 <numberBatchesFailed>0</numberBatchesFailed>
 <numberBatchesTotal>0</numberBatchesTotal>
 <numberRecordsProcessed>0</numberRecordsProcessed>
 <numberRetries>0</numberRetries>
 <apiVersion>23.0</apiVersion>
 <numberRecordsFailed>0</numberRecordsFailed>
 <totalProcessingTime>0</totalProcessingTime>
 <apiActiveProcessingTime>0</apiActiveProcessingTime>
 <apexProcessingTime>0</apexProcessingTime>
</jobInfo>
75160000001YF90AAG

Am I doing something wrong?

In the job file:
https://github.com/yatish27/salesforce_bulk_api/blob/master/lib/salesforce_bulk_api/job.rb#L205

You are using xml to download batches of large data sets.
This is unideal since salesforce allows you to specify csv [0] as the type.

This would have the following pros:

1. Simplifying the parsing code
2. Smaller download size
3. Less cpu time spent parsing
4. Less memory used trying to hold the xml

[0] - http://www.salesforce.com/us/developer/docs/api_asynch/
under Bulk Query -> Bulk Query Details

It appears as though that the current job.rb file errors out on occasion, claming:

salesforce_bulk_api-0.0.10/lib/salesforce_bulk_api/job.rb:35 undefined method `[]' for nil:NilClass

I experience this issue pretty consistently around 1AM PDT. It's unclear if this is a limitation of the API or there's some sort of issue with the salesforce_bulk_api gem itself.

Before any action can be taken, I think it might be useful to first get some more information about the status of each call by providing a success, error, and complete callback so that clients of salesforce_bulk_api can inspect the status of each job, respectively.

I noticed that although updates are still being merged to master, there hasn't been a new gem release since March 2015:

https://rubygems.org/gems/salesforce_bulk_api/versions/0.0.12

This was brought up in #69 but that ticket was closed even though no release was made.

I tried sending a request to salesforce using salesforce_bulk_api gem. I was successful in creating the connection and I was also able to query all the domains in salesforce. But I tried creating a ResourceRequest through the gem. I always get nil as the result. I got the following output when i executed the create statement. But no records where created :(. As far as i know, there are no mistakes with the data and i am very sure that it did not fail due to domain validations. Please help.

750R00000009aOMIAY insert pse__Resource_Request__c 00550000001DA2zAAG 2012-05-17T05:49:38.000Z 2012-05-17T05:49:38.000Z Open Parallel XML 0 0 0 0 0 0 0 23.0 0 0 0 0

I was having problems using Ruby 1.9.3 connecting to SalesForce using this gem. It looks like Net::HTTP.new takes different sets of arguments if a proxy is needed so I modified the https method in the SalesforceBulkApi::Connection class to the following:

def https(host)
      req = Net::HTTP.new(host, 443, @proxy_url, @proxy_port, @proxy_user, @proxy_pass)
      req.use_ssl = true
      req.verify_mode = OpenSSL::SSL::VERIFY_NONE
      req
 end

This seemed to fix the problem for me but I wasn't sure how you wanted this worked into the Connection class. I was thinking of setting proxy variables in the connection from the Restforce/Database.com client if the proxy_uri option is set. If you think this is a decent approach, I'll work it up and send in a PR for it.

When it sends a batch request I get this error when I go into the Bulk Data load Job detail within salesforce.com and click on view request.

    This page contains the following errors:

    error on line 1 at column 1: Document is empty
   Below is a rendering of the page up to the first error.

The SalesforceBulkApi module is a rename from SalesforceBulk but the code references SalesforceBulk . This causes the gem not to function at all.

Thanks

Hello everyone,

I currently am experiencing an error while upserting objects in SF from our backend.
Before sending the data to the Salesforce API (version 41), I log the xml that I send and it is well formatted. The job then fails on Salesforce side with the error: "XML Parsing Error: mismatched tag". Indeed, when I look at the XML processed by salesforce, I can see this:
<Client__c>a050X00000zLdPsQAK</Clieent__c> in the middle of the XML. I do not really understand why.

Did anyone ever face this problem and find a trick to solve it?

Here is the configuration that I use:

• ruby 2.6.0
• rails 5.2.2
• gem 'restforce', '~> 3.1.0'
• gem 'salesforce_bulk_api' , '~> 0.12.0'

Thank you very much

https://github.com/yatish27/salesforce_bulk_api/blob/master/lib/salesforce_bulk_api/connection.rb#L78

AFAIK "blank?" is an ActiveSupport thing, which doesn't seem to be a dependency of this gem at all. Attempting to authenticate with the SFDC API returns the following trace:

/Users/aashay/dev/labs/salesforce/ruby/vendor/bundle/gems/salesforce_bulk_api-0.0.9/lib/salesforce_bulk_api/connection.rb:78:in `parse_instance': undefined method `blank?' for "cs10":String (NoMethodError)
    from /Users/aashay/dev/labs/salesforce/ruby/vendor/bundle/gems/salesforce_bulk_api-0.0.9/lib/salesforce_bulk_api/connection.rb:35:in `login'
    from /Users/aashay/dev/labs/salesforce/ruby/vendor/bundle/gems/salesforce_bulk_api-0.0.9/lib/salesforce_bulk_api/connection.rb:20:in `initialize'
    from /Users/aashay/.rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/salesforce_bulk_api-0.0.9/lib/salesforce_bulk_api.rb:18:in `new'
    from /Users/aashay/.rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/salesforce_bulk_api-0.0.9/lib/salesforce_bulk_api.rb:18:in `initialize'
    from ./bulk.rb:19:in `new'
    from ./bulk.rb:19:in `<main>'

this is the response when a bulk job is submitted:

=> {"xmlns"=>"http://www.force.com/2009/06/asyncapi/dataload", "exceptionCode"=>["InvalidSessionId"], "exceptionMessage"=>["Invalid session id"]}

Restforce methods work, so login procedure may be ok.

force_client = Restforce.new :oauth_token => token,
        :refresh_token => refresh__token,
        :instance_url  => instance_url,
        :api_version   => "31.0",
        :client_id     => Rails.application.config.salesforce_app_id,
        :client_secret => Rails.application.config.salesforce_app_secret

    #instantiate salesforce bulk api gem
    salesforce_bulk = SalesforceBulkApi::Api.new(force_client)
    # here it fails!
    result = salesforce_bulk.create("Houses__c", houses_collection)

I suspect that this is due to salesforce sending a response that this gem doesn't expect.

/u/apps/pd_bizops_integration/shared/bundle/ruby/2.2.0/gems/activerecord-3.2.17/lib/active_record/associations/has_many_association.rb:80: warning: circular argument reference - reflection
/u/apps/pd_bizops_integration/shared/bundle/ruby/2.2.0/gems/railties-3.2.17/lib/rails/commands/runner.rb:54:in `eval': undefined method `[]' for nil:NilClass (NoMethodError)
        from /u/apps/pd_bizops_integration/shared/bundle/ruby/2.2.0/gems/salesforce_bulk_api-0.0.12/lib/salesforce_bulk_api/job.rb:175:in `each'
        from /u/apps/pd_bizops_integration/shared/bundle/ruby/2.2.0/gems/salesforce_bulk_api-0.0.12/lib/salesforce_bulk_api/job.rb:175:in `block in get_job_result'
        from /usr/local/rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/timeout.rb:89:in `block in timeout'
        from /usr/local/rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/timeout.rb:99:in `call'
        from /usr/local/rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/timeout.rb:99:in `timeout'
        from /u/apps/pd_bizops_integration/shared/bundle/ruby/2.2.0/gems/salesforce_bulk_api-0.0.12/lib/salesforce_bulk_api/job.rb:172:in `get_job_result'
        from /u/apps/pd_bizops_integration/shared/bundle/ruby/2.2.0/gems/salesforce_bulk_api-0.0.12/lib/salesforce_bulk_api.rb:79:in `do_operation'
        from /u/apps/pd_bizops_integration/shared/bundle/ruby/2.2.0/gems/salesforce_bulk_api-0.0.12/lib/salesforce_bulk_api.rb:29:in `update'
        from /u/apps/pd_bizops_integration/releases/20150821062234/lib/sfdc.rb:49:in `do_bulk!'
        from /u/apps/pd_bizops_integration/releases/20150821062234/lib/sfdc.rb:26:in `update_object!'
        from /u/apps/pd_bizops_integration/releases/20150821062234/lib/sfdc.rb:18:in `update_transaction_object!'
        from /u/apps/pd_bizops_integration/releases/20150821062234/app/models/tasks/sfdc_transaction_updater.rb:49:in `update'
        from /u/apps/pd_bizops_integration/releases/20150821062234/app/models/tasks/sfdc_transaction_updater.rb:6:in `block in run'
        from /u/apps/pd_bizops_integration/releases/20150821062234/lib/cron_utils.rb:26:in `call'
        from /u/apps/pd_bizops_integration/releases/20150821062234/lib/cron_utils.rb:26:in `block (2 levels) in run_with_lock_and_benchmark'
        from /u/apps/pd_bizops_integration/releases/20150821062234/lib/cron_utils.rb:16:in `block in filelock'
        from /u/apps/pd_bizops_integration/releases/20150821062234/lib/cron_utils.rb:14:in `open'
        from /u/apps/pd_bizops_integration/releases/20150821062234/lib/cron_utils.rb:14:in `filelock'
        from /u/apps/pd_bizops_integration/releases/20150821062234/lib/cron_utils.rb:25:in `block in run_with_lock_and_benchmark'
        from /usr/local/rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/benchmark.rb:303:in `realtime'
        from /u/apps/pd_bizops_integration/releases/20150821062234/lib/cron_utils.rb:24:in `run_with_lock_and_benchmark'
        from /u/apps/pd_bizops_integration/releases/20150821062234/app/models/tasks/sfdc_transaction_updater.rb:5:in `run'
        from (eval):1:in `<top (required)>'
        from /u/apps/pd_bizops_integration/shared/bundle/ruby/2.2.0/gems/railties-3.2.17/lib/rails/commands/runner.rb:54:in `eval'
        from /u/apps/pd_bizops_integration/shared/bundle/ruby/2.2.0/gems/railties-3.2.17/lib/rails/commands/runner.rb:54:in `<top (required)>'
        from /u/apps/pd_bizops_integration/shared/bundle/ruby/2.2.0/gems/railties-3.2.17/lib/rails/commands.rb:64:in `require'
        from /u/apps/pd_bizops_integration/shared/bundle/ruby/2.2.0/gems/railties-3.2.17/lib/rails/commands.rb:64:in `<top (required)>'
        from script/rails:12:in `require'
        from script/rails:12:in `<main>'

Edit: I added logging functionality around the issue at:
https://github.com/PagerDuty/salesforce_bulk_api/commit/401239a46641c6e7fda75e4d28e4b1701931c422

Travis should have a set of encrypted credentials to a developer org containing no modifications to salesforce's vanilla behavior.

This will make sure that tests pass on the same environment each run.

What is the license for using this gem? Can you add a license file to this project??

Can you please state under what license this gem is distributed?

👋

I just noticed that my bulk client is using API version 32. I took a look on this repo and I see that the API version was updated to 46, but I don't see those changes in the files that gem installed when I gem install salesforce_bulk_api

I tried uninstalling and reinstalling the gem:

❯ gem uninstall salesforce_bulk_api
Successfully uninstalled salesforce_bulk_api-0.0.12

❯ gem install salesforce_bulk_api
Fetching salesforce_bulk_api-0.0.12.gem
Successfully installed salesforce_bulk_api-0.0.12
Parsing documentation for salesforce_bulk_api-0.0.12
Installing ri documentation for salesforce_bulk_api-0.0.12
Done installing documentation for salesforce_bulk_api after 0 seconds
1 gem installed

❯ gem which salesforce_bulk_api
/Users/robgoodberry/.gem/ruby/2.6.5/gems/salesforce_bulk_api-0.0.12/lib/salesforce_bulk_api.rb

Then if I vi /Users/robgoodberry/.gem/ruby/2.6.5/gems/salesforce_bulk_api-0.0.12/lib/salesforce_bulk_api.rb I'm still seeing the older code with the @@ class variables and API version 32:

Bundler.require()
require 'salesforce_bulk_api/version'
require 'net/https'
require 'xmlsimple'
require 'csv'
require 'salesforce_bulk_api/concerns/throttling'
require 'salesforce_bulk_api/job'
require 'salesforce_bulk_api/connection'

module SalesforceBulkApi

  class Api
    attr_reader :connection

    @@SALESFORCE_API_VERSION = '32.0'

Am I messing something up here or has the most recent master just not been released?

Thanks for any help!

Hi Yatish,
What should be done with Results after performing this query :

res = salesforce.query("Account", "select id, name, createddate from Account")

. I am currently working on an application where I am saving the information retrieved from Salesforce Bulk API to a postgresql database. The result just shows the number of records that are being fetched ( which are exactly the number of records we have in Salesforce against that object ) but does not display the information in the form of Array or Hash.

I tried to browse through the code to figure this thing out myself but was not able to do it.

Thanks

Sadan Masroor.

We have a tailored sinatra app that is able to talk to salesforce using Databasedotcom gem. We also have some patching to faraday to tailor URI connection. The faraday gem we installed is version 0.7.6 and when adding in salesforce_bulk_api into Gemfile and then running bundle install, it is throwing back:

Bundler could not find compatible versions for gem "faraday":
  In Gemfile:
    salesforce_bulk_api (>= 0) ruby depends on
      faraday (~> 0.6.1) ruby

    faraday (0.7.6)

I guess I could downgrade faraday and give my app a try, but when checking rubygems site https://rubygems.org/gems/faraday/versions, even the 0.7.6 version of faraday is over 1 year old, 0.6.1 is back dated to April 2011. Is it possible to have salesforce_bulk_api use the latest faraday?

https://rubygems.org/gems/salesforce_bulk_api/versions/0.0.12

The tests currently rely on data being in the sandbox setup a specific way.

See https://github.com/yatish27/salesforce_bulk_api/blob/master/example_auth_credentials.yml
there is an account id in there

There are other tests that depend on data being in the org.

It would be nice for the tests to actually produce that data before using it.

We could use another gem version bump. Version 0.0.9 (current version on rubygems) does not include some important changes, such as getting rid of Rails calls:

8132635#diff-c54fc188605097c5989b3994bd6b5c75L38

Thank you in advance!

The README says this is a rewritten version of salesforce_bulk gem, but it doesn't say what was improved. Please provide an explanation of what's different between salesforce_bulk_api and salesforce_bulk.

Hello there, and thank you for maintaining this Salesforce Bulk API library! It has definitely helped me develop against Salesforce with its easy to use interface, so thanks for that :)

While writing testing code for my application that currently uses your utility, I noticed a significant performance decrease as I began integrating calls to the bulk api service. After a bit of investigation, I noticed that the call to sleep here, was the sole reason of our slowdown, even after using VCR to mock out our web requests.

It's not clear to me why this sleep statement is necessary, other than to give a small buffer of time for the next batch in sequence to change status; it doesn't guarantee whether or not the next batch will be finished.

If the library needs to ensure the processing of each batch, perhaps it could instead provide some sort of blocking operation on each batch?

For example:

@batch_ids.each do |batch_id|

  # block execution until batch is ready 
  while true 
    batch_state = self.check_batch_status(batch_id)
    if batch_state['state'][0] != "Queued" && batch_state['state'][0] != "InProgress"
      state << (batch_state)
      @batch_ids.delete(batch_id)

      # once the batch is added to `state`, break out
      # to the next batch in iteration
      break
    end
  end

end

This removes the need for a call to sleep, which bleeds into the implementation details of into client programs and can lead to race conditions in some cases.

I would be interested to learn more about your thoughts on the issue!

Thanks,
-Kelly.