Coder Social home page Coder Social logo

salesforce_bulk_api's People

Contributors

caioeps avatar edwinwills avatar gregormelhorn avatar idiogenes avatar intractablequery avatar jaurand avatar jthibeaux avatar kellydunn avatar neanias avatar ontarionick avatar patcheng avatar sherrychaizp avatar thomasdziedzic-pd avatar yatish27 avatar yatish27coupa avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

salesforce_bulk_api's Issues

undefined method `blank?' for String

https://github.com/yatish27/salesforce_bulk_api/blob/master/lib/salesforce_bulk_api/connection.rb#L78

AFAIK "blank?" is an ActiveSupport thing, which doesn't seem to be a dependency of this gem at all. Attempting to authenticate with the SFDC API returns the following trace:

/Users/aashay/dev/labs/salesforce/ruby/vendor/bundle/gems/salesforce_bulk_api-0.0.9/lib/salesforce_bulk_api/connection.rb:78:in `parse_instance': undefined method `blank?' for "cs10":String (NoMethodError)
    from /Users/aashay/dev/labs/salesforce/ruby/vendor/bundle/gems/salesforce_bulk_api-0.0.9/lib/salesforce_bulk_api/connection.rb:35:in `login'
    from /Users/aashay/dev/labs/salesforce/ruby/vendor/bundle/gems/salesforce_bulk_api-0.0.9/lib/salesforce_bulk_api/connection.rb:20:in `initialize'
    from /Users/aashay/.rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/salesforce_bulk_api-0.0.9/lib/salesforce_bulk_api.rb:18:in `new'
    from /Users/aashay/.rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/salesforce_bulk_api-0.0.9/lib/salesforce_bulk_api.rb:18:in `initialize'
    from ./bulk.rb:19:in `new'
    from ./bulk.rb:19:in `<main>'

Using restforce Oauth login, got Invalid Session Id

this is the response when a bulk job is submitted:

=> {"xmlns"=>"http://www.force.com/2009/06/asyncapi/dataload", "exceptionCode"=>["InvalidSessionId"], "exceptionMessage"=>["Invalid session id"]}

Restforce methods work, so login procedure may be ok.

force_client = Restforce.new :oauth_token => token,
        :refresh_token => refresh__token,
        :instance_url  => instance_url,
        :api_version   => "31.0",
        :client_id     => Rails.application.config.salesforce_app_id,
        :client_secret => Rails.application.config.salesforce_app_secret

    #instantiate salesforce bulk api gem
    salesforce_bulk = SalesforceBulkApi::Api.new(force_client)
    # here it fails!
    result = salesforce_bulk.create("Houses__c", houses_collection)

What should be done after select query

Hi Yatish,
What should be done with Results after performing this query :

res = salesforce.query("Account", "select id, name, createddate from Account")

. I am currently working on an application where I am saving the information retrieved from Salesforce Bulk API to a postgresql database. The result just shows the number of records that are being fetched ( which are exactly the number of records we have in Salesforce against that object ) but does not display the information in the form of Array or Hash.

I tried to browse through the code to figure this thing out myself but was not able to do it.

Thanks

Sadan Masroor.

Gem dependency issue when trying to install with databasedotcom gem

*1) With the following in my gemfile:
gem "databasedotcom", "> 1.3.2"
gem "salesforce_bulk_api", "
> 0.0.4

I get an error: Bundler could not find compatible versions for gem "multipart-post":
In Gemfile:
salesforce_bulk_api (> 0.0.4) ruby depends on
multipart-post (
> 1.1.0) ruby

databasedotcom (~> 1.3.2) ruby depends on
  multipart-post (1.2.0)

*2) With the following in my gemfile:
gem "databasedotcom", "~> 1.3.2"
gem "salesforce_bulk_api"

I get this error:
Bundler could not find compatible versions for gem "addressable":
In Gemfile:
salesforce_bulk_api (>= 0) ruby depends on
addressable (~> 2.2.4) ruby

launchy (>= 0) ruby depends on
  addressable (2.3.2)

API Version 32 in 0.0.12?

👋

I just noticed that my bulk client is using API version 32. I took a look on this repo and I see that the API version was updated to 46, but I don't see those changes in the files that gem installed when I gem install salesforce_bulk_api

I tried uninstalling and reinstalling the gem:

❯ gem uninstall salesforce_bulk_api
Successfully uninstalled salesforce_bulk_api-0.0.12

❯ gem install salesforce_bulk_api
Fetching salesforce_bulk_api-0.0.12.gem
Successfully installed salesforce_bulk_api-0.0.12
Parsing documentation for salesforce_bulk_api-0.0.12
Installing ri documentation for salesforce_bulk_api-0.0.12
Done installing documentation for salesforce_bulk_api after 0 seconds
1 gem installed

❯ gem which salesforce_bulk_api
/Users/robgoodberry/.gem/ruby/2.6.5/gems/salesforce_bulk_api-0.0.12/lib/salesforce_bulk_api.rb

Then if I vi /Users/robgoodberry/.gem/ruby/2.6.5/gems/salesforce_bulk_api-0.0.12/lib/salesforce_bulk_api.rb I'm still seeing the older code with the @@ class variables and API version 32:

Bundler.require()
require 'salesforce_bulk_api/version'
require 'net/https'
require 'xmlsimple'
require 'csv'
require 'salesforce_bulk_api/concerns/throttling'
require 'salesforce_bulk_api/job'
require 'salesforce_bulk_api/connection'

module SalesforceBulkApi

  class Api
    attr_reader :connection

    @@SALESFORCE_API_VERSION = '32.0'

Am I messing something up here or has the most recent master just not been released?

Thanks for any help!

Explain improvements against salesforce_bulk gem

The README says this is a rewritten version of salesforce_bulk gem, but it doesn't say what was improved. Please provide an explanation of what's different between salesforce_bulk_api and salesforce_bulk.

Web Proxy Capability

I was having problems using Ruby 1.9.3 connecting to SalesForce using this gem. It looks like Net::HTTP.new takes different sets of arguments if a proxy is needed so I modified the https method in the SalesforceBulkApi::Connection class to the following:

def https(host)
      req = Net::HTTP.new(host, 443, @proxy_url, @proxy_port, @proxy_user, @proxy_pass)
      req.use_ssl = true
      req.verify_mode = OpenSSL::SSL::VERIFY_NONE
      req
 end

This seemed to fix the problem for me but I wasn't sure how you wanted this worked into the Connection class. I was thinking of setting proxy variables in the connection from the Restforce/Database.com client if the proxy_uri option is set. If you think this is a decent approach, I'll work it up and send in a PR for it.

Support for streaming?

From the examples in the homepage README, it looks like any bulk query you submit must fit into memory as a Ruby array of hashes. Would it be possible to stream the results to a file for queries that pull back a lot of data?

Nils being inserted into updates that don't have them.

I'm trying to use the gem, but I can't because it is inserting NIL values where I don't want them.

The basic problem is this:

data = [ { id: '1', Field_1__c: nil}, { id: '2', Field_2__c: nil] } ]

bulk_api.update(:contact, data, true, true)

What I expect is for record 1 to have Field_1__c set to nil, and record 2 to have Field_2__c set to nil. What I get is both records having both fields blanked out.

Renamed module, broken code

The SalesforceBulkApi module is a rename from SalesforceBulk but the code references SalesforceBulk . This causes the gem not to function at all.

Setup travis to run continuous testing.

Travis should have a set of encrypted credentials to a developer org containing no modifications to salesforce's vanilla behavior.

This will make sure that tests pass on the same environment each run.

Not able to create objects in sales force

I tried sending a request to salesforce using salesforce_bulk_api gem. I was successful in creating the connection and I was also able to query all the domains in salesforce. But I tried creating a ResourceRequest through the gem. I always get nil as the result. I got the following output when i executed the create statement. But no records where created :(. As far as i know, there are no mistakes with the data and i am very sure that it did not fail due to domain validations. Please help.

750R00000009aOMIAY insert pse__Resource_Request__c 00550000001DA2zAAG 2012-05-17T05:49:38.000Z 2012-05-17T05:49:38.000Z Open Parallel XML 0 0 0 0 0 0 0 23.0 0 0 0 0

Doesn't indicate or elaborate on errors in SF (e.g. validation)

Leads to wasted hours investigating the below:

I'm able to push records to SF using the Databasedotcom gem itself, but I can't get bulk to work. Something like this:

records = [{"Ticket__c" => "100", "Subject__c" => "Foo"}, ...]
bulk.upsert("Redmine_Tickets__c", records, "Ticket__c")

prints out this, and no records are created or updated in SF:

na4
na4.salesforce.com
<?xml version="1.0" encoding="UTF-8"?><jobInfo xmlns="http://www.force.com/2009/06/asyncapi/dataload">
 <id>750600000011S5oAAE</id>
 <operation>upsert</operation>
 <object>Redmine_Tickets__c</object>
 <createdById>00560000001y43UAAQ</createdById>
 <createdDate>2013-10-03T16:07:34.000Z</createdDate>
 <systemModstamp>2013-10-03T16:07:34.000Z</systemModstamp>
 <state>Open</state>
 <externalIdFieldName>Ticket__c</externalIdFieldName>
 <concurrencyMode>Parallel</concurrencyMode>
 <contentType>XML</contentType>
 <numberBatchesQueued>0</numberBatchesQueued>
 <numberBatchesInProgress>0</numberBatchesInProgress>
 <numberBatchesCompleted>0</numberBatchesCompleted>
 <numberBatchesFailed>0</numberBatchesFailed>
 <numberBatchesTotal>0</numberBatchesTotal>
 <numberRecordsProcessed>0</numberRecordsProcessed>
 <numberRetries>0</numberRetries>
 <apiVersion>23.0</apiVersion>
 <numberRecordsFailed>0</numberRecordsFailed>
 <totalProcessingTime>0</totalProcessingTime>
 <apiActiveProcessingTime>0</apiActiveProcessingTime>
 <apexProcessingTime>0</apexProcessingTime>
</jobInfo>
75160000001YF90AAG

Am I doing something wrong?

XML Parsing Error: mismatched tag while upserting objects

Hello everyone,

I currently am experiencing an error while upserting objects in SF from our backend.
Before sending the data to the Salesforce API (version 41), I log the xml that I send and it is well formatted. The job then fails on Salesforce side with the error: "XML Parsing Error: mismatched tag". Indeed, when I look at the XML processed by salesforce, I can see this:
<Client__c>a050X00000zLdPsQAK</Clieent__c> in the middle of the XML. I do not really understand why.

Did anyone ever face this problem and find a trick to solve it?

Here is the configuration that I use:

  • ruby 2.6.0
  • rails 5.2.2
  • gem 'restforce', '~> 3.1.0'
  • gem 'salesforce_bulk_api' , '~> 0.12.0'

Thank you very much

@session_id is nil when using Restforce

Hi,

when I user Restforce client with username and password + client_id and client_secret the @session_id is is nil. Instead if i use a token from Salesforce OAuth2 authentication with no username and password the @session_id is correctly set.

Is there any workaround for this?

Add a license file

What is the license for using this gem? Can you add a license file to this project??

Gem dependency issue with Faraday

We have a tailored sinatra app that is able to talk to salesforce using Databasedotcom gem. We also have some patching to faraday to tailor URI connection. The faraday gem we installed is version 0.7.6 and when adding in salesforce_bulk_api into Gemfile and then running bundle install, it is throwing back:

Bundler could not find compatible versions for gem "faraday":
  In Gemfile:
    salesforce_bulk_api (>= 0) ruby depends on
      faraday (~> 0.6.1) ruby

    faraday (0.7.6)

I guess I could downgrade faraday and give my app a try, but when checking rubygems site https://rubygems.org/gems/faraday/versions, even the 0.7.6 version of faraday is over 1 year old, 0.6.1 is back dated to April 2011. Is it possible to have salesforce_bulk_api use the latest faraday?

Batch request error

When it sends a batch request I get this error when I go into the Bulk Data load Job detail within salesforce.com and click on view request.

    This page contains the following errors:

    error on line 1 at column 1: Document is empty
   Below is a rendering of the page up to the first error.

Occasionally getting "undefined method `[]' for nil:NilClass" errors in salesforce_bulk_api gem

I suspect that this is due to salesforce sending a response that this gem doesn't expect.

/u/apps/pd_bizops_integration/shared/bundle/ruby/2.2.0/gems/activerecord-3.2.17/lib/active_record/associations/has_many_association.rb:80: warning: circular argument reference - reflection
/u/apps/pd_bizops_integration/shared/bundle/ruby/2.2.0/gems/railties-3.2.17/lib/rails/commands/runner.rb:54:in `eval': undefined method `[]' for nil:NilClass (NoMethodError)
        from /u/apps/pd_bizops_integration/shared/bundle/ruby/2.2.0/gems/salesforce_bulk_api-0.0.12/lib/salesforce_bulk_api/job.rb:175:in `each'
        from /u/apps/pd_bizops_integration/shared/bundle/ruby/2.2.0/gems/salesforce_bulk_api-0.0.12/lib/salesforce_bulk_api/job.rb:175:in `block in get_job_result'
        from /usr/local/rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/timeout.rb:89:in `block in timeout'
        from /usr/local/rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/timeout.rb:99:in `call'
        from /usr/local/rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/timeout.rb:99:in `timeout'
        from /u/apps/pd_bizops_integration/shared/bundle/ruby/2.2.0/gems/salesforce_bulk_api-0.0.12/lib/salesforce_bulk_api/job.rb:172:in `get_job_result'
        from /u/apps/pd_bizops_integration/shared/bundle/ruby/2.2.0/gems/salesforce_bulk_api-0.0.12/lib/salesforce_bulk_api.rb:79:in `do_operation'
        from /u/apps/pd_bizops_integration/shared/bundle/ruby/2.2.0/gems/salesforce_bulk_api-0.0.12/lib/salesforce_bulk_api.rb:29:in `update'
        from /u/apps/pd_bizops_integration/releases/20150821062234/lib/sfdc.rb:49:in `do_bulk!'
        from /u/apps/pd_bizops_integration/releases/20150821062234/lib/sfdc.rb:26:in `update_object!'
        from /u/apps/pd_bizops_integration/releases/20150821062234/lib/sfdc.rb:18:in `update_transaction_object!'
        from /u/apps/pd_bizops_integration/releases/20150821062234/app/models/tasks/sfdc_transaction_updater.rb:49:in `update'
        from /u/apps/pd_bizops_integration/releases/20150821062234/app/models/tasks/sfdc_transaction_updater.rb:6:in `block in run'
        from /u/apps/pd_bizops_integration/releases/20150821062234/lib/cron_utils.rb:26:in `call'
        from /u/apps/pd_bizops_integration/releases/20150821062234/lib/cron_utils.rb:26:in `block (2 levels) in run_with_lock_and_benchmark'
        from /u/apps/pd_bizops_integration/releases/20150821062234/lib/cron_utils.rb:16:in `block in filelock'
        from /u/apps/pd_bizops_integration/releases/20150821062234/lib/cron_utils.rb:14:in `open'
        from /u/apps/pd_bizops_integration/releases/20150821062234/lib/cron_utils.rb:14:in `filelock'
        from /u/apps/pd_bizops_integration/releases/20150821062234/lib/cron_utils.rb:25:in `block in run_with_lock_and_benchmark'
        from /usr/local/rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/benchmark.rb:303:in `realtime'
        from /u/apps/pd_bizops_integration/releases/20150821062234/lib/cron_utils.rb:24:in `run_with_lock_and_benchmark'
        from /u/apps/pd_bizops_integration/releases/20150821062234/app/models/tasks/sfdc_transaction_updater.rb:5:in `run'
        from (eval):1:in `<top (required)>'
        from /u/apps/pd_bizops_integration/shared/bundle/ruby/2.2.0/gems/railties-3.2.17/lib/rails/commands/runner.rb:54:in `eval'
        from /u/apps/pd_bizops_integration/shared/bundle/ruby/2.2.0/gems/railties-3.2.17/lib/rails/commands/runner.rb:54:in `<top (required)>'
        from /u/apps/pd_bizops_integration/shared/bundle/ruby/2.2.0/gems/railties-3.2.17/lib/rails/commands.rb:64:in `require'
        from /u/apps/pd_bizops_integration/shared/bundle/ruby/2.2.0/gems/railties-3.2.17/lib/rails/commands.rb:64:in `<top (required)>'
        from script/rails:12:in `require'
        from script/rails:12:in `<main>'

Edit: I added logging functionality around the issue at:
https://github.com/PagerDuty/salesforce_bulk_api/commit/401239a46641c6e7fda75e4d28e4b1701931c422

Bump bulk api

I set the version to '32.0' in my own branch and it seems to work for the most part.

https://github.com/gostrc/salesforce_bulk_api/tree/bump-api

I did this because there are some fields that require this newer version bump.

e.g. I was getting this error before bumping the api:

ERROR at Row:1:Column:427 No such column 'CleanStatus' on entity 'Account'. If you are attempting to use a custom field, be sure to append the '__c' after the custom field name. Please reference your WSDL or the describe call for the appropriate names.

I didn't create a pull request in case you wanted to do some extra testing around the branch.
I noticed that null values came up with the following value:

"field"=>[{"xsi:nil"=>"true"}],

Sleep statement in job.rb causes performance losses, bleeds into client implementation details.

Hello there, and thank you for maintaining this Salesforce Bulk API library! It has definitely helped me develop against Salesforce with its easy to use interface, so thanks for that :)

While writing testing code for my application that currently uses your utility, I noticed a significant performance decrease as I began integrating calls to the bulk api service. After a bit of investigation, I noticed that the call to sleep here, was the sole reason of our slowdown, even after using VCR to mock out our web requests.

It's not clear to me why this sleep statement is necessary, other than to give a small buffer of time for the next batch in sequence to change status; it doesn't guarantee whether or not the next batch will be finished.

If the library needs to ensure the processing of each batch, perhaps it could instead provide some sort of blocking operation on each batch?

For example:

@batch_ids.each do |batch_id|

  # block execution until batch is ready 
  while true 
    batch_state = self.check_batch_status(batch_id)
    if batch_state['state'][0] != "Queued" && batch_state['state'][0] != "InProgress"
      state << (batch_state)
      @batch_ids.delete(batch_id)

      # once the batch is added to `state`, break out
      # to the next batch in iteration
      break
    end
  end

end

This removes the need for a call to sleep, which bleeds into the implementation details of into client programs and can lead to race conditions in some cases.

I would be interested to learn more about your thoughts on the issue!

Thanks,
-Kelly.

License question

Can you please state under what license this gem is distributed?

Use csv format to download batches.

In the job file:
https://github.com/yatish27/salesforce_bulk_api/blob/master/lib/salesforce_bulk_api/job.rb#L205

You are using xml to download batches of large data sets.
This is unideal since salesforce allows you to specify csv [0] as the type.

This would have the following pros:

  1. Simplifying the parsing code
  2. Smaller download size
  3. Less cpu time spent parsing
  4. Less memory used trying to hold the xml

[0] - http://www.salesforce.com/us/developer/docs/api_asynch/
under Bulk Query -> Bulk Query Details

How to make a bulk call with multiple queries?

The example for queries only shows a single query being submitted. How do you run multiple queries in one bulk call? When I follow the example in the README it runs immediately and synchronously returns the values for only the one call.

bulk = SalesforceBulkApi::Api.new(restforce_client)
response = bulk.query('Contact', "SELECT Id, AccountId FROM Contact WHERE Email = '[email protected]'")

response['batches'][0]['response'] returns:

[
    [0] {
         "xsi:type" => "sObject",
             "type" => [
            [0] "Contact"
        ],
               "Id" => [
            [0] "00336000004AVvHAAW",
            [1] "00336000004AVvHAAW"
        ],
        "AccountId" => [
            [0] "001360000027qGJAAY"
        ]
    }
]

What I would like to do is something like the example below, but it's unclear how:

array = []
array << "SELECT Id, AccountId FROM Contact WHERE Email = '[email protected]'"
array << "SELECT Id, AccountId FROM Contact WHERE Email = '[email protected]'"
bulk.query('Contact', array)

Thanks!

As a client of salesforce_bulk_api, I expect to recover from API errors.

It appears as though that the current job.rb file errors out on occasion, claming:

salesforce_bulk_api-0.0.10/lib/salesforce_bulk_api/job.rb:35 undefined method `[]' for nil:NilClass

I experience this issue pretty consistently around 1AM PDT. It's unclear if this is a limitation of the API or there's some sort of issue with the salesforce_bulk_api gem itself.

Before any action can be taken, I think it might be useful to first get some more information about the status of each call by providing a success, error, and complete callback so that clients of salesforce_bulk_api can inspect the status of each job, respectively.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.