brandwatchltd / bcr-api Goto Github PK
View Code? Open in Web Editor NEWPython Client Library for the Brandwatch Consumer Research API
License: MIT License
Python Client Library for the Brandwatch Consumer Research API
License: MIT License
Solved
I have been trying for long to download via api or in the app at least 7 days but all the methods are getting wrong data. Is there any way i can get the snippet via api?
Hi,
I'm curious if the search feature that you can apply additional queries to backfilled queries on Web UI is also available through API calls.
Thanks,
Q
Hi,
Can I choose "Legacy Instagram" in contentSources during using BWQueries(project).upload as it can be chose as an option of contentsources on brandwatch website UI?
Does BWSignals(project) send notification emails to subscribers when events like backfills are completed with notificationThreshold =1 ?
Is there any api feature that checks status of backfill because I would like to track how long some backfill takes ?
thanks,
Q
KeyError Traceback (most recent call last)
in
----> 1 filtered = queries.get_mentions(name = "Himelboim-COVID-19 Vaccine ",
2 startDate = start,
3 endDate = today)
~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\bcr_api\bwdata.py in get_mentions(self, name, startDate, max_pages, **kwargs)
34 key: value for (key, value) in kwargs.items() if key != "iter_by_page"
35 }
---> 36 all_mentions = list(
37 self.iter_mentions(
38 name=name,
~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\bcr_api\bwdata.py in iter_mentions(self, name, startDate, max_pages, iter_by_page, **kwargs)
78 else:
79 page_idx += 1
---> 80 next_cursor, next_mentions = self._get_mentions_page(params)
81 if len(next_mentions) > 0:
82 cursor = next_cursor
~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\bcr_api\bwdata.py in _get_mentions_page(self, params)
994 raise KeyError("Mentions GET request failed", mentions)
995
--> 996 return mentions.get("nextCursor", None), mentions["results"]
997
998 def _valid_input(self, param, setting):
KeyError: 'results'
When iterating through BWQueries.iter_mentions()
there's no way to keep track of current page, retry or skip a page that yields an error. What's more, internal server errors remain uncaught - there's error
key in the response, but the code looks for errors
: https://github.com/BrandwatchLtd/bcr-api/blob/master/src/bcr_api/bwdata.py#L993
Hello,
We are using this package to create a query over brandwatch. Previously we were able to do that with the same code base but right now it seem to be failing with the following error message, not sure if anything have been changed from Brandwatch side and if this API needs to be updated.
Tried the same code with 1.4.0, 1.5.1 version of bcr-api
Code:
project = BWProject(username=YOUR_ACCOUNT, project=YOUR_PROJECT, token_path=YOUR_TOKEN_PATH)
queries = BWQueries(project)
corporation_booleanQuery = '("Sasol" OR "Rio Tinto" OR "DRDGold" OR "Durban Roodepoort Deep Limited") AND (site:cer.org.za)'
queries.upload(name = 'testing 0' ,
booleanQuery = corporation_booleanQuery,
startDate = '2019-01-01',
languages = ['en'],
contentSources = ['news'])
Error message:
05:22:23 ERROR: There was an error with this request:
https://api.brandwatch.com/query-validation
{"booleanQuery": "(\"Sasol\" OR \"Rio Tinto\" OR \"DRDGold\" OR \"Durban Roodepoort Deep Limited\") AND (site:cer.org.za)", "language": ["en"]}
[{'code': 0, 'message': "Unrecognized field 'language'"}]
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
/var/tmp/ipykernel_6175/1702768516.py in <module>
17 startDate = query_start_date,
18 # languages = languages,
---> 19 contentSources = ['news'])
20 corporation_queries.append(f'{backfill_portfolio_name} Client Onboarding Mentions ' + str(idx)) # entities addition
21 # corporation_queries.append(f'Corporation Mentions ' + str(idx)) # domains addition
/opt/conda/lib/python3.7/site-packages/bcr_api/bwresources.py in upload(self, create_only, modify_only, **kwargs)
296 The uploaded query information in a dictionary of the form {query1name: query1id}
297 """
--> 298 return self.upload_all([kwargs], create_only, modify_only)
299
300 def upload_all(self, data_list, create_only=False, modify_only=False):
/opt/conda/lib/python3.7/site-packages/bcr_api/bwresources.py in upload_all(self, data_list, create_only, modify_only)
320
321 queries = super(BWQueries, self).upload_all(
--> 322 data_list, create_only=False, modify_only=False
323 )
324
/opt/conda/lib/python3.7/site-packages/bcr_api/bwresources.py in upload_all(self, data_list, create_only, modify_only)
162 for data in data_list:
163 # eventually make _fill_data() a BWResource func
--> 164 filled_data = self._fill_data(data)
165 name = data["name"]
166
/opt/conda/lib/python3.7/site-packages/bcr_api/bwresources.py in _fill_data(self, data)
506 # validating the query search - comment this out to skip validation
507 self.project.validate_query_search(
--> 508 booleanQuery=filled["booleanQuery"], language=["en"]
509 )
510 return json.dumps(filled)
/opt/conda/lib/python3.7/site-packages/bcr_api/bwproject.py in validate_query_search(self, **kwargs)
138
139 valid_search = self.request(
--> 140 verb=requests.post, address="query-validation", data=json.dumps(kwargs)
141 )
142 return valid_search
/opt/conda/lib/python3.7/site-packages/bcr_api/bwproject.py in request(self, verb, address, params, data)
182 access_token=self.token,
183 params=params,
--> 184 data=data,
185 )
186
/opt/conda/lib/python3.7/site-packages/bcr_api/bwproject.py in bare_request(self, verb, address_root, address_suffix, access_token, params, data)
238 )
239 )
--> 240 raise RuntimeError(response.json()["errors"])
241
242 logger.debug(response.url)
RuntimeError: [{'code': 0, 'message': "Unrecognized field 'language'"}]
After Downloading the data ,I could see that the twitter data is empty under fullText ,I cant see any data under that domain twitter in the dump downloaded
Maybe the field is removed in BCR? The upload method seems to work after commenting out this line:
bcr-api/src/bcr_api/bwresources.py
Line 487 in 05ece05
If I do queries.upload()
with samplePercentage=10
, I would expect to see the samplePercentage
field when I .get()
that query to be set to 10. However, it is actually set to 100. @ctafoyabwatch did some investigation, and it looks like the userRequestedSampling
field needs to be passed in as True
for a user to manually set the sample rate (as opposed to BCR deciding the sample rate).
A simple fix might be to use to see if a user has passed in samplePercentage
when uploading a query, and if so, set userRequestedSampling
to True
. We'll need to make sure this overrides the default value of 100.
Whether I authenticate in terminal with bcr-authenticate or with BWUser (both successful), I get an error with BWProject assigning either project name or ID to YOUR_PROJECT:
YOUR_ACCOUNT = "{name}@{domain}.com"
YOUR_PROJECT = {projectID}
project = BWProject(username=YOUR_ACCOUNT, project=YOUR_PROJECT)
17:26:04 DEBUG: https://api.brandwatch.com/projects
For the sorry souls who follow: This isn't a problem - if you get this debug message, you can just keep working normally!
@bw-matthew noticed that bwproject.py seems to override the logging level to:
logger.setLevel(logging.DEBUG)
. This doesn't seem right
Some people have raised that they don't know the date format expected by methods such as get_chart(). We could add this as a note in the docstring
When trying to instantiate BWProject with username and project i receive:
13:50:59 DEBUG: https://api.brandwatch.com/projects
{"error":"unauthorized","error_description":"Invalid authentication credentials found on request"}
I have a bearer token in tokens.txt and I've tried the same bearer token manually via requests on my own environment and can access the project and queries so the token works. Might be an issue with the version of requests?
I'm currently using:
python - 3.7.6
requests - 2.22.0
Hi,
I’ve been using Brandwatch actively nowadays for a project almost like requesting hundreds of backfills and extracting data every day.
However, I’m stuck while I was trying Brandwatch API calls to extract data more efficiently and requesting backfills.
First, can I choose "Legacy Instagram" in contentSources during using BWQueries(project).upload as it’s available on Web UI ?
contentSources = ["blog","forum","news","twitter","review", "instagram","facebook","qq","reddit","tumblr","youtube","lexis_nexis_licensed_news"]
Does BWSignals(project) send notification emails to subscribers when events like backfills are completed if I set notificationThreshold = 1 ?
signals.upload(name= "signal_test",
queries= ["sample_query"],
subscribers= [{"emailAddress": [email protected], "notificationThreshold": 1}])
Is there any API feature that checks status of backfill or get backfill time because I would like to track how long some backfill takes ?
Also, I found out that data extracted through UI which is dashboard and data extracted through APIs are slightly different. ( difference could be more than thousands in terms of mention volume )
I assume that timezone or time setting has caused the data difference when using API BWQueries.get_chart.
I guess if I set Date on Brandwatch Dashboards as from 2019-01-01 to 2021-01-31, it automatically applies as the project region timezone, which is Asia/Seoul that I set before and the date would be converted as internally like 2019-01-01 +0900 and 2021-01-31 +0900 and monthly sum data is calculated starting from every 01-01 +0900 not 01-01 +0000. causes the data difference
And I understand startDAte and endDate are not for setting the first date of each month when providing monthly data but
I tried queries.get_chart in two different ways like below
queries.get_chart(name=i,
startDate = "2019-01-01",
endDate = "2021-01-31",
y_axis= "volume",
x_axis= "months",
breakdown_by = "queries"
)
I also tried to set startDate and endDate as below to get date based on Asia/Seoul(KST) standard
queries.get_chart(name=i,
startDate = "2019-01-01T00:00:00.000+0900",
endDate = "2022-02-01T00:00:00.000+0900",
y_axis= "volume",
x_axis= "months",
breakdown_by = "queries"
)
Even though default timezone of my project in Brandwatch is already set as Asia/Seoul(KST), the results of API call “BWQueries.get_chart” below, which is for getting monthly data, says that the first date of each monthly data ( results part ) is from UTC 0 not UTC +9 which is Asia/Seoul(KST) standard
{'dimension1': 'months',
'dimension2': 'queries',
'aggregate': 'volume',
'results': [{'id': '2019-01-01T00:00:00.000+0000',
'name': '2019-01-01 00:00:00.0',
'data': {'timeUnit': 'months'},
'values': [{'id': 2000877612,
'name': 'C brand',
'data': {},
'value': 5}]},
Could you share how to set a first date of each month as Asia/Seoul(KST) standard for monthly data using API BWQueries.get_chart ?
Please let me know if you are available for a quick chat over Zoom
Thanks,
Q
The fix in this commit 7aba3eb is not in the pypi package of 1.3.1, thus the rule validation function is broken.
Can someone please republish it to pypi?
Dear manager,
i'm trying to connect "bcr-authenticate" in command line
but i got a message below
-----below----
Authenticating user: [email protected]
Traceback (most recent call last):
File "c:\programdata\anaconda3\lib\site-packages\urllib3\connection.py", line 159, in _new_conn
conn = connection.create_connection(
File "c:\programdata\anaconda3\lib\site-packages\urllib3\util\connection.py", line 84, in create_connection
raise err
File "c:\programdata\anaconda3\lib\site-packages\urllib3\util\connection.py", line 74, in create_connection
sock.connect(sa)
TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\programdata\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "c:\programdata\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 381, in _make_request
self._validate_conn(conn)
File "c:\programdata\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 978, in _validate_conn
conn.connect()
File "c:\programdata\anaconda3\lib\site-packages\urllib3\connection.py", line 309, in connect
conn = self._new_conn()
File "c:\programdata\anaconda3\lib\site-packages\urllib3\connection.py", line 171, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x000002BF7CAD04C0>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\programdata\anaconda3\lib\site-packages\requests\adapters.py", line 439, in send
resp = conn.urlopen(
File "c:\programdata\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 726, in urlopen
retries = retries.increment(
File "c:\programdata\anaconda3\lib\site-packages\urllib3\util\retry.py", line 446, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.brandwatch.com', port=443): Max retries exceeded with url: /oauth/token?username=myemail%40email.com&grant_type=api-password&client_id=brandwatch-api-client (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x000002BF7CAD04C0>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\programdata\anaconda3\lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "c:\programdata\anaconda3\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\ProgramData\Anaconda3\Scripts\bcr-authenticate.exe_main.py", line 7, in
File "c:\programdata\anaconda3\lib\site-packages\bcr_api\authenticate.py", line 66, in main
user = authenticate(args.username, args.password, credentials_path=args.store)
File "c:\programdata\anaconda3\lib\site-packages\bcr_api\authenticate.py", line 22, in authenticate
return BWUser(username=username, password=password, token_path=credentials_path)
File "c:\programdata\anaconda3\lib\site-packages\bcr_api\bwproject.py", line 60, in init
self.username, self.token = self._get_auth(
File "c:\programdata\anaconda3\lib\site-packages\bcr_api\bwproject.py", line 94, in _get_auth
token = requests.post(
File "c:\programdata\anaconda3\lib\site-packages\requests\api.py", line 119, in post
return request('post', url, data=data, json=json, **kwargs)
File "c:\programdata\anaconda3\lib\site-packages\requests\api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "c:\programdata\anaconda3\lib\site-packages\requests\sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "c:\programdata\anaconda3\lib\site-packages\requests\sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "c:\programdata\anaconda3\lib\site-packages\requests\adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.brandwatch.com', port=443): Max retries exceeded with url: /oauth/token?username=myemail%40email.com&grant_type=api-password&client_id=brandwatch-api-client (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x000002BF7CAD04C0>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond'))
------end
I took action solving this error
'curl -X POST --data-urlencode 'password=[mypassword]' \ 'https://api.brandwatch.com/oauth/token?username=[[email protected]]&grant_type=api-password&client_id=brandwatch-api-client'
but all ways failed. please help to solve this problem.
thank you
(installed python version : Python 3.8.5)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.