cfpb / ccdb5-api Goto Github PK
View Code? Open in Web Editor NEWAn API that provides an interface to search complaint data.
License: Creative Commons Zero v1.0 Universal
An API that provides an interface to search complaint data.
License: Creative Commons Zero v1.0 Universal
I see that there is an attempt to get total_record_count
here but when I run it, it turns up zero:
"_meta": {
"total_record_count": 0,
"last_updated": "2017-07-10T00:00:00.000Z",
"license": "CC0"
},
produces the same set as
But with frm=20
that shouldn't be the case
Would be a very useful feature to be able to aggregate complaints by complaints per company -- even limited to the top 10 or 20 companies for a particular filter.
Hi,
Is there a way to get all products and companies using one of the GET endpoint? (I would assume they would have different query parameters if this is possible).
When you go to
https://cfpb.github.io/api/ccdb/api/index.html#/Complaints/get_
click "search consumer complaints", then "Try it out" and "Execute"
If you don't specify the format the response will be an html-page with js.
If you specify the format as json or csv it will never not load (hangs).
I get the same result with urllib library from python. The only workaround was to specify "?format=default" which is not documented
Per the documentation:
Search by particular field. If this parameter is not specified, default behavior
will return only complaints with narratives.
Although, the first hit
when you perform a search with no criteria is the complaint with ID 2662569
, which does not have a narrative.
It seems the current implementation is not behaving as expected.
Steps to reproduce:
/_suggest/?text=Mort&size=6
{
"sgg": [
{
"text": "Mort",
"length": 4,
"options": [],
"offset": 0
}
],
}
I would expect "mortgage" as one example
When I search with state=WY, I get 166 matches, but aggregations.state
does not list WY
as one of the buckets.
Is this aggregation being limited? Notice sum_other_doc_count
"aggregations": {
... snip ...
"state": {
"doc_count": 166148,
"state": {
"buckets": [
{
"doc_count": 23505,
"key": "CA"
},
{
"doc_count": 15105,
"key": "FL"
},
{
"doc_count": 14685,
"key": "TX"
},
... snip ...
{
"doc_count": 245,
"key": "SD"
},
{
"doc_count": 204,
"key": "VT"
},
{
"doc_count": 193,
"key": "ND"
}
],
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 597
}
},
When I try http://localhost:8000/data-research/consumer-complaints/search/api/v1/?field=all&size=25&sort=created_date_asc
I get:
{"error":"Elasticsearch error: search_phase_execution_exception"}
The same happens for sort=created_date_desc
In API documentation for Trends, date_received_max is defined as "Return results with date <= date_received_max (i.e. 2017-03-04)"
https://cfpb.github.io/api/ccdb/api/index.html#/Trends/get_trends
However, I believe that this actually returns results with date < the entered date, not <=. Please double check, but a couple searches seems to confirm this.
Installation of this package with pip 10 fails due to the use of pip.req
here in setup.py.
$ pip --version
pip 10.0.1 from ...
$ pip install git+https://github.com/cfpb/[email protected]#egg=ccdb5-api
Collecting ccdb5-api from git+https://github.com/cfpb/[email protected]#egg=ccdb5-api
Cloning https://github.com/cfpb/ccdb5-api.git (to revision v1.0.5) to /private/var/folders/gg/wxm3673x2356ll65n0xbm_7r0000gn/T/pip-install-CacAhx/ccdb5-api
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/private/var/folders/gg/wxm3673x2356ll65n0xbm_7r0000gn/T/pip-install-CacAhx/ccdb5-api/setup.py", line 32, in <module>
install_requires = parse_requirements()
File "/private/var/folders/gg/wxm3673x2356ll65n0xbm_7r0000gn/T/pip-install-CacAhx/ccdb5-api/setup.py", line 21, in parse_requirements
requirements = pip.req.parse_requirements(
AttributeError: 'module' object has no attribute 'req'
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/gg/wxm3673x2356ll65n0xbm_7r0000gn/T/pip-install-CacAhx/ccdb5-api/
See this related discussion about use of pip internals; an alternative approach would be to define requirements in the setup.py install_requires
instead of in requirements.txt. See related setuptools documentation here.
Highlight should occur in the field searched.
There seems to be a discrepancy between the Swagger documentation and the actual response object. The documentation indicates that Hits.total should parse as an integer.
In practice, for the /
endpoint, I found that hits.total is an object that looks something like this:
{
"value": 3198,
"relation": "eq"
}
Example API call:
An API request with special characters (curly quotes โ) times out:
https://www.consumerfinance.gov/data-research/consumer-complaints/search/api/v1/?date_received_max=2022-01-04&date_received_min=2019-01-04&field=all&search_term=%E2%80%9Cmortgage%20default%E2%80%9D~3&size=25&sort=created_date_desc
Whereas one with regular quotes is fine:
https://www.consumerfinance.gov/data-research/consumer-complaints/search/api/v1/?date_received_max=2022-01-04&date_received_min=2019-01-04&field=all&search_term=%22mortgage%20default%22~3&size=25&sort=created_date_desc
The API should handle special characters in the request.
The Trends API seems broken.
Using the following code: curl -X GET "https://www.consumerfinance.gov/data-research/consumer-complaints/search/api/v1/trends?lens=overview&trend_interval=yearly" -H "accept: application/json"
And if used in command line, curl returns the page HTML, rather than anything useful.
Other APIs don't seem to give the same problem.
There's a typo
https://cfpb.github.io/api/ccdb/api/index.html#/Complaints/get__complaintId_
it says find "comsumer" but should be "consumer"
Also I believe that the trends endpoint https://cfpb.github.io/api/ccdb/api/index.html#/Trends/get_trends
is missing the "focus" parameter description
1.) I still cannot use the raw size of the dataset. When I include the full size, it errors:
/data-research/consumer-complaints/search/api/v1/?date_received_min=2017-01-03&field=all&format=json&no_aggs=true&size=135811&sort=relevance_desc
2.) I get an error when I try to use CSV format
/data-research/consumer-complaints/search/api/v1/?date_received_min=2017-01-03&field=all&format=csv&no_aggs=true&size=10000&sort=relevance_desc
Queries' results are cut to a maximum of 100 records and parameter from
doesn't work which prohibits most meaningful applications of the API presented
When doing some testing against a cf.gov environment that could not talk to Elasticsearch, I saw the following error at /data-research/consumer-complaints/search/api/v1/?field=all&size=25&sort=created_date_desc
:
Elasticsearch error: <urllib3.connection.HTTPConnection object at 0x7f5558fef650>: Failed to establish a new connection: [Errno 111] Connection refused
This was in a non-debug environment where I would not expect to see a raw exception leaking into a response like that. I thought I'd open an issue in case this is something that could be adjusted.
I'm also wondering if a 400
(see here) is appropriate in this scenario, since it seems more like a 500
situation to me.
When running tox
, during the py36-dj111
testing step, the tests pass, but they result in several warnings and errors printed to the testing output.
Here's a sample run of the test suite:
complaint_search/tests/test_view_suggest_company.py:2: RemovedInDjango20Warning: Importing from django.core.urlresolvers is deprecated in favor of django.urls.
from django.core.urlresolvers import reverse
Creating test database for alias 'default'...
ccdb5_api/urls.py:20: RemovedInDjango20Warning: Passing a 3-tuple to django.conf.urls.include() is deprecated. Pass a 2-tuple containing the list of patterns and app_name, and provide the namespace argument to include() instead.
url(r'^admin/', include(admin.site.urls)),
ccdb5_api/urls.py:21: RemovedInDjango20Warning: Specifying a namespace in django.conf.urls.include() without providing an app_name is deprecated. Set the app_name attribute in the included module, or pass a 2-tuple containing the list of patterns and app_name instead.
url(r'^', include('complaint_search.urls', namespace="complaint_search")),
System check identified no issues (0 silenced).
..............................................................tox/py36-dj111/lib/python3.6/site-packages/django/core/handlers/base.py:52: RemovedInDjango20Warning: Old-style middleware using settings.MIDDLEWARE_CLASSES is deprecated. Update your middleware and use settings.MIDDLEWARE instead.
"instead.", RemovedInDjango20Warning
TransportError(N/A, 'Error')
....TransportError(N/A, 'Error')
....Out of memory
.TransportError(N/A, 'Error')
................................................TransportError(N/A, 'Error')
.......TransportError(N/A, 'Error')
....TransportError(N/A, 'Error')
....
----------------------------------------------------------------------
Ran 133 tests in 3.675s
lint: commands succeeded
py36-dj111: commands succeeded
congratulations :)
According to the design specs, we need to search within the company
field.
Steps to reproduce:
?field=company&search_term=bank
{"field":["\"company\" is not a valid choice."]}
It says that
date_received_max
string
(query)
Return results with date < date_received_max (i.e. 2017-03-04)
But in fact it queries with date <= date_received_max
, for instance:
?date_received_min=2012-01-01&date_received_max=2012-01-01&format=default&size=50
Returns 14 records with date_received = 2012-01-01
I hope that this behavior won't change and the documentation will be fixed accordingly
The following GET call returns a response with 100 entries as oppose to the requested 150. It seems that any size parameter above 100 is limited to the first 100 entries.
Not sure if a feature or a bug but believe that any meaningful analysis would require more entries.
https://www.consumerfinance.gov/data-research/consumer-complaints/search/api/v1/?size=150&no_aggs=true
R chunck:
url <- 'https://www.consumerfinance.gov/data-research/consumer-complaints/search/api/v1/'
query_params <- list(size = 150, no_aggs = 'true')
response <- GET(url=url, query = query_params)
content <- jsonlite::fromJSON(httr::content(response, "text", encoding = 'UTF-8'))
My goal is to create an R API Client package for the complaint database and this issue limits most of its functionality.
Full API client package implementation can be found https://github.com/joseandresmontes/cfpbR
The first instructions in the setup section of the README are:
This repository assumes that you have an instance of elasticsearch running with complaint data set up and running.
I believe it would help get users off the ground if you could be more specific here, with links to any public data or even the expected schema of the data.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.