If you don't manage to scrape the value you are looking for, please enable debug logging and log_response. This will provide you with a lot of information for continued investigation. log_response will write all responses to files. If the value you want to scrape is not in the files with the output from BeautifulSoup (*-soup.txt), Multiscrape will not be able to scrape it. Most likely it is retrieved in the background by javascript. Your best chance in this case, is to investigate the network traffic in de developer tools of your browser, and try to find a json response containing the value you are looking for.
If all of this doesn't help, use the home assistant forum. I cannot give everyone personal assistance and please don't create github issues unless you are sure there is a bug. Check the wiki for a scraping guide and other details on the functionality of this component.
You and you alone, are accountable for your scraping activities. Be a good (web) citizen. Set reasonable scan_interval timings, seek explicit permission before scraping, and adhere to local and international laws. Respect website policies, handle data ethically, mind resource usage, and regularly monitor your actions. Uphold these principles to ensure ethical and sustainable scraping practices.
This Home Assistant custom component can scrape multiple fields (using CSS selectors) from a single HTTP request (the existing scrape sensor can scrape a single field only). The scraped data becomes available in separate sensors.
It is based on both the existing Rest sensor and the Scrape sensor. Most properties of the Rest and Scrape sensor apply.
Install via HACS (default store) or install manually by copying the files in a new 'custom_components/multiscrape' directory.
multiscrape:
- name: HA scraper
resource: https://www.home-assistant.io
scan_interval: 3600
sensor:
- unique_id: ha_latest_version
name: Latest version
select: ".release-date"
value_template: "{{ value | trim }}"
- unique_id: ha_release_date
icon: >-
{% if is_state('binary_sensor.ha_version_check', 'on') %}
mdi:alarm-light
{% else %}
mdi:bat
{% endif %}
name: Release date
select: ".release-date"
attribute: "title"
value_template: "{{ (value.split('released')[1]) }}"
binary_sensor:
- unique_id: ha_version_check
name: Latest version == 2021.7.0
select: ".release-date"
value_template: '{{ value | trim == "2021.7.0" }}'
attributes:
- name: Release notes link
select: ".release-date"
attribute: hrefBased on latest (pre) release.
| name | description | required | default | type |
|---|---|---|---|---|
| name | The name for the integration. | False | string | |
| resource | The url for retrieving the site or a template that will output an url. Not required when resource_template is provided. |
True | string | |
| resource_template | A template that will output an url after being rendered. Only required when resource is not provided. |
True | template | |
| authentication | Configure HTTP authentication. basic or digest. Use this with username and password fields. |
False | string | |
| username | The username for accessing the url. | False | string | |
| password | The password for accessing the url. | False | string | |
| headers | The headers for the requests. | False | template - list | |
| params | The query params for the requests. | False | template - list | |
| method | The method for the request. Either POST or GET. |
False | GET | string |
| payload | Optional payload to send with a POST request. | False | string | |
| verify_ssl | Verify the SSL certificate of the endpoint. | False | True | boolean |
| log_response | Log the HTTP responses and HTML parsed by BeautifulSoup in files. (Will be written to/config/multiscrape/name_of_config) | False | False | boolean |
| timeout | Defines max time to wait data from the endpoint. | False | 10 | int |
| scan_interval | Determines how often the url will be requested. | False | 60 | int |
| parser | Determines the parser to be used with beautifulsoup. Either lxml or html.parser. |
False | lxml | string |
| list_separator | Separator to be used in combination with select_list features. |
False | , | string |
| form_submit | See Form-submit | False | ||
| sensor | See Sensor | False | list | |
| binary_sensor | See Binary sensor | False | list | |
| button | See Refresh button | False | list |
Configure the sensors that will scrape the data.
| name | description | required | default | type |
|---|---|---|---|---|
| unique_id | Will be used as entity_id and enables editing the entity in the UI | False | string | |
| name | Friendly name for the sensor | False | string | |
| See Selector fields | True | |||
| attributes | See Sensor attributes | False | list | |
| unit_of_measurement | Defines the units of measurement of the sensor | False | string | |
| device_class | Sets the device_class for sensors or binary sensors | False | string | |
| state_class | Defines the state class of the sensor, if any. (measurement, total or total_increasing) (not for binary_sensor) | False | None | string |
| icon | Defines the icon or a template for the icon of the sensor. The value of the selector (or value_template when given) is provided as input for the template. For binary sensors, the value is parsed in a boolean. | False | string/template | |
| picture | Contains a path to a local image and will set it as entity picture | False | string | |
| force_update | Sends update events even if the value hasn’t changed. Useful if you want to have meaningful value graphs in history. | False | False | boolean |
Configure a refresh button to manually trigger scraping.
| name | description | required | default | type |
|---|---|---|---|---|
| unique_id | Will be used as entity_id and enables editing the entity in the UI | False | string | |
| name | Friendly name for the button | False | string |
Configure the attributes on the sensor that can be set with additional scraping values.
| name | description | required | default | type |
|---|---|---|---|---|
| name | Name of the attribute (will be slugified) | True | string | |
| See Selector fields | True |
Configure the form-submit functionality which enables you to submit a (login) form before scraping a site. More details on how this works can be found on the wiki.
| name | description | required | default | type |
|---|---|---|---|---|
| resource | The url for the site with the form | False | string | |
| select | CSS selector used for selecting the form in the html. When omitted, the input fields are directly posted. | False | string | |
| input | A dictionary with name/values which will be merged with the input fields on the form | False | string - dictionary | |
| input_filter | A list of input fields that should not be submitted with the form | False | string - list | |
| submit_once | Submit the form only once on startup instead of each scan interval | False | False | boolean |
| resubmit_on_error | Resubmit the form after a scraping error is encountered | False | True | boolean |
| variables | See Form Variables | False | list |
Configure the variables that will be scraped from the form_submit response. These variables can be used in the value_template of the main configuration of the current integration: a selector in sensors/attributes or in a header. A common use case is to populate the X-Login-Token header which is the result of the login.
| name | description | required | default | type |
|---|---|---|---|---|
| name | Name of the variable | True | string | |
| See Selector fields | True |
Example:
multiscrape:
- resource: "https://somesiteyouwanttoscrape.com"
form_submit:
submit_once: True
resource: "https://authforsomesiteyouwanttoscrape.com"
input:
email: "<email>"
password: "<password>"
variables:
- name: token
value_template: "{{ ... }}"
headers:
X-Login-Token: "{{ token }}"
sensor: ...Used to configure scraping options.
| name | description | required | default | type |
|---|---|---|---|---|
| select | CSS selector used for retrieving the value of the attribute. Only required when select_list or value_template is not provided. |
False | string/template | |
| select_list | CSS selector for multiple values of multiple elements which will be returned as csv. Only required when select or value_template is not provided. |
False | string/template | |
| attribute | Attribute from the selected element to read as value. | False | string | |
| value_template | Defines a template applied to extract the value from the result of the selector (if provided) or raw page (if selector not provided) | False | string/template | |
| on_error | See On-error | False |
Configure what should happen in case of a scraping error (the css selector does not return a value).
| name | description | required | default | type |
|---|---|---|---|---|
| log | Determines if and how something should be logged in case of a scraping error. Value can be either 'false', 'info', 'warning' or 'error'. | False | error | string |
| value | Determines what value the sensor/attribute should get in case of a scraping error. The value can be 'last' meaning that the value does not change, 'none' which results in HA showing 'Unkown' on the sensor, or 'default' which will show the specified default value. | False | none | string |
| default | The default value to be used when the on-error value is set to 'default'. | False | string |
For each multiscrape instance, a service will be created to trigger a scrape run through an automation. (For manual triggering, the button entity can now be configured.)
The services are named multiscrape.trigger_{name of integration}.
Multiscrape also offers a get_content and a scrape service. get_content retrieves the content of the website you want to scrape. It shows the same data for which you now need to enable log_response and open the page_soup.txt file.
scrape does what it says. It scrapes a website and provides the sensors and attributes.
Both services accept the same configuration as what you would provide in your configuration yaml (what is described above), with a small but important caveat: if the service input contains templates, those are automatically parsed by home assistant when the service is being called. That is fine for templates like resource and select, but templates that need to be applied on the scraped data itself (like value_template), cannot be parsed when the service is called. Therefore you need to slightly alter the syntax and add a ! in the middle. E.g. {{ becomes {!{ and %} becomes %!}. Multiscrape will then understand that this string needs to handled as a template after the service has been called.
If someone has a better solution, please let me know!
To call one of those services, go to 'Developer tools' in Home Assistant and then to 'services'. Find the multiscrape.get_content or multiscrape.scrape services and go to yaml mode. There you enter your configuration.
Example:
service: multiscrape.scrape
data:
name: HA scraper
resource: https://www.home-assistant.io
sensor:
- unique_id: ha_latest_version
name: Latest version
select: ".release-date"
value_template: "{!{ value | trim }!}"
- unique_id: ha_release_date
name: Release date
select: ".release-date"
attribute: "title"
value_template: "{!{ (value.split('released')[1]) }!}"Debug logging can be enabled as follows:
logger:
default: info
logs:
custom_components.multiscrape: debugDepending on your issue, also consider enabling log_response.
If you want to contribute to this please read the Contribution guidelines
This project was generated from @oncleben31's Home Assistant Custom Component Cookiecutter template.
Code template was mainly taken from @Ludeeus's integration_blueprint template
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
![renovate[bot] avatar](https://avatars.githubusercontent.com/in/2740?v=4 "renovate[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent