mapup / edit-osm-add-missing-bridge-for-truck-restriction

Automate adding bridge information to OSM using JOSM

edit-osm-add-missing-bridge-for-truck-restriction

Introduction

This repository contains Python and JavaScript scripts using which we plan to add missing bridge data to OSM and then add truck restriction information to the bridge data included in OSM. This will be a two-phase process. In the first phase, we will add all the missing bridges. In the second phase, we will add truck restriction data to all the bridges. This repository is currently focused on Phase One.

Setup

Install required Python packages from requirements.txt and install QGIS-LTR, Osmium tool with brew install osmium-tool, and GDAL library with brew install gdal.

Process Overview

For a comprehensive description of the process, read this guide: Overview-Add-missing-bridge-truck-restrictions-to-OSM

Hydrography Approach

1. Download Data:
   • OSM Ways Data: Downloaded from Geofabrik, this project uses updated OSM data extracts for Kentucky.
     • Data link: Kentucky-Latest.osm.pbf
   • NBI Bridge Dataset: Obtained from the Federal Highway Administration, containing detailed information on bridges and tunnels across the USA.
     • Data link: Kentucky-NBI-bridge-data.csv
   • National Hydrography Dataset (NHD): Provides essential water feature details for accurate bridge associations.
     • Data link: NHD-Kentucky-Streams-Flowline.gpkg
2. Filter & Process Data: Within the filter_data folder, we have two scripts that filter relevant OSM data and NBI bridge data.
   • Output:
     • Kentucky-filtered-highways.gpkg
     • Kentucky-NBI-bridges.gpkg
3. Tag Data: To ensure precise associations between NBI bridges and relevant OSM ways, data tagging processes are implemented within the tag_data folder:
   • Filter out bridges already in OSM data, near freeway interchanges, or near tunnel=culvert OSM ways.
   • Tag OSM ways and NBI bridges with nearby NHD streams and OSM ways.
   • Outputs:
     • Intersections among OSM ways and NHD streams: OSM-NHD-Intersections.csv
     • OSM ways data tagged with relevant NHD stream data: OSM-NHD-Join.csv
     • NBI bridge data tagged with nearby OSM ways: NBI-30-OSM-NHD-Join.csv
4. Associate Data: Within the associate_data folder, we have scripts that perform the following steps:
   • Create associations among NBI-OSM and OSM-NHD data, linking NBI bridges, OSM ways, and NHD water streams.
   • Determine final OSM ways for NBI bridges based on specified conditions and bridge attributes.
   • Project final coordinates of NBI bridges onto associated OSM ways.
   • Calculate match percentages for associated OSM ways' road names.
   • Remove nearby bridges within 30m based on fuzzy match scores.
   • Output: Final-bridges-with-percentage-match.csv

Mile-point Approach

The milepoint approach automates the addition of bridges to OpenStreetMap (OSM) using detailed Linear Referencing System (LRS) data. By leveraging milepoint information from bridge and road datasets, this method accurately locates and integrates missing bridges into OSM.

1. Integrate Data
   • Merge bridge and road datasets including milepoint information
   • Prepare datasets by cleaning and standardizing column names.
2. Associate bridge and road data
   • Associate bridges with road segments based on milepoint ranges.
   • Filter and process left-right lane bridges for accurate representation.
3. Interpolate bridges on OSM ways
   • Calculate precise bridge positions along road geometries using milepoint data.
   • Improve accuracy through fuzzy matching of road names and bridge descriptions.
4. Match OSM Data
   • Match interpolated bridge locations with existing OSM road network.
   • Select closest OSM ways to each bridge location for integration.
5. Export Data
   • Generate files containing interpolated bridge points and associated road segments.
6. Output
   • Kentucky-osm-road-points.gpkg

Merging Association Approaches

The merge-approaches folder contains script that merges data from both the NBI-OSM associations approaches and generates a combined output.

• Output: merged-approaches-association-output.csv

Split OSM ways using Python and JOSM

1. Obtain Bridge Coordinates on OSM Ways: Within the obtain_bridge_split_coordinates folder, we have the script that identifies and positions bridge coordinates equidistant from the midpoint along specified OSM ways.
   • Output: bridge-osm-association-with-split-coords.csv
2. Use JOSM to Add Bridge Tags: Within the split_ways_add_bridge_tag folder, we have the following scripts:
   • Add Tags to Bridge Spanning over Single OSM Way:
     • Script: 01-JOSM-1-split-way-in-place.js
     • Utilize the JOSM Scripting Plugin to accurately position bridge locations along existing ways and split ways to incorporate new nodes. This includes adding the "bridge=yes" tag to the identified way.
   • Determine OSM ways covered by bridges which span multiple ways using NetworkX.
     • Script: 02-shortest-route-between-two-ways.py
   • Add Tags to Bridge Spanning over Multiple OSM Ways:
     • Script: 03-JOSM-1-handle-multi-way-bridge.js
     • Using Python libraries Osmium and NetworkX alongside the JOSM Scripting Plugin to update OSM data. This involves finding all OSM way IDs that the bridge spans and ensuring accurate tagging.

Conclusion

This repository provides tools and scripts necessary to enhance OSM bridge data using publicly available datasets. By automating the identification, tagging, and association processes, it aims to improve the accuracy and completeness of bridge information within OpenStreetMap.

edit-osm-add-missing-bridge-for-truck-restriction's Issues

urllib3-2.0.7-py3-none-any.whl: 1 vulnerabilities (highest severity is: 4.4)

Vulnerable Library - urllib3-2.0.7-py3-none-any.whl

HTTP library with thread-safe connection pooling, file post, and more.

Library home page: https://files.pythonhosted.org/packages/d2/b2/b157855192a68541a91ba7b2bbcb91f1b4faa51f8bae38d8005c034be524/urllib3-2.0.7-py3-none-any.whl

Path to dependency file: /requirements.txt

Path to vulnerable library: /requirements.txt

Found in HEAD commit: 201ffccf9994bc4474564734be55f2d52d352c3a

Vulnerabilities

CVE	Severity	CVSS	Dependency	Type	Fixed in (urllib3 version)	Remediation Possible**
CVE-2024-37891	Medium	4.4	urllib3-2.0.7-py3-none-any.whl	Direct	2.2.2	❌

**In some cases, Remediation PR cannot be created automatically for a vulnerability despite the availability of remediation

Details

CVE-2024-37891

Vulnerable Library - urllib3-2.0.7-py3-none-any.whl

HTTP library with thread-safe connection pooling, file post, and more.

Library home page: https://files.pythonhosted.org/packages/d2/b2/b157855192a68541a91ba7b2bbcb91f1b4faa51f8bae38d8005c034be524/urllib3-2.0.7-py3-none-any.whl

Path to dependency file: /requirements.txt

Path to vulnerable library: /requirements.txt

Dependency Hierarchy:

• ❌ urllib3-2.0.7-py3-none-any.whl (Vulnerable Library)

Found in HEAD commit: 201ffccf9994bc4474564734be55f2d52d352c3a

Found in base branch: main

Vulnerability Details

urllib3 is a user-friendly HTTP client library for Python. When using urllib3's proxy support with ProxyManager, the Proxy-Authorization header is only sent to the configured proxy, as expected. However, when sending HTTP requests without using urllib3's proxy support, it's possible to accidentally configure the Proxy-Authorization header even though it won't have any effect as the request is not using a forwarding proxy or a tunneling proxy. In those cases, urllib3 doesn't treat the Proxy-Authorization HTTP header as one carrying authentication material and thus doesn't strip the header on cross-origin redirects. Because this is a highly unlikely scenario, we believe the severity of this vulnerability is low for almost all users. Out of an abundance of caution urllib3 will automatically strip the Proxy-Authorization header during cross-origin redirects to avoid the small chance that users are doing this on accident. Users should use urllib3's proxy support or disable automatic redirects to achieve safe processing of the Proxy-Authorization header, but we still decided to strip the header by default in order to further protect users who aren't using the correct approach. We believe the number of usages affected by this advisory is low. It requires all of the following to be true to be exploited: 1. Setting the Proxy-Authorization header without using urllib3's built-in proxy support. 2. Not disabling HTTP redirects. 3. Either not using an HTTPS origin server or for the proxy or target origin to redirect to a malicious origin. Users are advised to update to either version 1.26.19 or version 2.2.2. Users unable to upgrade may use the Proxy-Authorization header with urllib3's ProxyManager, disable HTTP redirects using redirects=False when sending requests, or not user the Proxy-Authorization header as mitigations.

Publish Date: 2024-06-17

URL: CVE-2024-37891

CVSS 3 Score Details (4.4)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: High
  • Privileges Required: High
  • User Interaction: None
  • Scope: Unchanged
• Impact Metrics:
  • Confidentiality Impact: High
  • Integrity Impact: None
  • Availability Impact: None

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: GHSA-34jh-p97f-mpxf

Release Date: 2024-06-17

Fix Resolution: 2.2.2

Step up your Open Source Security Game with Mend here

numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl: 1 vulnerabilities (highest severity is: 5.3)

Vulnerable Library - numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl

Fundamental package for array computing in Python

Library home page: https://files.pythonhosted.org/packages/6d/ad/ff3b21ebfe79a4d25b4a4f8e5cf9fd44a204adb6b33c09010f566f51027a/numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl

Path to dependency file: /requirements.txt

Path to vulnerable library: /requirements.txt

Found in HEAD commit: 201ffccf9994bc4474564734be55f2d52d352c3a

Vulnerabilities

CVE	Severity	CVSS	Dependency	Type	Fixed in (numpy version)	Remediation Possible**
CVE-2021-34141	Medium	5.3	numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl	Direct	1.22.0	❌

**In some cases, Remediation PR cannot be created automatically for a vulnerability despite the availability of remediation

Details

CVE-2021-34141

Vulnerable Library - numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl

Fundamental package for array computing in Python

Library home page: https://files.pythonhosted.org/packages/6d/ad/ff3b21ebfe79a4d25b4a4f8e5cf9fd44a204adb6b33c09010f566f51027a/numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl

Path to dependency file: /requirements.txt

Path to vulnerable library: /requirements.txt

Dependency Hierarchy:

• ❌ numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (Vulnerable Library)

Found in HEAD commit: 201ffccf9994bc4474564734be55f2d52d352c3a

Found in base branch: main

Vulnerability Details

An incomplete string comparison in the numpy.core component in NumPy before 1.22.0 allows attackers to trigger slightly incorrect copying by constructing specific string objects. NOTE: the vendor states that this reported code behavior is "completely harmless."
Mend Note: After conducting further research, Mend has determined that versions 1.12.0 through 1.21.6 of numpy are vulnerable to CVE-2021-34141

Publish Date: 2021-12-17

URL: CVE-2021-34141

CVSS 3 Score Details (5.3)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: Low
  • Privileges Required: None
  • User Interaction: None
  • Scope: Unchanged
• Impact Metrics:
  • Confidentiality Impact: None
  • Integrity Impact: None
  • Availability Impact: Low

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-34141

Release Date: 2021-12-17

Fix Resolution: 1.22.0

Step up your Open Source Security Game with Mend here

requests-2.31.0-py3-none-any.whl: 1 vulnerabilities (highest severity is: 5.6)

Vulnerable Library - requests-2.31.0-py3-none-any.whl

Python HTTP for Humans.

Library home page: https://files.pythonhosted.org/packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl

Path to dependency file: /requirements.txt

Path to vulnerable library: /requirements.txt

Found in HEAD commit: 201ffccf9994bc4474564734be55f2d52d352c3a

Vulnerabilities

CVE	Severity	CVSS	Dependency	Type	Fixed in (requests version)	Remediation Possible**
CVE-2024-35195	Medium	5.6	requests-2.31.0-py3-none-any.whl	Direct	requests - 2.32.2	❌

**In some cases, Remediation PR cannot be created automatically for a vulnerability despite the availability of remediation

Details

CVE-2024-35195

Vulnerable Library - requests-2.31.0-py3-none-any.whl

Python HTTP for Humans.

Library home page: https://files.pythonhosted.org/packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl

Path to dependency file: /requirements.txt

Path to vulnerable library: /requirements.txt

Dependency Hierarchy:

• ❌ requests-2.31.0-py3-none-any.whl (Vulnerable Library)

Found in HEAD commit: 201ffccf9994bc4474564734be55f2d52d352c3a

Found in base branch: main

Vulnerability Details

Requests is a HTTP library. Prior to 2.32.2, when making requests through a Requests Session, if the first request is made with verify=False to disable cert verification, all subsequent requests to the same host will continue to ignore cert verification regardless of changes to the value of verify. This behavior will continue for the lifecycle of the connection in the connection pool. This vulnerability is fixed in 2.32.2.

Publish Date: 2024-05-20

URL: CVE-2024-35195

CVSS 3 Score Details (5.6)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Local
  • Attack Complexity: High
  • Privileges Required: High
  • User Interaction: Required
  • Scope: Unchanged
• Impact Metrics:
  • Confidentiality Impact: High
  • Integrity Impact: High
  • Availability Impact: None

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: GHSA-9wx4-h78v-vm56

Release Date: 2024-05-20

Fix Resolution: requests - 2.32.2

Step up your Open Source Security Game with Mend here