Ambianic Edge turns your Raspberry Pi into a smart camera device.
Read the docs to learn how to run, configure and develop with Ambianic Edge.
Python ML library for people fall detection
License: Apache License 2.0
Ambianic Edge turns your Raspberry Pi into a smart camera device.
Read the docs to learn how to run, configure and develop with Ambianic Edge.
Is your feature request related to a problem? Please describe.
We continuously receive images from users with examples of false positives or missed positives. This is expected iterative process of improvement.
Describe the solution you'd like
We should add to the testsuite examples from publicly available elderly fall videos in order to improve the performance of our fall detection algorithm.
Additional context
Attached are additional images reported from ambianic users with false positives.
Is your feature request related to a problem? Please describe.
Currently save_detections expects a bounding box for detections. This is the case for object detection and face detection, but not for pose detection and fall detection.
Describe the solution you'd like
Enhance detection event schema for storage, notifications and UI display.
Describe alternatives you've considered
Currently fall_detection is forced to output a fake bounding box to fit the expected format, but that's not optimal.
Is your feature request related to a problem? Please describe.
Currently our pose detection API supports multiple models, which is great.
Describe the solution you'd like
It would be nice to align with the TFJS version to make it easier for a broader community of ML researchers to participate in the fall detection project.
Additional context
Is your feature request related to a problem? Please describe.
We currently have a notebook that shows how PoseNet and MoveNet perform on various fall data.
https://github.com/ambianic/fall-detection/blob/main/MoveNet_Vs_PoseNet.ipynb
We also have a testsuite that establishes a baseline for fall detection performance.
https://github.com/ambianic/fall-detection/tree/main/tests
However we don't have an interactive representation of data samples which demonstrates the cases where fall detection is accurate and where it fails.
Describe the solution you'd like
It would be helpful to have an interactive notebook that provides examples of fall detection performance in the various cases of challenges that we've discovered and documented in this blog post:
https://blog.ambianic.ai/2021/09/02/movenet-vs-posenet-person-fall-detection.html
Respectively we should label each sample in the interactive data set as True/False in order to begin establishing best practices for fall detection labeling that other researches can use to contribute additional data training and test data. Ultimately the goal is to use the accumulated community contributed data for a new fall detection DNN classification model.
For each known category with challenges we should have around 10 examples with positive and negative labeled detections.
Describe the bug
It appears that the newly merged PR ambianic/ambianic-edge#282 has tests that fail intermittently in CI on ARM CPU.
___________________ test_fall_detection_2_frame_back_case_2 ____________________
def test_fall_detection_2_frame_back_case_2():
"""
Expected to detect a fall using frame[t] and frame[t-2].
frame[t-2] : A person is in standing position.
frame[t-1] : A person is mid-way of fall.
frame[t] : A person is fall down.
"""
config = _fall_detect_config()
result = None
def sample_callback(image=None, inference_result=None, **kwargs):
nonlocal result
result = inference_result
fall_detector = FallDetector(**config)
output = _OutPipeElement(sample_callback=sample_callback)
fall_detector.connect_to_next_element(output)
# A frame at t-2 timestamp when person is in standing position.
img_1 = _get_image(file_name='fall_img_1.png')
# A frame at t-1 timestamp when person is mid-way of fall.
img_2 = _get_image(file_name='fall_img_2_2.png')
# A frame at t timestamp when person falls down.
img_3 = _get_image(file_name='fall_img_2.png')
fall_detector.receive_next_sample(image=img_1)
fall_detector.min_time_between_frames = 0.01
time.sleep(fall_detector.min_time_between_frames)
fall_detector.receive_next_sample(image=img_2)
fall_detector.min_time_between_frames = 0.01
time.sleep(fall_detector.min_time_between_frames)
assert not result
fall_detector.receive_next_sample(image=img_3)
> assert result
E assert []
To Reproduce
https://github.com/ambianic/ambianic-edge/runs/1810906695?check_suite_focus=true#step:3:3921
Expected behavior
Tests should pass reliably in CI.
Host environment (please complete the following information):
Additional context
In the past this has happened a few times usually because the ARM version of TFLite outputs lower confidence scores for inferences than the x86 version.
Describe the bug
An LGTM bot warning revealed a legitimate issue in the code.
Minor issue is unused import:
Import of 'stacktrace' is not used.
A more potentially impactful issue is not using the spinal vector to calculate leaning angles relative to vertical axis and previous body positions.
spinalVectorEstimate = find_spinalLine()
--
| The value assigned to local variable 'spinalVectorEstimate' is never used.
This may not have a significant impact on the heuristic algorithm results, but it does make the code confusing since we are introducing the concept of spinal vector and not making use of it as intended.
Expected behavior
Let's simplify the code and replace redundant angle calculations for left and right side vectors with the single spinal vector estimate.
Pull fall detection code out of ambianic-edge into this standalone repo as an independent python library.
The goal is to allow data scientists to use, collaborate and improve people the fall detection ML model independently of ambianic deployments.
All,
Testing the Edge/UI for the last 24 hours and I have notice that the Edge does not capture a great deal of activity ( me walking in and out of my office ) and I have "fallen" over 30 times, with the Edge capturing only 2 of them. Once on the IF camera at night, ( I spent from 11:00 PM to 2:00 AM staging "falls" in the darkness to test the IF camera. Edge only captured one of the falls. I spent 8 hours today during daylight staging falls, the Edge only capture one falls.
I have walked in and out of the room all day (daylight hours ) and the Edge does not record 50% of them.
There are a lot of captures that are "false positive" nobody in the room.
I have attached a file of the detections. There are 5 different directors. If you have any questions please ping me on slack.
Thanks
Archive.zip
Is your feature request related to a problem? Please describe.
Users report that fall detection triggers false positives when there are multiple persons in view. See attached screenshots.
This is a known issue for single person PoseNet models.
Describe the solution you'd like
A sophisticated solutions would be to implement multiple person tracking in a sequence of video frames.
An interim practical improvement would be to discard frames with multiple persons. This is a reasonable approach because:
For the interim solution we can use a multi person posenet model and simply discard frames when the model detects multiple persons in a frame. That will avoid potential false fall detections. Testing shows that multi-person posenet models take slightly more CPU time for inference than single person models (2-3 fps vs 3-4 fps on rpi4), which is a reasonable tradeoff since we only need 1fps for fall detection.
Describe alternatives you've considered
A sophisticated solutions would be to implement multiple person tracking in a sequence of video frames. Respectively track and detect falls for each individual person as they move between frames.
Additional context
Screenshots from users with false positives below.
Is your feature request related to a problem? Please describe.
Users report that our model detects falls better in well lighted areas than it does in areas with poor lighting conditions. Since falls can occur any time of the day, including night time, it is important to extend the range of lighting conditions we can handle. Night time is arguably more important because it is less likely for a fallen person to be supervised by a family caregiver.
Describe the solution you'd like
It appears that the current PoseNet models have been trained on images with good lighting.
Maybe we can pre-process images so their colors shift closer to daytime lighting range that the PoseNet model can pick up.
One relatively quick and simple approach to try is conventional image filter libraries such as ImageMagick. For example the -auto-level filter appears to have visible effect on dark image illumination while it leaves in place images with good lighting. See example attached of Mac Preview with Auto Levels.
The enhancement should be a particularly fast operation if we apply it only to the 20K (~300x300) thubmnail fed to the DNN model instead of the original HD image.
Here is a discussion on the comparable -auto-level ImageMagick operator.
Wand: Python bindings for ImageMagick.
Describe alternatives you've considered
New Deep neural models are also available for this task. See resources in a related issue: ambianic/ambianic-edge#233
Is your feature request related to a problem? Please describe.
After examining my timeline I noticed that my Ambianic camera reported a "Fall Detection" when I bent over to tie my running shoes.
Running latest release on Edge and UI.PWA
I do multiple "test" falls per day, sometimes per hour. The Edge does not capture the falls 50% of the time. No Frame is recorded in the time line.
Here is the Ambianic Log. It captures me getting up from the fall, but not the actually fall itself.
ambianic-edge | 2021-10-11 15:01:40.094 | INFO | main:join_peer_room:80 - myRoom members {'clientsIds': ['b4166b0d-d8c5-481e-a7db-e3a3590fde11']}
ambianic-edge | 2021-10-11 15:01:40 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(183): FallDetector inference time 833.72 ms,
0.87 fps in pipeline area_watch
ambianic-edge | 2021-10-11 15:01:41 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(183): ObjectDetector inference time 313.15 ms, 0.87 fps in pipeline area_watch
ambianic-edge | 2021-10-11 15:01:41 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/object_detect.py.convert_inference_result(53): label: person , confidence: 1, box: (0.33572521805763245, 0.5026041756014852, 0.631360650062561, 0.9994138274672468)
ambianic-edge | 2021-10-11 15:01:41 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/fall_detect.py.find_keypoints(224): A pose detected with
ambianic-edge | spinal_vector_score=0.29140514398636597 >= 0.25
ambianic-edge | confidence threshold.
ambianic-edge | Pose keypoints: {'right shoulder': [141.0, 103.0], 'right hip': [136.0, 145.0]}"
ambianic-edge |
ambianic-edge | 2021-10-11 15:01:41 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(183): FallDetector inference time 279.00 ms, 1.68 fps in pipeline area_watch
ambianic-edge | 2021-10-11 15:01:41 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(183): ObjectDetector inference time 310.77 ms, 1.68 fps in pipeline area_watch
ambianic-edge | 2021-10-11 15:01:41 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/object_detect.py.convert_inference_result(53): label: person , confidence: 1, box: (0.4862857460975647, 0.5147159099578857, 0.6158604025840759, 0.9832253470223331)
ambianic-edge | 2021-10-11 15:01:41 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(183): FallDetector inference time 0.02 ms, 3.17 fps in pipeline area_watch
ambianic-edge | 2021-10-11 15:01:41 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(183): ObjectDetector inference time 310.47 ms, 3.18 fps in pipeline area_watch
ambianic-edge | 2021-10-11 15:01:41 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(183): FallDetector inference time 0.02 ms, 3.19 fps in pipeline area_watch
ambianic-edge | 2021-10-11 15:01:42 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(183): ObjectDetector inference time 310.79 ms, 3.18 fps in pipeline area_watch
ambianic-edge | 2021-10-11 15:01:43 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(183): FallDetector inference time 815.36 ms, 0.88 fps in pipeline area_watch
ambianic-edge | 2021-10-11 15:01:43.200 | INFO | __
20211011-154321.251638-inference.json.txt
m
UPDATE: June 11, 2021
Is your feature request related to a problem? Please describe.
Currently we use mobilnetv2 with a 300x300 input tensor (image) by default for object detection.
It struggles with poses of people laying down on the floor. We experimented with rotating images +/-90', which improves overall detection rates, but it still misses poses of fallen people, even when the full body is clearly visible by a human eye.
Clearly the model has not been trained on fallen people poses.
Describe the solution you'd like
Google AI introduced MoveNet on May 17, 2021:
30fps on a mobile phone. Initially for TensorflowJS with a follow up model release coming to TFLite.
Google AI released PoseNet 2.0 with a Resnet50 in 2020 base model which has 5-6fps performance on desktop CPU and noticeably better detection rates. Interactive web demo here. However testing shows that even with these improvements, it still misses some poses of people laying down (fallen poses) that are otherwise easy for a human eye to recognize. See an example recorded video below that provides a reference for situations when resnet misses poses.
Google AI MediaPipe released a new iteration of BlazePose, which detects 33 (vs 15) keypoints at 25-55fps on desktop CPU (5-10 times faster than PoseNet 2 ResNet50). Testing shows that blazepose does a better job with horizontal people poses, although it still misses some laying positions. See attached video for reference. BlazePose interactive web demo here. Pose detection TFLite model here.
Additional context
See TensorFlow 2 Detection Model Zoo
https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf2_detection_zoo.md
Notice the high performance and dual purpose (object + keypoints) for CenterNet Resnet50 V1 FPN Keypoints 512x512 and CenterNet Resnet50 V2 Keypoints 512x512.
More on CenterNet and its various applications for object detection, posedetection and object motion tracking.
https://github.com/xingyizhou/CenterNet
Is your feature request related to a problem? Please describe.
Looking at Pypi and searching on google, I could not find a python version of the TFJS pose-detection API.
There are several pose estimation models (tf-pose) on github and pypi that are not optimized for running tflite on edge devices (such as RPI) with CPU only (no GPU) and do not seem to be actively maintained.
Describe the solution you'd like
Let's extract the pose detection part into its own repo and pypi package and offer it to the broader community for collaboration.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Is your feature request related to a problem? Please describe.
We currently use PoseNet 2.0 as our default pose detection model.
However benchmarking has shown that MoveNet performs better:
https://blog.ambianic.ai/2021/09/02/movenet-vs-posenet-person-fall-detection.html
Describe the solution you'd like
Keep PoseNet as an alternative option that can be configured for use, but switch default model to MoveNet.
Describe the bug
Pose engine throws intermittent runtime exceptions.
To Reproduce
Run pose engine through a variety of fall images.
Expected behavior
There should be no unexpected runtime exceptions caused by image processing during fall detections.
Host environment (please complete the following information):
Additional context
2021-02-07 13:09:02 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1077.19 ms, 0.93 fps in pipeline area_watch
2021-02-07 13:09:02 ERROR /opt/ambianic-edge/src/ambianic/pipeline/ai/fall_detect.py.process_sample(101): Error "only size-1 arrays can be converted to Python scalars" while processing sample. Dropping sample: {'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=1280x720 at 0xADB66570>}
Traceback (most recent call last):
File "/opt/ambianic-edge/src/ambianic/pipeline/ai/fall_detect.py", line 85, in process_sample
inference_result, thumbnail = self.fall_detect(image=image)
File "/opt/ambianic-edge/src/ambianic/pipeline/ai/fall_detect.py", line 344, in fall_detect
pose, thumbnail, pose_score, pose_dix = self.find_keypoints(image)
File "/opt/ambianic-edge/src/ambianic/pipeline/ai/fall_detect.py", line 146, in find_keypoints
poses, thumbnail = self._pose_engine.DetectPosesInImage(image)
File "/opt/ambianic-edge/src/ambianic/pipeline/ai/pose_engine.py", line 146, in DetectPosesInImage
kps = self.parse_output(template_heatmaps, template_offsets, 0.3)
File "/opt/ambianic-edge/src/ambianic/pipeline/ai/pose_engine.py", line 89, in parse_output
pose_kps[i, 0] = int(remap_pos[0] + offset_data[max_val_pos[0], max_val_pos[1], i])
TypeError: only size-1 arrays can be converted to Python scalars
2021-02-07 13:09:03 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1070.10 ms, 0.70 fps in pipeline area_watch
2021-02-07 13:09:04 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1003.87 ms, 0.99 fps in pipeline area_watch
2021-02-07 13:09:05 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1042.34 ms, 0.96 fps in pipeline area_watch
Is your feature request related to a problem? Please describe.
Currently we use a simple heuristic that takes a before and after images from a fall sequence and compares the angle between the spinal vectors of the poses of the person in each image.
Sometimes this results in false positives , for example when someone is leaning to tie shoes or squatting.
Describe the solution you'd like
We can use the simple heuristic model as a baseline and begin training a pose classification model to gradually reduce false positives. When a fall is detected with the current model, the user can be given an option to agree or disagree based on their own assessment of the before and after images. See attached diagram. The user can vote for each before and after image whether they agree or disagree with the estimation about the pose type (non-fallen person vs fallen person). In case they disagree, the use can choose one of two options, either pick the correct kind of pose (fallen vs non-fallen) or select Neither (not a person in the image or its hard to tell what the pose is - may only the head and shoulders are visible).
Once the end user provides feedback via PWA UI, the labeling data can be fed back to the Ambianic Edge for local training (potentially via TFLite Model Maker for image classification).
The custom pose classification model can be a type of image classification which is applied on black and white images that represent a person pose stick figure in white lines on a solid black background.
A further improvement on local model training, could be federated learning (with user opt-in) applied across all ambianic edge devices.
UPDATE, Aug 15, 2021: There is a good new tutorial on implementing a simple pose classification DNN that takes MoveNet key points as input. The example is for yoga poses. We can instead classify two labels: 'fallen person'/ 'not a fallen person'. We also need it done on TFLite Model Maker instead of TF/Keras.
Currently TFLite Model Maker does not support classification based on sparse inputs such as pose keypoints, but we can try using the image classification API as described above. Draw pose keypoints as simple black and white stickfigure images and classify those with fall/no-fall labels.
Describe alternatives you've considered
A more sophisticated approach will also improve missed fall detections. That requires further research since we want to keep user label effort to a minimum. Labeling missed detections means gathering user feedback on data samples which are not automatically detected. One potential approach is to occasionally ask user for feedback on fall detections with low confidence score.
Additional context
Flow diagram for labeling UX attached below.
Describe the bug
Currently the fall detection code saves debug images even if the log level is not DEBUG.
According to the python log level table, if the log level is equal or lower to DEBUG, the following if statement shall be true.
However it appears that the log level specified in the ambianic config.yaml file does not trickle down to the package level logger.
To Reproduce
Run a fall detection test case that initializes the config at INFO. You will see that temporary images are written down to the file system.
Expected behavior
Debug images should be only written to the file system if the log level is DEBUG or finer grained. Not when the log level is INFO, WARNING or above.
This bug needs to be fixed and covered with test cases for the CI.
Looking at the ambianic edge logs in a real world usage, there is a constant stream of attempts to detect a pose in the original image and +/-90' rotations. This happens because most of the time there is no person in the camera view at all.
This is a suboptimal 3x use of CPU. Normally a single posenet pass on Raspberry Pi takes about 300ms (3fps). However after 2 rotations the total inference time goes up to 1-1.2sec (0.8-1fps). See log excerpt below:
ambianic-edge | 2021-02-04 01:17:45 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1015.96 ms, 0.98 fps in pipeline area_watch
ambianic-edge | 2021-02-04 01:17:46 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1094.21 ms, 0.91 fps in pipeline area_watch
ambianic-edge | 2021-02-04 01:17:47 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1069.70 ms, 0.93 fps in pipeline area_watch
ambianic-edge | 2021-02-04 01:17:48 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1163.64 ms, 0.86 fps in pipeline area_watch
ambianic-edge | 2021-02-04 01:17:49 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1024.56 ms, 0.97 fps in pipeline area_watch
ambianic-edge | 2021-02-04 01:17:50 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1090.12 ms, 0.92 fps in pipeline area_watch
ambianic-edge | 2021-02-04 01:17:51 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1089.76 ms, 0.92 fps in pipeline area_watch
ambianic-edge | 2021-02-04 01:17:52 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1223.01 ms, 0.82 fps in pipeline area_watch
ambianic-edge | 2021-02-04 01:17:54 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1116.52 ms, 0.89 fps in pipeline area_watch
ambianic-edge | 2021-02-04 01:17:55 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1173.54 ms, 0.85 fps in pipeline area_watch
ambianic-edge | 2021-02-04 01:17:56 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1366.06 ms, 0.73 fps in pipeline area_watch
ambianic-edge | 2021-02-04 01:17:57 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1145.92 ms, 0.87 fps in pipeline area_watch
ambianic-edge | 2021-02-04 01:17:58 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1139.79 ms, 0.87 fps in pipeline area_watch
ambianic-edge | 2021-02-04 01:17:59 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1051.62 ms, 0.95 fps in pipeline area_watch
ambianic-edge | 2021-02-04 01:18:01 INFO /opt/ambianic-edge/src/ambianic/pipeline/ai/tf_detect.py.log_stats(178): FallDetector inference time 1107.74 ms, 0.90 fps in pipeline area_watch
Describe the bug
Currently all fall detection code is under a pipeline
package which is a legacy leftover from the ambianic-edge code. This is noisy and not helping with code readability.
Expected behavior
Clean up code and tests naming so it makes sense to a developer using it as an independent library outside of ambianic edge.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.