- NCNN Python
Table of contents generated with markdown-toc
This is a collection of python programs using the NCNN framework. NCNN is an open source implementation of Convolutional Neural Networks by TENCENT. NCNN is optimized for mobile platforms; often it is faster to run the model on CPU than GPU.
Code was converted to Python and accelerated with numpy and cython. A basic framework to handle models, objects, bounding boxes, keypoints and transformations was developed. Python versions for Non Maximum Suppression and image manipulation was implemented. In general developing code in python does not accelerate existing C programs. The motivation for this work was to provide optimized examples for the Python platform.
There are several sites listing current implementations of CNN models that have been converted to NCNN. However, there is no single authoritative repository:
To run the program you will need to install the following packages:
- NCNN
- numpy
- opencv
- cython is suggested
ncnn pip install ncnn
camera pip install camera-util
opencv pip install opencv-contrib-python
sudo pip3 install opencv-contrib-python==4.5.3.56
As time progresses the version number might need to be increased, but many newer version have installation issues.
To increase performance, the following rules were observed:
- Use OpenCV based algorithms whenever possible
- OpenCV data manipulations are faster than Numpy
- OpenCV is slightly faster than NCCN when manipulating images (e.g. resize)
- Avoid indexing over tensor/matrix dimensions
- Use Numpy boolean indexing when possible, avoid
np.where
np.append
,np.vstack
ornp.stack
should be avoided when adding small amounts of data and regular lists and append function can be used- If
np.concatenate
is needed, apply it once to a list of all objects that need concatenation - If
np.max
andnp.argmax
are needed for a 3D array use examples shown below
Torch dependencies were removed. A useful guide is listed here: torch to numpy
Numpy does not provide a function that provides both the maximum and its indices in a data matrix. Its necessary to rearrange the matrix, find the maximum location and then convert it back to indices. Often maximum is needed to threshold and indices are needed for further location calculations.
Often needed to threshold score and find keypoints or bounding boxes.
out2D = out3D.reshape(out3D.shape[0],-1) # convert n,m,o array to n,m*o
idx = out2D.argmax(1) # find max location in m*o range for each n
max_y,max_x = np.unravel_index(idx,out3D.shape[1:]) # unravel the location to m,o coordinates
max_out = out2D[np.arange(len(idx)),idx] # obtain max
The location of the maxima along axis is needed when anchors are used in object detection.
k = np.argmax(out3D,axis=0) # max class score location along n axis
n,o = out3D.shape[1:] #
I,J = np.ogrid[:n,:o] # there is max at each location of m,o
class_score = out_3D[k, I, J] # max class score
Implementation | Author | Website | Article | Image Size | Implementation | Extraction [ms] | Pipeline [ms] |
---|---|---|---|---|---|---|---|
Object Detection | |||||||
fastestdet | Xueha Ma | FastestDet | Artcile | 352x352 | anchor-free | 8.5 | 11.4 |
yolo5s | QEngineering | 640x640 | TBD | TBD | |||
yolo7-tiny | Xiang Wu | Original QEngineering | Article | 640, base=64 | variable input size | 40 | 51 |
yolo8 | TBD | TBD | |||||
yolox (nano) | Megvii-BaseDetection & FeiGeChuanShu | Original QEngineering Android | 416x416 | ||||
yolox (tiny) | Original QEngineering Android | 416x416 | TBD | TBD | |||
yolox (small) | Original QEngineering Android | 640x640 | TBD | TBD | |||
Hand Detector | |||||||
blaze (palm-lite/full) Hand model | Vidur Satija (blazepalm), FeiGeChuanShu (android) | blazepalm Android | 192x192 | 8.09(full) 7.1(light) | 8.9(full) | ||
nanodet (nanodet-hand) Hand model | FeiGeChuanShu QEngineering | ||||||
yolox (yolox_hand_relu/swish) Hand model | FeiGeChuanShu QEngineering | ||||||
pfld (handpose) | FeiGeChuanShu QEngineering | ||||||
Hand Skeleton | |||||||
mediapipe (hand-lite/full-op) Skeleton model | Vidur Satija (blazepalm), FeiGeChuanShu (android) | blazepalm Android | 224x224 | 3D | 6.8 (full) 5.1(light) | 7.1 (full) | |
Face Detector | |||||||
blazeface | |||||||
retinaface retinaface-R50 | Tencent | Tencent QEngineering | Origin | variable, 320x320 | no scaling or padding needed | 3.6 | 7 |
scrfd | |||||||
ultraface | |||||||
Face Detector Support | |||||||
live | yuan hao / Minivision_AI | Website | 80x80 | dual stage | 3.5 | 5.3 | |
mask | |||||||
Face Recognition | |||||||
arcface mobilefacenet | Xinghao Chen | QEngineering | Origin | 112x112 | no NMS, anchorfree | 6 | 6.4 |
Person Detector | |||||||
ultralightpose, Person | Xueha Ma | MobileNetV2-YOLOv3-Nano | 320x320 | no NMS, anchorfree | 8.2 | 9.4 | |
blazeperson - [x] | Valentin Bazarevsky / Google | Android Blazepose Google | Paper | 224x224 | 9.4 | 11 | |
Person Pose | |||||||
ultralightpose, Skeleton | Xueha Ma | UltralightPose | 192x256 | no NMS, anchorfree | 5.84 | 6.4 | |
blazepose - [x] | Valentin Bazarevsky / Google | Android Blazepose DepthAI Blazepose | Paper | 256x256 | 3D | 6.3ms(light), 7.9ms(full), 22ms(heavy) | 6.6ms (light) |
Name | Anchors | Application of Anchors | NMS | Softmax | Sigmoid, Tanh |
---|---|---|---|---|---|
age | not available | ||||
race | not available | ||||
arcface | no anchors | none | no | none | none |
blazeface | detectstride8,16 anchors | gnerate_proposals | NMS | none | sigmoid |
blazehand hand | anchors 8,16,16,16 | none | NMS | warp | sigmoid |
blazehand skelet | uses mediapipe hand | ||||
blazepose person | anchors 8,16,32,32,32 | decode_boxes | NMS | warp | sigmoid |
blazepose skelet | no anchors | none | no | unwarp | none |
blur | computes high frequency content in image | ||||
fastestdet | no anchors | none | NSM | none | sigmoid, tanh |
handpose hand | detectstride8,16,32 anchors | generate_proposals | NMS | softmax | none |
handpose skelet | no anchors | none | no | none | none |
live | 2 models | average of confidence of both models | |||
mask | not implemented yet | ||||
mediapipehandpose skeleton | no anchors | none | none | none | none |
retinaface | anchors, 8,16,32 | generateproposal | NMS | none | exp |
scrfd, 9 different models | anchors 8,16,32 | genreateproposal | NMS | none | none |
ultraface | anchors 8,16,32,64 | generateBBOX | NMS | none | none |
ultralightpose person | no anchors | none | no | none | none |
ultralightpose skelet | no anchors | none | no | none | none |
yolo5 | not implemented yet | ||||
yolo7 | detectstride8,16,32 anchors | detect_stride | NMS | sigmoid | |
yolo8 | grid_strides table | generate_proposal | NMS | softmax | sigmoid |
yolox | not implemented yet |
- test_arcface: runs retinaface to find faces then extracts ROI and applies arcface to compute embeddings
- test_blazehandpose: uses blaze palm to find palms then runs handpose to find sceleton
- test_balzeperson: finds people
- test_blazepersonpose: finds people then runs skeleton detection
- test_blur: extracts ROI and assess if image is blurred
- test_fastestdet
- test_gestures: detects palm then extracts ROI, calculates sceleton then inteprets hand sign
- test_handpose: finds palms then computes skeleton
- test_live: determines if image of face is live or fake
- test_retinaface: detects faces
- test_yolo7: detects objects
2023 - Initial Release
Urs Utzinger
Documentation of utility functions:
objectTypes = {'rect':0, 'yolo80':1, 'hand':2, 'palm7':3, 'hand21':4, 'face':5,
'face5':6, 'person':7, 'person4':8, 'person17':9, 'person39':10 }
Simple objects with bounding box: rect, hand, face, person
Object with keypoints, plam7, hand21, face5, person4, person17, person39, where number indicates the number of keypoints.
Ojects: yolo80 (80 classes)
object.type=objectTypes['rect'], # Object Type
object.bb = np.array( [ # Bounding Box, 4 or 2 points
[ [-1, -1] ],
[ [-1, -1] ],
[ [-1, -1] ],
[ [-1, -1] ]
], dtype=np.float32 )
object.p = -1. # Probability
object.l = -1 # Label number
object.k = [] # Keypoints
object.v = [] # Keypoints visibility
True or False:
- hasKeypoints, hasVisibility, isRotated, is1D, is2D, is3D
Regular:
-
extent: max-min of bounding box
-
center: center of bounding box
-
width_height: on rotated bounding box
-
relative2absolute: scale from 0..1 to 0.. width/height
-
transform: apply cv2.transfrom to bounding box and keypoints
-
intransform: inverse transform
-
resize: resize and shift bounding box
-
square: ensure square bounding box, takes largest dimension
-
angle: angle of keypoints face5, palm7, person4, person17, person 39
-
rotateBoundingBox: rotate rectangualr bounding box by angle
-
draw: draw the bounding boxe and keypoints
-
drawRect: draw bounding box
-
printText: prints text to top left corner of bounding box
-
drawObjects: draw multiple objects
-
calculateBox: phased out
- get_object_scale
- apply
- get_alpha
- reset
- get_alpha
- apply
- apply
- apply_with_alpha
- has_last_raw_value
- last_raw_value
- last_value
- reset
- resizeImage to new width and new height, padding optional
- resizeImage2TargetSize so that width or height is targetsize and pad so that width or height is multiple of base
- extractObjectROI extract image from any bounding box and scale to targetsize
- extractRectROI, extract image from un-roated bounding box
- gesture of handsceleton will select none, point, swear, thumbs up,down,left,right, vulcan, oath, paper, rock, victory, finger, hook, pinky, one, two, three, four, ok
- overlaymatch places found face on top of face
-
nms_cv, non maximum supppresion, filters bounding boxes based on overlap, uses openCV nms, fastest approach
-
nms, simple, fully python based
-
nms_combination, likely not needed
-
nms_weighted, uses weighted approach for overlapping bounding boxes
-
nms_blaze original python code for blaze, includes mathematical explanations
-
matchEmbeddings, given one embedding and comparing it to a list of embedings, finds the one closest matching
-
Zscore (data - mean(data)) / std(data)
-
CosineDistance between two embeddings
-
EuclidianDistnace between two embeddings
-
l2normalize x/sqrt(sum(x*x)), try cv2.norm instead
-
findThreshold, might be obsolete
- decode_boxes
- Anchor object
- Anchor Params
- generate_anchors
- generate_anchors_np
- composeAffine (T,R,Z,S) creates transformation matrix from translation, rotation, zoom/scale, shear
- decomposeAffine23 converts transformation matrix back to T,R,Z,S
- decomposeAffine44 converst transfomration matrix to T,R,Z,S
Instruction to cythonize subroutines
Region of interested can be extracted from images using affine or warped transformation.
-
Affine transformation inlucdes scaling, rotation, translation and shear. The transformation uses a 2x3 Matrix and preserves the parallelism of lines.
-
Warped transformation includes perspective and distortion.
trans_mat = cv2.getAffineTransform(srcPts, dstPts)
img = cv2.warpAffine(src=img, M=trans_mat)
src_pt = np.array([[(x, y)]], dtype=np.float32)
dst_pt = cv2.transform(src_pt,trans_mat)
R = [ [a11, a12, t13]
[a21, a22, t23]
[ 0, 0, 1] ]
dst = [ A | t ] {src}
[ 0 | 1 ] {1}
dst = A * src + t
dst_x = a11*x + a12*y + t13
dst_y = a21*x + a22*y + t23
Rotation around the origin
R = [ [cos(θ), -sin(θ), 0]
[sin(θ), cos(θ), 0]
[0 , 0 , 1] ]
Rotation around center cx,cy, this is the same as T(cx,cy)∗R(θ)∗T(−cx,−cy)
R = [ [cos(θ), -sin(θ), -cx⋅cos(θ)+cy⋅sin(θ)+cx]
[sin(θ), cos(θ), −cx⋅sin(θ)−cy⋅cos(θ)+cy]
[0 , 0 , 1] ]
Translation
T = [ [1, 0, dx],
[0, 1, dy],
[0, 0, 1]]
Scaling
S = [ [sx, 0, 0],
[0, sy, 0],
[0, 0, 1]]
Shear
H = [ [ 0, sx, 0],
[sy, 0, 0],
[ 0, 0, 1]]
There is also reflection and mirror image transformation.
R = [ [a11, a12, a13]
[a21, a22, a23]
[a31, a32, a33] ]
dst = [ a11, a12, a13 ] {src}
[ a21, a22, a23 ] {1}
----------------------
[ a31, a32 ,a33] {src}
{1}
Note to myself:
py -3 setup.py check
py -3 setup.py sdist
py -3 setup.py bdist_wheel
pip3 install dist/thenewpackage.whl
twine upload dist/*