hhk7734 / tensorflow-yolov4

YOLOv4 Implemented in Tensorflow 2.

tensorflow-yolov4

YOLOv4 Implemented in Tensorflow 2.

>=v3, ❗ Do not try to train. Not yet stable.

Docs: https://wiki.loliot.net/docs/lang/python/libraries/yolov4/python-yolov4-about

Installation

python3 -m pip install yolov4

Changelog

Ref: CHANGELOG

tensorflow-yolov4's Issues

bug ghost detections while using yolov4-tiny-relu

Hi, I encountered one bug while using the detector over video. I detects 'ghosts'. I noticed that it draw bounding boxes where people are not present. maybe it is something with the conversion to bounding boxes and it puts another for larger objects on wrong locations... or something with anchors.
It can be clearly see that the ghost detection follow large objects lower within the image.

Probably you can find out from the data I attach
debugging.zip

You can see 3 ghosts detections on the images in zip file, I also attach the output of the network, the candidates before and after postprocessing and also the predicted bounding boxes as pickle
so you can find the bug
if you need any further assistance from me, it will be my pleasure

ps: sorry for not doing the pull request I was on holiday

strange ground truth values for save_dataset_for_mAP when not loading dataset each time its run

I managed to find a strange behaviour with the save_dataset_for_mAP function.
you have to reload the dataset each time you want to use the function save_dataset_for_mAP, otherwise a strange multiplication (or something else) is happening to the ground truth values being outputted into the "mAP/input/ground-truth" folder:

test_data_set = yolo.load_dataset(
    "/content/cats/test.txt",
    dataset_type="yolo"
)
yolo.save_dataset_for_mAP("/content/mAP/", test_data_set)
yolo.save_dataset_for_mAP("/content/mAP/", test_data_set)
yolo.save_dataset_for_mAP("/content/mAP/", test_data_set)

first time:
cat_head 479 199 808 516
second time:
cat_head 486440 203929 820469 529350
third time:
cat_head 493737435 208823994 832776689 542054400
and so on...

This might seems like an artificial scenario, but it will introduce an error when using Google Colab and having loaded the dataset in one cell and try to call save in another cell multiple times.

if the dataset is loaded before each call to the save_dataset_for_mAP, this is not an issue

test_data_set = yolo.load_dataset(
    "/content/cats/test.txt",
    dataset_type="yolo"
)
yolo.save_dataset_for_mAP("/content/mAP/", test_data_set)

test_data_set = yolo.load_dataset(
    "/content/cats/test.txt",
    dataset_type="yolo"
)
yolo.save_dataset_for_mAP("/content/mAP/", test_data_set)

Support for video live stream webcam

Does tensorflow-yolov4 support video live stream from a webcam ?
I had a look in the base_class.py, it seems not.

Thank you.

Training Error:Model Doesn't Show Results After Training

After Training YOLOv4 on custom dataset consisting 3 different classes, it doesn't show any detections at all. I thought i might be having problem with dataset so i changed it, but still it doesn't show any detections, doesn't matter if i train for 25 epochs or 100.
Here is my Script which i used in Colab.. Can you help me with this?

!pip install yolov4
!unzip yolo-dataset.zip

from tensorflow.keras import callbacks, optimizers
from yolov4.tf import SaveWeightsCallback, YOLOv4
import time

yolo = YOLOv4()
yolo.classes = "classes.names"
yolo.input_size = 640
yolo.batch_size = 4

yolo.make_model()
yolo.load_weights(
    "/content/drive/My Drive/Colab Notebooks/yolo weights/yolov4.conv.137",
    weights_type="yolo"
)

train_data_set = yolo.load_dataset(
    "yolo_train.txt",
    image_path_prefix="yolo-animal-detection-small/train",
    label_smoothing=0.05
)
val_data_set = yolo.load_dataset(
    "yolo_val.txt",
    image_path_prefix="yolo-animal-detection-small/test",
    training=False
)

epochs = 25
lr = 1e-4

optimizer = optimizers.Adam(learning_rate=lr)
yolo.compile(optimizer=optimizer, loss_iou_type="ciou", loss_verbose=0)

def lr_scheduler(epoch):
    if epoch < int(epochs * 0.6):
        return lr
    if epoch < int(epochs * 0.8):
        return lr * 0.5
    if epoch < int(epochs * 0.9):
        return lr * 0.1
    return lr * 0.01

_callbacks = [
    callbacks.LearningRateScheduler(lr_scheduler),
    callbacks.TerminateOnNaN(),
    callbacks.TensorBoard(
        log_dir="logs",
    ),
    SaveWeightsCallback(
        yolo=yolo, dir_path="trained",
        weights_type="yolo", epoch_per_save=1
    ),
]

yolo.fit(
    train_data_set,
    epochs=epochs,
    callbacks=_callbacks,
    validation_data=val_data_set,
    validation_steps=1,
    validation_freq=1,
    steps_per_epoch=100,
)

from yolov4.tf import YOLOv4

model = YOLOv4()


model.classes = "classes.names"
model.input_size = 640

model.make_model()
model.load_weights("trained/yolov4-final.weights", weights_type="yolo")


from PIL import Image
model.inference('yolo-animal-detection-small/test/cats_000.jpg')
Image.open('out.jpg')

Here are some of the files...


yolo_train.txt
yolo_val.txt

digits in `.names`-file causes `prob_loss: 0`

Hi, first of all thank you so much for this awesome package!

I noticed, while training a model to detect handwritten symbols, that if you insert digits (or floats) into the .names file (to replace the default coco.names), the training will fail because prob_loss: 0. I bet this is because somewhere under the bonnet python reads the lines and casts them as integers (floats). Renaming the labels as something that python casts as strings will solve the issue. For example, if names has

• "0, 1, 2..." -> this will fail
• "digit0, digit1, digit2..." -> this is OK

I think it would be simple to fix this by enforcing the labels to be string-objects after reading them from names. However, I'm posting this mostly as a warning for others that might suffer from same behavior.

Question about 'txty' and 'bxby'

Hi, I have two questions about "txty_s = (txty_s - self.a_half[0]) * self.scales[0] + self.a_half[0]" in head.py

1. self.a_half is filled with 0.5, why minus 0.5 from txty_s, and later plus 0.5 again?
2. what does self.scales mean? I could not find any clues for it. bxby represents the value relative to image size?

"Model and weights file do not match" when using custom weights file from darknet

im trained a weights file by darknet yolov4-tiny,
there are 3 classes in my custom dataset, so my .cfg file for darknet:

[net]
# Testing
#batch=1
#subdivisions=1
# Training
batch=32
subdivisions=16
width=416
height=416
channels=3
momentum=0.9
decay=0.0005
angle=0
saturation = 1.5
exposure = 1.5
hue=.1

learning_rate=0.00261
burn_in=100
max_batches = 18000
policy=steps
steps=14400,16200
scales=.1,.1

...


[convolutional]
size=1
stride=1
pad=1
filters=24
activation=linear

[yolo]
mask = 3,4,5
anchors = 10,14,  23,27,  37,58,  81,82,  135,169,  344,319
classes=3
num=6
jitter=.3
scale_x_y = 1.05
cls_normalizer=1.0
iou_normalizer=0.07
iou_loss=ciou
ignore_thresh = .7
truth_thresh = 1
random=0
resize=1.5
nms_kind=greedynms
beta_nms=0.6

...

[convolutional]
size=1
stride=1
pad=1
filters=24
activation=linear

[yolo]
mask = 0,1,2
anchors = 10,14,  23,27,  37,58,  81,82,  135,169,  344,319
classes=3
num=6
jitter=.3
scale_x_y = 1.05
cls_normalizer=1.0
iou_normalizer=0.07
iou_loss=ciou
ignore_thresh = .7
truth_thresh = 1
random=0
resize=1.5
nms_kind=greedynms
beta_nms=0.6

the error occur when

from yolov4.tf import YOLOv4

yolo = YOLOv4(tiny=True)

yolo.classes = "tftf.names"

yolo.make_model()
yolo.model.summary()

yolo.load_weights("yolov4-tiny-tftf_best.weights", weights_type="yolo")

Custom model issue

I was unable to use an already trained model due to this error:

ValueError: Model and weights file do not match.

Is it possible that it's because i used a modified config to train it? Is there already a possibility to solve this problem?

When setting higher batch size during training, increases time taken per epoch?

I am using this package to train a model on a custom dataset of images. I have been training my models using settings of batch size = 1, but when I increase the batch size to let's say 2, the time per epoch increases, which goes against what I understand of how batch processing works. Or is the way how the batches work differently in this package?

modify or remove head and predict

PAN

... -> PAN -> Dim(batch, y, x, #anchors * (xywh + conf + #classes)) ~#size

Head

raw -> sigmoid -> raw, sig
raw, sig -> split -> Dim(batch, y, x, xywh + conf + #classes) * #anchors
... -> split -> Dim(batch, y, x, #anchors, xy) 
                Dim(batch, y, x, #anchors, wh)
                Dim(batch, y, x, #anchors, conf + #classes)

Dim(batch, y, x, #anchors, xy)  -> scale -> add grid
Dim(batch, y, x, #anchors, wh) -> exp -> mul anchor

... -> concat -> Dim(batch, y, x, xywh + conf + #classes)
... -> concat -> Dim(batch, y, x, #anchors * (xywh + conf + #classes))

Predict

... -> reshape -> Dim(1 , y * x * #anchors, (xywh + conf + #classes)) ~#size
... -> concat(#size) -> Dim(1 , y * x * #anchors ~ #size, (xywh + conf + #classes)) 
... -> find max prob(#classes) -> conf * prob -> -> threshold class_prob
                               -> Dim(1 , threshold, (xywh + class_id + class_prob))
... -> NMS -> result

It can do the head after finding max prob. I will change the order in a direction that can reduce operations or increase speed.

ValueError: Model and weights file do not match.

I'm suddenly getting this error... but it worked for me before with the same weights file.
using yolov4-tiny

ValueError: Model and weights file do not match.

How to Extract Detected Objects?

Hi,
Can you tell me how to extract the detected objects or simply crop those objects?
Thanks

EdgeTPU Compiler: Internal compiler error. Aborting.

First of all, thanks a lot for the great work/package/documentation!

I trained a YoloV4 Tiny model using this script:

from tensorflow.keras import callbacks, optimizers
from yolov4.tf import SaveWeightsCallback, YOLOv4
import time

yolo = YOLOv4(tiny=True)
yolo.classes = "my-project/my-classes.names"
yolo.input_size = 608
yolo.batch_size = 32

yolo.make_model(activation1="relu")
yolo.load_weights(
    "my-project/yolov4-tiny.conv.29",
    weights_type="yolo"
)

train_data_set = yolo.load_dataset(
    "my-project/train.txt",
    image_path_prefix="",
    label_smoothing=0.05
)
val_data_set = yolo.load_dataset(
    "my-project/val.txt",
    image_path_prefix="",
    training=False
)

epochs = 400
lr = 1e-4

optimizer = optimizers.Adam(learning_rate=lr)
yolo.compile(optimizer=optimizer, loss_iou_type="ciou")

def lr_scheduler(epoch):
    if epoch < int(epochs * 0.5):
        return lr
    if epoch < int(epochs * 0.8):
        return lr * 0.5
    if epoch < int(epochs * 0.9):
        return lr * 0.1
    return lr * 0.01

_callbacks = [
    callbacks.LearningRateScheduler(lr_scheduler),
    callbacks.TerminateOnNaN(),
    callbacks.TensorBoard(
        log_dir="my-project/logs",
    ),
    SaveWeightsCallback(
        yolo=yolo, dir_path="my-project/logs/trained",
        weights_type="yolo", epoch_per_save=10
    ),
]

yolo.fit(
    train_data_set,
    epochs=epochs,
    callbacks=_callbacks,
    validation_data=val_data_set,
    validation_steps=50,
    validation_freq=5,
    steps_per_epoch=100,
)

Using the produced weights file, I verified that this model works, looking very promising!

Then I converted to Quantized TFLite using this script:

from yolov4.tf import YOLOv4

yolo = YOLOv4(tiny=True, tpu=True)

yolo.classes = "my-project/coco.names"
yolo.input_size = (608, 608) # width, height
yolo.make_model(activation1="relu")

yolo.load_weights("my-project/yolov4-tiny-40.weights", weights_type="yolo")

dataset = yolo.load_dataset(
    "my-project/train.txt",
    training=False,
    image_path_prefix="JPEGImages/"
)

yolo.save_as_tflite(
    "my-project/yolov4-tiny-relu-int8.tflite",
    quantization="full_int8",
    data_set=dataset,
    num_calibration_steps=400
)

The produced TF Lite model seems fine:
yolov4-tiny-relu-int8.tflite.zip

But the EdgeTPU Compiler failed with:

Edge TPU Compiler version 15.0.340273435

Internal compiler error. Aborting!

I tried lower input resolutions (TF Lite conversion time), nu success. I also tried the conversion with both with the 2.0.1 release on PyPi and with the current master branch. Any clue on what I seem to be missing?

what about data augmentation?

Does this implementation use data augmentation, i actually wanna make it more accurate, i wonder if data augmentation can be used in this implementation to generate much more accurate results?

range(3) causes index out of range

Hi -- thanks so much for making this package, it's great.

I made a lite + tiny test:

yolo = YOLOv4(tiny=True)

yolo.classes = "coco.names"

yolo.make_model()
yolo.load_weights("/users/andy/downloads/yolov4-tiny.weights", weights_type="yolo")

yolo.save_as_tflite("yolov4-tiny.tflite")

from yolov4.tflite import YOLOv4

yolo = YOLOv4()

yolo.classes = "coco.names"

yolo.load_tflite("yolov4-tiny.tflite")
yolo.inference(media_path="/users/andy/downloads/vtest.avi", is_image=False)

When I ran it, I got two list index out of range errors in tflite/__init__.py.

I'm not sure if my fix is valid for all cases, so I didn't make a PR,but I made these changes to __init__.py, and then the lite + tiny test ran properly.

At line 72, i changed:
self.output_index = [output_details[i]["index"] for i in range(3)]
to:
self.output_index = [output_details[i]["index"] for i in range(len(output_details))]

At line 141, I changed:
self.interpreter.get_tensor(self.output_index[i]) for i in range(3)
to:
self.interpreter.get_tensor(self.output_index[i]) for i in range(len(self.output_index))

Training interrupted due to index out of bounds error

Hi,

My training began successfully but got interrupted with the following error after some 30 steps.

Traceback (most recent call last):
  File "train.py", line 10, in <module>
    pre_trained_weights=None,
  File "/data/mtare/darknet-py/venv/lib/python3.6/site-packages/yolov4/tf/__init__.py", line 286, in train
    for image_data, target in trainset:
  File "/data/mtare/darknet-py/venv/lib/python3.6/site-packages/yolov4/core/dataset.py", line 137, in __next__
    ) = self.preprocess_true_boxes(bboxes)
  File "/data/mtare/darknet-py/venv/lib/python3.6/site-packages/yolov4/core/dataset.py", line 373, in preprocess_true_boxes
    label[best_detect][yind, xind, best_anchor, :] = 0
IndexError: index 76 is out of bounds for axis 0 with size 76

I don't think my data is inaccurate. Let me know what you think. @hhk7734

bboxes as opencv input

What do you recommend in using for getting the bboxes/ROI for using with cv2.rectangle() function? So getting the original coordinates from yolo.predict()

YOLOv4-tiny is released

@hhk7734 Hi,
Nice work with pypi: https://pypi.org/project/yolov4/

Did you manage to realize CIoU-loss and Mosaic?

Also YOLOv4-tiny is released:

• source code: https://github.com/AlexeyAB/darknet
• cfg: https://raw.githubusercontent.com/AlexeyAB/darknet/master/cfg/yolov4-tiny.cfg
• weights: https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v4_pre/yolov4-tiny.weights
• discussion: AlexeyAB/darknet#6067

There is required only 1 feature:

1. Add groups= and group_id= to the [route] layer.

[route]
layers=-1
groups=2
group_id=1

So if input is WxHxC, it divides input into 2 groups WxHx(C/2) (there are 2 groups: 0 and 1), and loads the 2nd group_1 WxHx(C/2).

If there are many layers specified in layers= parameter, then this will be done for each of the input layers specified in layer=, then results will be concatenated across channels.

YOLOv4 Training error

Transfer learning from COCO with different classes

Hi @hhk7734 thank you for your amazing work!

I'm trying to load the yolov4-tiny-relu.weights into a model with three classes.
When I'm doing so, I'm getting a ValueError("Model and weights file do not match.") from the load_weights function.

To my understanding, the number of classes affects the PANet and the YoloHead, but on the weights.py file the distinction between the two in the way the weights are loaded are the layers' indexes.

Could you orient me on how to accommodate loading and saving weights for different class numbers?

Thanks
Raz

Training/Conversion with non-square input

Would it be possible to support rectangular inputs for training/conversion?

Questions about v4 tiny on Edge TPU

@hhk7734 Very interesting to see v4 tiny on Edge TPU. I have two questions

1. What ops where not mapped to the TPU?
2. Did you quantize (post-training/training-aware) to INT8?

Thanks

Originally posted by @ankandrew in #4 (comment)

would be better with more parameters

it is a great work, could you add more parameters that would get the lib more applicable.

such as:

output_image = draw_bbox(img, bbox, label, conf)

result = yolo.draw_bboxes(original_image, pred_bboxes)
cv2.imwrite("result.jpg", result) 
from IPython.display import Image
Image("result.jpg")

ValueError: Model and weights file do not match.

Hi,

I have trained a custom dataset with YOLOv4.
Now I am trying to use the trained model on test image but I am getting following error-

My env is as below-
Python 3.7.6
TensorFlow 2.0.0
Windows 10

Please suggest.

Yolov4 tiny conversion to tflite

Hi, thanks for this! I used AlexeyAB yolov4-tiny cfg, trained for 1 class. When trying to convert weights to tflite I get:

2020-07-16 08:15:29.339406: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 13962 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5)
loc(callsite("YOLOv4Tiny/PANetTiny/up_sampling2d/resize/ResizeBilinear"("/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/layers/convolutional.py":2245:0) at callsite("/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py":927:0 at callsite("/usr/local/lib/python3.6/dist-packages/yolov4/model/neck.py":236:0 at callsite("/usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/impl/api.py":309:0 at callsite("/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py":927:0 at callsite("/usr/local/lib/python3.6/dist-packages/yolov4/model/yolov4.py":110:0 at callsite("/usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/impl/api.py":309:0 at callsite("/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py":927:0 at callsite("/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/saving/saving_utils.py":132:0 at "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py":441:0)))))))))): error: 'tf.ResizeBilinear' op is neither a custom op nor a flex op
error: failed while converting: 'main'
Ops that can be supported by the flex runtime (enabled via setting the -emit-select-tf-ops flag): ResizeBilinear.
Traceback (most recent call last):
  File "/usr/local/bin/toco_from_protos", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/tensorflow_core/lite/toco/python/toco_from_protos.py", line 93, in main
    app.run(main=execute, argv=[sys.argv[0]] + unparsed)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow_core/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/usr/local/lib/python2.7/dist-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/usr/local/lib/python2.7/dist-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow_core/lite/toco/python/toco_from_protos.py", line 56, in execute
    enable_mlir_converter)
Exception: /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/layers/convolutional.py:2245:9: error: 'tf.ResizeBilinear' op is neither a custom op nor a flex op
        interpolation=self.interpolation)
        ^
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py:927:19: note: called from
                  outputs = call_fn(cast_inputs, *args, **kwargs)
                  ^
/usr/local/lib/python3.6/dist-packages/yolov4/model/neck.py:236:9: note: called from
        x1 = self.upSampling18(x1)
        ^
/usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/impl/api.py:309:7: note: called from
      return func(*args, **kwargs)
      ^
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py:927:19: note: called from
                  outputs = call_fn(cast_inputs, *args, **kwargs)
                  ^
/usr/local/lib/python3.6/dist-packages/yolov4/model/yolov4.py:110:9: note: called from
        x = self.panet_tiny(x)
        ^
/usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/impl/api.py:309:7: note: called from
      return func(*args, **kwargs)
      ^
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py:927:19: note: called from
                  outputs = call_fn(cast_inputs, *args, **kwargs)
                  ^
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/saving/saving_utils.py:132:7: note: called from
      outputs = model(inputs, training=False)
      ^
/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py:441:9: note: called from
        return weak_wrapped_fn().__wrapped__(*args, **kwds)
        ^
<unknown>:0: error: failed while converting: 'main'
Ops that can be supported by the flex runtime (enabled via setting the -emit-select-tf-ops flag): ResizeBilinear.

support for data augmentation?

is it possible to do any sort of data augmentation for training with this library?
if it's not possible yet, when will it be?
thanks

TypeError

Hi,

I use the following code:

from yolov4.tf import YOLOv4
yolo = YOLOv4(tiny=True)
yolo.classes = "coco.names"
yolo.make_model()

with tensorflow version: 2.1.0 and numpy version: 1.19.0, but i end up with the following error:

File "C:\Users\iwu\Documents\Python Scripts\YoloEnv\Scripts\TryAndError\PipYOLOv4.py", line 23, in <module>
    yolo.make_model()

  File "C:\Users\iwu\AppData\Local\Continuum\anaconda3\envs\tf2\lib\site-packages\yolov4\tf\__init__.py", line 76, in make_model
    activation1=activation1,

  File "C:\Users\iwu\AppData\Local\Continuum\anaconda3\envs\tf2\lib\site-packages\yolov4\model\yolov4.py", line 48, in __init__
    activation0=activation0, activation1=activation1

  File "C:\Users\iwu\AppData\Local\Continuum\anaconda3\envs\tf2\lib\site-packages\yolov4\model\backbone.py", line 148, in __init__
    super(CSPDarknet53, self).__init__(name="CSPDarknet53")

TypeError: super(type, obj): obj must be an instance or subtype of type

seems like there's an issue with the super charging. Same error occurs for YOLOv4(tiny=False).
Do you have any idea what causes this error?

Greetings,
Ilkay

Test steps_per_execution

tensorflow-yolov4/py_src/yolov4/tf/__init__.py

Lines 239 to 255 in fd45eea

	def compile(
	self,
	loss_iou_type: str = "ciou",
	loss_verbose=1,
	optimizer=optimizers.Adam(learning_rate=1e-4),
	**kwargs
	):
	# TODO: steps_per_execution tensorflow2.4.0-rc4
	self.model.compile(
	optimizer=optimizer,
	loss=train.YOLOv4Loss(
	batch_size=self.batch_size,
	iou_type=loss_iou_type,
	verbose=loss_verbose,
	),
	**kwargs
	)

bad results when using trained weight

When training i get good results and a mAP score of 91% on a custom dataset, i then save the model and try to load in a new yolo object:
yolo = YOLOv4(tiny=False)
yolo.classes = "/content/classes.names"
yolo.make_model()
yolo.load_weights("/content/backup/weights/yolov4-final.weights", weights_type="yolo")

calculating the again mAP with the loaded weight gives a mAP of 0% or in some cases low percentages. I don't understand why it is not working and giving consistent results ?

I have tried using the default saved weight after the training is done, and also saving it using:
yolo.save_weights("/content/backup/weights/custom.weights", weights_type="yolo")

it seems like it is not really saving or loading the weights properly. not sure it is a bug or if I am missing a step?

Model and weights file do not match

I installed yolov4 from PyPI, and attempting to run the 1st example I get "Model and weights file do not match" [I am using "yolov4.weights" file from the link in the description]

ModuleNotFoundError: No module named 'yolov4.tf'; 'yolov4' is not a package

Good evening, and thank you for your great work! I have an error ( it's the title of my issue ) while executing the script to run the conversion of YoloV4 to TFlite. Any ideas what I have been doing wrong? Sorry for the trouble. I am using Windows, and I have Python 3.7.0, using an anaconda environment with tensorflow-gpu/tensorflow version 2.3.0. Any help would be much appreciated.

Also quick question since I am a beginner; I have been trying to convert YOLOv3 and YOLOv4 weights to int8 quantized Tflite to run them to Coral for quite some time without success. Could I do it, at least for YOLOv4-tiny using this script? Again, I am a beginner and any help would be much appreciated!

Question about input size

Hello! Thank you for the amazing work, the package is quite easy to use!

I have a question about the input size - is it intentionally not permitted to specify the input size hence it is fixed to 608x608? This can be easily changed inside the constructor of YOLOv4 class and it is working properly. Unfortunately that directly changes the implementation and makes the PIP package not usable. :(

Thanks in advance for the answer!

Unable to train on Custom Dataset

I wanted to train it on my custom dataset, but it is consistently raising an error
"IndexError: index 224 is out of bounds for axis 1 with size 20"
and this error is changing values but it is consistent

Any Help would be appreciated.
Thanks

Non-maximum supression and conversion

Hi, thank you for created such a beautiful package.

I have trouble in converting Yolov4 tiny from weights to tflite with int8 quantization, but I don't know how to use the parameter in data_set in the function save_as_tflite.

Besides, I used the tflite float16 yolov4-tiny but it showed a lot of bounding boxes, can you suggest for me how to adding the non-maximum suppression so that the bounding box would be more accurate?

Thank you a lot.

Different results after saving and loading trained weights

Hey, thanks so much for creating this project! Was easy to use, I was able to customize it to my needs.

I've been trying to train tiny yolov4 on a custom single class. Currently, I'm getting pretty good results in terms of validation loss, and the model converges well. However, if the training session terminates and I either resume training or predict from a checkpoint (either manually from save_weights function or ModelCheckpoint callback), I get wildly different results. The loss jumps from 2 to over 200 as if the weights are untrained. Model predictions return zeros for the most part.

At first, I suspected that my preprocessing was causing this, but after double checking and searching online, it seems that there might be an issue with Keras and/or Tensorflow saving the network. Model.save() unfortunately doesn't work at all.

This post has long thread of others who seem to have a similar issue, and it appears the solution is architecture specific. From what I gathered, some possible causes are from Upsampling layers or Lambda layers in loops. Here's another post describing the solution as fixing a random seed.

For reference, I'm using yolo.model.load_weights() and yolo.model.save_weights().

TypeError when running code

I'm not able to run the code because of the following errormessage. Please help.

My code:

from yolov4.tf import YOLOv4
yolo = YOLOv4()
yolo.classes = './newtry/coco.names'
yolo.make_model()
yolo.load_weights('./newtry/yolov4.weights', weights_type="yolo")
yolo.inference(media_path="./newtry/test.jpg")

I get the following error when running my code:

TypeError                                 Traceback (most recent call last)
<ipython-input-14-9306638226c6> in <module>
      1 yolo.classes = './newtry/coco.names'
      2 
----> 3 yolo.make_model()
      4 
      5 yolo.load_weights('./newtry/yolov4.weights', weights_type="yolo")

~\anaconda3\lib\site-packages\yolov4\tf\__init__.py in make_model(self)
    165         )
    166         # [batch, height, width, channel]
--> 167         self.model(tf.keras.layers.Input([self.input_size, self.input_size, 3]))
    168 
    169     def load_weights(self, path: str, weights_type: str = "tf"):

~\anaconda3\lib\site-packages\tensorflow\python\keras\engine\base_layer.py in __call__(self, inputs, *args, **kwargs)
    552             # In graph mode, failure to build the layer's graph
    553             # implies a user-side bug. We don't catch exceptions.
--> 554             outputs = self.call(inputs, *args, **kwargs)
    555           else:
    556             try:

~\anaconda3\lib\site-packages\yolov4\model\yolov4.py in call(self, x, training)
    246         s_bboxes = self.conv93(s_bboxes)
    247         # (batch, 3x, 3x, 3, (4 + 1 + num_classes))
--> 248         s_bboxes = self.decode93(s_bboxes, training)
    249 
    250         x2 = self.conv94(x2)

~\anaconda3\lib\site-packages\tensorflow\python\keras\engine\base_layer.py in __call__(self, inputs, *args, **kwargs)
    582           if base_layer_utils.have_all_keras_metadata(inputs):
    583             inputs, outputs = self._set_connectivity_metadata_(
--> 584                 inputs, outputs, args, kwargs)
    585           if hasattr(self, '_set_inputs') and not self.inputs:
    586             # Subclassed network: explicitly set metadata normally set by

~\anaconda3\lib\site-packages\tensorflow\python\keras\engine\base_layer.py in _set_connectivity_metadata_(self, inputs, outputs, args, kwargs)
   1392             'Pass everything else as a keyword argument '
   1393             '(those arguments will not be tracked '
-> 1394             'as inputs to the layer).'.format(self.name))
   1395 
   1396     # If the layer returns tensors from its inputs, unmodified,

TypeError: This layer ("decode") takes a single positional argument in `call()`, which is by convention the `inputs` argument, and only this argument may be specified as a positional argument. Pass everything else as a keyword argument (those arguments will not be tracked as inputs to the layer).

when using tensorflow not tflite runtime the load delegate method is not found to use coral

in tflite/init
bool variable can be added

tflitePresent=True
try:
    import tflite_runtime.interpreter as tflite
except ModuleNotFoundError:
    tflitePresent=False
    import tensorflow.lite as tflite

and later

        if tflitePresent:
            self.interpreter = tflite.Interpreter(
                model_path=tflite_path,
                experimental_delegates=[
                    tflite.load_delegate("libedgetpu.so.1")
                ],
            )
        else:
            self.interpreter = tflite.Interpreter(
                model_path=tflite_path,
                experimental_delegates=[
                    tflite.experimental.load_delegate("libedgetpu.so.1")
                ],
            )

to prevent the error, but maybe there is more elegant way :)

No way to pass epochs, LR etc as parameters

So far there is no way to pass parameters to the train model while initiating a new training from scratch.

For now one can edit the following file from source.
tensorflow-yolov4/py_src/yolov4/tf/__init__.py

What is the purpose of Dataset.grid in py_src/yolov4/tf/dataset.py

In the Dataset object you defined a grid attribute which is only used in line 182:

tensorflow-yolov4/py_src/yolov4/tf/dataset.py

Lines 182 to 183 in fd45eea

	for i, _grid in enumerate(self.grid_xy):
	ground_truth[i][..., 0:2] = _grid

I don't understand the purpose of this assignment , because later the xywh gt vector is assigned to each grid point in line 222:

tensorflow-yolov4/py_src/yolov4/tf/dataset.py

Line 222 in fd45eea

ground_truth[i][0, _y, _x, j, 0:4] = xywh

And when calculating the xiou_loss only these xywh gt vectors are used. So there seems to be no use anywhere for the _grid values.

Issue with No detection?

Hello,

If there are no objects on the image, what happens?

I'm having an error where the box result is negative:
prediction --> [[0. 0. 0. 0. 0. 0.]]
after the call to fit_pred_bboxes_to_original: [[ 0. -1.156 0. 0. 0. 0. ]]
time: 3171.96 ms

If I need to then use the box; the negative result will bring an error.

I'm using the following script

def run_obstacle_detection(img):
    start_time=time.time()
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    resized_image = yolo.resize_image(img)
    # 0 ~ 255 to 0.0 ~ 1.0
    resized_image = resized_image / 255.
    #input_data == Dim(1, input_size, input_size, channels)
    input_data = resized_image[np.newaxis, ...].astype(np.float32)

    candidates = yolo.model.predict(input_data)

    _candidates = []
    result = img.copy()
    for candidate in candidates:
        batch_size = candidate.shape[0]
        grid_size = candidate.shape[1]
        _candidates.append(tf.reshape(candidate, shape=(1, grid_size * grid_size * 3, -1)))
        #candidates == Dim(batch, candidates, (bbox))
        candidates = np.concatenate(_candidates, axis=1)
        #pred_bboxes == Dim(candidates, (x, y, w, h, class_id, prob))
        pred_bboxes = yolo.candidates_to_pred_bboxes(candidates[0], iou_threshold=0.35, score_threshold=0.40)
        print(pred_bboxes)
        pred_bboxes = yolo.fit_pred_bboxes_to_original(pred_bboxes, img.shape)
        print(pred_bboxes)
        exec_time = time.time() - start_time
        print("time: {:.2f} ms".format(exec_time * 1000))
        result = yolo.draw_bboxes(img, pred_bboxes)
    return result, pred_bboxes

result, pred_bboxes = run_obstacle_detection(img_left)
plt.imshow(result)
plt.show() 

Thanks for helping!

Weights file not being read properly during training

Hi,

Thanks for this python implementation. I just tried it out on my custom data files and the initial weights file provided by you. I also tried it out on the weights file yolov4.conv.137 which is shipped by darknet.

On your file I am getting the below error:

File "train.py", line 10, in <module>
    pre_trained_weights="yolov4.weights",
  File "/data/mtare/darknet-py/venv/lib/python3.6/site-packages/yolov4/tf/__init__.py", line 152, in train
    self.load_weights(pre_trained_weights)
  File "/data/mtare/darknet-py/venv/lib/python3.6/site-packages/yolov4/tf/__init__.py", line 322, in load_weights
    utils.load_weights(self.model, weights_path)
  File "/data/mtare/darknet-py/venv/lib/python3.6/site-packages/yolov4/core/utils.py", line 135, in load_weights
    assert len(wf.read()) == 0, "failed to read all data"
AssertionError: failed to read all data

On yolov4.conv.137 I get the following error:

File "train.py", line 10, in <module>
    pre_trained_weights="yolov4.conv.137",
  File "/data/mtare/darknet-py/venv/lib/python3.6/site-packages/yolov4/tf/__init__.py", line 152, in train
    self.load_weights(pre_trained_weights)
  File "/data/mtare/darknet-py/venv/lib/python3.6/site-packages/yolov4/tf/__init__.py", line 324, in load_weights
    self.model.load_weights(weights_path).expect_partial()
  File "/data/mtare/darknet-py/venv/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 250, in load_weights
    return super(Model, self).load_weights(filepath, by_name, skip_mismatch)
  File "/data/mtare/darknet-py/venv/lib/python3.6/site-packages/tensorflow/python/keras/engine/network.py", line 1259, in load_weights
    with h5py.File(filepath, 'r') as f:
  File "/data/mtare/darknet-py/venv/lib/python3.6/site-packages/h5py/_hl/files.py", line 408, in __init__
    swmr=swmr)
  File "/data/mtare/darknet-py/venv/lib/python3.6/site-packages/h5py/_hl/files.py", line 173, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 88, in h5py.h5f.open
OSError: Unable to open file (file signature not found)

Technically both should work as they do with the C implementation of darknet yolov4.
Please help

Cannot add config file for newly trained custom weight file

I was trying to train a new custom detector by adding one additional new class object into the reduced COCO dataset (1 new class + 35 classes). And I have trained new weight file using yolov4.conv.137 with modified yolov4-custom.cfg from Darknet AlexeyAB. Is there any way to add my yolov4-custom.cfg into your code so that the detector could identify the new object? Thanks.

Significant FPS drop and issues with tracking

Good evening,
I have been trying using the converted model today for object detection with deepsort, without result. Before that, I tried testing it as underlined by you, using the inference command. However, when used with videos it takes a huge amount of time to change the frame and track the changes. As for deepsort, I referred to https://github.com/theAIGuysCode/yolov4-deepsort and his tracker script, only to provide the following error:
ValueError: Shapes (1, 19, 19) and (1, 38, 38) are incompatible
After that, I tried running the above script ( basically the same as hunglc007's script ) with the following correction for int8 models, as specified here hunglc007#214 ( I recall you have referenced someone at one issue at this ). I tried running it, and it got me the following error:

File "object_tracker.py", line 127, in main
    output_tensors = decode(pred[2], input_size // 8, NUM_CLASS, STRIDES, ANCHORS, i, XYSCALE, 'tflite')
IndexError: list index out of range

Should I swap the number 2 with 1 or 0, it will eventually bring up an image, with an extremely inaccurate detection. I haven't tried this with video, for safety purposes :P ...

These issues and the fps are the crucial issues for me. Thank you for your great work though, the conversion is successful and the model is working.

Enable mixed precision

Is it possible to use TensorFlow's mixed precision for a performance boost on newer nvidia graphics cards?

from tensorflow.keras.mixed_precision import experimental as mixed_precision

policy = mixed_precision.Policy('mixed_float16')
mixed_precision.set_policy(policy)

yolo.make_model()

Right now it throws a TypeError when trying to do this because the outputs and inputs will not match in the existing model.

TypeError: Input 'y' of 'AddV2' Op has type float32 that does not match type float16 of argument 'x'.

Am I doing something wrong in enabling it this way or is it just not possible with the current implementation?

detect customer class id

Hi,
I would like to know if it is possible to detect customer class id.

For example, I only interest in the person in an image, is it possible to only detect the person and draw the person bboxes?

thx in advance

How to improve tiny performance

Hello, first thanks for your work.

I am running yolo v4 tiny but seeing not very good performance. I am timing the predict function, but get worse results than what I expected. I created this Colab to replicate the results.

Results (best) are 42.7 ms or 23 FPS per inference w/ batch=1. How can I get better results?

Thanks

Test tf.int8 vs tf.uint8

tensorflow-yolov4/py_src/yolov4/tf/__init__.py

Lines 149 to 153 in fd45eea

	elif quantization == "full_int8":
	converter.representative_dataset = representative_dataset_gen
	_supported_ops += [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
	converter.inference_input_type = tf.int8
	converter.inference_output_type = tf.int8

clean up model

class Decode(Model):
    def __init__(self, anchors_ratio, num_classes: int, xyscale):
        super(Decode, self).__init__()
        self.anchors_ratio = anchors_ratio
        self.cell_ratio = None
        self.num_classes = num_classes
        self.xyscale = xyscale
        self.xy_grid = None

        self.reshape0 = layers.Reshape((-1,))
        self.concatenate = layers.Concatenate(axis=-1)

    def build(self, input_shape):
        self.cell_ratio = 1 / input_shape[1]

        self.reshape0.target_shape = (
            input_shape[1],
            input_shape[1],
            3,
            5 + self.num_classes,
        )

        # grid(1, i, j, 3, 2) => grid top left coordinates
        # [
        #     [ [[0, 0]], [[1, 0]], [[2, 0]], ...],
        #     [ [[0, 1]], [[1, 1]], [[2, 1]], ...],
        # ]
        self.xy_grid = tf.stack(
            tf.meshgrid(tf.range(input_shape[1]), tf.range(input_shape[1])),
            axis=-1,
        )  # size i, j, 2
        self.xy_grid = tf.reshape(
            self.xy_grid, (1, input_shape[1], input_shape[1], 1, 2)
        )
        self.xy_grid = tf.tile(self.xy_grid, [1, 1, 1, 3, 1])
        self.xy_grid = tf.cast(self.xy_grid, tf.float32)

    def call(self, x, training: bool = False):
        """
        @param x: Dim(batch, grid_y, grid_x,
                        anchors * (tx, ty, tw, th, score, classes))

        @return
            Dim(batch, grid_y, gird_x, anchors, (x, y, w, h, score, classes ))
        """
        x = self.reshape0(x)
        txty, twth, score, classes = tf.split(
            x, (2, 2, 1, self.num_classes), axis=-1
        )

        # x = (f(tx) + left_x) * strides / input_size
        # y = (f(ty) + top_y) * strides / input_size
        txty = (activations.sigmoid(txty) - 0.5) * self.xyscale + 0.5
        xy = (txty + self.xy_grid) * self.cell_ratio

        # w = (anchor_w * exp(tw)) / input_size
        # h = (anchor_h * exp(th)) / input_size
        wh = self.anchors_ratio * backend.exp(twth)

        if not training:
            score = activations.sigmoid(score)
            classes = activations.sigmoid(classes)

        x = self.concatenate([xy, wh, score, classes])
        return x

Installation error in Windows

Hi,

When I run command python -m pip install yolov4 in my virtual env, I am getting below error-

My system setup is as below-
Python 3.6.10
Windows 10
Cuda 10.1
cuDNN 7.6.0

cv_frame_size doesn't seem to work

when using yolo.inference() the cv_frame_size argument doesn't seem to have effect of the window opening up.

yolo.inference(IMAGE_PATH, cv_frame_size=(800, 600))