bbenligiray / stag

STag: A Stable Fiducial Marker System

STag: A Stable Fiducial Marker System

Important

Due to other commitments, this repository is not actively maintained anymore. For an updated fork and a corresponding Python package, please refer to: https://github.com/manfredstoiber/stag

Code used in the following paper:

B. Benligiray; C. Topal; C. Akinlar, "STag: A Stable Fiducial Marker System," Image and Vision Computing, 2019.

Markers (see ref/marker generator for reference code for marker generation):

https://drive.google.com/drive/folders/0ByNTNYCAhWbIV1RqdU9vRnd2Vnc?resourcekey=0-9ipvecbezW8EWUva5GBQTQ&usp=sharing

ROS Package

• STag ROS: A ROS package for the Stable Fiducial Marker System by Unmanned Systems & Robotics Lab - UofSC

• ROS wrapper for STag by Dartmouth Reality and Robotics Lab

Some figures from the paper:

stag's Issues

STag Marker not being detected on cropped image

Hi,

Firstly, thanks for making the library open-source. It's working great, especially on devices with limited computational resources like RPI-4. Currently, I am getting around an average of 9 FPS when the STAG marker detection algorithm is running on the images captured by the OAK-D camera at 720p resolution 60 FPS. Lowering the resolution to 400p increased the FPS. Hence I tried cropping a part of the image and then running the STAG marker detection algorithm but as you can see in the image below the marker was not detected. The marker is not detected for the cropped image but is detected on the uncropped image. If I downsize the image the marker is detected although it is not robust which is expected. I looked into the code to debug this issue but could not figure out why? Is there an input size limitation? Do you have any suggestions as to why this might be happening?

My final plan is to first run the marker detection on the whole image and then use some tracking algorithm to track the marker. The marker detection would run on the tracked part of the image. After every few seconds, I would run the marker detection algorithm on the whole image to update tracking. This way it should be less intensive on the CPU as the image size would be smaller.

Thanks

Problem with STag generstor

I tested the provided tag generator, but the output is something like this:

If I stop it before eroding and dilating, the output stag is

Could you provide any suggestions on solving this problem?

License

Hi Burak,

thank you for the great paper and for providing the code here.
Do you plan to add a license to your implementation?

Best,

Manuel

Compiler

Hi all,

Thanks very much for the contribution of the great works. Could I ask, what the compiler you use for compile this project and what is the os?

Thank you!
Sincerely,

CUDA speedup?

Hi, in the paper it mentions adding CUDA code to speed up the marker detection by 4 times. Is this implemented in this repo?

Thanks!

README Images

Several questions on extension to a ROS node for localization

First off, thank you for releasing this code base. I'm evaluating using STags for a mobile robot application where the stability of the reading from the tags is extremely important.

Pose estimation code?

I've got a simple ROS integration started here: https://github.com/dartmouthrobotics/stag. I'm starting out with finding the pose of the tag using opencv's solvePnP function with a hard coded camera and distortion configuration (used here https://github.com/dartmouthrobotics/stag/blob/master/src/Stag.cpp#L89). In your paper you mention using a more recent algorithm for finding the camera extrinsics with the refined homography matrix but as far as I can tell you don't provide your implementation of that algorithm. Do you have the implementation you used for your experiments handy? If so, I'd appreciate getting a copy!

Support for configurable tag sizes?

Would you be open to adding configuration for a mapping of tag id to tag size? If so, I'd be happy to look into doing that. Insight into the places that need to be changed in the code would help that go faster.

Marker decoded many times in single frames

When the marker is successfully decoded, the array of decoded markers will usually contain four duplicate readings for the same tag which I'm sure has performance implications. For example, in this frame the marker is decoded four times (the array markers on the Stag object has four markers in it here https://github.com/dartmouthrobotics/stag/blob/master/src/Stag.cpp#L50). I would guess that this is coming from the quad detector and that adding a simple step which filters out duplicate quads based on the sum of the distances between their corners or something would fix this. Output from that frame:

~~~~~~~~~~~~~~~~~
Marker 1 num marker 4
Center [970.951, 268.624]
Translation: [-0.5426630776763407;
 -1.027159407186172;
 2.694480358554943]
Rotation: [-0.0444761597568246;
 0.1080291508322071;
 0.01037789328024481]
Marker 1 num marker 4
Center [970.955, 268.627]
Translation: [-0.5433152177014651;
 -1.026926771242593;
 2.693072105601706]
Rotation: [-0.04121157508080839;
 0.1051655381738611;
 0.009573907338546446]
Marker 1 num marker 4
Center [970.951, 268.624]
Translation: [-0.5426630776763407;
 -1.027159407186172;
 2.694480358554943]
Rotation: [-0.0444761597568246;
 0.1080291508322071;
 0.01037789328024481]
Marker 1 num marker 4
Center [970.955, 268.627]
Translation: [-0.5433152177014651;
 -1.026926771242593;
 2.693072105601706]
Rotation: [-0.04121157508080839;
 0.1051655381738611;
 0.009573907338546446]

Memory Leak

Hi !
Really a fantastic project! It really makes great progress for my project!
But when I was using it, I noticed that the running memory on the server kept growing. I found a memory leak when using the tools "valgrind".
Does anyone else meet the same problem? Or just is the problem with the way I used it?
Many thanks !!

Is there a pose estimation for calculate distance from A marker from The other marker?

Hello, Thank you for sharing amazing marker system.
I tried to calculate distance between Stag markers on real world.
for that, I need pose estimation function.
Is there the function in your library?
if not, could you give tips or help about pose estimation using stag?

Inverted markers

Just wondering, how viable would it be to invert the marker colours in terms of detection? Right now the markers are set up to be a large square of black with a little white in the middle which is not exactly optimal in terms of printer cartridges.

I suppose it would take a major refactor to even try and check if the detection quality is comparable with an inverted colour scheme, but I'm just interested if this was ever considered or tried.

RVec and TVec

it's possible to extract rvec and tvec from markers?
do you have some example?
thanks
L

How to get the best result from multiple estimated results (cornerGroups/marker)?

Hi, Thank you for your work.

When I tried to use stag in my project, I found that each candidate marker would give multiple estimated results (most of the time four) which are almost identical. e.g. two markers in the sight, eight poses return. From your paper, I guess that is to ensure the detection stability when occlusion happens. My questions would be, how can I get only one result for each candidate marker. Should I use average, or there're some other evaluation metrics you suggested to get the best estimation.

Is there a way/ implemetetion to find out camera pose estimation.

Hi, I would like to know if there is any implementation to find out the camera pose estimation.

LibStag.lib file not getting generated

I am building the current version in Windows 10 using cmake 3.22.2 and also using Visual Studio 2019, the .dll, .pdb files are getting generated in debug and release but .lib files are not getting generated. Is the .cmake file correctly written because i tried building multiple times but in vain. Any help would be highly appreciated.
Thanks.

refinement issue

hi，thanks for sharing this repository.
i do have a question regarding the refinement using homography.
you define 4 errors: center x, center y, ellipse a and b.
however, the ellipse center is not the projection of its corresponding circular center, can we use the center position as a cost to refine the homography and further get the refined corners?

best regards,

Unit of corners in config file?

I'm trying to get accurate pose of markers, but I'm not sure what is the unit of the corners in config yaml files. Is it in meter or inch, or something else?

missing files "ED/EDLines.h" and "ED/EdgeMap.h"

Hi,
very great job, i'm trying to compile your source code on windows, but it seems missing two files:
#include "ED/EDLines.h"
#include "ED/EdgeMap.h"

STag dependencies

Hi,

Would you mind sharing the exact dependencies used in the project?
OpenCV version, OS version, compiler etc.

Changes to make this more useful as an external library

I'd like to get the ROS node I have running put into a more finished state by separating out the STag code and the ROS wrapper. To get that node running I had to make a couple of changes to this code:

• There was a memory leak in the EDInterface, so I added a destructor
• The code only compiles on a windows computer unless you remove the windows specific stuff (thankfully not necessary to detect markers!), so I just deleted it (Timer.cpp and the function that finds files in a directory)
• No CMakeLists.txt was provided, so I made one to compile Stag.cpp into a library
• I patched the marker being decoded multiple times problem
• The Stag class has a vector of Marker objects which would increase in size indefinitely over all frames, so I made it clear the markers
• Made detectMarkers return the number of markers detected
• Made the markers vector on the Stag class public to make the markers usable by other code
• Changed the define of NULL to STAG_NULL to avoid a compiler warning
• Put everything into the stag namespace to avoid name collisions in client code
• Added explicit termination criteria into the PoseRefiner.cpp code for the pose optimization

Are you interested in having these changes contributed back into your code? If so, I can do the separation work and improve the style of what I have then make a PR into this code. I could also add an alternate implementation for the windows specific code which would work on linux.

Here is a diff for your reference (messy but can be cleaned if you want the changes).
master...dartmouthrobotics:master

Source Code not Compatible with Opencv 4 higher

Hi,

I am trying to build the source on my windows system and i have a opencvconfig,cmake file from my opencv 4.x.x build. As i am looking at the source code, i can see there are lots of #ifndef BUILD_MACOS provided and some files are even using highly deprecated headers like opencv/cv.h, can you please make the source opencv 4 and higher compatible along with windows ?

Thanks.

Help compiling and running on mac os x

Hello,

Trying to run your detector on Mac OS X and so far I have managed to download the code, change some flags for new opencv and build the library by using CMake.
I want to build the library as a shared library so I can call it from python and use it to detect tags.
The Stag parts compile okay, but when the compilation gets to the ED/ED.cpp file it fails in the directory generally due to dependency on old opencv code (such as IplImage). Any ideas on how I can update the library to compile against new opencv?

Scanning dependencies of target STag
[  4%] Building CXX object CMakeFiles/STag.dir/src/ED/ED.cpp.o
In file included from /Users/Vishesh/repos/tui-table/tags/stag/src/ED/ED.cpp:18:
/Users/Vishesh/repos/tui-table/tags/stag/src/ED/Utilities.h:22:14: error: unknown type name
      'IplImage'
void RGB2Lab(IplImage *rgbImg, IplImage *labImg);
             ^
/Users/Vishesh/repos/tui-table/tags/stag/src/ED/Utilities.h:22:32: error: unknown type name
      'IplImage'
void RGB2Lab(IplImage *rgbImg, IplImage *labImg);
                               ^

There is also a dependency on windows.h in the Timer.h module. How do I convert this to mac?

Another question is whether this whole library can be wrapped for python or similar so that I can ignore main.cpp and write my own loop function there (and just use the library to run the detection algorithm).

Python Wrapper

Hey guys,

Just wanted to say thanks for the paper (a great read; very informative) and for uploading the code here; and that I can probably write the Python wrapper if you all aren’t already doing so.

Thoughts?

Thanks,
Brandon

Exe showing no output or errors

Hi,

I managed to generate a exe file using the darthmout robotics Cmakefile and have the project hooked up to OpenCV4.4.0 on windows. When i run the exe file no out put or errors occur. Do I need to include anything in the directory of the exe file apart from the marker images?

Markers generator

Hello, I am interested in your S-tag solution.
In the page, you mentioned about Markers generator.
When can will you update the software?
I would like to know how I can make the Markers?

Thanks.

How to Draw a Axes on the image from Stag?

How to Draw a Axes on the image from Stag?

After running your program, I had some result such as 1 edges.png ~ 8 ellipses.png, but there is no Homography Refinement image. You mentioned (j) Homography Refinement in Figure 7 on the paper. How can I make Homography Refinement Image such as Drawing a Axes.

How to generate the Stag makers?

Thank you for your really excellent stag library, it is very useful in my project. But I can not find how to generate the marks, so, could you put some marker in the github? Thank you so much!

How do you get the center position in estimateHomography() of Quad.cpp?

Hello, I deeply study your S-tag program.

I have a question. May I ask you some about making the center position of the mark?

// product of these transformations is the homography
H = HarInv * Haffsim;

// locate the projection of the center of the marker
Mat origCenter(3, 1, CV_64FC1);
origCenter.at<double>(0) = 0.5;
origCenter.at<double>(1) = 0.5;
origCenter.at<double>(2) = 1;

origCenter = H * origCenter;
center.x = origCenter.at<double>(0) / origCenter.at<double>(2);
center.y = origCenter.at<double>(1) / origCenter.at<double>(2);

Why do you divide "origCenter.at(2) " from each x, y Coordinator?

In addition, can I get Z Coordinator of the mark center in the camera pose if I take hand-eye calibration?

is it possible?