Comments (14)
Hi @nikitakaraevv, thank you for your reply! I haven't replaced PIPs with CoTracker in that project, I was using a different tracking method with SAM previously which is called Track-Anything, and realised that I could also try the CoTracker for point-based tracking. I will give it a go and let you know if there are any interesting results :)
from co-tracker.
Thanks, @lucasjinreal! CoTracker is currently unable to run in real-time on a CPU as it is still not fast enough for such a use-case. And thanks for your suggestion, I will try to add examples with CoTracker applications to this repo later.
from co-tracker.
Hi @GeorgeAdamon, thank you! Yes, we are working on that as well. You can check out our new paper https://vggsfm.github.io/
from co-tracker.
Hi @AshkanTaghipour, I haven't tried it myself, but it should be similar to v1, only much more efficient. As a rule of thumb, the more points you are tracking at a time, the better the performance.
from co-tracker.
Thank you, @lucasjinreal!
I believe models like CoTracker can be useful for many applications where motion estimation is needed. For example, video segmentation could be enhanced by combining point tracking with Segment Anything. I also think that CoTracker can improve 3D reconstruction of both static and dynamic objects, or even object tracking itself.
Furthermore, It can be directly applied in video editing to attach something, such as text or stickers, to a moving object.
from co-tracker.
@nikitakaraevv thanks, does the computation heavy (for example, able to run on mobile devices like snapchat visualeffects)?
Also wanna ask, Does CoTracker able to get real-world 3d tracking results which could be used in 3d body estimation (3d body estimation suffering on tracking people or moving camera) ?
from co-tracker.
Hi @lucasjinreal!
If you don't need to run it in real-time, I believe CoTracker can already work on mobile devices if you manage to convert it to the required format. However, a lot of work has to be done to optimize it for real-time applications on-device.
CoTracker currently estimates 2D tracks, but I think it should be possible to improve 3D body pose estimation even with such tracks. This looks like a research problem! :)
from co-tracker.
@nikitakaraevv Looks promising, so, is CoTracker able to run realtime on CPU? I think it can be more be usable if combined with appearing StableDiffusion's controlnet swap body from user's video.
from co-tracker.
Hi @nikitakaraevv,
Thank you for your amazing work. I am working on a project aimed at segmenting and reconstructing 3D objects from head-mounted AR device scanned data previously with mask-rcnn and tsdf integration. Recently I tried SAM and Co-tracker and found them to work very well and I'm considering integrating them into my work in the future.
You've previously noted that "CoTracker could improve the 3D reconstruction of both static and dynamic objects." Could you provide some specific insights into how CoTracker achieves this improvement? I'm particularly interested in understanding the underlying mechanics and the particular aspects where the enhancement is most noticeable. Your expertise on this subject would be greatly appreciated.
Thank you for your contribution to the field and for your time in addressing this inquiry.
from co-tracker.
Hi @DavidTu21, thank you!
Did you replace PIPs with CoTracker in this project? It would be interesting to see any numbers or results if you could share them!
We're currently working on 3D reconstruction using CoTracker, but we don't have any specific insights yet. This is just our current intuition. Stay tuned!
from co-tracker.
@nikitakaraevv Can co-tracker work real time on video stream from a webcam? Are there any examples for this?
from co-tracker.
Hi @lucasjinreal! If you don't need to run it in real-time, I believe CoTracker can already work on mobile devices if you manage to convert it to the required format. However, a lot of work has to be done to optimize it for real-time applications on-device.
CoTracker currently estimates 2D tracks, but I think it should be possible to improve 3D body pose estimation even with such tracks. This looks like a research problem! :)
This is a truly amazing project! My project involves the tracking of a moving camera (filming ice-skaters), and this algorithm truly outperforms anything that is out there in terms of point tracking accuracy and consistency. This is the first step in any camera tracking solution.
Unfortunately, the second step of camera tracking, what is commonly referred as "3D Solve" seems to be outside the scope of this research. Is anyone aware of a project that combines the outputs of co-tracker with a 3D camera solver ?
from co-tracker.
Hi @abhishekmonogram, yesterday, we released a new model that has an online mode, please see this issue.
from co-tracker.
Thank you, @lucasjinreal! I believe models like CoTracker can be useful for many applications where motion estimation is needed. For example, video segmentation could be enhanced by combining point tracking with Segment Anything. I also think that CoTracker can improve 3D reconstruction of both static and dynamic objects, or even object tracking itself.
Furthermore, It can be directly applied in video editing to attach something, such as text or stickers, to a moving object.
Hi,
Thank you for your comprehensive paper; I really enjoyed reading it. I have a question regarding the application of version 2 (v2) in point tracking with the "Segment Anything" feature. I've read some of your comments about the performance drop of v2 in the previous code base. Could you please clarify if it's still feasible to use v2 for point tracking with "Segment Anything"?
Thank you!
from co-tracker.
Related Issues (20)
- Inference with sequence of multiple images for a particular single pixel using co-tracker2 HOT 3
- cannot backward tracking in online model with specific queries in middle frame HOT 2
- About the training window size of stride 8 model. HOT 1
- About reproduction of training. HOT 1
- Tracking errors with cotracker2 HOT 2
- Question About GPU Memory Requirements for Model Training HOT 2
- question about backward_tracking HOT 1
- Efficency of V2 and adding more points? HOT 1
- Can not reproduce table3 with the provided checkpoint. HOT 2
- Code for Training and Testing on PointOdyssey HOT 2
- Is it possible to track specified point in the video from the webcam? HOT 8
- track fast moving objects HOT 1
- thx
- question about windowed inference HOT 3
- how to track a specified x-y point? HOT 2
- Add points as they get visible throughout the video HOT 5
- Sample occluded points during training HOT 3
- where can I find the confidence value of prediction?
- huggingface space down?
- ValueError: mutable default <class 'hydra.conf.JobConf.JobConfig.OverrideDirname'> for field override_dirname is not allowed: use default_factory HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from co-tracker.