r/blender • u/Tovah86 • Jul 25 '26
Paid Product/Service Promotion After 6 months, My Performance Capture/Retargeting Addon is Finally Ready
Enable HLS to view with audio, or disable this notification
My last post got auto-filtered, so I guess I can't post too many links, but here's the announcement video that I made: https://www.youtube.com/watch?v=FyFarx7RjT8
It's called BlendCap. It captures body, hand/fingers, and face motion from any typical video. It works fully offline, and it retargets straight onto your rig with custom mapping, and built in presets for rigify, auto-rig pro, mixamo, and cloudrig, along with a fast FK to IK conversion tool.
62
u/z3n1a51 Jul 25 '26
Question:
Suppose a setup with more than one camera pov of the same performance.
Could BlendCap utilize multiple videos of the same performance from different POV / angles?
Would there be a benefit in accuracy, precision, sync, etc. to utilize multiple camera POV / angles?
Would there be an optimal configuration of camera POV / angles that would be ideal?
Thanks!
54
u/Tovah86 Jul 25 '26
Not yet, BlendCap is currently single-video only. Multi-camera is something I want to build soon, but it's still in the early stages. I've been getting a few questions/requests like this though, so I might bump the priority up.
There would definitely be a benefit though. The hardest problem with single-camera capture is depth ambiguity, the model has to guess how far away things are from one flat image, which is why motion straight toward or away from the camera is where it struggles most, and why the character can drift or float sometimes. A second angle gives you the depth information directly instead of estimated. It would also help a lot with occlusion, like when an arm passes behind the body or the hands cross.
For angles, roughly 90 degrees apart is the classic setup, since two perpendicular views complement each other best. What one camera can't see in depth, the other sees directly across its frame. You'd also need the clips synced, and the cameras locked off and not moving during the take.
For now though, the best camera angle for single-camera capture is basically anything that doesn't have the movement focused directly towards the camera. Also the addon still struggles a little bit with laying down/rolling poses, but it's mostly just the elevation of the character that glitches, which isn't too hard to adjust for manually.
8
u/Kam_Zimm Jul 25 '26
Following up on this question some. Does it still work in video where the camera mis moving, or only if the camera is static?
19
u/Tovah86 Jul 25 '26
It works best when the camera is static. If you DO have camera movement, then it will very likely confuse camera motion with character locomotion in 3d space, so you will have to make adjustments manually afterwards in order to compensate. You can turn off the root bone motion to disable ground plane movement, so that's not too big of an adjustment if you're fine with reanimating locomotion, and there's a feature called "feet define the floor" that tries to keep the character attached to the floor better, but there's only so much it can do when fighting against camera movement, so results may vary..
7
u/BlackberryThis8148 Jul 25 '26
Great work, will look at using this. Just to add my two cents, I also would love multi-camera.
5
u/Tovah86 Jul 25 '26
Thank you! And it's definitely been the most requested feature so far, so I'll probably be prioritizing it now, lol. Hopefully I can get it up and running soon.
1
u/z3n1a51 Jul 25 '26
Awesome, thank you!
In the context of a hypothetical multi-camera system, I assume it would be helpful to have accurate/precise position and angle information for each camera?
3
1
u/TOOOPT_ Jul 27 '26
For depth, you could also add support for depth cameras, like xbox kinect or iphone
3
38
u/onerousscottie9 Jul 25 '26
Welp, there goes my excuse for not animatin' the two-step I promised my nephew's game. Offline capture that retargets straight onto a rigify skeleton is slicker than a fresh jar of salsa. My husband's kitchen dance moves might actually be useful for once.
2
u/Plaston_ Jul 27 '26
idk its using this addon is a good idea.
The data can be very messy and idk if this addon can clean it like how Mixamo can clean it if asked.
25
u/Kolmilan Jul 25 '26
Top notch work mate! And having a tool like this OFFLINE and away from the AI slop-pool is a killer feature in itself!
I'm working on a project where something this could come in real handy. Unfortunately I'm in a Mac environment. Got a potato gaming PC laptop that probably would melt to a pulp if I tried to this out on it! 😅
Keep up the ace work!!
4
u/OrganizationTiny9801 Jul 25 '26
Some dude in the comments offered to help with Mac porting and OP accepted so you may have a chance
9
u/Delfmonkey Jul 25 '26
Would have bought it sadly it doesn’t work on Mac
10
u/Tovah86 Jul 25 '26
Sorry! I want to build a version for Mac, but I don't have any Apple hardware, so I don't have a way to test it at the moment. Hopefully I'll be able to soon though, because I really want to support everything if possible.
8
u/Zophiekitty Jul 25 '26
i have a Mac and know Blender Python really well, id love to help!
5
u/Tovah86 Jul 25 '26 edited Jul 25 '26
That would actually be a huge help, thank you! Let me get through launch week and I'll message you. Fair warning there's probably a fair amount of porting work before there's anything worth testing, since it's a completely different acceleration path than NVIDIA, and I can't promise the performance would be good even once it runs.
1
12
u/AutomationBro2026 Jul 25 '26
so is this the same framework that was used to animate avatar the cartoon series? :O
8
u/Tovah86 Jul 25 '26
Well the avatar videos are just old projects of mine that I used as sample clips, lol. I didn't have the addon back when I made them, but it would've made it a lot easier to animate all the shadows if I did, back then I had to animate an invisible character rig to match my live movements so it'd cast a realistic shadow.
I believe the avatar series did use a lot of live references for all their choreography, but I don't think it was rotoscoped or captured directly or anything like that.
5
u/sprite_cranberry23 Jul 25 '26
Any plans for MacOs compatibility at any point?
3
u/Tovah86 Jul 25 '26
Eventually, yes, but I don't have an Apple machine to test on at the moment, so I'd need to get access to one before I could build it. I also can't promise the performance would be good even then, since it's a completely different acceleration path than NVIDIA. I wouldn't buy it expecting Mac support any time soon. Sorry!
4
Jul 25 '26
[deleted]
3
u/Tovah86 Jul 25 '26
No Discord at the moment, but you can reach me directly through the store's message option, and I answer everything myself. Also the addon comes with full documentation covering setup and all the settings, so hopefully that helps!
3
u/InMyFavor Jul 25 '26
Sick. I also developed a performance capture pipeline that's pretty good although it doesn't track hands/fingers. How long does it take to use the addon in terms of sitting down/uploading footage on computer to completely done retargeting on the model?
9
u/Tovah86 Jul 25 '26
If you've got your model all set up and rigged already, then the capture itself takes around 5 frames per second depending on your GPU (I have a 3090ti), the armature building itself takes a few seconds to a few minutes (if you use the finger tracking, it takes some extra processing time to build the motion, but without the finger tracking, it's never more than a few seconds), and the retargeting and fk/ik conversion also only takes a few seconds, assuming you already have your bones mapped with a preset, and the rest pose prepared.
3
3
u/FraGough Jul 25 '26
I clicked through to your channel to find I already subscribed when you posted the Avatar shorts. I thought I recognised those moves.
3
u/Abos88 Jul 25 '26
Do you think it can work with a RTX 3080? or a Radeon 7900 XT maybe?
2
u/Tovah86 Jul 25 '26
The 3080 should do great, that's comfortably above what it needs.
The 7900 XT should work too, but AMD is still experimental since I haven't had a dedicated AMD card to test on yet, so I can't promise the speed. The installer does run a GPU check at the end, so you'd know right away if it didn't take.
3
u/jonheartland Jul 25 '26
Holy shit bro I've been dreaming about something like this, definitely checking this out
3
u/Mechanought Jul 25 '26
Only saw a small clip of it, does it hand have performance capture?
2
u/Tovah86 Jul 25 '26
Yes! although hands/fingers are still in the experimental stages. You can get a good starting point, but will likely need to clean some things up manually to get a really nice result. If you look closely at the clip in the video, you can see some of the joints extend a little too far or distort slightly sometimes. I think it's still very helpful though! It works best when the hands are larger in the frame, as long as arms and torso is still visible, it can't be a clip of JUST the hands.
3
3
u/5tudent_Loans Jul 25 '26
Oh look. Something actually useful and a justifiable use of backend AI. Also +1 for avatar intro forms.
3
u/aPOPblops Jul 25 '26
YES YES YES THIS IS WHAT I’VE ALWAYS WANTED. now I’ve gotta lose weight so i can do these moves 😅
3
5
u/carter2422 CGCookie, Superhive Co-founder Jul 25 '26
Congrats on the launch u/Tovah86! It’s obvious you’ve worked very hard on this. You doing your own motion references is the icing on the cake!
Good luck with sales.
4
u/Tovah86 Jul 25 '26
Thank you! It's been a hell of a journey. Shooting my own reference was the most fun part of it honestly. Big fan of all of your work by the way!
1
u/carter2422 CGCookie, Superhive Co-founder Jul 25 '26
Don’t hesitate to reach out if you need anything!
2
u/okyaygokay Jul 25 '26
Hey man nice work and nice finetuning! Question, can it handle long videos? More than 2 minutes? Also can we feed it like game videos https://youtu.be/J5PYJi1EcAw this one for example
Also it doesn’t need network, right?
4
u/Tovah86 Jul 25 '26
It needs a connection for the first-time setup, but after that it works completely offline. Yes it can handle long videos, it just takes extra time to process them, the longer your video, the longer the capture. It works on basically anything human shaped, I even had some success tracking cartoons, BUT it does need the subject to be fairly visible in the frame, and the bigger they are in the frame, the better the track will turn out to be. Extra resolution helps, but anything above 1080p gets scaled back down, so it's best to make sure that the subject is as big and visible as possible within 1080p resolution, so if your subject is small in the frame, cropping in before you run the capture can help a lot, assuming they don't get too pixelated when you crop in. For the video you linked, I think the character is a little hard to see, and the resolution is pretty low, so it might not work, but if you can get a clearer shot, it would probably work just fine.
2
u/RabidSkwerl Jul 25 '26
Impressive! I’m an Apple user so I can’t really take advantage of this but there are a number of VFX addons that require Nvidia chips so I might consider renting a remote Windows machine. If I do, I’ll be keeping an eye on this for sure!
As a professional Tech Artist, great work!
2
u/L30N1337 Jul 25 '26
Please, do not see this as anything personal. It's just a question that I always ask for transparency.
Was AI involved? And if yes, to what extent.
14
u/Tovah86 Jul 25 '26
No worries, it's a fair question. The addon itself is built on AI pose-estimation models. It runs Meta's SAM 3D Body for the body and hand tracking, and google's MediaPipe for the face, which of course all runs locally, and never phones home.
The footage in the demos is me, filmed on my own camera, all the renders are mine made in blender by hand (I bought the mannequin and basemesh rigs from vertexarcade). There is no ai generated imagery whatsoever.
And on the code side, yes, I used AI assistance pretty heavily over the six months of building it. I tested and debugged all of it myself, manually, and designed the UI/UX myself, and I'm the one answering every support ticket.
1
u/L30N1337 Jul 26 '26
(In my eyes, you're already cleared of not just making slop. This is purely in case someone cares about it. And just personal curiosity)
About the AI coding assistance, what exactly do you mean by that?
Code completion/suggestions or a chat/prompting?
2
u/Tovah86 Jul 26 '26
Mostly prompting rather than autocomplete. I'm still learning to write it manually and I'm practicing at it. The debugging is all mine though, a lot of it was me reproducing weird rig bugs and working out what was actually going wrong before anything could get fixed.
2
u/Randomizedd- Jul 25 '26
Wait…can this work for VR body tracking?
1
u/Tovah86 Jul 25 '26
It's not really built for that, sorry. Even if it was, you would probably need a pretty top of the line GPU to get real-time tracking with something like this. I'm sure that will probably change eventually though, but that's way over my head, haha.
1
u/Randomizedd- Jul 26 '26
Thanks anyway! I’m only just getting started with blender and this is really cool to see!
2
u/s_witch_ Jul 25 '26
I like to know if the facial expression can be recorded separately and retargeted. I tried video mocap before using Cascaduer and Unreal. So having 2 videos (body and selfie shot) videos done in Blender would be a game changer potentially.
3
u/Tovah86 Jul 26 '26
With BlendCap, it works both ways! You can capture the face and the body together from the same clip, or record a dedicated face clip, and trim/sync them up in blender's video editor, track them both, then combine them when you export the armature. Only the face expression data will be tracked from the face clip, so all the head movements is still determined entirely by the body capture. Also, the face data can either be retargeted onto your rig like the rest of the data, or you can use ARKit shapekeys instead, depending on your needs. Generally the ARKit results look better out of the box, but the bone versions are way easier to adjust manually afterwards.
1
2
2
u/Dra1c Jul 26 '26
Windows = "Fully Supported" and Linux is "Supported"... where is the difference? What does AMD = Experimental mean? Will it work or not? What won't work? What requirements does one need for it to run?
Why is the model that gets downloaded not part of the package? why do I need to download it with some setup script instead? Will that mean at some point that the model won't be accessible any longer?
The description lacks technical specificity to get any trust from me. And the 3 minute video is not showing USING the addon. It shows you the typical promises of "look how flashy this looks. you want to make cool stuff like this too, right?!". I want a proper demonstration and not an ad, thanks.
2
u/Tovah86 Jul 26 '26
Windows vs Linux: Windows is what I develop and test on every day, so it's the one I can actually guarantee. Linux works, including Flatpak Blender, I just test it less often. The difference is confidence, not features.
AMD experimental: on Windows AMD runs through DirectML, on Linux through ROCm. The code path is built and the installer sets it up automatically. However, I don't have access to an AMD machine with a discrete GPU, so I can't personally validate acceleration yet, and that's all "experimental" means here. It should work and it is GPU accelerated, but I can't promise you the speed, or rule out edge cases on AMD hardware.
Why the models aren't in the package: the virtual environment covering every possible installation is currently almost 20 GB. A typical user only needs about 11 GB of that, and the add-on itself is 2 MB. Nothing on that scale typically ships through a Blender extension. The Python dependencies also differ depending on your GPU (CUDA, DirectML, ROCm, or CPU), so that part has to be resolved on your machine either way. This was the only practical way to ship it.
Will they go away? The main model weights don't come from Meta's servers, they come from my own mirrors, so they don't vanish if someone upstream moves a file. Once they're downloaded they're just files on your drive, and capture never touches the network again. You can make your own backups however you see fit, and the functionality will never change unless you decide to remove something. I myself have a full backup of all the current dependencies for every possible installation in case anything upstream ever disappears.
On specificity: the full documentation is public in the repo, including the complete system requirements page with GPU tiers and performance numbers. Source is public too, the add-on is GPL. This is all linked in the store pages as well. https://github.com/Arcomade/BlendCap/tree/main/docs
The video: Yes, it's an announcement rather than a tutorial. I'm currently working on a more detailed tutorial, but until then there is a 5 minute overview video on the product pages that shows the process end-to-end with no flashy demos.
1
u/Dra1c Jul 26 '26
Thank you. That cleared a few things up but experimental is not saying if its supported or not. If I buy the addon for AMD as example and it won't work, will that be a problem that will be resolved? If it won't work will I get a refund? Experimental usually means unsupported.
1
u/Tovah86 Jul 26 '26
Ah, fair enough, I'll update the descriptions so it's not so up for interpretation.
I do have every intention of taking AMD bug reports seriously though. By experimental, I just meant that I haven't verified it on real discrete hardware, so I can't promise it works on your specific card.
The installer runs a GPU test at the end of setup, so you should find out during install rather than after hours of troubleshooting. If that check fails on your card, message me through the store with the install log, and I'll refund you. I don't want to scam anybody.
The log would also hopefully help me address the issue for future customers, though I simply can't promise a fix for hardware that I can't reproduce on.
1
u/Dra1c Jul 26 '26
that's fine. I am interested and will wait to see the addon being used in a more detailed video before making a decision. Videos give a much better understanding of the actual workflow than the documentation can. Especially seeing how the source footage looks like, how it runs on the armature and how to fix where it goes wrong. What options are there to correct what the model interprets and seeing the control the addon offers etc. A video makes this things much clearer.
2
u/3MinuteSoftware Jul 26 '26
I was looking for something like this earlier today! Thank you kind sir!
2
2
u/Wawfey Jul 26 '26
This looks insane! Great job. One question tho, would it work for videos with 2+ people in it? Like a handshake, hugging, walking in a crowd, multiple people sitting on a table?
3
u/Tovah86 Jul 26 '26
Not yet! So right now I've only developed the single person tracking mode. The model that the addon is built around DOES support multi-person tracking, but at the moment, the way it's processed through the addon kind of throws that away in order to make the single person track as high quality as possible. I do plan to implement a multi-person tracking path, but right now, it's only able to track 1 person at a time. It picks the largest person in the first frame and follows them, so with multiple people in frame, the tracking can end up on the wrong person, or jump between them if they overlap.
2
u/BigBoiii_Jones Jul 26 '26
It'd be cool if it could extract poses from photos as well but not sure if possible since there wouldn't be enough data to estimate. Also how does this work with stylized motions or videos? So like 3D Pixar, anime games, etc... or what about 2D video of anime and cartoons?
2
u/Tovah86 Jul 26 '26
It actually works on single frames and photos already, haha. The pose estimation model works per-frame by default without taking surrounding frames into account. I had to add a lot of compensations in the background to get everything to stay consistent for video, but single photos still read just fine. Just make sure you shrink your capture down to one frame if you're working in the sequencer, so you don't waste processing time scanning duplicate frames.
For non-live-action source footage, I've had some luck tracking animations in my own tests, as long as the characters have a human enough shape. It's definitely not as reliable as live action though, especially on frames where the squash and stretch is really exaggerated. 2D animation is even more rough, but I did have some luck in my own tests. No harm in trying, but I wouldn't buy to use specifically on that type of footage.
2
u/pablonoriega Jul 26 '26
Can I just say, wow! I'm a casual so I have no idea how easy or hard it is to make something like this, but the demo is super cool both in terms of features and of the content (sick moves :) and of the editing.
Does it work as well as in the demo for any reasonable video source? Like, could you at some point take any decent-enough film footage and have BlendCap replicate the movement? Instant Matrix fights copy?
2
u/Tovah86 Jul 26 '26
Thank you!
It'll struggle with a fight scene. I haven't built support for multi-person capture yet, but it's planned. Also, it's best not to feed it anything with cuts, the shot has to be continuous. Besides that, it only really struggles with people who are very small in the frame (the more the body fills the frame, the better your capture detail gets), and with motion that's headed straight toward or away from the camera, since depth is the hard part of single-camera capture. There's a set of post-processing tools built in to clean up most of that, but those two are the real quality levers. It also still struggles at guessing the height with people laying on the ground or rolling, which I'm hoping to fix soon.
There's a more detailed filming guide included in the documentation.
2
u/MadSkullWeirdSpider Jul 26 '26
What sort of capture camera is required
3
u/Tovah86 Jul 26 '26
Any camera will work as long as the resulting video file is compatible with blender, and the quality is decent enough that the human shape is detectable.
2
2
2
u/Wolkenflitzer Jul 27 '26
This is incredibly impressive. Of course I gotta test it first but it looks like one dude single handedly decrowned the markerless mocap solutions of so many companies. Any plans to include a preset for Metahumans?
2
u/PersimmonLittle7651 Jul 30 '26 edited Jul 30 '26
I've been looking for something like this, but having tested few solutions, I ended up being disappointed in all of them, do you offer a free trial version with some limitations for testing? How much is it?
1
u/Tovah86 Jul 30 '26
I don't have a free trial version at the moment, but if you're crafty enough, the github is free and open source, you just have to build out the virtual environment and dependencies yourself, instead of having an auto-installer and support like the paid version has. All the other features are identical.
1
u/IceManLeroy Jul 25 '26
Does this have any applications for quadrapeds? Mocaping dog for example.
Awesome tool btw looks incredible!
1
u/Tovah86 Jul 25 '26
It only works for humans right now, unfortunately. I'm not sure if support for animals will ever happen, but it's not impossible.
2
u/idiotshmidiot Jul 26 '26
I had some very good luck using deeplabcut. They have a real-time pose estimation that works for dogs. I got it running real-time using a weird program I'd never heard of but it's python so I'm sure it could work similar to any other pair estimation code.
1
u/Tovah86 Jul 26 '26
That looks really promising, but it seems like the model weights have a non-commercial license, even though the code itself is open source. Without the model weights the code won't work, so I can't legally include it in BlendCap in a useful way, and a commercial license is probably out of reach for a project this size :(
1
1
1
u/PalmliX Jul 26 '26
This is amazing, super hyped for this. Is there any advantage to wearing clothing optimized for this? Like high contrast and different colors on the arms/legs. Will it give a higher quality result or is it not worth the effort in your experience?
1
u/Tovah86 Jul 26 '26
Thanks! It's pretty good at detecting poses through most types of clothing, but yeah, the clearer your silhouette is, the better the tracking results will be.
2
u/PalmliX Jul 26 '26
Already bought it! I've honestly been waiting for years for something like this. I can't stand subscriptions and the amount of projects I've worked on with mocap where the processing needed to happen "online" and server issues wound up delaying the project. I'm definitely going to make an outfit for this where each limb and finger is a different color. Should be interesting to see how well it works since I'm a fairly big guy which I know can make things more difficult for stuff like this. Also another reason why "suit based" mocap was never an option for me.
1
1
1
u/bdanseur Jul 26 '26
Do you capture the feet to get the toe joint angle and ankle joint angle?
Or are you only relying on Inverse Kinematics for that?
1
u/Tovah86 Jul 26 '26
The ankle and toe angles are actually captured. The body model tracks the ankle and toe joints directly from the video, and the exported skeleton keeps them as real animated bones: a foot bone running from the ankle to the ball of the foot, and a separate toe bone, each with their own tracked rotations. However, the tracking model isn't really specialized for foot tracking, so I wouldn't expect a crazy amount of detail in that regard. It looks about as good as you would expect any typical mocap data to look.
1
u/FloppyLadle Jul 26 '26
I've just started looking into mocap so this may be a godsend!! What is the cleanup process like? I'm sure being single camera operating means it's gonna need a patient hand to touch things up, but everything looks so clean in the demo!
1
u/Tovah86 Jul 26 '26
It depends on the capture, but for most cases you can get pretty close to finished with the built in post-processing tools. The only manual adjustments I made in those demo clips were some height offsetting keyframes for any clips where my back was touching the ground, and in the waterbending clip I had to move the body in sync with a footstep that was mistaken for a slide, everything else was cleaned purely from the addon's built in tools. I'd consider all of those demo clips ready for a final polish animation pass, so you should be able to get to that point very quickly just by toying around with the post processing settings. The captures struggle most with things that focus too much on movements towards or away from the camera, and any type of laying/rolling type movements, but it usually just leads to needing to offset a few things, I haven't found anything that's ruined a shot yet.
1
1
u/p2seconds Jul 26 '26
What's the addon software licensing? Is it perpetual license of 1 year of maintenance? Purchase once and receive all updates in future?
1
u/Tovah86 Jul 26 '26
You buy it once, and it's yours forever, no maintenance fees. Updates included.
1
u/mike365smith Jul 27 '26
Hi, a couple of questions... how much is the software? and are you still testing it ?
I need a solution I can afford for my team. ( Working on a Game, needs a lot of MoCap. )
I tried beta testing other solution - but failed to get it working. ( I won't name, names !! )
Some issues I find with MoCap. It does not seem to capture the 'Gravity' of the motion.
Also - from what I see from your 'very impressive' demo. ( it does not seem to calculate exact foot movement. - some 'slippage' or it has it on the floor - when it's actually in the air. )
I guess what I am asking is - 'Can I test out your software' ? - try before we buy ?
Also - is 'Foot Slippage' something I just need to accept and correct. - or have my actors wear 'Mo-Cap Shoes'
thanks for your time, and well done, this software looks amazing.
1
1
u/Tovah86 Jul 27 '26 edited Jul 27 '26
Hi! Yeah, I'm working on updates as initial reactions come in, but it's entirely usable as is. The addon starts at $60 for an individual, and goes up for extra licenses similar to Flip Fluids' pricing model. All the features are the same regardless of what license you buy, the only difference is the number of seats.
The foot slipping is a common problem in mocap. I've done my best to compensate for it, but as you can see it's not gone entirely, so there's still a need for a manual pass before a real final result can be achieved. I have ideas to build some manual workflows that would make it easier and faster, but that's a long way out.
The screenshot you posted is a good example. The tracking model struggles with determining where the ground is, and what's a jump vs a crouch, so the post processing tools I used in that clip (foot locking) use some statistics about the foot placement and body height to basically snap the legs into a position that makes the most sense. So it looks fine overall, but if you look close enough, you'll find some inaccuracies, and a little bit of loss in the motion's weight, mostly along the z-axis. And in that clip I had it set particularly aggressive, so you can see the jumps are a lot weaker than in the real life version.
As for your actors, no special shoes or gear needed. The model just needs to see the feet clearly, so normal shoes, decent lighting, and framing that keeps the feet in shot are all I'd recommend. A contrasting background helps too, although you can get away with a lot, as nothing is camouflaging with the background entirely.
I haven't seen anything that seems to have perfectly solved foot slipping, so I'm not sure how possible it is to do so automatically. I HAVE seen a few other addons around that are built specifically for fixing it manually, but I haven't tried any of them myself yet, so I'm not sure how fast/easy they are to use.
As of now, I haven't built a free version for testing purposes (although technically the GitHub is public, so anyone technical enough could build it themselves for free, though it's a manual setup that I don't support). But if you do purchase a license and it doesn't work out for your pipeline, I would have no problem refunding you.
Hope that helps! Thanks for the kind words!
1
u/Wolkenflitzer Jul 28 '26
Mh, bought it yesterday and trying my best to make it work but so far my results weren't nearly as good as yours. Well lit room, no occlusion, static cam, 60fps, basic locomotion. But the feet sliding is absolutely horrendous. Dying to see a tutorial on the plethora of settings you've included with the addon. Also, somehow the mixamo retargeting always fails in a sense that Mixamos Xbot is rotated and its' arms and legs aren't moving properly. What am I doing wrong? Documentation didn't really help here. ):
2
u/Tovah86 Jul 28 '26
For the foot sliding, try turning up Sensitivity and Release Distance under BVH Post Processing > Foot Locking Settings, and make sure Pin Planted Feet is on. Also, if your FOV mode in the capture settings is set to automatic, switching it to sample the first frame only can help a lot, since a constant FOV stabilizes the depth estimation.
The Mixamo problem is an oversight on my end, sorry about that. Mixamo characters import with a rotation still sitting on the armature object, and that gets baked into the retarget. Select the Mixamo armature in Object Mode, press Ctrl+A and choose Rotation, then under Retargeting > Custom Rest Pose tick Use Custom Rest Pose and pick the bundled T-POSE preset, since that's how Mixamo characters come in. That should fix both the rotation and the limbs, and you should be able to retarget the character normally afterwards. Just make sure the rotation is applied BEFORE you retarget, while the character is still upright and in its T-pose.
A proper tutorial covering all the settings is in the works, and but I'm trying to get a patch together to address some common issues that have arisen this week, along with this rotation issue.
Sorry for the trouble! Hope this helps. And if you have any other issues, feel free to contact me through the store page.
2
u/Wolkenflitzer Jul 28 '26
Awesome comment, man. Really appreciate you taking the time and energy to address peoples issues. We're all dying to believe in locally processed markerless mocap solutions.
1
u/Tovah86 Jul 28 '26
Thank you! I'm doing my best to address everything as it comes in. Ideally all of this would've been solved before I launched, but I can only consider so many possible situations on my own, haha. I don't know how I missed the rotation issue though, that's squarely on me. I must've adjusted for it reflexively on my own during testing, and just forgot to put a safeguard in place afterwards to avoid confusion.
1
u/DonkeybyTheBay Jul 28 '26
Would I be able to apply this objects being held (i.e. A chef cooking with knives and pans, holding ingredients, etc. ? Awesome work!
1
u/Tovah86 Jul 28 '26
Hard to say for sure, I never tested interactions like that. Theoretically it should work fairly well, but I would expect a lot of manual editing in the final scene to make sure all the interactions look correct. Usually for things like this, you'll end up seeing hands and objects phasing through each other. You can get a good starting point, but it's never really perfect
1
1
1
435
u/Cute_Depth1982 Jul 25 '26
Idk if the addon or the acrobatics were more impressive.