⭐️ If you like this project, please consider giving us a star on GitHub! ⭐️

Learn More:

Support DeepLabCut:

In the News:

Key Publications:

Primary research papers:

• Mathis et al, Nature Neuroscience 2018 or free link: rdcu.be/4Rep

• Nath*, Mathis* et al, Nature Protocols 2019 or free link: https://rdcu.be/bHpHN

• Kane et al, eLife 2020

• Lauer et al. Nature Methods 2022

Review & educational papers:

• Mathis & Mathis Deep learning tools for the measurement of animal behavior in neuroscience Current Opinion in Neurobiology 2020

• Mathis, Schneider, Lauer, Mathis A Primer on Motion Capture with Deep Learning: Principles, Pitfalls, and Perspectives Neuron 2020

Preprints/always open access versions:

[1] arXiv (April 2018): Markerless tracking of user-defined features with deep learning (published in Nature Neuroscience)

[2] bioRxiv(Oct 2018):On the inference speed and video-compression robustness of DeepLabCut
(presented at COSYNE 2019)

[3] bioRxiv (Nov 2018): Using DeepLabCut for 3D markerless pose estimation across species and behaviors (published in Nature Protocols)

[4] arXiv (Sept 2019): Pretraining boosts out-of-domain robustness for pose estimation

[5] arXiv (Oct 2019): Deep learning tools for the measurement of animal behavior in neuroscience (published in Current Opinion in Neurobiology)

[6] arXiv (Sept 2020): A Primer on Motion Capture with Deep Learning: Principles, Pitfalls, and Perspectives (Published in Neuron)

[7] arXiv (Nov 2020): Pretraining boosts out-of-domain robustness for pose estimation (published at WACV 2021 & ICML workshop 2020)

[8] bioRxiv (August 2020): Real-time, low-latency closed-loop feedback using markerless posture tracking (published in eLife)

[9] bioRxiv (April 2021) Multi-animal pose estimation and tracking with DeepLabCut (published in Nature Methods)

What’s New:

• July 2023! We released an updated preprint and new SuperAnimals into the Model Zoo, see modelzoo.deeplabcut.org for more information!

• April 2022! We released a major update to the code with strong multi-animal support, tracking, and many new tools and features! Please check out the new docs and video tutorials to get started, and it’s now published in Nature Methods!

• March 2020! We released the DLC Model Zoo. Use trained networks without any installation, coding, or retraining: DeepLabCut Model Zoo homepage

Example use cases:

(click on the image to see more details and other use cases!)

Easy Install:
Simply have Anaconda installed, click to download the appropriate file below and follow these instructions to launch your environment with DeepLabCut (plus GUIs) already installed! *But, for Mac M1 or M2 chip users, please see here.

Launch the program “Anaconda Prompt” and in the folder where you downloaded the file type:

conda env create -f DEEPLABCUT.yaml

*Note, if you have an M1 chip macbook, please see the main repo for a specialized install: https://github.com/DeepLabCut/DeepLabCut

Check out other applications and share yours with #DeepLabCut

DeepLabCut Model Zoo

Here we provide model weights that are already trained on a wide range of animals & scenarios.
You can use these models for video analysis (inference) without the need to train new models yourself. Or, if you need to fine-tune the model, you can use 10X less data the the original DeepLabCut weights 🔥.

NEWS: Read our first preprint about the Model Zoo:

SuperAnimal pretrained pose estimation models for behavioral analysis. Shaokai Ye, Anastasiia Filippova, Jessy Lauer, Maxime Vidal, Steffen Schneider, Tian Qiu, Alexander Mathis, Mackenzie Weygandt Mathis

Model Zoo SuperAnimals 🔥

We present a new framework for using DeepLabCut across species and scenarios, from the lab to the wild. We are alpha releasing these models to allow users to use DeepLabCut in two common scenarios with zero-training required. These models are trained in the TensorFlow 2.x framework (and PyTorch weights are downloadable and will soon be fully integrated into DLC3.0).

SuperAnimal-Quadruped: this has been trained on over 40K images of animals that have 4 legs; from mice, rats, horses, dogs, cats, to elephants and gazelles. It has 39 key points.

SuperAnimal-TopViewMouse: this has been trained on over 5K mice from videos across diverse lab settings; there is known a C57 bias, but some CD1 (white) mice are also included. It has 26 keypoints.

Test your videos on Google COLAB!

Simply click on the blue icon below to try it out on your videos now (and watch our tutorial if you need assistance):

Test your data on 🤗 HuggingFace!

You can take a frame from your video and test our models. Simply upload your image (we do not collect this data), and test a mode!

• https://huggingface.co/spaces/DeepLabCut/DeepLabCutModelZoo-SuperAnimals

• https://huggingface.co/spaces/DeepLabCut/MegaDetector_DeepLabCut This app was built during the 2022 DLC AI Residency Program

Hosted user-contributed models:

Below is a list of published models kindly provided in the DeepLabCut framework. All the models below are trained in the tensorflow 1.x framework (typically 1.13.1-1.15). If you use a model from the zoo in your work, please see how to cite the model below.

Primates:

• full_macaque: From MacaquePose! Model contributed by Jumpei Matsumoto, at the Univ of Toyama. See their paper for many details here! And if you use this model, please also cite their paper.

• primate_face: Model contributed by Claire Witham at Centre for Macaques, MRC Harwell, UK. This model is trained on photos and videos of rhesus macaque faces – mostly forward facing or in profile. Includes range of ages from infant to adult and both sexes. Shows reasonable transference to other primates especially other macaque species. Read more here!
  

• full_human: This model can be used to track points shown in full-body poses of humans. This is a network presented in DeeperCut, trained on the MPII-Pose dataset. It expects smaller images as input.

Other animals:

• horse_sideview: A pre-trained horse network! Dataset/networks from Mathis et al. 2019 arXiv/WACV 2021. Note, this currently works on videos as shown, horses of different sizes and coat colors, but walking left to right. If you use this model, please cite our paper.

• mouse_pupil_vclose: Model contributed by Jim McBurney-Lin at University of California Riverside, USA.A pre-trained mouse pupil detector. Video must be cropped around the eye (as shown)! Trained on C57/B6 mice ages 6-24 weeks (both sexes). Read more here!

• full_cheetah: A pre-trained full 25 keypoint cheetah model. This is trained in TF1.15 with a DLC-ResNet-152 w/a pairwise model (see Joska et al. 2021 ICRA for details and citation!). Note, the network was trained on large GoPro videos (2704x1520), so large videos are the expected input.

  Please note, that full_cat and full_dog released in March 2020 have been sunset as of Nov 2022; instead, use the SuperAnimal- Quadruped model.
  

Help us make this even better: more data, more models, more results!
(See who has generously contributed data already!)

How to cite a model: If you use a model from the zoo in a publication, we ask you please cite the model paper (i.e., see model description for citation), and/or for our unpublished models (non-commercial use only), please cite Ye et al 2023!

A huge thank you to those that have contributed to the DeepLabCut Model Zoo
by labeling data and/or providing models!

Models:

• Jim McBurney-Lin, University of California Riverside, USA (mouse eye)

• Claire Witham, Centre for Macaques, MRC Harwell, UK (primate face)

• Rollyn Labuguen, Jumpei Matsumoto, University of Toyama, Japan (full macaque)

Data:

• Sam Golden, University of Washington

• Jacob Dahan, Columbia University

• Katherine Shaw, U Ottawa, CAN

• Nick Hollon, Salk Institute for Biological Studies

• Loukia Katsouri, Sainsbury Wellcome Centre, UCL

• Lauren (Nikki) Beloate, St. Jude Children's Research Hospital

• Leandro Aluisio Scholz, Queensland Brain Institute, The University of Queensland

• Diego Giraldo, Johns Hopkins University

• Alice E Kane, Harvard Medical School

• Nina Harano, Columbia University (Columbia College)

• Vasyl Mykytiuk, Max Planck Institute for Metabolism Research

• Khaterah Kohneshin, University of Pittsburgh

• Michelle Joyce, Temple University

• Alejandra Del Castillo, Georgia Institute of Technology

• Tara O'Driscoll, UCL

• Alice Tang, Columbia University

Pretraining Boosts Out-of-Domain Robustness for Pose Estimation

This paper was published Winter Conference on Applications of Computer Vision (WACV) 2021! [arxiv version][WACV Paper][Suppl].

This paper introduces the new benchmarks, and new SOTA EfficientNets for DeepLabCut. Note, a shorter workshop version of our paper was presented at the Uncertainty & Robustness in Deep Learning Workshop at ICML '20.

If you use this data or are inspired by this work, we ask you to please cite:

@InProceedings{Mathis_2021_WACV, author = {Mathis, Alexander and Biasi, Thomas and Schneider, Steffen and Yuksekgonul, Mert and Rogers, Byron and Bethge, Matthias and Mathis, Mackenzie W.}, title = {Pretraining Boosts Out-of-Domain Robustness for Pose Estimation}, booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)}, month = {January}, year = {2021}, pages = {1859-1868} }

Motivation

Pose estimation is an important tool for measuring behavior, and thus widely used in technology, medicine and biology. Due to innovations in both deep learning algorithms and large-scale datasets pose estimation on humans has gotten very powerful. However, typical human pose estimation benchmarks, such as MPII pose and COCO, contain many different individuals (>10K) in different contexts, but only very few example postures per individual. In real world application of pose estimation, users want to estimate the location of user-defined bodyparts by only labeling a few hundred frames on a small subset of individuals, yet want this to generalize to new individuals. Thus, one naturally asks the following question: Assume you have trained an algorithm that performs with high accuracy on a given (individual) animal for the whole repertoire of movement - how well will it generalize to different individuals that have slightly or a dramatically different appearance? Unlike in common human pose estimation benchmarks here the setting is that datasets have many (annotated) poses per individual (>200) but only few individuals (1-25).

To allow the field to tackle this challenge, we developed a novel benchmark, called Horse-10, comprising 30 diverse Thoroughbred horses, for which 22 body parts were labeled by an expert in 8,114 frames. Horses have various coat colors and the “in-the-wild” aspect of the collected data at various Thoroughbred yearling sales and farms added additional complexity.

Moreover, we present Horse-C to contrast the domain shift inherent in the Horse-10 dataset with domain shift induced by common image corruptions.

The data:

• >8,000 expertly labeled frames across 30 individual thoroughbred horses (called Horse-30).

The Tasks:

• Horse-10: Train on a subset of individuals (10) and evaluate on held-out “out-of-domain” horses (20).

• Horse-C: We apply common image corruptions (Hendrycks et al., 2019) to the Horse-10 dataset. The resulting Horse-C images are corrupted with different forms of digital transforms, blurring filters, point-wise noise or weather conditions. All conditions are applied following the evaluation protocol and implementation by Michaelis et al. 2019. In total, we arrive at 75 variants of Horse-C (15 different corruptions at 5 different severities), totalling over 600K frames.

Train:

The ground truth training data is provided as 3 splits of 10 Horses each. The download provides you a project compatible with loading into the deeplabcut framework, but ground truth labels/training data can be easily loaded in pandas to accommodate your framework (example loader here).

Please do NOT train on all three splits simultaneously. You must train independently (as some horses can be considered out-of-domain in other splits for evaluation!). Integrity matters!

The download also includes all of Horse-30 images and annotations (thus is ~850MB).

NOTE: Horse-10/Horse-30 dataset (c) by Rogers, Byron and Mathis, Alexander and Mathis, Mackenzie W. Horse-10 is licensed under a Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) License.

Evaluation Metrics:

• average PCK@0.3, within domain, and out-of-domain (held out horses); what fraction of machine-applied points fall within a specific range of human-labeled ground-truth labels; we use a matching threshold of 30% of the head segment length (nose to eye for horse), which is computed by taking the median for all annotated images per horse.

• Normalized Error (RMSE). Due to the variant in sizes of the horse, we provide RMSE in relation to the eye-to-nose distance. 1 equalling the full eye-to-nose distance (on average of ~18 pixels).

Evaluation:

To evaluate on out-of-domain horses, please run evaluation on each splits held-out test images. If you have any questions, we are happy to help, so please reach out to: alexander.mathis@epfl.ch.

Horse-10 Leaderboard on PapersWithCode: