Latest 0.2.0
Homepage https://github.com/trevphil/TechniqueAnalysis
License MIT
Platforms ios 11.0
Frameworks UIKit, AVKit, CoreML, AVFoundation, Vision, CoreMedia
Authors

Description

This is a CocoaPod which processes video footage of a user doing an exercise, and provides feedback on the user’s form. The general idea is to capture a video and convert it to a timeseries, where each data point contains an array of body point locations in 2D space, along with a confidence level on the accuracy of each body point.

pipeline

It is already possible to estimate body points (called pose estimation), for example using the project OpenPose. tucan9389 has provided an excellent starting point for the code implemented here, based on his project which predicts body points in real-time.

To translate from pose estimation to technique analysis, we need to train a model on timeseries with correct and incorrect form for various exercises. Richard Yang and Steven Chen from Stanford University have had relative success with Dynamic Time Warping (DTW) to solve this type of problem, as described in this paper.

dtw

Additional research by Keogh et al. in their paper "Fast Time Series Classification Using Numerosity Reduction" provides a promising, efficient algorithm to train a kNN model with DTW (with k=1 and a dynamic warping window) for a use case similar to the one described here.

The ML frameworks compatible with CoreML on iOS (e.g. scikit-learn) are currently lacking a suitable model to pre-train a kNN-DTW model and bundle it into an iOS application. Because of this, the best option may be to implement the algorithm manually and ship it with the application. The kNN-DTW algorithm here is written in Swift, but writing the algo in C would provide a much faster runtime. For further explanation on kNN-DTW, Mark Regan has implemented a Python version of kNN-DTW similar to the one described by Keogh et al. It can be found here.

Example

To run the example project, clone the repo, and run pod install from the Example directory first. You should use Xcode’s Legacy Build System to correctly include the videos in Bundle Resources (File → Workspace Settings → Shared Workspace Settings → Build System).

demo

Requirements

iOS 11, Swift 4.2

Installation

TechniqueAnalysis is available through CocoaPods. To install
it, simply add the following line to your Podfile:

pod 'TechniqueAnalysis'

Author

trevphil, trevo[email protected]

License

TechniqueAnalysis is available under the MIT license. See the LICENSE file for more info.

Latest podspec

{
    "name": "TechniqueAnalysis",
    "version": "0.2.0",
    "summary": "A CocoaPod to track body motion and analyze exercise technique.",
    "description": "This is a CocoaPod which processes video footage of a user doing an exercise and provides feedback on the user's form. The general idea is to capture a timeseries, where each data point contains an array of body point locations in 2D space, along with a confidence level on the accuracy of each body point. Timeseries are compared using k-Nearest Neighbor and Dynamic Time Warping to find the nearest neighbor.",
    "homepage": "https://github.com/trevphil/TechniqueAnalysis",
    "license": {
        "type": "MIT",
        "file": "LICENSE"
    },
    "authors": {
        "trevphil": "[email protected]"
    },
    "source": {
        "git": "https://github.com/trevphil/TechniqueAnalysis.git",
        "tag": "0.2.0"
    },
    "swift_version": "4.2",
    "platforms": {
        "ios": "11.0"
    },
    "source_files": [
        "Models/**/*",
        "Utils/**/*",
        "Views/**/*"
    ],
    "frameworks": [
        "UIKit",
        "AVKit",
        "CoreML",
        "AVFoundation",
        "Vision",
        "CoreMedia"
    ]
}

Pin It on Pinterest

Share This