How can I smooth out posenet fluctuations?

user_13043070 0
Viewed 289

I'm trying to create a hand to mouse movement system, however I am experiencing value fluctuations due to the PoseNet model. The way it works is this: PoseNet runs in the browser, then sends request to a local Flask server, which moves the mouse. I'm not sure what the best way to fix this would be - some sort of smoothing system after the co-ordinates are recieved by Python, or a PoseNet adjustment. I can't increase the quality as this would slow the process and movements would be jagged without a major re-thinking of the system. My code is below:

poseNet = ml5.poseNet(video, {
  flipHorizontal: true
});

// When the model is loaded
function modelLoaded() {
  console.log('Model Loaded!');
}

// Listen to new 'pose' events
poseNet.on('pose', (results) => {
  console.log(results)
  poses = results[0]
  keypoints = poses["pose"]
  console.log(keypoints)
  var leftWrist = keypoints["leftWrist"]
  console.log(leftWrist)
  var leftX = leftWrist["x"]
  var leftY = leftWrist["y"]
  var rightWrist = keypoints["rightWrist"]
  var rightX = rightWrist["x"]
  var rightY = rightWrist["y"]
});
tensorflow python
1 Answers

My recommendation would be this: the raw outputs of any neural network are likely to be somewhat jagged and have fairly high random fluctuations as it is a highly non-linear function approximator especially when trained for extremely complex tasks such as image processing. Therefore, you should build in some post-processing to smooth out these fluctuations. Here are a few potential fixes:

  1. Average the last few values together to provide some basic rolling average smoothing.
  2. Add a cutoff threshold so that jitters and iterations below some value are rounded to 0.
  3. Use Kalman Filtering to smooth the outputs from the neural network. If you're not familiar with kalman filtering, it essentially assumes that the value you are trying to measure (in this case x and y coords I think) are a normal distribution, and when you sample measurements (i.e. when the network gives an output) this sample is drawn from the distribution so has some error. Depending on the error of the measurement and the relative difference between the measurement and the previously assumed state, the kalman filter decides how much to update the value based on the measurement. This is a really great technique for dealing with noisy inputs to control systems. There are nice, easy to use kalman filter packages for python as well (I used py-kalman for a while).
user_98317770
Related