## HandTracking Push

My tasks are to detect hand motion in certain exercises for stroke patients to re-learn their motor function. One of those is pushing something, a drawer, for example.

I try to make it simple, so I consider the case in which hand is spread and its side is parallel to Kinect. Now I gotta find the angles between fingers and palm, also between palms and lower arm.

At first, I thought it was easy. I noticed how the wrist should be the turning point of contour from pretty smooth lower arm’s part to that also pretty smooth part of palm. But think of case when the hand is straight from arm to palm, such turning point is hard to recognize.

My friend suggested ” Răng ông không đo khoảng cách từ ngón tay tới cổ tay, rồi kiếm điểm mô cách ngón tay chừng nớ thì nó là cổ tay”. He means, the distance between fingertip and knuckle, also between knuckle and wrist should be constant, so based on this fact we can find the knuckle and wrist.

I gave it a try. The distance between my fingertip and knuckle is around 105 mm, that from knuckle to wrist is around 100 mm. 1 problem is that, I don’t know which direction to go on the contour. Clockwise or anti-clockwise? To make it even simpler, I just do both ways and see if I can find out how to compare and take only one of them.

Here is my result :

As you can see, the tip is found pretty correct. The knuckle and the wrist are also cool. I also add another third point which I hope to use later on for comparison. The white dots should not be there, I am trying to get rid of them.

Ideas:

• In this task, I need a bigger tracking rectangle so I can find the lower arm. But this results in a bigger contour, including the lower arm, therefore the center of palm is moving downward to the lower arm (since this part is now having the bigger circle packed inside the contour). So I think, I can use to rectangles, one smaller to find the hand details, the other bigger to be able to work with arm.

I continued my work this afternoon. I make this change: No more trying to find which direction to go to find the knuckle and the wrist. I decided to go both ways. So I have 1 fingertip, 2 knuckle points and 2 wrist points. I also go further down to get 2 lower arm points. Use fingertip and 2 knuckles I can represent the fingers in skeleton style. Similarly, using 2 knuckles and 2 wrists represent palms, and 2 wrists with 2 lower arms represent the lower arm, also in skeletons.

Here are a few snapshots of what I got:

Video of the work can be found here.

It works better than I hope. I’ve not made any measuring to find its accuracy.

What are the limits?

• Fingertip detection is not very good. In the video, the fingertip is found neatly. But in fact, in the code I restrict the fingertip is the K-curvature point of as high as possible in picture frame. So if the hand is pointing down, it won’t be detected correctly. Need change!
• Knuckles and wrists are found based on the distance from the tips. This, again, is not good. For example, in case of palm facing the Kinect, the detection won’t work well.