Since 2012, I've been working in the USC Interaction Lab, a lab in USC's Center for Robotics and Embedded Systems that specializes in human-robot interaction. They use Willow Garage's ROS software, so I can draw upon my experience with PR Lite. (They even have a PR2, so I'm not even restricted to having "lite" capabilities!)

In summer 2012, I posted my updates online because there were many student workers that summer. Fall 2013 milestones and updates are posted below. I also worked for the Interaction Lab in the spring and winter break of 2012, but did not post updates online.

The ROS package normally used to calculate poses between cameras requires that all of the cameras see the checkerboard at the same time to be able to calibrate to each other. Since the Interaction Lab is working on a project to make many Kinects track people as they move around in the lab, I made a modified version of this package that does not have this requirement. My package also saves the calibrations to a yaml file instead of a bag file to allow manual editing of calibrations, and includes an openni_launch-specific "hack" to allow tf transforms between Kinect depth frames after their RGB cameras are calibrated to each other. (Cameras other than Kinects can still be calibrated to each other normally.) We plan to put the code in a public repository at some point, and when we do I will add download directions to this page. Below is an rviz screenshot showing the point clouds of 3 Kinects calibrated to each other 2 at a time using my package. (A person is visible near the center of the image.)

rviz screenshot

This is a ROS package I made in the 2012-2013 winter break to provide a dynamic_reconfigure GUI for the multi_calibrate ROS package described above. Below is a screenshot of the client portion of the package. A link will be provided on this page after the package is publicly available.

multi_calibrate_dynamic client screenshot

This is a ROS package I am working on in 2013 which builds and applies a statistical model of skeletons tracked by different sensors. The output of the model is a real-time list of the top N ways to group the tracked skeletons from different cameras together, indicating which skeletons in different cameras are likely to correspond to the same people. A link will be provided on this page after the package is publicly available.

Fall 2013-spring 2014 milestones:

  1. Fix remaining bugs in the multitrack_model package. (done 12/13/2013)
  2. Make a ROS package or node to evaluate the accuracy of multitrack_model.
  3. Work with Ross on a paper about the multi-Kinect skeleton tracking system.


12/17/2013 (worked 1.3 hours)
I talked with Ross about how we'll evaluate the person classifier models to determine their classification accuracy.

12/13/2013 (worked 3.4 hours)
I fixed some bugs in the discriminant classifier from 11/22, and now it works!

11/18/2013, 11/22/2013 (worked 3.5 hours)
I discussed the issues with the discriminant classifier with Ross, and he suggested a different, simpler way to make a discriminant classifier. However, when I implemented it, it still didn't work! I still need to think through why it didn't work, and I haven't ruled out a bug in my code.

11/15/2013 (worked 4.9 hours)
I discussed our plans going forward with Ross, and implemented our new idea for a discriminant person classifier. This version doesn't crash, but it still returns incorrect results for more subtle reasons explained in an email to Ross. The old non-discriminant classifier still works remarkably well.

I sat in the lab for an hour or two, but couldn't concentrate on anything (maybe because I had 2 tests that week?). Sorry about that.

11/1/2013 (worked 2.2 hours)
Earlier in the week, I went to a professor's office twice to ask about the statistics issue in the person tracker, since I still hadn't gotten a reply from Amin. Unfortunately the professor was very busy and I didn't get to talk to him much. On 11/1, I had a productive meeting with Ross and someone else (todo: what's his name?) and we came up with a couple potential ways to solve the issue, though I'm not sure I'm completely satisfied with either of them.

10/25/2013 (worked 1.9 hours)
In the skeleton tracker, I implemented letting you pick the Kinect either by index or bus/address, whereas previously you could only pick it by index. (I wrote this using the simple way I thought of and threw out the non-working code I wrote on 10/18.)

10/18/2013 (worked 2.0 hours)
I figured out a simple way to add support in openni_multitracker2 (the modified skeleton tracker I use) for specifying the Kinect by bus/port address instead of just by device number, which frequently changes when the Kinect is unplugged and replugged. However, I instead tried to implement this in a more complicated way since I also wanted to support specifying the Kinect by serial number, which I wasn't able to get working by the time I had to leave for a class. (The naive way of trying to get the serial number doesn't work for Kinects.)

10/8/2013, 10/11/2013 (worked 1.8 hours)
I talked to Amin about the statistics issue on 10/8, and he saw the problem (that the probability density function doesn't really output probability so it is allowed to output values greater than 1) but needed more time to think about what is the right way to fix it. (As of 11/13, he's been too busy to schedule a follow up meeting.) Since this issue was blocking work on the person classifier, on 10/11 I started looking into improving the skeleton tracker itself and integrating it with OpenNI 2.

10/3/2013 - 10/4/2013 (worked 4.0 hours)
I did some experimenting and found what occasionally caused bad checkerboard and joint tracking calibrations. I also implemented inverting some of the terms in the denominator of the person classifier algorithm, as discussed in a previous meeting. However, this caused frequent crashes, the details of which are explained in an email to Ross.

9/26/2013 (worked 2.0 hours)
I was tired, so I made a lot of mistakes and as a result didn't make much progress.

9/19/2013 - 9/20/2013 (worked 6.5 hours)
On 9/19, I modified the person classifier to consider the likelihoods that tracked skeletons are not associated with hypothetical skeletons. However, when running it, it grouped the skeletons incorrectly (in a different way than on 8/30), and some of the log likelihoods were well above 0. On 9/20, I added some hacks to make the classifier behave like it did on 8/30, while keeping the code reorganization that I did on 9/19. This version behaved reasonably, but running the 9/19 version still had the same problems after running it with a new calibration.

8/30/2013 (worked 4.2 hours)
I fixed the normalizing correction term in the multitrack_model person classifier. However, after running it, it had a tendency to incorrectly classify all skeletons from different Kinects as different people. As of 9/20, I now suspect we just had the Kinects badly calibrated to each other that day.