OpenNI + Depth & IR Compression + Pandaboard

[Update 4/14] I do not have access to a pandaboard yet, therefore I am working on this in Ubuntu with a Kinect.

I am looking to compress depth images from a Kinect or Asus Xtion using OpenNI. Currently I am trying to modify NiViewer to capture and save depth frames as images.

See https://groups.google.com/forum/?fromgroups=#!msg/openni-dev/iYtcrrA365U/wEFT2_-mH0wJ
I edited the files as listed on the google groups post. Alternate link: http://gremsi.blogspot.com/2013/04/saving-rgb-and-depth-data-from-kinect.html

Install OpenCV: http://mitchtech.net/raspberry-pi-opencv/

Edit the make file for NiViewer using this: http://ubuntuforums.org/showthread.php?t=1895678

[Update 4/7]
I had a little trouble with the makefile for NiViewer using the link above. Here is what I have for  OpenNI/Platform/Linux/Build/Samples/NiViewer/Makefile

include ../../Common/CommonDefs.mak
BIN_DIR = ../../../Bin
INC_DIRS = \
../../../../../Include \
../../../../../Samples/NiViewer \
/usr/local/lib \
/usr/local/include/opencv
SRC_FILES = ../../../../../Samples/NiViewer/*.cpp
ifeq ("$(OSTYPE)","Darwin")
LDFLAGS += -framework OpenGL -framework GLUT
else
USED_LIBS += glut GL opencv_core opencv_highgui
endif
USED_LIBS += OpenNI
EXE_NAME = NiViewer
CFLAGS        = -pipe -O2 -I/usr/local/include/opencv  -D_REENTRANT $(DEFINES)
CXXFLAGS      = -pipe -O2 -I/usr/local/include/opencv  -D_REENTRANT $(DEFINES)
include ../../Common/CommonCppMakefile

[Update 4/14]
I ran NiViewer, right click and start capture. So as of now, it saves depth images (in OpenNI/Platform/Linux/Bin/(your platform)/CaptureFrames). Remember to create the CapturedFrames folder.

Now, I want it to start capturing as soon as I ran the program so I edited the NiViewer.cpp file to this:

Comment this part out in the main method (the part that handles the user interface):

reshaper.zNear = 1;
reshaper.zFar = 100;

glut_add_interactor(&reshaper);
cb.mouse_function = MouseCallback;
cb.motion_function = MotionCallback;
cb.passive_motion_function = MotionCallback;
cb.keyboard_function = KeyboardCallback;
cb.reshape_function = ReshapeCallback;
glut_add_interactor(&cb);
glutInit(&argc, argv);
glutInitDisplayString("stencil double rgb");
glutInitWindowSize(WIN_SIZE_X, WIN_SIZE_Y);
glutCreateWindow("OpenNI Viewer");
glutFullScreen();
glutSetCursor(GLUT_CURSOR_NONE);
init_opengl();
glut_helpers_initialize();
glutIdleFunc(IdleCallback);
glutDisplayFunc(drawFrame);
drawInit();
createKeyboardMap();
createMenu();
atexit(onExit);
glutMainLoop();

 before the audioShutdown() command is called, I added these lines:

captureStart(0);
int i = 0;
while (i<10)
{
captureFrame();
i++;
}
captureStop(0);

I compiled and ran NiViewer, but I was only getting blank images. This is because the saveFrame_depth() function (from the google groups post) is using a Linear Histogram. It seems that the calculateHistogram method needs to be called before the Linear Histogram is used. Before, drawFrame() was calling the calculateHistogram function and that is why it worked. I just wanted to see if this worked in general so inside the saveFrame_depth() function, there is a line that says switch(g_DrawConfig.Streams.Depth.Coloring), change this to switch(PSYCHEDELIC); Now I am able to see something in my saved depth image.

[Update 4/17]
I wanted to see how much space it would take for depth data to be stored without any compression. I first started out writing the depth values to a binary file (in plain ascii). This is pretty simple but I am just writing everything out incase if anyone is confused. To do this, I created a new file in the beginning of the saveFrame_depth function:

ofstream myfile;
myfile.open("depthData_ascii");

Then inside the nested for loops (after the switch case statements), you will see data being assigned to red, blue and green pointers (e.g. Bptr[nX] = nBlue and etc). I added these lines under the red, green, blue pointer assignments:

myfile << *pDepth
myfile << " "; 

The lines above should be inside the 2nd for loop. Then I added

myfile << "\n" 

at the end of the first for loop. So basically it should look like this:

saveFrame_depth(...)
{
   ofstream myfile;
   myfile.open("depthData_ascii");
   //code
   for(...)
   {
      for(...)
      {
         //code
         myfile << *pDepth
         myfile << " ";
      }
      myfile << "\n"
   } 

   myfile.close()
   //code 

} 

This file seems to take up around 1.2 mb of space per frame.

[Update 4/21]
Saving it as a simple binary file seems to take up too much space. 1.2mb * 30fps * 60seconds/min * 60mins/hr is 126.562gb per hour. I tried to save the depth data inside a png as 16bit unsigned short integers. To do this,  create a new matrix at the beginning of saveFrame_depth file. The dimensions of the matrix should be pDepthMD->YRes() by pDepthMD->XRes(). Instead of creating it as an 8bit unsigned, do 16bit unsigned (CV_16U). The code for this would look like:

cv::Mat depthArray = cv::Mat(pDepthMD->YRes(), pDepthMD->XRes(), CV_16U)

Inside the first for loop, there are RGB pointers being created with the colorArr variable. Create a pointer for for your depthArray in that same location.

uchar* depthArrayPtr = depthArray.ptr<uchar>(nY); 

ushort* depthArrayPtr = depthArray.ptr<ushort>(nY);

 edit: I had to use ushort for the pointer type. Im not sure exactly why because the pointer type should stay the same length whether or not we are creating a depthArray of 16bit unsigned short or 8bit unsigned char. But using ushort works, where as if I used uchar it saved the image on the left half.

image when using uchar*

image when using  ushort*

Inside the second for loop (toward the end of it), there are values being assigned to the locations of those pointers (e.g. Bptr[nX] = nBlue), under those lines, add this:

depthArrayPtr[nX] = *pDepth

All that is happening is, I am storing the raw depth value in an matrix. Save this array as a png image (add these lines at the end of the function saveFrame_depth):

vector<int> compression_params;
compression_params.push_back(CV_IMWRITE_PNG_COMPRESSION);
compression_params.push_back(0);
imwrite(str_aux_raw, depthArray, compression_params);

I had to #include "cv.h", "highgui.h", <vector>, <iostream>, and <fstream>. In addition to that, I added using namespace std; after my include statements.

This will save the image as a png and it should have the original depth values. The total size comes out to be around 100kb - 200kb (depending on the depth image).

124kb

To see if you are saving your png image correctly and the values are correct, I used MatLab. In MatLab enter this command:

image = imread('/path/to/image.png'); imagesc(image); colorbar;

This should give you a scale of the values represented in your depth image. You could even click Tools->Data Cursor (in the image window) and select a point in that image to get the specific value at that point.

other information: ~1 min of recording depth as a png (with 0 compression) =  14.5mb. It could be that it is not doing 30fps. There were over all of 117 images so thats about 2 images/frames per second.

[Update 4/21]
I am now trying to get IR data and store it into the image. I found some code that could help: https://groups.google.com/d/msg/openni-dev/ytk-dRPDkoM/XKgoIhOxsv8J

[Update 4/26]
I dont think the code is the problem here, because viewing IR data is not working at all in NiViewer. However I do have some information about compression with depth data. I was able to use PNG_COMPRESSION in imwrite and set that value to 50. The image lowered about 10kb (so not that much). However if I had a binary/text file of 1.5mb and zipped that, it would go down to 67kb. So maybe I could use gzip to zip the files as I am saving them.

Updating...