2. Service & Support
  3. Technical information industrial cameras
  4. Baumer GAPI and OpenCV

In this how-to you will learn how to pass images from Baumer GAPI frame buffers to OpenCV for further processing.

We will use the Baumer GAPI SDK to set up the Baumer camera and to capture the images into memory. We will then pass the images in memory to OpenCV to save them to the hard disk. Once passed to OpenCV you can use the library as needed to further process the images.

Transfer images from Baumer GAPI to OpenCV

After receiving the image data in the Baumer GAPI buffer it can be assigned to an OpenCV matrix (Mat) using the overloaded constructor:

    new cv::Mat::Mat(int _rows, int _cols, int _type, Void * _data, size_t _step = 0Ui64); 
  
    new Emgu.CV.Mat.Mat(int rows, int cols, Emgu.CV.CvEnum.DepthType type, 
                    int channels, IntPtr data, int step);

The data of the Baumer GAPI memory buffer is not copied but a pointer is passed to the OpenCV Matrix. The OpenCV matrix can then be stored to disk using cv::imwrite(). After saving you can delete or reuse the OpenCV matrix and re-queue the Baumer GAPI buffer so it can be used to capture a new image.

Clone, copy, and convert Baumer GAPI buffer to OpenCV matrix

To work with a copy of the Baumer GAPI Buffer data, you can use Clone(), Copy(), or ConvertTo() methods of OpenCV. After copying, the Baumer GAPI Buffer can be re-queued and the copied data can be further processed using OpenCV.

Please note, that OpenCV matrix includes information of width, height, and type (CV_8UC1, CV_8UC3, CV_16UC1, CV_16UC3) while the Baumer GAPI buffer provides further useful information such as Buffer.FrameID, Buffer.PixelFormat, Buffer.XOffset, Buffer.YOffset as well as chunk data. If required you need to copy this information as well.

    pDevice->GetRemoteNode("PixelFormat")->SetString("Mono8");

BGAPI2::Buffer* pBufferFilled = pDataStream->GetFilledBuffer(1000);
if (pBufferFilled->GetPixelFormat() == "Mono8")
{
	cv::Mat* imOriginal = new cv::Mat((int)pBufferFilled->GetHeight(), 
	                                  (int)pBufferFilled->GetWidth(), 
	                                  CV_8UC1, 
	                                  (char *)pBufferFilled->GetMemPtr();

	//3 methods to copy
	cv::Mat imClone = imOriginal->clone();

	cv::Mat* imCopy = new cv::Mat((int)pBufferFilled->GetHeight(), 
                              (int)pBufferFilled->GetWidth(), 
                              CV_8UC1);
	imOriginal->copyTo(*imCopy);

	cv::Mat* imConvert = new cv::Mat((int)pBufferFilled->GetHeight(), 
                                 (int)pBufferFilled->GetWidth(), 
                                 CV_8UC1);

	imOriginal->convertTo(*imConvert, CV_8UC1, 1.0);

	delete imOriginal;
	pBufferFilled->QueueBuffer();

	//use copied image
	cv::imwrite("cv_image_Clone.png", imClone);
	cv::imwrite("cv_image_Copy.png", *imCopy);
	cv::imwrite("cv_image_Convert.png", *imConvert);

	delete imCopy;
	delete imConvert;
} 
  
    mDevice.RemoteNodeList["PixelFormat"].Value = "Mono8";

BGAPI2.Buffer mBufferFilled = mDataStream.GetFilledBuffer(1000);
if (mBufferFilled.PixelFormat == "Mono8")
{
    Emgu.CV.Mat imOriginal = new Emgu.CV.Mat((int)mBufferFilled.Height, 
                                             (int)mBufferFilled.Width, 
                                             Emgu.CV.CvEnum.DepthType.Cv8U, 1, 
                                             (IntPtr)mBufferFilled.MemPtr, 
                                             (int)mBufferFilled.Width * 1);

    //3 methods to copy
    Emgu.CV.Mat imClone = imOriginal.Clone();

    Emgu.CV.Mat imCopy = new Emgu.CV.Mat((int)mBufferFilled.Height, 
                                         (int)mBufferFilled.Width, 
                                         Emgu.CV.CvEnum.DepthType.Cv8U, 1);
    imOriginal.CopyTo(imCopy);

    Emgu.CV.Mat imConvert = new Emgu.CV.Mat((int)mBufferFilled.Height, 
                                            (int)mBufferFilled.Width, 
                                            Emgu.CV.CvEnum.DepthType.Cv8U, 1);
    
    imOriginal.ConvertTo(imConvert, Emgu.CV.CvEnum.DepthType.Cv8U, 1.0);

    mBufferFilled.QueueBuffer();

    //use copied image
    Emgu.CV.CvInvoke.Imwrite("cv_image_Clone.png", imClone);
    Emgu.CV.CvInvoke.Imwrite("cv_image_Copy.png", imCopy);
    Emgu.CV.CvInvoke.Imwrite("cv_image_Convert.png", imConvert);
}

Understanding PixelFormats

Baumer cameras support multiple PixelFormats such as Mono8, Mono12, BGR8 which need to be matched to their equivalent OpenCV format. You can see the matching OpenCV-Matrix-Types in the table below.

Baumer format

OpenCV format

Mono8

CV_8UC1, (8 bit, 1 channel)

Mono10

Convert to Mono16 (bit-shift)

Mono12

Convert to Mono16 (bit-shift)

Mono16

CV_16UC1, (16 bit, 1 channel)

BGR8

CV_8UC3, (8 bit, 3 channels)

BGR10

Convert to BGR16 (bit-shift)

BGR12

Convert to BGR16 (bit-shift)

BGR16

CV_16UC3, (16 bit, 3 channels (BGR) )

RGB8

Convert to BGR8 using cv::cvtColor

RGB10

Convert to BGR16 bit-shift and cv::cvtColor

RGB12

Convert to BGR16 bit-shift and cv::cvtColor

RGB16

Convert to BGR16 using cv::cvtColor

BayerGB8, BayerRG8,

BayerGR8, BayerBG8

Convert to BGR8 using cv::cvtColor

BayerGB10, BayerRG10,

BayerGR10, BayerBG10

Convert to BGR16 bit-shift and cv::cvtColor

BayerGB12, BayerRG12,

BayerGR12, BayerBG12

Convert to BGR16 bit-shift and cv::cvtColor

 

Working with images of PixelFormats which don’t need a conversion is very strait forward as you can see in the examples below.

Example Mono8
    pDevice->GetRemoteNode("PixelFormat")->SetString("Mono8");

BGAPI2::Buffer * pBufferFilled = pDataStream->GetFilledBuffer(1000);
if (pBufferFilled->GetPixelFormat() == "Mono8")
{
	cv::Mat* imOriginal = new cv::Mat((int)pBufferFilled->GetHeight(),
	                                  (int)pBufferFilled->GetWidth(),
	                                  CV_8UC1,
	                                  (char *)pBufferFilled->GetMemPtr();
	cv::imwrite("cv_Mono8_image.png", *imOriginal);
	delete imOriginal;
}
pBufferFilled->QueueBuffer(); 
  
    mDevice.RemoteNodeList["PixelFormat"].Value = "Mono8";

BGAPI2.Buffer mBufferFilled = mDataStream.GetFilledBuffer(1000);
if (mBufferFilled.PixelFormat == "Mono8")
{
    Emgu.CV.Mat imOriginal = new Emgu.CV.Mat((int)mBufferFilled.Height, 
                                             (int)mBufferFilled.Width, 
                                             Emgu.CV.CvEnum.DepthType.Cv8U, 1, 
                                             (IntPtr)mBufferFilled.MemPtr, 
                                             (int)mBufferFilled.Width * 1);
    Emgu.CV.CvInvoke.Imwrite("cv_Mono8_image.png", imOriginal);
}
mBufferFilled.QueueBuffer();
Example BGR8

The standard color PixelFormat in OpenCV is BGR with 8bit for each color channel (CV_8UC3). To match this format in the camera set PixelFormat BGR8 (BGR8Packed can be used as well).

    pDevice->GetRemoteNode("PixelFormat")->SetString("BGR8");

BGAPI2::Buffer * pBufferFilled = pDataStream->GetFilledBuffer(1000);
if ((pBufferFilled->GetPixelFormat() == "BGR8") || 
    (pBufferFilled->GetPixelFormat() == "BGR8Packed"))
{
	cv::Mat* imOriginal = new cv::Mat((int)pBufferFilled->GetHeight(),
	                                  (int)pBufferFilled->GetWidth(),
	                                  CV_8UC3,
	                                  (char *)pBufferFilled->GetMemPtr();
	cv::imwrite("cv_BGR8_image.png", *imOriginal);
	delete imOriginal;
}
pBufferFilled->QueueBuffer(); 
  
    mDevice.RemoteNodeList["PixelFormat"].Value = "BGR8";

BGAPI2.Buffer mBufferFilled = mDataStream.GetFilledBuffer(1000);
if ((mBufferFilled.PixelFormat == "BGR8") ||
    (mBufferFilled.PixelFormat == "BGR8Packed"))
{
    Emgu.CV.Mat imOriginal = new Emgu.CV.Mat((int)mBufferFilled.Height, 
                                             (int)mBufferFilled.Width, 
                                             Emgu.CV.CvEnum.DepthType.Cv8U, 
                                             3, 
                                             (IntPtr)mBufferFilled.MemPtr, 
                                             (int)mBufferFilled.Width * 3);
    Emgu.CV.CvInvoke.Imwrite("cv_BGR8_image.png", imOriginal);
}
mBufferFilled.QueueBuffer();
Converting PixelFormats to a 16 Bit OpenCV matrix type

Some cameras support PixelFormats with 12 bit (Mono12, RGB12). Storing the data in a 16 bit OpenCV Matrix Type would result in a much darker output-image than expected. In order to map the 12 bit PixelFormats to 16 bit OpenCV Matrix Type, the Baumer GAPI buffer memory data need to be shifted by 4 bits (multiply with 16) using the OpenCV ConvertTo() command.

Similarly, 10 bit PixelFormats such as Mono10 can be converted by shifting the data by 2 bits or multiplying with factor 64.

Warning: since we just pass a pointer to the Baumer GAPI buffer memory data, the original data is modified.

    pDevice->GetRemoteNode("PixelFormat")->SetString("Mono12");

BGAPI2::Buffer * pBufferFilled = pDataStream->GetFilledBuffer(1000);
if (pBufferFilled->GetPixelFormat() == "Mono12")
{
	cv::Mat* imOriginal = new cv::Mat((int)pBufferFilled->GetHeight(),
	                                  (int)pBufferFilled->GetWidth(),
	                                  CV_16UC1,
	                                  (char *)pBufferFilled->GetMemPtr();

	imOriginal.ConvertTo(imOriginal, Emgu.CV.CvEnum.DepthType.Cv16U, 64.0);
	cv::imwrite("cv_Mono12_as_Mono16_image.png", *imOriginal);
	delete imOriginal;
}
pBufferFilled->QueueBuffer(); 
  
    mDevice.RemoteNodeList["PixelFormat"].Value = "Mono12";

BGAPI2.Buffer mBufferFilled = mDataStream.GetFilledBuffer(1000);
if (mBufferFilled.PixelFormat == "Mono12")
{
    Emgu.CV.Mat imOriginal = new Emgu.CV.Mat((int)mBufferFilled.Height, 
                                             (int)mBufferFilled.Width, 
                                             Emgu.CV.CvEnum.DepthType.Cv16U, 
                                             1, 
                                             (IntPtr)mBufferFilled.MemPtr, 
                                             (int)mBufferFilled.Width * 2);
    
    imOriginal.ConvertTo(imOriginal, Emgu.CV.CvEnum.DepthType.Cv16U, 16.0);
    Emgu.CV.CvInvoke.Imwrite("cv_Mono12_as_Mono16_image.png", imOriginal);
}
mBufferFilled.QueueBuffer();

Working with Bayer PixelFormats

VT_BayerRGB_2.jpg

The Bayer pattern is widely used in CCD and CMOS cameras. It enables you to get color pictures from a single plane where R,G, and B pixels (sensors of a particular component) are interleaved as shown in the image.

The output RGB components of a pixel are interpolated from 1, 2, or 4 neighbors of the pixel having the same color. There are several modifications of the above pattern that can be achieved by shifting the pattern one pixel left and/or one pixel up. The two letters C1 and C2 in the conversion constants CV_Bayer C1C2 2BGR and CV_Bayer C1C2 2RGB indicate the particular pattern type. These are components from the second row, second and third columns, respectively. For example, the above pattern has a very popular "BG" type.

OpenCV color transformation using cvtColor

Color cameras include raw PixelFormats like BayerBG8, BayerGB8, BayerGR8, and BayerRG8.
The transformation of these formats to the format BGR8 is necessary for further image processing.
Baumer cameras name the Bayer pattern depending on the first two pixels of the first row of the image. In contrast to that, the OpenCV naming depends on the second & third pixel of the second row of the image, which results in a different naming:

Baumer BayerGB corresponds to OpenCV BayerGR
Baumer BayerRG corresponds to OpenCV BayerBG
Baumer BayerGR corresponds to OpenCv BayerGB
Baumer BayerBG corresponds to OpenCV BayerRG

    pDevice->GetRemoteNode("PixelFormat")->SetString("BayerRG8");

BGAPI2::Buffer * pBufferFilled = pDataStream->GetFilledBuffer(1000);
if (pBufferFilled->GetPixelFormat() == "BayerRG8")
{
	cv::Mat* imOriginal = new cv::Mat((int)pBufferFilled->GetHeight(),
	                                  (int)pBufferFilled->GetWidth(),
	                                  CV_8UC1,
	                                  (char *)pBufferFilled->GetMemPtr();
	cv::Mat* imTransformBGR8 = new cv::Mat((int)pBufferFilled->GetHeight(),
                                       (int)pBufferFilled->GetWidth(),
                                       CV_8UC3);

	//Baumer: RGrgrg  >>  OpenCV: rgrgrg
	//        gbgbgb              gBGbgb
	cv::cvtColor(*imOriginal, *imTransformBGR8, CV_BayerBG2BGR); //to BGR

	delete imOriginal;
	pBufferFilled->QueueBuffer();

	cv::imwrite("cv_BayerRG8_as_BGR8_image.png", *imTransformBGR8);
	delete imTransformBGR8;
} 
  
    mDevice.RemoteNodeList["PixelFormat"].Value = "BayerRG8";

BGAPI2.Buffer mBufferFilled = mDataStream.GetFilledBuffer(1000);
if (mBufferFilled.PixelFormat == "BayerRG8")
{
    Emgu.CV.Mat imOriginal = new Emgu.CV.Mat((int)mBufferFilled.Height, 
                                             (int)mBufferFilled.Width, 
                                             Emgu.CV.CvEnum.DepthType.Cv8U, 1, 
                                             (IntPtr)mBufferFilled.MemPtr, 
                                             (int)mBufferFilled.Width * 1);
    Emgu.CV.Mat imTransformBGR8 = new Emgu.CV.Mat();

    //Baumer: RGrgrg  >>  OpenCV: rgrgrg
    //        gbgbgb              gBGbgb
    Emgu.CV.CvInvoke.CvtColor(imOriginal, imTransformBGR8, 
                              Emgu.CV.CvEnum.ColorConversion.BayerBg2Bgr);

    mBufferFilled.QueueBuffer();

    Emgu.CV.CvInvoke.Imwrite("cv_BGR8_image.png", imTransformBGR8);
}
Transformation Bayer to BGR16

Other raw PixelFormats use more bits in the Bayer image, like BayerBG12, BayerGB12, BayerGR12, and BayerRG12. These raw data need to be converted from 12 bit to 16 bit in the first step. After that the 16 bit value can be transformed by cvtColor to the 48 bit format BGR16 (16 bit per color channel).
In case of 10 bit formats like BayerBG10, BayerGB10, BayerGR10, or BayerRG10, use a scaling factor of 64.0 instead of 16.0 to convert them to 16 bit Bayer format.

    pDevice->GetRemoteNode("PixelFormat")->SetString("BayerGB12");

BGAPI2::Buffer * pBufferFilled = pDataStream->GetFilledBuffer(1000);
if (pBufferFilled->GetPixelFormat() == "BayerGB12")
{
	cv::Mat* imOriginal = new cv::Mat((int)pBufferFilled->GetHeight(),
	                                  (int)pBufferFilled->GetWidth(),
	                                  CV_16UC1,
	                                  (char *)pBufferFilled->GetMemPtr();
	cv::Mat* imConvert = new cv::Mat((int)pBufferFilled->GetHeight(), 
	                                 (int)pBufferFilled->GetWidth(), 
	                                 CV_16UC1);
	//convert with scaling of 16.0 to convert 12-Bit to 16-Bit 
	imOriginal->convertTo(*imConvert, CV_16UC1, 16.0);

	delete imOriginal;
	pBufferFilled->QueueBuffer();

	cv::Mat* imTransformBGR16 = new cv::Mat((int)pBufferFilled->GetHeight(), 
	                                        (int)pBufferFilled->GetWidth(), 
	                                        CV_16UC3);

	//Baumer: GBgbgb  >>  OpenCV: gbgbgb
	//        rgrgrg              rGRgrg
	cv::cvtColor(*imConvert, *imTransformBGR16, CV_BayerGR2BGR); //to BGR
	delete imConvert;

	cv::imwrite("cv_BayerGB12_as_BGR16_image.png", *imTransformBGR16);
	delete imTransformBGR16;
} 
  
    mDevice.RemoteNodeList["PixelFormat"].Value = "BayerGB12";

BGAPI2.Buffer mBufferFilled = mDataStream.GetFilledBuffer(1000);
if (mBufferFilled.PixelFormat == "BayerGB12")
{
    Emgu.CV.Mat imOriginal = new Emgu.CV.Mat((int)mBufferFilled.Height, 
                                             (int)mBufferFilled.Width, 
                                             Emgu.CV.CvEnum.DepthType.Cv16U, 1, 
                                             (IntPtr)mBufferFilled.MemPtr, 
                                             (int)mBufferFilled.Width * 2);
    Emgu.CV.Mat imConvert = new Emgu.CV.Mat((int)mBufferFilled.Height, 
                                            (int)mBufferFilled.Width, 
                                            Emgu.CV.CvEnum.DepthType.Cv16U, 1);

    //convert with scaling of 16.0 to convert 12-Bit to 16-Bit 
    imOriginal.ConvertTo(imConvert, Emgu.CV.CvEnum.DepthType.Cv16U, 16.0);

    mBufferFilled.QueueBuffer();

    Emgu.CV.Mat imTransformBGR16 = new Emgu.CV.Mat();

    //Baumer: GBgbgb  >>  OpenCV: gbgbgb
    //        rgrgrg              rGRgrg
    Emgu.CV.CvInvoke.CvtColor(imConvert, imTransformBGR16, 
                              Emgu.CV.CvEnum.ColorConversion.BayerGr2Bgr);

    Emgu.CV.CvInvoke.Imwrite("cv_BayerGB12_as_BGR16_image.png", imTransformBGR16);
}

Downloads

Software examples
To the top