5.3. Application Demos¶

5.3.1. Dual Camera¶

Dual camera demo

Below diagram illustrates the video data processing flow of dual camera demo.

Dual camera example demo demonstrates following-

1. Video capture using V4L2 interface from up to two cameras.
2. QT QWidget based drawing of user interface using linuxfb plugin. linuxfb is fbdev based software drawn QPA.
3. Hardware accelerated scaling of input video from primary camera to display resolution using DSS IP.
4. Hardware accelerated scaling of input video from secondary camera (if connected) to lower resolution using DSS IP.
5. Hardware accelerated overlaying of two video planes and graphics plane using DSS IP.
6. Scaling of two video planes and overlaying with graphics plane happens on the fly in single pass inside DSS IP using DRM atomic APIs.
7. Snapshot of the screen using JPEG compression running on ARM. The captured images are stored in filesystem under /usr/share/camera-images/ folder
8. The camera and display driver shares video buffer using dmabuf protocol. Capture driver writes captured content to a buffer which is directly read by the display driver without copying the content locally to another buffer (zero copy involved).
9. The application also demonstrates allocating the buffer from either omap_bo (from omapdrm) memory pool or from cmem buffer pool. The option to pick omap_bo or cmem memory pool is provided runtime using cmd line.
10. If the application has need to do some CPU based processing on captured buffer, then it is recommended to allocate the buffer using CMEM buffer pool. The reason being omap_bo memory pool doesn’t support cache read operation. Due to this any CPU operation involving video buffer read will be 10x to 15x slower. CMEM pool supports cache operation and hence CPU operations on capture video buffer are fast.
11. The application runs in nullDRM/full screen mode (no windows manager like wayland/x11) and the linuxfb QPA runs in fbdev mode. This gives applicationfull control of DRM resource DSS to control and display the video planes.

Instructions to run dual camera demo

• Since the application need control of DRM resource (DSS) and there can be only one master, make sure that the wayland/weston is not running.

#/etc/init.d/weston stop

• Run the dual camera application

#dual-camera -platform linuxfb <0/1>

• When last argument is set as 0, capture and display driver allocates memory from omap_bo pool.
• When it is set to 1, the buffers are allocated from CMEM pool.

5.3.2. Video Analytics¶

Overview

The Video Analytics demo shipped with the Processor SDK Linux for AM57xx showcases how a Linux Application running on Cortex A-15 cluster can take advantage of C66x DSP, 3D SGX hardware acceleration blocks to process a real-time camera input feed and render the processed output on display - all using open programming paradigms such as OpenCV, OpenCL, OpenGL, and Qt and standard Linux drivers for capture/display.

A hand gesture is detected using morphological operators from OpenCV library. The gesture is used to control a parameter (wind speed) used in physical simulation of water waves. Result of simulation is visualized in real-time via 3D rendering of water surface. The hardware IP blocks, such as IVAHD, M4 cores, are not utilized for this demo.

Setup

In order to re-create the demo, user would need a standard AM572x GP EVM and Processor SDK Linux package. The demo is integrated into the Matrix GUI and can be launched by touching the “Video Analytics” icon on the LCD. The sources to the demo are packaged in the Processor SDK Linux Installer, under “example-applications” folder and can be compiled using the SDK’s top-level Makefile

Building Blocks

• Camera Capture: The camera module on AM572x GP EVM acquires color frames of 640x480 resolution. The images are received by the OpenCV framework using camera capture class, that depends on the standard V4L2 Linux driver (/dev/video1).
• Gesture Recognition: The first 30 frames captured are used to estimate the background, and later subtracted to extract the hand contour, using erode and dilute operations available in the OpenCV Library. Analysis of largest contour is performed to find convex hulls. Hand gesture classification is based on ratio of outer and inner contour. Several discrete values are sent to wave simulation algorithm.
• Wave Simulation: Wave simulation is done in two stages: calculation of initial (t=0)) Phillips spectrum followed by spectrum updates per each time instant. Height map is generated in both steps using 2D IFFT (of spectrum). The Gesture inputs are used to vary the wind direction for wave simulation
• Graphics Rendering: Finally, 2D vertex grid is generated with the above height map. Fragment rendering uses user-defined texture and height dependent coloring.
• Display: The displayed content includes - Camera feed, overlayed with the contour detection, and the generated water surface

Block diagram

Demo Internals

Programming Paradigm

• Code running on A15 Linux platform is written in C++ and links with QT5, OpenCV and OpenCL host side libraries.
• Code running on DSP C66x is written in OpenCL C
• Code running on GPU is written in GLSL.
• Standard V4L2 Linux driver is used for camera capture
• The demo uses OpenCV 3.1, OpenCL 1.2, OpenGL ES 2.0/GLSL and QT 5.5

OpenCV (running on ARM A15s)

Video analytics functionality includes simple algorithm to extract contours of hand, and detect how widely opened the hand is. A parameter which indicates ratio between convex hull and contours is used to control physical simulation of waves.This is an output argument converted to wind speed, in 2.5-10 ms range (4 discrete values). Initially image background is estimated (for 30 frames) using MOG2 (Mixture Of Gaussians), standard OpenCV algorithm. Later, the background is subtracted from camera frames, before morphological operations: erode and dilute are performed.Next two steps include contour detection (which is also presented in camera view window) and convex hull estimation. Ratio between outer and inner contour indicates (in 1.1-1.8 range) can be correlated with number of fingers and how widely are they spread. If they are more widely spread, parameter is converted to higher wind speed and waves become higher.

OpenCL (running on ARM A15s + C66x)

Wave surface simulation is based on generation of Phillips (2D) spectrum followed by 2D IFFT (256x256). Both operations are executed by OpenCL offload to single DSP core. Many details on this algorithm can be found in graphics.ucsd.edu/courses/rendering/2005/jdewall/tessendorf.pdf. This stage is controlled by wave speed (output of gesture detection algorithm) using fixed wind direction (an idea for improvement: wind direction can be controlled using hand gesture).

Height map in form of 256x256 float matrix is generated on output and used by OpenGL based vertex renderer (performed in next step). Wave surface simulation consists of two steps:

• initial setup defining starting conditions (wind speed and wind direction are used as input in this stage only)
• update of wave surface height map (Phillips spectrum modification generated at t=0, along time axe and 2D IFFT for each time step).

OpenGL ES (running on ARM A15s + GPU/SGX)

FLOW: 256x256 float height map matrix + fixed texture -> OpenGL ES based algorithm -> rendered frame buffers

OpenGL ES is a subset of Open GL for desktop devices. Important difference (for this project) is requirement to use vertex buffers and only triangle strips. Also Qt provides wrapper functions (QOpenGL functions) created with intention to hide differences between different OpenGL versions and also to slightly simplify programming. On the downside, it involves Qt specific syntax (not far from original OpenGL ES functions). Height Map data (256x256, but sub-sampled by 4x4, hence 64x64 vertices) received from previous stage, are rendered specific coloring and user supplied JPEG image. Fragment shader does mixing of texture and color derived from interpolated height giving more natural visualization. Currently lighting effects are not used (Implementing this can significantly improve the quality of rendering).

QT 5 (running on ARM A15)

FLOW: user input (mouse and keyboard) -> QT application -> windowing system and control of above threads

Directory Structure

The functionality is organized as shows in the files/description below.

Performance

The hand gesture detection/wave surface simulation/wave surface rendering demo pipeline runs at 18-20 fps. For other algorithms (e.g. smoothing, canny) the pipeline runs at 33-35 fps.

Licensing

The demo code is provided under BSD/MIT License

FAQ/Known Issues

• Brighter lighting conditions are necessary for noise-free camera input, to allow good contour detection. In poor lighting conditions, there would be false or no detection.
• OpenCV 3.1 version shows low FPS rate for Camera Capture. Hence, a custom solution based on direct V4L2 ioctl() calls is adopted (cameraGrab.cpp file) to boost the FPS

5.3.3. DLP 3D Scanner¶

This demo demonstrates an embedded 3D scanner based on the structured light principle, with am57xx. More details can be found at https://www.ti.com/tool/tidep0076

5.3.4. People Tracking¶

This demo demonstrates the capability of people tracking and detection with TI’s ToF (Time-of-Flight) sensor. More details can be found at https://www.ti.com/lit/pdf/tidud06

5.3.5. Barcode Reader¶

Introduction

Detecting 1D and 2D barcodes on an image, and decoding those barcodes are important use cases for the Machine-Vision. Processor SDK Linux has integrated the following open source components, and examples to demonstrate both of these features.

• Barcode detection: OpenCV
• Barcode Decoder/Reader: Zbar Library

OpenCV includes python wrapper to allow quick and easy prototyping. It also includes support for OpenCL offload on devices with C66 DSP core (currently OpenCV T-API cannot be used with python wrapper).

Zbar Barcode Decoder/Reader

Recipes for zbar barcode reader have been added to build the zbar library and test binary. Zbar is standalone library, which does not depend on OpenCV. Current release is not accelerated via OpenCL dispatch (obvious candidates are zbar_scan_y() and calc_tresh() functions consuming >50% of CPU resources).

Command to run zbar test binary:

barcode_zbar [barcode_image_name]

Barcode Region Of Interest (ROI) Detection with OpenCV and Python

Detecting Barcodes in Images using Python and OpenCV provides python scripts which run with OpenCV 2.4.x. For use with Process SDK Linux which has OpenCV 3.1, modifications have been made to the original python scripts so that they can run with OpenCV 3.1. Below please find the modified python scripts detect_barcode.py.

# import the necessary packages
import numpy as np
import argparse
import cv2

# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-i", "--image", required = True, help = "path to the image file")
args = vars(ap.parse_args())

# load the image and convert it to grayscale
image = cv2.imread(args["image"])
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

# compute the Scharr gradient magnitude representation of the images
# in both the x and y direction
gradX = cv2.Sobel(gray, ddepth = cv2.CV_32F, dx = 1, dy = 0, ksize = -1)
gradY = cv2.Sobel(gray, ddepth = cv2.CV_32F, dx = 0, dy = 1, ksize = -1)

# subtract the y-gradient from the x-gradient
gradient = cv2.subtract(gradX, gradY)
gradient = cv2.convertScaleAbs(gradient)

# blur and threshold the image
blurred = cv2.blur(gradient, (9, 9))
(_, thresh) = cv2.threshold(blurred, 225, 255, cv2.THRESH_BINARY)

# construct a closing kernel and apply it to the thresholded image
kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (21, 7))
closed = cv2.morphologyEx(thresh, cv2.MORPH_CLOSE, kernel)

# perform a series of erosions and dilations
closed = cv2.erode(closed, None, iterations = 4)
closed = cv2.dilate(closed, None, iterations = 4)

# find the contours in the thresholded image, then sort the contours
# by their area, keeping only the largest one
(_, cnts, _) = cv2.findContours(closed.copy(), cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE)
c = sorted(cnts, key = cv2.contourArea, reverse = True)[0]

# compute the rotated bounding box of the largest contour
rect = cv2.minAreaRect(c)
box = np.int0(cv2.boxPoints(rect))

# draw a bounding box arounded the detected barcode and display the
# image
cv2.drawContours(image, [box], -1, (0, 255, 0), 3)
cv2.imshow("Image", image)
cv2.waitKey(0)

python detect_barcode.py --image [barcode_image_name]

Barcode Region Of Interest (ROI) Detection with OpenCV and CPP implementation

Current version of OpenCV (3.1) Python wrapper does not support T-API which is needed for OpenCL dispatch. So Processor SDK Linux is including the same algorithm implemented in CPP (https://git.ti.com/apps/barcode-roi-detection), which can be executed on ARM platform only, or with DSP acceleration. CPP example includes more options for various types of input and output, and run-time control of OpenCL dispatch.

This example allows multiple command line options:

• Using static image (JPG or PNG) as input
• Live display or static image output (JPG or PNG)
• Use OpenCL offload or not

Target filesystem includes detect_barcode in “/usr/bin”, and test vector in “/usr/share/ti/image” folder. Again, after successful detection image with barcode in green bounding box is displayed or written to output file. Below are various use cases of detect_barcode.

• Static image input, no opencl dispatch, live display: detect_barcode sample_barcode.jpg 0 1
• Static image input, opencl ON, live display: detect_barcode sample_barcode.jpg 1 1
• Static image input, opencl ON, file output: detect_barcode sample_barcode.jpg 1 image_det.png

Majority of workload is in following lines:

ocl::setUseOpenCL(ocl_acc_flag);  /* Turn ON or OFF OpenCL dispatch  */

cvtColor(im_rgb,im_gray,CV_RGB2GRAY);
im_gray.copyTo(img_gray);

Sobel( img_gray, gradX, CV_16S, 1, 0, -1, 1, 0, BORDER_DEFAULT ); /* Input is 8-bit unsigned, output is 16-bit signed */
Sobel( img_gray, gradY, CV_16S, 0, 1, -1, 1, 0, BORDER_DEFAULT ); /* Input is 8-bit unsigned, output is 16-bit signed */
subtract(gradX, gradY, gradient);
convertScaleAbs(gradient, abs_gradient);

// blur and threshold the image
//GaussianBlur( abs_gradient, blurredImg, Size(7,7), 0, 0, BORDER_DEFAULT );
GaussianBlur( abs_gradient, blurredImg, Size(3,3), 0, 0, BORDER_DEFAULT ); /* 3x3 kernel */
threshold(blurredImg, threshImg, 225, 255, THRESH_BINARY);

Mat elementKernel = getStructuringElement( MORPH_RECT, Size( 2*10+1, 2*3+1 ), Point(10, 3));
ocl::setUseOpenCL(false); /* Turn OFF OpenCL dispatch */
morphologyEx( threshImg, closedImg, MORPH_CLOSE, elementKernel );

ocl::setUseOpenCL(ocl_acc_flag);   /* Turn ON or OFF OpenCL dispatch  */
erode(closedImg, img_final, UMat(), Point(-1, -1), 4); /* erode, 4 iterations */
dilate(img_final, img_ocl, UMat(), Point(-1, -1), 4);  /* dilate, 4 iteration */
ocl::setUseOpenCL(false); /* Turn OFF OpenCL dispatch */

Not all OpenCV kernels can be dispatched to DSP via OpenCL. Please refer to OpenCV#OpenCL_C_C66_DSP_kernels for the list of kernels which are currently DSP accelerated.

In order to use OpenCL dispatch, it is necessary to:

• Enable OpenCL use (by setting environment variables, and invoking ocl::setUseOpenCL(ocl_acc_flag))
• Use T-API: e.g. replace Mat types with UMat types

5.3.6. EVSE Demos¶

This demo showcases Human Machine Interface (HMI) for Electric Vehicle Supply Equipment(EVSE) Charging Stations. More details can be found at https://www.ti.com/tool/TIDEP-0087

5.3.7. Protection Relay Demo¶

Matrix UI provides out of box demo to showcase Human Machine Interface (HMI) for Protection Relays. More details can be found at https://www.ti.com/tool/TIDEP-0102

5.3.8. mmWave Gesture Controlled HMI¶

This demo showcases gesture controlled HMI, where the mmWave sensor detects presence and classifies natural hand gestures, and sends them to the Sitara processor to drive the Qt GUI display. More details can be found at https://www.ti.com/tool/TIDEP-01013.

5.3.9. Qt5 Thermostat HMI Demo¶

Qt-based Thermostat HMI demo

The morphing of devices like the basic thermostat into a breed of power smart thermostats has shown how the appliances in residences today must adapt, re-imagine and, in some cases, reinvent their role in the connected home of the future or risk being left behind. Thermostats in particular have come a long way since they were first created. There was a time when the mechanical dial thermostat was the only option. It was simple and intuitive to operate because what you saw was what you got. The user simply set the temperature and walked away. Unfortunately, it was not efficient. Energy and money could be wasted since its settings would only change when someone manually turned the dial to a new temperature. Modern thermostats provide a much richer Graphical Interface, with several features. Processor SDK now includes a Qt5 based thermostat with the following key features - that should easily enable customers to use as a starting point for further innovation.

• Display three-day weather forecast, daily temperature range for the selected city from openweathermap.org
• Display and adjust room temperature
• Pop-up menu to select city, temperature format and set network proxy
• Weekly temperature schedule
• Away Energy saving mode

Fig. 5.1 Figure1: three-day weather forecast

Fig. 5.2 Figure2: Select City for weather forecasts

Fig. 5.3 Figure3: Set Proxy for Internet connectivity

Fig. 5.4 Figure4: Weekly temperature schedule

Fig. 5.5 Figure5: Inside temperature, Away energy saving settings

The demo is hosted at https://git.ti.com/apps/thermostat-demo, and also the sources are located at

<SDK-install-dir>/example-applications/qt-tstat-2.0/

The code can be compiled, installed using top-level SDK Makefile

make qt-tstat-clean
make qt-tstat
make qt-tstat-install

5.3.10. Optical Flow with OpenVX¶

5.3.10.1. OpenVX Example Block Diagram¶

OpenVx tutorial example demontrates the optical flow based on image pyramids and Harris corner tracking. It builds upon TIOVX, and utilizes OpenCV for reading the input (from file or camera) and rending the output for display. Input frame from OpenCV invokes OpenVX graph, and the processing is done once per input frame. OpenVX defines the graph topology and all configuration details. All resources are allocated and initialzied before processing.

5.3.10.2. Run OpenVx Tutorial Example¶

The binary for OpenVX tutorial example is located at /usr/bin/tiovx-opticalflow. It is a statically linked Linux application running on Arm.

Before running the tutorial example, download the test clip and copy it to file system (e.g. ~/tiovx). Then, execute the commands below to load OpenVx firmware and run the optical flow example.

reload-dsp-fw.sh tiovx                                       # load openvx firmware and restart dsps
tiovx-opticalflow /home/root/tiovx/PETS09-S1-L1-View001.avi  # Run tutorial example

Screen capture after running the optical flow example:

Logs:

root@am57xx-evm:~# tiovx-opticalflow /home/root/PETS09-S1-L1-View001.avi
 VX_ZONE_INIT:Enabled
 VX_ZONE_ERROR:Enabled
 VX_ZONE_WARNING:Enabled
VSPRINTF_DBG:SYSTEM NOTIFY_INIT: starting
VSPRINTF_DBG: SYSTEM: IPC: Notify init in progress !!!

VSPRINTF_DBG:[0] DUMP ALL PROCS[3]=DSP1
VSPRINTF_DBG:[1] DUMP ALL PROCS[4]=DSP2
VSPRINTF_DBG:Next rx queue to open:QUE_RX_HOST

VSPRINTF_DBG:Just created MsgQue
VSPRINTF_DBG:Created RX task
VSPRINTF_DBG:Next tx queue to open:QUE_RX_DSP1, procId=3
VSPRINTF_DBG:Next tx queue to open:QUE_RX_DSP2, procId=4
VSPRINTF_DBG:Dump all TX queues: procId=3 name=QUE_RX_DSP1 queId=262272, msgSize=68, heapId=0
VSPRINTF_DBG:Dump all TX queues: procId=4 name=QUE_RX_DSP2 queId=196736, msgSize=68, heapId=0
VSPRINTF_DBG:SYSTEM: IPC: SentCfgMsg, procId=3 queuId=262272
VSPRINTF_DBG:SYSTEM: IPC: SentCfgMsg, procId=4 queuId=196736
VSPRINTF_DBG: SYSTEM: IPC: Notify init DONE !!!

VSPRINTF_DBG:>>>> ipcNotifyInit returned: 0

VSPRINTF_DBG:SYSTEM NOTIFY_INIT: done
OK: FILE /home/root/PETS09-S1-L1-View001.avi 768x480
init done
Using Wayland-EGL
wlpvr: PVR Services Initialised
LOG: [ status = -1 ] Hello there!

Run the optical flow example with camera input:

reload-dsp-fw.sh tiovx        # load openvx firmware and restart dsps
tiovx-opticalflow             # Run tutorial example with camera input

Logs:

root@am57xx-evm:~# tiovx-opticalflow
 VX_ZONE_INIT:Enabled
 VX_ZONE_ERROR:Enabled
 VX_ZONE_WARNING:Enabled
VSPRINTF_DBG:SYSTEM NOTIFY_INIT: starting
VSPRINTF_DBG: SYSTEM: IPC: Notify init in progress !!!

VSPRINTF_DBG:[0] DUMP ALL PROCS[3]=DSP1
VSPRINTF_DBG:[1] DUMP ALL PROCS[4]=DSP2
VSPRINTF_DBG:Next rx queue to open:QUE_RX_HOST

VSPRINTF_DBG:Just created MsgQue
VSPRINTF_DBG:Created RX task
VSPRINTF_DBG:Next tx queue to open:QUE_RX_DSP1, procId=3
VSPRINTF_DBG:Next tx queue to open:QUE_RX_DSP2, procId=4
VSPRINTF_DBG:Dump all TX queues: procId=3 name=QUE_RX_DSP1 queId=262272, msgSize=68, heapId=0
VSPRINTF_DBG:Dump all TX queues: procId=4 name=QUE_RX_DSP2 queId=196736, msgSize=68, heapId=0
VSPRINTF_DBG:SYSTEM: IPC: SentCfgMsg, procId=3 queuId=262272
VSPRINTF_DBG:SYSTEM: IPC: SentCfgMsg, procId=4 queuId=196736
VSPRINTF_DBG: SYSTEM: IPC: Notify init DONE !!!

VSPRINTF_DBG:>>>> ipcNotifyInit returned: 0

VSPRINTF_DBG:SYSTEM NOTIFY_INIT: done
OK: CAMERA#1 640x480
init done
Using Wayland-EGL
wlpvr: PVR Services Initialised
LOG: [ status = -1 ] Hello there!

After finishing running the OpenVX tutorial example, switch the firmware back to the default for OpenCL:

reload-dsp-fw.sh opencl        # load opencl firmware and restart dsps

5.3.11. ROS and Radar¶

5.3.11.2. HW Setup¶

• Sitara device, e.g. AM5728 GP EVM

• IWR1443 Radar EVM or IWR1642 Radar EVM

• USB to microUSB cable (connecting USB connector on Sitara side with microUSB connector on Radar EVM
• [optional] Kobuki mobile base (as used by Turtlebot2), http://kobuki.yujinrobot.com/

Fig. 5.6 Kobuki mobile base with Sitara, Radar EVMs and monitoring Ubuntu Linux box (PC desktop or laptop)

Compatibility

The ti_mmwave_rospkg ROS driver package on Sitara is tested with Processor Linux SDK which includes meta-ros layer, indigo, (from https://github.com/bmwcarit/meta-ros).

For visualization we use ROS indigo distro on Ubuntu Linux box, preffered for compatibility reasons. Run the commands below to install ROS indigo on Ubuntu Linux box. More details can be found from https://wiki.ros.org/indigo/Installation/Ubuntu.

sudo sh -c 'echo "deb http://packages.ros.org/ros/ubuntu trusty main" > /etc/apt/sources.list.d/ros-latest.list'
wget https://raw.githubusercontent.com/ros/rosdistro/master/ros.key -O - | sudo apt-key add -
sudo apt-get update
sudo apt-get install ros-indigo-desktop-full

For this demo, IWR EVM requires mmWave SDK firmware. If different firmware is used on Radar EVM, please follow procedure using UniFlash tool to install mmWave SDK.

root@am57xx-evm:/usr/bin# ln -sf python3 python.python
root@am57xx-evm:/usr/bin# ln -sf python3-config python-config.python

export ROS_ROOT=/opt/ros/indigo
export PATH=$PATH:/opt/ros/indigo/bin
export LD_LIBRARY_PATH=/opt/ros/indigo/lib
export PYTHONPATH=/usr/lib/python3.5/site-packages:/opt/ros/indigo/lib/python3.5/site-packages
export ROS_MASTER_URI=http://$SITARA_IP_ADDR:11311
export ROS_IP=$SITARA_IP_ADDR
export CMAKE_PREFIX_PATH=/opt/ros/indigo
export ROS_PACKAGE_PATH=/opt/ros/indigo/share
touch /opt/ros/indigo/.catkin

source /opt/ros/indigo/setup.bash
roscore

source /opt/ros/indigo/setup.bash
rosrun roscpp_tutorials talker

....[ INFO] [1516637959.231163685]: hello world 5295
[ INFO] [1516637959.331163994]: hello world 5296
[ INFO] [1516637959.431165605]: hello world 5297
[ INFO] [1516637959.531161359]: hello world 5298
[ INFO] [1516637959.631162807]: hello world 5299
[ INFO] [1516637959.731166207]: hello world 5300
[ INFO] [1516637959.831215641]: hello world 5301
[ INFO] [1516637959.931165361]: hello world 5302
[ INFO] [1516637960.031165019]: hello world 5303
[ INFO] [1516637960.131164027]: hello world 5304

export ROS_MASTER_URI=http://$SITARA_IP_ADDR:11311
export ROS_IP=$LINUXBOX_IP_ADDR
source /opt/ros/indigo/setup.bash
rosrun roscpp_tutorials listener

...
data: hello world 5338
---
data: hello world 5339
---
data: hello world 5340
---
data: hello world 5341
---
data: hello world 5342
---
data: hello world 5343
---
data: hello world 5344

5.3.11.5. mmWave ROS node on Sitara and RViz on Ubuntu Linux box¶

1st SSH terminal, to Sitara EVM

Start roscore, only if it is not already started

source /opt/ros/indigo/setup.bash roscore

2nd SSH terminal, to Sitara EVM

source /opt/ros/indigo/setup.bash
roslaunch  ti_mmwave_rospkg rviz_1443_3d.launch

Change "rviz_1443_3d.launch to "rviz_1642_2d.launch", based on Radar EVM type (1443 or 1642).

If Kobuki mobile is available, use the command below instead:

roslaunch  ti_mmwave_rospkg plsdk_rviz_1443_3d.launch

Sample log is included:

source /opt/ros/indigo/setup.bash
roslaunch ti_mmwave_rospkg plsdk_rviz_1443_3d.launch

... logging to /home/root/.ros/log/97dfe396-2711-11e8-bd4a-a0f6fdc25c34/roslaunch-am57xx-evm-7487.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://192.168.0.222:35481/

SUMMARY
========

PARAMETERS
 * /fake_localization/use_map_topic: True
 * /mmWave_Manager/command_port: /dev/ttyACM0
 * /mmWave_Manager/command_rate: 115200
 * /mmWave_Manager/data_port: /dev/ttyACM1
 * /mmWave_Manager/data_rate: 921600
 * /mmWave_Manager/max_allowed_azimuth_angle_deg: 90
 * /mmWave_Manager/max_allowed_elevation_angle_deg: 90
 * /rosdistro: b'<unknown>\n'
 * /rosversion: b'1.11.21\n'

NODES
  /
    fake_localization (fake_localization/fake_localization)
    mmWaveQuickConfig (ti_mmwave_rospkg/mmWaveQuickConfig)
    mmWave_Manager (ti_mmwave_rospkg/ti_mmwave_rospkg)
    octomap_server (octomap_server/octomap_server_node)
    static_tf_map_to_base_radar_link (tf/static_transform_publisher)
    static_tf_map_to_odom (tf/static_transform_publisher)

ROS_MASTER_URI=http://192.168.0.222:11311

core service [/rosout] found
process[mmWave_Manager-1]: started with pid [7505]
process[mmWaveQuickConfig-2]: started with pid [7506]
process[static_tf_map_to_odom-3]: started with pid [7507]
process[static_tf_map_to_base_radar_link-4]: started with pid [7508]
[ INFO] [1520981858.224293205]: mmWaveQuickConfig: Configuring mmWave device using config file: /opt/ros/indigo/share/ti_mmwave_rospkg/cfg/1443_3d.cfg
process[octomap_server-5]: started with pid [7509]
process[fake_localization-6]: started with pid [7517]
[ INFO] [1520981858.367713151]: waitForService: Service [/mmWaveCommSrv/mmWaveCLI] has not been advertised, waiting...
[ INFO] [1520981858.436009564]: Initializing nodelet with 2 worker threads.
[ INFO] [1520981858.480256524]: mmWaveCommSrv: command_port = /dev/ttyACM0
[ INFO] [1520981858.480407967]: mmWaveCommSrv: command_rate = 115200
[ INFO] [1520981858.497923263]: waitForService: Service [/mmWaveCommSrv/mmWaveCLI] is now available.
[ INFO] [1520981858.498667137]: mmWaveQuickConfig: Ignored blank or comment line: '% ***************************************************************'
[ INFO] [1520981858.499059815]: mmWaveQuickConfig: Ignored blank or comment line: '% Created for SDK ver:01.01'
[ INFO] [1520981858.499462577]: mmWaveQuickConfig: Ignored blank or comment line: '% Created using Visualizer ver:1.1.0.1'
[ INFO] [1520981858.505357942]: mmWaveQuickConfig: Ignored blank or comment line: '% Frequency:77'
[ INFO] [1520981858.506164932]: mmWaveQuickConfig: Ignored blank or comment line: '% Platform:xWR14xx'
[ INFO] [1520981858.506843089]: mmWaveQuickConfig: Ignored blank or comment line: '% Scene Classifier:best_range_res'
[ INFO] [1520981858.507514414]: mmWaveQuickConfig: Ignored blank or comment line: '% Azimuth Resolution(deg):15 + Elevation'
[ INFO] [1520981858.508289684]: mmWaveQuickConfig: Ignored blank or comment line: '% Range Resolution(m):0.044'
[ INFO] [1520981858.508999398]: mmWaveQuickConfig: Ignored blank or comment line: '% Maximum unambiguous Range(m):9.01'
[ INFO] [1520981858.509816310]: mmWaveQuickConfig: Ignored blank or comment line: '% Maximum Radial Velocity(m/s):5.06'
[ INFO] [1520981858.510520982]: mmWaveQuickConfig: Ignored blank or comment line: '% Radial velocity resolution(m/s):0.64'
[ INFO] [1520981858.518476684]: mmWaveQuickConfig: Ignored blank or comment line: '% Frame Duration(msec):33.333'
[ INFO] [1520981858.519262364]: mmWaveQuickConfig: Ignored blank or comment line: '% Range Detection Threshold (dB):9'
[ INFO] [1520981858.519957764]: mmWaveQuickConfig: Ignored blank or comment line: '% Range Peak Grouping:disabled'
[ INFO] [1520981858.520157681]: mmWaveDataHdl: data_port = /dev/ttyACM1
[ INFO] [1520981858.520252841]: mmWaveDataHdl: data_rate = 921600
[ INFO] [1520981858.520315142]: mmWaveDataHdl: max_allowed_elevation_angle_deg = 90
[ INFO] [1520981858.520375654]: mmWaveDataHdl: max_allowed_azimuth_angle_deg = 90
[ INFO] [1520981858.520943849]: mmWaveQuickConfig: Ignored blank or comment line: '% Doppler Peak Grouping:disabled'
[ INFO] [1520981858.521671945]: mmWaveQuickConfig: Ignored blank or comment line: '% Static clutter removal:disabled'
[ INFO] [1520981858.522412729]: mmWaveQuickConfig: Ignored blank or comment line: '% ***************************************************************'
[ INFO] [1520981858.523396537]: mmWaveQuickConfig: Sending command: 'sensorStop'
[ INFO] [1520981858.533674630]: mmWaveCommSrv: Sending command to sensor: 'sensorStop'
[ INFO] [1520981858.536083724]: DataUARTHandler Read Thread: Port is open
[ INFO] [1520981858.548926257]: mmWaveCommSrv: Received response from sensor: 'sensorStop
Done
mmwDemo:/>'
[ INFO] [1520981858.550875817]: mmWaveQuickConfig: Command successful (mmWave sensor responded with 'Done')
[ INFO] [1520981858.551745758]: mmWaveQuickConfig: Sending command: 'flushCfg'
[ INFO] [1520981858.559882020]: mmWaveCommSrv: Sending command to sensor: 'flushCfg'
[ INFO] [1520981858.562726084]: mmWaveCommSrv: Received response from sensor: 'flushCfg
Done
mmwDemo:/>'
[ INFO] [1520981858.564378289]: mmWaveQuickConfig: Command successful (mmWave sensor responded with 'Done')
[ INFO] [1520981858.565240748]: mmWaveQuickConfig: Sending command: 'dfeDataOutputMode 1'
[ INFO] [1520981858.573026625]: mmWaveCommSrv: Sending command to sensor: 'dfeDataOutputMode 1'
[ INFO] [1520981858.576915985]: mmWaveCommSrv: Received response from sensor: 'dfeDataOutputMode 1
Done
mmwDemo:/>'
...
mmwDemo:/>'
[ INFO] [1520981858.776118886]: mmWaveQuickConfig: Command successful (mmWave sensor responded with 'Done')
[ INFO] [1520981858.776938726]: mmWaveQuickConfig: Sending command: 'compRangeBiasAndRxChanPhase 0.0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0'
[ INFO] [1520981858.782736816]: mmWaveCommSrv: Sending command to sensor: 'compRangeBiasAndRxChanPhase 0.0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0'
[ INFO] [1520981858.792102024]: mmWaveCommSrv: Received response from sensor: 'compRangeBiasAndRxChanPhase 0.0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0
Done
mmwDemo:/>'
[ INFO] [1520981858.793846462]: mmWaveQuickConfig: Command successful (mmWave sensor responded with 'Done')
[ INFO] [1520981858.794657355]: mmWaveQuickConfig: Sending command: 'measureRangeBiasAndRxChanPhase 0 1.5 0.2'
[ INFO] [1520981858.800233568]: mmWaveCommSrv: Sending command to sensor: 'measureRangeBiasAndRxChanPhase 0 1.5 0.2'
[ INFO] [1520981858.806256139]: mmWaveCommSrv: Received response from sensor: 'measureRangeBiasAndRxChanPhase 0 1.5 0.2
Done
mmwDemo:/>'
[ INFO] [1520981858.807890614]: mmWaveQuickConfig: Command successful (mmWave sensor responded with 'Done')
[ INFO] [1520981858.808687680]: mmWaveQuickConfig: Sending command: 'sensorStart'
[ INFO] [1520981858.814534734]: mmWaveCommSrv: Sending command to sensor: 'sensorStart'
[ INFO] [1520981858.822598283]: mmWaveCommSrv: Received response from sensor: 'sensorStart
Done
mmwDemo:/>'
[ INFO] [1520981858.824211611]: mmWaveQuickConfig: Command successful (mmWave sensor responded with 'Done')
[ INFO] [1520981858.824545077]: mmWaveQuickConfig: mmWaveQuickConfig will now terminate. Done configuring mmWave device using config file: /opt/ros/indigo/share/ti_mmwave_rospkg/cfg/1443_3d.cfg
[mmWaveQuickConfig-2] process has finished cleanly

3rd SSH terminal, to Sitara EVM

Bring up all ROS components for communicting and controlling Kobuki

source /opt/ros/indigo/setup.bash
roslaunch kobuki_node minimal.launch

4th SSH terminal, to Sitara EVM

Start Kobuki teleop console (remotely control Kobuki movement using keyboard)

source /opt/ros/indigo/setup.bash
roslaunch kobuki_keyop safe_keyop.launch

        Operating kobuki from keyboard:
        Forward/back arrows : linear velocity incr/decr.
        Right/left arrows : angular velocity incr/decr.
        Spacebar : reset linear/angular velocities.
        d : disable motors.
        e : enable motors.
        q : quit.

5th SSH terminal, to Ubuntu Linux box

First, install ROS Indigo distribution on Ubuntu Linux box if it has not been done before.

Setup ROS variables on Ubuntu Linux box (to enable communication with ROS host on Sitara) then start RViz

export ROS_MASTER_URI=http://$SITARA_IP_ADDR:11311 (IP address of Sitara EVM, modify as needed)
export ROS_IP=$LINUX_BOX_IP_ADDR (IP address of Ubuntu Linux box, modify as needed)
source /opt/ros/indigo/setup.bash

rosrun rviz rviz

Alternatively, in order to get Kobuki avatar on the screen, install kobuki_description on Ubuntu Linux box and start RViz by launching view_model from kobuki_description.

git clone https://github.com/yujinrobot/kobuki
cd kobuki
cp -r kobuki_description /opt/ros/indigo/share

roslaunch kobuki_description view_model.launch

In RViz add point cloud from mmWave radar:

• Click Add->PointCloud2
• Select /mmWaveDataHdl/RScan from the Topic field dropdown for the PointCloud2 on the left hand panel
• Increase Size to 0.03, increase Decay Time to 0.25, and Select Style as “Spheres”.
• In rviz, select map for Fixed Frame in Global Options.
• If Kobuki is also started, set Reference Frame (left panel) to “map”.

You should see a point cloud image:

More information can be found in ROS driver document in chapters: “Visualizating the data”, “Reconfiguring the chirp profile”, and “How it works”

Starting GStreamer pipeline for streaming front view camer

It is possible to start GStreamer pipeline on Sitara and receive front-camera view on Ubuntu Linux box or Windows PC using VLC.

gst-launch-1.0 -e v4l2src device=/dev/video1  io-mode=5 ! 'video/x-raw, \
format=(string)NV12, width=(int)640, height=(int)480, framerate=(fraction)30/1' ! ducatih264enc bitrate=1000 ! queue ! h264parse config-interval=1 ! mpegtsmux  ! udpsink host=192.168.0.100 sync=false port=5000

E.g. on Windows PC (192.168.0.100), you can watch the stream using: “Program Files (x86)VideoLANVLCvlc.exe” udp://@192.168.0.100:5000

Fig. 5.7 Multiple windows on Ubuntu Linux box showing ROS RViz, front camera view and external camera view

5.3.12. TIDL Classification Demo¶

Refer to various TIDL demos documented at Examples and Demos

5.3.13. Arm NN Classification Demo¶

Refer to Arm NN MobileNet Demo

5.3.14. PRU-ADC Demo¶

cd /boot
cp am335x-boneblack.dtb am335x-boneblack.dtb.orig
cp am335x-boneblack-pru-adc.dtb am335x-boneblack.dtb

cd /lib/firmware
ln -sf /lib/firmware/pru/PRU_ADS8688_Controller.out am335x-pru0-fw
ln -sf /lib/firmware/pru/PRU_ADS8688_Interface.out am335x-pru1-fw

run-pru-adc.sh

5.3.15. Video Graphics Test¶

5.3.15.3. Application Description¶

In the video graphics test application, we have demonstrated embedded application displaying two different videos and graphics contents rendered on two different display monitors in full-screen mode. One of the video sources comes from camera input hooked to parallel capture port VIP. Captured video is submitted to QT eglfs_kms QPA to be rendered on Display1 device. Next, the application takes the relevant information from the camera input and passes two copies of the same image to the GC320. The GC320 takes one copy and does not modify the image. For the second copy, the GC320 will rotate the image by 90 degrees. Then, the GC320 will stitch both images together and the output is submitted to QT plugin for rendering on Display2 device. QT framework is also used to draw analog clock and Maroon game using QT QML. The analog clock is set to be rendered on Display1 and Maroon game to be rendered on Display2. QT uses OpenGL APIs to render the graphics contents using 3D graphics accelerator SGX544. The users submit the properties of video planes like scaling, overlaying, alpha-blending, number of display monitors and etc using the APIs exposed through QPlatformNativeInterface. It sets the video planes to blend with graphics planes and the resulting output renders to the size of the display on both the monitors. QT uses DSS hardware accelerator to accelerate the overlaying, alpha blending and scaling processing.

Video Graphics Test Block Diagram

5.3.15.4. eglfs-kms QPA User Exposed APIs¶

As part of eglfs_kms QPA enhancement, following APIs are exposed via QPlatformNativeInterface to allocate DSS hardware planes, to set the plane properties supported by DSS and to submit the buffer for display.

1. create_plane: Secure DSS hardware planes for overlay from QT QPA
2. distroy_plane: Free the reserved DSS hardware planes
3. export_buffer: Export user allocated buffer to the QT QPA
4. distroy_buffer: Free the buffers
5. set_plane_properties: Set the DSS plane properties for scaling of video planes and blending it with graphics planes. Properties can be set for both DRM plane type Primary and Overlay planes.
6. get_plane_property: Read default/pre-set property value of a plane
7. queue_plane: Queue the video buffer to a QPA plane to be overlayed/displayed
8. start_disp_plane: Begin displaying of the queued buffers
9. user_call_back_handle: Pass callback function pointer to the QT eglfs_kms QPA. The QPA will call this function to notify user the display completion of the overlay buffer

The APIs required to achieve scaling, overlaying and alpha-blending using the DSS IP owned by eglfs_kms QPA are demonstrated inside disp_obj.cpp and video_graphics_test.cpp file. An example of the software flow is shown below:

target # insmod /lib/modules/<kernel_version>/extra/galcore.ko baseAddress=0x80000000 physSize=0x80000000
target # /etc/init.d/weston stop
target # export QT_QPA_EGLFS_INTEGRATION=eglfs_km
target # video-graphics-test -platform eglfs