Smart IoT Vision

Participantes:

Milton Felipe Souza Santos
Gustavo Retuci Pinheiro
Eduardo Caldas Cardoso

Resumo do projeto:

SmartCities solution where the Snapdragon platform is a smart camera and also a camera gateway with edge processing, using cat-m and RF-mesh connectivities. The system detects people and cars to efficiently control smart photocells and light signals.

Descrição do projeto:

The proposed project could be a solution to many problems, but we are focusing on the smart city context. In this matter, there is three major problem we are solving:

1 – energy saving in public lighting; 2 – improve city security; 3 – improve traffic flow.

1 – By using LEDs lights on streets, the savings are already up to 50%, and with the addition of Telemanagement, we can have 30% more savings.
2 – With the use of processors together with cameras, that would be called “smart camera”, we can detect behaviour and control the lights to dim down where the people flow is absent and make the street section brighter where people are walking. It will not only save energy but increase the feeling of being watched, thus, intimidate bad intentioned people. Furthermore, visual alarms (blinking the lamps for example), could be used in case of suspicious behavior.
3 – The smart camera will watch the traffic, process locally its conditions, and control the light signals in order to best manage the traffic. In this way, traffic jams could be avoided, cars would not have to wait for long time red signals when there is no flow in the crossing, and so on.

Regarding technological problems, we are also solving what are common issues in IoT such as robust connectivity in city scale and camera integration for IoT Network, by using edge processing to transmit only relevant information.

See our Instructables and GitHub

Histórico do desenvolvimento:

Week 1 and 2 – Installing OS on the dragonboards

Installing Debian on Dragonboard820C (Fastboot Method)

Fastboot Method

Using a Linux OS, install the packages listed in: https://www.96boards.org/documentation/consumer/dragonboard/dragonboard820c/installation/build-linux-host.md.html

On the dragonboard:

make s4 OFF,OFF,OFF,OFF
Turn on pressing vol (-)
If you are using a serial monitor (highly recommended), you will get the message “fastboot: processing commands” (serial monitor at 115200)

Connect the micro-usb (J4) on the PC

On host PC:

Download (and unzip) from https://www.96boards.org/documentation/consumer/dragonboard/dragonboard820c/downloads/debian.md.html

$ sudo fastboot devices
452bb893 fastboot (example)

$ sudo fastboot flash boot boot-linaro-buster-dragonboard-820c-BUILD.img
$ sudo fastboot flash rootfs linaro-buster-alip-dragonboard-820c-BUILD.img

 

Installing Debian on Dragonboard410C

Steps on the computer (Linux)

1 – Download the image

$ cd ~

$ mkdir Debian_SD_Card_Install_image

$ cd Debian_SD_Card_Install_image

$ wget http://builds.96boards.org/releases/dragonboard410c/linaro/debian/latest/dragonboard410c_sdcard_install_debian*.zip

2 – Unzip the files

$ cd ~/Debian_SD_Card_Install_image

$ unzip dragonboard410c_sdcard_install_debian-233.zip

3 – Insert the microSD on your computer and check if it mounted

$ df -h

/dev/sdb1 7.4G 32K 7.4G 1% /media/3533-3737

4 – Unmount the microSD and burn the image

$ umount /dev/sdb1

$ sudo dd if=db410c_sd_install_debian.img of=/dev/sdb bs=4M oflag=sync status=noxfer

5 – Remove the microSD from you PC

 

Steps on the computer (Windows)

Download – SD Card image – (Option 1) SD Card Image – Install and boot from eMMC
https://www.96boards.org/documentation/consumer/dragonboard/dragonboard410c/downloads/debian.md.html

Unzip SD Card Install Image

Download and install the Win32DiskImager tool
https://sourceforge.net/projects/win32diskimager/files/latest/download

Open Win32DiskImager tool
Insert the SD card in the computer
Find the extracted .img file
Click on Write

Steps on the Dragonboard

Make sure DragonBoard™ 410c is unplugged from power
Set S6 switch on DragonBoard™ 410c to 0-1-0-0, “SD Boot switch” should be set to “ON”.

Connect an HDMI
Plug a USB keyboard
Insert the microSD
Plug power adapter

Select the image to install and click “Install”

wait for the installation to finish
Remove power adapter
Remove the microSD
Set S6 switch to 0-0-0-0

DONE!

Week 3 – 5 – Installing essential software modules

On the development computer

Note that some steps are hardware-dependent and should be adjusted to meet your actual computer specifications. The libraries may be installed with a single command.

sudo apt install build-essential git libatlas libgoogle-glog-dev libiomp-dev libleveldb-dev liblmdb-dev libopencv-dev libopenmpi-dev libsnappy-dev libprotobuf-dev libatlas libboost libgflags2 hdf5 openmpi-bin opnempi-doc protobuf-compiler python-dev python-pip python-numpy python-scipy python-matplotlib python-future python-protobuf python-typing python-hypotesis python-yaml

OpenCV

This framework is used to develop image based statistical algorithms on the development machine. Since most of our code is written in Python, the easiest installation method is to just

pip install opencv-python

Note, however, that these wheels will not use anything apart from your CPU and may not even use all of its cores, so you may want to compile from source to achieve maximum performance. To build the package in Linux, for example, you donwload the zip file form the OpenCV Releases page and unzip it. From the unzipped folder:

mkdir build && cd build
cmake ..
make all -j4
sudo make install

The -j4 command instructs make to use four threads. Use as many as your CPU has!

Caffe

To setup the Caffe framework from sources:

git clone https://github.com/BVLC/caffe.git && cd caffe
mkdir build
cmake ..
make all
make test
make runtest

If all tests run successfully then you are all set.

TensorFlow

Google does not let you compile TensorFlow with ordinary tools. It requires Bazel for it and chances are it won’t work, so avoid compiling it and just grab the pre-compiled module with:

pip install tensorflow

If your computer is bit old and doesn’t have AVX instructions, get the last non-AVX tensorflow with

pip install tensorflow==1.5

And you’re done.

SNPE – Snapdragon™ Neural Processing Engine

Setting Snappy up, as our Qualcomm friends call SNPE, is not hard but the steps should be followed closely. The installation outline is:

  1.  clone the neural network frameworks’ git repositories
    1. Caffe
    2. Caffe2
    3. TensorFlow
    4. ONNX
  2. run the scripts to check for dependencies
    1. snpe/bin/dependencies.sh
    2. snpe/bin/check_python_depends.sh
  3. for each installed framework run snpe/bin/envsetup.sh
    1. source $SNPE/bin/envsetup.sh -c $CAFFE_GIT
    2. source $SNPE/bin/envsetup.sh -f $CAFFE2_GIT
    3. source $SNPE/bin/envsetup.sh -t $TENSORFLOW_GIT
    4. source $SNPE/bin/envsetup.sh -o $ONNX_GIT

To source SNPE in every terminal instance you open, append step three’s four lines to the end of your ~/.bashrc file.

On the target board

Moving to arm64 from amd64 is not an effortless task, as many libraries will take advantage of x86 instructions to boost their performance. Luckily, it is possible to compile most of the necessary resources on the board itself. The needed libraries may be installed with a single command.

sudo apt install build-essential git libatlas libgoogle-glog-dev libiomp-dev libleveldb-dev liblmdb-dev libopencv-dev libopenmpi-dev libsnappy-dev libprotobuf-dev libatlas libboost libgflags2 hdf5 openmpi-bin opnempi-doc protobuf-compiler python-dev python-pip python-numpy python-scipy python-matplotlib python-future python-protobuf python-typing python-hypotesis python-yaml

Install them with apt and move on. Note that this step may take some time, as apt calls make to build the code that’s not pre-compiled.

OpenCV

Download the release from the OpenCV repository, unzip it somewhere and from the unzipped folder:

mkdir build && cd build
cmake ..
make all -j3
sudo make install

Note that we used the -j3 option. If you access the board via ssh, having all cores fully loaded may be enough to drop the connection. That’s not desirable. By limiting thread usage to three, we will always have at least one free thread to cope with ssh connections and general system housekeeping.

This is for the Dragonboard 820 and Inforce 6640 with the APQ8096 chip. On the Dragonboard 410 you will want to have some free virtual memory or limit the compile threads to one, since it has less physical RAM available.

It is also of note that cooling the chip will help increase performance by limiting thermal throttling. A heatsink does the trick at small loads but you will want a proper fan for compiling and other CPU-intensive loads.

Why not install OpenCV with apt or pip? Because compiling it in the target machine makes every available processor instruction visible to the compiler, improving execution performance.

SNPE – Snapdragon™ Neural Processing Engine

We installed Snappy just like it was on a desktop computer, even though there was no actual neural network framework installed (SNPE only needs the git repos, not the actual binaries).

However, since all we need are the binaries and headers for the snpe-net-run command, there’s a possibility that just having the following files on a folder and adding this folder to the PATH works:

  • Neural network binary
    • snpe/bin/aarch64-linux-gcc4.9/snpe-net-run
  • CPU libraries
    • snpe/lib/aarch64-linux-gcc4.9/libSNPE.so
    • snpe/lib/aarch64-linux-gcc4.9/libsymphony-cpu.so
    • /usr/lib/aarch64-linux-gnu/libatomic.so.1
  • DSP libraries
    • snpe/lib/dsp/libsnpe_dsp_skel.so
    • snpe/lib/aarch64-linux-gcc4.9/libsnpe_adsp.so
  • Results viewer
    • snpe/models/alexnet/scripts/show_alexnet_classifications.py

The bold item, /usr/lib/aarch64-linux-gnu/libatomic.so.1, is provided with Linaro on this path and must be copied to this hypothetical minimal folder.

We have yet to test whether this works or not.

Hardware development begins

The first camera chassis were designed and 3D printed during these weeks to check for basic hardware demands. Cameras and primitive algorithms were also tried during this phase.

Our very first prototype was made to be compatible with the CMUCam 5/Pixy Camera.

Other utilities

sudo apt-get install net-tools
sudo apt-get install gedit
sudo apt install nodejs
sudo apt install openvpn

Week 6 – Installing connectivity modules

Installing Cat-m and 3G

Using graphical environment

Connect the modem (oneRF_Modem_v04 – HE910)

Open Network Connections

Click on + to add new connection

Select Mobile Broadband

Select the correct device

Select the Country

Select the provider

Select the plan and Save

Remove the Modem

Reconnect the Modem

Using terminal

apt-get install pppconfig
pppconfig

provider = vivo
dinamico
CHAP
vivo
vivo
115200
Tone
*99#
no (manual)
/dev/ttyUSB0
salvar

cat /etc/ppp/peers/vivo
cat /etc/chatscripts/vivo

pon vivo

If you are using Cat-M module, just use the following commands before:

echo 1bc7 1101 > /sys/bus/usb-serial/drivers/option1/new_id

apt-get install comgt

comgt -d /dev/ttyUSB0
comgt info -d /dev/ttyUSB0

 

Week 6 – 8 – Final hardware development

Smart Cam

As the software platforms were chosen, so did the hardware design evolve. Our halfway to the finished camera was ready by this week.

 

Our final chassis shape was done by the 6th week. We selected a Logitech USB camera and added mount points for most of the electronics.

Camera Gateway

 

Week 9 – Final adjustments and testing

Gallery

Final Result

Hardware:

Dragonboard 410

(https://www.embarcados.com.br/qualcomm-dragonboard-410c/)

(https://www.96boards.org/product/dragonboard410c/)

Dragonboard 820

(https://www.embarcados.com.br/dragonboard-820c/)

(https://developer.qualcomm.com/hardware/dragonboard-820c)

Modem Telit

(https://www.embarcados.com.br/apresentando-o-kit-telit-evk2/)

 

IoT mezzanine

See complete documentation here 

OneRF RF mesh module

OneRF Smart photocell

Usb Camera

(https://pixycam.com/pixy-cmucam5/)

Software/Firmware:

AT commands:

AT#SIMDET?  //  check SIM presence
#SIMDET: 2,0 // sim not inserted
#SIMDET: 2,1 // sim inserted

AT+CREG? //chec if it is registered
+CREG: 0,1 //(disable network registration unsolicited result code (factory default), registered home network)

AT+COPS?
+COPS: 0,0,”VIVO”,2 // (mode = automatic choice, format = alphanumeric, oper, ?)

AT+CPAS //Phone Activity Status
+CPAS: 0 //ready

AT+CSQ  // check quality of service
+CSQ: 16,3 //(rssi, bit error rate)

AT+CGATT? //state of GPRS attachment
+CGATT: 1 //attached

AT+CGDCONT=1,”IP”,”zap.vivo.com.br”,,0,0 // configure context
OK

AT+CGDCONT? // check context
+CGDCONT: 1,”IP”,”zap.vivo.com.br”,””,0,0

AT#SGACT=1,1 //Context Activation
#SGACT: 100.108.48.30
OK

Now you are connected!

Important AT commands

AT#PING=8.8.8.8

AT+CNUM //check SIM number

AT+CMGL // list SMSs

 

Referências:

http://caffe.berkeleyvision.org/install_apt.html

http://caffe.berkeleyvision.org/installation.html#compilation

https://developer.qualcomm.com/docs/snpe/setup.html

https://opencv.org/releases.html

https://github.com/BVLC/caffe

https://github.com/pytorch/pytorch.git

https://github.com/tensorflow/tensorflow

https://github.com/onnx/onnx

https://www.tensorflow.org/install/

https://caffe2.ai/

http://caffe.berkeleyvision.org/

https://onnx.ai/