Smart IoT Vision
Participantes:
Milton Felipe Souza SantosResumo do projeto:
SmartCities solution where the Snapdragon platform is a smart camera and also a camera gateway with edge processing, using cat-m and RF-mesh connectivities. The system detects people and cars to efficiently control smart photocells and light signals.Descrição do projeto:
The proposed project could be a solution to many problems, but we are focusing on the smart city context. In this matter, there is three major problem we are solving:
1 – energy saving in public lighting; 2 – improve city security; 3 – improve traffic flow.
1 – By using LEDs lights on streets, the savings are already up to 50%, and with the addition of Telemanagement, we can have 30% more savings.
2 – With the use of processors together with cameras, that would be called “smart camera”, we can detect behaviour and control the lights to dim down where the people flow is absent and make the street section brighter where people are walking. It will not only save energy but increase the feeling of being watched, thus, intimidate bad intentioned people. Furthermore, visual alarms (blinking the lamps for example), could be used in case of suspicious behavior.
3 – The smart camera will watch the traffic, process locally its conditions, and control the light signals in order to best manage the traffic. In this way, traffic jams could be avoided, cars would not have to wait for long time red signals when there is no flow in the crossing, and so on.
Regarding technological problems, we are also solving what are common issues in IoT such as robust connectivity in city scale and camera integration for IoT Network, by using edge processing to transmit only relevant information.
See our Instructables and GitHub
Histórico do desenvolvimento:
Week 1 and 2 – Installing OS on the dragonboards
Installing Debian on Dragonboard820C (Fastboot Method)
Fastboot Method
Using a Linux OS, install the packages listed in: https://www.96boards.org/documentation/consumer/dragonboard/dragonboard820c/installation/build-linux-host.md.html
On the dragonboard:
make s4 OFF,OFF,OFF,OFF
Turn on pressing vol (-)
If you are using a serial monitor (highly recommended), you will get the message “fastboot: processing commands” (serial monitor at 115200)
Connect the micro-usb (J4) on the PC
On host PC:
Download (and unzip) from https://www.96boards.org/documentation/consumer/dragonboard/dragonboard820c/downloads/debian.md.html
$ sudo fastboot devices
452bb893 fastboot (example)
$ sudo fastboot flash boot boot-linaro-buster-dragonboard-820c-BUILD.img
$ sudo fastboot flash rootfs linaro-buster-alip-dragonboard-820c-BUILD.img
Installing Debian on Dragonboard410C
Steps on the computer (Linux)
1 – Download the image
$ cd ~
$ mkdir Debian_SD_Card_Install_image
$ cd Debian_SD_Card_Install_image
2 – Unzip the files
$ cd ~/Debian_SD_Card_Install_image
$ unzip dragonboard410c_sdcard_install_debian-233.zip
3 – Insert the microSD on your computer and check if it mounted
$ df -h
/dev/sdb1 7.4G 32K 7.4G 1% /media/3533-3737
4 – Unmount the microSD and burn the image
$ umount /dev/sdb1
$ sudo dd if=db410c_sd_install_debian.img of=/dev/sdb bs=4M oflag=sync status=noxfer
5 – Remove the microSD from you PC
Steps on the computer (Windows)
Download – SD Card image – (Option 1) SD Card Image – Install and boot from eMMC
https://www.96boards.org/documentation/consumer/dragonboard/dragonboard410c/downloads/debian.md.html
Unzip SD Card Install Image
Download and install the Win32DiskImager tool
https://sourceforge.net/projects/win32diskimager/files/latest/download
Open Win32DiskImager tool
Insert the SD card in the computer
Find the extracted .img file
Click on Write
Steps on the Dragonboard
Make sure DragonBoard™ 410c is unplugged from power
Set S6 switch on DragonBoard™ 410c to 0-1-0-0, “SD Boot switch” should be set to “ON”.
Connect an HDMI
Plug a USB keyboard
Insert the microSD
Plug power adapter
Select the image to install and click “Install”
wait for the installation to finish
Remove power adapter
Remove the microSD
Set S6 switch to 0-0-0-0
DONE!
Week 3 – 5 – Installing essential software modules
On the development computer
Note that some steps are hardware-dependent and should be adjusted to meet your actual computer specifications. The libraries may be installed with a single command.
sudo apt install build-essential git libatlas libgoogle-glog-dev libiomp-dev libleveldb-dev liblmdb-dev libopencv-dev libopenmpi-dev libsnappy-dev libprotobuf-dev libatlas libboost libgflags2 hdf5 openmpi-bin opnempi-doc protobuf-compiler python-dev python-pip python-numpy python-scipy python-matplotlib python-future python-protobuf python-typing python-hypotesis python-yaml
OpenCV
This framework is used to develop image based statistical algorithms on the development machine. Since most of our code is written in Python, the easiest installation method is to just
pip install opencv-python
Note, however, that these wheels will not use anything apart from your CPU and may not even use all of its cores, so you may want to compile from source to achieve maximum performance. To build the package in Linux, for example, you donwload the zip file form the OpenCV Releases page and unzip it. From the unzipped folder:
mkdir build && cd build cmake .. make all -j4 sudo make install
The -j4 command instructs make to use four threads. Use as many as your CPU has!
Caffe
To setup the Caffe framework from sources:
git clone https://github.com/BVLC/caffe.git && cd caffe mkdir build cmake .. make all make test make runtest
If all tests run successfully then you are all set.
TensorFlow
Google does not let you compile TensorFlow with ordinary tools. It requires Bazel for it and chances are it won’t work, so avoid compiling it and just grab the pre-compiled module with:
pip install tensorflow
If your computer is bit old and doesn’t have AVX instructions, get the last non-AVX tensorflow with
pip install tensorflow==1.5
And you’re done.
SNPE – Snapdragon™ Neural Processing Engine
Setting Snappy up, as our Qualcomm friends call SNPE, is not hard but the steps should be followed closely. The installation outline is:
- clone the neural network frameworks’ git repositories
- Caffe
- Caffe2
- TensorFlow
- ONNX
- run the scripts to check for dependencies
- snpe/bin/dependencies.sh
- snpe/bin/check_python_depends.sh
- for each installed framework run snpe/bin/envsetup.sh
-
source $SNPE/bin/envsetup.sh -c $CAFFE_GIT
-
source $SNPE/bin/envsetup.sh -f $CAFFE2_GIT
-
source $SNPE/bin/envsetup.sh -t $TENSORFLOW_GIT
-
source $SNPE/bin/envsetup.sh -o $ONNX_GIT
-
To source SNPE in every terminal instance you open, append step three’s four lines to the end of your ~/.bashrc file.
On the target board
Moving to arm64 from amd64 is not an effortless task, as many libraries will take advantage of x86 instructions to boost their performance. Luckily, it is possible to compile most of the necessary resources on the board itself. The needed libraries may be installed with a single command.
sudo apt install build-essential git libatlas libgoogle-glog-dev libiomp-dev libleveldb-dev liblmdb-dev libopencv-dev libopenmpi-dev libsnappy-dev libprotobuf-dev libatlas libboost libgflags2 hdf5 openmpi-bin opnempi-doc protobuf-compiler python-dev python-pip python-numpy python-scipy python-matplotlib python-future python-protobuf python-typing python-hypotesis python-yaml
Install them with apt and move on. Note that this step may take some time, as apt calls make to build the code that’s not pre-compiled.
OpenCV
Download the release from the OpenCV repository, unzip it somewhere and from the unzipped folder:
mkdir build && cd build cmake .. make all -j3 sudo make install
Note that we used the -j3 option. If you access the board via ssh, having all cores fully loaded may be enough to drop the connection. That’s not desirable. By limiting thread usage to three, we will always have at least one free thread to cope with ssh connections and general system housekeeping.
This is for the Dragonboard 820 and Inforce 6640 with the APQ8096 chip. On the Dragonboard 410 you will want to have some free virtual memory or limit the compile threads to one, since it has less physical RAM available.
It is also of note that cooling the chip will help increase performance by limiting thermal throttling. A heatsink does the trick at small loads but you will want a proper fan for compiling and other CPU-intensive loads.
Why not install OpenCV with apt or pip? Because compiling it in the target machine makes every available processor instruction visible to the compiler, improving execution performance.
SNPE – Snapdragon™ Neural Processing Engine
We installed Snappy just like it was on a desktop computer, even though there was no actual neural network framework installed (SNPE only needs the git repos, not the actual binaries).
However, since all we need are the binaries and headers for the snpe-net-run command, there’s a possibility that just having the following files on a folder and adding this folder to the PATH works:
- Neural network binary
- snpe/bin/aarch64-linux-gcc4.9/snpe-net-run
- CPU libraries
- snpe/lib/aarch64-linux-gcc4.9/libSNPE.so
- snpe/lib/aarch64-linux-gcc4.9/libsymphony-cpu.so
- /usr/lib/aarch64-linux-gnu/libatomic.so.1
- DSP libraries
- snpe/lib/dsp/libsnpe_dsp_skel.so
- snpe/lib/aarch64-linux-gcc4.9/libsnpe_adsp.so
- Results viewer
- snpe/models/alexnet/scripts/show_alexnet_classifications.py
The bold item, /usr/lib/aarch64-linux-gnu/libatomic.so.1, is provided with Linaro on this path and must be copied to this hypothetical minimal folder.
We have yet to test whether this works or not.
Hardware development begins
The first camera chassis were designed and 3D printed during these weeks to check for basic hardware demands. Cameras and primitive algorithms were also tried during this phase.
Other utilities
sudo apt-get install net-tools
sudo apt-get install gedit
sudo apt install nodejs
sudo apt install openvpn
Week 6 – Installing connectivity modules
Installing Cat-m and 3G
Using graphical environment
Connect the modem (oneRF_Modem_v04 – HE910)
Open Network Connections
Click on + to add new connection
Select Mobile Broadband
Select the correct device
Select the Country
Select the provider
Select the plan and Save
Remove the Modem
Reconnect the Modem
Using terminal
apt-get install pppconfig
pppconfig
provider = vivo
dinamico
CHAP
vivo
vivo
115200
Tone
*99#
no (manual)
/dev/ttyUSB0
salvar
cat /etc/ppp/peers/vivo
cat /etc/chatscripts/vivo
pon vivo
If you are using Cat-M module, just use the following commands before:
echo 1bc7 1101 > /sys/bus/usb-serial/drivers/option1/new_id
apt-get install comgt
comgt -d /dev/ttyUSB0
comgt info -d /dev/ttyUSB0
Week 6 – 8 – Final hardware development
Smart Cam
As the software platforms were chosen, so did the hardware design evolve. Our halfway to the finished camera was ready by this week.
Our final chassis shape was done by the 6th week. We selected a Logitech USB camera and added mount points for most of the electronics.
Camera Gateway
Week 9 – Final adjustments and testing
Gallery
Final Result
Hardware:
Dragonboard 410
(https://www.embarcados.com.br/qualcomm-dragonboard-410c/)
(https://www.96boards.org/product/dragonboard410c/)
Dragonboard 820
(https://www.embarcados.com.br/dragonboard-820c/)
(https://developer.qualcomm.com/hardware/dragonboard-820c)
Modem Telit
(https://www.embarcados.com.br/apresentando-o-kit-telit-evk2/)
IoT mezzanine
See complete documentation here
OneRF RF mesh module
OneRF Smart photocell
Usb Camera
(https://pixycam.com/pixy-cmucam5/)
Software/Firmware:
AT commands:
AT#SIMDET? // check SIM presence
#SIMDET: 2,0 // sim not inserted
#SIMDET: 2,1 // sim inserted
AT+CREG? //chec if it is registered
+CREG: 0,1 //(disable network registration unsolicited result code (factory default), registered home network)
AT+COPS?
+COPS: 0,0,”VIVO”,2 // (mode = automatic choice, format = alphanumeric, oper, ?)
AT+CPAS //Phone Activity Status
+CPAS: 0 //ready
AT+CSQ // check quality of service
+CSQ: 16,3 //(rssi, bit error rate)
AT+CGATT? //state of GPRS attachment
+CGATT: 1 //attached
AT+CGDCONT=1,”IP”,”zap.vivo.com.br”,,0,0 // configure context
OK
AT+CGDCONT? // check context
+CGDCONT: 1,”IP”,”zap.vivo.com.br”,””,0,0
AT#SGACT=1,1 //Context Activation
#SGACT: 100.108.48.30
OK
Now you are connected!
Important AT commands
AT#PING=8.8.8.8
AT+CNUM //check SIM number
AT+CMGL // list SMSs
Referências:
http://caffe.berkeleyvision.org/install_apt.html
http://caffe.berkeleyvision.org/installation.html#compilation
https://developer.qualcomm.com/docs/snpe/setup.html
https://opencv.org/releases.html
https://github.com/BVLC/caffe
https://github.com/pytorch/pytorch.git
https://github.com/tensorflow/tensorflow
https://github.com/onnx/onnx
https://www.tensorflow.org/install/
https://caffe2.ai/
http://caffe.berkeleyvision.org/
https://onnx.ai/