BigDataProject_Guide

BigData Project Guide

1. Android

1.1 Compile c-ares

点击此处下载官方 Github ,关于 c-ares 的介绍可查看 README.MD 文件,此处示例版本为 c-ares-1.15.0

1
# 放入 /external/bigdata/ 目录下,添加 Android.mk 和 ares_config.h

Android.mk

1
2
3
4
5
6
7
8
9
10
LOCAL_PATH := $(call my-dir)

include $(LOCAL_PATH)/Makefile.inc

include $(CLEAR_VARS)
LOCAL_MODULE := libcares
LOCAL_CFLAGS += -DHAVE_CONFIG_H
LOCAL_EXPORT_C_INCLUDE_DIRS := $(LOCAL_PATH)
LOCAL_SRC_FILES := $(CSOURCES)
include $(BUILD_SHARED_LIBRARY)

编译 c-ares 模块生成 libcares.so ,此文件用于编译下文的 data_servicemosquitto

1
$ mmm external/bigdata/c-ares

1.2 Compile mosquitto

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
$ ll mosquitto/
total 1904
drwxr-xr-x 14 ranger ranger 4096 Dec 13 14:02 ./
drwxr-xr-x 6 ranger ranger 4096 Dec 18 09:57 ../
-rw-r--r-- 1 ranger ranger 83 Dec 13 14:02 Android.mk
-rw-r--r-- 1 ranger ranger 2745 Dec 13 14:02 CMakeLists.txt
-rw-r--r-- 1 ranger ranger 3941 Dec 13 14:02 CONTRIBUTING.md
-rw-r--r-- 1 ranger ranger 64076 Dec 13 14:02 ChangeLog.txt
-rw-r--r-- 1 ranger ranger 155 Dec 13 14:02 LICENSE.txt
-rw-r--r-- 1 ranger ranger 2593 Dec 13 14:02 Makefile
-rw-r--r-- 1 ranger ranger 637 Dec 13 14:02 README.txt
-rw-r--r-- 1 ranger ranger 2026 Dec 13 14:02 about.html
-rw-r--r-- 1 ranger ranger 230 Dec 13 14:02 aclfile.example
drwxr-xr-x 5 ranger ranger 4096 Dec 13 14:02 certs/
drwxr-xr-x 2 ranger ranger 4096 Dec 13 17:00 client/
-rw-r--r-- 1 ranger ranger 873 Dec 13 14:02 compiling.txt
-rw-r--r-- 1 ranger ranger 782 Dec 13 14:02 config.h
-rw-r--r-- 1 ranger ranger 7889 Dec 13 14:02 config.mk
-rw-r--r-- 1 ranger ranger 36920 Dec 13 14:02 dash2-mosquitto.conf
-rw-r--r-- 1 ranger ranger 1569 Dec 13 14:02 edl-v10
-rw-r--r-- 1 ranger ranger 11695 Dec 13 14:02 epl-v10
-rw-r--r-- 1 ranger ranger 13127 Dec 13 14:02 host-build-output.txt
drwxr-xr-x 2 ranger ranger 4096 Dec 13 14:02 installer/
drwxr-xr-x 3 ranger ranger 4096 Dec 13 17:00 lib/
drwxr-xr-x 3 ranger ranger 4096 Dec 13 14:02 logo/
-rw-r--r-- 1 ranger ranger 37117 Dec 13 14:02 m2.conf
drwxr-xr-x 2 ranger ranger 4096 Dec 13 14:02 man/
drwxr-xr-x 3 ranger ranger 4096 Dec 13 14:02 misc/
-rw-r--r-- 1 ranger ranger 37068 Dec 13 14:02 mosq-test.conf
-rw-r--r-- 1 ranger ranger 37049 Dec 13 14:02 mosquitto.conf
-rw-r--r-- 1 ranger ranger 36999 Dec 13 14:02 mrb-mosquitto.conf
-rw-r--r-- 1 ranger ranger 36920 Dec 13 14:02 nexus-mosquitto.conf
-rw-r--r-- 1 ranger ranger 9230 Dec 13 14:02 notice.html
-rw-r--r-- 1 ranger ranger 23 Dec 13 14:02 pskfile.example
-rw-r--r-- 1 ranger ranger 355 Dec 13 14:02 pwfile.example
-rw-r--r-- 1 ranger ranger 696 Dec 13 14:02 qnx-env-setup
-rw-r--r-- 1 ranger ranger 2096 Dec 13 14:02 readme-windows.txt
-rw-r--r-- 1 ranger ranger 3026 Dec 13 14:02 readme.md
drwxr-xr-x 2 ranger ranger 4096 Dec 13 14:02 security/
drwxr-xr-x 5 ranger ranger 4096 Dec 13 14:02 service/
drwxr-xr-x 3 ranger ranger 4096 Dec 13 17:01 src/
-rw-r--r-- 1 ranger ranger 268 Dec 13 14:02 tcu-env-setup
drwxr-xr-x 5 ranger ranger 4096 Dec 13 14:02 test/
drwxr-xr-x 2 ranger ranger 4096 Dec 13 14:02 tls/
-rw-r--r-- 1 ranger ranger 1460649 Dec 13 14:02 understand-mosquitto.udb

1.3 Compile data_service

此模块编译需要 external/bigdata/protobuf 同时存在,data_service 依赖 libpbdata 。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
$ ll data_service/
total 208
drwxr-xr-x 3 ranger ranger 4096 Dec 13 18:08 ./
drwxr-xr-x 6 ranger ranger 4096 Dec 18 09:57 ../
-rw-r--r-- 1 ranger ranger 905 Dec 13 14:02 Android.mk
-rw-r--r-- 1 ranger ranger 9698 Dec 13 15:24 android_data_retriever.cpp
-rw-r--r-- 1 ranger ranger 852 Dec 13 14:02 android_data_retriever.h
-rw-r--r-- 1 ranger ranger 7831 Dec 13 14:02 data_binder.cpp
-rw-r--r-- 1 ranger ranger 93 Dec 13 14:02 data_binder.h
-rw-r--r-- 1 ranger ranger 528 Dec 13 14:02 data_pool.h
-rw-r--r-- 1 ranger ranger 1931 Dec 13 14:02 data_report.cpp
-rw-r--r-- 1 ranger ranger 1506 Dec 13 14:02 data_report.h
-rw-r--r-- 1 ranger ranger 11098 Dec 13 14:02 data_retriever.cpp
-rw-r--r-- 1 ranger ranger 2408 Dec 13 14:02 data_retriever.h
-rw-r--r-- 1 ranger ranger 478 Dec 13 14:02 data_service.cpp
drwxr-xr-x 2 ranger ranger 4096 Dec 13 14:02 data_svc/
-rw-r--r-- 1 ranger ranger 89 Dec 13 14:02 data_svc.rc
-rw-r--r-- 1 ranger ranger 11155 Dec 13 14:02 data_types.cpp
-rw-r--r-- 1 ranger ranger 6086 Dec 13 14:02 data_types.h
-rw-r--r-- 1 ranger ranger 7979 Dec 13 15:37 error_detector.cpp
-rw-r--r-- 1 ranger ranger 172 Dec 13 14:02 error_detector.h
-rw-r--r-- 1 ranger ranger 9436 Dec 13 14:02 event_data.cpp
-rw-r--r-- 1 ranger ranger 231 Dec 13 14:02 event_data.h
-rw-r--r-- 1 ranger ranger 5898 Dec 13 14:02 idc_client_base_api.h
-rw-r--r-- 1 ranger ranger 8892 Dec 13 18:08 linux_data_retriever.cpp
-rw-r--r-- 1 ranger ranger 901 Dec 13 14:02 linux_data_retriever.h
-rw-r--r-- 1 ranger ranger 937 Dec 13 14:02 mqtt_data_publisher.cpp
-rw-r--r-- 1 ranger ranger 602 Dec 13 14:02 mqtt_data_publisher.h
-rw-r--r-- 1 ranger ranger 13787 Dec 13 14:02 mqtt_data_report.cpp
-rw-r--r-- 1 ranger ranger 1453 Dec 13 14:02 mqtt_data_report.h
-rw-r--r-- 1 ranger ranger 7891 Dec 13 14:02 usb_writer.cpp
-rw-r--r-- 1 ranger ranger 198 Dec 13 14:02 usb_writer.h
-rw-r--r-- 1 ranger ranger 39 Dec 13 14:02 usb_writer_cfg.h
-rw-r--r-- 1 ranger ranger 7563 Dec 13 14:02 utils.cpp
-rw-r--r-- 1 ranger ranger 1659 Dec 13 14:02 utils.h

$ ll protobuf/
total 20
drwxr-xr-x 2 ranger ranger 4096 Dec 13 17:23 ./
drwxr-xr-x 6 ranger ranger 4096 Dec 18 09:57 ../
-rw-r--r-- 1 ranger ranger 573 Dec 13 14:02 Android.mk
-rw-r--r-- 1 ranger ranger 0 Dec 13 14:02 README.txt
-rw-r--r-- 1 ranger ranger 6350 Dec 13 14:02 big_data.proto

Note:

a. linux_data_retriever.cpp and android_data_retriever.cpp

Add “namespace android{}” before “using namespace android”

b. error_detector.cpp

Add “static_cast()” at line 102

2. Ubuntu

此处示例版本为 mosquitto-1.5.5

Download:

https://github.com/eclipse/mosquitto — Build Dependencies

https://mosquitto.org/download/

2.1 Install mosquitto

1
2
$ make
$ sudo make install

Note:

$ make WITH_SRV=no WITH_UUID=no

$ sudo make install

Build Dependencies

  • c-ares (libc-ares-dev on Debian based systems) - disable with make WITH_SRV=no
  • libuuid (uuid-dev) - disable with make WITH_UUID=no
  • libwebsockets (libwebsockets-dev) - enable with make WITH_WEBSOCKETS=yes
  • openssl (libssl-dev on Debian based systems) - disable with make WITH_TLS=no
  • xsltproc (xsltproc and docbook-xsl on Debian based systems) - only needed when building from git sources - disable with make WITH_DOCS=no

安装完成后检查是否生成如下文件:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
$ ll /usr/local/bin/mosquitto*
-rwxr-xr-x 1 root root 53016 Dec 17 19:08 /usr/local/bin/mosquitto_passwd*
-rwxr-xr-x 1 root root 117312 Dec 17 19:08 /usr/local/bin/mosquitto_pub*
-rwxr-xr-x 1 root root 131000 Dec 17 19:08 /usr/local/bin/mosquitto_sub*

ll /usr/local/sbin/mosquitto*
-rwxr-xr-x 1 root root 1017600 Dec 17 19:08 /usr/local/sbin/mosquitto*

$ ll /usr/local/lib/*mosq*
lrwxrwxrwx 1 root root 17 Dec 17 19:08 /usr/local/lib/libmosquitto.so -> libmosquitto.so.1*
-rwxr-xr-x 1 root root 403376 Dec 17 19:08 /usr/local/lib/libmosquitto.so.1*
lrwxrwxrwx 1 root root 19 Dec 17 19:08 /usr/local/lib/libmosquittopp.so -> libmosquittopp.so.1*
-rwxr-xr-x 1 root root 64912 Dec 17 19:08 /usr/local/lib/libmosquittopp.so.1*

# 旧版本 /usr/local/lib/ 下有如下6个文件:
ll /usr/local/lib/*mos*
-rw-r--r-- 1 root root 1269118 Dec 13 17:02 /usr/local/lib/libmosquitto.a
lrwxrwxrwx 1 root root 17 Dec 13 17:02 /usr/local/lib/libmosquitto.so -> libmosquitto.so.1*
-rwxr-xr-x 1 root root 64392 Dec 13 17:02 /usr/local/lib/libmosquitto.so.1*
-rw-r--r-- 1 root root 80912 Dec 13 17:02 /usr/local/lib/libmosquittopp.a
lrwxrwxrwx 1 root root 19 Dec 13 17:02 /usr/local/lib/libmosquittopp.so -> libmosquittopp.so.1*
-rwxr-xr-x 1 root root 18864 Dec 13 17:02 /usr/local/lib/libmosquittopp.so.1*

事实证明 libmosquitto.alibmosquittopp.a 在编译 data_receiver 时会用到,所以还是用旧版本 mosquitto 编译。

2.2 Install protobuf

https://developers.google.com/protocol-buffers/docs/downloads

报错:

1
2
autogen.sh:31:# The absence of a m4 directory in googletest causes autoreconf to fail when
autogen.sh:37:autoreconf -f -i -Wall,no-obsolete

原因是 linux 缺少 autoreconf 工具,安装后即可解决:

1
sudo apt install autoconf automake libtool
1
2
3
4
5
6
7
8
$ ./autogen.sh
$ ./configure –disable-shared
$ make
$ sudo make install
$ sudo ldconfig

$ protoc --version
libprotoc 3.6.1

2.3 Install LAMP(Linux, Apache, MySQL, PHP) Server

2.3.1 Apache

1
2
3
4
5
6
# 安装 apache
sudo apt install apache2
$ apache2 -v
Server version: Apache/2.4.29 (Ubuntu)
Server built: 2018-10-10T18:59:25
# 打开 http://10.244.6.199/

打开 http://your_server_IP_address ,例如 http://10.244.6.199/ ,会显示 Apache2 Ubuntu Default Page 页面,如果您看到此页面,则您的Web服务器现在可以正确安装并通过防火墙访问。

查找服务器公网 IP :

1
2
3
4
$ ip addr show eno1 | grep inet | awk '{ print $2; }' | sed 's/\/.*$//'
10.244.6.199
fe80::882b:520d:3097:6d0c
# 或者 ifconfig

2.3.2 MySQL

1
2
3
4
# 安装 mysql
$ sudo apt install mysql-server
$ mysql --version
mysql Ver 14.14 Distrib 5.7.24, for Linux (x86_64) using EditLine wrapper

2.3.3 PHP

1
2
3
4
5
6
7
# 安装 php
$ sudo apt install php libapache2-mod-php php-mysql
$ php -v
PHP 7.2.10-0ubuntu0.18.04.1 (cli) (built: Sep 13 2018 13:45:02) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
with Zend OPcache v7.2.10-0ubuntu0.18.04.1, Copyright (c) 1999-2018, by Zend Technologies
1
2
3
4
5
6
7
8
9
10
11
12
13
# 重新启动Apache Web服务器
$ sudo /etc/init.d/apache2 restart
# 或者
sudo systemctl restart apache2

# 查看 apache2 服务的状态
sudo systemctl status apache2

# 检查 PHP
$ php -r 'echo "\n\nYour PHP installation is working fine.\n\n\n";'


Your PHP installation is working fine.

在 Web 服务器上测试 PHP 处理

为了测试您的系统配置是否适合 PHP ,请创建一个名为info.php的非常基本的 PHP 脚本。 为了让 Apache 找到这个文件并正确提供它,它必须保存到一个非常特定的目录中,这个目录称为“ web 根目录”。

在 Ubuntu 18.04 中,这个目录位于/var/www/html/ 。 通过运行以下位置在该位置创建文件:

1
2
3
4
$ sudo vim /var/www/html/info.php
<?php
phpinfo();
?>

现在您可以测试您的Web服务器是否能够正确显示由此PHP脚本生成的内容。 要尝试此操作,请在您的Web浏览器中访问此页面,例如 http://10.244.6.199/info.php

1
http://your_server_ip/info.php

2.4 Login mysql and import the specific data schema

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
$ sudo mysql -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 760
Server version: 5.7.24-0ubuntu0.18.04.1 (Ubuntu)

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> create database iviData;
Query OK, 1 row affected (0.00 sec)

mysql> use iviData;
Database changed
mysql> source /home/ranger/work/Ubuntu_tools/10031472_data_receiver/database_schema/iviData.sql
Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.00 sec)
... ...

2.5 Install data_receiver

报错一:

1
2
3
4
$ make clean
make: pkg-config: Command not found
rm -rf ./generated/cpp ./generated/java ./bin ./lib ./obj
rm -f .gen_cpp .gen_java

pkg-config 是一个在源代码编译时查询已安装的的使用接口的计算机工具软件。pkg-config原本是设计用于Linux的,但现在在各个版本的BSDwindowsMac OS XSolaris上都有着可用的版本。

它输出已安装的库的相关信息,包括:

安装 pkg-config 解决如上问题:

1
sudo apt install pkg-config

报错二:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
$ sudo make clean
rm -rf ./generated/cpp ./generated/java ./bin ./lib ./obj
rm -f .gen_cpp .gen_java

$ sudo make all
... ...
/bin/sh: 1: mysql_config: not found
sub_bigdata_src/sub_bigdata.cpp:4:10: fatal error: mysql.h: No such file or directory
#include <mysql.h>
^~~~~~~~~
compilation terminated.
sub_bigdata_src/mqtt_bigdata_client.cpp:10:10: fatal error: my_global.h: No such file or directory
#include <my_global.h>
^~~~~~~~~~~~~
compilation terminated.
Makefile:52: recipe for target 'bin/sub_bigdata' failed
make: *** [bin/sub_bigdata] Error 1

报错,安装如下解决:

1
sudo apt install libmysqlclient-dev

报错三:

1
2
3
4
/usr/bin/ld: cannot find -lcares
collect2: error: ld returned 1 exit status
Makefile:53: recipe for target 'bin/sub_bigdata' failed
make: *** [bin/sub_bigdata] Error 1

/usr/bin/ld: cannot find -lxxx 意思是缺少 libxxx.so 文件,即缺少 libcares.so ,此 so 为 c-ares 编译生成,c-ares Github 地址 ,下载后运行如下编译:

1
2
3
$ sudo ./configure
$ sudo make
$ sudo make install

如此 libcares.so 就生成到 /usr/local/lib/ 目录下。

编译 data_receiver

1
2
$ sudo make clean
$ sudo make all

3. Run the application

1
2
$ ./SocketServer 
./SocketServer: error while loading shared libraries: libjson-c.so.2: cannot open shared object file: No such file or directory

解决(安装libjson-c2):https://ubuntu.pkgs.org/16.04/ubuntu-main-amd64/libjson-c2_0.11-4ubuntu2_amd64.deb.html

Da Ye Lai Ma ~

本文标题:BigDataProject_Guide

文章作者:Ranger Zhou

发布时间:2018年12月14日 - 13:31

最后更新:2018年12月24日 - 09:22

原始链接:http://rangerzhou.top/2018/12/14/BigDataProject_Guide/

许可协议: 署名-非商业性使用-RangerZhou 转载请保留原文链接及作者。

-------------本文结束感谢您的阅读 ^_^-------------