大数据实战之App管理平台日志分析（一）-项目区-About云-梭伦科技

levycui 发表于 2019-4-17 17:02:15

大数据实战之App管理平台日志分析（一）

问题导读：
1、如何进行项目分析流程？
2、如何初始化web日志收集程序？
3、如何创建app日志收集web模块app-log-collect-web ？
4、如何创建UploadUtil类，模拟日志上传到web？

static/image/hrline/line7.png

一、项目介绍
-------------------------------------------------
1.App开发商每个开发商可以有多个App产品
2.App软件
3.数据服务平台提供商 Umeng，向App开发商提供服务。提供App使用情况的统计服务。
4.SDK 数据服务平台提供商提供给App开发商的软件包。内置log上报程序。
5.用户每个使用App的设备。
6.租户购买了数据服务平台提供商服务的App开发商。

二、项目分析
------------------------------------------------------
1.用户设备id,唯一性
2.新增用户首次打开应用的用户。卸载再安装不是新增
3.活跃用户指定时间段内打开过app的用户即为活跃用户。多次打开算一次。
4.月活率活跃用户 / 截止到当月累计用户总数。
5.沉默用户两天时间没有启动过app的用户就算沉默用户。
6.版本分布计算各版本的新增用户、活跃用户、启动次数。
7.本周回流用户上周没启动，本周启动的用户
8.连续n周活跃用户连续n周，每周至少启动一次。
9.忠诚用户连续5周以上活跃用户
10.连续活跃用户连续2周以上
11.近期流失用户连续n(2<= n <= 4)周没有启动应用的用户。
12.留存用户某段时间内的新增用户，在经过一段时间后，仍然使用app的用户。
13.用户新鲜度每天启动app的新老用户比例
14.单次使用时长每次启动使用的时间长度。
15.日使用时长每天的使用累加值。
16.启动次数计算标准两次之间<30s,算作一次启动.

三、日志组成
--------------------------------------------------
1.启动日志
2.页面访问日志
3.事件日志
4.用户使用日志
5.错误日志

四、开始项目 -- 初始化web日志收集程序
----------------------------------------------------
1.创建新项目UmengProject
2.创建公共模块app-analyze-common，存放用于跨模块间访问的类。添加maven依赖 a.创建日志类AppLogEntity

package com.test.app.common;

         /**
         * AppLog实体类
         * 内部含有各种日志时间的集合。
         */
         public class AppLogEntity {
            private String appId;             //应用唯一标识
            private String tenantId;          //租户唯一标识,企业用户
            private String deviceId;          //设备唯一标识
            private String appVersion;          //版本
            private String appChannel;          //渠道,安装时就在清单中制定了，appStore等。
            private String appPlatform;          //平台
            private String osType;             //操作系统
            private String deviceStyle;          //机型

            private AppStartupLog[] appStartupLogs;    //启动相关信息的数组
            private AppPageLog[] appPageLogs;          //页面跳转相关信息的数组
            private AppEventLog[] appEventLogs;          //事件相关信息的数组
            private AppUsageLog[] appUsageLogs;          //app使用情况相关信息的数组
            private AppErrorLog[] appErrorLogs;          //错误相关信息的数组

            /** get / set**/
         }

b.抽取共性，创建日志基本父类AppBaseLog
package com.test.app.common;

         import java.io.Serializable;

         /**
            * AppBaseLog
            */
         public class AppBaseLog implements Serializable {
               private Long createdAtMs;          //日志创建时间
               private String appId;             //应用唯一标识
               private String tenantId;          //租户唯一标识,企业用户
               private String deviceId;          //设备唯一标识
               private String appVersion;          //版本
               private String appChannel;          //渠道,安装时就在清单中制定了，appStore等。
               private String appPlatform;          //平台
               private String osType;             //操作系统
               private String deviceStyle;          //机型

               /*省略get set*/
         }

c.创建日志具体分类的相关类AppStartupLog等
===================================================
package com.test.app.common;

         /**
         * 启动日志
         */
         public class AppStartupLog extends AppBaseLog {
            private String country;             //国家，终端不用上报，服务器自动填充该属性
            private String province;             //省份，终端不用上报，服务器自动填充该属性
            private String ipAddress;             //ip地址

            private String network;             //网络
            private String carrier;             //运营商

            private String brand;             //品牌
            private String deviceStyle;          //机型
            private String screenSize;          //分辨率
            private String osType;             //操作系统

            //省略getset

         }

====================================================================
package com.test.app.common;

         /**
         * 应用上报的app错误日志相关信息
         */
         public class AppErrorLog extends AppBaseLog {

            private static final long serialVersionUID = 1L;

            private String errorBrief; //错误摘要
            private String errorDetail;    //错误详情
         }

=======================================================================
package com.test.app.common;

         import java.util.Map;

         /**
         * 应用上报的事件相关信息
         *
         */
         public class AppEventLog extends AppBaseLog {

            private static final long serialVersionUID = 1L;

            private String eventId;       //事件唯一标识
            private Long eventDurationSecs; //事件持续时长
            private Map<String,String> paramKeyValueMap;    //参数名/值对
         }

=========================================================================
package com.test.app.common;

         /**
         * 应用上报的页面相关信息
         *
         */
         public class AppPageLog extends AppBaseLog {

            private static final long serialVersionUID = 1L;

            /*
            * 一次启动中的页面访问次数(应保证每次启动的所有页面日志在一次上报中，即最后一条上报的页面记录的nextPage为空)
            */
            private int pageViewCntInSession = 0;

            private String pageId;    //页面id
            private int visitIndex = 0; //访问顺序号，0为第一个页面
            private String nextPage; //下一个访问页面，如为空则表示为退出应用的页面
            private Long stayDurationSecs = (long) 0;//当前页面停留时长
         }

=============================================================================
package com.test.app.common;

         /**
         * 应用上报的使用时长相关信息
         */
         public class AppUsageLog extends AppBaseLog {

            private static final long serialVersionUID = 1L;

            private Long singleUseDurationSecs; //单次使用时长(秒数),指一次启动内应用在前台的持续时长
            private Long singleUploadTraffic; //单次使用过程中的上传流量
            private Long singleDownloadTraffic;    //单次使用过程中的下载流量
         }

d.创建util工具包com.test.app.util和属性拷贝工具类PropertiesUtil
package com.test.app.util;

         import java.beans.BeanInfo;
         import java.beans.IntrospectionException;
         import java.beans.Introspector;
         import java.beans.PropertyDescriptor;
         import java.lang.reflect.Method;

         /**
         * 通过内省实现属性复制
         */
         public class PropertiesUtil {

            //复制属性
            //要将AppLogEntity中的所有相关属性的值赋值给对用的具体log的属性值[因为具体log的属性值是空的]
            public static void copyProperties(Object src, Object des)
            {
               try {
                     BeanInfo srcInfo = Introspector.getBeanInfo(src.getClass());
                     //属性描述符
                     PropertyDescriptor[] sarr = srcInfo.getPropertyDescriptors();
                     for(PropertyDescriptor p : sarr)
                     {
                        Method getter = p.getReadMethod();
                        Method setter = p.getWriteMethod();
                        //获取set方法描述符的name
                        String setName = setter.getName();
                        //获取set方法的参数类型
                        Class [] param = setter.getParameterTypes();
                        try {
                           //通过get方法描述符，获取src的属性值
                           Object value = getter.invoke(src);
                           //通过src set方法描述符找到des的set方法，给des的属性赋值
                           Method desSettrer = des.getClass().getMethod(setName, param);
                           desSettrer.invoke(des, value);

                        } catch (Exception e) {
                           //出现异常说明des中没有src的属性
                           continue;
                        }
                     }
               } catch (Exception e) {
                     e.printStackTrace();
               }
            }
         }

3.创建app日志收集web模块app-log-collect-web
a.添加maven依赖
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion>
<groupId>com.test</groupId>
<artifactId>app-logs-collect-web</artifactId>
<version>1.0-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
</dependency>

<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<version>2.8.8</version>
</dependency>

<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.8.3</version>
</dependency>

<dependency>
<groupId>com.maxmind.db</groupId>
<artifactId>maxmind-db</artifactId>
<version>1.0.0</version>
</dependency>

<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-webmvc</artifactId>
<version>4.3.5.RELEASE</version>
</dependency>

<dependency>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
<version>2.5</version>
</dependency>

<dependency>
<groupId>com.test</groupId>
<artifactId>app-analyze-common</artifactId>
<version>1.0-SNAPSHOT</version>
</dependency>

<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId> <version>1.2.24</version>
</dependency> </dependencies>
</project>

b.添加app-analyze-common模块依赖，共享模块资源
Project-Structure --> dependencies --> 3.Module De...
别忘记打钩，不然引用不了
c.创建java包
com.test.applogs.collect.web.controller
d.编写WEB-INF/web.xml
<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns="http://xmlns.jcp.org/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee http://xmlns.jcp.org/xml/ns/javaee/web-app_3_1.xsd" version="3.1">
<servlet>
<servlet-name>controller</servlet-name>
<servlet-class>org.springframework.web.servlet.DispatcherServlet </servlet-class>
</servlet>

<servlet-mapping>
<servlet-name>controller</servlet-name>
<url-pattern>/</url-pattern>
</servlet-mapping>

</web-app>

e.创建新文件WEB-INF/controller-servlet.xml
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:mvc="http://www.springframework.org/schema/mvc" xmlns:context="http://www.springframework.org/schema/context" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/mvc
http://www.springframework.org/schema/mvc/spring-mvc-4.3.xsd
http://www.springframework.org/schema/context
http://www.springframework.org/s ... ing-context-4.3.xsd">

<context:component-scan base-package="com.test.applogs.collect.web.controller"/>

<mvc:annotation-driven/>

<bean class="org.springframework.web.servlet.view.InternalResourceViewResolver">
<property name="prefix" value="/views"/>
<property name="suffix" value=".jsp"/>
</bean>

<bean id="jsonMapping" class="org.springframework.http.converter.json.MappingJackson2HttpMessageConverter"/> <bean class="org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter">
<property name="messageConverters">
<list> <ref bean="jsonMapping"/>
</list>
</property>
</bean>
</beans>
f.创建日志收集控制器
package com.test.applogs.collect.web.controller;

         import com.alibaba.fastjson.JSONObject;
         import com.test.app.util.PropertiesUtil;
         import com.test.app.common.*;
         import com.test.app.common.AppLogEntity;
         import com.test.app.common.AppStartupLog;
         import org.springframework.stereotype.Controller;
         import org.springframework.web.bind.annotation.RequestBody;
         import org.springframework.web.bind.annotation.RequestMapping;
         import org.springframework.web.bind.annotation.RequestMethod;
         import org.springframework.web.bind.annotation.ResponseBody;

         import javax.servlet.http.HttpServletRequest;

         /**
         */
         @Controller()
         @RequestMapping("/coll")
         public class CollectLogController {

            /**
            * 启动日志收集
            */
            @RequestMapping(value = "/index", method = RequestMethod.POST)
            @ResponseBody
            public AppLogEntity collect(@RequestBody AppLogEntity e, HttpServletRequest req) {

               System.out.println("=============================");
               //server时间
               long myTime = System.currentTimeMillis() ;
               //客户端时间
               long clientTime = Long.parseLong(req.getHeader("clientTime"));
               //时间校对
               long diff = myTime - clientTime ;

               //对e进行处理，将具体日志分类的属性值填充完毕
               processLogs(e);
               //修正日志时间
               verifyTime(e,diff);
               String json = JSONObject.toJSONString(e);

               System.out.println(json);
               return e;
            }

            /**
            * 校对各个具体日志的创建时间（使用服务器时间差diff）
            */
            private void verifyTime(AppLogEntity e, long diff)
            {
               //启动修正
               //startuplog
               for(AppBaseLog log : e.getAppStartupLogs()){
                     log.setCreatedAtMs(log.getCreatedAtMs() + diff );
               }
               for(AppBaseLog log : e.getAppUsageLogs()){
                     log.setCreatedAtMs(log.getCreatedAtMs() + diff );
               }
               for(AppBaseLog log : e.getAppPageLogs()){
                     log.setCreatedAtMs(log.getCreatedAtMs() + diff );
               }
               for(AppBaseLog log : e.getAppEventLogs()){
                     log.setCreatedAtMs(log.getCreatedAtMs() + diff );
               }
               for(AppBaseLog log : e.getAppErrorLogs()){
                     log.setCreatedAtMs(log.getCreatedAtMs() + diff );
               }
            }

            /**
            * 将Log的属性分类复制到各个具体的log中
            */
            private void processLogs(AppLogEntity e){

               for(AppStartupLog log : e.getAppStartupLogs()){
                     PropertiesUtil.copyProperties(e,log);
               }
               for(AppErrorLog log : e.getAppErrorLogs()){
                     PropertiesUtil.copyProperties(e,log);
               }
               for(AppEventLog log : e.getAppEventLogs()){
                     PropertiesUtil.copyProperties(e,log);
               }
               for(AppPageLog log : e.getAppPageLogs()){
                     PropertiesUtil.copyProperties(e,log);
               }
               for(AppUsageLog log : e.getAppUsageLogs()){
                     PropertiesUtil.copyProperties(e,log);
               }
            }
         }

g.创建tomcat-server启动程序
Run --> Edit Configurations --> + --> tomcat server --> ...
Project Structure --> Artifacts --> Add All Maven Jars --> ...
[注意！]Project Structure --> Artifacts --> Add Common模块到out class --> 不然依赖的其他模块是加载不到的，同时添加maven依赖的支持

4.创建app-logs-phone模块，用于模拟手机生成日志
a.添加maven依赖[可以通过maven的手段，达到访问其他模块的内容，比如common模块]
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.test</groupId>
<artifactId>app-logs-phone</artifactId>
<version>1.0-SNAPSHOT</version>
<dependencies>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
<version>1.2.24</version>
</dependency>
<dependency>
<groupId>com.test</groupId>
<artifactId>app-analyze-common</artifactId>
<version>1.0-SNAPSHOT</version>
</dependency>
</dependencies>
</project>
b.创建海量日志生成模拟程序

package com.test.app.client;

         import com.alibaba.fastjson.JSONObject;
         import com.test.app.common.*;
         //import com.test.app.util.PropertiesUtil;

         import java.io.File;
         import java.io.FileWriter;
         import java.io.InputStream;
         import java.net.HttpURLConnection;
         import java.net.URL;
         import java.util.HashMap;
         import java.util.Map;
         import java.util.Random;

         /**
         * 数据生成程序
         */
         public class TestGenData {

            /**
            *
            */
            private static String url = "http://localhost:8080/coll/index";

            private static Random random = new Random();

            private static String appId = "sdk34734";
            private static String[] tenantIds = {"cake"};
            private static String[] deviceIds = initDeviceId();
            private static String[] appVersions = {"3.2.1", "3.2.2"};
            private static String[] appChannels = {"youmeng1", "youmeng2"};
            private static String[] appPlatforms = {"android", "ios"};

            private static Long[] createdAtMsS = initCreatedAtMs();
            //国家，终端不用上报，服务器自动填充该属性
            private static String[] countrys = {"America", "china"};
            //省份，终端不用上报，服务器自动填充该属性
            private static String[] provinces = {"Washington", "jiangxi", "beijing"};
            //网络
            private static String[] networks = {"WiFi", "CellNetwork"};
            //运营商
            private static String[] carriers = {"中国移动", "中国电信", "EE"};
            //机型
            private static String[] deviceStyles = {"iPhone 6", "iPhone 6 Plus", "红米手机1s"};
            //分辨率
            private static String[] screenSizes = {"1136*640", "960*640", "480*320"};
            //操作系统
            private static String[] osTypes = {"8.3", "7.1.1"};
            //品牌
            private static String[] brands = {"三星", "华为", "Apple", "魅族", "小米", "锤子"};
            //事件唯一标识
            private static String[] eventIds = {"popMenu", "autoImport", "BookStore"};
            //事件持续时长
            private static Long[] eventDurationSecsS = {new Long(25), new Long(67), new Long(45)};

            static Map<String, String> map1 = new HashMap<String, String>() {
               {
                  put("testparam1key", "testparam1value");
                  put("testparam2key", "testparam2value");
               }
            };
            static Map<String, String> map2 = new HashMap<String, String>() {
               {
                  put("testparam3key", "testparam3value");
                  put("testparam4key", "testparam4value");
               }
            };
            private static Map[] paramKeyValueMapsS = {map1, map2};

            //单次使用时长(秒数),指一次启动内应用在前台的持续时长
            private static Long[] singleUseDurationSecsS = initSingleUseDurationSecs();

            private static String[] errorBriefs = {"at cn.lift.dfdf.web.AbstractBaseController.validInbound(AbstractBaseController.java:72)", "at cn.lift.appIn.control.CommandUtil.getInfo(CommandUtil.java:67)"};    //错误摘要
            private static String[] errorDetails = {"java.lang.NullPointerException\\n " + "at cn.lift.appIn.web.AbstractBaseController.validInbound(AbstractBaseController.java:72)\\n " + "at cn.lift.dfdf.web.AbstractBaseController.validInbound", "at cn.lift.dfdfdf.control.CommandUtil.getInfo(CommandUtil.java:67)\\n " + "at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\\n" + " at java.lang.reflect.Method.invoke(Method.java:606)\\n"};    //错误详情
            //页面id
            private static String[] pageIds = {"list.html", "main.html", "test.html"};
            //访问顺序号，0为第一个页面
            private static int[] visitIndexs = {0, 1, 2, 3, 4};
            //下一个访问页面，如为空则表示为退出应用的页面
            private static String[] nextPages = {"list.html", "main.html", "test.html", null};
            //当前页面停留时长
            private static Long[] stayDurationSecsS = {new Long(45), new Long(2), new Long(78)};

            //启动相关信息的数组
            private static AppStartupLog[] appStartupLogs = initAppStartupLogs();
            //页面跳转相关信息的数组
            private static AppPageLog[] appPageLogs = initAppPageLogs();
            //事件相关信息的数组
            private static AppEventLog[] appEventLogs = initAppEventLogs();
            //app使用情况相关信息的数组
            private static AppUsageLog[] appUsageLogs = initAppUsageLogs();
            //错误相关信息的数组
            private static AppErrorLog[] appErrorLogs = initAppErrorLogs();

            private static String[] initDeviceId() {
               String base = "device22";
               String[] result = new String;
               for (int i = 0; i < 100; i++) {
                  result = base + i + "";
               }
               return result;
            }

            private static Long[] initCreatedAtMs() {
               Long createdAtMs = System.currentTimeMillis();
               Long[] result = new Long;
               for (int i = 0; i < 10; i++) {
                  result = createdAtMs - (long) (i * 24 * 3600 * 1000);
               }
               result = createdAtMs;
               return result;
            }

            private static Long[] initSingleUseDurationSecs() {
               Random random = new Random();
               Long[] result = new Long;
               for (int i = 1; i < 200; i++) {
                  result = (long) random.nextInt(200);
               }
               return result;
            }

            //启动相关信息的数组
            private static AppStartupLog[] initAppStartupLogs() {
               AppStartupLog[] result = new AppStartupLog;
               for (int i = 0; i < 10; i++) {
                  AppStartupLog appStartupLog = new AppStartupLog();
                  appStartupLog.setCountry(countrys);
                  appStartupLog.setProvince(provinces);
                  appStartupLog.setNetwork(networks);
                  appStartupLog.setCarrier(carriers);
                  appStartupLog.setDeviceStyle(deviceStyles);
                  appStartupLog.setScreenSize(screenSizes);
                  appStartupLog.setOsType(osTypes);
                  appStartupLog.setBrand(brands);
                  appStartupLog.setCreatedAtMs(createdAtMsS);
                  result = appStartupLog;
               }
               return result;
            }

            //页面跳转相关信息的数组
            private static AppPageLog[] initAppPageLogs() {
               AppPageLog[] result = new AppPageLog;
               for (int i = 0; i < 10; i++) {
                  AppPageLog appPageLog = new AppPageLog();
                  String pageId = pageIds;
                  int visitIndex = visitIndexs;
                  String nextPage = nextPages;
                  while (pageId.equals(nextPage)) {
                     nextPage = nextPages;
                  }
                  Long stayDurationSecs = stayDurationSecsS;

                  appPageLog.setPageId(pageId);
                  appPageLog.setStayDurationSecs(stayDurationSecs);
                  appPageLog.setVisitIndex(visitIndex);
                  appPageLog.setNextPage(nextPage);
                  appPageLog.setCreatedAtMs(createdAtMsS);
                  result = appPageLog;
               }
               return result;
            }

            ;

            //事件相关信息的数组
            private static AppEventLog[] initAppEventLogs() {
               AppEventLog[] result = new AppEventLog;
               for (int i = 0; i < 10; i++) {
                  AppEventLog appEventLog = new AppEventLog();
                  appEventLog.setEventId(eventIds);
                  appEventLog.setParamKeyValueMap(paramKeyValueMapsS);
                  appEventLog.setEventDurationSecs(eventDurationSecsS);
                  appEventLog.setCreatedAtMs(createdAtMsS);
                  result = appEventLog;
               }
               return result;
            }

            ;

            //app使用情况相关信息的数组
            private static AppUsageLog[] initAppUsageLogs() {
               AppUsageLog[] result = new AppUsageLog;
               for (int i = 0; i < 10; i++) {
                  AppUsageLog appUsageLog = new AppUsageLog();
                  appUsageLog.setSingleUseDurationSecs(singleUseDurationSecsS);
                  appUsageLog.setCreatedAtMs(createdAtMsS);
                  result = appUsageLog;
               }
               return result;
            }

            ;

            //错误相关信息的数组
            private static AppErrorLog[] initAppErrorLogs() {
               AppErrorLog[] result = new AppErrorLog;
               for (int i = 0; i < 10; i++) {
                  AppErrorLog appErrorLog = new AppErrorLog();
                  appErrorLog.setErrorBrief(errorBriefs);
                  appErrorLog.setErrorDetail(errorDetails);
                  appErrorLog.setCreatedAtMs(createdAtMsS);
                  appErrorLog.setOsType(osTypes);
                  appErrorLog.setDeviceStyle(deviceStyles);
                  result = appErrorLog;
               }
               return result;
            }

            private static void httpPost(String urlString, String params) {
               URL url;

               try {
                  url = new URL(urlString);
                  HttpURLConnection conn = (HttpURLConnection) url.openConnection();
                  conn.setRequestMethod("POST");
                  conn.setDoOutput(true);
                  conn.setDoInput(true);
                  conn.setUseCaches(false);
                  conn.setInstanceFollowRedirects(true);
                  conn.setRequestProperty("User-Agent",
                        "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0");
                  conn.setRequestProperty("Content-Type", "application/json");
                  conn.setConnectTimeout(1000 * 5);
                  conn.connect();
                  conn.getOutputStream().write(params.getBytes("utf8"));
                  conn.getOutputStream().flush();
                  conn.getOutputStream().close();
                  byte[] buffer = new byte;
                  StringBuffer sb = new StringBuffer();
                  InputStream in = conn.getInputStream();
                  int httpCode = conn.getResponseCode();
                  System.out.println(in.available());
                  while (in.read(buffer, 0, 1024) != -1) {
                     sb.append(new String(buffer));
                  }
                  System.out.println("sb:" + sb.toString());
                  in.close();
                  System.out.println(httpCode);
               } catch (Exception e) {
                  e.printStackTrace();
               }
            }

            public static void main(String[] args) {
               Test1();
            }

            private static void Test1() {
               Random random = new Random();
               try {
                  //发送数据
                  for (int i = 1; i <= 2000; i++) {
                     AppLogEntity logEntity = new AppLogEntity();
                     //渠道
                     logEntity.setAppChannel(appChannels);
                     //appid
                     logEntity.setAppId(appId);
                     //platform
                     logEntity.setAppPlatform(appPlatforms);
                     logEntity.setAppVersion(appVersions);
                     String tenantId = tenantIds;
                     if (tenantId != null) {
                        logEntity.setTenantId(tenantId);
                     }
                     logEntity.setTenantId(tenantIds);
                     logEntity.setDeviceId(deviceIds);

                     //模拟startup log集合
                     logEntity.setAppStartupLogs(new AppStartupLog[]{appStartupLogs});
                     logEntity.setAppEventLogs(new AppEventLog[]{appEventLogs});
                     logEntity.setAppErrorLogs(new AppErrorLog[]{appErrorLogs});
                     logEntity.setAppPageLogs(new AppPageLog[]{appPageLogs});
                     logEntity.setAppUsageLogs(new AppUsageLog[]{appUsageLogs});
                     try {
                        //将对象转换成json string
                        String json = JSONObject.toJSONString(logEntity);
                        UploadUtil.upload(json);
                        Thread.sleep(2000);
                     } catch (Exception ex) {
                        System.out.println(ex);
                     }
                  }
               } catch (Exception ex) {
                  ex.printStackTrace();
               }
            }

            private static void Test2() {
               boolean result = map1.isEmpty();
               System.out.println(result);
            }
         }

c.创建UploadUtil类，模拟日志上传到web
package com.test.app.client;

      import java.io.InputStream;
      import java.io.OutputStream;
      import java.net.HttpURLConnection;
      import java.net.URL;

      /**
         * 模拟手机上报日志程序
         */
      public class UploadUtil {

            /**
            * 上传日志
            */
            public static void upload(String json) throws Exception {
               try{
                  //输入流
                  InputStream in = ClassLoader.getSystemResourceAsStream("log.json");

                  URL url = new URL("http://localhost:8080/coll/index");
                  HttpURLConnection conn = (HttpURLConnection) url.openConnection();
                  //设置请求方式为post
                  conn.setRequestMethod("POST");

                  //时间头用来供server进行时钟校对的
                  conn.setRequestProperty("clientTime",System.currentTimeMillis() + "");
                  //允许上传数据
                  conn.setDoOutput(true);
                  //设置请求的头信息,设置内容类型
                  conn.setRequestProperty("Content-Type", "application/json");

                  //输出流
                  OutputStream out = conn.getOutputStream();
                  out.write(json.getBytes());
                  out.flush();
                  out.close();
                  in.close();
                  int code = conn.getResponseCode();
                  System.out.println(code);
               }
               catch (Exception e){
                  e.printStackTrace();
               }
            }
      }
d.生产环境下特别注意时钟问题手机客户端的时间可能不准，所以不能按照客户端提供的时间来算所以。服务器端收集文件之后要进行时间校对

1)client 发送数据同时，写入clientTime头。 conn.setRequestProperty("clientTime",System.currentTimeMillis() + "");
2)web server //server时间 long myTime = System.currentTimeMillis() ; //客户端时间 long clientTime = Long.parseLong(req.getHeader("clientTime")); //时间校对 long diff = myTime - clientTime ;
3)完成Log实体中公共部分属性和Log类中间属性复制。

作者：葛红富
来源：https://blog.csdn.net/xcvbxv01/article/details/84243540
最新经典文章，欢迎关注公众号http://www.aboutyun.com/data/attachment/forum/201406/15/084659qcxzzg8n59b6zejp.jpg

美丽天空 发表于 2019-4-19 00:03:47

感谢分享

若无梦何远方 发表于 2019-9-5 20:51:06

感谢分享

页: [1]

About云-梭伦科技's Archiver

大数据实战之App管理平台日志分析（一）