求一段日志分析代码

本人spark菜鸟，现需要分析以下的日志，需要将日志转换为日期 程序packageName 日志等级 描述 ，并存入mysql数据库中，求代码

2016-11-05 20:35:23 [com.marstor.controller.LoginController.handleRequest(259)]-[WARN] username is null or password is null
2016-11-05 23:55:42 [com.marstor.controller.LoginController.loadUserCookieIdentify(1416)]-[INFO] JSESSIONID : 251F7E0588801B2FC2BE3591D045544B
2016-11-05 23:55:42 [com.marstor.controller.LoginController.loadUserCookieIdentify(1416)]-[INFO] td_cookie : 18446744070238079273
2016-11-05 23:55:42 [com.marstor.controller.LoginController.loadUserCookieIdentify(1416)]-[INFO] td_cookie : 18446744072203084016
2016-11-05 23:55:42 [com.marstor.controller.LoginController.loadUserCookieIdentify(1470)]-[INFO] CERT_IDENTIFY : null
2016-11-05 23:55:42 [com.marstor.controller.LoginController.loadUserCookieIdentify(1472)]-[INFO] End cookie
2016-11-05 23:56:50 [com.marstor.service.MailEngine.send(78)]-[ERROR] Mail server connection failed; nested exception is javax.mail.MessagingException: Unknown SMTP host: smtp.163.com (java.net.UnknownHostException: smtp.163.com). Failed messages: javax.mail.MessagingException: Unknown SMTP host: smtp.163.com (java.net.UnknownHostException: smtp.163.com)
2016-10-09 23:57:42 [com.marstor.controller.LoginController.handleRequest(417)]-[INFO] null, IP: 192.168.0.1

这个其实就是对日志的一个处理，可能需要用到正则等对数据的识别。下面有段代码楼主仅供参考
LogRecord 类：



[mw_shl_code=java,true]case class LogRecord (
    clientIpAddress: String,      
    rfc1413ClientIdentity: String,   
    remoteUser: String,             `
    dateTime: String,              //[day/month/year:hour:minute:second zone]
    request: String,                 
    httpStatusCode: String,         
    bytesSent: String,              
    referer: String,               
    userAgent: String               
)[/mw_shl_code]

LogParser 解析类


[mw_shl_code=java,true]import java.util.regex.Pattern
import java.text.SimpleDateFormat
import java.util.Locale
import scala.util.control.Exception._
import java.util.regex.Matcher
import scala.util.{Try, Success, Failure}

@SerialVersionUID(100L)
class LogParser extends Serializable {

    private val ddd = "\\d{1,3}"                     
    private val ip = s"($ddd\\.$ddd\\.$ddd\\.$ddd)?"
    private val client = "(\\S+)"                    
    private val user = "(\\S+)"
    private val dateTime = "(\\[.+?\\])"            
    private val request = "\"(.*?)\""               
    private val status = "(\\d{3})"
    private val bytes = "(\\S+)"                     
    private val referer = "\"(.*?)\""
    private val agent = "\"(.*?)\""
    private val regex = s"$ip $client $user $dateTime $request $status $bytes $referer $agent"
    private val p = Pattern.compile(regex)


    def parseRecord(record: String): Option[AccessLogRecord] = {
        val matcher = p.matcher(record)
        if (matcher.find) {
            Some(buildAccessLogRecord(matcher))
        } else {
            None
        }
    }

    def parseRecordReturningNullObjectOnFailure(record: String): AccessLogRecord = {
        val matcher = p.matcher(record)
        if (matcher.find) {
            buildAccessLogRecord(matcher)
        } else {
            AccessLogParser.nullObjectAccessLogRecord
        }
    }

    private def buildAccessLogRecord(matcher: Matcher) = {
        AccessLogRecord(
            matcher.group(1),
            matcher.group(2),
            matcher.group(3),
            matcher.group(4),
            matcher.group(5),
            matcher.group(6),
            matcher.group(7),
            matcher.group(8),
            matcher.group(9))
    }
}

/**
* 例子：
* 94.102.63.11 - - [21/Jul/2009:02:48:13 -0700] "GET / HTTP/1.1" 200 18209 "http://acme.com/foo.php" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)"
*/
object AccessLogParser {

    val nullObjectAccessLogRecord = AccessLogRecord("", "", "", "", "", "", "", "", "")

    def parseRequestField(request: String): Option[Tuple3[String, String, String]] = {
        val arr = request.split(" ")
        if (arr.size == 3) Some((arr(0), arr(1), arr(2))) else None
    }
    def parseDateField(field: String): Option[java.util.Date] = {
        val dateRegex = "\\[(.*?) .+]"
        val datePattern = Pattern.compile(dateRegex)
        val dateMatcher = datePattern.matcher(field)
        if (dateMatcher.find) {
                val dateString = dateMatcher.group(1)
                println("***** DATE STRING" + dateString)
                // HH is 0-23; kk is 1-24
                val dateFormat = new SimpleDateFormat("dd/MMM/yyyy:HH:mm:ss", Locale.ENGLISH)
                allCatch.opt(dateFormat.parse(dateString))  // return Option[Date]
            } else {
            None
        }
    }

}[/mw_shl_code]

Log日志分析源码.zip (29.12 KB, 下载次数: 3)

2016-11-8 20:49 上传

点击文件名下载附件
还有段另外代码

做的流程，推荐参考这个

Spark 日志解析示例包括运行、输出结果、源代码
http://www.aboutyun.com/forum.php?mod=viewthread&tid=20238