Log management solutions based on big data technology

Scheme background

At present, the insurance industry is gradually expanding with the development of social ideology and objective economy. By considering the continuous expansion of the scale of insurance business, and the continuous increase of business information system and hardware equipment for service providing, all insurance companies are facing greater and greater pressure on system operation and maintenance. As the main reference source data for system operation and maintenance, the log information has shown an explosive growth trend no matter from the type or the data volume. It is difficult for traditional log management platform to handle massive log data, and even more difficult to do relevant statistical analysis based on these log data, so the value of the log data cannot be mined and realized. Therefore, we have launched a log management solution based on big data technology.

Scheme content

The log management platform based on big data technology includes the log data acquisition, creation of log index data, storage, statistics, analysis and mining of massive log data, calculation and presentation of real-time log data, which supports almost all major log information generated during system operation and maintenance, including logs from server, security device, network device, middleware, database, and application system, comprehensively covering the log query, real-time monitoring and warning, statistical analysis of reports, and data aggregation mining during operation and maintenance. The platform adopts a variety of big data technology components, Flume and Kafka for data acquisition terminal, ElasticSearch component for log retrieval, Storm for real-time stream processing, while HDFS, Hive and Spark for analysis. The whole platform is with the characteristics of distribution, high reliability and high availability.

Scheme value

The insurance industry log management platform based on big data technology is superior to the traditional log management platform in terms of data volume, data type, acquisition ability, processing ability, and analysis ability, which greatly improves the system operation and maintenance ability, reduces the operation and maintenance pressure, and realizes the increment of log data. Break an island of log information: the different types of log information is collected in batch and real time, and then stored in a unified big data platform, completing the centralized management of all types of log data. Improve the ability of real-time warning and prediction: the real-time streaming data processing technology is used to calculate the result of key log items in real time through specific algorithm rules during the real-time generation of log data, realizing the real-time warning and prediction capability of the system. Discover potential value of log information: massive log data is associated, aggregated and analyzed to discover valuable information points in log data for guiding the operation and maintenance of business system, realizing the increment of log data.