一种Spark作业配置参数智能优化方法

In order to solve the problem of low efficiency for manual configuration parameter tuning of Spark jobs caused by the large number of parameters, large parameter search space and mutual influence among parameters, a smart configuration parameter tuning method was proposed. Firstly, a set of core configuration parameters of Spark that had great effort to jobs’ performance were selected, then the performance prediction model was built based on the selected configuration parameters and the corresponding jobs’ performance information using support vector regression algorithm. By changing the size of the data set, the predicted value of the model was compared with the real running time of the job, and the evaluation index proved the effectiveness and accuracy of the jobs’ performance prediction model. Secondly, several optimization algorithms of configuration parameters based on hill climbing, simulated annealing, recursive random search and particle swarm were designed and implemented. The solution quality of these algorithms was analyzed by experiments. The experiment results showed unique advantage of recursive random search against other algorithms on both convergence result and convergence variance, which indicate recursive random search had both good stability and adaptability to different types of jobs. Compared with experienced optimization configuration, the experiment results showed that the smart optimization configuration can bring 4%, 15% and 22% average performance improvement to typical Spark jobs, which demonstrate that smart optimization configuration can obtain more suitable parameters for given jobs efficiently and improve running performance of jobs.