To the problem of nonlinear system identification, the nonlinear system is expanded in a linear regression form under one basis function set.When the basis function set is known, the problem of how to identify the nonlinear system is transformed to estimate the unknown parameter vector exists in the linear regression form.One penalty term about the unknown parameter vector is added in the cost function and the regularized least square method is applied to estimate the unknown parameter vector.When this basis function set is unknown, the constraint about prediction error is considered in the cost function.As to this constrain optimization problem, the optimal solution is solved from the primal-dual point and its bias is also analyzed.To avoid the priori information about the basis function, the multiply operation coming from the linear regression matrix is replaced by one kernel function defined in the least square support vector machine theory.Then the dual vector in the constrain optimization problem is denoted by this kernel function directly and further the formal nonlinear function can be also approximated by one weighted sum form which is consisted of the kernel function and dual vector.Finally the simulation example results confirm the effectiveness of approximating the nonlinear system based on kernel function from support vector machine theory.