Levenberg-Marquardt method

Concept

this method is combination of Gauss Newton Method and Steepest Descent Method:
solve

$$(J_k^T{J}_k+{\mu }_{k}I)s_k=-J_k^T{r}_k$$

for $s_k$ where ${\mu }_k$ is from Steepest Descent Method.

Cost

When ${\mu }_k\to 0$ as iterating, this method behaved like Gauss Newton Method(quadratic convergence rate), otherwise it behaved like Steepest Descent Method(linear convergence rate).