LoRA is a very important algorithm in PEFT and also an algorithm that I have found to be very effective in practice when training LLM models. LoRA is not complex but is very cleverly designed: firstly, LoRA does not increase any inference time. Secondly, LoRA does not change the original model but only trains an additional new parameter, and this parameter is only used to adapt to the current task. However, this also means that LoRA needs multiple different ΔW when training for multiple tasks, and learning multiple tasks is relatively difficult for LoRA, unless they are treated as the same task.
LoRA is a very important algorithm in PEFT and also an algorithm that I have found to be very effective in practice when training LLM models. LoRA is not complex but is very cleverly designed: firstly, LoRA does not increase any inference time. Secondly, LoRA does not change the original model but only trains an additional new parameter, and this parameter is only used to adapt to the current task. However, this also means that LoRA needs multiple different ΔW when training for multiple tasks, and learning multiple tasks is relatively difficult for LoRA, unless they are treated as the same task.