I noticed that depending on boot timings, the USB gadget weren't detected at all by hosts. After few hours of debugging and using the at91clk output, I found that the UDP clock wasn't enabled from time to time.
This patch fix the startup of UDC by adding a check into the at91_update_vbus().