I don't really have the answer.
In most case, everything should be OK.
The problem is that there will be 2 different 5V in contact. And it's never exactly 5V.
The one from the USB and the one after the 7805 regulator (wall adaptor).
That must not be very good neither for the 7805 nor for the 5V source of the USB.
What is your USB source ? Computer ? Ipad ?