I don't think swapping out "standardized" battery packs will catch on, because of ALL the reasons mentioned
I think we will see more on the supercapacitors(Ultracapacitors) as they are perfected
These can take a charge, basically as fast as the electrical feed can provide it, so very fast charging
I think there would be a trade off in Range
500mile range battery only 6 hour charge
200mile range battery only 4 hour charge, +200miles capacitor 10min charge, 400mile total
So you lose a bit of range but trade off is time to recharge if in a hurry
And the capacitor would recharge batteries when vehicle was parked, so quick charge to "fill up" capacitor again
I would say 80% of people(probably more) drive less than 200miles a day, so moot point for most
And you can rent a liquid fuel vehicle for longer trips