Time of exposure matters. It's called viral load. There are two ways you get enough viral load to be infected. Either a ton of virus enters your body in a short amount of time via large droplets, or you spend a lot of time in an enclosed space with a lot of aerosolized particles. So, the CDC says that it takes about 10 minutes within 6ft of an infected person to be considered "high risk" of exposure.
If you're passing a random person on the street, or in the aisle of a store, you're not likely to transmit the virus unless one of coughs or sneezes into the other's face, hence masks/social distancing in public.
However, if people spend enough time in an enclosed area there can be enough aerosolized droplets in the air from regular breathing, etc to transmit the virus, particularly if at least one person is unmasked. The guidance from virologists has said pretty consistently that you're far more likely to get the virus in places/situations where multiple people are together in an enclosed space for a lengthy amount of time. This means you're more likely to get the virus from family, friends, or coworkers than a short interaction with a random stranger.