When you downshift you usually have your foot off the gas so throttle plate is closed but RPM is up.
This creates a high vacuum in the intake, this can pull oil thru the valve stem seals(under the valve covers), this oil is pulled into the cylinders via the intake valves, and burned with the fuel, this usually produces a blue smoke and oil smell from the exhaust.
To limit this check and clean PCV valve and hose, also the Vent hose that runs from valve cover/oil filler to the air intake/filter, clean and no cracks, both of these lower the pressure in the valve cover area so oil is not easily pulled in to the intake via the valve seals.
This also decreases oil leaks from valve covers and oil pans, as it reduces the pressure that pushes out the oil.
If the oil smell is not coming from the exhaust pipe then wash off all areas where you think oil is leaking and the headers and exhaust pipe, then drive it and look for oil leaking near exhaust system.