Thats usually when you'll hear a rod knock.
If the bearings are wore to the point where the crank and rod are just a little loose, then when you have torque on the crank its holding everything togther, but when you let off itlooses that torque and lets stuff slop.....see what im saying?
Drop the pan and change the bearings....before it chews up the crank.
later,
Dustin