it shouldnt be slowing you down that much, I ran it for about a month with little to no slow downs(usually when trying to run multiple apps and itunes), I used it for email(outlook) and instant messaging(AIM), I did not mess with the WDE.
AMD 3500+ w/ 2 gigs ram.
your specs should handle it easily, main guess is still other apps running that you may not need, or make sure you have all the updates for outlook, could be an MS thing. Vista could also be a complication if you are running it