Swap the plug wire around with a Cyl 4. See if your misfire becomes a Cyl 4 misfire. If so, theres ur problem. Otherwise, I distinctly remember watching the tech @ Lexus do the compression test, and one of the cylinders were very low.
P.S.- Don’t do the 2.4 swap…its not worth it. Trust me.