You get more trade ships with the level of the other persons port.
a level 1 port, regardless of how much they have in the town, you can only get 300 goods max. A level 3 port, you can get 4 trade ships…
I somehow got 12 trade ships at one place, and Im not sure what level his port was. Poor guy… must’ve been pissed when I went back for seconds and thirds 
I attack any level player, solely based on their military size, the amount of gold I will get from the pillage, and i only go for town level 6+. The gold matters, because if I need to send 2 or 3 guys to fend of the wall (a military of 0 with a wall that gives defense requires more than one guy) I want to get my gold back that it costs to send the troops.