Some will definitely look off, buildings, for example, have different door sizes in gen 4 vs gen 5, so if you have a building from HGSS next to a building from BW you will be able to see the doorway difference.
With smaller things like rocks, land (grass, dirt, sand, etc) as long as the colors don't clash they will most likely merge fine enough, though the gen 5 patterns for these are much more detailed than the stylized Gen 4 tiles.