I personally would need a bit more info but here are some things to consider.
Replacing a texture (depending on its size) and the machines abilities can take a beat. How larger are your PNGs and are they already loaded in memory when you do the swap?
The number of materials is not going to be the issue typically when it comes to scene load efficiency…it has a roll but insignificant in comparison to the number of Images/textures you have. If you want to be efficient, you use one texture for both objects.
Alternatively if you want to keep it the way it is…you may want to preload your textures that are not currently loaded. Basically create small triangle faces and assign a material you want to use as the swapping material. This will insure the mat is loaded in memory…then you just change materials…..make sure the original material also has a small hidden triangle somewhere. This will eliminate the loading lag you are getting.
Although not your exact problem the process is described here: https://www.xeons3dlab.com/post/assign-materials-puzzles