How did they know the contacts were enemy ships?

The first indication would be a lack of information about any friendly ships in the area. In WW2, submarines were operating deep into enemy areas and could start with the assumption that any ship was likely to be hostile. If there were friendly ships in the area, did the potential targets match any of that information?

Then it was a matter of identification. The U. S. had books that described the known Japanese warships, merchant ships and tankers. Captains would use features such as the shape of the bow and stern, the types of masts and funnels to compare what was seen in the periscopes to what could be found in the book. Captains would be very familiar with the outlines of the Japanese warships, and that made identification faster and easier.

One could also be more confident that it was an enemy ship if it was shooting at the boat. Likely, but not guaranteed.

However, by the end of the war, we had sunk most of the Japanese merchant fleet. It then became easier to find another American submarine than it was a Japanese ship to attack. (The joke was that someone could cross the South China Sea just stepping from one American periscope to another.) Obviously, at that point more caution was needed.