The fort overlooking the harbour was built in the 16C to protect the town against pirates.