Section 19. The mayor of a city, except Boston, or the selectmen of a town where a harbor is situated, unless otherwise specially provided, may, and for all harbors that have been improved by the expenditure of money by the commonwealth shall, appoint a harbor master and assistant harbor masters and fix their compensation, to be paid by their respective cities or towns. Said appointment shall remain in force unless the harbor master is removed for neglect of duty, negligence or conduct unbecoming a harbor master. Assistant harbor masters shall be appointed for terms of three years. Any appointment or re-appointment of assistant harbor masters shall be on the recommendation of the harbor master.