Section 24. A harbor master may, at the expense of the master or owners thereof, cause the removal of any vessel which lies in his harbor and is not moved when directed by him, and upon the neglect or refusal of such master or owners on demand to pay such expense, he may recover the same from them in contract, to the use of the town where the harbor is situated.