Section 39. Whoever abandons an infant under the age of ten within or without any building, or, being its parent, or being under a legal duty to care for it, and having made a contract for its board or maintenance, absconds or fails to perform such contract, and for four weeks after such absconding or breach of his contract, if of sufficient physical and mental ability, neglects to visit or remove such infant or notify the department of his inability to support such infant, shall be punished by imprisonment in a jail or house of correction for not more than two years; or, if the infant dies by reason of such abandonment, by imprisonment in a jail or house of correction for not more than two and one half years or in the state prison for not more than five years.