Section 6. Unless all the towns interested in such infirmary agree to choose a different number, each of them shall choose three members of the joint board; and upon the death of a director, or his removal from the place for which he was chosen, the vacancy may be filled by such town. If a town neglects to choose directors, those chosen by the other towns shall have charge of the infirmary.