Section 23. Upon the expiration of the term of office of a sheriff, or upon his resignation or removal, he shall deliver to his successor all the prisoners in his custody, but he shall retain the keeping of the jails and houses of correction and of the prisoners therein under his care until his successor has qualified.