In 1746, apparently for the full sum of 500 pounds, the Charity Farm was purchased. A brief description of the land indicates a house, and probably other buildings, were already on the site, therefore by implication it was a working farm when bought. The property almost certainly stood on land once belonging to Lord Abergavenny and, indeed, was more than likely tenanted at the time of purchase. It has been considered that the tenant farmer then could well have been the David Dadswell we have mentioned before, whom we now know married the schoolmaster's daughter, Ann, and who together brought up their family at the farm which David managed for the Fermor Trust

By 1783, George, Lord Abergavenny, who was made Earl, remained the sole surviving trustee. David Dadswell was then schoolmaster and tenant of Charity Farm, and no doubt had been part of a committee welcoming John Fermor's son, the Rev John Shirley Fermor, to the Crowborough chapel in 1776.

Under the terms of the original Scheme, the Governors of the Sir Henry Fermor's Foundation were to take charge of the old Fermor Trust until, following negotiations with the Ecclesiastical Commissioners, a final severence and settlement of the endowment fund was completed. This aspect, the formal separation of the church or chapel from the school, came about on 17 September 1880 - the founding date of All Saints' Church.

Although 23 October 1877 was the date the school could operate as an independent body from the Trust, it was not the date the new school appeared on the Charity Farm site. Construction began in 1878, and completion came near the end of 1880, possibly coinciding with 17 September, the date the old schoolhouse became the property of the Church. Up until that time, the schoolhouse had been in use under the mastership of Mr Jeremiah Davis, who then became the first headmaster of the new Sir Henry Fermor School.