The Milltown of Ilchester and the Thistle Manufacturing Company were located along the east bank of the Patapsco River between Ellicott City and Catonsville. The Baltimore and Ohio Railroad (B&O) was built through Ilchester in 1830. Ilchester did not, however, become a major stop for the B&O due to its steep grades and the tavern was unsuccessful.
Ellicott sold his tavern and 110 acres of land to a Redemptorist church order in 1866. The church used the Ellicott estate to start a college, initially called Mount Saint Clemens and several additions to the original building were made, including the construction of a chapel. The parish moved the church to a new facility in 1958 and eventually, enrollment at the college declined, leading to its closure in 1972. The State of Maryland purchased a portion of the campus in 1987 and added the land to adjacent Patapsco Valley State Park.