Many college courses have digital content that does not comply with accessibility best practices, from readings that have been scanned from a physical book to video clips that lack captions. To help address this, my team at Barnard College turned to Blackboard Ally, an automated accessibility checker that flags accessibility issues and provides guidance on how to fix them. During the Spring 2021 semester, three courses participated in a pilot program to evaluate the efficacy, ease-of-use and impact of using Blackboard Ally within our learning management system. In this presentation, I will discuss what we discovered including reactions from faculty and students, the opportunities and pitfalls of using an automated tool, and the extent to which Blackboard Ally helped us realize our goal of making course content more accessible.