Structural Justice in Student Analytics, or, the Silence of the Bunnies

Presented at the 2017 Digital Sociology Mini-Conference
Eastern Sociological Society Annual Meeting
Philadelphia – February 24, 2017

Download the full paper. Download the PowerPoint.

The explosion of student learning and behavioral analytics raises deep questions about whether it can be done within a meaningful frame of information justice. These questions that came to the forefront of public discourse in 2016 when Mount St. Mary’s University President Simon Newman described using predictive student analytics to weed out students unlikely to be retained as a way to “drown the bunnies . . . put a Glock to their heads.” Using the Mount St. Mary’s University incident as a touchstone case and generalizing that to a broadly applicable model of predictive student analytics, this paper suggests that these concerns can best be understood within a framework of structural justice, which focuses on the ways in which the structures of predictive student analytics influence students’ capacities for self-development and self-determination.

Predictive student analytics are algorithmic systems that use data from student behavior and performance to generate individual predictions for student outcomes. Mount Saint Mary’s University attempted to use predictive student analytics to improve the university’s first-to-second year retention rates reported to IPEDS by manipulating membership in the reporting cohort. Emails obtained by The Mountain Echo, Mount St. Mary’s University’s student newspaper, show that, at the suggestion of then-president and private equity investor Simon Newman, Mount Saint Mary’s University instituted a locally developed survey intended, ostensibly, to develop better metrics for student analytics to be administered at the student orientation bu in fact intented by the president to support to dismissing 20 to 25 students before the IPEDS reporting date. The survey consists of approximately 14 sections and 110 individual responses. It was developed locally but consisted of both locally developed individual items and items taken from a hodgepodge of psychometric instruments, broadly addressing courses and programs, non-cognitive student characteristics, and student activities. In the course of the controversy among leaders at the university, Newman made his now infamous “drown the bunnies” remark.

While the specifics of Mount St. Mary’s University are certainly problematic, the importance of the case for information justice in higher education comes in the context of initiatives like the EAB Student Success Collaborative and Degree Compass. This combination of broad collection of academic and non-cognitive data, predictive analytics methods, and triage-driven intervention intended to directly influence key quantitative indicators of success is now the state of the art in student services for retention and completion. Many of its characteristics can be seen in applications of predictive analytics in other areas such as criminal justice, where modelling identifies defendants who have the highest likelihood of failing to appear in court or of offending again and sets higher bail amounts, longer sentences, or recommends against parole, in some cases effectively denying bail or imposing life sentences for relatively minor crimes. This represents a significant new challenge in higher education management, especially from the perspective of meeting universities’ responsibilities to students, and is a major area of application for information justice. In honor of former President Newman, I call this combination the “drown the bunnies” model.

At Mount St. Mary’s University, knowledge of what is ethical—clearly present in abundance among everyone involved other than Newman—was insufficient by itself to produce a just outcome. This raises a consideration that ethics, with its focus on the justification of individual action, is poorly suited to engage. The decisive factor at Mount St. Mary’s University was the balance of political power: the authority of the university president and the inherent gaps in the principal-agent relationship between him and his employees. This reflects fundamentally a changed conception of the terrain of student analytics: a movement from pedagogy to politics. It is thus necessary to examine not only the myriad ethical considerations presented by “drown the bunnies” models but also these political relationships from the perspective of a coherent conception of justice itself.

In Justice and the Politics of Difference, Iris Marion Young presents a structural theory of justice: a society is just to the extent that social structures and relationships facilitate both the capacity to develop and exercise one’s human capacities (that is, self-development) and supports one’s participation in determining their actions (self-determination). Likewise, injustice has two corresponding forms. The denial of self-development is oppression; that of self-determination, domination. These conditions are not matters of either distribution or of individual ethics; they are part of the social structure:

Oppression in this sense is structural, rather than the result of a few people’s choices or policies. Its causes are embedded in unquestioned norms, habits, and symbols, in the assumptions underlying institutional rules and the collective consequences of following those rules. . . . In this extended structural sense oppression refers to the vast and deep injustices some groups suffer as a consequence of often unconscious assumptions and reactions of well-meaning people in ordinary interactions, media and cultural stereotypes, and structural features of bureaucratic hierarchies and market mechanisms—in short, the normal processes of everyday life. We cannot eliminate this structural oppression by getting rid of the rulers or making some new laws, because oppressions are systematically reproduced in major economic, political, and cultural institutions. (Young, 1990, p. 41)

Young uses this framework to understand and articulate the claims of injustice posed by various emancipatory social movements, suggesting that it is a particularly valuable approach for information justice. The key to sound predictive student analytics, then, is attention to the structural conditions in them that determine the extent to which student analytics supports the students’ self-development and self-determination.

The structural determinants of the Mount St. Mary’s University case, directions characteristic of most instances of predictive student analytics, take many different forms. The organizational authority structure and heirarchies of the university are the most readily apparent structure that influences predictive student analytics. Student analytics is a management process, one that affirms the authority of the institution—a social structure in which the student participates—over the student. Student analytics make the institution’s actions more reliable and more likely to achieve their ends, enhancing its capacity to act on—thus to exercise authority over—its students. This is a significant shift in self-determination, and to the extent that it is driven by institutional rather than student interests, a significant limitation on students’ self-development.

Simon Newman’s beliefs about the best way to manage student retention were irrelevant to the fortunes of Mount St. Mary’s University students until he became President Simon Newman and could translate those beliefs into intentions and institutional actions. They remained relevant only as long as he had the backing of other organizational structures (the Board of Trustees), who in turn could provide that support only as long as they were supported by empowered stakeholders such as donors who could vote with their pocketbooks and students who could vote with their feet.

Newman’s demand that the university dismiss students unlikely to be retained reflected his (perceived) position of authority in the institution, a perception informed by a view of the university as a business organization. This left students with neither the opportunity to participate in the decision (either directly or through the faculty members who were voicing the students’ interests) nor the ability to make an informed decision about participation: a clear form of domination, in Young’s framework.

And yet, even with Newman’s assertion of absolute authority backed by the Board of Trustees, the president did not get his way. Organizational hierarchies are more complex than might appear. Other norms at work in the university, especially those of shared governance and responsibility to serve the students rather than the university, empowered the faculty and other senior administrators to oppose Newman through the organizational authorities, protections, and relationships created by shared governance principles. These structures ultimately safeguarded the students’ self-development as empowered faculty members to kill the plan by, as one faculty member put it, running out the clock on the IPEDS reporting deadline.

Bureaucratic hierarchy is by no means the only organizational structure that shapes the justice of “drown the bunnies” models. The institution, the state, and the political economy all structure the content and use of student analytics. Analytics processes may use hundreds of variables, but institutions and the state have chosen what variables will be available to the system by choosing what data to collect, how to store it, and what to make public. These are guided not just by technical factors of political ones. For example, The federal policy context, especially to the extent that it is influenced by non-democratic factors such as industry lobbying and policy biases toward traditional students from traditional families attending traditional institutions rather than the contemporary non-traditional student majority, incentivizes institutions to undermine student self-determination in the name of retention rates.

The political economy of predictive analytics both situates systems within intellectual property law that makes them “black boxes” opaque to examination and, as development is often a commercialization of one institution’s system, makes generalizability an assumption rather than demonstrating it: Political and economic power thus reinforce scientism. Under such conditions institutions—and the students on whom their policies act—are unable to interrogate these systems’ algorithms or results and understand the basis of its recommendations not because of technical limitations, but because of how a profit-making entity uses law to protect its economic interests, undermining the students’ informed participation in a process that could easily lead to their dismissal from the university.

Knowledge structures found at intersections of policy, science, and technology are equally important in protecting students from oppression and domination. The broader structural context of analytics algorithms is built on the assumption that predictive learning analytics is, in fact, predictive. This belief is upheld in many cases by scientism, the ideology that science is the only path to true knowledge and that scientific knowledge is inherently and unquestionably objective. Model choice depends on assumptions about reality and intent, but these are rarely interrogated because of hyperpositivist beliefs about the efficacy of predictive analytics.

Scientism then hides other knowledge structures in which justice interests are found. An analytics process is part of a nexus in which problems, data and models, and interventions mutually support and inform each other. The underlying data is not an objective representation of reality but rather the end result of a translation process that is as much technical as it is social. These regimes of knowledge are structures laden with questions of justice. These structures confer intellectual authority on developers while shutting down critical inquiry with flippant injunctions against arguing with facts and dismissive contrasts between sound data and unfounded instinct.

This may be the most oppressive aspect of the “drown the bunnies” model: We believe our methods accurately tell us which bunnies to drown, for we have science. Those who suggest otherwise—for example, faculty members at Mount St. Mary’s University who claimed that the data were inadequate to the intended use or students who can speak to considerations that are not quantifiable or even simply not collected because it didn’t occur to anyone to do so—are denied the legitimacy of their claims by structures of knowledge that exclude information external to the process as non-knowledge. Students’ self-development is hampered because data science says, with high confidence in precise quantitative scores and an acceptably low error (or, in Newman’s words, “collateral damage”) rate, that they are unsuccessful students.

This structural analysis suggests that the “drown the bunnies” model fails because it is structurally unjust; it oppresses and dominates students. Students’ self-determination is undermined by organizational forms that establish paternalistic—literally, in loco parentis—authority over them. Using this authority, students’ self-development is subordinated to the needs of institutions, governments, and vendors. The well noted ethical concerns are most often a consequence of the organizational, political, and knowledge structures of student analytics: Privacy, individuality, autonomy, and discrimination are likely to be addressed most effectively where analytics processes aim at self-development and support self-determination.