Big Data Has Potential to Both Hurt and Help Disadvantaged Communities

In the future, many aspects of daily urban life may be tracked and translated into data points. Local governments and private companies are already testing potential uses for this information. According to watchdog groups, however, without a holistic understanding of how data collection and algorithms affect different communities, these systems risk reinforcing existing structural barriers to economic and physical mobility.

The Chicago Police Department’s experiments with predictive policing have raised concerns about unfair profiling in Black communities. Boston has tested situational awareness software that uses mass surveillance and facial-recognition technology for large-scale events, including queries that capture skin tone on a numeric scale. Elsewhere, data could enable a form of 21st-century redlining, with banks and healthcare companies using informational leverage to deny services to people in low-income communities.

“A big part of what’s happening across society today is that major institutions are increasingly using computers to make decisions that shape people’s lives. These processes are often opaque,” says David Robinson of Robinson + Yu, a firm that provides technical expertise to social justice advocates working on big data issues. His organization recently released a report, Civil Rights, Big Data, and Our Algorithmic Future, outlining both the risks and opportunities of this data-driven future. “People need to feel empowered around these algorithmic processes. There’s a cultural tendency to defer to computer decisions as inherently fair or beyond scrutiny.”

While intentional discrimination is clearly illegal, identifying unintentional discrimination remains legally and analytically complex, says Solon Barocas of Princeton’s Center for Information Technology Policy. A recent report he co-authored examines how big data can produce disparate impacts on vulnerable communities. “We need to be extremely sensitive to the subtle ways that data-driven systems can produce inequitable outcomes,” Barocas says. “That sensitivity starts with understanding the data itself.”

The report cites Boston’s Street Bump app as an example. The app allows smartphone users to report potholes automatically, but differences in smartphone ownership across populations may unintentionally bias infrastructure improvements toward wealthier neighborhoods.

“Historically disadvantaged communities tend to be simultaneously over-surveilled and underrepresented,” Barocas explains. While individuals interacting with welfare systems generate extensive data, they are often less visible as consumers. Credit scoring illustrates this imbalance: people outside the formal economy struggle to generate sufficient data to qualify for loans, while newer alternative metrics introduce their own risks.

The questions analysts ask—and how results are interpreted—are critical. Barocas points to an anecdote involving Evolv, a San Francisco–based company that builds hiring models. Evolv found that employees who lived farther from call centers were more likely to quit. However, because distance could be correlated with race, the company declined to use this variable to avoid potential violations of equal employment laws.

“Data mining can be used differently,” Barocas notes. “Instead of screening people out, you could ask whether adjusting workplace policies or conditions might help recruit or retain a broader group of employees.” In this way, data can be used to counteract—not reinforce—historical bias.

Robinson adds, “When powerful institutions design decision-making systems, we need to ensure they align with human rights. We can’t simply assume that algorithmic decisions are fair.”

The future does not need to resemble The Minority Report. With technical expertise and oversight, civil rights organizations can play an advisory role as governments and companies deploy large-scale data systems. Barocas envisions a future in which big data becomes a tool for exposing—rather than obscuring—persistent inequality.

“These techniques may force a long-overdue conversation,” he says. “Not just about conscious prejudice, but about structural inequality embedded in our systems.”

To access the original article, click here.