Americans are likely to think of New Year’s Eve and New Year’s Day as a time to celebrate the fresh start that a new year represents, but there is also a troubling side to the holiday’s history. In the years before the Civil War, the first day of the new year was often a heartbreaking one for enslaved people in the United States.

In the African-American community, New Year’s Day used to be widely known as “Hiring Day” — or “Heartbreak Day,” as the African-American abolitionist journalist William Cooper Nell described it — because enslaved people spent New Year’s Eve waiting, wondering if their owners were going to rent them out to someone else, thus potentially splitting up their families. The renting out of slave labor was a relatively common practice in the antebellum South, and a profitable practice for white slave owners and hirers.

“‘Hiring Day’ was part of the larger economic cycle in which most debts were collected and settled on New Year’s Day,” says Alexis McCrossen, an expert on the history of New Year’s Eve and New Year’s Day and a professor of history at Southern Methodist University, who writes about Hiring Day in her forthcoming book Time’s Touchstone: The New Year in American Life.

Some enslaved people were put up for auction that day, or held under contracts that started in January. (These transactions also took place all year long and contracts could last for different amounts of time.) These deals were conducted privately among families, friends and business contacts, and slaves were handed over in town squares, on courthouse steps and sometimes simply on the side of the road, according to Divided Mastery: Slave Hiring in the American South by Jonathan D. Martin.

Accounts of the cruelty of Hiring Day come from records left by those who secured their freedom, who described spending the day before January 1 hoping and praying that their hirers would be humane and that their families could stay together.

“Of all days in the year, the slaves dread New Year’s Day the worst of any,” a slave named Lewis Clarke said in an 1842 account.

“On New Year’s Day, we went to the auctioneer’s block, to be hired to the highest bidder for one year,” Israel Campbell wrote in a memoir published in 1861 in Philadelphia, in which he describes being hired out three times.

“That’s where that sayin’ comes from that what you do on New Year’s Day you’ll be doin’ all the rest of the year,” a former slave known as Sister Harrison said in an interview in 1937.