Bringing Color to Presidents Past

Armchair historians — and actual ones — have always enjoyed ranking American presidents, recasting Mount Rushmore, debating who was greatest of them all. There are the perennial favorites —Washington, Lincoln, FDR — and the eternally scorned: James Buchanan, Millard Fillmore, Franklin Pierce. Then there are those who voters abused but history redeemed: Herbert Hoover was hung in effigy, his motorcades pelted with rotten fruit, as he left office in the midst of the depression in 1933; yet twenty years later, after his epic humanitarian missions leading post-war disaster relief, he ranked among the most admired men in in America. Harry Truman had a 22% approval rating his last year in office, yet is now exalted for his common sense and steady hand during the dangerous birth of the atomic age.

Among those least inclined to judge and rank the presidents are the presidents themselves. They know the deep and often damaging toll the job takes on all those who hold it, successful or not: the toll on health, on family, on any kind of normalcy. A few weeks after his reelection in 2004, I asked George W. Bush whether he thought more or less highly of his predecessors, now that he’d been in the job a while.

“Of my predecessors? Very interesting,” he replied, and then, without hesitation, “More highly of them all.”

Why? Because “I’ve got a much better appreciation of what they’ve been through.”

Bill Clinton says much the same. “There’s just a general sympathy,” Clinton told Michael Duffy and me, when we interviewed him for our book The Presidents Club, and as if to prove it, he launched into an astonishingly detailed exploration of Franklin Pierce’s presidency, the impact of his son’s tragic death, the challenge of dealing with his party’s inherently untenable coalition: “I’ve thought he had a really admirable life and if he’d had a chance to serve at a different time, he might’ve been different. And you know, I’ve always been fascinated by all this,” Clinton told us, and pointed to the long wall of presidential biographies and memoirs and collections of letters in his office. “I read a really interesting biography of Rutherford Hayes when I was president.”

The President’s Club: The Inside Story of the World’s Most Exclusive Fraternity

One president who was especially reluctant to judge his brethren was John F. Kennedy. Once, when asked by Arthur Schlesinger Sr. to participate in a poll of historians that would rank past presidents, Kennedy’s reaction was ferocious: he threw the poll across the room.

“No one,” Kennedy told historian David Herbert Donald early in 1962, “has a right to grade a President — not even poor James Buchanan — who has not sat in his chair, examined the mail and information that came across his desk, and learned why he made decisions.”

On Presidents Day—and as we re-imagine, through Sanna Dullaway’s eye-opening colorizations, some of the figures who have occupied that august, lonely office — it is worth remembering what a democracy asks of the men and women who lead it. That does not mean we suspend judgment: voting is the ultimate ranking of presidential performance. But on a day we celebrate the presidency, we are celebrating service and sacrifice as much as success.

Nancy Gibbs is TIME’s managing editor. The Presidents Club, Gibbs’ latest book (co-authored with Washington Bureau Chief Michael Duffy), is available through Amazon.

Sanna Dullaway is a photo editor based in Sweden. TIME previously featured Dullaway’s colorizations of Abraham Lincoln.