These Lost Photos of Andy Warhol Show an Unseen Side

“Let’s put it this way, I had no love whatsoever for Andy Warhol, none. N-O-N-E. As far as I was concerned, he was a self-seeker, a clever artist for sure, basically an advertising man [who] basically advertises for himself with all of his Factory of self-promotional expertise,” photographer Larry Fink bluntly tells TIME.

In his new book, Fink on Warhol: New York Photographs of the 1960s, Fink takes images that he shot of Warhol and his crew in 1966 and juxtaposes those with other photos he was shooting at the time in the rest of New York City, from working class life to anti-war and Civil Rights demonstrations. “Andy is now known by all kinds of people as a renaissance man and I glommed onto that renaissance and tried to see if I could demystify it,” says Fink.

The photos of Warhol and his crew, shot around Manhattan’s Lower East Side, included Edie Sedgwick, Lou Reed, John Cale, Ingrid Superstar and Gerard Malanga and were part of a fashion shoot for the literary magazine East Side Review. The magazine went out of business before the images were even published and as the decades went on, they had been shelved in Fink’s vast archive, only to be rediscovered two years ago. Friend and agent Marcello Marvelli pushed Fink to finally publish the lost photographs, but Fink had one condition: the book shouldn’t solely be about Warhol. That’s when the interwoven contrasting images of a much different New York City came into play.

“In the 60s of course, both in the Civil Rights and the anti-war marches, the pain had already been inflicted,” says Fink. Describing himself as a “left wing guy,” he goes on to point out: “So all the passion of Malcolm X and the demonstrations, were all my own passions. The hopes of so many of us at that time, anti-Vietnam War, pro-Civil Rights, the idea that some kind of odd revolution would be impacted, even though we were delusional, we were still profoundly muscular in our delusion.”

New York City at that time was anything but glamorous, it wasn’t a rich city and was much more culturally diverse. As Fink puts it, “Today to be a hipster, you have to be a millionaire in order to buy a t-shirt.” At the time, he notes that he was paying $35/month in rent for his apartment. “The extent of your dreams are based on the cost of your rent. If your rent is cheap, you can dream more. If your rent is very expensive, you don’t dream at all.”

Today, Fink is still very active in capturing social change through his photographs. In January, he covered the Women’s March on Washington, after Donald Trump was elected president. Comparing the demonstrations of the 1960s to today, Fink says: “There was pain depth and soul, whereas when Trump got elected, even though the American culture still had its own contradictions, obviously, the women and the men who went to those marches across the country had not yet been inflicted by the pain that was soon to come. It was just more or less a march of innocence, if you will, with incredulity as its template.”

Fifty-one years after first encountering Warhol for that 1966 shoot, Fink looks back on what he wants to communicate through this book. “[I’d want someone to] take away the sense of what it was in the 60s, from this man’s perspective anyhow, in terms of the urgencies that were on the street and within the people and the lack of urgency that was all in the Andy Warholian crowd. Bourgeois nihilists, nothing more.”

Larry Fink is an accomplished photographer and educator, who has been making images for over 55 years.

Michelle Molloy, who edited this photo essay, is a senior photo editor at TIME.

Kenneth Bachor is TIME.com’s associate photo editor, overseeing culture and entertainment.