It began with a great outpouring of images. At 15, Harold Feinstein borrowed his neighbour’s Rolleiflex camera and started shooting scenes of everyday life on the streets and boardwalks of south Brooklyn. The year was 1946 and Coney Island, where Feinstein grew up, was still popular with New Yorkers, who flocked to its amusement parks and beaches in the summertime to let their hair down.

Feinstein found compelling dramas wherever he looked: the sergeant in full uniform flirting with an older woman on the boardwalk; the gypsy girl with a dirty face loitering by the carousel. In one shot, a man with a pencil moustache and a “bad luck” tattoo glowers menacingly down at the pint-sized photographer. In another, a cluster of sunbathing teenagers, including a radiantly smiling girl with a radio, bask in the camera’s attention.

His talent was quickly apparent. At 18, Feinstein was accepted into the Photo League, a co-operative of socially conscious New York photographers whose members and supporters included Weegee and Richard Avedon. A year later, he walked into the Museum of Modern Art and presented his work to photography director Edward Steichen, who purchased two prints for the museum’s collection.

Over the following decade, Feinstein established himself as an eagle-eyed observer of city life, matching exquisite black-and-white compositions with a rare command of printing – one New York Times review praised the “rich blacks, brilliant middle tones and subdued highlights” in his work. His lens sought out human stories behind the windows of beauty parlours, in smoke-filled diners and on subway trains. As the critic AD Coleman puts it, the resulting images exude “a childlike wonder at the amazing diversity of the city”.

By the end of the 1950s, however, Feinstein’s photography career was waning. In 1960, as other street chroniclers such as Garry Winogrand and Diane Arbus were gaining wider attention, Feinstein left New York for a teaching post in Philadelphia. He continued to take photographs over the ensuing decades, and found unexpected commercial success in 2000 with a book of high-definition scans of flowers, but for half a century his street work was largely forgotten.

In 2011, the British film-maker Andy Dunn noticed a Kickstarter campaign by gallery owner Jason Landry and collector Jim Fitts to fund a hardcover retrospective of Feinstein’s classic black-and-white photography, and duly pledged $50. A year later, when the finished book arrived in the post, he thought: “There’s definitely a film in this.” By the time he tracked Feinstein down and secured an interview, in 2014, the photographer was 83 and ailing. He suffered a heart attack shortly before Dunn was due to visit him at his Massachusetts home, but the interview went ahead, and Dunn was able to hear about the ups and downs of Feinstein’s life first-hand, before the photographer died in 2015.

Over the following few years, Dunn spoke to friends and family, dealers and critics, and some of the many people who had benefited from Feinstein’s teaching over the decades – his students included the photographers Mary Ellen Mark and Mariette Pathy Allen. This testimony formed the picture of a vivacious man who was inexhaustibly curious about the people around him and who often saw the world in an optimistic light, despite the darknesses in his own life.

Feinstein’s upbringing wasn’t entirely happy: his meat-dealer father, Louis, was physically abusive and Harold left home at 16 to get away from him. Later, he struggled with alcoholism. “There was certainly a dark side, a needy side, a lonely side,” one of Feinstein’s former students tells Dunn in the film. “I guess his photography was working against that.”

While many of his contemporaries sought out angst, gloom and grit, Feinstein preferred to see the beauty and hopefulness of the world. That, suggests Carrie Scott, who is curating the first UK exhibition of his work later this month at the Store X in London, might partly explain why his photography failed to reach a wider audience during his lifetime. “Everything about Harold’s work lifts you up,” she says. “Was that the problem? Did everyone want him to be a bit more edgy, pissed off and grumpy?”

It didn’t help, either, that Feinstein was unwilling to compromise his artistic principles for exposure and commercial gain. In the early 50s, Edward Steichen had invited him to participate in The Family of Man, a group celebration of humanistic photography at MoMA, which went on to become one of the most popular touring shows of all time, with a bestselling book to match. When Steichen asked to have the negatives so that he could control how the images were cropped and sized, Feinstein withdrew his submissions. It was, he later admitted, “probably… the most foolish decision of my career”.

But he didn’t seem to mind all that much. Retreating to Philadelphia and later Vermont, Feinstein never paid close attention to career imperatives, preferring to preach a gospel of creativity and proto-mindfulness to generations of aspiring photographers. “He was very much about being in the moment, appreciating what’s in front of you and not worrying about anything else,” says Dunn.

Commercial success did eventually catch up with Feinstein, via his late-career experiments with scanography, as he termed it, which led to the publication of seven coffee-table books beginning with One Hundred Flowers in 2000. (Feinstein’s close-up of a white rose went on to become a top-selling item at Ikea.) With the proceeds, he and his third wife, Judith Thompson, bought a house in Merrimac, Massachusetts.

Feinstein lived long enough to witness the resurgence of interest in his earlier work. Following the publication of Harold Feinstein: A Retrospective in 2012, curators started getting in touch about his archive, and exhibitions were staged in Los Angeles, Moscow, Istanbul and, posthumously, in Paris.

The resurgence continues with the London show and Dunn’s film, Last Stop Coney Island: The Life and Photography of Harold Feinstein, which premiered at Doc NYC festival last November. After two days of filming in Merrimac in July 2014, Dunn took Feinstein on a road trip into his past. They travelled to Coney Island, where the 83-year-old brought his camera to the boardwalk and started snapping. He may have been at the end of his life, and barely able to walk, but the childlike wonder Feinstein had exuded 70 years earlier was undiminished.

Found: A Harold Feinstein Exhibition runs 15-19 May at the Store X, London WC2 (and until 26 May by appointment only). Last Stop Coney Island has its UK premiere (plus a Q&A with Andy Dunn) at Curzon Bloomsbury, London, on 15 May