Meaning of the quote according to a pamphlet by Raleigh Addington of the London Oratory, pdf, cited by wikipedia:
When summoned to hear confessions or to see someone who had called, he came down instantly with the words “We must leave Christ for Christ”. Philip was a mystic of the highest order, a man of ecstasies and visions, whose greatest happiness was to be alone with God. Yet at the call of charity he gave up the delight of prayer and, instead, sought God by helping his neighbour. His whole life is that of the contemplative in action.
Here's another anecdote from Addington:
Some young men were playing games in the warm Roman sunshine outside Philip Neri's room. When Baronius, the learned Church historian intent on his writing protested at the noise, Philip told them to go on. “So long as they don't sin”, he said, “they can chop wood on my back”.
