Possible Clues:

• Passing through extreme poverty with Heather
• Do you have to have fish so as to be so annoying?
• One had to have fish to be so irritating
• Doing embroidery work could be irksome
• To be a pest you've got to have fish
• Seems you have to have fish so as to be such an annoyance
• It's annoying to be wanting about a pound
• How irritating to be in want of fish!
• You have to have the fish to get 24 across
• How irritating to have to have the end of 27 across!