As stressors impacting coral reefs increase, understanding the role of herbivorous fish in controlling the proliferation of macroalgae on coral reefs is vital. However, previous studies investigating fish herbivory of macroalgae (particularly Sargassum spp.) have only investigated removal of the ‘leafy’ biomass and have not investigated whether fish remove Sargassum’s holdfast, despite Sargassum’s ability to rapidly regrow from holdfasts. To investigate the impact of herbivory on the different components of Sargassum (in particular the holdfast), we placed Sargassum attached to coral rubble (rocks) on the reef crest at Lizard Island, Great Barrier Reef, for 3.5 weeks, identifying the fish species responsible for Sargassum removal and quantifying rates of removal using video recordings. All blades were removed from deployed Sargassum within two days, predominantly by Naso unicornis. However, only one out of 54 holdfasts was removed during the experiment and none of the remaining holdfasts decreased in size relative to caged controls, despite recording over 800 bites on the rocks by 29 fish species. With no fish species obviously targeting Sargassum holdfasts, these findings indicate that holdfast removal is unlikely to be rapid on coral reefs, which may contribute to the persistence of Sargassum biomass, even in the presence of herbivores.