The San Luis Obispo Tribune says filming is taking place this week at the indoor blue-and-gold tiled Roman Pool and the outdoor Neptune Pool.
The pop diva had yet to be seen as crewmembers carried in bags full of fake flowers and a giant plaster seashell Tuesday.
Shoots at the state historical site are extremely rare. Gaga is donating $250,000 to the Hearst Castle foundation in addition to underwriting a $25,000 water supply study and paying special fees.
She will also make a water conservation public service announcement and a short feature on the castle built by newspaper publisher William Randolph Hearst.