Worried about train travel crowds during Golden Week

My husband and I will be vacationing in Osaka from April 30 to May 4. During that time, we had planned to go to Hiroshima on May 2 and spend the night in Miyajima returning to Osaka on May 3. We plan to use a JR Pass. I understand that I will not be able to make seat reservations until I arrive in Japan on April 30, and I am worried that I will not be able to get a seat on the train and that the unreserved cars will be too full and the lines too long. I am also worried that the trip from Hiroshima to Miyajima will be very crowded and we may not be able to get the train and the ferry there and back.

Do you recommend that we cancel our plans to go from Osaka to Hiroshima and Miyajima because it will be too difficult? Thank you, Roni