Stay in Kanazawa or Takayama

Hi,

Can anyone give me some advice on whether it would be better to spend 2 or 3 nights in Kanazawa or Takayama?

We are going to Japan for 3 weeks, this includes:

1 night in Tokyo on arrival
3 nights in Yakushima
1 night Osaka
1 night Koyasan
5 nights Kyoto
2-3 nights in Takayama or Kanazawa
6 nights in Tokyo

I was thinking that if we stay in either Kanazawa or Takayama we could do a day trip to the other destination and also a day trip to Shirakawa-go.

I would prefer to stay in 1 location rather than in both just to save the headache of taking luggage.

Does this sound feasible and which would be a better base location?

Any help would be appreciated.
Thanks
Deniz