I did a three week trip in February this year, definitely agree that it's too ambitious to see the whole country. You can easily cover the main popular areas around Japan though. I would suggest Tokyo, Kansai (Osaka, Kobe, Kyoto) and Hiroshima/Miyajima as well.
I noticed that many have suggested using the JR Pass. I have used it myself during my February trip and recommend it as well if the price is not a problem. Otherwise, I suggest catching a bus (I have attached a link to the company I used down below). Personally, I found it very comfortable and just as convenient as the bullet trains, however time does have to be considered. I took the bus at night so I could sleep and not waste any of my time exploring on travel. If I had to compare it to something, I would say the seats reminded me of business class on a plane but a bit smaller.
Going back to areas of Japan to visit, I have written a list of places I enjoyed down below. Please keep in mind that a few are the popular tourist sites while others are more local spots that I preferred when I lived there for two months.
Tourist spots: Sensoji Temple, Tokyo Tower, Tokyo Sky Tree, Ramen Museum
Personal Favourites: Tamagawa River (especially during sunset and Sakura season), Yushima Shrine (there are quite a lot of stairs though so please keep that in mind), Ikegami Baien Park, Kiyosumi Garden, World Porters (the food on the top floor is delicious!)
Tourist sports: Dotonbori, Osaka/Umeda Station
Personal Favourites: Nakanoshima, Osaka Castle
Tourist sports: Any place with Kobe beef, Arima Onsen, Kobe Harbour
Personal Favourites: Nunobiki Falls/Herb Garden (you can choose to walk or take the cable car), Mt Rokko (tickets are available for purchase at Umeda Station)
Tourist spots: Gion Street (Each icecream store has different depths and levels of bitterness for their green tea icecream. I suggest the ones on the right side of the street if you want sweeter icecream and the left side of the street for a less sweet taste.), Arashiyama, Kinkakuji, Fushimi Inari-taisha
Personal Favourite: Kyoto Tower (I don't suggest the restaurant there though)
Tourist spots: Atomic Dome (easily accessed via bus), Peace Museum, Sadako's statue, Miyajima Island, Itsukushima
Personal Favourite: Mine align with the tourist spots mainly because I didn't spend a lot of days there, but do try the oyster curry buns on Miyajima Island. They taste absolutely amazing!
Link to buses: http://willerexpress.com/en/
Also, public transport is the most convenient in Japan. I suggest getting a SUICA to travel since it is accepted in the areas I have listed, but if you go to Hiroshima I suggest getting an ICOCA card when you get there as well (based off memory the SUICA doesn't work on their streetcars). I had a PASMO card from my previous trip but that didn't work in Hiroshima.
If you would also like some other recommendations, I loved Yamadera in the Yamagata Prefecture as well as Sendai (they have a loop bus that goes around to all the tourist spots for 600 yen for the day). Unfortunately I have yet to really travel anywhere else apart from these areas but I hope this helps.
I think three weeks is plenty to cover these places. Don't worry too much about the amount of walking, most times on public transport you will be able to sit and the areas I suggested have plenty of sitting areas as well (exc. Sensoji Temple). If you are worried about accommodation as well, I used booking.com and ended up spending around AUD$3000 for my friend and I combined (mainly due to staying in 4/5 star hotels and travelling over 8 different cities). If you look for 3 star hotels or apartment style hotels that amount can drop to under AUD$2500.
P.S. Personally I think you have time to go even further south or north if you wanted.
Good luck and have lots of fun!