Onsen, and public baths are probably the main place you'd notice any tattoo rules. Recently some of the beaches near Tokyo have a "no tattoo visible" summer rule (and also no alcohol), but it doesn't seem to be enforced!
Also some gyms have a cover up your tattoo rule, though this might also be weakly enforced.
You don't need to cover up unless there's explicitly a rule stating you should do some, and people won't be offended or anything. Japan slowly shifting towards tattoo acceptance, and you'll see quite a few young people in Tokyo with (usually small) tattoos now.
Here's article on finding hot springs that are tattoo friendly:
Disclaimer: I have no tattoos!