But he’s not complaining.
The ‘Pirates of the Caribbean actor is reportedly thankful his profession allows him to take long breaks and care for their laaities Leo, six, and Luna, three, while his Mrs is busy filming.
Comparing the situation to the late Robin Williams’ famous nanny character, he said: “Penelope is shooting [in London] so I am more into the Mrs Doubtfire role – kids, chaos, you know.
“It’s one of the things this job allows for, long periods of intense work and long periods of intense time off.
“We just did ‘Escobar’, had three months in Colombia together. The dream.”
The 48-year-old actor has always grown up with a strong sense of family because his own upbringing with a single mother in Madrid.
Javier told GQ magazine: “I grew up in a very small family.
“My parents separated when I was small, but we were so close – my mother, my brother and my sister. We were like a gang of wolves.
“We would attack anyone we felt was a threat.”
He said his mom, a single parent, was an actress who “had to make money where she could”.
“In the morning she would do television, then theatre in the evening and cabaret at night, coming home to sleep for two hours before going back to work.
“As a kid I was left to my own devices. My father was absent.
“I had no dominant male figure to look up to. This isn’t a complaint, just a fact. It meant I had to test myself.
“I had to figure out my own boundaries. It meant I made mistakes.”