Box Lunch Lifestyle is all about moving toward things that are "better"--not sacrificing what makes you happy.
Does this time with your friends feel like an investment in yourself--in the person you'll be in the future? Do you have a goal or aspiration your friends at work have, too, and you could lead everyone in taking it on together? Is there a non-work-related skill you'd like to learn from someone?
This doesn't have to be a solitary exercise. It could actually be a great way to make connections--and show yourself and other people that you're dedicated to more than just your job.