This problem needs pious company. Allah the Almighty says:

﴾And keep yourself (O Muhammad PBUH) patiently with those who call on their Lord (i.e. your companions who remember their Lord with glorification, praising in prayers, etc., and other righteous deeds, etc.) morning and afternoon, seeking His Face, and let not your eyes overlook them, desiring the pomp and glitter of the life of the world﴿ 

[Al Kahf-28]

In another Ayah, Allah the Almighty says:

﴾ O you who believe! Be afraid of Allah, and be with those who are true (in words and deeds) .﴿

[At Taubah-119]

You need pious company, and my advice to you is to evaluate your friends like being in a team in the game of rope pulling. If your friends in any evening, picnic or meeting make you leave Salah (they pull you against Salah), you should leave them, and if they make you offer Salah (they pull you towards Salah), stay with them.