There is an increasing interest in the application of fluorescence lifetime imaging (FLIM) for medical diagnosis. Central to the clinical translation of FLIM technology is the development of compact and high-speed clinically compatible systems. We present a handheld probe design consisting of a small maneuverable box fitted with a rigid endoscope, capable of continuous lifetime imaging at multiple emission bands simultaneously. The system was characterized using standard fluorescent dyes. The performance was then further demonstrated by imaging a hamster cheek pouch in vivo, and oral mucosa tissue both ex vivo and in vivo, all using safe and permissible exposure levels. Such a design can greatly facilitate the evaluation of FLIM for oral cancer imaging in vivo.