In order to increase the intelligence of the family service robot, a human action recognition method was put forward. First, the scheme of Motion History Image was used to adaptively update the background model. Then, YCbCr images combined with gray images were used to segment the images to decrease the shadow. At last, space-time interest points based on high entropy changes were brought forward to recognize the human action. Experiments showed that the background model can obtain the family environment background satisfactorily. And the recognition rate of the action recognition scheme combined with the environmental information could reach 95%.