Aside from its inclusion in P'sukei D'zimra, the entirety of Psalm 150 is inserted into the Shofarot section of the Rosh Hashana Musaf Amida, since it explicitly mentions the shofar, along with a list of other instruments we use to praise God. Additionally, its message that we should praise God with all of our faculties is strikingly relevant on the day we crown God as King.
While David Montague Davis is best known for his role in editing "The Voice of Prayer and Praise" (1899) together with Rabbi Francis Lyon Cohen, he also wrote some wonderful compositions. His Halleluyah was originally intended for the choral wedding service, but its infectious energy makes for an equally fine addition to the Rosh Hashana service.