There is no one sound that makes up musical theatre, as it is influenced by every other style and genre of music. The musicals of the early 20th Century were strongly influenced by the popular singing styles of that time, and the same is true today. This means that if you want to specialize in musical theatre singing that you need to work on a number of different vocal qualities.
Recently I have been working with students on their legit musical theatre sound. I don't think anyone really likes the term "legit" as it comes from the word "legitimate". We know that there is no one legitimate way to sing, but it is still the term that is widely known for this style of music. It is a vocal quality grounded in classical tradition and arose in the early 20th century from the popularity of operetta. This sound was very prominent in the musicals written before the 1960s but is still required for roles in some contemporary musicals such as The Light in the Piazza. It is also a vocal quality people are often required to demonstrate when they are auditioning for musical theatre degrees at universities around the world.
If you want to listen the sound Julie Andrews can be heard here singing "I Could Have Danced All Night" from My Fair Lady.
What are the vocal characteristics of legit singing?
The sound of the voice is similar to that heard in classical singing, with the caveat that the words become more important than having a beautiful sound. This does change things as having too wide a vibrato or making the vowels too dark and round can make it hard for the listener to understand the words and follow the story. That being said the tone will still be clean and resonant and legato will be used to deliver the long lyrical lines. The voice will contain vibrato from the onset, the soft palette will be lifted*, and the vowels will be round and open. The larynx will be in a stable position that may be slightly lower than the speech type singing that occurs in more contemporary musicals.
Female voices will be CT-dominant and will be in mix or head voice. Male adult voices will sing predominantly in M1, that is TA-dominant or chest voice. There will be a smooth transition between the registers of the voice giving a one voice sound.
What are the challenges that people face to produce the sound?
I have found that students struggle with the onset that is required for this style. Instead of using a coordinated (simultaneous) onset for words starting with vowels they have a tendency to use either an aspirate or glottal onset, as you may do stylistically in contemporary singing. This can lead to a certain level of frustration when your song starts with a word like "If". Students enjoy singing through songs and don't always want to spend a lesson learning how to correctly sing one word, though of course this onset is required for all singing.
In order to start with a coordinated onset, think of a gentle ‘ah’ with no breath preceding the sound, and no hard attack on the ‘a’. If you allow pressure to build up under the folds, while they are closed, and then have the sound produced you will end up with a glottal and the sort of sound you get when you say "uh uh", which is not how you want to start a legit song. Another way, that I have found helps me, is to sing through the piece on a "ya" paying close attention to how it feels to sing that sound. (If you sing a "ga" for example you will most likely feel the glottal and a "ha" will release air first giving you an aspirate onset.)
Students also struggle to produce a legato sound. I usually recommend that they try to sing through the line using only the vowel sounds in the words and that they let their voices slide between the sounds. Once we have done that we can try singing it all on one vowel that works particularly well for their voice or this style, such as the "ee" which helps with the tongue and palette position. And after that we start to add back in the consonants and talk about the ones that cause them problems and how to approach them.
Changes to the Style
The sound of legit singing is changing and becoming more contemporary as tastes in audiences move further away from classical singing. You can hear this by comparing original cast recordings with some of the recent Broadway revivals of these shows. For example there is a distinct difference in articulation when you compare Julies Andrews singing "In My Own Little Corner" and the version that Laura Onses sang. I believe that to modern ears Julie Andrews sounds beautiful, but also very proper and distant from the audience, in a way that would still work for Mary Poppins, but feels slightly wrong for the character of Cinderella.
* Our understanding of voice science is changing all the time. We know that the mechanism of the soft palette would be better described as moving back and forward, but singing teachers have been talking about lifting the soft palette for a very long time and most people understand what is meant by this description and what impact it has on the sound of the voice.