Hollywood kept experimenting with other variations on the bluescreen process including the ultraviolet matte as used in The Old Man and The Sea. But the real challenger to blue screen was created in the late 50s and credited to one of the giants in world of compositing Petro Vlahos.
Developed by Vlahos in the mid 50s and used extensively by the Walt Disney Studios in the 60s and 70s: The Sodium vapor process used actors, who were lit normally, standing in front of a white screen which was lit by powerful sodium vapor lights - those are the orange lights you see on street corners. Sodium vapor emits light in a very specific wavelength - averaging 589.3 nanometers - and nothing else.
Using a specially coated prism in an old three strip Technicolor camera, the very specific wavelength of the sodium vapor light was split off and captured on special black and white film - automatically creating the black and white traveling matte. The remaining light would be captured by regular three strip Technicolor Film which was relatively unaffected by the yellow/orange sodium vapor lights.
This technique produced some of the best travelling mattes of the time and was used by Disney first on film The Parent Trap and then The Absent Minded Professor both in 1961. Mary Poppins in 1964 demonstrated the capability of the sodium vapor process winning an academy award for best special effects.
There was just one problem. Only One Sodium Vapor prism was ever made so there was only one camera that was capable of this process. Disney owned the camera and they didn’t let it rent for cheap and they were too cheap to order a more prisms made up.
In the late 50s, When MGM was ready to produce Ben Hur in the MGM Camera 65 format (a 65mm film process) they turned to Petro Vlahos, the inventor of the sodium vapor process for help on the compositing. They didn’t want the problems that Ten Commandments had with bluescreen but The sodium vapor process wouldn’t work as it prism it used was been made for 35mm film, not 65mm. So Vlahos was asked to see if he could do something about trying to improve the bluescreen process.
After six months of hard work, Vlahos had a discovery. And this is where it gets pretty complicated..
Most colors that aren’t purely green or purely blue have about equal amounts of blue and green in them. So when creating a matte from bluescreen, Vlahos used a Green Cancellation separation (or positive), ran it though with the original color negative exposing both pieces of film together under a blue to light to create a “blue difference matte”. This matte was clear where the blue and green were the same - Then the blue separation positive was combined with the original negative and exposed under red light to get a cover matte. This cover matte was applied back to the original color separations except that the blue separation was replaced with a composite of the green and the green difference mask - essentially a synthetic blue separation.
This complicated process required 12 film elements to get from the composite negative to the composite internegative but it was remarkable in the way it single handedly solved the edge and fine detail problems that plagued blue screen.
It was so successful in fact that the process remained in popular use for almost 40 years. Developments like microprocessor controlled quad optical printers, employed by Richard Edlund for The Empire Strikes Back made the process faster and more accurate but the next big change to come would be in the form of digital.