The case, a major test of the separation of church and state, was an unexpected loss for religious rights advocates after a string of victories.