Her other early film roles included George Stevens' Giant (1956), playing the love interest of James Dean, and in the romantic comedy But Not for Me (1959). In 1961, Baker appeared in the controversial independent film Something Wild, directed by her then-husband Jack Garfein, playing a traumatized rape victim. She went on to star in several critically acclaimed Westerns in the 1950s and 1960s, such as The Big Country (1958), How the West Was Won (1962), and Cheyenne Autumn (1964).
In the mid-1960s, as a contract player for Paramount Pictures, Baker became a sex symbol after appearing as a hedonistic widow in The Carpetbaggers (1964). The film's producer, Joseph E. Levine, cast her in the potboiler Sylvia before giving her the role of Jean Harlow in the biopic Harlow (1965). Despite significant prepublicity, Harlow was a critical failure, and Baker relocated to Italy in 1966 amid a legal dispute over her contract with Paramount and Levine's overseeing of her career. In Europe, she spent the next 10 years starring in hard-edged giallo and horror films, including Romolo Guerrieri's The Sweet Body of Deborah (1968), Umberto Lenzi's Orgasmo (1969) and Knife of Ice (1972), and Corrado Farina's Baba Yaga (1973), before re-emerging for American audiences as a character actress in the Andy Warhol-produced dark comedy Bad (1977).
Baker appeared in supporting roles in several acclaimed dramas in the 1980s, including the true-crime drama Star 80 (1983) as the mother of murder victim Dorothy Stratten, and the racial drama Native Son (1986), based on the novel by Richard Wright. In 1987, she had a supporting part in Ironweed (1987). Through the 1990s, Baker had guest roles on several television series such as Murder, She Wrote; L.A. Law, and Roswell. She also had supporting parts in several big-budget films, such as Kindergarten Cop (1990), and the David Fincher-directed thriller The Game (1997). She formally retired from acting in 2002. In addition to acting, Baker is also the author of two autobiographies and a novel.