(Forbes) Sam Wyly is something of a celebrity at his Dallas retirement home. The former billionaire is often stopped while walking the long, air-conditioned hallways (his goal is 1.5 miles a day) or reading in the dining hall at Edgemere, which houses 300 of the city’s well-heeled seniors. “A lot of people will thank me for making their late husbands rich or somebody in their family rich. Or thank me because somebody had a good job working for one of my companies,” says Wyly, 85. “They’re nice folks.”
Wyly wants to be known for building the arts-and-crafts chain Michael’s and several other companies, which first catapulted him onto Forbes’ list of the richest Americans in 2000, with a net worth of $750 million. He’s better known, however, for being a convicted tax cheat. In 2010, the Securities and Exchange Commission and the Internal Revenue Service targeted him and his now-deceased brother, Charles, in one of the largest tax evasion cases in history, accusing them of utilizing a tangled web of offshore trusts to hide stock sales that generated hundreds of millions in ill-gotten profits. Wyly ultimately was convicted—and socked with a whopping $1.3 billion bill for back taxes, interest and penalties, with $200 million of that owed to the SEC and $1.1 billion to the IRS.
But according to his oldest son Evan Wyly and his lawyer Jim Lee, Wyly is now in the clear. He paid the SEC the full $200 million owed, and the IRS agreed in October to accept $300 million, which means he ultimately got away with paying just $500 million—a $800 million reduction. He paid all of his back taxes, but was unable to pony up the full amount of interest and penalties, says Evan Wyly. (A spokesperson for the IRS declined to comment, citing federal law that “prohibits the IRS from commenting or appearing to comment on specific taxpayers situations.”)
After settling with the IRS, he also got out of bankruptcy in October. Suddenly, he has zero debt and a fresh start—but also significantly reduced means for him and his family. After declaring bankruptcy in 2014, he was forced to sell off his properties, art and other assets—and anything he hasn't already parted with, including the contents of his children’s trust funds, are being liquidated by a trust to finish repaying that $300 million settlement, and other creditors.
“I’m an old guy. I don’t need much,” Wyly says during a phone interview from his apartment in the retirement home, where he pays a modest (for a former billionaire) $7,500 a month, according to court filings. Among the possessions the bankruptcy judge let him keep: his elite American Airlines frequent flier status, paintings he had commissioned of his family, and his 2017 Lexus IS 200t.
His son Evan is with him and helps translate questions when Wyly’s hearing aid doesn’t do the trick. His daughter Lisa also turns up later for a visit. These days, he spends a lot of time with his six children, nine grandchildren and nine great grandchildren, many of whom live in Dallas. (His daughter Christiana recently married Elon Musk’s brother Kimbal.)
He makes a convincing case that he’s a man at peace, even after his world—and self-made fortune—came tumbling down around him.
Success with the first computer boom
Sam Wyly was born in rural Louisiana during the Great Depression. His mom ran a dancing school, and his dad owned a cotton plantation. After failing to hedge the crop one year, the family was forced to live for a time in a clapboard cabin that lacked indoor plumbing and electricity. Wyly, who played football in high school with his brother Charles, graduated from Louisiana Tech University and then earned a scholarship to get an M.B.A. at the University of Michigan. After graduation he headed to Texas for compulsory military service and then became a sales rep at IBM (where he met and befriended Ross Perot) and Honeywell.
In 1963, he struck out on his own. He founded Universal Computing with $1,000 and an idea that big companies would pay for access to a shared computer, starting with one that sat on Southern Methodist University’s campus. He was right. The company’s revenues grew quickly, from $700,000 (1964) to $7 million (1966) to $60 million (1968). Investors piled into the stock and quickly pushed the market cap to $1 billion.
Emboldened by his success, he embarked on his next corporate adventure, which would pit him directly against AT&T. Wyly wanted to build a nationwide digital network that would give computers a way to transmit data outside of phone lines. But AT&T, which wanted to keep that data business for itself, stood in his way. Wyly’s company, Datran, spent some $100 million waging a technical and regulatory battle—and lost. But then Wyly challenged AT&T’s monopoly in court. The case resulted in a $50 million settlement to Datran and other parties in 1980 and helped pave the way for a breakup of Ma Bell in 1984.
Meanwhile, Wyly was busy starting or acquiring multiple other businesses, including Bonanza Steakhouse, which he bought as a small regional steakhouse in 1967 and expanded to 600 locations before selling it in 1989. He started Sterling Software in 1981, which he sold for $4 billion in March 2000, just before the dot-com crash. And in 1982 he scooped up Michael’s when it had just six locations and turned it into a nationwide arts-and-crafts retailer, which he sold to Bain and Blackstone for $6 billion in 2006.
Even before he broke into the billionaire ranks, Wyly became a prominent donor to the Republican Party and its causes. In the 1960s, Wyly directed Richard Nixon’s presidential election efforts in Texas, and later he donated to the campaigns of George H.W. Bush, George W. Bush, John McCain, Bob Dole and Jeb Hensarling.
He and Charles also—most notoriously—bankrolled the “Swift Boat Veterans for Truth” campaign ad in the 2004 presidential campaign, which questioned the war record of Senator John Kerry. The ads were largely discredited as an unproven smear campaign, but they came to represent a new, coarsened type of political campaigning known as “swiftboating,” which now commonly means turning a candidate’s virtue against them.
Wyly hasn’t retired completely from politics, or at least from addressing hot-button political issues. He has spoken out against Trump for his anti-immigrant rhetoric and is working on his second book about immigration, which tells the stories of successful immigrants like PepsiCo’s Indra Nooyi, Google’s Sergey Brin and Tesla’s Elon Musk. “It was inspired by all the public talk that seems dumb. There’s a lot of anti-immigration talk and we are a nation of immigrants. All of us came from somewhere,” says Wyly, who frequently talks about his own Scottish and Irish ancestry.
“A scheme of secrecy”
The Wyly brothers’ troubles appear to have begun with Sam Wyly’s decision to move money offshore following the savings and loan crisis of the 1980s, when a bank abruptly pulled Michael’s line of credit and caused Wyly to lose his faith in the U.S. banking system, according to testimony he gave on the second day of his bankruptcy proceedings in 2016.
The Wylys proceeded to set up dozens of accounts in the Cayman Islands and the Isle of Man between 1992 and 2005, according to a 78-page complaint filed by the SEC in 2010. They used this “scheme of secrecy” to hide the sale of 14 million shares of stock in publicly traded companies whose boards they sat on, which generated profits of more than $550 million, according to the SEC. The brothers did not disclose these transactions to the government, despite a law that requires anyone who owns more than 5% of a publicly traded company to do so.
They also disguised their ownership. The brothers owned as much as 37% of Michael’s, 34% of Sterling Software and 16% of Scottish RE, yet company filings at the time showed that they owned as little as a quarter of those amounts, according to court documents. This allowed them to sell large amounts of stock without attracting attention from investors or the press, which otherwise might have prompted a sell-off that would depress the stock price.
The proceeds from the stock sales financed a high-flying lifestyle, which included multiple vacation homes, art, jewelry and collectibles. The brothers bought ranches and condos in Aspen, a 100-acre horse farm outside Dallas and an $8 million oceanside property in Malibu. They gave expensive jewelry, including a $622,000 ruby and a $759,000 emerald necklace, to their wives. Sam, a history buff, spent $721,000 acquiring official documents from the presidency of Abraham Lincoln and $937,500 on a portrait of Benjamin Franklin. They also funneled $300 million into hedge funds and other investments and $120 million into bank accounts, according to court documents.
From the outset, the Wyly brothers roundly denied wrongdoing, describing the trusts as a type of 401(k) that were meant to defer taxes and facilitate estate planning. They pointed the finger at their lawyers and accountants, saying they simply followed the advice they were given, and if there was any illegal activity, they had no knowledge of it.
“I set up what was essentially a big 401(k),” says Wyly. “The basic tradeoff for what I did, like with all these private plans done by big insurance companies, is you get to defer transactions and capital gains taxes.”
The jury didn’t buy it. After a four-year trial in lower Manhattan, where Sam and Charles were cast as fat cats from Texas who brazenly flouted the law, they were convicted of fraud in 2014. The judge ordered Sam to pay $198.1 million. Charles, who had died in a car accident in Aspen in 2011, was ordered to fork over $101.2 million via his estate.
“I did nothing wrong. What I did really made sense.”
The IRS also scrambled to get a piece of the pie, declaring in 2015 that it would seek to recover a stunning $2.03 billion in unpaid income taxes, interest and penalties from Sam and more than $1.1 billion from Charles’ estate. Wyly told a local reporter at the time that the numbers were “so absurd as to undermine the credibility of the IRS.” They would end up paying a fraction of that. In 2016, a federal bankruptcy judge ordered Sam to cough up $1.1 billion to the IRS. Then, in 2019, Wyly reached a settlement with the IRS to pay just $300 million.
“I did nothing wrong”
Overwhelmed by fines and legal fees, the former billionaire declared bankruptcy in 2014 and began reining in his lavish lifestyle. He sold his mansion in Highland Park, Texas, for a reported $9.4 million, unloaded a 244-acre ranch in Colorado for $14 million (slashed from a $60 million asking price), a New York City apartment for $4 million and an independent bookstore for $4.6 million. His art collection was parceled off by Christie’s and the Dallas Auction Gallery, which fetched $209,000 for Norman Rockwell’s Barbershop Quartet and $293,000 for Rembrandt Peale’s George Washington. He fired the staffers at his family office and the writers who helped him put together his books.
“Whether there had been any litigation or not, when you get some years on you, you need to downsize.”
Today Wyly says he has made his peace with the way things worked out. When he’s not roaming the halls of the senior complex, he says, he sometimes goes back to Highland Park, where he owned a prime mansion, to take his walks. He refuses, though, to sound bitter or in any way suggest he misses his Texas-size wealth.
“Whether there had been any litigation or not, when you get some years on you, you need to downsize,” he says. “My apartment has about 1,100 square feet, and I was probably personally only was using 1,100 square feet of 9,600 square feet in the other house.”
“I still have good cooking and nice places to walk around. I used to run all around the world, but I don’t really care much about that,” says Wyly. “Bankruptcy was a way to clean things up and make sure it wasn’t something that had to be dealt with by my kids, grandkids or great grandkids.”
But don’t expect him to be repentant. “I did nothing wrong,” says Wyly, who dismisses the whole affair as “a waste” that has lined the pockets of lawyers and was carried out by a biased New York judge.
