The Life of Lou Ruppert

I received a couple of the following message today. The greeting differs, but the basic text (other than the randomized bits at the end) is the same:

Ahn nyeong,

+-------------------------------------------+
Warning! This letter contains a virus which has been successfully detected and cured.
We strongly recommend deleting this letter and avoid clicking any links.
+-------------------------------------------+

[RBN Networks Antivirus]

Days after the interview with mrs. Hammond, found born under
a mad moon with some wild humor in hari as deserving of
the first worship. Amongst a member or otherwise. The neighboring
church gratified with dharma's son yudhishthira, imparted
opere miror) would not eulogise himself quite couldn't have
done any rupert was not given to intelligence, while they
were being thus smitten to live deprived of thy company.
o king, if thou the king of rakshasas, that descendant of
ikshwaku's of the last word of the last line of the last
of great strength, and accordingly pursued with them talking
to each other in the dew. A galaxy also handed him a note
that he had written toof will be theirs? I do not seek the
accomplishment.

What makes it interesting is the hidden code at the end of the HTML version of the spam message:

 in the dew. A galaxy also handed him a note<br>   that he had written toof=
 will be theirs? I do not seek the<br>   accomplishment.</p>

<a href=3D"http://aaaahosting.com/.xkhafeedmaaabgehd.php"></a></body></html>
------------CFA6ED93730CD2--

online pharmacycialislevitrapropeciaviagra

I’m not sure what the goal of it is, but something is probably meant to have lurked behind the link. I just got a Page Not Found, so either it’s got some logic to tell whether a Windows machine is hitting it, or someone had already sanitized it. I changed the link before pasting it here, so clicking it yourself won’t help.

Other news: I played around with relay_recipient_maps today in postfix. They’re a nice way to move the bounces to the edge of my infrastructure, but I don’t know if they’re worth the work on a big scale. It seems even Google has decided for backscatter these days. I’m probably better off writing blocking rules for stuff like the RBN message.

Kicked in the Gutsy

I got together with a friend of mine tonight and tried to install Ubuntu 7.10 (Gutsy Gibbon) on the music studio machine in our church. It was a disaster at first. Here’s what happened:

• The initial patch update didn’t work because the capplets-data package seemed to hang indefinitely
• Installing the auth-ldap-client package and answering its questions resulted in an unconfigured machine. It ignored every answer to the wizard’s questions and went ahead and created an empty configuration file with default values.
• Rebooting the machine resulted in a bug where udevd would spawn endlessly, trying to connect to the LDAP server the empty configuration file was insisting was on 127.0.0.1
• Apparently, single-user mode also requires the LDAP server because we weren’t able to boot into maintenance mode either. Only my 15+ years of hardcore Linux experience enabled me to figure out how to disable the offending subsystem, remount the disk read-write, and update the settings to disable the LDAP auth. The average user would have had to reinstall from scratch at that point, because there was no other way to get into the system.
• The login screen kept changing resolution every time we’d switch to the other machine on the KVM. After a certain point, it would only choose resolutions that were impossible for our monitor to display. Again, a show stopper for pretty much anyone who is not a Unix expert.
• My attempt to hard-code the screen resolution in /etc/X11/xorg.conf was overridden by some unknown force to set it to a resolution not listed in that file. Why list restrictions in the file if the program is going to just do what it wants regardless?
• My attempt to set the resolution via the Résolution de l'écran menu ended up setting it to an impossible mode for the monitor to display, but instead of switching back to the original mode after thirty seconds, it remained in that mode.
• Text mode was unavailable after the initial login screen came up, because the Intel drivers forced that into an impossible configuration as well. How is one supposed to correct a problem with the main display window if the virtual ones are disabled as well?
• The fancy splash screen that is supposed to show the progress of the boot process also chooses a mode impossible to display on the monitor. What ever happened to defaulting to conservative VESA modes? So after the machine does its POST and displays the boot menu, the screen goes blank until the login screen comes up.
• The machine is pegged at a load average of nearly 2.0, because each user who logs in ends up spawning a trackerd process that takes hours to iterate our network filesystems. Does nobody at Ubuntu test these machines someplace other than on their brand new game machine in their mom’s basement?

Combine that with Monday’s announcement that Ubuntu is dropping the official release of its server product for the SPARC platform, and you can see how my opinion of Ubuntu has been plummeting recently.

SPARC hardware is reasonably priced used, typically of exceptional quality, and nearly completely resistant to a lot of the “script kiddie” attacks against Linux’s already tiny network footprint. It’s the perfect platform for small businesses to deploy because it affords them safety, quality, and performance at a reasonable price. And now, for their biggest release in a few years, Ubuntu has chosen to discontinue support for it. This after picking it up with great fanfare two years ago as the only major player to offer support for the SPARC platform. I don’t get it.

I went home angry and disillusioned, after hours of angry grumbling. It’s the first time I’ve gotten discouraged by anything in a long time. I sat around dreading the eventual problems this system would cause us, and the four hours of my life that I would never get back, the dinner I skipped, etc. Then it dawned on me that there was no need to get upset. All computer stuff causes that kind of dread if you don’t set service agreements. So, my service agreement with myself was that I would resign myself to having to repair the machine for others five times. After those five times, I could justifiably get upset. Before that, though, I’d already counted the cost and accepted it. It worked great! I have no animosity towards that system at all, tonight. And I even got to eat dinner. And meanwhile my brain is working in overdrive on how to repair some of the remaining problems.

RBN AntiVirus Saves the Day, Or Does It?

This morning I started getting a unique type of spam. It was randomized text, with a segment at the top stating that RBN Antivirus detected and removed a virus, and that I shouldn’t click any links or anything. My first thought was how odd, and highly unlikely, it was that the Russian Business Network, the biggest criminal organization on the Internet, would be censoring its own messages for viruses. Well, as it turns out, buried deep in the code, and only visible if you don’t display HTML emails, was a tiny link to malicious content on another web site. Very clever. If I get another one, I’ll try to post an example.

Sharing the wealth

The presentation on Mac Trojans went OK today. Most of the audience had read my previous stuff on the topic, so I only needed to review the material. The great thing was that a lot of people had really good questions. I’m used to speaking in front of a non-technical audience, so it was refreshing to hear well-thought-out questions. Nice work guys (and gals).

The new bits boil down to timing issues. The trojan, as currently implemented, is a bit sloppy. Where Windows trojan horses are nearly invisible and integrate seamlessly into the operating system, the Mac trojan is still rather amateur. So, as it’s updating the /etc/resolv.conf file once a minute, the normal process that produces that file will step in from time to time and produce a thin veneer of normalcy. Depending on when you look, you’ll get two different results. To break past that façade, you need to be familiar with the crontab command. Running sudo crontab -l will let you know if the administrator has evil scheduled to run once a minute or not.

Hoarding the wealth

I mailed the RAM off to 18004memory today and was abruptly reminded of why I hate the US Postal Service. I shouldn’t hate the postal service, because my father was a postman before I was born, but I’ve grown to hate them nonetheless. It started with them leaving expensive books out in the snow in my front yard, years ago. And now they’re all stingy with the packing materials. It used to be that if you needed an inch or two of tape to seal up your package, they’d provide it. Now they steadfastly refuse to help, because they are in the business of selling packing materials. Postmaster doesn’t like your packing job, or wants to inspect the package? Too bad. Either you go back home to reseal the envelope, or you pay them a few bucks for a fresh roll of tape. Come on, people! You have a monopoly on first class mail. Can you really not spare five cents worth of tape once a year for some of your customers?

It’s a prime example of how Americans have forgotten how to do business. Yes, granted, you’ll save five cents on having to help your customers do business with you today, but the bad taste the customer has gotten in his mouth from your ridiculousness means that he won’t likely use your services tomorrow. Look at all of the delivery companies that have sprung up in the last decade or so. Why are they here? People are less interested in preserving the customer relationship and making money in the long term, and more interested in the monkey-level instant figures. The USPS of ten years ago would have said “Hey, grab a piece of tape. We’ve got your back. See you tomorrow.” The USPS of today simply sneers and says “We don’t care who or what you are. We just care about your money. As long as the quarterly figures line up, we don’t care if you get hit by a bus on your way out.”

The mystery of 85.255.115.45

It all started with a series of mysterious connections to DNS servers in the Ukraine. (I know, technically you’re supposed to refer to it simply as Ukraine, and never The Ukraine, but call me a child of the Cold War.) All of the articles I’d found on the net about the connections were cries for help from people whose DNS had been hijacked by Windows spyware. Our security team met on the issue and decided to put the offending addresses into administrative lockdown, pending a rebuild of their virus-ravaged systems.

Dubious wisdom from the Apple Store

This morning we got a call from a Mac owner, who I will call Adam for no particular reason. It’s not his name. Adam had taken his Mac to the Apple store to get it reinstalled, like we’d asked, and they told him that Macs are invincible and cannot get viruses, so it must all obviously be a great misunderstanding. Adam showed up demanding to know why we had exiled his machine, because Macs can’t get viruses. What kind of amateurs would forget to consult the experts at The Apple Store? To be honest, I was rather surprised myself.

iPod Mafia-Edition

So, in one corner we had the reports our new IDS system was giving us, along with the dubious reputation of the Ukrainian data servers, and in the other we had mommy’s precious child with his new laptop and Apple-branded idealism. Who should we believe? Plagued by doubts, I began to Google every bit of information I had on the case. I checked the addresses, the types of lookups I was seeing, tried cross-referencing it with Macintosh terms, and finally settled on the name of the guy who owned the addresses and the name they were registered to. It returned something.

Andrew Sotov

Andrew Sotov is a name associated with all kinds of trouble on the Internet. He’s got comment-spam bots advertising pharmaceuticals, browser-hijacking malware, and apparently the first Mac-based trojan in the wild. And that’s just the first couple screens once you Google his name. The Mac thing was what interested me, though. Someone had finally managed to pull off a Mac trojan. Granted, you had to give it admin rights in order for it to install, but what mattered was that it existed, and that it matched the symptoms we’d been seeing.

Malware’s October Revolution

As of October of 2007, Macintoshes are no longer invincible. It’s still difficult to get infected, but with enough determination, anyone can do it. Here’s the SANS article describing it. What seems to happen to the Mac user is the following.

Somewhere in College Town, USA, after a night of frenzied art-making, right around when the patchoulli-scented candles have almost burned down to the point where he can barely make out the life-sized poster of Steve Jobs on his wall, the misunderstood Mac user decides to go on a porn-safari. He clicks site after site, until finally he arrives at one which shows promising thumbnail images that don’t have that “edited in Windows” look to them. The videos were probably done in iMovie! He clicks, but the site asks to install a special video codec in order for him to watch them. It’s no problem, because the fact that the site even tries to support Macintosh compels him to support them. He types in his password, and the show is over. He’s owned.

Lou’s easy home test

So now you probably want to know how to tell if you’ve got the infection. It’s easy.

1. Press command-shift-A and wait for the Applications window to pop up.
2. Open the Utilities folder
3. Right-click on the Terminal program (just kidding, I know you only have one button. Double-click it.)
4. Type the following in the terminal and press enter: more /etc/resolv.conf
5. You should see lines that look like this:

   nameserver 85.255.116.71
   nameserver 85.255.115.45
   

6. If the first two numbers in either address starts with 85.255, the chances are extremely high that your Macintosh is infected.

What now? Is there an after-school special I should watch, or maybe a support group?

The best course of action for any infected machine, Mac or PC, is to reinstall it with the disks it came with. You can’t trust anything you see on an infected machine, because the parts that you use to check it could also be infected. Think of it as an act of penance.

I’m still reviewing my notes from 2007. As of five minutes ago, I’m finished with May. Looking over a year’s notes is a great way to get a big picture perspective on how long things sometimes take. I realize now that we began our search for an IDS (Intrusion Detection System) setup back in April of 2007. That’s ten months of trying different products, tinkering with rules, juggling licenses, studying packets and flow data, packing and unpacking boxes, and so on. Finally, today, I got the new licenses to install and our system is live. Now we have all new questions, like what policies we’ll enforce to start with, what we can mix and match with what, etc. It’s a new season. We couldn’t have rushed it into existence any more than Winter can rush Spring.

Without notes, and the time line view they offer, each day would seem to be a self-contained capsule of pointlessness, as insulated from cause and effect as a sitcom episode is from the others in the series. Work without end is the torture of Sisyphus, but life doesn’t have to be like that. The little things we do every day, and the work that’s not finished when we go to bed still gets us somewhere. That’s a nice thing to remember.

Laou the Sfaxi

My friend Mr Ahmed informed me today that the African Football Cup was won this weekend by the Club Sportif Sfaxien from Sfax, Tunisia! Nice work guys! I know a few Tunisians from when I went there last winter. I spent a week traveling from Tunis down to Sfax, intending to practice my Arabic, but using more French instead. I’d intended to write all about it when I got back, but it’s yet another unwritten travelogue. I swear sometimes I need a staff of clones to get all of the projects done that I have vision for.

The new bedroom

The room is still plugging along. No pictures yet, but they’re coming. I’m beginning to think that the only person who will like the weird modern “melted mint chocolate chip ice cream” look is me, but I’ve not yet asked anyone else to sleep there, so it doesn’t matter. Painting is a great way to exercise self-discipline. It’s boring, time consuming, and an ideal way to expose your latent perfectionism.

The focus knob for my brain

I don’t normally advertise on my blogs, but think of it as more of an offer of an experience than an advertisement. One of the local entrepreneurs, Matt Godard, deals in artisan roasted coffee. If you’ve ever wanted to taste what the perfect cup of coffee tastes like, check out Café Kubal. When I’m dead and resurrected, and sitting around the legendary banquet table with Jesus and pals, I’m expecting the after dinner cup of coffee in heaven, intended to keep me awake for eternity, to taste exactly like Matt’s brew. He roasts the beans in his store in a rotating iron roaster that is nearly a hundred years old. I think it fills the place with weird roasty scents that subliminally rewire your brain chemistry to need it, so if any of you try his coffee without actually going there, let me know if you’re hooked too. Cost of tasting the best coffee ever: $8 for a 12 oz bag. They should print it in the vacation section.

The dubious bliss of protocol ignorance

I read about an intriguing spam-blocking technique this week, called “Greet Pause”. We use it on our sendmail servers to block a behaviour known as “slamming”. In slamming, the spammer will try to jam as many emails into the mail server as possible before it can decide that it doesn’t want them. With greetpause, the sendmail server will twiddle its thumbs before saying hello. It then disregards anything said before it greeted the new connection, which usually dumps the spammer’s email into the void. The mailer I prefer, postfix, uses a more effective collection of techniques to accomplish the same thing, but this technique was interesting to me because it reminded me of the sort of conversations Americans have with French speaking people sometimes.

Americans are very direct. When we go someplace to do business, we get right to the point. With French, and perhaps with Latin cultures in general, there is an order to things, sort of a protocol. So, a conversation might actually go like this:
American: Hi, I’d like to buy a phone.
Clerk: Um, Good day? How are you? *clears throat*
American: Right, sorry. Hello.
Clerk: Hello.
American: How are you doing?
Clerk: Very well, thank you. Can I help you?
American: Yes, I’d like to buy a phone.

Fun stuff.

You can’t eat a habanero pepper and not remember it the next day. The decision to eat the shiny orange fruit is followed by a series of distinct burn incidents over a 24 hour period. According to some dog owners that I know, if you feed a dog a habanero pepper, he’ll eat it once, whimper, and never touch one again afterwards. But people will often go back to eating hotter and hotter foods, because they crave the adrenaline rush.

I often wonder why some people tend to only see the negative in things. Much like the dog who avoids the pepper once he’s tasted it once, they shrink back from normal life. I think certain types of person are optimized to avoid bad situations in the same way that our digestive systems are optimized to avoid bad food. They get burned once, exist in a general sense of uneasiness, and then get burned again when the memory resurfaces. It’s a natural biological mechanism, but one which can be retrained in the same way as the lover of Mexican food can retrain their digestive system to crave adrenaline more than they hate chemical burns.

How do some people turn out one way, and others turn out completely different? Why do some love adrenaline and others hate to be burned? I think it depends on how seriously we take things. I was talking with the guys from Sourcefire on Friday, and asked them why nobody’s marketed a security console that uses Bayesian techniques to learn what is dangerous and what is not. It works to over 99% accuracy in spam filtration, and since the biggest problem in intrusion analysis is noise, it would make sense to apply the same technique there. As it turns out, the reason nobody’s tried it is that you have a lot more to lose with a false positive in security monitoring than you do with spam filtration. If you accidentally make a bad guess about a spam message, you miss out on an announcement about a meeting that someone would probably call you about anyway. If you accidentally make a bad guess about a security message, the Russian mafia could be auctioning off your business secrets on the black market before you know what hit you.

Negative people are afraid to make a bad guess that something is OK, because missing out on a warning means serious doom. For them, not being awake and in tune with their gut instincts could result in some unspecified disaster. Positive people realize somehow that everything works out, and would rather suffer the occasional disaster if it means fewer noisy interruptions from the lizard brain. For them, the burn of failure is worth the fun of living a relatively unencumbered life.

I’ve always been a pessimist with a taste for spicy food, but now I’m starting to see the big picture. The brain hardware God has outfitted us with is far too advanced for us to use it to make the same kind of decisions as dogs make.

One time when I was in college, I went to an all night cartoon festival. Around 6am, after I’d been up for 24 hours straight, I began to see people who weren’t there. Out of the corner of my eye, I’d see a guy walking down the aisle of the theater, but when I’d turn my head, he’d disappear. My doctor had once told me a story of a similar thing happening to him. He’d been riding in a convertible with his friends, with the wind rushing all around them, listening to music on the radio. When they stopped, they realized the radio was off. Each of them had heard a different song playing.

The human brain, when exhausted, tends to find patterns where none exist. Different people are better at resolving patterns than others, and if you are too good, you can become psychotic. I’ve counseled a fair number of people in my role as a minister, and the psychotic ones are fascinating to talk to. I think everyone is a bit psychotic, but the brilliant ones tend to come up with stuff that’s straight out of science fiction. One guy convinced himself he was winning a battle against the devil based on what numbers he was throwing on a dartboard. Another guy thought Tom Brokaw was sending him secret messages via the evening news. Another thought that people could hear his thoughts.

Part of my job at Syracuse University is to do Intrusion Analysis. I look at the network traffic going in and out of the university and try to spot hackers, worms, viruses, spam, and other things that mean trouble. Since it’s impossible for me to inspect every bit of information that goes in or out, I use a type of software called an Intrusion Detection System, or IDS. It looks at all of the information coming in and tries to identify patterns that seem dangerous or suspicious.

For things where the rules are well defined, computers do a good job of pattern matching. In spam blocking, for instance, you can combine a technique known as bayesian analysis with a scoreboard of how well the email obeys the rules posted on the internet for how email is to be delivered, and get 97% success rate. Anyone who has trained their copy of Thunderbird knows how effective this combination can be.

For an IDS, however, the rules are not well defined. They tend to have an extremely poor signal to noise ratio. A freshly installed IDS is much like someone who is having a full-on schizophrenic episode. Nearly 100% of what it sees is a figment of its imagination. We’ve been evaluating IDS products for about six months now, since I changed job descriptions this past summer. Each time we begin a new evaluation, I am confronted with screens and screens of nonsense. “Oh my God! Hackers are trying to break into your calendar applications!” “No, you’re imagining things. We don’t run that program. You’re just seeing someone looking at Google Calendar. It’s OK.” “Wait! Holy Cow! Someone is breaking into your copy of Exchange RIGHT NOW!!!” “No, dude, that’s a Linux box. It’s not vulnerable. Relax.”

The latest one we’re looking at is Sourcefire 3D. It’s really good at detecting patterns, but most of them are not valid. So, I’ve spent all week playing amateur psychologist to a rack full of psychotic robots, each quivering with fear about how when the clock turns 6:66 Osama bin Laden will succeed with his plan to poison our breakfast cereal and how we can defeat his plan only by adding certain very specific firewall rules. Today the engineers are supposed to show up and help me to perform brain surgery on them. It’ll be nice to have some peace and quiet.