« September 2004 | Main | November 2004 »
October 29, 2004
Log file mistakes
From Computerworld
A nice article that exposes the top five mistakes people make when watching log files for security issues.
No. 1: Not looking at the logs
No. 2: Storing logs for too short a time
No. 3: Not normalizing logs
No. 4: Failing to prioritize log records
No. 5: Looking for only the bad stuff
I find at a high level this would be true for watching for other things as well.
Go to Advanced Search
Computerworld Home
You may retrieve this story by entering QuickLink# 49539
> Return to story
Five mistakes of log analysis
Opinion by Anton Chuvakin, netForensics Inc.
OCTOBER 21, 2004 (COMPUTERWORLD) - As the IT market grows, organizations are deploying more security solutions to guard against the ever-widening threat landscape. All those devices are known to generate copious amounts of audit records and alerts, and many organizations are setting up repeatable log collection and analysis processes.
However, when planning and implementing log collection and analysis infrastructure, the organizations often discover that they aren't realizing the full promise of such a system. This happens due to some common log-analysis mistakes.
This article covers the typical mistakes organizations make when analyzing audit logs and other security-related records produced by security infrastructure components.
No. 1: Not looking at the logs
Let's start with an obvious but critical one. While collecting and storing logs is important, it's only a means to an end -- knowing what 's going on in your environment and responding to it. Thus, once technology is in place and logs are collected, there needs to be a process of ongoing monitoring and review that hooks into actions and possible escalation.
It's worthwhile to note that some organizations take a half-step in the right direction: They review logs only after a major incident. This gives them the reactive benefit of log analysis but fails to realize the proactive one -- knowing when bad stuff is about to happen.
Looking at logs proactively helps organizations better realize the value of their security infrastructures. For example, many complain that their network intrusion-detection systems (NIDS) don't give them their money's worth. A big reason for that is that such systems often produce false alarms, which leads to decreased reliability of their output and an inability to act on it. Comprehensive correlation of NIDS logs with other records such as firewalls logs and server audit trails as well as vulnerability and network service information about the target allow companies to "make NIDS perform" and gain new detection capabilities.
Some organizations also have to look at log files and audit tracks due to regulatory pressure.
No. 2: Storing logs for too short a time
This makes the security team think they have all the logs needed for monitoring and investigation (while saving money on storage hardware) and then leading to the horrible realization after the incident that all logs are gone due to its retention policy. The incident is often discovered a long time after the crime or abuse has been committed.
If cost is critical, the solution is to split the retention into two parts: short-term online storage and long-term off-line storage. For example, archiving old logs on tape allows for cost-effective off-line storage, while still enabling future analysis.
No. 3: Not normalizing logs
What do we mean by "normalization"? It means we can convert the logs into a universal format, containing all the details of the original message but also allowing us to compare and correlate different log data sources such as Unix and Windows logs. Across different application and security solutions, log format confusion reigns: some prefer Simple Network Management Protocol, others favor classic Unix syslog. Proprietary methods are also common.
Lack of a standard logging format leads to companies needing different expertise to analyze the logs. Not all skilled Unix administrators who understand syslog format will be able to make sense out of an obscure Windows event log record, and vice versa.
The situation is even worse with security systems, because people commonly have experience with a limited number of systems and thus will be lost in the log pile spewed out by a different device. As a result, a common format that can encompass all the possible messages from security-related devices is essential for analysis, correlation and, ultimately, for decision-making.
No. 4: Failing to prioritize log records
Assuming that logs are collected, stored for a sufficiently long time and normalized, what else lurks in the muddy sea of log analysis? The logs are there, but where do we start? Should we go for a high-level summary, look at most recent events or something else? The fourth error is not prioritizing log records. Some system analysts may get overwhelmed and give up after trying to chew a king-size chunk of log data without getting any real sense of priority.
Thus, effective prioritization starts from defining a strategy. Answering questions such as "What do we care about most?" "Has this attack succeeded?" and "Has this ever happened before?" helps to formulate it. Consider these questions to help you get started on a prioritization strategy that will ease the burden of gigabytes of log data, collected every day.
No. 5: Looking for only the bad stuff
Even the most advanced and security-conscious organizations can sometimes get tripped up by this pitfall. It's sneaky and insidious and can severely reduce the value of a log-analysis project. It occurs when an organization is only looking at what it knows is bad.
Indeed, a vast majority of open-source tools and some commercial ones are set up to filter and look for bad log lines, attack signatures and critical events, among other things. For example, Swatch is a classic free log-analysis tool that's powerful, but only at one thing -- looking for defined bad things in log files.
However, to fully realize the value of log data, it needs to be taken to the next level -- to log mining. In this step, you can discover things of interest in log files without having any preconceived notion of what you need to find. Some examples include compromised or infected systems, novel attacks, insider abuse and intellectual property theft.
It sounds obvious: How can we be sure we know of all the possible malicious behavior in advance? One option is to list all the known good things and then look for the rest. It sounds like a solution, but such a task is not only onerous, but also thankless. It's usually even harder to list all the good things than it is to list all the bad things that might happen on a system or network. So many different events occur that weeding out attack traces just by listing all the possibilities is ineffective.
A more intelligent approach is needed. Some of the data mining (also called "knowledge discovery in databases") and visualization methods actually work on log data with great success. They allow organizations to look for real anomalies in log data, beyond "known bad" and "not known good."
Avoiding these mistakes will take your log-analysis program to the next level and enhance the value of your company's security and logging infrastructures.
Anton Chuvakin is a security strategist at netForensics Inc., a security information management company in Edison, N.J. His areas of expertise include intrusion detection, Unix security, forensics and honeypots. Chuvakin is the co-author of Security Warrior (O'Reilly, 2004) and a contributor to Know Your Enemy: Learning About Security Threats, Second Edition by the Honeynet Project (Addison-Wesley Professional, 2004) and Information Security Management Handbook (Auerbach Publishing, 2004). In his spare time, he maintains his security portal www.info-secure.org.
Copyright © 2004 Computerworld Inc. All rights reserved. Reproduction in whole or in part in any form or medium without express written permission of Computerworld Inc. is prohibited. Computerworld and Computerworld.com and the respective logos are trademarks of International Data Group Inc.
Changing ftp client from passive to active
I am not sure why I didn't know this......
local: strace.tar remote: strace.tar
227 Entering Passive Mode ()
ftp: connect: Connection timed out
ftp> passive
Passive mode off.
October 26, 2004
Miller's view of the Senate
From The National Review my friend John Miller is prediciting a two seat gain by the Republicans. I sure hope so. I would like to see the majorities in both houses increase. Especially since the election looks close.
Don't forget to buy John's great book Our Oldest Enemy: A History of America's Disastrous Relationship with France
A cuppa a day...
From Yahoo News:
A nice cup of tea could hold back Alzheimer's, scientists say
LONDON (AFP) - A steaming cup of tea, the relaxing drink of choice for millions in countries such as Britain and China, could help ward off the effects of Alzheimer's disease (news - web sites), scientists said.
Laboratory tests found that regular cups of green and black tea inhibit the activity of certain enzymes in the brain which bring on Alzheimer's, a form of generative dementia that affects an estimated 10 million people worldwide.
The research by the Medicinal Plant Research Centre at Newcastle University, northeast England, is published in academic journal Phytotherapy Research.
Scientists tested coffee as well as green and black tea, the latter of which -- the variety enjoyed by most Britons -- is derived from the same plant as the green variety but has a different taste and appearance as it is fermented.
The results found that while coffee had no significant effect, both green and black tea inhibited the activity of enzymes associated with the development of Alzheimer's.
According to the journal, tea inhibited the activity of the enzyme acetylcholinesterase (AChE), which breaks down the chemical messenger, or neurotransmitter, acetylcholine. Alzheimer's is characterised by a drop in acetylcholine.
Green tea and black tea also hinder the activity of the enzyme butyrylcholinesterase (BuChE), seen in protein deposits found on the brains of patients with Alzheimer's.
However green tea alone had a further effect, obstructing the activity of beta-secretase, which has a role in the production of protein deposits in the brain associated with Alzheimer's.
The effects of green tea also last for a week, scientists found, as against only a day for black tea.
"Although there is no cure for Alzheimer's, tea could potentially be another weapon in the armoury which is used to treat this disease and slow down its development," said head researcher Dr Ed Okello.
"It would be wonderful if our work could help improve the quality of life for millions of sufferers and their carers.
"Our findings are particularly exciting as tea is already a very popular drink, it is inexpensive, and there do not seem to be any adverse side effects when it is consumed.
"Still, we expect it will be several years until we are able to produce anything marketable."
October 25, 2004
Tool of the day
Tv allowance looks like a neat tool to limit tv viewing. I need this ;-)
October 24, 2004
Psalm of the day.
ESV Bible Online: Passage: Psalm 5:11-12
Psalm 5
11 But let all who take refuge in you rejoice;
let them ever sing for joy,
and spread your protection over them,
that those who love your name may exult in you.
12 For you bless the righteous, O Lord;
you cover him with favor as with a shield.
Bad ideas pt. 2
Is this really a great idea? Influencing a country's election indirectly is one thing. Advocating assassination and voter intimidation is another.
At what point
does it become a little counter productive? While it's no secret that
The Guardian is hoping for Kerry to win isn't there a level of rhetoric
that hurts their cause? As their recent forays into influenceing the US vote might have shown them. Is a columnist hoping for assasination over the line? "... John Wilkes Booth, Lee Harvey Oswald, John Hinckley Jr - where are you now that we need you?"
October 23, 2004
Security related
October 20, 2004
Cheap n easy way to kill em all.
Description
This recipe is useful when one wants to kill all the processes belonging to one user or having a particular regular expression in the process name. One can use different options of "ps" and "grep" to kill the undesired process(es).
Directions
The following is the general syntax of this recipe
ps -u <username> | grep <processname_pattern> | awk '{print $1}' | xargs kill -9
ps -u will find all the processes of the user username.
This output is then greped for processname_pattern which is then piped to awk.
awk '{print $1}' will print only the first column of the output (the process-id, in this case).
This is then xargd to the kill with sure kill -9 signal.
As a result,all the undesired processes will be killed.
Note : One should give the processname_pattern carefully as an incorrect
regular expression may lead to even desired processes being killed.
October 18, 2004
More from the Miller front.
John is interviewed in Frontpage Magazine about Our Oldest Enemy
Hat tip to Dhimmi Watch
Daytime hours for Atlanta Radio
From http://www.radio-info.com/
Here are the "Day Time" hours for Atlanta. This is when a station is at it's highest power.
These are all in EST so during Daylight Saving Time add an hour.
Jan 7:45 - 5:45
Feb 7:30 - 6:15
March 6:45 - 6:45
April 6:15 - 7:15
May 5:30 - 7:30
June 5:30 - 7:45
July 5:30 - 7:45
Aug 6:00 - 7:30
Sept 6:15 - 6:45
Oct 6:45 - 6:00
Nov 7:15 - 5:30
Dec 7:30 - 5:30
Run Amy Run
My friend John Miller and his wife Amy made it into today's Washington Times.
Church campaigner
"My wife went totally Catholic on me this morning — not an inappropriate thing, when you consider that it's Sunday and we were leaving Mass," John J. Miller writes in the Corner, a daily blog compiled by contributors at National Review Online (www.nationalreview.com).
"Now, you need to understand a couple things. First, Amy is not a very confrontational person (except when I forget to do the dishes). She certainly isn't confrontational with strangers. She's also a hard-core pro-lifer. So it really got her Irish up (as we say around the Miller household) to see a lady passing out Kerry-Edwards bumpers stickers on church property," Mr. Miller said.
"At Amy's urging, I rolled down the window and offered a few choice words. But then Amy insisted that we turn around and tell the priest. So we did, and he marched out and asked Kerry's minion not to harass his parishioners after Mass on church grounds. She refused. Our priest didn't have time to bicker — there was another Mass to give, and more unborn children to pray for — and so he left the scene. Then we told the Kerry lady that she really ought to leave because of Kerry's views on issues vital to the Catholic Church. She told us she respected our views and didn't budge.
"It seems to me that if you disobey a Catholic priest's request to quit pamphleteering after Mass on church property, then no, you don't respect Catholic views."
Better weekend for football
It was a nice weekend for football. The weather was really nice, and the two teams I follow here did well. Georgia did what they should have beating Vandy and the Falcons pulled it out in the 4th quarter. I want to know who was wearing Vick's jersey and playing for him in the first three quarters. I am glad he took it back in the 4th.
October 17, 2004
Hymn of the day
We sang Come, O thou Traveler unknown
today in church. I hadn't heard it before. It is a poem by Charles Wesley about Jacob wrestling with the angel. The music for the hymn was written by Erik Routley.
The text courtesy of Oremus.org
Come, O thou Traveler unknown,
whom still I hold, but cannot see;
my company before is gone,
and I am left alone with thee,
with thee all night I mean to stay,
and wrestle till the break of day.
I need not tell thee who I am,
my misery or sin declare;
thyself hast called me by my name,
look on thy hands, and read it there.
But who, I ask thee, who art thou?
Tell me thy name, and tell me now.
Yield to me now, for I am weak
but confident in self-despair;
speak to my heart, in blessings speak,
be conquered by my instant prayer.
Speak, or thou never hence shalt move,
and tell me, if thy name is Love.
'Tis Love, 'tis Love! Thou diedst for me!
I hear thy whisper in my heart:
the morning breaks, the shadows flee.
Pure universal Love thou art;
thy mercies never shall remove,
thy nature and thy name is Love.
Our Offertory Anthem was "Teach Me, O Lord" by William Byrd. It was absolutely wonderful, as anything by Byrd usually is.
From Christ Church Cathedral:
The 20th Sunday after Pentecost: Oct. 17
Genesis 32: 3-8, 22-30 ** Psalm 121
II Timothy 3: 14-4: 5 ** Luke 18: 1-8a
The Genesis reading is the story of Jacob’s struggle with an angel. All his life, Jacob has gotten things by devious means, and now he must wrestle honestly in the darkness before he can rely on God in his upcoming meeting with Esau his brother, whom he has terribly wronged.
ANTHEM: “Te Deum laudamus” ....John Ireland
HYMNS: 388 O worship the King
453 As Jacob with travel was weary one day
513 Like the murmur of the dove's song
522 Glorious things of thee are spoken
639 O come, thou traveler unknown
This last hymn is perhaps the crowning achievement of the late Rev’d Erik Routley, universally renowned as teacher, theologian, poet, and musician. The Charles Wesley text is deeply moving and evocative. Routley’s lifelong love of this poem (a recounting of Jacob’s wrestling with the angel, today’s Old Testament lesson) led him to compose his most unusual and interesting tune. “Not that there are no good tunes for this incomparable text, but that whoever sets it won’t get all the juice out of it.”
The golden thread
From the CSM:
Christians call it 'the Golden Rule,' and it is found in every major world religion:
Christianity: Therefore all things whatsoever ye would that men should do to you, do ye even so to them: for this is the law and the prophets.
- The King James Bible, Matt. 7:12
Judaism: That which you hold as detestable, do not do to your neighbor. That is the whole Law; the rest is but commentary.
- Talmud, Sabbat, 31a
Islam: None of you is a believer if he does not desire for his brother that which he desires for himself.
- Sunnah
Brahmanism (orthodox Hinduism): Such is the sum of duty: Do not do to others that which, to you, would do harm to yourself.
- Mahabharata 5:1517
Buddhism: Injure not others in the manner that would injure you.
- Udana-Varga 5:18
Confucianism: Here certainly is the golden maxim: Do not do to others that which we do not want them to do to us.
- Analects 15:23
Taoism: Regard your neighbor's gain as your gain, and your neighbor's loss as your own loss.
- T'ai Shang Kan Ying P'ien
Scripture of the day.
The Lord is not slow to fulfill his promise as some count slowness, but is patient toward you, not wishing that any should perish, but that all should reach repentance.
October 15, 2004
Grass fed beef
Three websites selling grass fed (and finished) beef:
Ketchum Farms Natural Beef
US Wellness Meats
Bent Tree Farms
Recovering a lost root passwd
Linux Gazette has a helpful document on How to Reset forgotten Root passwords
Remember that as they say in this article:
Physical Access is Root Access. Meaning, if you give someone physical access to a system, then you are giving them a very good chance of getting root access on your box. This is true for Windows, Linux, or any other OS out there.
Booting into single user mode from GRUB
Follow these steps to reset the password when using GRUB:
* Reboot the system, and when you are at the selection prompt (See Fig. 2 below), highlight the line for Linux and press 'e'. You may only have 2 seconds to do this, so be quick.
* This will take you to another screen where you should select the entry that begins with 'kernel' and press 'e' again.
* Append ' single' to the end of that line (without the quotes). Make sure that there is a space between what's there and 'single'. If your system requires you to enter your root password to log into single-user mode, then append init=/bin/bash after 'single'. Hit 'Enter' to save the changes.
* Press 'b' to boot into Single User Mode.
* Once the system finishes booting, you will be logged in as root. Use passwd and choose a new password for root.
* Type reboot to reboot the system, and you can login with the new password you just selected.
2.1.2 Reseting passwords by using a boot disk and editing the password file
This method is a little bit more complicated than the previous one and has a very high chance of success (assuming your filesystem is not encrypted and you didn't forget the password to decrypt it if it is). As before, get permission before you do this.
To start, you need a Linux boot disk or a rescue disk. (If you didn't create one when prompted during the installation then let this be a lesson for you.) You can use your installation CD as a rescue disk; most distros have an option to allow you to boot into rescue mode. With my Redhat Linux CD, I have to enter linux rescue to start the rescue mode. But this might be a bit different in each distro. You can also use a live linux CD like Knoppix or Gnoppix for system recovery. (Click here for a list of all the live Linux CD's). In this tutorial I will use Knoppix as my rescue CD but the process is almost the same for any rescue CD you might use.
[ You can also download one of the many single-floppy Linux distributions (e.g., Tom's RootBoot ), and use it to bring up the machine as described. This is, of course, much faster than downloading and burning a rescue CD, especially on a slow connection. -- Ben ]
Follow these steps to reset the password using Knoppix:
* Reboot the system and configure it to boot from the Knoppix CD (instructions available here)
* At the Knoppix Boot Prompt (See Fig. 3 below) enter: knoppix lang=us to start boot Knoppix using the english locale. If you understand German, feel free to just hit 'Enter' to boot into Knoppix.
* Once the system finishes booting, press
* Type mkdir mountplace to create a directory called 'mountplace'. This is where we will mount the filesystem.
* Type mount /dev/hdaX mountplace, where /dev/hdaX is your root partition. More information on Linux partitions is available here.
* Change to the "/etc" directory on your root partition by typing cd mountplace/etc.
* Use your favorite text editor and open the 'shadow' file for editing. I use 'vi', so I type vi shadow (If you have a really old system, you won't have a shadow file, in which case you need to edit the 'passwd' file.)
* Scroll down to the line containing the root user's information, which looks something like:
root:dsfDSDF!s:12581:0:99999:7:::
* Delete everything between the first and second colons, so that the line looks like:
root::12581:0:99999:7:::
* Save the file and exit your editor.
* Type cd to return to your home directory.
* Type umount mountplace to unmount the partition.
* Type reboot to reboot your system, and remove the Knoppix CD from the drive.
* Now you can log into your system as root with no password. Make sure you change the password immediately.
YAHR
Another hummus recipe from the daily bread
Drain some chickpeas, and chuck them in the processor with a couple peeled cloves of garlic, the juice of one lemon, a spoonful or two (depending on how solid v liquid you like your hummus) of non-fat plain yoghurt, a teaspoon of ground coriander, a whole bunch of chopped coriander leaf, and salt to taste. I often add some cayenne pepper or chilli powder for extra heat. Just blitz everything until it reaches the consistency you want, scraping half-processed bits from the sides of the bowl as necessary. Adjust seasoning and eat with toasted wholemeal pita bread or raw vegetables for dipping. And brush your teeth when you're done - this stuff is seriously garlicky.
October 14, 2004
Do you think he wishes he had backed more bedsore research?
With the passing of Christopher Reeve we have to ask ourselves if his courage was used in the wrong way. I laud his wish to walk again, but would his notoriety have been better used to improve the day to day lives of the paralyzed. Addressing things like bed sore treatment, wound care, access and transportation, better wheel chairs, more community services might have done more in the long run.
October 12, 2004
More Marillion pictures
Pictures from the Philly
and DC gigs
October 11, 2004
WTF were they thinking?
October 10, 2004
Email from my friend John Miller
Hello,
This Tuesday is the official publication date for my book, Our Oldest Enemy: A History of America's Disastrous Relationship with France.
http://www.amazon.com/exec/obidos/tg/detail/-/0385512198/qid=1088332584/sr=1-1/ref=sr_1_1/002-7485360-0654400?v=glance&s=books
The first big promotional event is Monday night, when I'm scheduled to be a guest on The O'Reilly Factor. The show is an hour long and starts at 8 pm EST. I'm not sure exactly when I'll be on. Check it out if you have a chance.
Also, I've set up a website for the book. New postings on French perfidy are added each day.
http://www.oldestenemy.com/
--JJM
The Fox appearance was later postponed.
Sans top 20 vulnerabilities
A nice gut check on the top 20 Internet securitiy vulnerabilities. Are you patched for all these?
Using ssh keys
SSH keys can make your life easier. If you have multiple systems you can use them in a similar manner to the old (and dangerous) rlogin command. Buyer beware though. If someone compromises one of the systems in a a ring of systems, then all of them will be compromised.
[hat tip to Lockergnome]
Listening to: Quartz - Marillion
October 08, 2004
Linuxquestions wiki
Sample Veritas commands
October 07, 2004
You ever dance with the devil in the pale moonlight?
Man he get scarier every day. Should the joker sue for copyright infringement. You figure a lawyer would know better.
Photo cribbed from Low Culture
How do I list my linux firewall rules.
Assuming you are running either iptable or ipchains run:
'iptables -L' or 'ipchains -L'
Does SuSE have anything like an /etc/rc.local?
/etc/init.d/boot.local plus my SuSE 9.1 distro also has boot.localfs and boot.localnet.
Nmap example
nmap -sS -p 22 -oG /tmp/test 17.0.0.0/8 &
This looks for hosts on the 17 network with port 22 open.
New Iraqi secret weapon
Davids Medienkritik has a link to a video showing the testing of Iraqi terrorists new secret weapon. Talk about a quagmire.
Ever imagined a photo based map?
Multimap is a great resource for maps, but they have gone one step beyond. Now you can have a satellite photo on screen and overlay it with a map.
via boing boing
Missing volboot file on hpux using VxVM
How to cope with a missing volboot file under hp-ux.
October 06, 2004
Gmail adds feed
Gmail has been busy adding new features. This includes my favorite, an atom feed. I wonder why no rss feed as well. I still am not sure why atom would be superior. I will have to research. I also ran across this cool link to create your own gmail icon.
Neat gadget of the day.
I want one. It is just too easy to throw a clock.
Is MS MCE worth the hype?
Two reviews of Micro$oft Media Center Edition.
Interview with Ian Mosley.
A nice and comprehensive interview with Ian Mosley from Marillion. I saw them in concert Sunday in Cincinnati, and they were great (as usual).
Guide to using live cd to repair system
Repairing your linux system was never easier. ;-)
New styles and a redesign at Movablestyle.
The new MovableStyle.com looks great and has a few nice looking mt3 styles.
Unix tools free from Microsoft?
Check out What's New in Windows Services for UNIX 3.5. Micro$oft has released the latest version of Windows Services for UNIX free (as in beer). It looks like a neat set of tools. I will have to test it vs. cygwin.