count - Python script that performs line matching over stale files generates inconsistent output -


i created python script parse mail (exim) logfiles , execute pattern matching in order top 100 list send domains on smtp servers. however, everytime execute script different count. these stale logfiles, , cannot find functional flaw in code.

example output:     1:     70353 gmail.com     68337 hotmail.com     53657 yahoo.com     2:     70020 gmail.com     67741 hotmail.com     54397 yahoo.com     3:     70191 gmail.com     67917 hotmail.com     54438 yahoo.com  code:  #!/usr/bin/env python  import os import datetime import re collections import defaultdict  class domaincounter(object): def __init__(self):     self.base_path = '/opt/mail_log'     self.tmp = []     self.date = datetime.date.today() - datetime.timedelta(days=14)     self.file_out = '/var/tmp/parsed_exim_files-'  + str(self.date.strftime('%y%m%d')) + '.decompressed'  def parse_log_files(self):     sub_dir = os.listdir(self.base_path)     directory in sub_dir:         if re.search('smtp\d+', directory):             fileinput = self.base_path + '/' + directory + '/maillog-' + str(self.date.strftime('%y%m%d')) + '.bz2'             if not os.path.isfile(self.file_out):                  os.popen('touch ' + self.file_out)             proccessfiles = os.popen('/bin/bunzip2 -cd ' + fileinput + ' > ' + self.file_out)             accessfilehandle =  open(self.file_out, 'r')             readfilehandle = accessfilehandle.readlines()             print "proccessing %s." % fileinput             line in readfilehandle:                 if '<=' in line , ' ' in line , '<>' not in line:                     distinctline = line.split(' ')                     recipientaddresses = distinctline[1].strip()                     recipientaddresslist = recipientaddresses.strip().split(' ')                     if len(recipientaddresslist) > 1:                         emailaddress in recipientaddresslist:                             # since syslog messages transmitted on udp messages dropped , needs filtered out.                             if '@' in emailaddress:                                 (login, domein) = emailaddress.split("@")                                 self.tmp.append(domein)                                 continue     else:         try:                                 (login, domein) = recipientaddresslist[0].split("@")                                 self.tmp.append(domein)         except exception e:              print e, '<<no valid email address found, skipping line>>'      accessfilehandle.close()     os.unlink(self.file_out) return self.tmp   if __name__ == '__main__': domaincounter = domaincounter() result = domaincounter.parse_log_files() domaincounts = defaultdict(int) top = 100 domain in result:     domaincounts[domain] += 1  sorteddict = dict(sorted(domaincounts.items(), key=lambda x: x[1], reverse=true)[:int(top)]) w in sorted(sorteddict, key=sorteddict.get, reverse=true):     print '%-3s %s' % (sorteddict[w], w) 

proccessfiles = os.popen('/bin/bunzip2 -cd ' + fileinput + ' > ' + self.file_out)

this line non-blocking. therefore start command, few following lines reading file. concurrency issue. try wait command complete before reading file.

also see: python popen command. wait until command finished since os.popen deprecated since python-2.6 (depending on version using).

sidenote - same happens line below. file may, or may not, exist after executing following line:

os.popen('touch ' + self.file_out)

Comments

Post a Comment

Popular posts from this blog

Android layout hidden on keyboard show -

i have strange problem layout. when tap on edittext , keyboard shown edittext stay on bottom , when tap on to hide keyboard , keyboard hides , edittext go should when want input text. when tap again, keyboard shown , edittext goes down behind keyboard... <relativelayout android:layout_width="match_parent" android:layout_height="match_parent" > <scrollview android:layout_width="match_parent" android:layout_height="fill_parent" android:above="@+id/rl_commands"> <textview android:layout_width="match_parent" android:layout_height="wrap_content"/> </scrollview> <relativelayout android:layout_width="match_parent android:layout_height="24dp" android:id="@+id/rl_commands" android:alignparentbottom="true">
Read more

google app engine - 403 Forbidden POST - Flask WTForms -

i have flask application running on google app engine. trying submit form built using wtform , keep getting following error. 403 forbidden: don't have permission access requested resource. either read-protected or not readable server. location.html (part of code) <form method=post action="/home/location"> .... <button type="submit" class="btn btn-primary" value="submit"> directions </button> </form> main.py @app.route('/home/location', methods=['post', 'get']) def location(): form = cfcdirections.direction(request.form) print(request.method) if request.method == 'post' , form.validate(): print("after if") directions = cfcdirections() street_no = directions.no street = directions.street suburb = directions.suburb postcode = directions.postcode
Read more

c - Why would PK11_GenerateRandom() return an error -8023? -

i looking through internet trying find source of pk11_generaterandom() function see why function fail. have program uses function when moved new flavor of linux, fails after forking ( fork() ) since not believe there problem nss, suspect doing incorrectly disregarded in older versions of linux new 1 there issue. the openssl package same on 'good' , 'bad' server: openssl 0.9.8e-fips-rhel5 01 jul nss rpm differs though. 'good' has nss-3.12.2.0-2.el5 and bas has version nss-3.15.3-4.el5_10 the 'good' server uses quite obsolete linux: linux 2.6.18-128.el5 #1 smp wed jan 21 08:45:05 est 2009 x86_64 x86_64 x86_64 gnu/linux enterprise linux enterprise linux server release 5.3 (carthage) red hat enterprise linux server release 5.3 (tikanga) the 'bad' server newer: linux bad 2.6.18-371.4.1.el5 #1 smp wed jan 29 11:05:49 pst 2014 x86_64 x86_64 x86_64 gnu/linux oracle linux server release 5.10 red hat enterprise linux server release
Read more