java - How to take out all duplicated files in subfolders from a huge set of files/subfolders in shortest time span? -


situation: have build application in java runs on lan on pc use server. pc dedicated application , has configuration (core-i 7 8 gb ram). app aimed :

  • store data huge file manipulation project (around 12 million files 200 kb each) in database (mysql).
  • server pc acts database server also, , 20 pc's, interacts database day long.
  • project process consists of multiple stages. each stage there separate folders, software moves files on network on separate storage server of size 16 tb.
  • the network cable between server pc , storage server giga cable while other network cables normal ones.
  • in process everyday 60,000 new files generated, , 100,000 records inserted in database , @ end of day app sends mail report client.

problem: our client has asked provide mechanism identify files same name, , submit them in seperately. can not rely on database records, since on different stages, users delete or modify files. other hand need think time. process of duplicate check done everyday before sending report. prefered solution use java and/or mysql based solution. tried:

  • searching , indexing files storage server, changed mind when program running 3 hours , still going on.
  • also database accept duplicate names, can not put unique constraint on it, can not make column file count , put unique constraint on file names, since slows down data enteries (they use insert batch). not want create table keep unique file names purpose only, since redundency.
  • mutli tasking tried.

question: best solution in java/mysql check files same name on huge amount of files/folders, on busy network in minimum amount of time, considering in mind entered database records not accurate?

i had similar situation before, well, not of de-duplication, of categorization. not many ready-made tools available free , open-source , can take information database. but, after long hunt, did find 1 great useful tool, directorylistprint.

http://download.cnet.com/directory-list-print-pro/3000-2248_4-10911895.html

the last saw, had free version can dump data csv-like format in own window or csv file. there, take on simple database access or sql server or something, run query find duplicate files. if have repetitively, use automation tool such autoit or autohotkeys automate task.


Comments

Post a Comment

Popular posts from this blog

Android layout hidden on keyboard show -

i have strange problem layout. when tap on edittext , keyboard shown edittext stay on bottom , when tap on to hide keyboard , keyboard hides , edittext go should when want input text. when tap again, keyboard shown , edittext goes down behind keyboard... <relativelayout android:layout_width="match_parent" android:layout_height="match_parent" > <scrollview android:layout_width="match_parent" android:layout_height="fill_parent" android:above="@+id/rl_commands"> <textview android:layout_width="match_parent" android:layout_height="wrap_content"/> </scrollview> <relativelayout android:layout_width="match_parent android:layout_height="24dp" android:id="@+id/rl_commands" android:alignparentbottom="true">
Read more

google app engine - 403 Forbidden POST - Flask WTForms -

i have flask application running on google app engine. trying submit form built using wtform , keep getting following error. 403 forbidden: don't have permission access requested resource. either read-protected or not readable server. location.html (part of code) <form method=post action="/home/location"> .... <button type="submit" class="btn btn-primary" value="submit"> directions </button> </form> main.py @app.route('/home/location', methods=['post', 'get']) def location(): form = cfcdirections.direction(request.form) print(request.method) if request.method == 'post' , form.validate(): print("after if") directions = cfcdirections() street_no = directions.no street = directions.street suburb = directions.suburb postcode = directions.postcode
Read more

c - Why would PK11_GenerateRandom() return an error -8023? -

i looking through internet trying find source of pk11_generaterandom() function see why function fail. have program uses function when moved new flavor of linux, fails after forking ( fork() ) since not believe there problem nss, suspect doing incorrectly disregarded in older versions of linux new 1 there issue. the openssl package same on 'good' , 'bad' server: openssl 0.9.8e-fips-rhel5 01 jul nss rpm differs though. 'good' has nss-3.12.2.0-2.el5 and bas has version nss-3.15.3-4.el5_10 the 'good' server uses quite obsolete linux: linux 2.6.18-128.el5 #1 smp wed jan 21 08:45:05 est 2009 x86_64 x86_64 x86_64 gnu/linux enterprise linux enterprise linux server release 5.3 (carthage) red hat enterprise linux server release 5.3 (tikanga) the 'bad' server newer: linux bad 2.6.18-371.4.1.el5 #1 smp wed jan 29 11:05:49 pst 2014 x86_64 x86_64 x86_64 gnu/linux oracle linux server release 5.10 red hat enterprise linux server release
Read more