python - Recommendation for how to best handle information for random access? -


imagine have filesystem tree:

root/aa/aadata root/aa/aafile root/aa/aatext root/ab/abinput root/ab/aboutput root/ac/acinput ...

totally around 10 million files. each file around 10kb in size. key-value storage, separated folders in order improve speed (fs die if put 5 million files in single folder).

now need to:

  1. archive tree single big file (it must relatively fast have compression ratio - thus, 7z slow)

  2. seek result big file - so, when need content of "root/ab/aboutput", should able read quickly.

i won't use redis because in future amount of files might increase , there no space them in ram. on other side, can use ssd-powered servers data access relatively fast (compared hdd).

also should not exotic file system, such squashfs or similar file systems. should work in ordinary ext3 or ext4 or ntfs.

i thought storing files simple zlib-compressed strings, remembering file offset each string, , creating map in ram. each time need file, read content offset form map, , - using offsets - actual file. maybe there easier or done?

assumptions (from information in contents). might use following strategy: use 2 files (one "index", second actual content. simplicity, make second file set of "blocks" (say each of 8196). process files, read them programmatic structure file name (key) along block number of second file content begins. write file content second file (compressing if storage space @ premium). save index information.

to retrieve, read index file programmattic storage , store binary tree. if search times problem, might hash keys , store values table , handle collision simple add next available slot. retrieve content, block number (and length) index find; read content second file (expand if compressed).


Comments

Post a Comment

Popular posts from this blog

Android layout hidden on keyboard show -

i have strange problem layout. when tap on edittext , keyboard shown edittext stay on bottom , when tap on to hide keyboard , keyboard hides , edittext go should when want input text. when tap again, keyboard shown , edittext goes down behind keyboard... <relativelayout android:layout_width="match_parent" android:layout_height="match_parent" > <scrollview android:layout_width="match_parent" android:layout_height="fill_parent" android:above="@+id/rl_commands"> <textview android:layout_width="match_parent" android:layout_height="wrap_content"/> </scrollview> <relativelayout android:layout_width="match_parent android:layout_height="24dp" android:id="@+id/rl_commands" android:alignparentbottom="true">
Read more

google app engine - 403 Forbidden POST - Flask WTForms -

i have flask application running on google app engine. trying submit form built using wtform , keep getting following error. 403 forbidden: don't have permission access requested resource. either read-protected or not readable server. location.html (part of code) <form method=post action="/home/location"> .... <button type="submit" class="btn btn-primary" value="submit"> directions </button> </form> main.py @app.route('/home/location', methods=['post', 'get']) def location(): form = cfcdirections.direction(request.form) print(request.method) if request.method == 'post' , form.validate(): print("after if") directions = cfcdirections() street_no = directions.no street = directions.street suburb = directions.suburb postcode = directions.postcode
Read more

c - Why would PK11_GenerateRandom() return an error -8023? -

i looking through internet trying find source of pk11_generaterandom() function see why function fail. have program uses function when moved new flavor of linux, fails after forking ( fork() ) since not believe there problem nss, suspect doing incorrectly disregarded in older versions of linux new 1 there issue. the openssl package same on 'good' , 'bad' server: openssl 0.9.8e-fips-rhel5 01 jul nss rpm differs though. 'good' has nss-3.12.2.0-2.el5 and bas has version nss-3.15.3-4.el5_10 the 'good' server uses quite obsolete linux: linux 2.6.18-128.el5 #1 smp wed jan 21 08:45:05 est 2009 x86_64 x86_64 x86_64 gnu/linux enterprise linux enterprise linux server release 5.3 (carthage) red hat enterprise linux server release 5.3 (tikanga) the 'bad' server newer: linux bad 2.6.18-371.4.1.el5 #1 smp wed jan 29 11:05:49 pst 2014 x86_64 x86_64 x86_64 gnu/linux oracle linux server release 5.10 red hat enterprise linux server release
Read more