multithreading - Writing a file using multiple threads in java -


i trying write single huge file in java using multiple threads.

i have tried both filewriter & bufferedwriter classes in java.

the content being written entire table (postgres) being read using copymanager & written. each line in file single tuple table & writing 100s of lines @ time.

approach write:

the single to-be-written file opened multiple threads in append mode. each thread thereafter tries writing file file.

following issues face:

  1. once while, contents of file gets overwritten i.e: 1 line remains incomplete & next line starts there itself. assumption here buffers writer getting full. forces writer write data onto file. data written may not complete line & before can write remainder, next thread writes content onto file.

  2. while using filewriter, once while see single black line in file.

any suggestions, how avoid data integrity issue?

shared resource == contention

writing normal file definition serialized operation. gain no performance trying write multiple threads, i/o finite bounded resource @ orders of magnitude less bandwidth slowest or overloaded cpu.

concurrent access shared resource can complicated ( , slow )

if have multiple threads doing expensive calculations have options, if using multiple threads because think going speed up, going opposite. contention i/o slows down access resource, never speeds because of lock waits , other overhead.

you have have critical section protected , allows single writer @ time. source code logging writer supports concurrency , see there single thread writes file.

if application primarily:

  1. cpu bound: can use locking mechanism/data construct let 1 thread out of many write file @ time, useless concurrency standpoint naive solution; if these threads cpu bound little i/o might work.

  2. i/o bound: common case, must use messaging passing system queue of sort , have threads post queue/buffer , have single thread pull , write file. scalable , easiest implement solution.

journaling - async writes

if need create single super large file order of writes unimportant , program cpu bound can use journaling technique.

have each process write separate file , concat multiple files single large file @ end. old school low tech solution works , has decades.

obviously more storage i/o have better perform on end concat.


Comments

Post a Comment

Popular posts from this blog

php - SPIP: From Tag directly to an article -

i work spip , plugin called tagsphere works fine tags in spip. have tag groupe put 1 article tag. when click on tag in tagsphere send me before menu can see articles linked tag. want send me directly article without see tag menu before. here code model of thagsphere: <div id="tagsphere-#env{id_article}"> <ul> <boucle_mot(mots){id_groupe ?}> <li>[<a href="#url_mot">(#titre)</a>]</li> </boucle_mot> </ul> </div> anybody idea? if want skip tag page, have link directly article. <b_article> block prevents display tag if no article linked. <div id="tagsphere-#env{id_article}"> <ul> <boucle_mot(mots){id_groupe ?}> <b_article> <li> <a<boucle_article(articles){...
Read more

jquery - isAjaxRequest always return false -

i got error message 404,'model has not been saved' following code my action controller public function actionextend($id) { $model=$this->loadmodel($id); if(yii::app()->request->isajaxrequest) { $model->todate = date('y-m-d h:i:s a',(strtotime('next month',strtotime(date('y-01-d'))))); if($model->save()) { yii::app()->end(); } else { throw new chttpexception(404,'model has not been saved'); } } } the jquery call $("#buttonextend<?php echo $this->post_row;?>").click(function(e){ $.ajax({ type: "post", url: "<?php echo yii::app()->createurl('post/extend', array('id' => $data['id'])); ?>", }); }); why not ajax call ? thank help i found response, missed } , function ...
Read more

ruby on rails - In a controller spec, how to find a specific tag in the generated view? -

i know integration tests preferred need ran in controller test, i'm testing gem injecting html code in view, xhr can't run in feature spec (if can, please explain me how :) ) so rspec controller tests can assert selector present (with capybara) : response.body.should have_selector('#foobar') has_selector? call all method capybara find selector. what want last child of body , assert id in particular. afaik it's not possible have_selector. what : all('body:first-child').first.id.should == '#foobar' however, capybara dsl, defined (more or less): def all(*args) page.all(*args) end and page empty unless use visit it's integrations specs. how can use capybara all method inside rspec controller test ? i can't test right after googling seems trick def page capybara::node::simple.new(response.body) end source
Read more