heroku - Node.js worker memory spike Error R14 when using request & pkgcloud -


this 1 has me totally baffled because occurs seemingly @ random. have node.js app streams tweets (using ntwitter), adding hundreds of rows per second mongodb. part works fine , memory use steady. i've started getting url of twitter avatars , saving them our rackspace cdn (using "request" , "pkgcloud" modules). cycle thru max of 10 @ time (but way less) every 20 seconds , run code:

request(idocs[ikey].tweet.user.profile_image_url.replace('_normal.', '_bigger.')).on('response', function(response) {     console.log(response.request.path);     avatars.push(path.basename(path.dirname(response.request.path)) + '/' + path.basename(response.request.path));     var headers = {};     if (response.headers && response.headers['content-type']) {         headers['content-type'] = response.headers['content-type'];     }     if (response.headers && response.headers['content-length']) {         headers['content-length'] = response.headers['content-length'];     }     response.headers = headers; }).pipe(client.upload({     container: process.env.rackspace_container,     remote: path.basename(path.dirname(idocs[ikey].tweet.user.profile_image_url)) + '/'         + path.basename(idocs[ikey].tweet.user.profile_image_url.replace('_normal.', '_bigger.')) }));

everything works okay while, following madness in nodetime:

memory spike!!!!

and here's output of "heroku logs -p worker -t" when things go haywire (notice how it's going along fine @ around 600 mb leaps on 1gb after 1 particular image--and btw images standard twitter dimensions of 73x73, can't file size issue):

2014-03-10t00:27:28.425915+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#load_avg_1m=0.50 sample#load_avg_5m=0.24 sample#load_avg_15m=0.12 2014-03-10t00:27:28.426216+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#memory_total=630.87mb sample#memory_rss=599.24mb sample#memory_cache=0.00mb sample#memory_swap=31.63mb sample#memory_pgpgin=14872576pages sample#memory_pgpgout=33324565pages 2014-03-10t00:27:48.353902+00:00 app[worker.1]: /profile_images/434575414246539265/noy_be24_bigger.jpeg 2014-03-10t00:27:48.570496+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#load_avg_1m=0.36 sample#load_avg_5m=0.22 sample#load_avg_15m=0.12 2014-03-10t00:27:48.570793+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#memory_total=638.78mb sample#memory_rss=607.07mb sample#memory_cache=0.00mb sample#memory_swap=31.71mb sample#memory_pgpgin=14872576pages sample#memory_pgpgout=33403153pages 2014-03-10t00:28:08.451643+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#load_avg_1m=0.54 sample#load_avg_5m=0.27 sample#load_avg_15m=0.14 2014-03-10t00:28:08.452211+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#memory_total=601.35mb sample#memory_rss=569.55mb sample#memory_cache=0.00mb sample#memory_swap=31.79mb sample#memory_pgpgin=14872576pages sample#memory_pgpgout=33482782pages 2014-03-10t00:28:28.616784+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#load_avg_1m=0.67 sample#load_avg_5m=0.32 sample#load_avg_15m=0.16 2014-03-10t00:28:28.617032+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#memory_total=605.14mb sample#memory_rss=573.27mb sample#memory_cache=0.00mb sample#memory_swap=31.87mb sample#memory_pgpgin=14872576pages sample#memory_pgpgout=33560778pages 2014-03-10t00:28:48.332406+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#load_avg_1m=0.76 sample#load_avg_5m=0.36 sample#load_avg_15m=0.18 2014-03-10t00:28:48.332628+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#memory_total=594.71mb sample#memory_rss=562.75mb sample#memory_cache=0.00mb sample#memory_swap=31.95mb sample#memory_pgpgin=14872576pages sample#memory_pgpgout=33647657pages 2014-03-10t00:29:08.501121+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#load_avg_1m=0.83 sample#load_avg_5m=0.40 sample#load_avg_15m=0.20 2014-03-10t00:29:08.501351+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#memory_total=599.86mb sample#memory_rss=567.83mb sample#memory_cache=0.00mb sample#memory_swap=32.03mb sample#memory_pgpgin=14872576pages sample#memory_pgpgout=33730317pages 2014-03-10t00:29:28.606331+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#load_avg_1m=0.88 sample#load_avg_5m=0.44 sample#load_avg_15m=0.21 2014-03-10t00:29:28.606565+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#memory_total=630.37mb sample#memory_rss=598.26mb sample#memory_cache=0.00mb sample#memory_swap=32.11mb sample#memory_pgpgin=14872576pages sample#memory_pgpgout=33823519pages 2014-03-10t00:29:29.054831+00:00 app[worker.1]: /profile_images/1857857435/251396_10150278547536115_50300066114_9027856_3493590_n_bigger.jpg 2014-03-10t00:29:48.392172+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#load_avg_1m=0.91 sample#load_avg_5m=0.48 sample#load_avg_15m=0.23 2014-03-10t00:29:48.392490+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#memory_total=1056.09mb sample#memory_rss=1023.97mb sample#memory_cache=0.00mb sample#memory_swap=32.13mb sample#memory_pgpgin=28008448pages sample#memory_pgpgout=33948237pages 2014-03-10t00:29:48.392958+00:00 heroku[worker.1]: process running mem=1056m(103.1%) 2014-03-10t00:29:48.393236+00:00 heroku[worker.1]: error r14 (memory quota exceeded) 2014-03-10t00:30:08.453128+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#load_avg_1m=0.65 sample#load_avg_5m=0.45 sample#load_avg_15m=0.23 2014-03-10t00:30:08.453346+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.18856063.9fde816b-1162-457c-aba0-b9b73584f61b sample#memory_total=1055.90mb sample#memory_rss=1023.71mb sample#memory_cache=0.00mb sample#memory_swap=32.19mb sample#memory_pgpgin=177823744pages sample#memory_pgpgout=34011963pages 2014-03-10t00:30:08.453806+00:00 heroku[worker.1]: process running mem=1055m(103.1%) 2014-03-10t00:30:08.454034+00:00 heroku[worker.1]: error r14 (memory quota exceeded)

so ask you, ladies , gentlemen: wtf!!!


Comments

Post a Comment

Popular posts from this blog

Android layout hidden on keyboard show -

i have strange problem layout. when tap on edittext , keyboard shown edittext stay on bottom , when tap on to hide keyboard , keyboard hides , edittext go should when want input text. when tap again, keyboard shown , edittext goes down behind keyboard... <relativelayout android:layout_width="match_parent" android:layout_height="match_parent" > <scrollview android:layout_width="match_parent" android:layout_height="fill_parent" android:above="@+id/rl_commands"> <textview android:layout_width="match_parent" android:layout_height="wrap_content"/> </scrollview> <relativelayout android:layout_width="match_parent android:layout_height="24dp" android:id="@+id/rl_commands" android:alignparentbottom="true">
Read more

google app engine - 403 Forbidden POST - Flask WTForms -

i have flask application running on google app engine. trying submit form built using wtform , keep getting following error. 403 forbidden: don't have permission access requested resource. either read-protected or not readable server. location.html (part of code) <form method=post action="/home/location"> .... <button type="submit" class="btn btn-primary" value="submit"> directions </button> </form> main.py @app.route('/home/location', methods=['post', 'get']) def location(): form = cfcdirections.direction(request.form) print(request.method) if request.method == 'post' , form.validate(): print("after if") directions = cfcdirections() street_no = directions.no street = directions.street suburb = directions.suburb postcode = directions.postcode
Read more

c - Why would PK11_GenerateRandom() return an error -8023? -

i looking through internet trying find source of pk11_generaterandom() function see why function fail. have program uses function when moved new flavor of linux, fails after forking ( fork() ) since not believe there problem nss, suspect doing incorrectly disregarded in older versions of linux new 1 there issue. the openssl package same on 'good' , 'bad' server: openssl 0.9.8e-fips-rhel5 01 jul nss rpm differs though. 'good' has nss-3.12.2.0-2.el5 and bas has version nss-3.15.3-4.el5_10 the 'good' server uses quite obsolete linux: linux 2.6.18-128.el5 #1 smp wed jan 21 08:45:05 est 2009 x86_64 x86_64 x86_64 gnu/linux enterprise linux enterprise linux server release 5.3 (carthage) red hat enterprise linux server release 5.3 (tikanga) the 'bad' server newer: linux bad 2.6.18-371.4.1.el5 #1 smp wed jan 29 11:05:49 pst 2014 x86_64 x86_64 x86_64 gnu/linux oracle linux server release 5.10 red hat enterprise linux server release
Read more