hadoop - reading a file into array list in scala spark -


i new spark , scala.

i want read file array list.

this how done in java.

list<string> sourcerecords; sourcerecords = new arraylist<string>(); bufferedreader sw; sw = new bufferedreader(new filereader(srcpath[0].tostring())); string srcline ; while ((srcline = sw.readline()) != null)  { sourcerecords.add(srcline.tostring()); }

how in scala in spark

it's easy. example,

val rdd = sc.textfile("your_file_path") val sourcerecords = rdd.toarray

however, don't need convert rdd array. can manipulate rdd array.

you can find more information in https://spark.incubator.apache.org/examples.html


Comments

Post a Comment

Popular posts from this blog

google app engine - 403 Forbidden POST - Flask WTForms -

i have flask application running on google app engine. trying submit form built using wtform , keep getting following error. 403 forbidden: don't have permission access requested resource. either read-protected or not readable server. location.html (part of code) <form method=post action="/home/location"> .... <button type="submit" class="btn btn-primary" value="submit"> directions </button> </form> main.py @app.route('/home/location', methods=['post', 'get']) def location(): form = cfcdirections.direction(request.form) print(request.method) if request.method == 'post' , form.validate(): print("after if") directions = cfcdirections() street_no = directions.no street = directions.street suburb = directions.suburb postcode = directions.postcode ...
Read more

Android layout hidden on keyboard show -

i have strange problem layout. when tap on edittext , keyboard shown edittext stay on bottom , when tap on to hide keyboard , keyboard hides , edittext go should when want input text. when tap again, keyboard shown , edittext goes down behind keyboard... <relativelayout android:layout_width="match_parent" android:layout_height="match_parent" > <scrollview android:layout_width="match_parent" android:layout_height="fill_parent" android:above="@+id/rl_commands"> <textview android:layout_width="match_parent" android:layout_height="wrap_content"/> </scrollview> <relativelayout android:layout_width="match_parent android:layout_height="24dp" android:id="@+id/rl_commands" android:alignparentbottom="true"> ...
Read more

Parse xml element into list in Python -

i've been trying days store list of coordinates list in python. i've tried minidom, xmlreader, , few others. i'm doing wrong, i'm on deadline , need help. <?xml version="1.0" encoding="utf-8"?> <kml xmlns="http://www.opengis.net/kml/2.2" xmlns:gx="http://www.google.com/kml/ext/2.2" xmlns:kml="http://www.opengis.net/kml/2.2" xmlns:atom="http://www.w3.org/2005/atom"> <document> <name>kmlfile</name> <style id="s_ylw-pushpin_hl"> <iconstyle> <scale>1.3</scale> <icon> <href>http://maps.google.com/mapfiles/kml/pushpin/ylw-pushpin.png</href> </icon> <hotspot x="20" y="2" xunits="pixels" yunits="pixels"/> </iconstyle> </style> <stylemap id="m_ylw-pushpin"> ...
Read more