Veri Bilimi Okulu

Üye Ol
Giriş Yap
Frame 1000005891
Apache Spark ile Parquet Formatında Veri Yazma ve Okuma (Scala)
Anasayfa
Apache Spark ile Parquet Formatında Veri Yazma ve Okuma (Scala)
Erkan ŞİRİN|
| Yorumlar 0
spark_parquet

Loading

Merhaba. Bu yazımızda Apache Spark ile 

csv
csv uzantılı bir veri dosyasını okuyup 
parquet
parquet uzantılı olarak diske kaydedeceğiz ve kaydettiğimiz parquet dosyasını tekrar Spark ile okuyacağız. Spark, birçok formatta veriyi okuyup yazabiliyor. Parquet, csv dosyalarına göre daha az yer kapladığı gibi okuma performansı da daha yüksek. Ayrıca okuma yaparken inferSchema veya header gibi ilave seçenekler kullanmıyoruz. Bu bilgiyi parquet formatından alıyor. Bu kısa açıklamadan sonra uygulamaya geçebiliriz. Uygulama esnasında kullanacağımız veri setini buradan edinebilirsiniz.

Benim yaptığım çalışma esnasında kullandığım ortam bilgileri:

İşletim sistemi: Windows 10 64 bit Pro

IDE: Intellij IDEA Community Edition

Spark 2.3.1

Dil: Scala

Kodlara buradan erişebilirsiniz.

Kütüphaneler:

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
import org.apache.log4j.{Logger, Level}
import org.apache.spark.sql.SparkSession
import org.apache.log4j.{Logger, Level} import org.apache.spark.sql.SparkSession
import org.apache.log4j.{Logger, Level}
import org.apache.spark.sql.SparkSession

SparkSession:

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
Logger.getLogger("org").setLevel(Level.ERROR) // prevent info logs
val spark = SparkSession.builder()
.master("local[4]")
.appName("ReadFromParquet")
.config("spark.driver.memory","2g")
.config("spark.executor.memory","4g")
.getOrCreate()
Logger.getLogger("org").setLevel(Level.ERROR) // prevent info logs val spark = SparkSession.builder() .master("local[4]") .appName("ReadFromParquet") .config("spark.driver.memory","2g") .config("spark.executor.memory","4g") .getOrCreate()
Logger.getLogger("org").setLevel(Level.ERROR) // prevent info logs

   val spark = SparkSession.builder()
     .master("local[4]")
     .appName("ReadFromParquet")
     .config("spark.driver.memory","2g")
     .config("spark.executor.memory","4g")
     .getOrCreate()

Csv kaynağından veri okuma:

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
// read from csv
val df = spark.read.format("csv")
.option("header","true")
.option("inferSchema","true")
.option("sep",";")
.load("D:\\Datasets\\OnlineRetail.csv")
// read from csv val df = spark.read.format("csv") .option("header","true") .option("inferSchema","true") .option("sep",";") .load("D:\\Datasets\\OnlineRetail.csv")
// read from csv
  val df = spark.read.format("csv")
    .option("header","true")
    .option("inferSchema","true")
    .option("sep",";")
    .load("D:\\Datasets\\OnlineRetail.csv")

Okuduğumuz csv dosyasına göz atma:

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
// show csv read data
df.show()
// show csv read data df.show()
// show csv read data
    df.show()

Çıktı:

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
+---------+---------+--------------------+--------+---------------+---------+----------+--------------+
|InvoiceNo|StockCode| Description|Quantity| InvoiceDate|UnitPrice|CustomerID| Country|
+---------+---------+--------------------+--------+---------------+---------+----------+--------------+
| 536365| 85123A|WHITE HANGING HEA...| 6|1.12.2010 08:26| 2,55| 17850|United Kingdom|
| 536365| 71053| WHITE METAL LANTERN| 6|1.12.2010 08:26| 3,39| 17850|United Kingdom|
| 536365| 84406B|CREAM CUPID HEART...| 8|1.12.2010 08:26| 2,75| 17850|United Kingdom|
| 536365| 84029G|KNITTED UNION FLA...| 6|1.12.2010 08:26| 3,39| 17850|United Kingdom|
| 536365| 84029E|RED WOOLLY HOTTIE...| 6|1.12.2010 08:26| 3,39| 17850|United Kingdom|
| 536365| 22752|SET 7 BABUSHKA NE...| 2|1.12.2010 08:26| 7,65| 17850|United Kingdom|
| 536365| 21730|GLASS STAR FROSTE...| 6|1.12.2010 08:26| 4,25| 17850|United Kingdom|
| 536366| 22633|HAND WARMER UNION...| 6|1.12.2010 08:28| 1,85| 17850|United Kingdom|
| 536366| 22632|HAND WARMER RED P...| 6|1.12.2010 08:28| 1,85| 17850|United Kingdom|
| 536367| 84879|ASSORTED COLOUR B...| 32|1.12.2010 08:34| 1,69| 13047|United Kingdom|
| 536367| 22745|POPPY'S PLAYHOUSE...| 6|1.12.2010 08:34| 2,1| 13047|United Kingdom|
| 536367| 22748|POPPY'S PLAYHOUSE...| 6|1.12.2010 08:34| 2,1| 13047|United Kingdom|
| 536367| 22749|FELTCRAFT PRINCES...| 8|1.12.2010 08:34| 3,75| 13047|United Kingdom|
| 536367| 22310|IVORY KNITTED MUG...| 6|1.12.2010 08:34| 1,65| 13047|United Kingdom|
| 536367| 84969|BOX OF 6 ASSORTED...| 6|1.12.2010 08:34| 4,25| 13047|United Kingdom|
| 536367| 22623|BOX OF VINTAGE JI...| 3|1.12.2010 08:34| 4,95| 13047|United Kingdom|
| 536367| 22622|BOX OF VINTAGE AL...| 2|1.12.2010 08:34| 9,95| 13047|United Kingdom|
| 536367| 21754|HOME BUILDING BLO...| 3|1.12.2010 08:34| 5,95| 13047|United Kingdom|
| 536367| 21755|LOVE BUILDING BLO...| 3|1.12.2010 08:34| 5,95| 13047|United Kingdom|
| 536367| 21777|RECIPE BOX WITH M...| 4|1.12.2010 08:34| 7,95| 13047|United Kingdom|
+---------+---------+--------------------+--------+---------------+---------+----------+--------------+
only showing top 20 rows
+---------+---------+--------------------+--------+---------------+---------+----------+--------------+ |InvoiceNo|StockCode| Description|Quantity| InvoiceDate|UnitPrice|CustomerID| Country| +---------+---------+--------------------+--------+---------------+---------+----------+--------------+ | 536365| 85123A|WHITE HANGING HEA...| 6|1.12.2010 08:26| 2,55| 17850|United Kingdom| | 536365| 71053| WHITE METAL LANTERN| 6|1.12.2010 08:26| 3,39| 17850|United Kingdom| | 536365| 84406B|CREAM CUPID HEART...| 8|1.12.2010 08:26| 2,75| 17850|United Kingdom| | 536365| 84029G|KNITTED UNION FLA...| 6|1.12.2010 08:26| 3,39| 17850|United Kingdom| | 536365| 84029E|RED WOOLLY HOTTIE...| 6|1.12.2010 08:26| 3,39| 17850|United Kingdom| | 536365| 22752|SET 7 BABUSHKA NE...| 2|1.12.2010 08:26| 7,65| 17850|United Kingdom| | 536365| 21730|GLASS STAR FROSTE...| 6|1.12.2010 08:26| 4,25| 17850|United Kingdom| | 536366| 22633|HAND WARMER UNION...| 6|1.12.2010 08:28| 1,85| 17850|United Kingdom| | 536366| 22632|HAND WARMER RED P...| 6|1.12.2010 08:28| 1,85| 17850|United Kingdom| | 536367| 84879|ASSORTED COLOUR B...| 32|1.12.2010 08:34| 1,69| 13047|United Kingdom| | 536367| 22745|POPPY'S PLAYHOUSE...| 6|1.12.2010 08:34| 2,1| 13047|United Kingdom| | 536367| 22748|POPPY'S PLAYHOUSE...| 6|1.12.2010 08:34| 2,1| 13047|United Kingdom| | 536367| 22749|FELTCRAFT PRINCES...| 8|1.12.2010 08:34| 3,75| 13047|United Kingdom| | 536367| 22310|IVORY KNITTED MUG...| 6|1.12.2010 08:34| 1,65| 13047|United Kingdom| | 536367| 84969|BOX OF 6 ASSORTED...| 6|1.12.2010 08:34| 4,25| 13047|United Kingdom| | 536367| 22623|BOX OF VINTAGE JI...| 3|1.12.2010 08:34| 4,95| 13047|United Kingdom| | 536367| 22622|BOX OF VINTAGE AL...| 2|1.12.2010 08:34| 9,95| 13047|United Kingdom| | 536367| 21754|HOME BUILDING BLO...| 3|1.12.2010 08:34| 5,95| 13047|United Kingdom| | 536367| 21755|LOVE BUILDING BLO...| 3|1.12.2010 08:34| 5,95| 13047|United Kingdom| | 536367| 21777|RECIPE BOX WITH M...| 4|1.12.2010 08:34| 7,95| 13047|United Kingdom| +---------+---------+--------------------+--------+---------------+---------+----------+--------------+ only showing top 20 rows
+---------+---------+--------------------+--------+---------------+---------+----------+--------------+
|InvoiceNo|StockCode|         Description|Quantity|    InvoiceDate|UnitPrice|CustomerID|       Country|
+---------+---------+--------------------+--------+---------------+---------+----------+--------------+
|   536365|   85123A|WHITE HANGING HEA...|       6|1.12.2010 08:26|     2,55|     17850|United Kingdom|
|   536365|    71053| WHITE METAL LANTERN|       6|1.12.2010 08:26|     3,39|     17850|United Kingdom|
|   536365|   84406B|CREAM CUPID HEART...|       8|1.12.2010 08:26|     2,75|     17850|United Kingdom|
|   536365|   84029G|KNITTED UNION FLA...|       6|1.12.2010 08:26|     3,39|     17850|United Kingdom|
|   536365|   84029E|RED WOOLLY HOTTIE...|       6|1.12.2010 08:26|     3,39|     17850|United Kingdom|
|   536365|    22752|SET 7 BABUSHKA NE...|       2|1.12.2010 08:26|     7,65|     17850|United Kingdom|
|   536365|    21730|GLASS STAR FROSTE...|       6|1.12.2010 08:26|     4,25|     17850|United Kingdom|
|   536366|    22633|HAND WARMER UNION...|       6|1.12.2010 08:28|     1,85|     17850|United Kingdom|
|   536366|    22632|HAND WARMER RED P...|       6|1.12.2010 08:28|     1,85|     17850|United Kingdom|
|   536367|    84879|ASSORTED COLOUR B...|      32|1.12.2010 08:34|     1,69|     13047|United Kingdom|
|   536367|    22745|POPPY'S PLAYHOUSE...|       6|1.12.2010 08:34|      2,1|     13047|United Kingdom|
|   536367|    22748|POPPY'S PLAYHOUSE...|       6|1.12.2010 08:34|      2,1|     13047|United Kingdom|
|   536367|    22749|FELTCRAFT PRINCES...|       8|1.12.2010 08:34|     3,75|     13047|United Kingdom|
|   536367|    22310|IVORY KNITTED MUG...|       6|1.12.2010 08:34|     1,65|     13047|United Kingdom|
|   536367|    84969|BOX OF 6 ASSORTED...|       6|1.12.2010 08:34|     4,25|     13047|United Kingdom|
|   536367|    22623|BOX OF VINTAGE JI...|       3|1.12.2010 08:34|     4,95|     13047|United Kingdom|
|   536367|    22622|BOX OF VINTAGE AL...|       2|1.12.2010 08:34|     9,95|     13047|United Kingdom|
|   536367|    21754|HOME BUILDING BLO...|       3|1.12.2010 08:34|     5,95|     13047|United Kingdom|
|   536367|    21755|LOVE BUILDING BLO...|       3|1.12.2010 08:34|     5,95|     13047|United Kingdom|
|   536367|    21777|RECIPE BOX WITH M...|       4|1.12.2010 08:34|     7,95|     13047|United Kingdom|
+---------+---------+--------------------+--------+---------------+---------+----------+--------------+
only showing top 20 rows

Gayet güzel okundu. Şimdi bu df’i parquet formatında diske yazalım:

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
// save df as parquet
df // df to write
.coalesce(1) // as a 1 partition not many
.write // write
.mode("overwrite") // do I overwrite: yes
.parquet("D:\\Datasets\\OnlineRetailParquet") // where to write (need to write permission)
// save df as parquet df // df to write .coalesce(1) // as a 1 partition not many .write // write .mode("overwrite") // do I overwrite: yes .parquet("D:\\Datasets\\OnlineRetailParquet") // where to write (need to write permission)
// save df as parquet
    df // df to write
      .coalesce(1) // as a 1 partition not many
      .write // write
      .mode("overwrite")  // do I overwrite: yes
      .parquet("D:\\Datasets\\OnlineRetailParquet") // where to write (need to write permission)

Yazdığımızı okuyalım:

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
// read from parquet
val df_parquet = spark.read.format("parquet")
.load("D:\\Datasets\\OnlineRetailParquet")
// read from parquet val df_parquet = spark.read.format("parquet") .load("D:\\Datasets\\OnlineRetailParquet")
// read from parquet
    val df_parquet = spark.read.format("parquet")
      .load("D:\\Datasets\\OnlineRetailParquet")

Okuduğumuzu görelim:

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
// show parquet read data
df_parquet.show()
// show parquet read data df_parquet.show()
// show parquet read data
  df_parquet.show()

Çıktı:

Plain text
Copy to clipboard
Open code in new window
EnlighterJS 3 Syntax Highlighter
+---------+---------+--------------------+--------+---------------+---------+----------+--------------+
|InvoiceNo|StockCode| Description|Quantity| InvoiceDate|UnitPrice|CustomerID| Country|
+---------+---------+--------------------+--------+---------------+---------+----------+--------------+
| 536365| 85123A|WHITE HANGING HEA...| 6|1.12.2010 08:26| 2,55| 17850|United Kingdom|
| 536365| 71053| WHITE METAL LANTERN| 6|1.12.2010 08:26| 3,39| 17850|United Kingdom|
| 536365| 84406B|CREAM CUPID HEART...| 8|1.12.2010 08:26| 2,75| 17850|United Kingdom|
| 536365| 84029G|KNITTED UNION FLA...| 6|1.12.2010 08:26| 3,39| 17850|United Kingdom|
| 536365| 84029E|RED WOOLLY HOTTIE...| 6|1.12.2010 08:26| 3,39| 17850|United Kingdom|
| 536365| 22752|SET 7 BABUSHKA NE...| 2|1.12.2010 08:26| 7,65| 17850|United Kingdom|
| 536365| 21730|GLASS STAR FROSTE...| 6|1.12.2010 08:26| 4,25| 17850|United Kingdom|
| 536366| 22633|HAND WARMER UNION...| 6|1.12.2010 08:28| 1,85| 17850|United Kingdom|
| 536366| 22632|HAND WARMER RED P...| 6|1.12.2010 08:28| 1,85| 17850|United Kingdom|
| 536367| 84879|ASSORTED COLOUR B...| 32|1.12.2010 08:34| 1,69| 13047|United Kingdom|
| 536367| 22745|POPPY'S PLAYHOUSE...| 6|1.12.2010 08:34| 2,1| 13047|United Kingdom|
| 536367| 22748|POPPY'S PLAYHOUSE...| 6|1.12.2010 08:34| 2,1| 13047|United Kingdom|
| 536367| 22749|FELTCRAFT PRINCES...| 8|1.12.2010 08:34| 3,75| 13047|United Kingdom|
| 536367| 22310|IVORY KNITTED MUG...| 6|1.12.2010 08:34| 1,65| 13047|United Kingdom|
| 536367| 84969|BOX OF 6 ASSORTED...| 6|1.12.2010 08:34| 4,25| 13047|United Kingdom|
| 536367| 22623|BOX OF VINTAGE JI...| 3|1.12.2010 08:34| 4,95| 13047|United Kingdom|
| 536367| 22622|BOX OF VINTAGE AL...| 2|1.12.2010 08:34| 9,95| 13047|United Kingdom|
| 536367| 21754|HOME BUILDING BLO...| 3|1.12.2010 08:34| 5,95| 13047|United Kingdom|
| 536367| 21755|LOVE BUILDING BLO...| 3|1.12.2010 08:34| 5,95| 13047|United Kingdom|
| 536367| 21777|RECIPE BOX WITH M...| 4|1.12.2010 08:34| 7,95| 13047|United Kingdom|
+---------+---------+--------------------+--------+---------------+---------+----------+--------------+
only showing top 20 rows
+---------+---------+--------------------+--------+---------------+---------+----------+--------------+ |InvoiceNo|StockCode| Description|Quantity| InvoiceDate|UnitPrice|CustomerID| Country| +---------+---------+--------------------+--------+---------------+---------+----------+--------------+ | 536365| 85123A|WHITE HANGING HEA...| 6|1.12.2010 08:26| 2,55| 17850|United Kingdom| | 536365| 71053| WHITE METAL LANTERN| 6|1.12.2010 08:26| 3,39| 17850|United Kingdom| | 536365| 84406B|CREAM CUPID HEART...| 8|1.12.2010 08:26| 2,75| 17850|United Kingdom| | 536365| 84029G|KNITTED UNION FLA...| 6|1.12.2010 08:26| 3,39| 17850|United Kingdom| | 536365| 84029E|RED WOOLLY HOTTIE...| 6|1.12.2010 08:26| 3,39| 17850|United Kingdom| | 536365| 22752|SET 7 BABUSHKA NE...| 2|1.12.2010 08:26| 7,65| 17850|United Kingdom| | 536365| 21730|GLASS STAR FROSTE...| 6|1.12.2010 08:26| 4,25| 17850|United Kingdom| | 536366| 22633|HAND WARMER UNION...| 6|1.12.2010 08:28| 1,85| 17850|United Kingdom| | 536366| 22632|HAND WARMER RED P...| 6|1.12.2010 08:28| 1,85| 17850|United Kingdom| | 536367| 84879|ASSORTED COLOUR B...| 32|1.12.2010 08:34| 1,69| 13047|United Kingdom| | 536367| 22745|POPPY'S PLAYHOUSE...| 6|1.12.2010 08:34| 2,1| 13047|United Kingdom| | 536367| 22748|POPPY'S PLAYHOUSE...| 6|1.12.2010 08:34| 2,1| 13047|United Kingdom| | 536367| 22749|FELTCRAFT PRINCES...| 8|1.12.2010 08:34| 3,75| 13047|United Kingdom| | 536367| 22310|IVORY KNITTED MUG...| 6|1.12.2010 08:34| 1,65| 13047|United Kingdom| | 536367| 84969|BOX OF 6 ASSORTED...| 6|1.12.2010 08:34| 4,25| 13047|United Kingdom| | 536367| 22623|BOX OF VINTAGE JI...| 3|1.12.2010 08:34| 4,95| 13047|United Kingdom| | 536367| 22622|BOX OF VINTAGE AL...| 2|1.12.2010 08:34| 9,95| 13047|United Kingdom| | 536367| 21754|HOME BUILDING BLO...| 3|1.12.2010 08:34| 5,95| 13047|United Kingdom| | 536367| 21755|LOVE BUILDING BLO...| 3|1.12.2010 08:34| 5,95| 13047|United Kingdom| | 536367| 21777|RECIPE BOX WITH M...| 4|1.12.2010 08:34| 7,95| 13047|United Kingdom| +---------+---------+--------------------+--------+---------------+---------+----------+--------------+ only showing top 20 rows
+---------+---------+--------------------+--------+---------------+---------+----------+--------------+
|InvoiceNo|StockCode|         Description|Quantity|    InvoiceDate|UnitPrice|CustomerID|       Country|
+---------+---------+--------------------+--------+---------------+---------+----------+--------------+
|   536365|   85123A|WHITE HANGING HEA...|       6|1.12.2010 08:26|     2,55|     17850|United Kingdom|
|   536365|    71053| WHITE METAL LANTERN|       6|1.12.2010 08:26|     3,39|     17850|United Kingdom|
|   536365|   84406B|CREAM CUPID HEART...|       8|1.12.2010 08:26|     2,75|     17850|United Kingdom|
|   536365|   84029G|KNITTED UNION FLA...|       6|1.12.2010 08:26|     3,39|     17850|United Kingdom|
|   536365|   84029E|RED WOOLLY HOTTIE...|       6|1.12.2010 08:26|     3,39|     17850|United Kingdom|
|   536365|    22752|SET 7 BABUSHKA NE...|       2|1.12.2010 08:26|     7,65|     17850|United Kingdom|
|   536365|    21730|GLASS STAR FROSTE...|       6|1.12.2010 08:26|     4,25|     17850|United Kingdom|
|   536366|    22633|HAND WARMER UNION...|       6|1.12.2010 08:28|     1,85|     17850|United Kingdom|
|   536366|    22632|HAND WARMER RED P...|       6|1.12.2010 08:28|     1,85|     17850|United Kingdom|
|   536367|    84879|ASSORTED COLOUR B...|      32|1.12.2010 08:34|     1,69|     13047|United Kingdom|
|   536367|    22745|POPPY'S PLAYHOUSE...|       6|1.12.2010 08:34|      2,1|     13047|United Kingdom|
|   536367|    22748|POPPY'S PLAYHOUSE...|       6|1.12.2010 08:34|      2,1|     13047|United Kingdom|
|   536367|    22749|FELTCRAFT PRINCES...|       8|1.12.2010 08:34|     3,75|     13047|United Kingdom|
|   536367|    22310|IVORY KNITTED MUG...|       6|1.12.2010 08:34|     1,65|     13047|United Kingdom|
|   536367|    84969|BOX OF 6 ASSORTED...|       6|1.12.2010 08:34|     4,25|     13047|United Kingdom|
|   536367|    22623|BOX OF VINTAGE JI...|       3|1.12.2010 08:34|     4,95|     13047|United Kingdom|
|   536367|    22622|BOX OF VINTAGE AL...|       2|1.12.2010 08:34|     9,95|     13047|United Kingdom|
|   536367|    21754|HOME BUILDING BLO...|       3|1.12.2010 08:34|     5,95|     13047|United Kingdom|
|   536367|    21755|LOVE BUILDING BLO...|       3|1.12.2010 08:34|     5,95|     13047|United Kingdom|
|   536367|    21777|RECIPE BOX WITH M...|       4|1.12.2010 08:34|     7,95|     13047|United Kingdom|
+---------+---------+--------------------+--------+---------------+---------+----------+--------------+
only showing top 20 rows

Dosya büyüklüğü kontrolü yaptığımızda csv uzantılı dosya  45.024 KB yer kaplarken parquet dosyasının 3.446 KB yer kapladığını görüyoruz. Aynı bilgi bu kadar küçük bir dosyada nasıl saklanır gerçekten çok şaşırtıcı bir durum. Bu yazımızda bu kadar. Hoşçakalın…

Erkan Şirin
0

Related Posts:

Bir yanıt yazın

Password Requirements:

  • At least 8 characters
  • At least 1 lowercase letter
  • At least 1 uppercase letter
  • At least 1 numerical number
  • At least 1 special character