org.apache.spark.sql.util

Class ArrowUtils

Object

org.apache.spark.sql.util.ArrowUtils

public class ArrowUtils
extends Object

Constructor Summary

Constructors

Constructor and Description
ArrowUtils()

Method Summary

Modifier and Type	Method and Description
static DataType	fromArrowField(org.apache.arrow.vector.types.pojo.Field field)
static StructType	fromArrowSchema(org.apache.arrow.vector.types.pojo.Schema schema)
static DataType	fromArrowType(org.apache.arrow.vector.types.pojo.ArrowType dt)
static scala.collection.immutable.Map<String,String>	getPythonRunnerConfMap(org.apache.spark.sql.internal.SQLConf conf) Return Map with conf settings to be used in ArrowPythonRunner
static org.apache.arrow.memory.RootAllocator	rootAllocator()
static org.apache.arrow.vector.types.pojo.Field	toArrowField(String name, DataType dt, boolean nullable, String timeZoneId) Maps field from Spark to Arrow.
static org.apache.arrow.vector.types.pojo.Schema	toArrowSchema(StructType schema, String timeZoneId) Maps schema from Spark to Arrow.
static org.apache.arrow.vector.types.pojo.ArrowType	toArrowType(DataType dt, String timeZoneId) Maps data type from Spark to Arrow.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ArrowUtils

public ArrowUtils()

Method Detail

rootAllocator

public static org.apache.arrow.memory.RootAllocator rootAllocator()

toArrowType

public static org.apache.arrow.vector.types.pojo.ArrowType toArrowType(DataType dt,
                                                                       String timeZoneId)

Maps data type from Spark to Arrow. NOTE: timeZoneId required for TimestampTypes

fromArrowType

public static DataType fromArrowType(org.apache.arrow.vector.types.pojo.ArrowType dt)

toArrowField

public static org.apache.arrow.vector.types.pojo.Field toArrowField(String name,
                                                                    DataType dt,
                                                                    boolean nullable,
                                                                    String timeZoneId)

Maps field from Spark to Arrow. NOTE: timeZoneId required for TimestampType

fromArrowField

public static DataType fromArrowField(org.apache.arrow.vector.types.pojo.Field field)

toArrowSchema

public static org.apache.arrow.vector.types.pojo.Schema toArrowSchema(StructType schema,
                                                                      String timeZoneId)

Maps schema from Spark to Arrow. NOTE: timeZoneId required for TimestampType in StructType

fromArrowSchema

public static StructType fromArrowSchema(org.apache.arrow.vector.types.pojo.Schema schema)

getPythonRunnerConfMap

public static scala.collection.immutable.Map<String,String> getPythonRunnerConfMap(org.apache.spark.sql.internal.SQLConf conf)

Return Map with conf settings to be used in ArrowPythonRunner