2,115 questions
1
vote
1
answer
62
views
Pyspark - Resolve nested json file into multiple columns using inbuilt spark functions
I want to parse a JSON request and create multiple columns out of it in pyspark as follows:
{
"ID": "abc123",
"device": "mobile",
"Ads": [
{
...
- 659
0
votes
1
answer
48
views
Explode and duplicate rows in df and create a new column with the correct count of the values
In input, I have this dataframe :
numberType
NumberInst
Type
1
None
Car
2
1
Bus
3
1
Plane
I did that to explode and duplicate the rows :
df= df.with_columns(pl.col("NumberInst").alias("...
- 43
1
vote
2
answers
82
views
Explode Function and Reshaping in one Step
I have:
keys = ["panda1", "panda2", "panda3"]
values = [["eats", "shoots"], ["shoots", "leaves"], ["eats", "leaves&...
0
votes
1
answer
31
views
array into multiple rows in hive
I have a hive table with one of the example row:
A, B, [“11”, “12”, “13”], [“1”, “2”]
The array lengths of columns 3 and 4 are sometimes the same, and sometimes not.
Output:
A, B, 11, 1
A, B, 12, 2
A, ...
- 1
0
votes
1
answer
42
views
spark sql explode not giving correct result
I have a spark data frame with columns id and list of dates.
example below:
id | dates
aa | 2019-02-24, 2019-02-27, 2019-02-30
ab | 2023-04-02, 2023-04-06, 2023-04-09, 2023-04-12
b | 2012-07-15, ...
0
votes
1
answer
57
views
Explode multiple columns in pandas dataframe [duplicate]
I am trying to explode below dataframe based on a delimiter '|'.
preferred_title_symbol mim_number MDR_code MDR_term
0 17-BETA HYDROXYSTEROID DEHYDROGENASE III ...
- 1,495
2
votes
2
answers
93
views
create a subset array-of-struct column without exploding
new to pyspark. i have this example dataframe:
df = spark.createDataFrame(
(("7dc88", "D21", 14.14, 2, 10, [["msgA", 15, "a"],["msgB", 9, "g&...
- 335
0
votes
2
answers
103
views
Explode Pandas dataframe with mismatched number of elements
I work on a dataset like this one and I want to explode it with the content of each list.
index col1 col2 col3
1 [A,B] [,xx] [1,2]
2 [A,C] [zz,xx] [3,4]
3 ...
- 1
0
votes
1
answer
233
views
Explode array of maps with values in HIVE or Impala
I have a data table with two columns ID and Details that looks like:
Here is the code to create the above dataset:
select stack(4,
115,array(map('AA_Desc', 'Sale', 'AA_Date', '202403', '...
- 320
-2
votes
1
answer
116
views
Should I use explode or Unpivot?
I have a table consist of No Contract, Past, Current, and Expired columns.
|No.Cont| Past |Current|Expired|
+-------+------+-------+-------+
|113 | X | | |
|114 | | X | ...
- 1
0
votes
0
answers
101
views
Pyspark Json Extract Values
I am working with a PySpark DataFrame that has a JSON column from CSV file budgetthresholdaction. The JSON structure looks like this:
{
"budgetThresholdAction0threshold0": {
"...
- 543
3
votes
0
answers
4k
views
How is Spark's "exploding" of array/map fields a SELECT operation?
I am new to Python a Spark, currently working through this
tutorial
on Spark's explode operation for array/map fields of a DataFrame.
Based on the very first section 1 (PySpark explode array or map
...
- 1,265
1
vote
1
answer
4k
views
Exploding a StructType column in PySpark
Is there a way to explode a Struct column in a Spark DataFrame like you would explode an Array column? Meaning to take each element of the Struct (a key-value pair) value and create a separate row for ...
1
vote
4
answers
144
views
Explode string to array in PHP (strings are similar to WP shortcodes)
I'm working on a type of page builder with Laravel and try to implement shortcodes WordPress style. I have HTML code as a string like this:
$string = '<div class="temp">
[title ...
- 21
-3
votes
2
answers
62
views
Why explode shows the above results in a while php
I make an explode inside a while to separate words with a comma (,) and fix it so that it can be put in a sql query, in the first sample of the while it correctly outputs what I want but in the second ...
