46 of 55.
A data engineer is implementing a streaming pipeline with watermarking to handle late-arriving records.
The engineer has written the following code:
inputStream \
.withWatermark("event_time", "10 minutes") \
.groupBy(window("event_time", "15 minutes"))
What happens to data that arrives after the watermark threshold?

• A. Any data arriving more than 10 minutes after the watermark threshold will be ignored and not included in the aggregation.
• B. Data arriving more than 10 minutes after the latest watermark will still be included in the aggregation but will be placed into the next window.
• C. Records that arrive later than the watermark threshold (10 minutes) will automatically be included in the aggregation if they fall within the 15-minute window.
• D. The watermark ensures that late data arriving within 10 minutes of the latest event time will be processed and included in the windowed aggregation.

An MLOps engineer is building a Pandas UDF that applies a language model that translates English strings into Spanish. The initial code is loading the model on every call to the UDF, which is hurting the performance of the data pipeline.
The initial code is:

def in_spanish_inner(df: pd.Series) -> pd.Series:
model = get_translation_model(target_lang='es')
return df.apply(model)
in_spanish = sf.pandas_udf(in_spanish_inner, StringType())
How can the MLOps engineer change this code to reduce how many times the language model is loaded?

• A. Convert the Pandas UDF to a PySpark UDF
• B. Convert the Pandas UDF from a Series → Series UDF to an Iterator[Series] → Iterator[Series] UDF
• C. Run the in_spanish_inner() function in a mapInPandas() function call
• D. Convert the Pandas UDF from a Series → Series UDF to a Series → Scalar UDF

47 of 55.
A data engineer has written the following code to join two DataFrames df1 and df2:
df1 = spark.read.csv("sales_data.csv")
df2 = spark.read.csv("product_data.csv")
df_joined = df1.join(df2, df1.product_id == df2.product_id)
The DataFrame df1 contains ~10 GB of sales data, and df2 contains ~8 MB of product data.
Which join strategy will Spark use?

• A. Broadcast join, as df2 is smaller than the default broadcast threshold.
• B. Shuffle join, as the size difference between df1 and df2 is too large for a broadcast join to work efficiently.
• C. Shuffle join, because AQE is not enabled, and Spark uses a static query plan.
• D. Shuffle join because no broadcast hints were provided.

A data engineer wants to create a Streaming DataFrame that reads from a Kafka topic called feed.

Which code fragment should be inserted in line 5 to meet the requirement?
Code context:
spark \
.readStream \
.format("kafka") \
.option("kafka.bootstrap.servers", "host1:port1,host2:port2") \
.[LINE 5] \
.load()
Options:

• A. .option("subscribe.topic", "feed")
• B. .option("subscribe", "feed")
• C. .option("topic", "feed")
• D. .option("kafka.topic", "feed")