Trigger.AvailableNow support for OpenSharing streaming (GA)
docs.databricks.com databricks.com
This is what was read here when it was published. databricks.com no longer carries it in what it publishes now.
Listen to this summary
Streaming reads from OpenSharing tables now support the Trigger.AvailableNow trigger. This trigger captures the shared table's current version when it starts, processes all available data across one or more micro-batches (honoring maxFilesPerTrigger and maxVersionsPerRpc), and then stops, which is well suited to scheduled, batch-like incremental processing. Previously, Trigger.AvailableNow on a OpenSharing source was converted to Trigger.Once, which processed a single micro-batch per run. This feature requires Databricks Runtime 18.0 or above.
See Query a table using Apache Spark Structured Streaming for Databricks-to-Databricks recipients and Access a shared table using Spark Structured Streaming for open sharing recipients.
More ways to use this
Open on another device
Scan to open this page. Your saved notes and account access are not included.