Azure Data Lake Writer Activity¶
Writes specified view to Azure Data Lake as a file.
Parameters¶
Name |
Description |
Type |
Default |
|---|---|---|---|
Title |
Title of the activity. Its displayed on the designer. |
text |
|
Source View |
Name of the DataRow view that is going to be written into the file. |
text |
|
Path |
File path in adl[s]://<account>.azuredatalakestore.net/<path> format. |
text |
|
Format |
File format. Please see below for supported file formats. |
enum |
|
Partition By |
List of columns to partition the output by. |
enum |
error |
Mode |
Write mode. Please see blow for list modes and their meanings. |
enum |
error |
Options |
Additional options (e.g. delimiter for CSV file could be specified as [delimiter,\t] if file is tab delimited |
key-value |
Supported File Formats¶
Name |
Description |
|---|---|
Parquet |
Apache Parquet is a columnar storage format. |
ORC |
ORC is also columnar storage format. |
Avro |
Apache Avro is a data serialization format. |
CSV |
CSV is a delimited text file format. |
JSON |
JSON is a lightweight data-interchange format. |
Supported Write Modes¶
Name |
Description |
|---|---|
error |
Fail if file already exists. |
append |
Append to existing file or create new. |
overwrite |
Overwrite existing file or create new. |
ignore |
Ignore if file already exists or create new. |
Job Parameters¶
Note
Following job parameters need to be set for proper Azure Data Lake authentication.
Key |
Value |
|---|---|
dfs.adls.oauth2.access.token.provider.type |
e.g. ClientCredential |
dfs.adls.oauth2.refresh.url |
under Azure Portal/Azure Active Directory/App Registrations/Endpoints/OAUTH 2.0 TOKEN ENDPOINT |
dfs.adls.oauth2.client.id |
AppId |
dfs.adls.oauth2.credential |
App Key |