User Tools

Site Tools


transformations:filesplitter

SPLIT DELIMITED FILE

Category: Import / File


Description

This action aims to help in processing very large text files. It splits a text file into smaller chunks (which are also text files) that can further be processed using iterations. The chunks can be either of fixed length (by row count), or split by unique values of a column (e.g. one chunk per unique Date).


Action settings

Setting Description
Input file*Fully-qualified file name of the text file to be split (includes relative or absolute path).
EncodingASCII, ANSI (with code page), and other types of encoding. If you're not sure what to choose, try UTF-8 as it's the most
common Unicode encoding.
Skip first lines*This option specifies how many lines to skip from the beginning of a file. It can be helpful in cases where the first rows contain
some metadata (file header) and actual tabular data starts only after that metadata. This setting can be specified
using a parameter as well.
Output folder*The folder location the split files will be saved to.
Output modeThe method by which the input file will be split.\\Options: Split by fixed row count (define the number of rows), or
Split by column value (define delimiter and column)

* Setting can be specified using a parameter.

Output mode settings

SettingDescription
Split by fixed row countThe input file will be split into a new file every specified number of rows. The Chunk size (rows)* setting defines the
number of rows. The output files will include an 8-digit sequence number appended to the filenames.
Split by column valueThe input file will be split into a new file for every unique value found in the specified column. Define the delimiter
the file uses with the Separator setting. Select the column used to define the unique values with the Column setting.
The output files will include the unique value names the files were split into appended to the filenames.

* Setting can be specified using a parameter.


Remarks

Since this action doesn't load files in memory, it can be used for splitting files so large that they can't fit in RAM.

In addition to splitting the input file, this action outputs a single-column dataset containing a list of the split files' filenames.


Examples

Example 1: Splitting a comma-delimited text file with 10,000 rows, and splitting every 1,000 rows.

(Split by fixed row count) Chunk size (rows) is 1000

Output:

  • 10 1000-row files with "00000001" through "00000010" appended to the filenames
  • A workflow dataset containing the fully-qualified filenames of the split files.



Example 2: Splitting a comma-delimited text file with 10,000 rows, and splitting by each unique Region value.

(Split by column value) Separator is Comma
(Split by column value) Column is Region

Output:

  • One file for each unique Region value, with the Region values appended to the filenames.
  • A workflow dataset containing the fully-qualified filenames of the split files.


See also

transformations/filesplitter.txt · Last modified: 2021/04/08 23:48 by craigt