User Tools

Site Tools


overview

Overview of EasyMorph


Data import

In order to start transforming data in EasyMorph, all data subject to manipulations has to be imported into EasyMorph. As of version 4.7 EasyMorph can import the following data sources/formats:

  • Native and ODBC database drivers (see the full list of database drivers)
  • REST APIs (JSON/XML)
  • File formats
    • Text files with delimiters (e.g. CSV)
    • Text files with fixed width columns
    • Excel spreadsheets (.xlsx, .xlsm, .xls)
    • XML and JSON files
    • Qlik data files (.qvd)
    • SAS files (.sas7bdat)
    • SPSS/PSPP files (.sav)
    • EasyMorph datasets (.dset)
  • Cloud applications
    • Google Sheets
    • Google Analytics
    • SharePoint (lists)
    • Airtable
    • Pipedrive

Once tables are loaded into EasyMorph, they can be modified using actions (transformations) that are applied sequentially one after another.

Databases can be queried using the visual Query Editor intended for users who don't know SQL.

Files can be loaded by simply dragging them into EasyMorph. It's possible to load and automatically append multiple files at once.

Tutorial: Loading data from files
Tutorial: Loading data from databases


Actions

All manipulations with tables are done with the help of various actions. E.g. renaming a column is an action. Filtering, joining, aggregations – all are actions. As of version 4.5, EasyMorph offers more than 120 various actions that can be divided into 5 categories:

  • Create – actions that create or generate a table
  • Import – actions that import data into EasyMorph
  • Transform – actions that transform previously loaded data
  • Export – actions that export data into files, databases, applications and external systems
  • Workflow – actions that organize project workflow, perform various operations with files and external systems

Actions are executed consequently, step-by-step. One action is one step. An action is applied to the result (output) of its previous action. Therefore the order of actions is important. The same actions ordered differently would usually produce a different result (although not always).

The final state (result) of a table is the result of all actions of this table applied consequently, one after another.

Some actions (e.g. Append Table) require another table. These actions have two inputs – one input is the result of the previous action in the same table, and the other input is the final state (i.e. the result of the last action) of the other table.


Expressions

Some actions can use expressions in order to calculate new columns or filter rows. These expressions can be composed using arithmetical operators, column names, parameters as well as functions.

As of version 4.7 EasyMorph more than 150 functions can be used in expressions:

Tutorial: Type system and expressions


Derived tables

Derived tables are a powerful feature that allows transforming data in a non-linear way. When one table is derived from another (source) table, it replicates the final result of the latter. So if the source table changes, the derived table changes automatically. In everything else derived tables is just a regular table and can further be modified using various actions.

Since you can derive any number of tables from a table and then keep transforming every derived table in its own way, you can create non-linear action chains. For instance, you can produce two derived tables from a table, transform each of them in its own way and then append or merge them back into one table.

Derived tables can be calculated conditionally, which allows designing conditionally branched workflows. For instance, if a certain data quality check failed, send an email with bad records attached.

Tutorial: Derived tables


Data export

Exporting data is performed using export actions. They can be inserted at any point of workflow. Export actions do not modify tables. They export them into a specified file format or external system. Therefore the output of an export action is always the same as the input.

As of version 4.7 EasyMorh can export into the following data formats and systems:

  • Native and ODBC database drivers (see the full list of database drivers)
  • Web APIs (as JSON/XML payload)
  • Files
    • Text files with delimiters (e.g. CSV)
    • Excel spreadsheets (.xlsx)
    • XML files (.xml)
    • Tableau extracts (.tde, .hyper)
    • Qlik data files (.qvd)
    • EasyMorph datasets (.dset)
  • Cloud/enterprise applications
    • Google Sheets
    • Power BI
    • Tableau Server / Online
    • Airtable
    • EasyMorph Server

Tutorial: Exporting data into a file
Tutorial: Exporting data into a database


Workflow automation

EasyMorph workflows can perform external actions, for instance:

  • File and folder operations (copy/rename/delete/unzip/zip/download)
  • Running external applications, scripts and batch files
  • Execution of Windows shell commands
  • Embedded PowerShell scripting

A workflow can be composed using advanced patterns:

  • FOR..EACH type of loops (iterations)
  • DO..WHILE/UNTIL type of loops (iterations)
  • IF..THEN..ELSE type of conditional branching
  • Subroutines
  • Error recovery

Workflows can have a complex hierarchical structure when a project can consist of multiple modules, and each module can consist of multiple table groups. Such a structure allows designing and navigating workflows with hundreds of transformation steps (actions).


Integration with enterprise / cloud applications

EasyMorph workflows can also integrate and interact with external systems and applications.

External system Integration
IMAP email server Send / receive emails
SSH server Execute commands
Microsoft Exchange Send / receive emails
SFTP/SCP Download / upload files
REST API Send HTTP requests and receive responses
Microsoft SharePoint Download / upload files
Google Drive Download / upload files
Amazon S3 Download / upload files
Power BI Publish push-datasets, refresh datasets, trigger flows
Tableau Server / Online Publish extracts, trigger extract refreshes, trigger flows
Qlik Sense Trigger task reloads
Airtable Import, export, update, delete tables


Auto-documentation

EasyMorph can automatically generate human-readable documentation in plain English for the current project. Read more about Auto-documentation.


Scheduling

EasyMorph Launcher is a free complementary scheduling/triggering utility that comes with EasyMorph Desktop. The Launcher allows creating tasks with EasyMorph projects that are triggered manually when needed or scheduled on various schedules (Once, Continuous, Daily, Weekly). The Launcher resides in the system tray, starts on Windows start, and is tightly integrated with Desktop.

EasyMorph Server is used for robust, production-ready scheduling of heavy ETL workloads.

Tutorial: Scheduling


Command-line execution

EasyMorph projects can be executed from the command line, thus allowing integration with third-party applications. Read more about command line parameters.

A more robust integration option is the "ems-cmd" command-line utility that allows triggering EasyMorph Server tasks from external applications running on Windows or Linux.

Tutorial: Running projects from the command line


EasyMorph Server

EasyMorph Server is a Windows service with a web interface that employs under the hood the same in-memory engine as EasyMorph Desktop. The Server allows scheduled execution of projects, sharing files and datasets between users, and hosting connector repositories. It also provides a REST API for programmatic integration with external systems.


EasyMorph Community

The EasyMorph Community is an online forum for EasyMorph users. It contains hundreds of practical questions answered, frequently with downloadable, ready-to-use examples.

See also

overview.txt · Last modified: 2021/05/06 07:09 by dmitry