Compare that to ADF - if your ADF pipeline does not need any complex transformations, the underlying Azure Databricks cluster is not provisioned and you can run your pipeline with a few clicks. When you run an AWS Glue job for the first time, Glue builds the underlying serverless environment and that can take some significant amount of time. With ADF, I did not have to deal with any of that.Įven for a simple use case with minimum transformation, there were a lot more clicks and a lot more page navigation in AWS Glue.
Azure Data Factory has a similar quickstart.īefore jumping into the AWS Glue tutorial, I read through the documentation to setup the required IAM roles for AWS Glue. AWS Glue has a tutorial using the same use case and I just followed the steps in the tutorial. My comparison was based on a very simple (and contrived) use case - take CSV files from a cloud storage location, modify the layout i.e change the schema, write it to a cloud storage location. There are quite a few comparisons out there, like this one from Stitch for instance. I liked the visual low-code development platform I think it is very well organized and I found it very easy to navigate.Ĭontinuing my tech exploration in times of COVID-19, I decided to try out AWS Glue as well.
The documentation and tutorials are very helpful, and there are many pipeline templates to reference. Getting started on ADF was fairly straightforward. The transformed data was sent to the SaaS application using a REST endpoint that was invoked by an Azure Function as part of an ADF pipeline activity. The proof-of-concept used Azure Blob Storage as a data source and Azure Data Factory (ADF) pipelines staged and transformed the data using ADF native capability and Azure Functions. Both are serverless offerings and both use Spark as an underlying tech stack.Ī few months ago, I had the opportunity to try out Azure Data Factory to build a data integration flow. Both are PaaS products focused on ETL/ELT. or how I spent my time exploring technology in times of COVID-19)Īzure Data Factory and AWS Glue are competing products from competing cloud service providers.
0 kommentar(er)
