AWS Glue Studio has not too long ago added the potential of including customized transforms that you should use to construct visible jobs to make use of them together with the AWS Glue Studio elements supplied out of the field. Now you can outline customized visible remodel by merely dropping a JSON file and a Python script onto Amazon S3, which defines the part and the processing logic, respectively.

Customized visible remodel helps you to outline, reuse, and share business-specific ETL logic amongst your groups. With this new function, information engineers can write reusable transforms for the AWS Glue visible job editor. Reusable transforms improve consistency between groups and assist hold jobs up-to-date by minimizing duplicate effort and code.

On this weblog put up, I’ll present you a fictional use case that requires the creation of two customized transforms for example what you may accomplish with this new function. One part will generate artificial information on the fly for testing functions, and the opposite will put together the information to retailer it partitioned.

Use case: Generate artificial information on the fly

There are a number of the explanation why you’ll wish to have a part that generates artificial information. Possibly the actual information is closely restricted or not but obtainable, or there’s not sufficient amount or selection in the intervening time to check efficiency. Or possibly utilizing the actual information imposes some price or load to the actual system, and we wish to scale back its utilization throughout growth.

Utilizing the brand new customized visible transforms framework, let’s create a part that builds artificial information for fictional gross sales throughout a pure yr.

Outline the generator part

First, outline the part by giving it a reputation, description, and parameters. On this case, use salesdata_generator for each the title and the operate, with two parameters: what number of rows to generate and for which yr.

For the parameters, we outline them each as int, and you may add a regex validation to ensure the parameters supplied by the consumer are within the right format.

There are additional configuration choices obtainable; to be taught extra, consult with the AWS Glue Consumer Information.

That is how the part definition would appear to be. Put it aside as salesdata_generator.json. For comfort, we’ll match the title of the Python file, so it’s vital to decide on a reputation that doesn’t battle with an current Python module.
If the yr just isn’t specified, the script will default to final yr.

  "title": "salesdata_generator",
  "displayName": "Artificial Gross sales Knowledge Generator",
  "description": "Generate artificial order datasets for testing functions.",
  "functionName": "salesdata_generator",
  "parameters": [
      "name": "numSamples",
      "displayName": "Number of samples",
      "type": "int",
      "description": "Number of samples to generate"
      "name": "year",
      "displayName": "Year",
      "isOptional": true,
      "type": "int",
      "description": "Year for which generate data distributed randomly, by default last year",
      "validationRule": "^d{4}$",
      "validationMessage": "Please enter a valid year number"

Implement the generator logic

Now, you could create a Python script file with the implementation logic.
Save the next script as Discover the title is identical because the JSON, simply with a distinct extension.

from awsglue import DynamicFrame
import pyspark.sql.features as F
import datetime
import time

def salesdata_generator(self, numSamples, yr=None):
    if not yr:
        # Use final yr
        yr = - 1
    year_start_ts = int(time.mktime((yr,1,1,0,0,0,0,0,0)))
    year_end_ts = int(time.mktime((yr + 1,1,1,0,0,0,0,0,0)))
    ts_range = year_end_ts - year_start_ts
    departments = ["bargain", "checkout", "food hall", "sports", "menswear", "womenwear", "health and beauty", "home"]
    dep_array = F.array(*[F.lit(x) for x in departments])
    dep_randomizer = (F.spherical(F.rand() * (len(departments) -1))).solid("int")

    df = self.glue_ctx.sparkSession.vary(numSamples) 
      .withColumn("sale_date", F.from_unixtime(F.lit(year_start_ts) + F.rand() * ts_range)) 
      .withColumn("amount_dollars", F.spherical(F.rand() * 1000, 2)) 
      .withColumn("division", dep_array.getItem(dep_randomizer))  
    return DynamicFrame.fromDF(df, self.glue_ctx, "sales_synthetic_data")

DynamicFrame.salesdata_generator = salesdata_generator

The operate salesdata_generator within the script receives the supply DynamicFrame as “self”, and the parameters should match the definition within the JSON file. Discover the “yr” is an non-compulsory parameter, so it has assigned a default operate on name, which the operate detects and replaces with the earlier yr. The operate returns the reworked DynamicFrame. On this case, it’s not derived from the supply one, which is the frequent case, however changed by a brand new one.

The remodel leverages Spark features in addition to Python libraries as a way to implement this generator.
To maintain issues easy, this instance solely generates 4 columns, however we might do the identical for a lot of extra by both hardcoding values, assigning them from a listing, searching for another enter, or doing no matter is smart to make the information life like.

Deploy and utilizing the generator remodel

Now that we have now each information prepared, all we have now to do is add them on Amazon S3 beneath the next path.

s3://aws-glue-assets-<account id>-<area title>/transforms/

If AWS Glue has by no means been used within the account and Area, then that bucket won’t exist and must be created. AWS Glue will robotically create this bucket once you create your first job.

You’ll need to manually create a folder referred to as “transforms” in that bucket to add the information into.

After getting uploaded each information, the subsequent time we open (or refresh) the web page on AWS Glue Studio visible editor, the remodel needs to be listed among the many different transforms. You’ll be able to seek for it by title or description.

As a result of this can be a remodel and never a supply, after we attempt to use the part, the UI will demand a guardian node. You should utilize as a guardian the actual information supply (so you may simply take away the generator and use the actual information) or simply use a placeholder. I’ll present you the way:

  1. Go to the AWS Glue, and within the left menu, choose Jobs beneath AWS Glue Studio.
  2. Depart the default choices (Visible with a supply and goal and S3 supply and vacation spot), and select Create.
  3. Give the job a reputation by modifying Untitled job on the high left; for instance, CustomTransformsDemo
  4. Go to the Job particulars tab and choose a job with AWS Glue permissions because the IAM position. If no position is listed on the dropdown, then observe these directions to create one.
    For this demo, you too can scale back Requested variety of employees to 2 and Variety of retries to 0 to attenuate prices.
  5. Delete the Knowledge goal node S3 bucket on the backside of the graph by choosing it and selecting Take away. We are going to restore it later after we want it.
  6. Edit the S3 supply node by choosing it within the Knowledge supply properties tab and choosing supply sort S3 location.
    Within the S3 URL field, enter a path that doesn’t exist on a bucket the position chosen can entry, for example: s3://aws-glue-assets-<account id>-<area title>/file_that_doesnt_exist. Discover there is no such thing as a trailing slash.
    Select JSON as the information format with default settings; it doesn’t matter.
    You may get a warning that it can not infer schema as a result of the file doesn’t exist; that’s OK, we don’t want it.
  7. Now seek for the remodel by typing “artificial” within the search field of transforms. As soon as the consequence seems (otherwise you scroll and search it on the checklist), select it so it’s added to the job.
  8. Set the guardian of the remodel simply added to be S3 bucket supply within the Node properties tab. Then for the ApplyMapping node, change the guardian S3 bucket with transforms Artificial Gross sales Knowledge Generator. Discover this lengthy title is coming from the displayName outlined within the JSON file uploaded earlier than.
  9. After these adjustments, your job diagram ought to look as follows (in case you tried to avoid wasting, there is perhaps some warnings; that’s OK, we’ll full the configuration subsequent).
  10. Choose the Artificial Gross sales node and go to the Rework tab. Enter 10000 because the variety of samples and depart the yr by default, so it makes use of final yr.
  11. Now we’d like the generated schema to be utilized. This could be wanted if we had a supply that matches the generator schema.
    In the identical node, choose the tab Knowledge preview and begin a session. As soon as it’s working, you need to see pattern artificial information. Discover the sale dates are randomly distributed throughout the yr.
  12. Now choose the tab Output schema and select Use datapreview schema That approach, the 4 fields generated by the node will probably be propagated, and we are able to do the mapping based mostly on this schema.
  13. Now we wish to convert the generated sale_date timestamp right into a date column, so we are able to use it to partition the output by day. Choose the node ApplyMapping within the Rework tab. For the sale_date discipline, choose date because the goal sort. This may truncate the timestamp to only the date.
  14. Now it’s a very good time to avoid wasting the job. It ought to allow you to save efficiently.

Lastly, we have to configure the sink. Observe these steps:

  1. With the ApplyMapping node chosen, go to the Goal dropdown and select Amazon S3. The sink will probably be added to the ApplyMapping node. Should you didn’t choose the guardian node earlier than including the sink, you may nonetheless set it within the Node particulars tab of the sink.
  2. Create an S3 bucket in the identical Area as the place the job will run. We’ll use it to retailer the output information, so we are able to clear up simply on the finish. Should you create it by way of the console, the default bucket config is OK.
    You’ll be able to learn extra details about bucket creation on the Amazon S3 documentation 
  3. Within the Knowledge goal properties tab, enter in S3 Goal Location the URL of the bucket and a few path and a trailing slash, for example: s3://<your output bucket right here>/output/
    Depart the remaining with the default values supplied.
  4. Select Add partition key on the backside and choose the sector sale_date.

We might create a partitioned desk on the similar time simply by choosing the corresponding catalog replace choice. For simplicity, generate the partitioned information presently with out updating the catalog, which is the default choice.

Now you can save after which run the job.

As soon as the job has accomplished, after a few minutes (you may confirm this within the Runs tab), discover the S3 goal location entered above. You should utilize the Amazon S3 console or the AWS CLI. You will notice information named like this: s3://<your output bucket right here>/output/sale_date=<some date yyyy-mm-dd>/<filename>.

Should you rely the information, there needs to be near however no more than 1,460 (relying on the yr used and assuming you’re utilizing 2 G.1X employees and AWS Glue model 3.0)

Use case: Enhance the information partitioning

Within the earlier part, you created a job utilizing a customized visible part that produced artificial information, did a small transformation on the date, and saved it partitioned on S3 by day.

You is perhaps questioning why this job generated so many information for the artificial information. This isn’t preferrred, particularly when they’re as small as on this case. If this information was saved as a desk with years of historical past, producing small information has a detrimental impression on instruments that eat it, like Amazon Athena.

The rationale for that is that when the generator calls the “vary” operate in Apache Spark with out specifying a variety of reminiscence partitions (discover they’re a distinct sort from the output partitions saved to S3), it defaults to the variety of cores within the cluster, which on this instance is simply 4.

As a result of the dates are random, every reminiscence partition is more likely to comprise rows representing all days of the yr, so when the sink wants to separate the dates into output directories to group the information, every reminiscence partition must create one file for every day current, so you may have 4 * 365 (not in a bissextile year) is 1,460.

This instance is a bit excessive, and usually information learn from the supply just isn’t so unfold over time. The problem can typically be discovered once you add different dimensions, akin to output partition columns.

Now you will construct a part that optimizes this, attempting to cut back the variety of output information as a lot as potential: one per output listing.
Additionally, let’s think about that in your crew, you’ve gotten the coverage of producing S3 date partition separated by yr, month, and day as strings, so the information might be chosen effectively whether or not utilizing a desk on high or not.

We don’t need particular person customers to should take care of these optimizations and conventions individually however as an alternative have a part they’ll simply add to their jobs.

Outline the repartitioner remodel

For this new remodel, create a separate JSON file, let’s name it repartition_date.json, the place we outline the brand new remodel and the parameters it wants.

  "title": "repartition_date",
  "displayName": "Repartition by date",
  "description": "Break up a date into partition columns and reorganize the information to avoid wasting them as partitions.",
  "functionName": "repartition_date",
  "parameters": [
      "name": "dateCol",
      "displayName": "Date column",
      "type": "str",
      "description": "Column with the date to split into year, month and day partitions. The column won't be removed"
      "name": "partitionCols",
      "displayName": "Partition columns",
      "type": "str",
      "isOptional": true,
      "description": "In addition to the year, month and day, you can specify additional columns to partition by, separated by commas"
      "name": "numPartitionsExpected",
      "displayName": "Number partitions expected",
      "isOptional": true,
      "type": "int",
      "description": "The number of partition column value combinations expected, if not specified the system will calculate it."

Implement the remodel logic

The script splits the date into a number of columns with main zeros after which reorganizes the information in reminiscence in accordance with the output partitions. Save the code in a file named

from awsglue import DynamicFrame
import pyspark.sql.features as F

def repartition_date(self, dateCol, partitionCols="", numPartitionsExpected=None):
    partition_list = partitionCols.break up(",") if partitionCols else []
    partition_list += ["year", "month", "day"]
    date_col = F.col(dateCol)
    df = self.toDF()
      .withColumn("yr", F.yr(date_col).solid("string"))
      .withColumn("month", F.format_string("%02d", F.month(date_col)))
      .withColumn("day", F.format_string("%02d", F.dayofmonth(date_col)))
    if not numPartitionsExpected:
        numPartitionsExpected = df.selectExpr(f"COUNT(DISTINCT {','.be part of(partition_list)})").accumulate()[0][0]
    # Reorganize the information so the partitions in reminiscence are aligned when the file partitioning on s3
    # So every partition has the information for a mix of partition column values
    df = df.repartition(numPartitionsExpected, partition_list)    
    return DynamicFrame.fromDF(df, self.glue_ctx, self.title)

DynamicFrame.repartition_date = repartition_date

Add the 2 new information onto the S3 transforms folder such as you did for the earlier remodel.

Deploy and use the generator remodel

Now edit the job to utilize the brand new part to generate a distinct output.
Refresh the web page within the browser if the brand new remodel just isn’t listed.

  1. Choose the generator remodel and from the transforms dropdown, discover Repartition by date and select it; it needs to be added as a toddler of the generator.
    Now change the guardian of the Knowledge goal node to the brand new node added and take away the ApplyMapping; we now not want it.
  2. Repartition by date wants you to enter the column that incorporates the timestamp.
    Enter sale_date (the framework doesn’t but permit discipline choice utilizing a dropdown) and depart the opposite two as defaults.
  3. Now we have to replace the output schema with the brand new date break up fields. To take action, use the Knowledge preview tab to examine it’s working accurately (or begin a session if the earlier one has expired). Then within the Output schema, select Use datapreview schema so the brand new fields get added. Discover the remodel doesn’t take away the unique column, nevertheless it might in case you change it to take action.
  4. Lastly, edit the S3 goal to enter a distinct location so the folders don’t combine with the earlier run, and it’s simpler to check and use. Change the trail to /output2/.
    Take away the present partition column and as an alternative add yr, month, and day.

Save and run the job. After one or two minutes, as soon as it completes, study the output information. They need to be a lot nearer to the optimum variety of one per day, possibly two. Think about that on this instance, we solely have 4 partitions. In an actual dataset, the variety of information with out this repartitioning would explode very simply.
Additionally, now the trail follows the standard date partition construction, for example: output2/yr=2021/month=09/day=01/run-AmazonS3_node1669816624410-4-part-r-00292

Discover that on the finish of the file title is the partition quantity. Whereas we now have extra partitions, we have now fewer output information as a result of the information is organized in reminiscence extra aligned with the specified output.

The repartition remodel has extra configuration choices that we have now left empty. Now you can go forward and check out completely different values and see how they have an effect on the output.
As an example, you may specify “division ” as “Partition columns” within the remodel after which add it within the sink partition column checklist. Or you may enter a “Variety of partitions anticipated” and see the way it impacts the runtime (it now not wants to find out this at runtime) and the variety of information produced as you enter the next quantity, for example, 3,000.

How this function works beneath the hood

  1. Upon loading the AWS Glue Studio visible job authoring web page, all of your transforms saved within the aforementioned S3 bucket will probably be loaded within the UI. AWS Glue Studio will parse the JSON definition file to show remodel metadata akin to title, description, and checklist of parameters.
  2. As soon as the consumer is completed creating and saving his job utilizing customized visible transforms, AWS Glue Studio will generate the job script and replace the Python library path (additionally referred as —extra-py-files job parameters) with the checklist of remodel Python file S3 paths, separated by comma.
  3. Earlier than working your script, AWS Glue will add all file paths saved within the —extra-py-files job parameters to the Python path, permitting your script to run all customized visible remodel features you outlined.


With the intention to keep away from working prices, in case you don’t wish to hold the generated information, you may empty and delete the output bucket created for this demo. You may additionally wish to delete the AWS Glue job created.


On this put up, you’ve gotten seen how one can create your personal reusable visible transforms after which use them in AWS Glue Studio to reinforce your jobs and your crew’s productiveness.

You first created a part to make use of synthetically generated information on demand after which one other remodel to optimize the information for partitioning on Amazon S3.

Concerning the authors

Gonzalo Herreros is a Senior Massive Knowledge Architect on the AWS Glue crew.

Michael Benattar is a Senior Software program Engineer on the AWS Glue Studio crew. He has led the design and implementation of the customized visible remodel function.

By admin

Leave a Reply

Your email address will not be published. Required fields are marked *