Processing Twitter JSON Data

Open the DMExpress Job Editor on a Windows workstation, click on File->Save Job As… to create a new job, name it TweetSorter.dxj, and save it to the desired folder on the Windows file system.

In this example, the JSON Reader takes the JSON file tweets.txt as its source, and writes the CSV output directly to stdout. Create the JSON Reader task to perform the JSON to CSV conversion as follows:

1. Select Edit ->Add JSON Reader… to create the task.
2. In the Source pane, select File path and enter the path to the file tweets.txt.
3. In the Target pane, select Standard output.
4. In the Target layout pane, generate a delimited record layout corresponding to the source layout and save it to a new DMExpress metadata file:
   1. Enter metadata.dxt in the Link to DMExpress metadata file field.
   2. Enter tweetLayout in the Map input fields to delimited layout field.
   3. Click on Create layout by scannping input file.

   This layout will be linked in as external metadata in the subsequent task. The populated window should look similar to this:

   

5. Click OK, then click to place the task on the job canvas.

You can simply add the provided SortTweet.dxt task to the job flow, or you can create the task yourself as follows:

1. Select Edit->Add New DMExpress Task... to create a task, name it SortTweet.dxt, and save it to the same folder as the job. Click to place the task on the canvas to the right of the JSON Reader task.
2. Link in the record layout generated by the JSON Reader as external metadata:
   1. Select Metadata->External->Link….
   2. Click Browse…, navigate to the metadata file created previously, and click OK.
   3. Ensure the record layout generated by the JSON Reader is selected, and click OK.
3. Add the JSON source:
   1. Select Add->Source->Pipe….
   2. Select Standard input for the Type of pipe.
   3. Ensure the record layout generated from the JSON file is associated with this source, and click OK.
4. Change the task type to Sort and define the Sort key to be <recordlayout>.id.
5. Add a new file target in a directory designated as $LOCAL_TARGET_DIR, which you will define later, and create a reformat which preserves the desired fields.

Once the task has been defined, click on the Sequence toolbar button and draw an arrow from the JSON Reader task to the SortTweet.dxt task as shown below:

Define the environment variable $LOCAL_TARGET_DIR to point to a location to store the sorted Twitter data. For example:

Run the job from the command line by navigating to the directory containing the job, and entering: